Created

I have been wondering about information distribution, feeling inefficient.
This page reflects on how to build a seemless, integrated information structure for it, and this repo is where it's built.

Towards spirit stream

More refined

Spirit Stream

When I follow curiosity, leaning into my wonder about computer and internet, I sense dissolution: The edges of my body fade and I float through the associations between all possible spaces. Into the space, through trees, stories, videogames, temples, beauty, destruction. I see my questions answered, not definitively, but as completely as the collective mind can. I sense connection, trusting and deep. I sense the conflicts in the world as internal, not as foggy, distant pain. There are no external barriers to my being and I stream into others to see exactly what they mean. And in my action, the structure of the world, not merely my individual life, is reflected.
Returning to my body, I can't help but wonder what the fuck this is: Me, stopping with my fingertips, not being able to walk through walls, facing the endless world in mist asking hello with no answer.

Floating head, slightly tilted. Partly shaded, mask-like. Reminiscent of a skull. Dark background and dark lines across the white head, partly messy, decorative and minimally defining facial features. Closer eye is white, farther eye is black. Red-magenta lines overlay the skull, partly straight and splitting the image, partly squiggly.
Spirit of curiosity
i can see the hive mind on the horizon.
there are hammers, sculptors, hammer blows and sculptures.
grouped and isolated
places to stay.

like gazing into a painting, being guided through the impression
coming along to find a question asked at myself, seeing it unfold and building a response

There seems to be a structure behind notes, memes, essays, books, stories, exhibitions, spaces, products, worlds. They consists of simple patterns in the mind their process might be reflected like this:

Sense, explore, parse circling patterns from
chaotic and far-reaching,
through filtered and coherent knowledge,
to linearized, actionable and accessible stories.

Express in sound, body,
writing, graphs, images, products,
and any tools to build them,
in process,
from memory.

Share in truth,
to test, be surprised, dance, reproduce, join the cutting edge, offer, cohere, support, punish,
into common languages per context,
into scopes of recipients,
under optional conditions.

Where I am,
When I am.

If the process can be reflected in the machine, the patterns and me with them can seamlessly enter the machine and then its abilities...

Execute machine code within instruction sets,
functions, programs, function approximators compiled from many languages,
from and to memory.

Store, retrieve anything, persistently, quickly

Express in any electric actuator, speaker, display, motor.
Receiving from any electric sensor, keyboard, mouse, camera, microphone

Exchange anything at up to ~storage speeds,
to any available address,
through ports, through fundamentally unreliable, insecure connections,
made reliable, secure through protocol and server

...become my own. The spiritstream aims to be the passage into the machine.

How to merge?

Since I construct meaning from the individual, I view all higher structures of power (groups, communes, cities, states, religions) as services to the individual. Build free, accessible, independent, private, no login, no lock in.

The flow of patterns in the mind precedes common language and translating them shrinks and distorts them. A direct neural interface may simplify the process. An AI could interpret the stream and seamlessly induce useful patterns. Should I now try and build a neural interface?
I suspect not. The patterns are repetitive and I have seen myself expressed in words almost entirely. Simple words or images can speak so loudly and clearly, throwing open the doors to the world with their precision and closeness. Finally, the problem of organizing symbols - patterns or words - remains. The neural interface may follow.
When the interface becomes seamless and reflective of the user,
when the spirits of computer and internet reveal themselves,
when I speak with clarity and the extension answers with clarity - we merge.

Create, reuse, organize and share arbitrary symbols in 3D space, which is the most dimensions I can understand and use intuitively. Links to other spaces enable arbitrarily many dimensions.
Remember a continuous stream of actions to provide a seamless safety net and past data to learn from. It encourages revisiting instead of starting anew, improving quality, and allowing fearless, even reckless editing.

Eventually, the spiritstream becomes the primary interface to the computer, an operating system. Motivation for depth and vertical integration is meet the computer and let higher structures fall into their place. Assume hardware is good + it's expensive to build -> don't go into hardware until software is insanely great.
Initially, it's a desktop client using python, OpenGL and GLFW to refine abstractions before setting them into C and becoming standalone.

Synchronizes with untrusted remote servers. Uses end-to-end encryption for non-public data. Public data is available in the web view.

To discover and be discovered, search engines, LLM pretraining or dynamic research through an LLM seem sufficient.
Algorithms and humans have equal capability when interacting with the spiritstream: A local or remote server handles all file modifications, while a trusted client is the interface.

To be become scary, serious and more real, therefore adventurous, payment must be supported. And later an interface to robots.

Current process for publishing:

Writing in pure html, css, javascript: VS Code, prerendering to static sites locally (table of contents): node.js
keeping history in a git repository,
Synchronizing with remote server over SFTP using python paramiko: Hetzner VPS: OpenSSH, nginx (HTTP, HTTPS)
Discoverable through Cloudlfare DNS.

Simplified specification

Data Model

At the bottom are text nodes. They are grouped into nodes for UI and for formal semantic differentiation. Scenes (worlds) are potentially cyclic node graphs. The UI of choice for differentiating nodes is markdown. It offers basic node types like paragraphs, headings, links. For more and custom node types, use html syntax.

fuzzy search entire node text or search by links that must overlap.

Links can either allow me to GO to or SHOW (embed) their target. Valid targets are headings, files or URLs. Only the content of headings or image files can be SHOWn. Files that can't be opened natively, use the system default to open them and are available to be opened by other software.
BACK and FORWARD buttons let me step through navigation history after following links.
Pasting a file produces a GO link to that file or a SHOW link if it's an image file. Links can point to void targets which can help with search.
Delete is the same as removing all references to a piece of data, gives warning when deleting last reference.

The appearance of nodes is defined with CSS syntax. The top level nodes (scenes) have infinite dimensions, while all other ones are limited and must be contained in their respective parent node.

Changes in the node graph are recorded down to 1 second resolution and any past state can be seamlessly revisited.

Individual nodes can be opened with external software, exported as standalone files and shared to recipients from various backends (web server, email, social networks, SMS...). Changes can be set to stream to available recipients or be sent on save. The recipient can access through the spirit stream software or see a high compatibility version that may have limited features.
Similarly, messages can be set to be received from the same backends and placed into nodes as defined through a template.
Streamed volume might become economically unviable and require bandwidth and access control.

UI, API

All UI is created with the same tools as are available to the user for creating their sites.

All actions are available both through UI to the user and API to external programs and in an AI-friendly format. Connecting to an already opened instance or creating a new one with or without showing UI is trivial through any terminal.

By default all content set to be public is available, seeing the rest requires authentication.

Window displays a portion of 2D SCENE where I can pan using SPACE+DRAG and zoom using MOUSEWHEEL. Nodes are positioned in it, using their style information. Nodes can be selected using CLICK or DRAG to select any in an areaor SHIFT+CLICK to select / unselect while keeping the current selection. Selected nodes can be moved using DRAG. If I select using CLICK and nodes overlap, the nearest, smallest node is selected. DOUBLECLICK a node places my cursor in it near the mouse position. All visible nodes are text nodes. They are differentiated using MARKDOWN and styled using CSS. DOUBLECLICK outside of node creates a new node in that place. DOUBLECLICK also activates EDIT MODE, where comments and raw markdown of selected nodes is visible. Selecting no node equals PREVIEW MODE, where comments are not rendered. A button is available to switch to SOURCE MODE to see raw markdown for all nodes.
Delete selection using BACKSPACE or DELETE.
CTRL+A selects all in current node or all nodes in current scene if no node selected. CTRL+A again selects all nodes in scene or all scenes with all nodes.
CLICK outside any node or press ESC to remove selection.
Reopening the program returns me to where I left off.

editor supports hyphenation, jpg, gif, png (LATEX), code highlighting in python, glsl, css.

METAINFO for any node can be viewed in a toggleable sidepanel. This includes editable styling and/or a link to the CSS file, file metadata, sharing status and history.
HISTORY is a graph with present at the top and past below, including branches. Time where nothing in the selected nodes was changed is compressed and commented with a date. LABELS can be added into the otherwise continous history graph and can be visited directly. They act like links. (Triple press CTRL+A to select all shows complete history in metainfo)

Select any node or Scene to share. To see what is and isn't shared, I either view METAINFO or view the content from any recipient's perspective via the VIEW menu. Selecting a different perspective also changes HOME to the recipient's home. Upon saving, shared data is automatically updated for its recipients.

Less refined

Implementation

Editor

Create and place symbols into sequences
use markdown interface for common formatting and differentiating semantic blocks
Blocks are styled using CSS
position into 3D space (x, y, z, rx, ry, rz)

Everything is stored in a piecechain without a starting piece, only big append buffer. It stores the serialized node tree, which the render tree is built from. When assembled, it's a JSON file. 

{
    "scenes": [
        {
            "id": ... // optional, target for file links
            "name": "style.css",
            "nodes": [
                {
                    "name": 
                }
            ]
        },
        {
            "name": "inbox.html",
            "data": ...,
        }
    ]
}

The current pieces are stored as a csv of index ranges into the append buffer, encoded using Base64 to save space while keeping it a text file: 

0,5
10,40
...

Clicking on nodes always selects the first deepest node in the tree. This makes layers a direct translation of the tree. It's also inverted, so the deepest nodes are the top layers.
Hold space and drag to move. Shift select other frames too.
EDIT MODE: Double click to edit, ESC to exit. Scroll in all directions. Shift scroll for horizontal scrolling, ctrl+scroll for zooming.

Views:
Strict WYSIWYG necessitates a preview, without any UI hints.
Edit (Markdown mode): view formatting syntax only for selected nodes, comments are visible.
Source: full formatting syntax, html.
3 Views: how to reduce?

Tree of nodes.
Node(x, y, z, rx, ry, rz, w, h, style)
Scene(title, camera, children). Camera (x, y, z, rx, ry, rz, fov)

Links:
may be GO or SHOW, either displaying a "door" to the data or embedding the data in its place.
Links reference piece indices. This requires updating all links when pieces are inserted before their target. SHOW links have start and end node and display everything inbetween. If either node is deleted, notify user / show hints for backlinks next to start pieces / for cursor position. SHOW links require META interaction to change the linked content but show no difference in preview mode.

Log

Continuously store individual, reversible changes in the current pieces. Occasionally, check if pieces can be merged or if the overlap to reduce size.

PIECEINDEX, START, LENGTH and if something is unchanged, leave empty, like ,,10 changes end of current piece to 10. changing start and end to 0 means it was deleted.

Make it reversible and add timestamp: TIMESTAMP,OLDPIECEINDEX,NEWPIECEINDEX,OLDSTART,NEWSTART,OLDLENGTH,NEWLENGTH

Uncompressed with Base64 the size would be:
TIMESTAMP: 31 bits -> 6B
PIECEINDICES: 16 bits if max 65536 pieces -> 3B each
START / LENGTH: if max length up to 1TB (40 bits) -> 7B each
COMMA / NEWLINE: 1B each
makes for 47B for each change that uses all fields. 14B if changing only START / LENGTH.
makes max ~4MB for uncompressed data after 24 hours of one maximum data change each second

version number for extendability.

To reduce size: Create new append buffer that consists of sections of the original one. empty at first and extended with new range every time a piece does not fit in the existing one. for each piece in history, try to fit it inside the existing append buffer while not including its current position. if that fails, try overlapping them. if that fails too, use current position. Repat until no savings.

Pieces -> append buffer
a, bad -> bad
a, bad, addendum -> baddendum

Integrity check: Rerun history against stored states. Separate and hash append buffer and log files above certain size.

file structure:

The log can store write, erase, seek (time/file+cursor) and meta (non-essential)

Upon loading, the log checks integrity and adjusts to any unrecorded changes:
- get latest view according to log file
- get difference to home files
- log events to catch up with home files.

class Log:
    def __init__(self, filepath):
        """Initialize log with the given file path."""

    def write(self, text, cursor=None, author=None):
        """Insert text at cursor."""

    def erase(self, count, cursor=None, author=None):
        """Remove count characters at cursor."""

    def save(self):
        """Save changes to the log."""
    
    def view(self, date):
        """Return the state as of the given date, without altering current state."""
    
    # internal methods
    def _seek(self, position):
        """Record a seek event if cursor position changes."""

    def _meta(self, **kwargs):
        """Record a meta event storing keys and values like user:alice or comment:"hamedi is coming for you" """

    def _elapsed(self, seconds):
        """Record elapsed time since last event."""

File system

Privacy map informs client what to en-/decrypt and server what to send. Akin to datatypes.
determines which piece indices / range require authentication / decryption. Necessitates splitting pieces at privacy rule perimeters. A privacy rule is a WHITE or BLACK list. Empty blacklist = public. Empty whitelist = private. Should be able to reuse privacy groups.

To maintain links, the offset they point to must be updated. To minimize updates, organize in a binary tree with offsets on each node and relative to parent node. Bytes at the bottom. Can either point to (GO) or include (SHOW) piece index / range. Any display is SHOW links. Any UI is SHOW links. Can contain nested SHOW links. Editing content necessitates visiting the original. My location is defined as where my edits write directly to file, no META.
SHOW links just show content somewhere else. Linking to it, formatting it as a quote is optional, must be explicit.

Server-Client

Client:

Server

other

Karpathys ideal blogging platform
file over app

"lazy payment" - unrealized costs become realized when someone pays transaction costs? optionally anonymously.
payment + PAYG
tor
interplanetary file system

discord
plausible analytics
atom/rss

"Digital gardens"
https://github.com/MaggieAppleton/digital-gardeners
https://simonewebdesign.it/

markdeep
libh2o PowerDNS.org with nice articles

Maybe merging physical and virtual world means replicating my room in the computer and the computer in the room. Most of the room is static -> low network load; focus on moving and important parts instead: Faces. Computer should compress data by "scanning" environment and face and storing/transmitting transformations to save space.

text with properties: x, y, font, width
current font is in global state
text(x,y width, height, text) places that text box

scene:

html is hierarchy for search engines and relative positioning with css.
llms like markdown, which is linear like html.
LLMs still struggle with 3D understanding.

assinging a style to a box is a meta op because it goes there and copies that style. default is always I just edit my style here. Editing the "original" is also meta op. Assign stlye like in browser debugger.
CTRL+CLICK to follow a link, else edit markdown for that link.

The process of language from far-reaching to linearized stories is analagous to the the process from abstract and ambiguous memory towards linear, factual history.
But history is told in language too. but there are ways to record direct experience of history. no such way to record thoughts? yet?
machines can probably translate brain signals into languages and back into experience.
In computers, all abstractions exist for efficiency, so there is very little given that I must build on.
it is less obvious what to build on in interaction with the mind? language? it is not the bottom: my experience exists in higher resolution and volume than what I can reproduce in language.
most experience quickly becomes irrelevant. Would arguably be rather burdensome if recorded. How to efficiently and accurately encode discoveries in the mind. thinking is a compression algorithm. Most attempts are shit, the remaining should be expressed in high quality to save others from doing the same.
problems are lack of introspection about why something appears meaningful or useful and lack of empathy for other people, the student, who does not understand the news.

it seems that the communication problem is solved.
no, the properties of the computer are still useful, like perfect memory, communication speed, processing autonomous processing. but the interface is so slow and tedious that it is not worth it to put most of my mind into it because I have to manage it too.
at some point it might interpret brain signals and aid them directly. an obvious problem here is that I don't understand most quotes and can not learn from them. I am not attracted to books because its hundreds of pages of nothing to use or remember. what to do with this?
the ideal publishing platform does not exist. there is no bottom up independent easy publishing app. there is obsidian but it can't contain worlds. Narrow and ugly under the hood?