Zubra docs

Sync: component state that lives on the device#

'use collection <name>' at the top of a component's script keeps its state in a local store. It needs @zubra/sync installed (npm install @zubra/sync). With a bundler, that's all. With @zubra/server, too: it finds the package, serves it, and adds it to the page's import map, so import { connect } from '@zubra/sync/connect' works in a page as written. If it isn't installed, the build or the page says so, naming the component and the command to run.

<!-- task-item.zb -->
<script>
  'use collection tasks';
  import { ephemeral } from '@zubra/sync';

  let completed = false;            // a field of the document
  let title = 'Untitled';           // another
  let editing = ephemeral(false);   // state, but not stored
  const label = completed ? 'done' : 'todo';   // derived: not stored
</script>

<task-item doc="">
  <input type="checkbox" checked={completed} onchange={(event) => (completed = event.target.checked)} />
  {title} <small>{label}</small>
</task-item>
<task-item doc="t42"></task-item>   <!-- the document tasks/t42 -->

examples/standup is a whole app built this way: a plain Node server, a SQLite log, offline edits and presence, tested with two browsers.

What the compiler writes#

One line at the end of the element's setup, and one import:

import { collection as $collection } from "@zubra/sync";
…
$collection("tasks", ["doc"], $p, { completed, title });

The fields are the script's top-level lets, minus the ones wrapped in ephemeral(). Derived consts, props and functions aren't stored.

The rules#

  • The key is props. doc by default; 'use collection rows by list, item' keys by two props (list/item). A key prop the host doesn't declare is a compile error. It isn't id: that's the element's own DOM id, which must be unique on the page, and two elements may show the same document.
  • A write is an assignment. completed = true, tags.push('x'): the DOM updates now, the store a moment later. Nothing waits on a disk or a network.
  • Loading never overwrites, and nothing typed is lost. Until an element's document has loaded, it shows the defaults, so the defaults never replace stored state. A field someone changed while it loaded keeps their change, which is then saved; the others take the stored values.
  • Values are JSON. Strings, numbers, booleans, null, arrays and plain objects (nested changes included). A Date is stored as its ISO string.
  • Same document, same state. Elements showing one document follow each other, in the page and in other tabs of the same site (BroadcastChannel).
  • Merging is last-writer-wins, per field. Two tabs writing the same field: the later write is the value. Different fields merge.
  • A document can hold fields a component doesn't declare. The server (or another component on the same collection) may write a field with no let here, such as a hold's since. It's kept on the device and in the log, and this component's edits push only the fields it changed, so they never overwrite or drop it.
  • The server renders the defaults (and an ephemeral() value, which it can evaluate); the element loads its document once it starts.

Stores#

The browser default is IndexedDB, in a database called zubra-sync, one record per document. Where there's no IndexedDB, a memory store. Another store is two functions:

import { configure, memoryStore } from '@zubra/sync';
configure({ store: { get: async (collection, key) => doc, set: async (collection, key, fields) => {} } });

Sharing between devices#

Tier 0 keeps a document on one device. To share it, start a connection on the page and serve the log:

// in the page
import { connect } from '@zubra/sync/connect';
connect({ url: '/_zb/sync' });
// on the server (a fetch handler: Node through a small adapter, Bun, Deno, workers)
import { createSyncServer } from '@zubra/sync/server';
const sync = createSyncServer({ authorize: ({ request, collection, key, field, kind }) => kind === 'read' || isSignedIn(request) });
const response = await sync.fetch(request); // null if it isn't a sync request

A page that only reads#

A page whose server writes the documents and never lets a page write (a menu with live stock, an order's status) can follow the log instead:

import { follow } from '@zubra/sync/follow';
follow(); // the same url, interval, live and scope options as connect()

It pulls and listens exactly as connect() does, and leaves out what only a writing page uses: the outbox, pushes, presence and peers. That is about 1.4 KB less to download (3.5 KB against 4.9, compressed). An edit made on such a page stays on that device, and the first one is reported in the console; use connect() on any page that writes. The server is the same either way, and authorize still decides what the page may read.

Who may read and write#

Without authorize, pages may neither read nor write. Pulls and live streams are refused with a 403, as are pushes, presence and signals; connect() and follow() report a refused pull once in the console. When an explicit authorize policy hides every change in a collection, the development server warns once and says how to open it. The server's own write() and documents() are never asked. This is deliberate: a log that takes any change from anyone lets a visitor set a price, mark an order paid or take every seat, and one that hands every page every document shows a visitor what was never on their screen. Before 6.0 reads were open by default; see the 6.0 guide.

Collections anyone may read and nobody may write (a menu, an order board the server writes) are named:

import { createSyncServer, allowReads } from '@zubra/sync/server';
createSyncServer({ authorize: allowReads(['menu', 'stock']) });   // allowReads() for every collection

Anything more, say who may see and change what:

createSyncServer({
  authorize: ({ request, collection, field, kind }) => {
    if (kind === 'read') return true;                       // or check, for private documents
    if (collection === 'payments' || field === 'price') return false;   // the server's alone
    return isSignedIn(request);
  },
});

field and value let an app keep fields, or whole collections, for the server: a page may move a card but not change what it costs. A push is all or nothing, so one refused change refuses the push.

A page on another site is let in only if it's listed: createSyncServer({ origins: ['https://shop.example'] }). Its writes are taken (a page on any other site is refused, as a browser says where a request comes from), and its answers carry CORS headers, so its browser lets it read its pulls and its live stream and send JSON at all. That's the same as the zubra-sync binary's --origin. authorize still decides what it may read and change. Cookies aren't sent across sites, so a page on another site is anonymous to authorize unless the app knows it some other way.

For public reads without page writes, use createSyncServer({ authorize: ({ write }) => !write }). For a page everyone may edit (a shared board on a trusted network, a demo), say so: createSyncServer({ authorize: allowAnyone }), which opens reading and writing everything. A Durable Object takes no reads or writes from pages unless its class has an authorize() (or options.authorize), and the zubra-sync binary reads nothing unless it's started with --read <collection>, --read-all or --open, and takes writes only with --open.

Components don't change. The protocol is plain HTTP, so any server can keep the log:

POST /_zb/sync/push { "changes": [Change] } → { "ok": true }
GET /_zb/sync/pull?since=<cursor>[&scope=<json>] { "changes": [Change], "cursor": … }
Change { "collection", "key", "field", "value", "clock" }
  • The device is the source of truth. A write lands in the local store first. The connection stamps each changed field with a hybrid logical clock (<ms>-<counter>-<device>), queues it (in localStorage, so a reload or a closed tab doesn't lose it) and pushes it when it can. Offline, it waits, for hours if need be: the queue keeps one change per field (a newer one replaces what's waiting, since this device's later write wins anyway), so a long session of typing waits as one change per field edited, not one per keystroke. Whatever another device wrote later still wins field by field when it gets through. And what this device wrote later always wins on this device: an older change that arrives while an edit is still being stored, or after the tab closed with writes queued, doesn't replace it.
  • A request is small. A push, presence update or signal is at most maxBytes (default 1 MiB, the same as the zubra-sync binary), and a push carries at most maxChanges changes (default 1000); past either the server answers 413 without asking authorize. A body that declares a larger length isn't read at all. connect() sends its outbox in batches under both limits (500 changes, about 512 KB), so a device back from hours offline still gets everything through.
  • A clock can't run far ahead. A push whose clock is more than maxSkew (default 60 s) ahead of the server's is refused with a 400, since a change stamped far in the future would win every merge for ever. A device a little fast is fine, and the server's own writes still come after it. Devices don't follow a far-future stamp they hear either, and a clock's counter carries into the next millisecond rather than outgrowing its five digits.
  • Two field names are taken: $clocks (where a document keeps its clocks) and key (a listed document's own key). A push with either is refused, and write() throws.
  • The server keeps a log, in order, and doesn't merge. Every device applies the same rule to the same changes: for each field, the change with the later clock wins. So they converge, whichever order the changes arrive in. The clock of each field's winning write is stored beside the document ($clocks).
  • authorize decides who may write a document and who sees its changes. It's told what's asked: kind is 'change' (a push), 'presence' (joining a document's room; nothing is stored) or 'read' (a change about to be sent to this request), and write is true for the first two. A 'change' also has the field and value it sets. Without it pages read and write nothing. See Who may read and write.
  • The log is in memory by default. sqliteLog(path) from @zubra/sync/sqlite keeps it in a SQLite file through the runtime's own SQLite (node:sqlite on Node 22.5+, bun:sqlite on Bun), so it survives a restart with nothing to install. Any { append(changes), after(cursor, limit) } works too.
  • A log that doesn't grow with every change: sqliteLog(path, { keep: 'latest' }) (and memoryLog, durableObjectLog, with the same option) keeps each field's latest value only. A change with a newer clock replaces the field's row and takes the next cursor; one with an older clock is dropped. A new device reads what's there now, not the evening's history of every seat held and released. Devices merge the same either way: what's dropped would have lost to the newer write on every device anyway. Opening a file that has the full log as keep: 'latest' carries each field's latest value across, numbered after the old cursors, so devices that had read the old log miss nothing. The zubra-sync binary keeps every change by default; --keep-latest uses the same latest-only table. The original full log remains in the file after migration, and future starts must use the same mode.
  • Cross-site pushes are refused, like actions.

Fresh first. On a connected page, a component waits for the connection's first pull before it applies what this device stored last time, so a returning visitor never sees last visit's state over what the log (or the page the server just rendered) says now. Until then it shows what it was rendered with. Offline, the pull fails at once and the stored document shows straight away; a pull that doesn't answer holds it back for 1.5 seconds at most. Call connect() in the page's own scripts (they all run before DOMContentLoaded, which is what a component waits for before looking).

A cursor past the end of the log starts over. If a device asks for changes after a cursor the log has never reached (the log was replaced, or wiped, or the device's cursor came from another one), pull and the live stream answer with everything in its scope from the beginning, with "reset": true, and the device takes the new cursor. Without this, it would hear nothing until the new log grew past its old cursor. Logs say where they end with head(); the built-in ones and the zubra-sync binary do. What a device already has with newer clocks than the new log's stays, by the same last-writer-wins rule.

Changes arrive over a live channel: GET /_zb/sync/live?since=<cursor> is a server-sent event stream of changes as they're pushed (and presence, below). While it's open there's nothing to poll; when it drops, the connection polls (every 2 seconds by default, and when the browser comes back online) and reopens it from the current cursor. connect({ live: false }) polls only. A server with several processes needs a shared log and a pub/sub between them; the reference server's subscribers are its own process's.

A refused read is silent on the page, by design, and the server says so in development. A page isn't told what it may not read: whether a document exists can be the very thing authorize hides. So a collection whose every read is refused looks, to the page, exactly like an empty one, and its components show their defaults with nothing in the console and no failed request. To keep that from being a mystery to whoever wrote authorize, the server logs once per collection when a page read the whole log and was sent none of a collection that has documents:

@zubra/sync: a page read "secrets" (GET /_zb/sync/pull) and was sent none of its 2 documents: `authorize` refused every read. …

It names the cause (authorize, the server's scope leaving them all out, or no authorize at all, with the allowReads([...]) that would open the collection). It's logged where process.env.NODE_ENV is development (an unset NODE_ENV is production, as for @zubra/server); createSyncServer({ warn }) takes a function to send it elsewhere, or false to turn it off (a server where one tenant rightly sees nothing of another's collection, say). A collection the page's own scope leaves out, a page that sees some of the collection, and later batches (other people's changes, refused as a matter of course) say nothing.

Only what the page shows: scope#

By default a page receives, and stores, every change in the log. A page that shows one showing's seats, or one business's orders, should ask for just those:

connect({ scope: { seats: 'showing-12/', holds: 'showing-12/', brands: true } });
  • Per collection, true or a key prefix, or a list of prefixes. A collection the scope doesn't name isn't sent. Keys made from several props (by showing, seat) are joined with /, so a prefix like 'showing-12/' takes every seat in that showing.
  • What's out of scope isn't sent: not by pull, not by the live stream (its backlog or new changes), and not as presence. The page's store never has it.
  • A page can connect to more than one server, each with its own scope (one for the app's rooms, one for its chat, say). Each connection sends a server the presence of its own scope's rooms only, so neither is sent the other's.
  • A connection started after another stopped takes up its live stream. The server keeps a stopped stream's peer id a while, for it to come back; stop() leaves its key for the next connection to the same server, which takes the id over instead of being refused (409) and giving the whole page a new one.
  • Each scope keeps its own cursor (in localStorage). A page that moves to another showing reads that showing from the start, not from wherever the last one got to.
  • The server can impose one: createSyncServer({ scope: ({ request }) => ({ orders: ${business(request)}/ }) }). A page's own scope can only narrow it, and pushes and presence outside it are refused with 403. That's tenancy: each business's documents under its own key prefix, on one log. authorize still runs on what's in scope.
  • The zubra-sync binary honours a page's scope. It has no notion of users, so a server-imposed scope is the JavaScript server's; put the binary behind an app that sets one.

The server taking part#

Devices decide most things themselves, and last writer wins is right for what one person owns. Some things two people can both want: the last item in stock, a job two drivers tap at once. The server decides those, in a server action, and its decision is a write to the same documents, so every page showing them follows.

const docs = await sync.documents();   // every document, as the devices merge it, kept current

export async function claim(formData, context) {
  'use server'
  const order = docs.get('orders', formData.get('order'));
  if (order?.driver) return { error: 'Taken' };
  // No await between the check and the write: no other request can come in between.
  await sync.write('orders', formData.get('order'), { driver: context.user.id, status: 'claimed' });
}
  • documents() reads the log once, then keeps up with every push and write. get(collection, key) is a document's fields (a copy); all(collection, prefix) is every document in a collection with its key, only those whose key starts with prefix when it's given (all('seats', 'show-12/')), in the order they were first written to; collections() names the collections that have any. Reads are synchronous.

  • write(collection, key, fields) stamps each field with the server's clock (createSyncServer({ device }), server by default), which is always later than any clock it has seen. So the write wins over everything before it, and a device that writes afterwards wins over it, as between any two devices. The view has it at once; then it's logged and sent to every live page. A server that hasn't read its log yet (it just started) reads it before its first write, so that write comes after every clock in it. If the log refuses a write, write() rejects and the documents take it back out; a push the log refuses never reaches them.

  • To keep a document the server's alone, refuse kind: 'change' for it in authorize, and let kind: 'presence' through if people should see who else has it open.

  • One process: the view is this process's, like the live subscribers. Several processes need a shared log and should decide such things in one place.

  • As a synchronous store for code that keeps its rules apart from where documents live (get, under, write), the three are a few lines over these, and keep its one rule: a write is in the documents before its promise settles, so a check and a write with no await between are one step.

    const docs = await sync.documents();
    const store = {
      get: (collection, key) => docs.get(collection, key) ?? null,
      under: (collection, prefix = '') => docs.all(collection, prefix).sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0)),
      write: sync.write,
    };
    

    On Workers, build it inside the Durable Object's own methods (await this.documents()), not in the Worker: the stub's documents are a copy. sync/test/store.test.js checks this on each log and inside an object.

On Cloudflare Workers: a Durable Object#

A Worker has many isolates and no memory they share, so the sync server can't live in one: a change pushed to one isolate would never reach a page listening on another. @zubra/sync/durable puts it in a Durable Object instead, one per app, with the log in the object's own SQLite (durableObjectLog(ctx.storage), the table and cursors of sqliteLog). The Worker forwards /_zb/sync/* to it.

import { SyncObject, syncStub } from '@zubra/sync/durable';

export class Market extends SyncObject {
  authorize({ request, collection, kind }) { /* as on Node; secrets are this.env's */ }
  // The object runs one call at a time: what must not interleave goes in its methods.
  async claim(order, driver) {
    if ((await this.documents()).get('orders', order)?.driver) return { error: 'Taken' };
    await this.write('orders', order, { driver, status: 'claimed' });
    return { ok: true };
  }
}

export default {
  async fetch(request, env) {
    const sync = syncStub(env.MARKET);             // the object named 'sync'
    return (await sync.fetch(request))              // Sync's requests, or null
      ?? render(await sync.documents('orders'));    // a copy, for this page
  },
};
// wrangler.jsonc: "durable_objects": { "bindings": [{ "name": "MARKET", "class_name": "Market" }] },
//                 "migrations": [{ "tag": "v1", "new_sqlite_classes": ["Market"] }]
  • One difference from Node: on Node, a server action reads documents() and writes in the same process, with no await between, so nothing interleaves. In a Worker the documents are in the object, and syncStub(…).documents() is a copy, not kept current. A read-then-write that must not interleave goes in a method of your SyncObject (sync.stub.claim(…)), where this.documents() is the live view. Don't fetch() in the middle of one: waiting on the network lets the object take the next request.
  • A page that vanishes is noticed a little later. A page that closes says it's leaving, and is gone from its rooms at once. One whose connection is simply cut (a crash, a dropped network) isn't reported to a Durable Object: it finds out when a keep-alive can't be written, which on workerd is the third after the cut. So SyncObject sends one every 2 seconds (static options = { heartbeat }), and such a page leaves within about 6.
  • Options: static options = { mount, heartbeat, device, table, origins }; syncStub(namespace, { name, mount }) must use the same mount. A different name is a separate object with its own log.
  • The object is awake while any page listens. An open live stream holds it in memory, which a Durable Object is billed for by duration; see Zubra on Cloudflare Workers.

Without Node: the zubra-sync binary#

sync-server/ in the repository builds zubra-sync, the same protocol as one binary with SQLite built in: zubra-sync --db sync.db --port 4000 --open. It reads and writes the same table as sqliteLog(), so a log can move between them. It has no notion of users, so it serves pages nothing until told: --read <collection> (repeatable) lets pages read those collections, --read-all every one, and --open reads and writes; with --open, put it behind your app at /_zb/sync (a reverse proxy after the app's sign-in), or allow a page's origin with --origin on a trusted network. The same protocol and browser tests run against it.

Presence: who else is here#

Some state should be seen by the other viewers of a document but never stored: where their cursor is, what they've selected, that they're typing.

<!-- board-view.zb -->
<script>
  'use collection boards';
  import { presence, peers } from '@zubra/sync';

  let strokes = [];                          // stored
  let cursor = presence({ x: 0, y: 0 });     // shared live, never stored
  let others = peers();                      // everyone else's presence
</script>

<board-view doc="">
  <div class="pad" onpointermove={(event) => (cursor = { x: event.offsetX, y: event.offsetY })}></div>
  <zb-each items={others} as="other" key={other.peer}>
    <span class="cursor" style:left={`${other.cursor.x}px`} style:top={`${other.cursor.y}px`}></span>
  </zb-each>
</board-view>
  • A room is a document. The board boards/b1's viewers see each other; another document's don't.
  • peers() is an array of the other viewers, { peer, ...presence }: each one's id and its presence fields by name (other.cursor above). Each tab is one peer; your own tab isn't in it. On the server it's empty, and a presence() value renders as its starting value.
  • Without a server, presence reaches the other tabs of the same site (BroadcastChannel). With connect(), it reaches other devices too: POST /_zb/sync/presence { room, peer, state }, and the live stream's presence events.
  • It's cheap on purpose. At most one message a frame, however fast the pointer moves; a heartbeat every 10 seconds; a viewer not heard from for 30 seconds has gone. Closing a tab says goodbye (pagehide), and the server says it for a device whose connection drops.
  • Nothing is stored, on the device or the server. authorize decides who may join a room (kind: 'presence') and who sees its presence (kind: 'read').
  • A peer id is its own stream's. The live stream's first event, hello, carries a secret key for its peer id; presence and signals sent as that peer must carry it, or they're refused with 403. A second stream asking for a peer id that's in use gets a 409, unless it presents the key (a reconnect, which takes over without a goodbye). So nobody can speak as another viewer, leave a room for them, or receive the signals meant for them. connect() does all of this itself.

Device to device: connect({ webrtc: true })#

Presence changes far more often than documents, and nobody needs it stored, so it doesn't have to go through a server at all once two viewers have found each other. With webrtc, each pair of viewers in a room opens a WebRTC data channel and sends presence over it; the server carries only what a channel can't:

  • introductions: a newcomer's first presence, which is how the others learn it's there;
  • signaling: the offer, answer and ICE candidates that open a channel (POST /_zb/sync/signal { room, from, to, data, key }, from a peer with a live stream, delivered as a signal event on that one peer's live stream, and only if that peer's scope has the room);
  • goodbyes, and presence to anyone without a channel yet.

On one network, the pointer moves never leave it. Across networks, peers need help finding a path: connect({ webrtc: { iceServers: [{ urls: 'stun:stun.example.org' }] } }). Some pairs can't connect directly at all (strict NATs, some corporate networks); that takes a TURN relay, which the app provides, since it carries traffic and costs bandwidth. Without one, those pairs simply keep using the server, so presence still works, just not directly.

  • Documents never go over channels. Their changes go through the log, which is what makes them durable and lets offline devices catch up.
  • It isn't end-to-end encrypted in the sense of "the server knows nothing". DTLS encrypts each channel, but the server still sees who is in which room, and carries the introductions.
  • link.direct() lists the peers presence currently goes to directly.

Edit this page on GitHub · Zubra 6.0.4