UTCP-SBI Wire Protocol

SPEC-710 — Production

UTCP-SBI defines how SBI (Sovereign Binary Interface) frames are carried over UTCP datagrams for mesh transfer, peer coordination, and inter-node communication. It replaces the JSON-based NIP protocol wire format with zero-encoding binary frames.

Normative reference: SPEC-039 Sovereign Binary Interface (Libertaria Core)

Design Principles

Principle	Implementation
Wire = Memory	SBI containers are raw `extern struct` bytes. No serialization step on same-architecture peers.
Transport owns framing	SBI defines containers, not framing. The 5-byte UTCP envelope is a transport concern.
Closed compression enum	Three blessed algorithms. Adding more requires a protocol version bump.
One exchange, done or dead	Handshake is a single message pair. No negotiation rounds.
Reject before allocate	Frame size validated before any buffer allocation. Two comparisons close the DoS vector.

Envelope Format

Every UTCP-SBI message is a 5-byte envelope followed by a complete SBI frame:

┌─────────────────┬──────────┬─────────────────────────────────────┐
│ frame_len: u32  │ op: u8   │ SBI Frame (preamble + container)   │
│ (big-endian)    │          │                                     │
└─────────────────┴──────────┴─────────────────────────────────────┘

frame_len is the byte length of the SBI frame only; it does NOT include the 5-byte envelope itself
op identifies the message type for routing before SBI decode
The SBI frame inside is a vanilla SPEC-039 frame (24-byte preamble + container payload)

Frame Size Constraints

UTCP_SBI_PREAMBLE_MIN = 24       // SBI preamble only, no payload
UTCP_SBI_MAX_FRAME    = 262144   // 256 KB hard cap

Validation is the first operation after reading the envelope – before any buffer allocation:

if frame_len < UTCP_SBI_PREAMBLE_MIN or frame_len > UTCP_SBI_MAX_FRAME:
    send NACK("invalid_frame_size")
    close connection

Two comparisons. Both cheap. Both non-negotiable.

Op Code Table

Op Code	Name	SBI Type	Size	Direction
`0x01`	`HANDSHAKE`	`SbiHandshake`	52B fixed	Bidirectional
`0x10`	`BLOCK_WANT`	`SbiBlockWant`	33B fixed	Request
`0x11`	`BLOCK_PUT`	`SbiBlockPut`	Variable (spine + DATA)	Response
`0x20`	`DAG_SYNC`	`SbiDagSync`	Variable (spine + DATA)	Bidirectional
`0xF0`	`ACK`	`SbiAck`	Fixed	Control
`0xF1`	`NACK`	`SbiNack`	Variable (error string)	Control

Op code ranges:

0x01–0x0F — Control (handshake, keepalive, future)
0x10–0x2F — Data transfer (blocks, chunks, DAG nodes)
0xF0–0xFF — Responses (ack, nack, status)

Message Definitions

HANDSHAKE (0x01)

Exchanged once at connection establishment. One message each direction – no negotiation round-trip.

SbiHandshake (52 bytes, extern struct, zero-pad aligned):

Offset  Field               Type      Size   Description
──────────────────────────────────────────────────────────────
0x00    peer_id             [32]u8    32     BLAKE3 peer identity (CellID)
0x20    capabilities        u32       4      Capability bitfield
0x24    required_features   u32       4      Features this peer requires
0x28    optional_features   u32       4      Features this peer supports optionally
0x2C    block_size          u32       4      Volume block size (for chunk alignment)
0x30    version             u16       2      Protocol version (1)
0x32    replica_count       u8        1      Scattered SB replica count
0x33    _pad                u8        1      Alignment padding
──────────────────────────────────────────────────────────────
        Total: 52 bytes (1 byte padding)

Capability bitfield:

Bit	Flag	Meaning
0	`HAS_DEFLATE`	Supports deflate compression
1	`HAS_ZSTD`	Supports ZSTD compression
2	`HAS_DEDUP`	Content-addressed deduplication on store
3	`HAS_RECOMPRESS`	Can recompress across algorithms
4–31	Reserved	Must be 0

Negotiation logic:

intersection = local.capabilities & remote.capabilities
missing = remote.required_features & ~intersection
if missing != 0:
    send NACK("missing_required_features")
    close connection

Optional features that aren't mutually supported are silently skipped. No negotiation round-trip; one exchange, one comparison, done or dead.

BLOCK_WANT (0x10)

Request a CAS chunk by content ID.

SbiBlockWant (33 bytes, extern struct):

Offset  Field         Type      Size   Description
──────────────────────────────────────────────────────────────
0x00    block_hash    [32]u8    32     BLAKE3-256 CID (of uncompressed content)
0x20    priority      u8        1      0=normal, 1=high, 2=critical
──────────────────────────────────────────────────────────────
        Total: 33 bytes

BLOCK_PUT (0x11)

Deliver a CAS chunk. Variable-length – spine + DATA region.

SbiBlockPut spine (40 bytes, extern struct):

Offset  Field         Type      Size   Description
──────────────────────────────────────────────────────────────
0x00    block_hash    [32]u8    32     BLAKE3-256 CID (of uncompressed content)
0x20    chunk_index   u32       4      Chunk position within file
0x24    comp_algo     u8        1      Compression algorithm (closed enum)
0x25    comp_level    u8        1      Compression level (0=default)
0x26    _pad          [2]u8     2      Alignment padding
──────────────────────────────────────────────────────────────
        Spine: 40 bytes

DATA region: compressed (or raw) chunk bytes immediately following spine

Receiver verification: BLAKE3(decompress(data)) == block_hash. If mismatch, send NACK.

DAG_SYNC (0x20)

Merkle tree synchronization. Peers exchange root hashes and missing node lists.

SbiDagSync spine (36 bytes, extern struct):

Offset  Field         Type      Size   Description
──────────────────────────────────────────────────────────────
0x00    root_hash     [32]u8    32     Merkle DAG root CID
0x20    depth         u16       2      Tree depth (for bounded traversal)
0x22    node_count    u16       2      Number of node hashes in DATA region
──────────────────────────────────────────────────────────────
        Spine: 36 bytes

DATA region: array of [32]u8 node hashes (node_count * 32 bytes)

ACK (0xF0)

Acknowledge successful receipt of a message.

SbiAck (8 bytes, extern struct):

Offset  Field         Type      Size   Description
──────────────────────────────────────────────────────────────
0x00    ref_seq       u32       4      Sequence number being acknowledged
0x04    status        u8        1      0=ok
0x05    _pad          [3]u8     3      Alignment padding
──────────────────────────────────────────────────────────────
        Total: 8 bytes

NACK (0xF1)

Reject a message or signal an error. Variable-length – optional error string in DATA region.

SbiNack spine (8 bytes, extern struct):

Offset  Field         Type      Size   Description
──────────────────────────────────────────────────────────────
0x00    ref_seq       u32       4      Sequence number being rejected
0x04    error_code    u16       2      Machine-readable error code
0x06    error_len     u16       2      Length of error string in DATA region (0 = none)
──────────────────────────────────────────────────────────────
        Spine: 8 bytes

DATA region: UTF-8 error string (error_len bytes, optional)

Mesh Transfer Integration

UTCP-SBI is the wire protocol for the Mesh Transfer system. The SBI codec and transport layer are wired directly to the CAS (Content-Addressable Storage) backend:

Op Code	Handler	CAS Operation
`BLOCK_WANT`	`MeshTransfer.handleBlockWant()`	`CasManager.get(block_hash)` → BLOCK_PUT reply
`BLOCK_PUT`	`MeshTransfer.handleBlockPut()`	Decompress → verify BLAKE3 → `CasManager.put(data)`
`DAG_SYNC`	`MeshTransfer.handleDagSync()`	Walk local DAG → compute missing set → respond
`HANDSHAKE`	`SbiTransport.handleHandshake()`	Capability negotiation, peer registration

The mesh daemon dispatches frames by op code to the appropriate handler. Each handler is a self-contained function that reads the SBI container, performs the CAS operation, and sends a response. No shared mutable state between handlers.

Compression

Negotiation

Compression support is declared in the HANDSHAKE capability bitfield. The sender picks the best mutually-supported algorithm:

Sender preference order:
1. ZSTD (if both HAS_ZSTD) — best ratio, acceptable speed
2. Deflate (if both HAS_DEFLATE) — Graf compatibility
3. None — raw bytes

The comp_algo field in BLOCK_PUT tells the receiver which algorithm was used. The receiver decompresses before BLAKE3 verification – the content hash is always over uncompressed data.

Algorithm Enum

Closed set. Adding algorithms requires a protocol version bump.

Value	Algorithm	Notes
`0x00`	None	Raw, uncompressed
`0x01`	Deflate	Graf compatibility
`0x02`	ZSTD	Levels 1–22
`0x03–0x0E`	Reserved	Future blessed algorithms
`0x0F`	Experimental	Receiver MUST reject unless out-of-band agreement exists

Zstd/Deflate Codec

The compression codec handles both algorithms through a unified interface:

Zstd: Default level 3 for BLOCK_PUT (good ratio, low latency). Level configurable per-connection via comp_level field.
Deflate: Level 6 (zlib default). Used primarily for Graf interop where Zstd is not available.
Passthrough: comp_algo = 0x00 skips compression entirely. Used for already-compressed content or tiny blocks where compression overhead exceeds savings.

BLAKE3 Content Addressing

All content hashes in UTCP-SBI use real BLAKE3 – 32-byte (256-bit) digests computed via C FFI to the reference BLAKE3 library. This is not a placeholder or substitute:

Property	Value
Algorithm	BLAKE3 (via C library FFI)
Digest size	32 bytes (256 bits)
Used in	`block_hash` (BLOCK_WANT, BLOCK_PUT), `root_hash` (DAG_SYNC), schema fingerprints
Verification	Every BLOCK_PUT is verified: `BLAKE3(decompress(data)) == block_hash`

CLD Schema Fingerprints

SBI preambles carry a schema fingerprint – a BLAKE3 hash of the canonical CLD (Container Layout Descriptor). This binds each wire message to its exact struct layout:

fingerprint = BLAKE3(canonical_cld_bytes)

If a peer sends a frame with an unknown fingerprint, the receiver knows the schema has changed and can NACK with "unknown_schema" instead of silently misinterpreting fields. Schema evolution without silent corruption.

Mechanism over Policy does not mean infinite extension points. It means the mechanism is honest about its boundaries.

Connection Lifecycle

Peer A                                  Peer B
──────                                  ──────
  │                                       │
  │──── HANDSHAKE(caps, features) ───────→│
  │←─── HANDSHAKE(caps, features) ────────│
  │                                       │
  │  [capability intersection check]      │
  │  [if missing_required → NACK, close]  │
  │                                       │
  │──── BLOCK_WANT(cid, priority) ───────→│
  │←─── BLOCK_PUT(cid, data) ────────────│
  │──── ACK(seq) ────────────────────────→│
  │                                       │
  │──── DAG_SYNC(root, hashes) ──────────→│
  │←─── DAG_SYNC(root, missing) ─────────│
  │──── BLOCK_PUT(cid1, data) ───────────→│
  │──── BLOCK_PUT(cid2, data) ───────────→│
  │←─── ACK(seq) ─────────────────────────│
  │                                       │

Version Negotiation

The HANDSHAKE carries version: u16. If a peer sends JSON (old protocol), the receiver reads 4 bytes of frame_len – JSON starts with { (0x7B), which would indicate a 2GB+ frame. The max-frame-size check rejects it instantly. Clean cut, no ambiguity, no graceful degradation.

Relationship to Existing Specs

Spec	Relationship
SPEC-039 (Libertaria)	SBI container format – normative. UTCP-SBI uses vanilla SPEC-039 frames without modification.
SPEC-093 (RUMPK)	UTCP transport layer – the L2 frame header (EtherType 0x88B5, CellID addressing, SYN/ACK/FIN/DATA flags). UTCP-SBI rides on top.
SPEC-101 (NexFS)	Feature flags – `required_features` / `optional_features` bitmasks in HANDSHAKE match NexFS superblock fields for interoperability.
ADR N2	UTCP over QUIC decision – architectural context for why UTCP exists.
ADR ST3	CBOR wire format – UTCP-SBI supersedes CBOR for inter-node wire encoding. CBOR remains the on-disk format for DAG nodes inside NexFS.

Implementation Notes

Nim (Mesh Daemon)

The mesh daemon uses the SBI binary codec for all wire communication. Nim does copy-decode, not zero-copy – the daemon is a coordination layer, not a hot path. Wire compatibility with Zig and JS SBI implementations is what matters.

Read path: read(4B) → frame_len → validate → read(1B) → op → read(frame_len) → decode_sbi_frame().

The daemon wires decoded frames to MeshTransfer handlers that operate against the CAS backend. See Mesh Transfer for the full architecture.

Zig (NexFS CAS FFI)

NexFS exposes CAS chunks for export/import via C-FFI. The mesh daemon (Nim) calls nexfs_cas_get() to read chunks for BLOCK_PUT responses and nexfs_cas_put() to store received chunks.

JavaScript (Graf Workers)

Graf already has an SBI implementation (graf-forge/workers/sbi.js). Browser-based mesh peers can participate using the same wire format.

Security Considerations

Frame size cap (256 KB) prevents memory exhaustion attacks
Minimum frame size (24 bytes) prevents underflow/parsing attacks
BLAKE3 content verification on every BLOCK_PUT prevents data corruption and substitution
Schema fingerprint in SBI preamble prevents type confusion between struct versions
Closed compression enum prevents algorithm negotiation attacks
CellID authentication is handled by the UTCP layer (SPEC-093), not by UTCP-SBI

UTCP-SBI Wire Protocol ​

Design Principles ​

Envelope Format ​

Frame Size Constraints ​

Op Code Table ​

Message Definitions ​

HANDSHAKE (0x01) ​

BLOCK_WANT (0x10) ​

BLOCK_PUT (0x11) ​

DAG_SYNC (0x20) ​

ACK (0xF0) ​

NACK (0xF1) ​

Mesh Transfer Integration ​

Compression ​

Negotiation ​

Algorithm Enum ​

Zstd/Deflate Codec ​

BLAKE3 Content Addressing ​

CLD Schema Fingerprints ​

Connection Lifecycle ​

Version Negotiation ​

Relationship to Existing Specs ​

Implementation Notes ​

Nim (Mesh Daemon) ​

Zig (NexFS CAS FFI) ​

JavaScript (Graf Workers) ​

Security Considerations ​