improvise/context/code/zset-ffi.md

zset-ffi: the reasoning

crates/zset-ffi is a C ABI over zset for CFFI callers (the first is Common Lisp). It is a separate crate so zset stays std-only, serde-free and wasm32-clean (ADR 009 clause 1). It is native only: it is not in nix/ci.nix's wasmCrates, and the workspace-wide native build (--workspace) already compiles it, so nothing in the nix files changes. The contract is crates/zset-ffi/include/zset_ffi.h; this file records the decisions behind it. The evidence is in context/test-evidence/zset-ffi.md.

Build

cargo build -p zset-ffi --release writes target/release/libzset_ffi.{so,dylib} (the cdylib), libzset_ffi.a (the staticlib) and the rlib. The rlib is what the Rust tests link, so they call the extern "C" functions as plain functions.

Decisions

1. Monomorphic. Keys are byte strings (ptr, len), weights are int64_t. The generic ZSet<K> is instantiated once, at Vec<u8>. A caller encodes whatever it keys by.

2. Opaque handles, explicit create and free. ZsZSet, ZsCircuit and ZsRebuild are Box::into_raw pointers; zs_X_new(&out) makes one and zs_X_free releases it. Freeing null is a no-op. A double free or a use after free is undefined, as in C. Circuit streams, inputs and sinks are uint32_t ids local to their circuit, not handles: they own nothing.

3. Error codes, never unwinding. Every function but the frees and two accessors returns int32_t: 0, or a ZS_ERR_*. Every entry point body runs in catch_unwind (guard). A zset weight-overflow panic is recognised by its message prefix and becomes ZS_ERR_OVERFLOW; any other panic is ZS_ERR_PANIC. Results come back through out-pointers written only on success. zs_last_error() is a thread-local, NUL-terminated message, never null, valid until the next failure on that thread. The Rust panic hook still prints the overflow message to stderr before the code returns, because replacing a process-wide hook from a library is not this library's call. A host that wants silence installs its own hook, or avoids the overflow.

4. Failure leaves the Z-set unchanged. add, sub, neg, scale and decode_add compute a new value and then assign it, so an overflow half way through cannot leave a half-applied update (zs_zset_insert is one addition, so it has no half).

5. A failed step poisons the circuit, and a rollback revives it. A panic in step leaves operator state half updated. So the first failure in a step, or in a rebuild step, marks the circuit poisoned, and every later call that uses it returns ZS_ERR_POISONED. The one way out is zs_circuit_rollback of a transaction that the failed step ran in (decision 13); a poison from outside a transaction is permanent, and the caller drops the circuit and builds another. Ids that do not exist, and malformed arguments, are refused before any state changes and do not poison.

6. Buffers. The caller owns every buffer it passes and may reuse it once the call returns; the library copies. The library hands out no memory except handles. Output of variable size uses a two-call protocol: zs_zset_encode(z, buf, cap, &needed) always sets needed, and writes only if cap >= needed (else ZS_ERR_BUFFER_TOO_SMALL).

7. Wire shape. A Z-set encodes as entries ascending by key, each [u32 LE key length][key] [i64 LE weight]. A keyed element is pair(k, v) = [u32 LE length of k][k][v]. These are this crate's own, not clause 9's wire form (that lives in improvise-protocol).

8. What the circuit offers. Input, output sinks, delay, integrate, differentiate, distinct, plus, filter and flat_map (C callbacks), index_by (a callback emits the one group key), join over two pair streams, and aggregate with count, sum, min and max over int64 values. map is flat_map that emits once. aggregate's fold is fixed to those four because a C callback cannot return an Option of a group result without a second encoding; richer folds belong to a caller that wants one. The Trace interface and the _in constructors are Rust-side seams and are not exposed.

9. Callbacks are extern "C" function pointers with a void *ctx; the library never dereferences ctx, and keeps both for the circuit's life. A callback reports failure by its return value (flat_map/index_by non-zero, keep and expired negative), and the step then fails with ZS_ERR_CALLBACK. A callback must return normally. A Rust panic across extern "C" aborts, and a Lisp handler-case outside zs_circuit_step that unwinds through the Rust frames skips destructors and leaves the circuit half way through a step. The ABI cannot stop the unwinding, so it detects it: ZsCircuit::in_step is set for the duration of step and rebuild_step, and cleared only when they return. Every entry point first reads it, so a circuit whose frames were skipped answers ZS_ERR_POISONED forever, and a callback that re-enters its own circuit gets ZS_ERR_POISONED without the library touching the engine. zs_rebuild_step reads circuit(c) before it borrows the job, so re-entry from an expired callback is refused first. zs_circuit_free on a mid-step circuit and zs_rebuild_free on a job being stepped do nothing: the object is leaked, not freed under the frames that are running. The first fault recorded in a step is the one reported (note), so a failing index_by callback is ZS_ERR_CALLBACK, not the consequent BAD_VALUE. The CL rule is to put handler-case inside the callback and return the failure value (tests/smoke.lisp does). index_by needs exactly one emitted key and a keyed stream valid pairs, else ZS_ERR_BAD_VALUE; these faults are recorded in a shared cell rather than panicking.

10. Threads (zset.md, "Threads"). A ZsCircuit may be moved between threads and is used by one at a time; it is Send and not Sync, with no lock. Callbacks run on whichever thread calls step or rebuild_step, with that thread's thread-local state (Lisp dynamic bindings, errno, zs_last_error), so the caller's ctx must be usable from every thread that may step (the wrapper Ctx is unsafe impl Send on that contract). The first-fault cell is the one lock, and it is the FFI's own. A ZsRebuild no longer shares a reference count with its circuit (the handles are ids), so it is a separate object, and a zs_rebuild_step call uses both. The caller still keeps each to one thread at a time. Z-sets are separate handles. The error message is per thread (tests/threads_abi.rs: callbacks ran on the thread that stepped).

11. The header is hand-written (cbindgen is not a dependency of this repo). tests/header.rs keeps it honest. It maps each Rust parameter and return type of every exported function (including the unary_op! expansions) to its C spelling, through a table that panics on an unlisted type, and compares them with the header's declarations. Every Code is declared by the codes! macro, which generates the enum, Code::ALL and the header name, so a new code cannot exist without a #define line and a matching value.

12. Versioning. zs_abi_version() and ZS_ABI_VERSION are 1. A change that breaks a signature or a code bumps it.

13. Transactions (improvise-bw4p.5; zset.md, "Transactions"). zs_circuit_begin, zs_circuit_commit and zs_circuit_rollback wrap Circuit::begin/commit/rollback; a handle-returning snapshot was not chosen, because the circuit logs what changes and there is no snapshot object to free. The new code is ZS_ERR_TRANSACTION (10) for a begin on an open transaction, a commit or rollback with none open, and a zs_circuit_take while one is open (take after commit: a rollback subtracts the step's output from the sink). begin on a poisoned circuit is ZS_ERR_POISONED, and so is commit after a failed step: the host rolls back. rollback is the only call that clears the poison, and only a poison that came from a step inside the transaction. This is additive, so ZS_ABI_VERSION stays 1. What a rollback cannot undo is a callback's own side effects and a ZsRebuild stepped in the transaction, finished or not (it answers ZS_ERR_INTERRUPTED after, and the host starts another job; rollback_abi.rs). A circuit left mid-step by a non-local exit answers ZS_ERR_POISONED to a rollback like to everything else: the frames are skipped and the engine is not touched.

A rebuild is one logical tick however many zs_rebuild_step calls it takes (zset.md, decision 8): a delay fed by it shows the previous tick once, and done is 1 as soon as the last chunk ran, whatever the deadline callback says.

Ids, handles and unsafety

All exported functions are pub unsafe extern "C" fn: the rlib is public API, and a safe Rust caller could otherwise pass any pointer. Output pointers are checked before any work (with_out): a refused call changes nothing and leaks nothing. Stream ids and sink ids are different spaces (sink ids carry the top bit). A ZsRebuild records its circuit's id (a process-wide counter, not an address, which can be reused) and is refused on another. A Circuit::step between two zs_rebuild_step calls is ZS_ERR_INTERRUPTED, without poisoning: zset's Rebuild::interrupted, checked first. An id from another circuit is not detected. zs_zset_add(a, a) is defined: the sum is computed through shared references and then stored. Keys of 4 GiB or longer are ZS_ERR_BAD_VALUE, not truncated. A non-empty aggregate group always has a row, even when its count or sum is zero.

Miri

Miri checks the aliasing, leak and null rules the C compiler cannot. It runs tests/abi.rs on a nightly built through the flake's rust-overlay input. The recipe below is not pinned: builtins.getFlake "github:oxalica/rust-overlay" and <nixpkgs> (hence --impure) resolve to whatever is current, and only the nightly date is fixed. It passed at the date of this commit (about 12s):

nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi --test rollback_abi

It is not a CI job. A proposed one: a miri job in .github/workflows/ci.yml that takes the nightly from the flake's own pinned rust-overlay input in nix develop .#ci.

Calling it from Common Lisp

crates/zset-ffi/tests/smoke.lisp is a runnable CFFI program (defcfun for each call, a foreign string for keys, with-foreign-object for out-parameters, the overflow code checked and the message read back with zs-last-error). It loads the built library and passes. It is not a CI gate: sbcl and cffi come from the user's profile and Quicklisp here, and are not in the nix dev shell. The command to run it is in the file's header. Sketch of the definitions it uses:

(defcfun "zs_zset_new" :int32 (out :pointer))
(defcfun "zs_zset_insert" :int32 (z :pointer) (key :pointer) (len :size) (weight :int64))
(defcfun "zs_zset_weight" :int32 (z :pointer) (key :pointer) (len :size) (out :pointer))
(defcfun "zs_last_error" :string)
(defcfun "zs_zset_free" :void (z :pointer))

A callback (filter, flat_map) is a defcallback passed with (callback name), with a handler-case inside it that returns the failure value; its emit argument is a foreign function pointer to call with foreign-funcall-pointer.