# `zset-ffi`: the reasoning

`crates/zset-ffi` is a C ABI over `zset` for CFFI callers (the first is Common Lisp). It is a
separate crate so `zset` stays std-only, serde-free and wasm32-clean (ADR 009 clause 1). It is
native only: it is **not** in `nix/ci.nix`'s `wasmCrates`, and the workspace-wide native
build (`--workspace`) already compiles it, so nothing in the nix files changes. The contract
is `crates/zset-ffi/include/zset_ffi.h`; this file records the decisions behind it. The
evidence is in `context/test-evidence/zset-ffi.md`.

## Build

`cargo build -p zset-ffi --release` writes `target/release/libzset_ffi.{so,dylib}` (the
`cdylib`), `libzset_ffi.a` (the `staticlib`) and the rlib. The `rlib` is what the Rust tests
link, so they call the `extern "C"` functions as plain functions.

## Decisions

1. **Monomorphic.** Keys are byte strings `(ptr, len)`, weights are `int64_t`. The generic
   `ZSet<K>` is instantiated once, at `Vec<u8>`. A caller encodes whatever it keys by.
2. **Opaque handles, explicit create and free.** `ZsZSet`, `ZsCircuit` and `ZsRebuild` are
   `Box::into_raw` pointers; `zs_X_new(&out)` makes one and `zs_X_free` releases it. Freeing
   null is a no-op. A double free or a use after free is undefined, as in C. Circuit streams,
   inputs and sinks are `uint32_t` ids local to their circuit, not handles: they own nothing.
3. **Error codes, never unwinding.** Every function but the frees and two accessors returns
   `int32_t`: 0, or a `ZS_ERR_*`. Every entry point body runs in `catch_unwind` (`guard`).
   A `zset` weight-overflow panic is recognised by its message prefix and becomes
   `ZS_ERR_OVERFLOW`; any other panic is `ZS_ERR_PANIC`. Results come back through
   out-pointers written only on success. `zs_last_error()` is a thread-local, NUL-terminated
   message, never null, valid until the next failure on that thread.
   The Rust panic hook still prints the overflow message to stderr before the code returns,
   because replacing a process-wide hook from a library is not this library's call. A host that
   wants silence installs its own hook, or avoids the overflow.
4. **Failure leaves the Z-set unchanged.** `add`, `sub`, `neg`, `scale` and `decode_add`
   compute a new value and then assign it, so an overflow half way through cannot leave a
   half-applied update (`zs_zset_insert` is one addition, so it has no half).
5. **A failed step poisons the circuit.** A panic in `step` leaves operator state half
   updated, and the circuit has no rollback (ADR 009 clause 11 is the host's; `zset.md`). So the first
   failure in a step, or in a rebuild step, marks the circuit poisoned, and every later call
   that uses it returns `ZS_ERR_POISONED`. A caller drops it and builds another. Ids that do
   not exist, and malformed arguments, are refused before any state changes and do not poison.
6. **Buffers.** The caller owns every buffer it passes and may reuse it once the call
   returns; the library copies. The library hands out no memory except handles. Output of
   variable size uses a two-call protocol: `zs_zset_encode(z, buf, cap, &needed)` always sets
   `needed`, and writes only if `cap >= needed` (else `ZS_ERR_BUFFER_TOO_SMALL`).
7. **Wire shape.** A Z-set encodes as entries ascending by key, each `[u32 LE key length][key]
   [i64 LE weight]`. A keyed element is `pair(k, v) = [u32 LE length of k][k][v]`. These are
   this crate's own, not clause 9's wire form (that lives in `improvise-protocol`).
8. **What the circuit offers.** Input, output sinks, `delay`, `integrate`, `differentiate`,
   `distinct`, `plus`, `filter` and `flat_map` (C callbacks), `index_by` (a callback emits the
   one group key), `join` over two pair streams, and `aggregate` with count, sum, min and max
   over `int64` values. `map` is `flat_map` that emits once. `aggregate`'s fold is fixed to
   those four because a C callback cannot return an `Option` of a group result without a
   second encoding; richer folds belong to a caller that wants one. The `Trace` interface and
   the `_in` constructors are Rust-side seams and are not exposed.
9. **Callbacks** are `extern "C"` function pointers with a `void *ctx`; the library never
   dereferences `ctx`, and keeps both for the circuit's life. A callback reports failure by
   its return value (`flat_map`/`index_by` non-zero, `keep` and `expired` negative), and the
   step then fails with `ZS_ERR_CALLBACK`. A callback must return normally. A Rust panic across
   `extern "C"` aborts, and a Lisp `handler-case` outside `zs_circuit_step` that unwinds
   through the Rust frames skips destructors and leaves a `RefCell` inside `zset` borrowed.
   The ABI cannot stop the unwinding, so it detects it: `ZsCircuit::in_step` is set for the
   duration of `step` and `rebuild_step`, and cleared only when they return. Every entry
   point first reads it, so a circuit whose frames were skipped answers `ZS_ERR_POISONED`
   forever, and a callback that re-enters its own circuit gets `ZS_ERR_POISONED` without the
   library touching the engine. `zs_rebuild_step` reads `circuit(c)` before it borrows the job,
   so re-entry from an `expired` callback is refused first. `zs_circuit_free` on a mid-step
   circuit and `zs_rebuild_free` on a job being stepped do nothing: the object is leaked, not
   freed under the frames that are running. The first fault recorded in a step is the one
   reported (`note`), so a failing `index_by` callback is `ZS_ERR_CALLBACK`, not the
   consequent `BAD_VALUE`. The CL rule is to put `handler-case` *inside* the callback and
   return the failure value (`tests/smoke.lisp` does). `index_by` needs exactly one emitted
   key and a keyed stream valid pairs, else `ZS_ERR_BAD_VALUE`; these faults are recorded in a
   shared cell rather than panicking.
10. **Thread confinement.** A circuit and every rebuild job made from it share an `Rc`
    (`Input`), so they are one group: keep the group on one thread at a time. Z-sets are
    separate handles. The error message is per thread.
11. **The header is hand-written** (cbindgen is not a dependency of this repo). `tests/header.rs`
    keeps it honest. It maps each Rust parameter and return type of every exported function
    (including the `unary_op!` expansions) to its C spelling, through a table that panics on
    an unlisted type, and compares them with the header's declarations. Every `Code` is
    declared by the `codes!` macro, which generates the enum, `Code::ALL` and the header
    name, so a new code cannot exist without a `#define` line and a matching value.
12. **Versioning.** `zs_abi_version()` and `ZS_ABI_VERSION` are 1. A change that breaks a
    signature or a code bumps it.

A rebuild is one logical tick however many `zs_rebuild_step` calls it takes (`zset.md`,
decision 8): a `delay` fed by it shows the previous tick once, and `done` is 1 as soon as the
last chunk ran, whatever the deadline callback says.

## Ids, handles and unsafety

All exported functions are `pub unsafe extern "C" fn`: the rlib is public API, and a safe
Rust caller could otherwise pass any pointer. Output pointers are checked before any work
(`with_out`): a refused call changes nothing and leaks nothing. Stream ids and sink ids are
different spaces (sink ids carry the top bit). A `ZsRebuild` records its circuit's id (a process-wide counter, not an address, which can be
reused) and is refused on another. A `Circuit::step` between two `zs_rebuild_step` calls is
`ZS_ERR_INTERRUPTED`, without poisoning: zset's `Rebuild::interrupted`, checked first. An id from another circuit is not detected. `zs_zset_add(a, a)` is
defined: the sum is computed through shared references and then stored. Keys of 4 GiB or
longer are `ZS_ERR_BAD_VALUE`, not truncated. A non-empty aggregate group always has a row,
even when its count or sum is zero.

## Miri

Miri checks the aliasing, leak and null rules the C compiler cannot. It runs `tests/abi.rs`
on a nightly built through the flake's `rust-overlay` input. The recipe below is **not
pinned**: `builtins.getFlake "github:oxalica/rust-overlay"` and `<nixpkgs>` (hence `--impure`)
resolve to whatever is current, and only the nightly date is fixed. It passed at the date of
this commit (about 12s):

```
nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi
```

It is not a CI job. A proposed one: a `miri` job in `.github/workflows/ci.yml` that takes the
nightly from the flake's own pinned `rust-overlay` input in `nix develop .#ci`.

## Calling it from Common Lisp

`crates/zset-ffi/tests/smoke.lisp` is a runnable CFFI program (`defcfun` for each call, a
foreign string for keys, `with-foreign-object` for out-parameters, the overflow code checked
and the message read back with `zs-last-error`). It loads the built library and passes. It is
**not a CI gate**: `sbcl` and `cffi` come from the user's profile and Quicklisp here, and are
not in the nix dev shell. The command to run it is in the file's header. Sketch of the
definitions it uses:

```lisp
(defcfun "zs_zset_new" :int32 (out :pointer))
(defcfun "zs_zset_insert" :int32 (z :pointer) (key :pointer) (len :size) (weight :int64))
(defcfun "zs_zset_weight" :int32 (z :pointer) (key :pointer) (len :size) (out :pointer))
(defcfun "zs_last_error" :string)
(defcfun "zs_zset_free" :void (z :pointer))
```

A callback (`filter`, `flat_map`) is a `defcallback` passed with `(callback name)`, with a
`handler-case` inside it that returns the failure value; its `emit` argument is a foreign
function pointer to call with `foreign-funcall-pointer`.