feat(io): OPFS-backed Storage for the browser, plus list/delete
Storage gains list and delete: a browser build has no OS file manager standing behind it, so enumerating and removing documents has to be the backend's job. Both are implemented for PhysicalFs and MemoryFs, and a shared storage_contract test runs against both so they cannot drift. OpfsFs (wasm32 only) implements the same *synchronous* trait. That works because createSyncAccessHandle() yields a handle whose read, write, truncate, flush and close are genuinely blocking -- available only inside a dedicated Web Worker, which is where improvise-9x6 puts the app. So the trait did not have to become async. What is still async is pulled out into four methods the worker awaits from its message loop: open(), attach(path), refresh() and commit_deletes(). attach must precede any read or write of a document, because acquiring a handle is async even though using it is not. Two consequences worth knowing. delete() drops the name from the index and queues a removeEntry, so it is visible to read and list immediately and lands in OPFS when commit_deletes is awaited -- the alternative was making the whole trait async for one operation. And handles are exclusive per file entry, so attach closes any handle it replaces and detach exists to release the lock. Checked before writing it: Safari 16.4+ has synchronous access handle methods (an early spec wrongly called them async, and every current browser implements them sync). So the IndexedDB fallback the ticket called for is not needed, per the user's "not unless recent Safari needs it" -- and the async question it would have forced does not arise. Compile- and clippy-clean for wasm32; there is no browser here, so improvise-xf1 tracks verifying it against one. storage_contract is the sequence it must satisfy. Closes improvise-i34 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V69PGQfMJuGRmtSWpfVAqn
This commit is contained in:
Edward Langley
Generated
+24
@@ -836,6 +836,7 @@ dependencies = [
|
||||
"improvise-core",
|
||||
"improvise-formula",
|
||||
"indexmap",
|
||||
"js-sys",
|
||||
"pest",
|
||||
"pest_derive",
|
||||
"pest_meta",
|
||||
@@ -843,6 +844,9 @@ dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tempfile",
|
||||
"wasm-bindgen",
|
||||
"wasm-bindgen-futures",
|
||||
"web-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2085,6 +2089,16 @@ dependencies = [
|
||||
"wasm-bindgen-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-futures"
|
||||
version = "0.4.77"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6b7777d5cc23d0e91404e53ce2d5e8ec7acae3026b16233dba62cd3246457950"
|
||||
dependencies = [
|
||||
"js-sys",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-macro"
|
||||
version = "0.2.127"
|
||||
@@ -2117,6 +2131,16 @@ dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "web-sys"
|
||||
version = "0.3.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c435338968042f4f59a557f690a253676d47ce13ceb55d70100e7facf6620a30"
|
||||
dependencies = [
|
||||
"js-sys",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "web-time"
|
||||
version = "1.1.0"
|
||||
|
||||
+7
-3
@@ -29,7 +29,7 @@ Architectural intent lives in `context/design-principles.md` — read that for t
|
||||
| Ask "what data is visible" vs "what can the user page" | `View::filtering_cats()` vs `categories_on(Axis::Page)` |
|
||||
| Change grid layout | `view/layout.rs` → `GridLayout` |
|
||||
| Change `.improv` format | `persistence/improv.pest` + `persistence/mod.rs` |
|
||||
| Change where documents are stored | `storage.rs` → `Storage` impls |
|
||||
| Change where documents are stored | `storage.rs` → `Storage` impls (`opfs.rs` for wasm) |
|
||||
| Change number display | `format.rs` → `format_f64` |
|
||||
| Change CLI args | `main.rs` (clap) |
|
||||
| Change import logic | `import/wizard.rs` → `ImportPipeline` |
|
||||
@@ -173,7 +173,9 @@ Region=West, Measure=Revenue = |pending| # pipe-quoted text
|
||||
|
||||
27. **`Axis::Filter` constrains without being navigable.** Drilling fixes its coordinates on `Filter`, not `Page` (improvise-dz8): both restrict which records are visible, but only `Page` is something the user may move or page through. The split shows up as two different questions, and picking the wrong one is the trap: **"what data is visible" is `View::filtering_cats()`** (Page ∪ Filter — used by `GridLayout` and by `add-record-row` when it builds a key from the current constraints), while **"what can the user act on" is `categories_on(Axis::Page)`** (used by `[`/`]` page cycling) or `View::tileable_cats()` (used by all three tile-select sites). `Axis::next()` maps `Filter` to itself and no navigable axis rotates into it, so tile select cannot hand one out. The `.improv` grammar accepts `filter` and round-trips a selection with it — a Filter axis only ever occurs in the never-persisted `_Drill` view, but the alternative was writing it out as some other axis.
|
||||
|
||||
28. **Two tests gate `ViewState` membership.** `ViewState` is the *portable* session slice — the one thing serialized into the Snapshot a newly-connected client gets (improvise-q08). A field may live there only if it is **portable** (plain data from the allowlisted crates, no `Rc`/`Arc`) and **snapshot-safe** (a fresh client inheriting the value is correct). `wizard` fails both — `ImportWizard` is an `improvise-io` type, and a half-open dialog is not something to inherit — and `drill_state` fails both — it holds an `Rc`, and `pending_edits` are uncommitted writes belonging to whoever staged them. Both live on `App` under the **session-local** residue tag (improvise-h0b). The `ViewState: Send` static assertion in `app.rs`'s tests is the compile-time proxy for portability: `Rc` is not `Send`, so a re-added shared pointer fails the build (the `Deserialize` derive from gotcha 25 now rejects it too, serde's `rc` feature being off). `Send` does *not* catch the origin-crate or snapshot-safety halves — check those by hand.
|
||||
28. **The browser backend is sync on purpose, with an async seam.** `OpfsFs` (`improvise-io/src/opfs.rs`, wasm32 only) implements the same synchronous `Storage` trait as `PhysicalFs`, because `FileSystemFileHandle.createSyncAccessHandle()` yields a handle whose read/write/truncate/flush **are** blocking — available only inside a dedicated Web Worker, which is where improvise-9x6 puts the app. So `Storage` never became async. What is still async splits out into four explicit methods the worker awaits from its message loop: `open()`, `attach(path)` (must precede any read/write of that document — acquiring a handle is async even though using it is not), `refresh()` (repopulates the index `list` answers from), and `commit_deletes()`. Consequence: **`delete` is visible immediately but lands in OPFS at the end of the batch** — it drops the name from the index and queues a `removeEntry`. Handles are exclusive per file, so `attach` closes any handle it replaces and `detach` releases the lock. `storage_contract` in `storage.rs` is the behaviour every backend owes; it runs against both host backends and is the written-down spec `OpfsFs` must satisfy in a browser, where the harness cannot reach.
|
||||
|
||||
29. **Two tests gate `ViewState` membership.** `ViewState` is the *portable* session slice — the one thing serialized into the Snapshot a newly-connected client gets (improvise-q08). A field may live there only if it is **portable** (plain data from the allowlisted crates, no `Rc`/`Arc`) and **snapshot-safe** (a fresh client inheriting the value is correct). `wizard` fails both — `ImportWizard` is an `improvise-io` type, and a half-open dialog is not something to inherit — and `drill_state` fails both — it holds an `Rc`, and `pending_edits` are uncommitted writes belonging to whoever staged them. Both live on `App` under the **session-local** residue tag (improvise-h0b). The `ViewState: Send` static assertion in `app.rs`'s tests is the compile-time proxy for portability: `Rc` is not `Send`, so a re-added shared pointer fails the build (the `Deserialize` derive from gotcha 25 now rejects it too, serde's `rc` feature being off). `Send` does *not* catch the origin-crate or snapshot-safety halves — check those by hand.
|
||||
|
||||
---
|
||||
|
||||
@@ -202,8 +204,10 @@ ast.rs 77 Expr, BinOp, AggFunc, Formula, Filter
|
||||
|
||||
### `improvise-io` (`crates/improvise-io/src/`)
|
||||
```
|
||||
storage.rs 130 / 5t Storage trait, PhysicalFs, MemoryFs (byte-level I/O)
|
||||
storage.rs 288 / 12t Storage trait (read/write/list/delete), PhysicalFs, MemoryFs
|
||||
injected into App as `storage`; effects use it, never std::fs
|
||||
opfs.rs 253 / 0t OpfsFs: Storage over the browser's Origin Private File
|
||||
System. wasm32 only. Compile-checked, not browser-tested.
|
||||
persistence/improv.pest 124 PEG grammar — single source of truth
|
||||
persistence/mod.rs 2783 / 97t save/load/gzip/legacy-JSON, CSV export
|
||||
import/wizard.rs 1121 / 38t ImportPipeline + ImportWizard
|
||||
|
||||
@@ -1331,6 +1331,12 @@ mod tests {
|
||||
fn write(&self, path: &Path, bytes: &[u8]) -> anyhow::Result<()> {
|
||||
self.0.write(path, bytes)
|
||||
}
|
||||
fn list(&self, dir: &Path) -> anyhow::Result<Vec<std::path::PathBuf>> {
|
||||
self.0.list(dir)
|
||||
}
|
||||
fn delete(&self, path: &Path) -> anyhow::Result<()> {
|
||||
self.0.delete(path)
|
||||
}
|
||||
}
|
||||
|
||||
let fs = Arc::new(MemoryFs::new());
|
||||
|
||||
@@ -19,6 +19,28 @@ pest_derive = "2.9.0"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
|
||||
# The browser backend. Only the wasm build pulls these in; the native
|
||||
# binary never sees them (improvise-i34).
|
||||
[target.'cfg(target_arch = "wasm32")'.dependencies]
|
||||
js-sys = "0.3"
|
||||
wasm-bindgen = "0.2"
|
||||
wasm-bindgen-futures = "0.4"
|
||||
|
||||
[target.'cfg(target_arch = "wasm32")'.dependencies.web-sys]
|
||||
version = "0.3"
|
||||
features = [
|
||||
"FileSystemDirectoryHandle",
|
||||
"FileSystemFileHandle",
|
||||
"FileSystemGetFileOptions",
|
||||
"FileSystemReadWriteOptions",
|
||||
"FileSystemRemoveOptions",
|
||||
"FileSystemSyncAccessHandle",
|
||||
"Navigator",
|
||||
"StorageManager",
|
||||
"WorkerGlobalScope",
|
||||
"WorkerNavigator",
|
||||
]
|
||||
|
||||
[dev-dependencies]
|
||||
pest_meta = "2.9.0"
|
||||
proptest = "1"
|
||||
|
||||
@@ -13,5 +13,9 @@ pub use improvise_core::{format, model, view, workbook};
|
||||
pub use improvise_formula as formula;
|
||||
|
||||
pub mod import;
|
||||
/// Browser storage. Only built for wasm — it is `Storage` over OPFS, and
|
||||
/// the native binary has `PhysicalFs` (improvise-i34).
|
||||
#[cfg(target_arch = "wasm32")]
|
||||
pub mod opfs;
|
||||
pub mod persistence;
|
||||
pub mod storage;
|
||||
|
||||
@@ -0,0 +1,253 @@
|
||||
//! Browser-backed [`Storage`] over the Origin Private File System.
|
||||
//!
|
||||
//! # Why this can be a synchronous backend
|
||||
//!
|
||||
//! OPFS has two APIs. The promise-based one (`getFileHandle`,
|
||||
//! `createWritable`) is async, but `FileSystemFileHandle`'s
|
||||
//! `createSyncAccessHandle()` yields a `FileSystemSyncAccessHandle` whose
|
||||
//! `read`/`write`/`truncate`/`flush`/`close` are ordinary blocking calls.
|
||||
//! It exists so wasm can do blocking file I/O — it is what SQLite's wasm
|
||||
//! build uses — and it is available **only inside a dedicated Web Worker**.
|
||||
//! improvise-9x6 puts the whole app in a worker, so the sync path is the
|
||||
//! one available to us and [`Storage`] does not have to become async.
|
||||
//!
|
||||
//! # The seam
|
||||
//!
|
||||
//! *Acquiring* a handle is still async, and so are directory listing and
|
||||
//! removal. So this backend splits into two halves:
|
||||
//!
|
||||
//! - **async, explicit**: [`OpfsFs::open`], [`OpfsFs::attach`],
|
||||
//! [`OpfsFs::refresh`], [`OpfsFs::commit_deletes`]. The worker calls
|
||||
//! these from its message loop, where awaiting is free.
|
||||
//! - **sync, via the trait**: `read`/`write` use an already-attached
|
||||
//! handle; `list` answers from the index `refresh` populated; `delete`
|
||||
//! drops the document from the index and queues the removal for
|
||||
//! `commit_deletes`.
|
||||
//!
|
||||
//! A `delete` is therefore visible to `read` and `list` immediately and
|
||||
//! lands in OPFS at the end of the batch. That ordering is the price of
|
||||
//! keeping the trait synchronous, and it is invisible to a single worker.
|
||||
|
||||
use std::cell::RefCell;
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
use wasm_bindgen::JsCast;
|
||||
use wasm_bindgen_futures::JsFuture;
|
||||
use web_sys::{
|
||||
FileSystemDirectoryHandle, FileSystemGetFileOptions, FileSystemReadWriteOptions,
|
||||
FileSystemSyncAccessHandle,
|
||||
};
|
||||
|
||||
use crate::storage::Storage;
|
||||
|
||||
/// Turn a `JsValue` error into something `anyhow` can carry. `JsValue` is
|
||||
/// neither `Send` nor `std::error::Error`, so it cannot cross into an
|
||||
/// `anyhow::Error` directly.
|
||||
fn js_err(context: &str, err: wasm_bindgen::JsValue) -> anyhow::Error {
|
||||
anyhow!("{context}: {err:?}")
|
||||
}
|
||||
|
||||
/// A document name as OPFS sees it: one flat entry in the root directory.
|
||||
/// OPFS has directories, but improvise stores documents side by side, and
|
||||
/// flattening keeps `Path` semantics from leaking into the browser build.
|
||||
fn entry_name(path: &Path) -> Result<String> {
|
||||
path.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
.map(str::to_owned)
|
||||
.ok_or_else(|| anyhow!("{} is not a usable document name", path.display()))
|
||||
}
|
||||
|
||||
pub struct OpfsFs {
|
||||
root: FileSystemDirectoryHandle,
|
||||
/// Sync access handles for attached documents. Exclusive per file:
|
||||
/// a second handle on the same entry fails until the first is closed,
|
||||
/// so `attach` closes any handle it replaces.
|
||||
handles: RefCell<HashMap<String, FileSystemSyncAccessHandle>>,
|
||||
/// Document names as of the last `refresh`, minus anything `delete`
|
||||
/// has since removed.
|
||||
index: RefCell<Vec<String>>,
|
||||
/// Removals awaiting `commit_deletes`.
|
||||
pending_deletes: RefCell<Vec<String>>,
|
||||
}
|
||||
|
||||
impl OpfsFs {
|
||||
/// Open the origin's private root directory. Async, once, at worker
|
||||
/// startup.
|
||||
pub async fn open() -> Result<Self> {
|
||||
let global = js_sys::global();
|
||||
let storage = global
|
||||
.dyn_into::<web_sys::WorkerGlobalScope>()
|
||||
.map_err(|_| anyhow!("OPFS sync access handles require a dedicated Web Worker"))?
|
||||
.navigator()
|
||||
.storage();
|
||||
let root = JsFuture::from(storage.get_directory())
|
||||
.await
|
||||
.map_err(|e| js_err("navigator.storage.getDirectory()", e))?
|
||||
.dyn_into::<FileSystemDirectoryHandle>()
|
||||
.map_err(|_| anyhow!("getDirectory() did not yield a directory handle"))?;
|
||||
Ok(Self {
|
||||
root,
|
||||
handles: RefCell::new(HashMap::new()),
|
||||
index: RefCell::new(Vec::new()),
|
||||
pending_deletes: RefCell::new(Vec::new()),
|
||||
})
|
||||
}
|
||||
|
||||
/// Acquire the sync access handle for one document, creating the file
|
||||
/// if it is not there. Must be awaited before `read` or `write` names
|
||||
/// that document.
|
||||
pub async fn attach(&self, path: &Path) -> Result<()> {
|
||||
let name = entry_name(path)?;
|
||||
if let Some(old) = self.handles.borrow_mut().remove(&name) {
|
||||
old.close();
|
||||
}
|
||||
let opts = FileSystemGetFileOptions::new();
|
||||
opts.set_create(true);
|
||||
let file = JsFuture::from(self.root.get_file_handle_with_options(&name, &opts))
|
||||
.await
|
||||
.map_err(|e| js_err(&format!("getFileHandle({name})"), e))?
|
||||
.dyn_into::<web_sys::FileSystemFileHandle>()
|
||||
.map_err(|_| anyhow!("getFileHandle({name}) did not yield a file handle"))?;
|
||||
let handle = JsFuture::from(file.create_sync_access_handle())
|
||||
.await
|
||||
.map_err(|e| js_err(&format!("createSyncAccessHandle({name})"), e))?
|
||||
.dyn_into::<FileSystemSyncAccessHandle>()
|
||||
.map_err(|_| anyhow!("createSyncAccessHandle({name}) did not yield a handle"))?;
|
||||
self.handles.borrow_mut().insert(name.clone(), handle);
|
||||
let mut index = self.index.borrow_mut();
|
||||
if !index.contains(&name) {
|
||||
index.push(name);
|
||||
index.sort();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Close and forget one document's handle, releasing the exclusive
|
||||
/// lock so another context can open it.
|
||||
pub fn detach(&self, path: &Path) -> Result<()> {
|
||||
let name = entry_name(path)?;
|
||||
if let Some(handle) = self.handles.borrow_mut().remove(&name) {
|
||||
handle.close();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Re-read the directory listing that `list` answers from.
|
||||
pub async fn refresh(&self) -> Result<()> {
|
||||
let mut names = Vec::new();
|
||||
let iter = js_sys::Reflect::get(&self.root, &"keys".into())
|
||||
.ok()
|
||||
.and_then(|f| f.dyn_into::<js_sys::Function>().ok())
|
||||
.ok_or_else(|| anyhow!("directory handle has no keys()"))?
|
||||
.call0(&self.root)
|
||||
.map_err(|e| js_err("keys()", e))?;
|
||||
let next = js_sys::Reflect::get(&iter, &"next".into())
|
||||
.map_err(|e| js_err("iterator.next", e))?
|
||||
.dyn_into::<js_sys::Function>()
|
||||
.map_err(|_| anyhow!("keys() iterator has no next()"))?;
|
||||
loop {
|
||||
let step = JsFuture::from(js_sys::Promise::from(
|
||||
next.call0(&iter).map_err(|e| js_err("next()", e))?,
|
||||
))
|
||||
.await
|
||||
.map_err(|e| js_err("await next()", e))?;
|
||||
let done = js_sys::Reflect::get(&step, &"done".into())
|
||||
.ok()
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(true);
|
||||
if done {
|
||||
break;
|
||||
}
|
||||
if let Some(name) = js_sys::Reflect::get(&step, &"value".into())
|
||||
.ok()
|
||||
.and_then(|v| v.as_string())
|
||||
{
|
||||
names.push(name);
|
||||
}
|
||||
}
|
||||
names.sort();
|
||||
let pending = self.pending_deletes.borrow();
|
||||
names.retain(|n| !pending.contains(n));
|
||||
*self.index.borrow_mut() = names;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Apply the removals that `delete` queued. Call at the end of a
|
||||
/// worker message batch.
|
||||
pub async fn commit_deletes(&self) -> Result<()> {
|
||||
let queued: Vec<String> = self.pending_deletes.borrow_mut().drain(..).collect();
|
||||
for name in queued {
|
||||
JsFuture::from(self.root.remove_entry(&name))
|
||||
.await
|
||||
.map_err(|e| js_err(&format!("removeEntry({name})"), e))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn with_handle<T>(
|
||||
&self,
|
||||
path: &Path,
|
||||
f: impl FnOnce(&FileSystemSyncAccessHandle) -> Result<T>,
|
||||
) -> Result<T> {
|
||||
let name = entry_name(path)?;
|
||||
let handles = self.handles.borrow();
|
||||
let handle = handles.get(&name).ok_or_else(|| {
|
||||
anyhow!("{name} is not attached; await OpfsFs::attach before reading or writing it")
|
||||
})?;
|
||||
f(handle)
|
||||
}
|
||||
}
|
||||
|
||||
impl Storage for OpfsFs {
|
||||
fn read(&self, path: &Path) -> Result<Vec<u8>> {
|
||||
self.with_handle(path, |handle| {
|
||||
let size = handle.get_size().map_err(|e| js_err("getSize()", e))? as usize;
|
||||
let mut buf = vec![0u8; size];
|
||||
let opts = FileSystemReadWriteOptions::new();
|
||||
opts.set_at(0.0);
|
||||
let read = handle
|
||||
.read_with_u8_array_and_options(&mut buf, &opts)
|
||||
.map_err(|e| js_err("read()", e))? as usize;
|
||||
buf.truncate(read);
|
||||
Ok(buf)
|
||||
})
|
||||
}
|
||||
|
||||
fn write(&self, path: &Path, bytes: &[u8]) -> Result<()> {
|
||||
self.with_handle(path, |handle| {
|
||||
// Truncate first: a shorter document must not leave the tail of
|
||||
// the previous one behind.
|
||||
handle
|
||||
.truncate_with_f64(0.0)
|
||||
.map_err(|e| js_err("truncate()", e))?;
|
||||
let opts = FileSystemReadWriteOptions::new();
|
||||
opts.set_at(0.0);
|
||||
handle
|
||||
.write_with_u8_array_and_options(bytes, &opts)
|
||||
.map_err(|e| js_err("write()", e))?;
|
||||
handle.flush().map_err(|e| js_err("flush()", e))?;
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn list(&self, _dir: &Path) -> Result<Vec<PathBuf>> {
|
||||
Ok(self.index.borrow().iter().map(PathBuf::from).collect())
|
||||
}
|
||||
|
||||
fn delete(&self, path: &Path) -> Result<()> {
|
||||
let name = entry_name(path)?;
|
||||
let mut index = self.index.borrow_mut();
|
||||
let Some(pos) = index.iter().position(|n| *n == name) else {
|
||||
return Err(anyhow!("Cannot delete {}", path.display()));
|
||||
};
|
||||
index.remove(pos);
|
||||
drop(index);
|
||||
if let Some(handle) = self.handles.borrow_mut().remove(&name) {
|
||||
handle.close();
|
||||
}
|
||||
self.pending_deletes.borrow_mut().push(name);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,10 @@
|
||||
//! Byte-level storage behind the persistence layer.
|
||||
//!
|
||||
//! Saving and loading a workbook only ever reads or writes one whole
|
||||
//! document, so that is all a backend has to provide. Keeping the trait to
|
||||
//! those two operations means a backend that is not a filesystem — browser
|
||||
//! OPFS, an in-memory fixture — implements it without pretending to have
|
||||
//! directories, metadata or seeking (improvise-6mq).
|
||||
//! document, so a backend never needs seeking, metadata or partial reads
|
||||
//! (improvise-6mq). It does need to enumerate and remove documents, since
|
||||
//! a browser build has no OS file manager standing behind it — hence
|
||||
//! `list` and `delete` alongside `read` and `write` (improvise-i34).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::io::{Read, Write};
|
||||
@@ -22,6 +22,12 @@ pub trait Storage {
|
||||
fn read(&self, path: &Path) -> Result<Vec<u8>>;
|
||||
/// Write a whole document, replacing anything already there.
|
||||
fn write(&self, path: &Path, bytes: &[u8]) -> Result<()>;
|
||||
/// Document names directly inside `dir`, sorted. Not recursive, and
|
||||
/// backends that are not trees treat `dir` as a name prefix.
|
||||
fn list(&self, dir: &Path) -> Result<Vec<PathBuf>>;
|
||||
/// Remove a document. Removing one that is not there is an error, so a
|
||||
/// caller that means "make sure it is gone" checks first.
|
||||
fn delete(&self, path: &Path) -> Result<()>;
|
||||
}
|
||||
|
||||
/// The real filesystem. What the native build wires up.
|
||||
@@ -44,6 +50,22 @@ impl Storage for PhysicalFs {
|
||||
file.write_all(bytes)
|
||||
.with_context(|| format!("Cannot write {}", path.display()))
|
||||
}
|
||||
|
||||
fn list(&self, dir: &Path) -> Result<Vec<PathBuf>> {
|
||||
let entries =
|
||||
std::fs::read_dir(dir).with_context(|| format!("Cannot list {}", dir.display()))?;
|
||||
let mut paths: Vec<PathBuf> = entries
|
||||
.filter_map(|e| e.ok())
|
||||
.filter(|e| e.file_type().is_ok_and(|t| t.is_file()))
|
||||
.map(|e| e.path())
|
||||
.collect();
|
||||
paths.sort();
|
||||
Ok(paths)
|
||||
}
|
||||
|
||||
fn delete(&self, path: &Path) -> Result<()> {
|
||||
std::fs::remove_file(path).with_context(|| format!("Cannot delete {}", path.display()))
|
||||
}
|
||||
}
|
||||
|
||||
/// An in-memory backend. Lets persistence be exercised without touching a
|
||||
@@ -90,12 +112,77 @@ impl Storage for MemoryFs {
|
||||
.insert(path.to_path_buf(), bytes.to_vec());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn list(&self, dir: &Path) -> Result<Vec<PathBuf>> {
|
||||
let mut paths: Vec<PathBuf> = self
|
||||
.files
|
||||
.lock()
|
||||
.expect("not poisoned")
|
||||
.keys()
|
||||
.filter(|p| p.parent() == Some(dir))
|
||||
.cloned()
|
||||
.collect();
|
||||
paths.sort();
|
||||
Ok(paths)
|
||||
}
|
||||
|
||||
fn delete(&self, path: &Path) -> Result<()> {
|
||||
self.files
|
||||
.lock()
|
||||
.expect("not poisoned")
|
||||
.remove(path)
|
||||
.map(|_| ())
|
||||
.with_context(|| format!("Cannot delete {}", path.display()))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The behaviour every backend owes, exercised against each host
|
||||
/// backend so they cannot drift apart. `OpfsFs` has to satisfy the same
|
||||
/// sequence in a browser, where this harness cannot reach it — so this
|
||||
/// doubles as the written-down contract for it (improvise-i34).
|
||||
fn storage_contract(fs: &dyn Storage, dir: &Path) {
|
||||
let a = dir.join("a.improv");
|
||||
let b = dir.join("b.improv");
|
||||
|
||||
assert!(fs.list(dir).unwrap().is_empty(), "starts empty");
|
||||
|
||||
fs.write(&a, b"first").unwrap();
|
||||
assert_eq!(fs.read(&a).unwrap(), b"first");
|
||||
assert_eq!(fs.list(dir).unwrap(), vec![a.clone()]);
|
||||
|
||||
fs.write(&a, b"second").unwrap();
|
||||
assert_eq!(fs.read(&a).unwrap(), b"second", "write replaces");
|
||||
assert_eq!(fs.list(dir).unwrap(), vec![a.clone()], "no duplicate entry");
|
||||
|
||||
fs.write(&b, b"other").unwrap();
|
||||
assert_eq!(fs.list(dir).unwrap(), vec![a.clone(), b.clone()], "sorted");
|
||||
|
||||
fs.delete(&a).unwrap();
|
||||
assert!(fs.read(&a).is_err(), "deleted document is unreadable");
|
||||
assert_eq!(fs.list(dir).unwrap(), vec![b.clone()]);
|
||||
assert!(fs.delete(&a).is_err(), "deleting twice is an error");
|
||||
|
||||
// A shorter document must not leave the tail of a longer one.
|
||||
fs.write(&b, b"0123456789").unwrap();
|
||||
fs.write(&b, b"xy").unwrap();
|
||||
assert_eq!(fs.read(&b).unwrap(), b"xy");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_fs_satisfies_the_storage_contract() {
|
||||
storage_contract(&MemoryFs::new(), Path::new("docs"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn physical_fs_satisfies_the_storage_contract() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
storage_contract(&PhysicalFs, dir.path());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_fs_round_trips_bytes() {
|
||||
let fs = MemoryFs::new();
|
||||
@@ -121,6 +208,65 @@ mod tests {
|
||||
assert!(err.contains("nope.improv"), "got {err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_fs_lists_only_the_named_directory() {
|
||||
let fs = MemoryFs::new();
|
||||
fs.write(Path::new("docs/a.improv"), b"a").unwrap();
|
||||
fs.write(Path::new("docs/b.improv"), b"b").unwrap();
|
||||
fs.write(Path::new("other/c.improv"), b"c").unwrap();
|
||||
assert_eq!(
|
||||
fs.list(Path::new("docs")).unwrap(),
|
||||
vec![
|
||||
PathBuf::from("docs/a.improv"),
|
||||
PathBuf::from("docs/b.improv")
|
||||
]
|
||||
);
|
||||
assert!(fs.list(Path::new("empty")).unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_fs_delete_removes_from_read_and_list() {
|
||||
let fs = MemoryFs::new();
|
||||
let path = Path::new("docs/a.improv");
|
||||
fs.write(path, b"a").unwrap();
|
||||
fs.delete(path).unwrap();
|
||||
assert!(fs.read(path).is_err());
|
||||
assert!(fs.list(Path::new("docs")).unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_fs_delete_missing_names_the_path() {
|
||||
let fs = MemoryFs::new();
|
||||
let err = fs.delete(Path::new("nope.improv")).unwrap_err().to_string();
|
||||
assert!(err.contains("nope.improv"), "got {err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn physical_fs_lists_files_but_not_subdirectories() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let fs = PhysicalFs;
|
||||
fs.write(&dir.path().join("b.improv"), b"b").unwrap();
|
||||
fs.write(&dir.path().join("a.improv"), b"a").unwrap();
|
||||
std::fs::create_dir(dir.path().join("sub")).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
fs.list(dir.path()).unwrap(),
|
||||
vec![dir.path().join("a.improv"), dir.path().join("b.improv")],
|
||||
"sorted, files only"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn physical_fs_delete_removes_the_file() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("m.improv");
|
||||
let fs = PhysicalFs;
|
||||
fs.write(&path, b"disk").unwrap();
|
||||
fs.delete(&path).unwrap();
|
||||
assert!(fs.read(&path).is_err());
|
||||
assert!(fs.list(dir.path()).unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn physical_fs_round_trips_bytes() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
|
||||
@@ -493,3 +493,32 @@ decision('improvise-dz8',filter_persists_in_the_format,'added "filter" to axis_k
|
||||
work_done('improvise-dz8',feature,'Axis::Filter added with filters_data/1 and is_user_assignable/1; View gained filtering_cats and tileable_cats. DrillIntoCell fixes coordinates on Filter instead of Page. GridLayout and add-record-row switched to filtering_cats; the three tile-select sites to tileable_cats; page cycling deliberately left on categories_on(Axis::Page).').
|
||||
work_unblocked('improvise-rml','improvise-dz8','adding the variant produced exactly ONE compile error outside axis.rs (the persistence writer), because rml had already collapsed the four display matches into one file. Evidence that the consolidation paid for itself.').
|
||||
doctrine(two_questions_one_enum,'when a new enum variant shares one property with an existing variant but not another, name both properties as predicates (filters_data, is_user_assignable) and route every call site through the one it means; testing for the variant directly is what goes stale when a third variant arrives').
|
||||
|
||||
|
||||
% ---------------------------------------------------------------------------
|
||||
% improvise-i34: OPFS-backed Storage. The trait stayed synchronous.
|
||||
% ---------------------------------------------------------------------------
|
||||
storage_op(list,'enumerate document names in a directory').
|
||||
storage_op(delete,'remove one document').
|
||||
storage_backend(opfs_fs,'crates/improvise-io/src/opfs.rs','wasm32 only; Storage over the Origin Private File System').
|
||||
|
||||
% Which OPFS operations are genuinely async, and how each is reconciled with
|
||||
% a synchronous trait. This is the design, not a note about it.
|
||||
opfs_async(get_directory, worker_init, 'OpfsFs::open').
|
||||
opfs_async(get_file_handle, per_document, 'OpfsFs::attach').
|
||||
opfs_async(create_sync_access_handle, per_document, 'OpfsFs::attach').
|
||||
opfs_async(directory_iteration, refreshed, 'OpfsFs::refresh populates the index list/1 answers from').
|
||||
opfs_async(remove_entry, deferred, 'delete/1 queues it; OpfsFs::commit_deletes drains at end of batch').
|
||||
opfs_sync(read).
|
||||
opfs_sync(write).
|
||||
opfs_sync(truncate).
|
||||
opfs_sync(flush).
|
||||
opfs_sync(close).
|
||||
|
||||
browser_fact(sync_access_handle_safari,'Safari 16.4+ supports createSyncAccessHandle, and all current browsers implement the handle methods synchronously despite an early spec that wrongly called them async. Checked 2026-08-31, so the IndexedDB fallback improvise-i34 called for is not needed.').
|
||||
browser_fact(sync_access_handle_worker_only,'available only in dedicated Web Workers, and exclusive per file entry: a second handle on the same file fails until the first is closed. Hence attach closing what it replaces, and detach existing at all.').
|
||||
decision('improvise-i34',no_indexeddb_fallback,'user decision 2026-08-31: skip the IndexedDB fallback unless recent Safari needs it. It does not -- Safari 16.4+ has synchronous access handles -- so the fallback is out of scope and the async question it would have forced does not arise.').
|
||||
decision('improvise-i34',delete_is_deferred,'removeEntry is async and the trait is sync, so delete/1 drops the name from the index and queues the removal. A delete is therefore visible to read and list at once and lands in OPFS when commit_deletes is awaited. Invisible to a single worker; the alternative was making the whole trait async for one operation.').
|
||||
work_done('improvise-i34',feature,'Storage gained list/delete, implemented for PhysicalFs and MemoryFs with a shared storage_contract test both satisfy. OpfsFs added behind cfg(target_arch=wasm32) using web-sys sync access handles, with open/attach/refresh/commit_deletes as the async seam. Compile- and clippy-clean for wasm32.').
|
||||
work_followup('improvise-i34','browser_verification','OpfsFs is compile-checked only -- no browser in this environment. storage_contract in storage.rs is the sequence it must satisfy. User will verify at the web MVP.').
|
||||
doctrine(contract_test_for_the_unreachable_backend,'when one implementation of a trait cannot be exercised by the test harness, write the contract as a function over &dyn Trait and run it against the reachable implementations; it becomes the executable specification the unreachable one is held to').
|
||||
|
||||
Reference in New Issue
Block a user