Follow the curiosity#

Aretta's curiosity-guided bug discovery loop enables this connection between the developer who knows where to look and the expert who knows how to verify. The loop has three surfaces: Ask, Campaign, and Finding. In Ask, a developer poses questions about their verification record in plain English: which parts have been modelled, and the set of guarantees checked so far. When the record has no answer, because no guarantee on file covers the question, the question becomes a campaign: a newly scoped piece of modelling and verification work, priced up front and tied to a concrete outcome, a new guarantee added to the record and a verdict on whether the code upholds it. If the campaign finds that a guarantee fails, it produces a finding: a bug, described in plain language, on the developer's dashboard.

The customer sets the direction. They decide which question is worth a campaign, they see the price before approving it. We verify where developers point us, rather than based on our guesswork.

A question about short reads#

For Turso, one of our design partners building a modern rewrite of SQLite in Rust, we can put the following question to the Ask surface:

If a disk read comes up short mid-recovery, could it silently lose data I have already committed?

A disk read can return fewer bytes than requested; correct code checks for this and reads the rest. The danger is a short read during recovery, when the engine reads its write-ahead log to restore committed data after a crash. Mishandled there, committed data can be dropped, with no error raised.

Ask draws its answers from the verification record, the models and proofs we have already built, but sometimes the record has none. Here it covered durability once data is checkpointed, but nothing on it addressed a short read during recovery, before the data is folded into the main database.

We close a gap like this by expanding the modelling and verification, and here that meant a campaign: Verify Turso surfaces I/O errors instead of silently swallowing them, seeded by a lead from the durability work we wrote about earlier.

What the check found#

Once approved, the campaign wrote a machine-checked specification of one guarantee: a failed or short disk read on a durability-critical path must be surfaced, never quietly treated as success. Further it attempted to verify the guarantee against the live engine. The check failed. When injecting a short read at the moment recovery reads the log, the above guarantee was violated. On the other hand, when running the identical fault against SQLite as a control, the guarantee held.

The failure of this guarantee is a consequential data-loss class bug. The database engine keeps recently committed rows in a write-ahead log (WAL). This is a side file it replays at open to restore anything not yet folded into the main database. The write-head log begins with a small header. When the read of that header is fragmented, by even a few bytes, the whole log is read as empty, and every committed row in it is discarded. Moreover, this data loss is silent as the query returns, the process exits 0, and PRAGMA integrity_check, the built-in health check, reports ok.

What it means#

We reported this data loss bug on the project's public tracker, turso#8593, with a reproduction. The launched campaign is now associated with a result: 1 bug found. Further the test capturing the bug durably persists in CI/CD and re-runs on every later change. Thus, the bug cannot be re-introduced unnoticed.

The specification behind this bug was not represented in any tests. It surfaced because of a curiousity about one specific scenario of disk behaviour. Asking a plain natural language question rather than a week of harness-building resulted in the bug discovery. This is the whole idea: the most valuable checks are aimed where a developer already suspects a weak spot, so the job is to make aiming them cheap. We add to the verification record one question at a time, guided by where developers think the risk lies.

Why networking#

Network protocol code is critical for two key reasons. Firstly, it provides load-bearing guarantees and abstractions that the software infrastructure built above it depends on. Secondly, it is subject to untrusted inputs that come from the outside world. Further, these inputs are highly unpredictable, since how bytes arrive depends on the network path, the current system load, and other conditions nobody controls. In a handwritten test, the input arrives in one write; on a real network, the same bytes can arrive in many pieces. That gap holds a whole class of behavior that example-based test suites cannot reach.

Mistakes here are not always benign either. Protocol code sits at a trust boundary, and when an implementation interprets input differently from the components around it, the disagreement itself becomes attack surface: entire attack classes, request smuggling among them, are built on exactly such disagreements, and differential studies of HTTP implementations keep surfacing new ones. Making protocol code interpret input correctly removes the disagreements these attacks depend on, so verifying its correctness is also a security measure.

Underlying method#

Aristo achieves network verification with the same interface described in our blogpost on the verification loop: an intent (a guarantee written as a plain claim), a model that makes the claim precise, and checks that hold the code to the claim on every input it can reach, with the ability to re-run verification as the code changes.

Consider HTTP framing as an example. The intent is a single sentence:

The messages parsed from a connection depend on the bytes alone, never on how those bytes were split across reads.

Our technique developed a formal model of HTTP/1.1 framing and proved that the model satisfies this claim. We then tested hyper against the same claim, replaying byte streams under every possible split and checking whether the outcomes matched.

The first result#

The outcomes matched on nearly every input. They differed on one: a request with an uncommon but accepted header ending. The result is a refusal in the form of a dropped connection. The cause is a fast-path scan that runs only when a request arrives split: it catches some endings of a header block, but not the one in question. We reported it upstream with this diagnosis, and a community fix was merged that extends the fast-path scan to cover the missing ending.

A model-based test systematically asks questions of the code that ad-hoc reviews and example-based tests alone cannot. For hyper, it took a couple of hours to catch a bug. The bug we found is a correctness issue, but as the smuggling discussion above suggests, protocol code carries security guarantees as well. Aristo checks both the same way: stated precisely, checked under every scenario, and re-checked as the code changes.

Work with us#

Aristo is free and open source. The full service runs this method at depth and continuously as the code changes. If you build proxies, gateways, protocol libraries, or anything that parses bytes off a wire, apply to join, or reach us at hello@aretta.ai.

And yet a data race lived in its write-ahead log for sixteen years. It corrupted a production database at Tailscale nineteen times over six months before anyone understood why. It was so rare that the SQLite developers, once they had found it, had to add code to their own test harness just to provoke it.

Proofs have a reputation as the slow option, the heavyweight tool you reach for in avionics or cryptography when you have time to spare. This is a case for the opposite view, argued on the least sympathetic example we could find. We checked one property, taken straight from SQLite's own WAL specification, by reasoning rather than by running the code, and that was enough to locate the exact window this bug lives in, without the race ever having to happen.

The bug, briefly#

SQLite records recent writes in a write-ahead log and later folds them into the main database file, a step called a checkpoint; once a log is fully checkpointed, a writer may reset it and start again at the beginning. The bug is a rare data race between a checkpoint and a concurrent reset: the checkpoint misses the reset, keeps working from a stale view of the log, and writes stale log data back over the database file. Committed writes disappear, and the file can fail PRAGMA integrity_check. SQLite documents the bug in its own WAL spec; Tailscale, who hit it repeatedly in production, spent months tracing it with the SQLite developers and wrote that story up; Carl Sverre, writing at Antithesis, reproduced it by searching schedules. SQLite fixed it in version 3.51.3. Ours is a fourth account, by a different method.

Why testing can miss a bug like this#

The bug survived sixteen years of excellent testing for one reason: it is rare in time, and testing samples time. To see it, the exact interleaving has to occur: one connection resets the log in the same narrow instant another is midway through a checkpoint. You can run that workload for months and never land on it.

It is not rare in place, though. The interleaving corresponds to a fixed, small region of the code, and that is the opening: enumerate the schedules a piece of code admits, instead of sampling them by running it, and you never need the rare moment to arrive.

The property, taken from the spec#

One sentence in, a runnable bug out. The developer writes the intent; the Aretta backend does the rest, automatically:

Aristo, the verification tool we build, starts from an intent: a property stated in plain language, the way you would put it in a design doc. The durability bug we found in Turso started from an intent too. Here, the intent is a standard durability claim, and it is the claim SQLite's own WAL documentation already makes:

After a checkpoint, the database file reflects every committed frame up to the checkpoint point. A checkpoint never makes committed data invisible.

That sentence, an Aristo intent lifted straight from the WAL spec, is the whole input. Everything after it runs on Aretta's verification service. The service maps the intent to a formal statement and to a formal model of the checkpoint and reset protocol, then checks that model against the property, and that check is what surfaced the bug. Stated over the value a reader returns for a page, the property becomes the model's durability invariant:

-- durability: an actor with an open read transaction reads, for every page,
-- exactly the committed value as of when its transaction began. A lost
-- committed write, or a value resurrected from an overwritten log frame,
-- makes this false.
def durInv (c : Config) : Bool :=
  c.th.all (fun t => readerDurable c.shm t.loc) && freshDurable c

The check asks one question: does every interleaving of the checkpoint and reset protocol preserve durInv? The objection to preempt here is that we knew the answer and steered toward it, so it is worth being exact: the input was that one sentence. We did not write the formal statement, the model, or the interleavings, and nothing we gave the service named a bug, a place to look, a symptom, or a fix. We read the fix only afterward, to confirm the window matched. We brought exactly one piece of prior knowledge: we pointed the service at the WAL and checkpoint code rather than at all of SQLite, and durability is where any verification of a write-ahead log begins. Neither choice says anything about this defect. Everything else came from the infrastructure.

The answer is no, and the failing case is concrete: a fifty-three-step interleaving of a writer, a passive checkpointer, and a reader ends in a state where durInv is false. That is a theorem, checked by the Lean kernel:

theorem ce1_violates_durability :
    (run initReset schedCE1).map durInv = some false := by decide

The counterexample is small enough to print. A schedule in the model is nothing but a list of actor ids, one per sub-operation, and schedCE1 is fifty-three of them. Laid out on a timeline, the race is visible to the eye: the checkpointer caches its view of the log just before the reset lands, and everything that follows executes faithfully against that stale snapshot.

The counterexample also violates an invariant SQLite's own source comments assert, that "the nBackfill number is never greater than WalIndexHdr.mxFrame" (src/wal.c:342), evidence that the model tracks the real protocol rather than a caricature of it. And the fix guards exactly this window: version 3.51.3 adds a check that, after the checkpoint takes its read lock, detects that the log was reset out from under it and declines to write the stale data back.

From the formal model to the binary#

A counterexample in a formal model is a mathematical refutation: the Lean kernel certifies that the modeled protocol can reach a state where durInv is false. What it refutes, though, is the model, and a model can misdescribe the code it stands for. Aretta's backend discharges that doubt automatically: it replays the counterexample's schedule against the real library, holding a checkpoint inside the window while a writer resets the log. What reaches the developer is a runnable reproduction on a stock build, not a claim about a model. Under that schedule, a stock build of 3.51.2 loses committed rows, and in most runs the file is corrupted as well. A stock build of 3.51.3, under the identical schedule, does neither.

The stall the schedule needs is not exotic. A pause of a few milliseconds at the right point, the kind an ordinary scheduler hands out for a page fault or a slow syscall, is enough. That is how the bug reached production.

Search and reasoning#

Antithesis reproduced this bug by searching schedules; we located it by reasoning about them. The two routes are complementary. Search needs no formal model and is the right tool when you cannot build one: it explores executions until one trips an assertion. Reasoning needs a model, and in exchange it hands you the mechanism itself: the violated property, a machine-checked counterexample, and the exact window in the code, all before the code runs once. To be clear about credit, we did not find this bug; SQLite and Tailscale did. We showed that one property, taken from the spec, is enough to pin it down.

The mindset this should change#

The slow path to this bug was real: months of production corruption, forensic telemetry in a live system, and in the end, test-harness code written just to force the bug. The reasoning path was a property and a formal model, and it skipped all of that: the incident, the irreproducible flake, the triage.

Formal methods have long been filed under caution, a luxury for teams with time and budget to spare. What made verification elusive was its cost, and that cost is falling. As it falls, the arithmetic turns over: reasoning about where a bug must be is cheaper than waiting for it, reproducing it, and triaging it after an outage. The deterministic test the reasoning leaves behind is a standing guard that re-runs on every change. On software tested as thoroughly as any on Earth, a failure that took months to run down in production fell out of a single sentence of its own spec. Testing samples time. A proof does not have to.

A closing indulgence: the property above is not new. It is the WAL specification, which SQLite has published all along. So, at the risk of hindsight bravado, the bug was checkable the day it was written in 2010, and finding it never needed the race to happen even once. Aristo did not exist in 2010. Alas.

The gap that regression tests cannot fill#

The fix was correct for the reported bug; however, it broke a neighboring, unreported durability specification. Detecting the second break is beyond the scope of an ordinary regression test suite.

An ordinary regression test would have been enough for the reported bug. After this fix, Turso's suite would gain a test that reproduces the checkpoint bug, the fix would make that test pass, and the test would stay in the suite to detect the same behavior if it ever returned. The limit is coverage. A regression test covers exactly the property it was written for, and a suite built this way has tests only for bugs that have already been found. So after a fix that repairs the known bug but breaks something else, every existing test still passes: there is no test for the property that broke.

How Aristo caught it#

Aristo starts from the guarantee instead of from the bug history. In the first post about this engagement we showed how that works. The top-level promise is decomposed into a conjunction of smaller specifications, S ⟺ A1 ∧ A2 ∧ …, and those into smaller ones, until every leaf is something you can check against the code. A bug, in this setting, is a leaf specification found to be false.

Instantiated for Turso's write-ahead-log subsystem (the symbols are schematic, so these leaves differ from the first post's tree), the top-level promise S is that a crash never corrupts or loses committed data, and two of its leaves are:

• A1: checkpointing is crash-atomic. A crash during a checkpoint never results in a torn database file. The reported bug (issue #7952) was a failure of this specification.
• A2: a COMMIT under PRAGMA synchronous=FULL does not return until its WAL frames are fsync-durable, so an acknowledged commit survives a crash. This one had passed every check so far.

Sibling specifications like A2 are the point: nobody wrote all the leaf specifications in anticipation of a particular failure; they exist because the proof of the top-level guarantee needs them, so the model already has coverage wherever a fix might have side effects.

Re-checking the model at the fix commit#

When the PR merged, we re-checked the whole model, not only the reproduction for the reported bug. At the fix commit, the check of A1 passed for the first time: the reported bug was gone. But the check of A2, which had passed at every revision before, now failed. Hence S still did not hold. That automated result is what the bot posted when it reopened the issue. The failure of A2 comes down to one observable requirement: under synchronous=FULL, COMMIT must fsync the WAL before it returns. The table below shows that requirement checked on both revisions, with and without a checkpoint immediately before the COMMIT:

Under PRAGMA synchronous=FULL, a COMMIT that follows a WAL-resetting checkpoint does not fsync the WAL. The frames are written and COMMIT returns success, but nothing forces them to disk. Without the preceding checkpoint the fsync happens correctly, which is why this is easy to miss in review and in ordinary test suites. In fact, this regression was more severe than the original bug: a torn file is corruption a checker can detect, while this one means an acknowledged commit is silently lost, with PRAGMA integrity_check still reporting "ok". None of it reached production; the failure was demonstrated in a crash simulator during verification.

Resolution#

Subsequently, Pekka Enberg at Turso confirmed the mechanism and implemented the full fix in a PR, including the regression test recommended in our reopening comment. Turso closed #7952 again, this time citing the fix commit. Less than a day passed between our reopening of the issue and the merged upstream fix. Enberg also described the episode on X, noting that "in hindsight I should have just run the verification step on the original fix to catch this earlier."

Both A1 and A2 remain in the model, and each new revision of Turso is re-checked against them as part of the standing report we maintain. That is the proof maintenance problem in the title, and the last leg of the verification loop we described earlier: re-running the proof whenever the code changes. A fix is a code change like any other, which is why re-running the whole proof, and not just the reported bug's test, is what caught the regression.

Work with us#

Aristo, the SDK our verification service is built on, is free and open source. Deeper verification as described here is what we run with design partners, and we are onboarding 2 to 3 more. If the hardest guarantees in your system are on the crash path, apply to join, or reach us at hello@aretta.ai.

The trust-building verification loop#

To answer this question, verification must be seen as a process loop that builds trust in the software over time:

1. Specification: deciding and communicating what to prove.
2. Verification: scalably performing the proof.
3. Proof reporting and maintenance: exposing proof results to the user and maintaining them.

Most of the advances in verification to date have been concentrated on developing novel proof languages and scalable verification algorithms; these are tools that live on the verification side. The core bottleneck to adoption is interfacing these tools with developers in the wild. This interface is where the rubber hits the road.

Specification: deciding and communicating what to prove#

While developers usually have specifications or invariants in their heads, traditionally verification has required exporting specifications into formal languages. First, this is challenging for the majority of developers due to unfamiliarity with formal languages. Second, formal specification languages just aren't very interpretable, making formal specifications a poor way of communicating claims and intents between developer teams. This is why the spec either never gets written or quietly rots after the first refactor. Specification, in practice, is an authoring problem long before it is a math problem.

At Aretta we recognized this problem early and built our SDK Aristo to remove it. Aristo puts the claim in the code, in plain English, directly above the function it describes:

#[aristo::assume("lo <= hi")]
#[aristo::intent("returns a value within [lo, hi] for any input", verify = "neural", id = "clamp_in_range")]
pub fn clamp(value: i64, lo: i64, hi: i64) -> i64 {
    value.max(lo).min(hi)
}

The assumption records the precondition the function relies on (an interval only makes sense when lo <= hi), and the intent records what the function guarantees under that assumption. The verify field selects how rigorously the claim is checked, from a fast local AI critic (neural) up to machine-checked proof (full). Together they address both halves of the problem: authoring needs no formal language, just plain sentences any engineer can write, and the claims are visible to every teammate and every tool, in the diff, next to the code they describe.

Agent-enabled: Because Aristo ships as agent skills, it works out of the box with coding agents like Claude Code. Your agent writes these annotations as it works, and you only attest to them. Attesting is simple: the skill proposes the claim, and you as the developer accept or reject it, since the final agreement on what the code should do has to involve the developer.

Verification: performing the proof#

Once a claim exists, checking it has to run against the real codebase, and the obstacle there is observability. The state a property talks about (e.g., a private counter inside a cache, or a flag inside a write-ahead log) is deliberately hidden behind the API, and rightly so. Encapsulation is good engineering, but it also means a checker cannot see the state it needs to inspect. Aristo provides observability hooks that expose exactly this state without changing the API you ship: derive-style snapshots that let a harness read private state directly. Beyond observability, Aristo also provides test-only visibility for crate-private functions and labeled fault points for exercising failure paths. All of these hooks are feature-gated, so they compile away entirely in production builds.

#[derive(Inspect)]          // feature-gated: compiles away in production builds
pub struct Cache {
    #[inspect]
    evictable: usize,       // gains inspect_evictable()
    // ...
}

The proving itself runs on Aretta's backend. The same claim can be checked today by a free local critic, then escalated to tests, fuzzing, and machine-checked proof, which is the full service we run with design partners. To make it concrete, here is one guarantee we check for Turso, a production database engine, verbatim:

The cache's materialized evictable-page counter stays equal to the true number of evictable resident pages after every cache operation.

That counter is private state inside the cache, and the guarantee is checkable against the running engine only because the observation points exist. No fields were made public and no fork was needed.

Reporting and maintenance: keeping the proof alive#

The last leg of the loop is the one most verification efforts never build, and it comes down to two abilities: communicating what was proved in terms the developer already speaks, and re-running the proof whenever the code changes.

Aristo handles the first by keeping every claim's status with the code. A claim reads as a plain sentence, its verdict is stored alongside the function, and there is no formal report to decode. It handles the second by hashing each claim against the function body, so that editing verified code flips the claim to stale, triggers the checks to run again, and puts the fresh result in the same place. Proofs go stale as soon as the code changes; keeping them current is what this leg of the loop is for.

Turso, the team building a modern rewrite of SQLite in Rust, is one of our design partners; we wrote earlier about a durability bug we found in their engine. Their deployment shows what this leg of the loop looks like in practice. Every week, the machine-checked guarantees we maintain for them (currently 31, 17 of them on the page cache, and growing over time) are re-checked against the current revision and published as a report their team reads in plain English. When an upstream change breaks a guarantee, its row flips red the same week with a reproduction attached; when it is fixed, the row turns green and the check remains in place as a regression guard.

Where this leaves us#

The provers are ready, as results like seL4 and CompCert show. Adoption has been waiting on the loop's other two legs: specifications that developers can write and read, and results that reach them and stay current as the code changes. Build those two legs, and the provers we already have can finally be applied to the code that actually ships.

Aristo's core is free and open source. The full service, which takes claims all the way to machine-checked proof, is what we run with design partners, and we are onboarding 2 to 3 more: apply to join, or write to us at hello@aretta.ai.

Working with the team at Turso, we found a case where the guarantee had a gap. It's a small bug with a sharp consequence, and a good illustration of what Aristo is for. Turso have written up their own account, generously and in detail, in "The Final Boss of Reliability"; this is ours.

The bug#

Turso can run as a local replica that syncs from a cloud master database. As it commits synced pages, it writes them to disk. But writing to disk isn't the same as making data durable: the operating system can hold your bytes in a cache, and only fsync forces them down to the media. So the property is simple:

A transaction may be reported durable only after its commit frame has reached stable storage via fsync.

On the replica-sync path, that second step was missing. The ordering the property requires, and the ordering the code actually took:

correct   write frame  ──▶  fsync  ──▶  report durable
bug       write frame  ───────────────▶  report durable      (fsync skipped)

So if a crash or power loss lands before that data reaches the disk, a write the database has already acknowledged can be gone on restart: silent data loss, or a corrupted copy of the synced database. No one ever hit it; it was caught first. But it was reachable.

Why it's hard to catch#

So why is something reachable so hard to actually catch? Because the bug lives on the failure path. The happy path is flawless: a commit completes, everyone moves on. It only bites if a crash lands in one narrow window, and ordinary testing never goes there. Tests run on healthy hardware, where the power doesn't cut out between two lines of code; you can't unit-test a bug that only exists mid-crash.

So you reach for stronger testing, and Turso has some of the strongest there is. Their Deterministic Simulation Testing drops the whole system into a simulated world and injects crashes and power failures on purpose, precisely to expose bugs like this. It is genuinely excellent, and unlike ordinary tests, it can reach this class of failure at all.

It still didn't find this one, and not because it's weak. Testing, even this kind, works by sampling: it runs the scenarios its generators happen to produce, out of a space of crash timings and interleavings far too large to enumerate. In Turso's words, the generators "were not generating the particular situation" that triggered it. The scenario was reachable; it just wasn't reached. Extensive isn't exhaustive, and that gap is where this bug lived.

How Aristo found it#

Aristo starts from intent. You state what a piece of code is supposed to do as a plain-language claim, written right above it:

#[aristo::intent(
    "a transaction is reported durable only after its commit frame is fsynced",
    verify = "full",
    id = "commit_is_durable",
)]
fn commit_dirty_pages_inner(/* … */) { /* … */ }

Three parts: the claim in plain English, an id so it can be tracked as the code changes, and a verify mode. The verify mode is the bridge to how the claim gets checked. neural is a fast, local check: an AI critic reads the code against the claim. full hands the claim to Aristo's backend, the engine that reasons about the states the code can reach, not just the ones a test happens to run.

For a durability claim like this one, full is what matters. The backend looked for any reachable execution in which a transaction is reported durable before its frame is fsynced, and it found one. In minutes, and deterministically: the same answer every time, not a matter of the right random seed. How the backend does that is the part we keep to ourselves; what you see is the claim going in and the exact violating case coming back.

That's the shape of the whole product. You declare intent next to your code, Aristo tells you whether the code can break it, and if the code later drifts from the claim, it says so, before the change ships.

A bug is a spec that didn't hold#

What we're really after is a proof: that an acknowledged write survives a crash. You can't establish that in one shot, so the proof breaks it into smaller specs, and those into smaller ones, down to leaves you can check against the code. The tree isn't a checklist written from memory; it's what proving the top spec forces onto the table. Every leaf is there in service of the proof above it.

S   ⟺   A1 ∧ A2
A1  ⟺   B1 ∧ B2 ∧ B3
A2  ⟺   C1 ∧ C2

S   an acknowledged write survives any crash
A1  ack only after the commit frame is on stable storage      (write side)
      B1  frame bytes written to the OS                 ✓
      B2  fsync after the write, before the ack         ✗   ← the bug
      B3  ack issued only after fsync returns           ✓
A2  recovery restores every persisted frame                   (recovery side)
      C1  scan the WAL from the last checkpoint         ✓
      C2  re-apply every frame up to the last commit    ✓

Notice what that buys you. Nobody had to think to test that the replica calls fsync. Proving durability required the obligation "fsync before the ack," so it appeared on the tree on its own, and it was the leaf that came back false. The bug wasn't something we went looking for; it was a spec the proof demanded, and the code didn't hold it.

That's all a bug is here: a spec raised in service of a larger proof, that didn't hold.

(Every split here is an exact ⟺, so a red leaf is a real hole in S, not a dead end in one proof attempt.)

What this doesn't mean#

We checked one guarantee: that acknowledged writes survive a crash. We did not verify all of Turso, and we won't imply otherwise. Verification also doesn't replace testing. Tests exercise the real system on real hardware, and Turso's simulation testing will keep finding things a single guarantee can't. The two are complementary; Turso framed it the same way, and that's the honest read. What verification adds is coverage of the failure paths you can't afford to leave to chance.

Try Aristo, and what's next#

You can try Aristo today. The core, writing intents and checking them locally, is free and open source, so you can start putting claims on your own code right away. The backend proofs that caught this bug (verify = "full") are what we're onboarding design partners for, and as part of our initial launch we're taking on three or four more. If you're building a database, a storage engine, a consensus layer, or anything where a wrong answer on the failure path is unacceptable, we'd like to hear from you.

We're glad to be working with Turso as they head toward their 1.0 release. Getting durability right is easy to overlook and expensive to get wrong, and a team that goes looking for these bugs before their users do is exactly who you want to build alongside.

Join the waitlist: forms.gle/AYoqpeXJq2YQv57S6

Or reach us directly: hello@aretta.ai

The setup#

Both implementations target the same spec: a key-value store with put, get, delete, scan, and close. Writes must survive process crashes (via a write-ahead log), background compaction must never change what queries return, and deleted keys must never come back.

Both received the same architectural constraints: sequence numbers on every entry, a memtable keyed by (key, reverse sequence number), leveled compaction, and bloom filters with no-false-negatives guarantees. These constraints were extracted from Verus proof skeletons to ensure the proofs could apply to either implementation.

The difference is what happened next:

• Claude Code went straight from the prompt to code. No intermediate specification, no proofs — just the architectural constraints and an implementation.
• Aretta first translated the high-level prompt into a low-level formal specification, then built Verus proof skeletons for that specification, and only then implemented the code. The proofs cover 13 machine-checked properties across data-structure invariants and store-level ordering guarantees. (The full list is in the appendix below.)

A torture test suite of 45 tests catches zero bugs#

We wrote 45 tests across seven categories: latest-write-wins, tombstone correctness, scan ordering, crash recovery, compaction transparency, scale, and edge cases. The suite includes overwrite storms of 100,000 puts to the same key, mass deletion of 10,000 keys followed by a full-range scan, and round-trip checks for binary keys containing null bytes. Every test passed on both implementations. This is the part where most teams would ship.

The bug that tests can't find#

LSM databases periodically run compaction — merging and reorganizing on-disk files (SSTables) to reclaim space and maintain read performance. Here is the database compaction code from Claude Code:

// Step 1: Remove old SSTables from in-memory tracking
levels[level].retain(|h| !source_ids.contains(&h.meta.id));
manifest.remove_sstables(&all_removed_ids);

// Step 2: Write merged output to new SSTables
let new_handles = write_split_sstables(dir, manifest, &merged)?;

See full source — agent-cc/src/compaction.rs

Do you see the bug?

Step 1 removes references to the old SSTables. Step 2 writes their replacement. If step 2 fails due to an I/O error, the references are already gone and there's nothing to replace them. From this point on, get() silently returns None for every key that lived in those files.

The data is still on disk. A restart recovers it. But the running process has no idea it's broken. It just serves wrong answers until someone notices.

What the proof found#

Aretta's proof skeleton specifies that new SSTables must be durable before the manifest is updated, and the manifest must be durable before old files are deleted. The underlying invariant is: at every reachable state during compaction, either old SSTables are present or new SSTables are present (or both). Aretta first proved this property on an abstract model of the compaction phases, then used that proof to guide the implementation — ensuring the code maintains the invariant by construction.

When we tried proving this property on Claude Code's generated code, Verus found the property to be violated during compaction.

Confirming the bug: fault injection#

To confirm this causes real harm, we built a fault injection hook that fails the N-th SSTableWriter::finish() call with a simulated ENOSPC error.

pub fn fail_nth_sst_finish(n: u32) {
    SST_FINISH_COUNTDOWN.with(|c| c.set(n));
}

See full source — agent-cc/src/testing.rs

We wrote four sentinel keys into separate L0 SSTables, then triggered compaction with the fault armed. On Claude Code's implementation, all four sentinel keys vanished from the live instance — get() returned None for every one of them. A reopen recovered the data (the files were still on disk), but the running process had been silently serving wrong answers with no indication of failure.

On Aretta's implementation, the same fault injection had no effect. All four sentinel keys remained visible throughout. The proof-guided implementation writes output first, then calls manifest.atomic_swap() which only updates in-memory state after the new SSTables are confirmed on disk:

// Step 1: Write new SSTables (confirmed on disk)
let new_sstables = self.write_level_sstables(manifest, merge_iter, 1)?;

// Step 2: Atomic manifest swap — ADDs before REMOVEs, fsync,
//         THEN update in-memory state
manifest.atomic_swap(new_sstables, old_file_numbers)?;

// Step 3: Delete old files (manifest already durable)
for entry in old_entries { fs::remove_file(...)?; }

See full source — agent-a/src/compaction.rs

If the write fails at step 1, atomic_swap is never called and the in-memory state is unchanged. The invariant holds at every phase.

Why this bug is hard to catch#

This isn't a logic error in the traditional sense. The happy path is correct — when compaction succeeds, the store works perfectly. The bug only manifests when an I/O operation fails between two specific lines of code. That makes it invisible to every standard testing technique:

• Unit tests only see the happy path. Compaction works, data looks right, test passes. Nobody writes a test that says "now make the disk full between these two lines."
• Integration tests run on healthy local SSDs. The I/O failure that triggers this bug simply never happens in CI.
• Fuzz testing throws random keys and values at the API. It doesn't fuzz the filesystem. You could fuzz for a year and never trigger ENOSPC at the right moment.
• Code review reads "clear old state, write new state" as perfectly reasonable cleanup. Its very easy to miss the two-line gap between remove and write.

The proof catches it because it reasons about all reachable states, not just the states that tests happen to visit. It asks: "is there any point during compaction where the invariant doesn't hold?" And it finds one.

What this means#

Tests are essential, and the 45-test torture suite we built provides real value as ongoing regression coverage. But tests exercise the code with working disks, successful writes, and cooperative hardware. The bugs that matter in production — the ones that cause silent data loss at 3 AM when a disk fills up — live in failure paths that no test in our suite could reach.

Formal verification works differently. Rather than running the code and checking outputs, it reasons about the full semantics of the program. The Verus proof examined every reachable state during compaction and determined that the invariant is violated at a specific program point. It did not need to simulate a disk failure to reach that conclusion.

Neither technique subsumes the other. Tests caught bugs that the proofs did not cover (like the missing file lock and the scan panic on empty input). The proof caught a bug that tests could not reach. In practice, the two approaches are complementary: tests cover breadth across the API surface, while proofs cover depth on critical invariants. Using both together gives stronger assurance than either one alone.

The full proof trees are available for Aretta here and Claude Code here.

The full source code, proof skeletons, and fault injection tests are available on GitHub. If you're building infrastructure that needs stronger correctness guarantees than testing alone can provide, reach out.

Appendix: proof properties#

The 13 machine-checked properties verified by Aretta's Verus proof skeletons: