A Portfolio of Oracles

Every engineer has seen the pyramid. Lots of unit tests on the bottom, fewer integration tests in the middle, a handful of end-to-end tests balanced precariously on top. It is drawn on whiteboards, printed in onboarding decks, and cited in code review as though it were a law of nature.

I have come to think it is a mildly useful diagram of the wrong thing. The pyramid describes cost: cheap tests at the bottom, expensive ones at the top, so keep the expensive ones few. What it does not describe is capability. It never asks which defects each layer can actually see. And that, quite frankly, is the only question that matters when you are deciding whether a layer earns its keep.

So over the last couple of years, across a few of our own projects, we have replaced the pyramid with something I have taken to calling a portfolio of oracles. The worked example throughout is yasss, Yet Another Service Scheduling System — one of our open-source applications, offered as a product through CrowdEase. The written version of the methodology lives in reaper, our ephemeral test-VM harness. An oracle, in the testing sense, is whatever tells you whether the system did the right thing. The portfolio framing says: every tier you keep must answer a question that no cheaper tier can answer. If it cannot, delete it.

This post is the whole methodology, tier by tier, with the reasoning attached, and then where it has paid off — including as the framework a coding agent works inside when one of us pairs with one. It is written to be transferable. The yasss details are there because a concrete example beats an abstract one, not because you should copy them.

The organizing question

Here is the portfolio as a table. The right-hand column is the entire justification for the left-hand one.

Tier	The question only it can answer
Pure unit	Is this calculation right, at its boundaries?
Cross-language vectors	Do two independent implementations agree?
Component conformance	Did the rendered output drift?
Source-as-data	Does the code's structure still hold its claims?
Server contract	Do the authorization and enum rules hold, in isolation?
Browser vs. fake API	What does the app do when a request fails?
Full stack	Does it work against a real database, real mail, real charset?
Seeded fuzz	Does any ordinary-but-untried request crash it?
Concurrency	Does the invariant survive simultaneous writers?
Simulated users	What breaks only after history accumulates?
Live browser audit	Does the UI hold up over a world somebody else built?
Human evidence	Does this actually make sense to a newcomer?

The table says nothing about how many of each you should have. That is the pyramid's question, and it comes second: work out what each layer can see, and the counts follow.

The rules that matter more than any test

Before the tiers, the process rules. These have caught more real problems than any individual suite, because they govern what happens on the day a test turns red and somebody is in a hurry.

Never weaken a test, check, assertion, or lint to route around a defect. Not by disabling a phase, adding an exclusion, lowering a threshold, marking skip or ignore or allow, appending || true, catching and swallowing, or quietly narrowing a suite's scope. Every one of those is a permanent decision about what the project stops noticing, made under exactly the conditions — deadline, fatigue, red terminal — in which nobody should be making permanent decisions.

Every narrowing needs a stated reason, and the reason must cover exactly what it narrows. A lint exclusion that silences four warnings behind a comment justifying two of them is a bug in the justification, and two of those warnings are now suppressed with nobody's sign-off. Narrowings get reported in the human-facing summary, not buried in a code comment where the next person will treat them as furniture.

Every fix ships with a test that would have caught it. Where a change is honestly untestable in isolation — a dead-code removal, a transposed log argument, a version bump — say so explicitly. Do not invent a test that asserts a constant so that the pull request has a green checkbox next to "tests added."

A pre-existing failure must be proven pre-existing. Stash the working tree, re-run, and name it in the report. Otherwise "that one always fails" becomes the sentence that hides a regression, and it will hide one.

New assertions get mutation-checked. After you write a test, break the thing it covers and confirm the test fails. A test that has never been observed failing has unmeasured value. This one habit has repeatedly caught assertions that were accidentally tautological: matching an explanatory comment rather than the code, or matching the test's own fixture. The suite was green the whole time.

Fix the cause, not the symptom. A quiet terminal is not the goal. Optimize for it directly and you will get it by the cheapest available means, which is rarely the correct one.

On to the tiers.

Tier 1 — Pure unit tests, and the vector file

No DOM, no network, no framework. Pure functions and plain data: arithmetic, validation, serialization, parsers, layout math, date handling.

Two habits pay disproportionately here. First, keep testable logic out of components. Label strings, state vocabularies, and layout arithmetic live in plain modules so they can be asserted without a renderer and shared between the app and its tests. When a test needs a DOM just to check a string, that is a design smell, not a test problem. Second, test the boundary, not the middle. Off-by-one. Empty, one, many. The cap, and the cap plus one. The value that lands exactly on the interval.

There is a refinement to this tier that I think is underused: cross-language golden vectors. Where two implementations must agree — a client and a server both computing a credential derivation, a code alphabet, a signature — neither one is the oracle. A committed vector file is: fixed inputs, fixed expected outputs, consumed by both sides' suites. It catches the failure mode where each implementation is perfectly self-consistent and they disagree with each other, which no amount of testing either side alone will ever find.

There is a second trick here that I would steal outright: run the interop vectors a second time under a hostile environment default. In yasss the credential vectors run once normally and once with a non-UTF-8 default charset. The reason is that under a UTF-8 default, the suite passes whether or not the encoding is actually pinned in the code. The ordinary run cannot see the bug at all.

Tier 2 — Component conformance

Render a component; assert its output exactly. Whole-string equality on class attributes. Exact element counts. Exact text.

This is unusual — most advice says assert behavior, not markup — and it is right here for one specific reason: for a design system, the rendered structure is the contract. is-outlined is-light and is-light are different components on screen, and a containment check happily accepts either.

The consequence has to be accepted honestly rather than worked around: these tests make markup expensive to change, which is the point, but it means you add a sibling component rather than generalizing an existing one. Growing a well-covered component a set of props to serve a second use case puts every one of its assertions up for renegotiation. A sibling that shares the logic modules costs a little duplicated markup and keeps the conformance suite meaningful. And when a shared component genuinely must gain an option, gain it in a form that renders nothing at all when unused, so the existing output is byte-identical and "the old suite passes untouched" becomes the acceptance gate on the change.

Tier 3 — Source-as-data

This is the least conventional tier, the cheapest to run, and the one that catches rot: code that is still correct about a product that has since changed.

The technique is to parse the project's own source as data and assert structural claims about it. No browser, no framework, no runtime. Some of the claims this tier holds for us:

• every declared item has corresponding content, and no content is orphaned (rename a key and its copy silently vanishes otherwise);

• every selector referenced anywhere names a hook the application actually produces — attribute and value, because a numeric value like "0" matches almost any file on its own;

• a tutorial step that points at a form field must also open the dialog that field lives in — the thing being described must be on screen;

• a control named in user-facing copy must exist verbatim somewhere in the application;

• every emphasized span in copy is classified as either a control name or prose, so a new one has to be classified by whoever wrote it instead of quietly landing in whichever bucket a heuristic guessed.

That last group takes a subjective question — is this documentation coherent? — and turns it into a set of mechanical ones. It will not tell you whether a sentence lands. It will tell you that the sentence describes a button that no longer exists, or a screen the reader is not on, and in my experience that is most of what actually goes wrong with docs.

Two design rules keep this tier from decaying into self-agreement. Assert against the source, not against a re-export of it — a check derived from the same structure it is checking will agree with itself no matter what, so where a test needs a mapping the app also has, duplicate it deliberately in the test and let the duplication be the check. And exclude the content being checked from the corpus being searched, or copy can satisfy a check by quoting itself.

Tier 4 — Server contract tests

Fast, in-process, no database. The targets are the rules that are pure functions in disguise.

The big one is the authorization matrix: every role, times every resource state, times every operation, as a table. The payoff arrives at refactor time. When a second entity type gains the same ownership semantics as the first, the existing rows passing unchanged is the gate on the refactor — and that is what stops the rule from existing in two copies that drift apart. Alongside it: enum and boundary decoding (ordinals that clamp rather than throw, string bounds, id parsing), and visibility rules asserted as pure functions so the expensive tiers are not enumerating thirty-six cases through HTTP.

Tier 5 — A real browser against a fake API

Real browser, real application build, fake server in-process. Fast, deterministic, parallelizable, and it runs the cross-engine matrix. This is where the bulk of UI behavior gets asserted.

Its unique capability is transport error injection. Route interception lets any request fail on demand:

route → 500 { status: "error", info: "boom" }

Failure paths in a well-built client are written deliberately: toast, return false, and only then update the local model, so a failed save leaves the screen showing the truth rather than an optimistic lie. Without injection, none of that code is ever exercised. A real server will not produce a 500 when you ask it to. A fake one will produce exactly the one you asked for.

Injection belongs on the fake tier, not the live one, and the reason is a small instance of the rules above. The live suite's watchdog fails any test that observes a 5xx, and that watchdog is what makes the live suite worth running. Injecting faults there would mean disabling it — which is precisely the kind of narrowing the rules forbid.

The fake server must also import the real implementations of anything that normalizes or validates — code normalization, id parsing — rather than re-implementing them. Otherwise the fake drifts into testing itself, and you have two systems that agree with each other and nothing else.

Tier 6 — Full stack, containerized, staged

A real database, a real mail catcher, the real application artifact, all in containers, driven by one script with named stages that can be run individually. In yasss the stage list looks like this, with one additional stage per feature where <feature> sits:

fuzz  accounts  sessions  reminders  text  <feature>  concurrency  regressions  journeys  browser  health

Three habits from the yasss battery that transfer to anything:

Start the environment hostile. The database boots in latin1, not the modern utf8mb4 default, because the schema's charset migration is only load-bearing on a server that does not already do the right thing. Boot in the friendly configuration and the migration and its assertions pass whether or not they work — the same shape of blindness as the charset vectors in Tier 1. Then push astral-plane text through the newest text column in every feature stage, on the theory that the newest column is the one somebody forgot.

Assert idempotence of anything that re-runs. Migrations execute on every boot, so a stage asserts that a second boot rebuilds nothing and that backfills are no-ops the second time.

No backdoors. State is built through the real API and the real mail flow — registration, emailed verification link, sign-in — because a seeding shortcut tests a path production does not have.

Tier 7 — Seeded fuzzing

Cheap, general, and aimed at a specific empirical observation from our own defect history: most defects found in practice were unguarded dereferences surfacing as 500s. A UUID belonging to a different parent. A null actor on an anonymous request. An empty collection. A validator that was never constructed. Not exotic inputs. Ordinary requests that nobody happened to try.

So the oracle is not "is the answer correct" — that is expensive and specific — but the far cheaper and more general:

1. no 5xx, ever;

2. every response is a well-formed envelope;

3. the server is still alive afterwards.

The corpus targets seams that have actually broken: type confusion at the deserializer, unparseable ids, boundary values on unsigned columns, oversized strings, and text that is only a problem if something concatenates it into SQL or HTML.

Determinism is the whole game. The seed is printed on every run and replayable through an environment variable. A fuzzer you cannot replay is a fuzzer you cannot act on. And the corpus is shared upward, so the same hostile values reach the API directly and through the UI without anyone re-typing them.

Tier 8 — Concurrency harnesses

Targeted, scenario-driven, N-way simultaneous requests against one resource. The design decision that matters here is what to assert, and the obvious choice is the wrong one.

Say an event has twenty seats. Ten actors fire RSVPs at the same instant against a nearly-full event. The tempting oracle is to count how many requests came back 200 and check it matches the remaining capacity. That oracle passes a system that answers correctly and stores wrongly — one that returns the right number of acceptances and then persists twenty-three attendees. So the harness instead reads the resource's own state back afterwards: the count the user will see, and the count the rule is about.

Scenarios cover both the race and its non-racing sibling, for a reason we learned the hard way: the same capacity rule was enforced in one endpoint and not in another, and the second one needed no concurrency at all to violate.

Tier 9 — Simulated users

This is the tier with the most machinery, and the one I get asked about most, so it gets the most room.

The shape: several actors, each with a real account and a real session, take turns doing whatever they are currently able to do, for hundreds of actions. A shadow model records what should be true. Invariants diff the model against the server as the run proceeds. The whole thing is driven by a seeded PRNG, so it is quasi-nondeterministic: the action sequence looks random and is fully reproducible from a seed. Anything that must vary between runs — a tag distinguishing this process's data from an earlier run's — is drawn outside the seeded stream, so replay reproduces the sequence exactly.

Six parts, borrowed loosely from the Jepsen vocabulary:

Part	Role
Generator	Seeded PRNG choosing a weighted action from those currently applicable
Actors	Independent identities with their own rotating credentials
Shadow model	A deliberately partial second copy of the truth
Checker	Invariants diffing model against server, periodically and at the end
Nemesis	Adversarial action classes
Shrinker	Reduces a failing run to something a person can read

What this tier sees that nothing else can is state rot: the bug that only exists after history accumulates. A listing that returns a duplicate once a user has enough of something. A permission that was correct on a fresh account and wrong after a role change three hundred actions ago. You cannot write that test by hand, because you do not know which history triggers it. You can only generate histories and check invariants against them.

The nemesis is not a mode

Error injection here is not a separate switch; it is a set of action classes in the same weighted pool, so faults land at arbitrary depths in an accumulated history rather than against a clean fixture:

Class	What it simulates
Stale credential	A second tab holding a retired session ticket
Act-on-deleted	A page rendered before someone else deleted its subject
Hostile input	The shared fuzz corpus, aimed at a write endpoint
Double submit	Save pressed twice before the first answer returns
Typo-then-correct	A value corrected, checked later from another actor's view
Abandonment	Half-finished work left behind

Each asserts the coherent outcome rather than a specific one. A stale ticket must produce a refusal, not a crash, and must not disturb the live session. Two simultaneous submits may both legitimately succeed — what is asserted is that the model and the server agree afterwards about how many there are.

Shrinking

A seed tells you a bug exists; a trace tells you what it is. On failure, the shrinker first bisects the length — the shortest prefix of the same seed that still fails — and then removes action kinds one at a time, keeping each removal if the run still fails. That answers "is this action actually involved, or merely present," which is the question you would otherwise spend an afternoon on.

One caveat: shrink re-runs land on a stack that already holds the first run's data. So a successful reproduction is a strong signal, and a failure to reproduce is a weak one. The output says so, rather than papering over it.

Self-testing the oracle

If you take one sentence from this post, take this one. An invariant that never fires is indistinguishable from a passing suite. It is exactly the mistake that leaves a whole team believing they are covered.

So the invariants are fed the responses a broken server would send — modeled on the real defects that motivated them — and each one must complain. This runs with no stack at all, in about a second, and it is the check that makes the expensive stage mean anything. It is mutation testing pointed at the oracle instead of the code.

The suite runs a small committed set of fixed seeds by default so the cost is known; long hunting runs are opt-in; and any seed that ever found a defect is promoted into the fixed set permanently.

Tier 10 — Live browser audit over an accumulated world

The browser suite from Tier 5, pointed at the real stack, run as an actor who has accumulated something — hundreds of prior actions' worth of world, courtesy of Tier 9.

It sees two things nothing else can. First, behavior that only appears once a user's own data is deep: a paging control that only exists past a threshold, a listing that misbehaves at scale. Second, whether the deployed configuration is actually being read — which a fake answering generically cannot distinguish from a real server that is ignoring its config.

Two oracles for one claim

Where a claim really matters, assert it from both sides. The tutorial's containment claim in yasss — nothing the practice mode does reaches the server — is checked by counting the browser's requests, and afterwards by asking the database whether anything arrived. The first proves the page made no call. The second catches a leak by a route nobody thought to watch. Neither is sufficient alone.

Tier 11 — Human evidence

Some questions cannot be automated and should not be faked. "Would a first-timer understand this?" is a judgment call. What automation can do is produce the evidence rather than the verdict: a per-step transcript and screenshot, written to a directory, so a person answers from evidence instead of memory.

Alongside that, a zero-dependency reader that prints the user-facing content as continuous prose, annotated with the structural transitions a reader cannot infer from the words — where the page changes, where a dialog opens. Reading a flow cold for five minutes finds more than any checker written for the purpose.

And on color: computed, not eyeballed, and not pixel-diffed. Resolve the painted colors through the browser's own computed styles, convert to relative luminance, and assert the distinctions survive with hue discarded — in the same units the accessibility guidelines use. That answers "is this conveyed by color alone" as a number and names the element that failed. A pixel diff would say "these images differ by 3%" and demand a regenerated baseline on every legitimate design change and every font substitution between machines.

Cross-cutting practices

A few things every tier shares.

Shared corpora. One hostile-input list, one set of label constants, consumed by every tier that needs them. A value that broke the API should reach the UI too, without anybody re-typing it.

Determinism and replay everywhere randomness appears. Print the seed. Accept it back through the environment. Keep anything that must vary out of the seeded stream.

Assert the absence of things, with time allowed to pass. Proving nothing happened needs a wait. And assert the requests before the success indicator, so a leak presents as "it made this call" rather than as "the toast never appeared" thirty seconds later, three inferences from the cause.

Name what a test does not prove. Where a suite's name overstates it — a walk through a flow that touches nothing proves passivity, not containment — write that down in the file and point at the test that carries the stronger claim.

Match effort to change. A comment does not warrant the containerized stack. A schema change does, and it is not optional. Say which suites were skipped and why.

If you are starting from zero

In order of return on effort:

1. Pure unit tests on logic deliberately kept out of components.

2. Seeded fuzz with the no-5xx / well-formed / still-alive oracle. The highest defect-per-line ratio of anything here, and about a weekend to build.

3. Browser-vs-fake with transport error injection. The failure paths are usually the least-exercised code you have.

4. Full stack, staged, started hostile.

5. Source-as-data structural checks, once the project has structures that can rot: routes, permissions, copy decks, generated clients.

6. Concurrency harnesses for each rule with a capacity or uniqueness constraint.

7. Simulated users, last, because it is the most machinery — and write the oracle self-test first, or you will not know whether it works.

And the acceptance test for the entire exercise, which I would put above everything else in this post: take defects you have already found and fixed by hand, revert the fixes, and confirm the harness rediscovers them. A methodology that cannot rediscover your known bugs is not yet measuring anything, however green and however extensive it is.

Where this earns its keep

A methodology is only interesting if it changes what you ship. Here is where this one has.

When the team chooses to pair with a model

Developers do not like to test. It is the reason QA departments exist, and it is the real danger of vibe coding: the model's code is usually fine, and the person driving it does not check it. Add to that the fact that language models are amplifiers, magnifying whatever you seed them with, and you get the actual problem with agentic coding. A model asked to "add tests" will add tests. Whether those tests can fail is a separate question, and it is the question nobody asks, because the terminal is green and the diff is long.

When an engineer on our team decides a language model is the right pair for a given piece of work, the portfolio is what makes that a sound decision. Every rule in it was written for a human under pressure who wants the red to go away — and a coding agent is exactly that, minus the fatigue. So the rules transfer without modification. What changes is how you deliver them.

A build delegated to a coding agent goes out as a plan with testing-methodology.md vendored into the repository and marked normative. Each phase of the plan lists its deliverables, the test additions by tier, and an acceptance gate the agent has to clear before an engineer signs off on the phase. The agent reads the methodology the same way a new hire would, and it is held to it the same way. A few of the rules do disproportionate work in that setting:

The non-negotiables are the guardrails an agent cannot route around. When a test goes red mid-run, an agent's cheapest path is the same as a tired human's: mark it skip, append || true, narrow the suite. The methodology forbids every one of those by name, and it requires the narrowing — if one is truly needed — to be justified in the human-facing summary. In practice that means the agent stops and reports rather than quietly editing the project's field of view. The rule that every fix ships with the test that would have caught it works the same way: it converts "I fixed it" into "here is the test that was red and is now green," which is a claim a reviewer can check in thirty seconds.

Mutation-checking is how you find out whether the agent's assertions mean anything. Models are very good at writing an assertion that matches the comment above the function instead of the function. The rule that every new assertion is broken once and observed failing catches this mechanically, and it is cheap enough that the agent can do it on its own before it reports. Same for the oracle self-test: an agent-written invariant suite is fed the responses a broken server would send, and each invariant has to complain. Without that step you have a simulated-user tier that is elaborate, green, and blind.

The cheap oracles let the agent check work it does not fully understand. No-5xx, well-formed envelope, still alive: none of those require knowing the right answer. That matters when the agent is working in a codebase it did not write, on a feature whose correct behavior lives in a product decision it was not part of. It can still run the fuzz stage. It can still assert the resource's state read back after a concurrency scenario. It can still ask whether the second boot rebuilt anything. Those tiers give a model a way to be rigorous about things it is not yet expert in.

The rediscovery test is the acceptance gate for the harness the agent built. Once the project has real, fixed defects, revert one and see whether the battery finds it. If it does not, the harness is not done, however much of it there is. This is the one check I would never let an agent — or a person — skip, because it is the only one that measures the harness rather than the code.

Phase gates are where a second reviewer earns its keep. On those builds, implementation goes to one model and phase-gate review to another before it reaches a person — the reviewer reads the phase's tests against the methodology and against the mutation-check results, not against the implementer's summary. This is the same reason a book has an editor: familiarity with the draft is what prevents seeing what is wrong with it.

The practical result is the thing people find hard to believe until they have watched it: given this framework, a model can produce a full test battery in one pass, and the battery is trustworthy, because the methodology gives it a way to check itself that does not depend on it already knowing the answer. That self-check is the line between vibe coding and AI-driven pair programming: on one side, a model producing code that nobody verifies; on the other, a model working inside an architecture that was designed, by engineers, to verify it.

The ephemeral machine that makes it affordable

The expensive tiers — full stack, concurrency, simulated users, the live browser audit — need a real database, real mail, real containers, and a way to roll back between runs. Nobody runs those on a laptop before every push, whether the one pushing is a person or an agent. This is what reaper is for: it syncs the uncommitted working tree into a disposable VM, runs the battery there, rolls the guest's state back by ZFS snapshot in a few seconds, and tears the machine down. The loop — for an engineer or for an agent — becomes edit, sync, run, read results, and the cost of running Tier 6 through Tier 10 drops to something you do on every iteration rather than on release day.

The two are designed together. The methodology says what to prove; reaper is where you afford to prove it; and whoever is doing the proving, human or model, works the same loop.

Handoffs, reviews, and inheriting somebody else's mess

The same document does a second job on the consulting side. When a client hands us a system in disrepair, the portfolio's first question — which defects can each existing suite actually see? — is a faster audit than reading the tests. Usually the answer is that the existing suite can see calculations and cannot see failure paths, concurrency, or state rot, and that tells you where the next month goes. The order-of-return list above is, more or less, our engagement plan for those projects.

And on review: a pull request that says which tiers it touched and which it skipped, and why, is a pull request you can review in proportion to its risk. A schema change that skipped the containerized stack is a review that starts with a question. A comment fix that ran only the unit suite is fine, and the author said so.

Closing

None of this is a pyramid, so we've stopped drawing one. What we have instead is a list of questions, a tier that can answer each, and a set of rules about what happens when a tier says no. The questions are the part that transfers; the tiers are how we happen to answer them on our own projects, and yours may differ. The rules do not. And whatever you build, hold it to the same test we hold ours to: revert a bug you already fixed, and see whether it gets found. If it does not, you have a green suite and no oracle, and the difference is the whole subject of this post.

Lineage

We say on the front page of our site that most technical problems were solved years ago by someone who isn't charging you for it, and this methodology is no exception. The portfolio is ours; most of the oracles in it are not. Here is where each came from, as far back as I can trace it.

The framing. "Test oracle" is William Howden's term, from Miller & Howden, Software Testing and Validation Techniques (IEEE, 1978); Elaine Weyuker's "The Oracle Assumption of Program Testing" (1980) is the follow-on, and Barr, Harman, McMinn, Shahbaz & Yoo, "The Oracle Problem in Software Testing: A Survey" (IEEE TSE, 2015) is the modern survey. The pyramid I am arguing against is Mike Cohn's, from Succeeding with Agile (2009), popularized by Martin Fowler's 2012 bliki entry.

Tier 1, boundaries. Boundary value analysis and equivalence partitioning are Glenford Myers, The Art of Software Testing (1979); Boris Beizer, Software Testing Techniques (1990), for the second pass.

Tier 1, golden vectors. The practice is borrowed from cryptography: NIST's CAVP test vectors and the vectors published in RFCs (RFC 4231 for HMAC-SHA2 is the canonical example), with Google's Wycheproof (2016) as the adversarial form. The principle that neither implementation is the oracle is William McKeeman, "Differential Testing for Software," Digital Technical Journal (1998).

Tier 2, exact-output conformance. Characterization tests are Michael Feathers, Working Effectively with Legacy Code (2004); approval testing is Llewellyn Falco's ApprovalTests (c. 2008); Jest's snapshot testing (2016) is the form most people have met.

Tier 3, source-as-data. The nearest named ancestor is the architectural fitness function — Ford, Parsons & Kua, Building Evolutionary Architectures (2017) — and tooling in that family (ArchUnit, NDepend). Extending it to copy decks and tutorial steps is our own stretch; I have not found a prior source for asserting that documentation names real controls.

Tier 4, server contracts. Consumer-driven contracts are Ian Robinson, "Consumer-Driven Contracts: A Service Evolution Pattern" (2006), and Pact (2013). The authorization matrix is plain table-driven testing, which is folk practice; the Go project's testing documentation is the usual citation.

Tier 5, transport error injection. Software fault injection is Voas & McGraw, Software Fault Injection (1998). The operational lineage runs through Netflix's Chaos Monkey (2011) and the Principles of Chaos Engineering (2015). Route interception is a Playwright feature, not a paper.

Tier 6, full stack in containers. Testcontainers (Richard North, 2015). "Start the environment hostile" has no originator I can point to; it is Weyuker's oracle assumption applied to the environment rather than the code — a friendly configuration cannot observe the defect.

Tier 7, seeded fuzz. Barton Miller, Lars Fredriksen & Bryan So, "An Empirical Study of the Reliability of UNIX Utilities," CACM (1990) — which coins "fuzz" and uses exactly the oracle above: it did not crash, it did not hang. AFL (Michał Zalewski, 2013) for the coverage-guided form. Our observation that most defects were unguarded dereferences is a close echo of Miller's original findings.

Tier 8, concurrency. Linearizability is Herlihy & Wing, "Linearizability: A Correctness Condition for Concurrent Objects," TOPLAS (1990). "Assert the state read back, not the responses" is the lesson of Jepsen (Kyle Kingsbury, 2013 onward).

Tier 9, simulated users. Four sources feed this one. Stateful property-based testing: Claessen & Hughes, "QuickCheck," ICFP 2000, and John Hughes's "Testing the Hard Stuff and Staying Sane" (2014) for the state-machine form; Hypothesis (David MacIver) for the idea that seeds which found a defect are kept forever. Deterministic simulation testing: FoundationDB, presented in Will Wilson's 2014 Strange Loop talk, with TigerBeetle's VOPR and Antithesis as descendants. The generator/nemesis/checker/history vocabulary: Jepsen. Shrinking: QuickCheck, with the systematic form being Zeller & Hildebrandt, "Simplifying and Isolating Failure-Inducing Input," IEEE TSE (2002). The shadow model is model-based testing, which goes back at least to Dalal et al., "Model-Based Testing in Practice," ICSE 1999.

Tier 9, the oracle self-test — and mutation-checking every assertion. Mutation testing is DeMillo, Lipton & Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," IEEE Computer (1978); Richard Lipton's 1971 class paper is the folklore origin. PIT and Stryker for tooling. Pointing it at the oracle instead of the code is our inversion.

Tier 11, human evidence. Heuristic evaluation is Nielsen & Molich, "Heuristic Evaluation of User Interfaces," CHI 1990; the think-aloud protocol is Ericsson & Simon, Protocol Analysis (1984). The luminance arithmetic is WCAG 2.0 (W3C, 2008), success criterion 1.4.3 and its relative-luminance definition.

The rediscovery test. This has the best pedigree of anything here. "Bebugging" first appears in Gerald Weinberg's The Psychology of Computer Programming (1970), and Harlan Mills's fault seeding — "On the Statistical Validation of Computer Programs," IBM Federal Systems Division (1972) — formalized inserting known faults to measure what a test process catches. Reverting real fixes instead of planting synthetic faults is, I would argue, the honest descendant.

---

The methodology document is docs/testing-methodology.md in axonibyte/reaper. The yasss implementation it cites is at axonibyte/yasss.