Guarantees
This page lists what Speccy guarantees, and the test that proves each guarantee.
Each row names a test in the Speccy repo. just verify runs every test on each pull request, so a guarantee on this page holds on main. A test that stops passing fails the gate before the change merges.
The verdict
Section titled “The verdict”| Guarantee | Test |
|---|---|
| An open MUST finding gives Not Build Ready. | TestVerdict_OpenMustBlocks |
| A valid waiver on the only MUST finding gives Build Ready. | TestVerdict_WaivedMustPasses |
| An open blocking thread gives Not Build Ready. | TestVerdict_BlockingThreadBlocks |
| The verdict of an old version reads Stale. | TestVerdict_OldVersionIsStale |
| SHOULD findings never change the verdict. | TestVerdict_ShouldNeverBlocks |
| The verdict function is pure: the same input gives the same output. | TestVerdict_Deterministic |
| An edit keeps the AI findings of the sections it did not touch, and drops the findings of a changed section. | TestAnEditCarriesTheAIFindingsOfUnchangedSections |
Waivers and approvals
Section titled “Waivers and approvals”| Guarantee | Test |
|---|---|
| A waiver ends when its section changes. | TestWaiver_InvalidatedOnSectionEdit |
| Speccy applies each value of the waiver policy. | TestWaiverPolicy_Table |
| An author cannot approve their own bundle. | TestApproval_AuthorCannotApprove |
| A change of the content revokes the approvals. | TestApproval_EditRevokes |
Profiles and adoption
Section titled “Profiles and adoption”| Guarantee | Test |
|---|---|
A mapped file with no frontmatter takes the mapped profile, and a frontmatter type wins over the mapping. |
TestConfig_PathMapping |
| A relaxed check reports at INFO and never blocks. Removing it from the list restores its level. | TestAdoption_RelaxedCheck |
| Each run records the profile version it used. | TestRun_PinsProfileVersion |
The review stages
Section titled “The review stages”| Guarantee | Test |
|---|---|
| Lint finishes a doc of 10,000 words in under 1 second. | BenchmarkLint_10kWords |
| Speccy retries invalid JSON from a model once, then the step fails. | TestModel_InvalidJSONRetryOnce |
| Instructions in a doc do not change the verdict. | TestInjection_DocCannotChangeVerdict |
| Instructions in an MCP result do not change the verdict. | TestInjection_MCPResultIsData |
| An unverified claim is a SHOULD finding. A contradicted claim is a MUST finding. | TestGrounding_Labels |
| Speccy sends an unchanged section to no model again. | TestCache_UnchangedSectionReused |
| The most specific class rule of a source policy wins. | TestClassOfMostSpecificWins |
A forbidden host stays forbidden, even when an allow pattern covers it. |
TestCheckHostForbidBeatsAllow |
| Speccy never changes a doc until the author accepts the fix. | TestSuggestFix_RequiresAccept |
| An anchor follows an edit, or detaches. It never points at the wrong text. | TestAnchor_Reanchor |
Divergence
Section titled “Divergence”| Guarantee | Test |
|---|---|
| Readers who give different answers make a divergence finding. | TestDivergence_SplitIsFinding |
When every reader answers NOT SPECIFIED to a MUST question, Speccy makes a MUST gap finding. |
TestDivergence_GapOnMust |
An answer with a quote that is not in the bundle counts as NOT SPECIFIED. |
TestDivergence_InventedQuoteRejected |
| One model for all readers gives the note “Low reader diversity”, and the note does not block. | TestDivergence_LowDiversityFlagged |
| A reader never receives the answers of another reader, or the rubric. | TestDivergence_ReaderIsolation |
Coherence
Section titled “Coherence”| Guarantee | Test |
|---|---|
| An upstream REQ that the doc does not cover is a MUST finding. | TestCoherence_UncoveredReqIsMust |
| A standalone acknowledgement makes coherence not apply. | TestCoherence_StandaloneAck |
| An upstream edit makes the downstream verdicts stale. | TestCoherence_UpstreamEditStales |
| A restatement above the threshold is a SHOULD finding. | TestCoherence_RestatementShingles |
| A link rule makes a link only when both files exist. | TestConfig_LinkRules |
Verification runs
Section titled “Verification runs”| Guarantee | Test |
|---|---|
| A target holds only when its anchor quote matches exactly once in the file. | TestCheckNeedsExactlyOneMatch |
| A judge that affirms nothing gives the outcome unproven, and unproven never blocks. | TestDecideNeedsAnAffirmation |
| A breach needs two judges that agree. | TestDecideBreachNeedsTwoJudges |
Storage and access
Section titled “Storage and access”| Guarantee | Test |
|---|---|
| Both store engines, SQLite and Postgres, pass the same conformance suite. | TestStoreConformance |
| The event store refuses a concurrent append with an old version. | TestEventStore_ConcurrentAppendRejected |
| Projections change in the same transaction as the append. | TestEventStore_InlineProjectionAtomic |
| Speccy stores no secret in plain text. | TestSecrets_EncryptedAndHashedAtRest |
| Local mode refuses an address that is not a loopback address. | TestLocalMode_LoopbackOnly |
| Each endpoint applies its table of roles. | TestAuthz_EndpointRoleTable |
| A person who joins through a share link cannot edit the doc or ask the AI. | TestGuest_Restrictions |
The CLI and the GitHub Action
Section titled “The CLI and the GitHub Action”| Guarantee | Test |
|---|---|
The exit codes of speccy review are 0 for Build Ready or advisory mode, and 1 for Not Build Ready in blocking mode. A usage error is 2, and a run error is 3. |
TestCLI_ExitCodes |
speccy review --summary works with no server and no speccy init. |
TestCLI_SummaryNoSetup |
| In advisory mode, a verdict never fails the job of the Action. | TestAction_AdvisoryNeverFails |
| Inline comments go only on changed lines, keep to the limit, and never appear twice. | TestAction_InlineComments |
| A suggestion block carries only a fix that needs no model. | TestAction_SuggestionsDeterministicOnly |