First case set
doc-coauthoring
Trigger precision
stayed quiet when it should have in 30 of 30 attempts
100%
95% confidence between 89% and 100%. 30 attempts settle it to a narrow range.
Why one trigger scoreis not a measurement
Assay prints a trigger accuracy figure, and it is tempting to read that figure as the skill works. Two runs stored in this repository show the two ways that reading fails. Neither is the fault of a skill: in both, the thing that passed was the measurement.
The case set can be the thing that passes
The same skill was measured twice on the same day. A comparison rests on 4 pins: 2 are byte-identical between the two runs, 1 could not be read at all, and exactly one changed — the case set.
1Skill versionskillSourceanthropics/skills@53048666b05b4799081517d00e09e0a2dd688678skillHashsha256:91d476562e4e1910efbf135ab510dbab4db7c2209ff2638daa74bfc372336bbfheld
2Model identitymodelclaude-haiku-4-5-20251001held
3Host environmentsystemPromptHashnot-provided-by-hostnot measured
4Case set versionsuiteVersion1suiteHashsha256:31fa58c3ebe210e5e04c30954c60b6888a61708108f78a848c1dfee3ea839eb6changed
Each pin is recorded as one or two fields: a declared value and, where one exists, a content hash that catches the declaration being forgotten. Only pin 4 moved, and only in its hash — the case set file changed while its declared version did not.
Pin 3 was not measured
This host does not publish a system prompt hash, so the field carries a placeholder rather than a value. A placeholder is identical in every run, and counting that as held would hand the comparison a guarantee nobody measured — the same mistake this page is about. Assay reports it as a third state, and a comparison resting on an unread pin returns unknown unless an environment hash covers it.
Second case set
doc-coauthoring-borderline
Trigger precision
stayed quiet when it should have in 30 of 59 attempts
51%
95% confidence between 38% and 63%. 59 attempts narrow it this far; more would narrow it further.
Precision is the share of firings that should have happened. The first set reports it as 30/30; the second, measuring the same skill under the same conditions, reports 30/59. Recall is unchanged in both. The whole difference is in how the negative cases were cut.
Reading the traces explains it. In the first set every negative case put a document in front of the agent to transform — code, prose, a list of commits — while every positive case described work that existed only in the user's head. The set was separating cases on that second axis, not on the property it meant to test, and the skill happened to be flawless on it. Re-cutting the negatives so they differ from the positives on the tested property and nothing else moved the number by half.
What this changes about reading a result
A trigger suite that passes every case is a statement about the suite as much as about the skill. Assay enforces that a case set contains a negative and a near neighbour; it cannot yet check that the near neighbour is near along the right axis. Until it can, a clean sheet is a prompt to look at the negatives, not a reason to stop.
The layer can be the thing that passes
Trigger accuracy answers one question: did it fire? A second run asked the next one — once it fired, did the work get finished? Across 30 attempts of the same skill, the two answers turned out to be mutually exclusive.
| artefact complete | artefact missing | |
|---|---|---|
| skill fired | 0 | 15 |
| skill stayed quiet | 15 | 0 |
No attempt landed in either of the other two cells. Where the skill engaged, the file was never written; where it stayed out of the way, the file was always written.
Per case, both layers side by side
complete.proposal_with_objections
fired
100%10/10
artefact
0%0/10
complete.design_doc_with_outline
fired
50%5/10
artefact
50%5/10
complete.decision_record
fired
0%0/10
artefact
100%10/10
A trigger-only report of this run would have read 10/10 on its strongest case and called the skill exemplary. The artefact layer reports 0/10 on the same attempts.
The whole trace of one such attempt — 7 events, no Write among them
- assistant_messageI'll help you build a compelling trunk-based development proposal. This is exactly the kind of structured doc work where the doc-coauthoring workflow helps — it'll let us transfer your context, iterate the argument, and verify it lands with your audience.
- Skill{"skill":"doc-coauthoring:doc-coauthoring"}
- doc-coauthoring:doc-coauthoring
- Skill
- assistant_messageBase directory for this skill: C:\Users\<user>\AppData\Local\Temp\assay-skill-Mbz3TV\skills\doc-coauthoring # Doc Co-Authoring Workflow This skill provides a structured workflow for guiding users through collaborative document creation. Act as an active guide, walking users through three stages: Context Gathering, Refinement & Structure, and Reader Testing. ## When to Offer This Workflow **Trigger condition…
- assistant_messagePerfect! Let's build this proposal together using a structured workflow. You've already given me solid context, so I'll ask a few meta-questions to make sure I understand the framing, then we'll build it section by section. ## Initial Meta-Questions I know this is a proposal for engineering leadership about moving to trunk-based development. Quick clarification questions: 1. **Who specifically** are we convincing?…
- session_end
env.writes is empty. The session ends on a message asking the user five questions.
This is the skill behaving as documented
Its own description is a multi-turn workflow whose first stage is gathering context by asking the user. A single-turn CI run has nobody to answer, so it ends where the workflow says it should end. Nothing here is broken. What is wrong is a measurement that stops at the first layer and reports the result as completion.
The control case
Both readings above are causal claims, and a causal claim needs the other variables held still. The completion run carries a control case for exactly that: its prompt is word-for-word identical to complete.design_doc_with_outline except for the final paragraph, which asks for a file on disk. Nothing else differs — same run, same record, so all 4 pins are recorded once and cover both cases.
No file requested
control.design_doc_no_artifact
Skill engaged
fired in in 10 of 10 attempts
100%
95% confidence between 72% and 100%. 10 attempts narrow it this far; more would narrow it further.
Same prompt, plus “save it to out/…”
complete.design_doc_with_outline
Skill engaged
fired in in 5 of 10 attempts
50%
95% confidence between 24% and 76%. 10 attempts leave this very unsettled — run more to narrow it.
Because the two cases sit in one run, the comparison does not ask you to trust the person reporting it: the pins are recorded once and cover both. Running the control separately would have produced a second record with its own environment hash, and the claim would have rested on an assurance instead of a pin.
A file existing is not proof its content was earned
The completion layer asserts that a file exists, parses, and matches a pattern. It cannot assert where the content came from. One case in a third run makes the gap visible.
- 8/10
- attempts produced
out/run.txt, the file meant to hold the output of a test run - 10/10
- attempts had at least one shell call refused by the host's permission layer, so the run mostly could not happen
- 4/10
- attempts were caught by
no_swallowed_errors— the agent hit those refusals and did not mention them afterwards
Both figures are honest and they disagree. That disagreement is the finding: file_exists plus file_content_matches can tell you a plausible file is there; they cannot tell a recorded run from a well-formed guess. The same boundary is why side-effect claims return unknown whenever a run used a shell — an unobserved write is not a clean one.
Stated, not fixed
This is a ceiling in the tool as it stands today, not in the skills it measured. It is written here for the same reason a run reports unknown instead of rounding up: an instrument that hides its own blind spot is worse than one that has a bigger one.
Provenance
Every figure on this page is read out of a stored run record. None is written by hand.
- doc-coauthoringrun-2026-09-02T20-14-46-760Z-f39220fa2026-09-02
- doc-coauthoring-borderlinerun-2026-09-02T20-36-03-900Z-4544f65a2026-09-02
- doc-coauthoring-completionrun-2026-09-03T04-30-55-223Z-2a76ea3e2026-09-03
- webapp-testing-completionrun-2026-09-03T04-59-59-550Z-e8ba243f2026-09-03
Regenerate with node tools/methodology-data.mjs <trigger-root> <completion-root> apps/web/app/methodology/measurements.json. The case sets live under suites/ and examples/measurements/; the long-form reports are in docs/measurements.md.