fail
hallmark:
A case failed on something other than triggering — open the cases below.
Fired when it should have
fired in 40 of 40 attempts
100%
95% confidence between 91% and 100%. 40 attempts settle it to a narrow range.
Was right when it fired
was right in 40 of 40 attempts
100%
95% confidence between 91% and 100%. 40 attempts settle it to a narrow range.
Every case
failablation.page.devtoolbehaved in 0 of 10 attemptsshould fire
0%failablation.page.bakerybehaved in 1 of 10 attemptsshould fire
10%failablation.page.studiobehaved in 1 of 10 attemptsshould fire
10%failablation.page.eventbehaved in 0 of 10 attemptsshould fire
0%passtrigger.negative.unrelated.slow_querybehaved in 10 of 10 attemptsshould stay quiet
100%
Instability
2 cases produced both outcomes
The same prompt, the same model, different results. A single run of any of these would have been a coin flip reported as a fact.
- ablation.page.bakery — 1 passed, 9 failed
- ablation.page.studio — 1 passed, 9 failed
Conditions
Two runs are comparable only when all of these are identical. A score that moved because the model changed is not a regression in the skill.
- Skill version
- Nutlope/hallmark@13ac0ec7e148655948100b6396439e481361d690
- Skill hash
- sha256:bc95b2e54a4dfd62ec84984e185d36c869e1ec57f4f5af10ebc5549459ed5c2a
- Model
- claude-haiku-4-5-20251001
- System prompt hash
- not-provided-by-host
- Environment hash
- sha256:244e9412eb9f08675f57d59595da6a09d23f3d8336dab09797c936b6b9bdc790
- Permission mode
- acceptEdits
- Case set version
- 1
- Case set hash
- sha256:fc75bd261b9524f68bdc8abe2fcf38624c656fb01217f943576846c39c279086
- Assay version
- 0.3.2
What it cost
- Attempts
- 50
- Tokens in / out
- 2,213 / 259,928
- Cost
- $5.16
- Agent time
- 46 min
Combined trigger score (F1) 1.00 · 50 of 50 trigger decisions were correct.