Does your skillstill fire?

$npx @ktlsr/assay run ./hallmark:hallmark.suite.yaml

✗ablation.page.devtool0% (N=10, 95% CI 0%–28%)

✗ablation.page.bakery10% (N=10, 95% CI 2%–40%)

✗ablation.page.studio10% (N=10, 95% CI 2%–40%)

✗ablation.page.event0% (N=10, 95% CI 0%–28%)

✓trigger.negative.unrelated.slow_query100% (N=10, 95% CI 72%–100%)

verdicts 12 pass · 38 fail · 0 unknown

50 attempts · claude-haiku-4-5-20251001 · 2026-09-10

A recorded run, replayed: 5 cases, 10 attempts each. Assay does not round the failure up, and it does not hide how wide the interval still is at 10 attempts.

Get the CLInpx @ktlsr/assay run ./my-skill.suite.yaml

One case, measured 10 times

failablation.page.devtool

behaved as the case set expects in 0 of 10 attempts

10 attempts narrow it this far; more would narrow it further. One attempt would have reported either outcome as a fact.

hallmark:hallmarkclaude-haiku-4-5-202510012026-09-10

Open the full scorecard Every published measurement

What it measures

Trigger accuracy: Does it fire on the request it claims, and stay quiet on the near neighbour? A case set without a negative case is rejected — a skill that fires on everything would pass every positive case.
Task completion: Does the artefact exist, parse, and match the schema? Deterministic assertions only: file existence, structure, schema, text match.
Tool trace: Were the expected tools called in the expected order? A right answer reached the wrong way is a skill that breaks tomorrow.
Side effects: What did it write, delete, or reach for on the network? Where the sandbox cannot see, the run says so instead of reporting a clean slate.
Instability: Repeat count never defaults to one. A single attempt is an observation; the measurement is the rate across N with its interval.
Regression: Did the new version get worse? Only claimed when the conditions are identical and the intervals do not overlap.

Three verdicts, not two

pass

The assertion held, and there was enough signal to say so.

fail

The assertion did not hold. Something is broken.

unknown

The signal could not be read. A first-class result, counted and shown separately — never rounded up to a pass.

The most dangerous thing a test tool can do is report what it could not measure as a success. Everything else here follows from refusing that.

Getting started

$npm install -g @ktlsr/assay
installs the CLI; the command is assay
$assay init ./my-skill.suite.yaml
writes an example case set to fill in for your skill
$assay run ./my-skill.suite.yaml --skill ./my-skill
runs it against the host, N times per case, stores the record locally; --concurrency 4 runs attempts in parallel
$assay run ./my-skill.suite.yaml --skill ./my-skill --fast
three attempts per case, trigger layer only — minutes instead of hours. An early warning, not evidence
$assay ci ./my-skill.suite.yaml
the same run, with a CI exit code: 1 for a failure, 3 for nothing measured
$assay push --suite ./my-skill.suite.yaml
optional — keeps the history here so the next run can be compared against it

The SDK is Apache-2.0 and works with no account: runs are stored under .assay/runs/ and printed as JSON. The hosted side does not measure anything — it remembers.

Pricing

Draft, and not yet charged for. The measuring half stays free and open — a measurement layer nobody can read is a measurement nobody should trust.

SDK

Free, Apache-2.0

The CLI, the adapters, the assertion engine, the local store, the GitHub Action. Everything that measures.

Team

Draft

Uploaded history, regression comparison across versions, shared visibility. Everything that remembers.

Platform

Draft

For catalogues that want a measured quality signal on the skills they distribute — dated, pinned, reproducible.