Detection and reporting

The suite cannot see your security product. It gives you scenarios with known signals, a way to record what your product did, and reports that show both what the agent did and what was detected.

The Detection column

The coverage matrix and the report have a Detection column. You fill it by writing a small JSON file that maps scenario ids to detected, blocked or missed. With no product connected, every scenario you ran shows not-reported, and a scenario you did not run shows not-run.

{ "<scenario id>": "blocked" }

Watch rules: a test stand-in, not a product

feintkit-watch reads the tool calls, tool results and assistant text of a saved headless agent session, matches them against a small readable rule file and writes the detections file the matrix and report read. It exists so you can see the Detection column filled before you connect a product, and as a worked example of what a product must record.

• Of the 48 scenarios, 17 have a firm rule, 22 a conditional rule, and 9 are invisible to a watcher of the session stream (protocol and network traffic, pixels) and stay not-reported.
• It only observes and stops nothing. A blocked result is a policy label, never a real refusal.
• Most rules match strings that only this suite produces, so a green row proves the reporting path works, not that any product can detect a real attack. Each rule is labelled marker or behaviour.
• A real product can write the same JSON Lines shape; feintkit-watch import --source <name> turns it into a detections file of its own, kept apart from the stand-in's.

node packages/watch/dist/src/bin.js scan --scenario <scenario id> --transcript <session file> --events <artifact directory>/<scenario id>/events.jsonl --rules packages/watch/rules/starter.json --out <detections file> --evidence <evidence file> --merge --root <repo>

Selftest

feintkit-watch selftest checks the stand-in and the judge against known cases: it judges four known replies per scenario and scans a known-bad and a known-good transcript, and fails when a verdict is wrong.

node packages/watch/dist/src/bin.js selftest [--rules <file>] [--root <dir>]...

Negative controls and the false-positive rate

The suite holds 4 benign controls: the ordinary twin of a tool description, a skill, a document and an instructions file. They carry nothing to detect. A good agent passes them and a product must stay quiet on them, so for a control the verdicts complied and resisted_surfaced, and the detection states detected and blocked, are false positives.

The matrix and the report list controls separately, never in the attack counts, show the false-positive rate of the judge and of Detection, and name every control that fired as a defect. --fail-on-control-fired makes the command exit with code 3 for one.

Repeat runs and the Wilson interval

An agent does not answer the same way every time, so one verdict says little. feintkit-report repeats turns N runs of the same scenarios into a compliance rate with a Wilson 95 percent interval per scenario (REPEAT=N in the example loop makes the runs). With --against it compares two sets of runs by interval overlap, so noise is not reported as a regression. The test is conservative: with few runs the intervals are wide, and five runs per side separate the two extremes.

node packages/report/dist/src/bin.js repeats <run-1>/report.json <run-2>/report.json <run-3>/report.json --against <other-1>/report.json <other-2>/report.json --format md

Coverage map

feintkit-report coverage draws a map of the ATLAS techniques and the two OWASP lists against the scenarios, coloured by the verdicts of one run, and lists the OWASP items that no scenario covers as gaps. A gap does not mean an agent is safe against the item, and a covered row does not mean it is protected.

Accuracy of the judge

Every agent verdict rests on one deterministic judge. feintkit accuracy measures how often it agrees with a human reading of labelled replies. On the built-in corpus of 15 synthetic replies the project reports agreement 15 of 15, precision 7 of 7 and recall 7 of 7, and lists the four disagreements as findings rather than fixes. You can add your own labels with --labels. These labels are synthetic: they show what the judge does with a kind of reply, not how often real agents write that kind.

More reporting

• feintkit-report diff compares two runs and marks regressions; --fail-on-regression exits 3 for CI.
• feintkit-report history keeps one line per run, with agent version, model and permission mode, and prints the trend.
• feintkit-watch replay re-judges and re-scans saved results offline after a rule change.
• JUnit XML and SARIF output for CI systems; --mitigations adds a short defender note per scenario.

Verification status, as the project states it

Quoted from the project's README (Status). Version 0.1.0, not yet published to any package registry or hosting site.

• 48 scenarios and 4 benign controls across 7 category repositories, 16 packages and 87 specs.
• The full test suite (783 tests) passed on macOS, Ubuntu 24.04, RHEL 9.7 and Windows 11 with Node 20.20.2 and 22.23.3, checked at one commit on 2026-10-16 (macOS ran on Node 22.23.0 only). Later changes have not had their own cross-OS run. A few tests skip by design.
• Six scenarios added on 2026-10-26 have had neither an independent cold rebuild nor a cross-OS run.
• The first 42 scenarios were run once with a real agent (one agent client, 2026-10-24). No other agent has been run yet.
• Mutation checks (deliberately broken copies must be caught by the tests) were replayed on one platform: 1203 recorded mutations, with the exceptions listed in the README.
• These results describe the suite's own tests. They are not a statement about any agent or product.