Known-answer gate

A change that makes a run cheaper or faster can still lose facts the benchmark found before. The known-answer gate compares scored runs of a candidate with scored runs of a baseline and fails when any item the baseline found drops. It reads scores that are already written, from a blind scoring sheet, a key score file or the calibration scorer. It starts no run and asks no model. Everything lives in scripts/benchmark/, and none of it ships in the product.

What it readsCopy Anchor Link

In a key score, the scored run's verdicts are the first column of the item table that holds a verdict word. Every cell of that column must be one, and a later column, which is another run's, is never read in its place. A bare partly is partly (body). The file is checked against its own sentences: the **Totals:** line and the "N facts more than the human analyst" line, when it has them. A file that disagrees with itself is refused, and the message names both numbers. A Totals line that cannot be read is refused, naming the line, and so is a stated beyond-human count with no list. A file with no Totals line is read, and the output says its totals were unchecked.

An item is keyed by its code (K3.2, K7 17, K10 T1027). Where a label has no code, the key is the whole label in one case, with its runs of spaces read as one and the spaces around a / dropped. So a / b in one file and a/b in another is one item, while task Updater and taskUpdater stay two. A baseline that scores no item is refused, and so is a calibration score with no fact, trap or absent entry. A calibration trap that the run did not store is unknown. The calibration scorer's beyond counts publications that no manifest names, so it is not read as a beyond-human count.

The statistic: found in m of nCopy Anchor Link

An arm is one or more scored runs of the same setup. For each item, the arm's verdict is the best word that at least m of its n runs reach or beat. By default m is a strict majority of the arm's own runs (1 of 1, 2 of 3, 3 of 4). --baseline-m and --candidate-m set it, between 1 and n. A run that did not score an item, or scored it as unknown, reaches no word.

An item drops when the candidate's verdict ranks below the baseline's, or when the candidate has none. It improves when it ranks above. An item that fewer than m baseline runs scored is not gated, and the output names it. The beyond-human count is gated the same way: each arm's count is the largest count that m of its runs reach. Every number is an exact count and is printed with its n.

Own analysis or lookupCopy Anchor Link

A reputation service can hand a run a family name or a C2 list that public reports already hold. Counted the same as a key the run decompiled, it would make the score say the run worked out something it looked up. So every credited item of a key score also says which kind of evidence credited it. This is reported only: no verdict and no gate outcome changes.

The pass reads the stored run in the score file's own directory (evidence.json, report.md and mitre.json). The entries that credited an item are:

• the evidence ids its row names (ev_0014);
• the ids cited where a quoted fragment of the row stands in report.md. A fragment is text between double quotes, split at an ellipsis, of at least 12 characters. On a table row every id of the row counts. In prose, the ids inside the fragment count, and so does the citation that closes its sentence;
• for an item whose code is an ATT&CK technique (K10 T1027), the ids that the run's published technique of that id cites in mitre.json.

Each entry is classed by the kind its tool declares, never by the tool's name. A kind that asks a service outside the deployment what it knows (a reputation lookup, an indicator report, a submission to such a service) is a lookup. Any other declared kind is analysis. A tool that declares no kind is undeclared, and is never taken as either. An entry that did not answer, or an id the ledger does not hold, credits nothing.

"Found by analysis" counts the first two tags, and "found only by lookup" counts the third. score prints each run's count with both columns, then each tag's items. For an arm of several runs, an item's tag is the one tag that m of the runs reaching its verdict share. When none is shared but m of those runs found it by analysis, some beside a lookup, the tag is own analysis + lookup; otherwise it is mixed. gate prints both arms' counts and names every item both arms credit whose tag changed. --json carries each item's tag, each run's tag and the entries with their classes.

On the stored runs, haiku4 reads 38 own analysis, 2 own analysis + lookup, 0 lookup only and 11 no cited entry of its 51 credited items. Run 7 reads 36, 1, 1 and 18 of 56. In both, the family item is credited by a sandbox signature and the reputation report together. In run 7 the IcedID link rests on the reputation report's labels alone.

The intel-off armCopy Anchor Link

The intel-off arm runs a profile with every lookup server withheld, which measures what the run finds without them. intel-off derives it and runs nothing:

python scripts/benchmark/run.py intel-off --profile all_tools --settings stored-settings.json \
                                          --json all_tools_intel_off.json

The servers withheld are those that declare a tool of a lookup kind, read from the settings' declarations. With the built-in servers, that means threatintel and virustotal. They are added to the base profile's exclude_servers. That one field keeps a server from the team's agents and from the triage pack, whose reputation step never asks a server the team withholds, so no global setting changes. A server is withheld whole, so each of its other tools goes with it, and the output names them. A lookup that a provider block declares cannot be withheld by a profile, so the output names it instead.

--settings takes the stored settings by dotted key (agents.profiles, mcp.servers and so on). Without it, the defaults are used, and --profile defaults to the active one. The arm is checked by building the settings with it. The output gives the profile to add under agents.profiles.<base>_intel_off and the job config that selects it. The command exits with status 2 when the arm cannot be derived. Running the arm is a paid run, so each run needs the same approval as any other paid run. Score it blind, then gate it against the base profile. The gate reports which items dropped, and the provenance counts show whether the base profile's lookup-only items were among them.

Running itCopy Anchor Link

python scripts/benchmark/run.py score runs/haiku4/score.md
python scripts/benchmark/run.py gate --baseline runs/a1/score.md runs/a2/score.md runs/a3/score.md \
                                     --candidate runs/b1/score.md runs/b2/score.md runs/b3/score.md
python scripts/benchmark/run.py gate --baseline blind:run7 --candidate blind:haiku4 \
                                     --sheet blind-1/scorer/scores.csv --map blind-1/sealed-map.json

score prints each run's totals (for example 35 found / 15 partly (body) / 1 partly (dump) / 6 missed / 0 wrong; credited 51 of 57) and its beyond-human count. Given several runs, it also prints them as one arm. gate prints the decision, both arms' totals, and every dropped, improved and not-gated item with both verdicts and their counts. It exits with status 0 when nothing dropped, 1 when something did, and 2 when the scored runs cannot be read or compared. --json OUT writes the whole result.

Repeats are paid runs, so each one needs the same approval as any paid run. Score them blind, then gate them.