Benchmark

This page reports how Malspark's teams perform on a small set of real samples, how their reports compare, item by item, with published human analyses of the same sample, and which facts they establish that the human analyses do not hold. It covers two series:

• The default-team iterations. The default team ran six times between 2026-09-23 and 2026-09-24, with platform fixes between the runs. Every number in this series comes from a single run on a laptop, with nothing executed: all runs used the mock sandbox, so every finding is static.
• The all-tools series. The importable all-tools team — three static analysts, a Ghidra reverser, then the dynamic and network stages — ran on the reference sample with a real sandbox (Hatching Triage), twice on a hosted model and several times on the local one. See The all-tools series.

SummaryCopy Anchor Link

The reference sample, 57 core items:

"Beyond human" counts the facts a run's report states that no human-report item holds and that the run's own evidence proves; the rule is in Beyond the human reports. The default-team iterations (—) were scored before that rule existed, and their score files do not itemise such facts.

In short, for the default-team iterations:

• The verdict was right in every completed run: 4 of 4 samples on both models in iteration 1, and every sample in every later iteration.
• Against the human report, the best run (iteration 4) found 15 of 57 core items, partly found 25, missed 17 and got none wrong. Before decoded strings and VirusTotal labels reached the model, the same sample scored 3 / 7 / 46 / 1.
• The score has plateaued within rerun noise. Iterations 5 and 6 found 14 and 12 items, and across the five scored runs of the default model found stayed at 12–15 and wrong at 0.
• The remaining gap is on the model's side. What Malspark still misses is mostly control flow: the anti-analysis checks, the bot-ID derivation, the beacon interval and the command IDs. Strings alone cannot recover these, and a disassembler did not help in the one run that had one. The rest is the purpose of values the report already prints, such as the mutex and the scheduled task's name.
• On the benign control (signed PuTTY), iterations 4 and 5 published no ATT&CK techniques. In iteration 6 the verdict fell back to text extraction and seven techniques were published from the analyst's hedged claims. No run since iteration 1 has produced detection drafts on it.

In short, for the all-tools series:

• In run 4 on the hosted model the report found 45 of 57 core items, partly found 9, missed 2 and got 1 wrong (command 0xf labelled "uninstall"), and stated 19 facts beyond the human reports, from the decoder and handler addresses to the TLS flags the request routine sets.
• A second hosted run, with no spend ceiling and parallel analysts, found 43, partly found 11, missed 2 and got 1 wrong (command 0xf again), and stated 27 facts beyond the human reports. It cost 3.07 USD against 1.39 for run 4 counted the same way (1.36 charged), and part of that came from platform defects the run exposed, which were fixed afterwards.
• On the local model the best run found 15, with none wrong, and stated 12 facts beyond the human reports. Its reverser decompiled every function the key items' decoded strings are used in and described none of them in a claim: the gap to the hosted run is the model's reading of code it already has.

MethodCopy Anchor Link

SamplesCopy Anchor Link

Iteration 1 ran all four samples on both models. Later iterations ran the default model only, on the samples that the fixes in between were meant to move: iteration 2 ran samples 1–3 and a second run of sample 1 with a disassembler attached; iterations 3, 4, 5 and 6 ran samples 1 and 3.

ModelsCopy Anchor Link

The small model was set as both the expert model and the judge model. Every timeout and budget stayed as shipped for both models.

Host, sandbox and profileCopy Anchor Link

• Host. One laptop with an 8 GB GPU, shared with other desktop work. The benchmark stopped any run in which the host's available memory fell below 6 GB; such a run was not counted.
• Sandbox. The local mock sandbox, which has no fixture for any of these samples. Nothing was executed, and the dynamic and network analysts were skipped in every run.
• Team. The default profile: the deterministic triage pack (including FLOSS decoded strings, capa and VirusTotal labels), the static analyst, the debate, the judge's verdict and the report. No setting was changed between runs of an iteration.
• Model server. From iteration 2 on, llama.cpp ran with a 1 GiB prompt cache instead of the 8 GiB default, to fit the host's memory. Iterations 4 to 6 also capped the per-slot context checkpoints at 8 instead of 32. Wall times are therefore like for like only across iterations 4 to 6.
• Output caps. In iterations 5 and 6 the judge's answer was capped at 8,192 tokens and each analyst's at 4,096. In iteration 6 the judge's and the report model's 8,192 was derived from the context window (a quarter of 32,768), while the analysts' 4,096 was a value stored in the settings, which wins over the derivation.

The human report and the scoring keyCopy Anchor Link

The reference sample is scored against Bitsight, Latrodectus, are you coming back? (João Batista, 2024-06-17), which lists this sample's sha256 among the bot samples it analysed. Elastic's and Proofpoint's public analyses of the family serve as corroboration. The score files quote the human report only by section name.

From that report the key draws 57 core items in ten groups:

K1–K6 and K9 form the main comparison (39 items); K7, K8 and K10 are depth (18 items). Build-specific values (the C2 domains, the group name, the RC4 key) are checked only for consistency with the human report's lists, because that report describes the family across about 80 samples. Detection drafts and later-stage items are not scored.

Each item gets one verdict, and each verdict quotes the report sentence it rests on and names the human report section it is scored against:

• Found: the report body states the item, with its purpose right.
• Partly: split in two so a reader can discount the weaker kind.
  • Partly (body): the report body states part of the item, or states it with the wrong purpose.
  • Partly (appendix): the item's own value (an IOC, a format string, a command keyword) is printed only in the report's appendix of FLOSS strings, with no sentence about it.
• Missed: the report does not state the item. A behaviour that a string in the appendix only hints at counts as missed, not partly.
• Wrong: the report states the item in a way the human report contradicts.

Each score file also lists the reverse direction: claims in Malspark's report that no human report supports, marked as contradicted, unsupported, or correct but new.

Beyond the human reportsCopy Anchor Link

The key measures how much of the human analysis a report recovers. It cannot show what a report establishes that the human analysis does not hold, so every score from the all-tools series on carries a second list, built by one rule:

• What is counted. Every fact the run's report states that no item of the key holds (K1–K10, and K11, the human reports' detection content), that none of the human reports states for this hash, and that the run's own evidence proves: a tool's answer, a decoded string, a sandbox record, a decompile.
• How it is listed. One fact per line, each with the evidence id it rests on and the tool that produced it, then the total: "N facts beyond the human reports".
• What is never counted. An extra claim that is wrong, or that the evidence does not prove — a model's unsupported sentence, a sandbox signature read as the sample's behaviour, a value cited to an entry that does not hold it. These are listed separately under Not counted and never added to the total.

A fact in the human reports' build items (the RC4 key, the group, the beacon version) is a key item, checked for consistency, and is not counted as beyond.

The all-tools seriesCopy Anchor Link

The same reference sample, key and verdict words as the iterations, with five differences:

• Team. The importable all-tools team (docs/examples/profiles/all-tools.json, see An all-tools team): the triage pack and the triage agent; three static analysts, each on a tool of its own (the analysis and knowledge servers, radare2, and a generic analysis server); a reverser on Ghidra; the dynamic and network stages; the debate, the verdict and the report.
• Sandbox. Hatching Triage, with a 600 s run, so the dynamic and network analysts had a report and a capture to read.
• Models. The hosted run used deepseek-flash (DeepSeek, hosted) through the OpenAI-compatible provider (compat = deepseek, thinking on), with a spend ceiling. The local runs used Qwen3.6-35B-A3B on ik_llama.cpp with a 131,072-token window, on the same laptop as the iterations.
• Scoring surface. The report body, the STIX bundle and the /iocs feed — what an analyst receives. Where the analysts' claims or the deterministic evidence held more than the report, the score file says so.
• Earlier runs. A run is submitted with the job option ignore_earlier_runs (python scripts/benchmark/run.py job-config prints the config), so it is not shown what an earlier run on the same sample stored, and its run summary says earlier_runs_ignored: true. A run submitted without it may be shown the earlier run's technique list as hypotheses, and its score is then partly that run's; its score file has to say so on a line of its own, **Earlier runs:** used. The scoring tools (scripts/benchmark/run.py score and gate) read the run summary stored beside each score and name every run that neither says earlier_runs_ignored: true nor states this, under EARLIER RUNS NOT RULED OUT. The console's job form cannot set the switch, so a benchmark run is submitted through the API (POST /api/v1/jobs with the printed config). Each run still stores its own case when it completes; with the printed config it is stored under the run's job alone (below), so the case of the sample held by the account that submitted it is left as it was. A run submitted without isolated_memory replaces that case (memory belongs to a job's creator).
• Isolated memory. The earlier-run switch covers the same sample only: the judge's similar cases, the attribution's similar samples and the function-hash families read every case the account stored, so run k of a series would still read what runs 1..k-1 stored for the other samples. Every benchmark run is therefore also submitted with the job option isolated_memory (the printed config carries both): its long-term memory is filed under the job alone, so it reads nothing from the long-term memory that any other job stored, for any sample, and its own case never replaces the account's case of the sample. Its run summary says isolated_memory: true. --with-earlier-runs leaves out both switches, since a job with memory of its own reads no earlier run either. A case corpus mined from the memory (scripts/knowledge/build_attck_case_kb.py, read through the knowledge server's similar_cases) is not owner-scoped: a run reads it whatever its switches. It is built from unscoped cases alone, so no benchmark run's case is copied into it. Each isolated job still stores its case and its function hashes, under its own key, where no later job reads them: every benchmark run adds one case plus its function hashes to the store, and no purge by owner exists yet. score and gate count a run as independent only when its run summary says both earlier_runs_ignored: true and isolated_memory: true, or when its score states, on a line of its own, **Memory:** fresh account (a run on an empty store of its own); every other run is named under EARLIER RUNS NOT RULED OUT, with the switch its summary does not say.

ResultsCopy Anchor Link

The default model across the iterationsCopy Anchor Link

Default model, default profile, one run per cell.

¹ From iteration 2 on, llama.cpp ran with a 1 GiB prompt cache instead of the 8 GiB default, and iterations 4 to 6 also bounded the per-slot context checkpoints (8 instead of 32). Times are therefore like for like only across iterations 4 to 6. Iteration 5's PuTTY run spent most of its extra time in the verdict stage (337 s against 45 s in iteration 4), where the judge's answer ran to its output cap twice.

The verdicts per sample: Latrodectus Malware (family Latrodectus), sample A Malware, PuTTY Benign (family PuTTY), the ELF Malware (family Snowlight). In iteration 1 the default model's judge wrote non-JSON twice on sample A, so its verdict carried no confidence, severity or family; in iteration 2 the judge was asked again after its answer was cut and stated Malware, 0.95, High, Filisto.

The small model (iteration 1 only)Copy Anchor Link

qwen3.8:27b on Ollama, not re-run since:

• Verdicts right on 4 of 4 samples, each stated with a confidence, and the family named on all four (Filisto included).
• PuTTY: 1 malicious sentence, 0 techniques, 20 template detection drafts.
• Mean wall time 63.3 min over Latrodectus, sample A and PuTTY, against a mean of 545 s for the default model over its four runs.
• It ran 76% on the CPU with an 8 GB GPU. Every run on a Windows PE sample hit the static analyst's 1,500 s time cap, and on those three samples the analyst produced no claims; only the small ELF's analysis finished in time.

The small model scored almost as well as the default model against the human report (10 found against 12) without a single analyst claim, because in iteration 1 the triage pack did the work that mattered: FLOSS decoded the Latrodectus strings, the VirusTotal labels named the family, and the judge and report model of both models read the same pack.

Against a human analystCopy Anchor Link

Latrodectus, 57 core items, found / partly / missed / wrong:

The same scores by group (found / partly / missed / wrong):

How much of "partly" sits only in the appendix, and the scores with those items counted as missed:

Iteration 4 is the best-scored reference run, with 15 found and 0 wrong against 12 and 0 in iteration 2:

• A new host-identifier section put the mutex runnung, the Custom_update folder, the group Littlehw, the RC4 key 12345, the C2 instruction keywords and the beacon format into the report body, so appendix-only items fell from 5 to 2.
• The scheduled task was stated with its name (Updater) and its logon trigger, and T1053.005 and T1218.011 were published.
• The eight core IOC values are all stated in the body as values (iteration 2: 5 of 8), 5 of 8 with a purpose.

The same run shows what is still weak:

• Most host identifiers were listed without their purpose. The mutex, for example, appears as "Unknown string, possibly part of a key or identifier", so those items moved from partly (appendix) to partly (body), not to found.
• The judge's structured answer overran its 8,192-token output cap twice. The verdict (Malware, Latrodectus) was then read from text, and the two decoded C2 domains were not published as indicators, as they had been in iteration 2. Only the sample's hashes (1 of the 8 core IOC values) were published.
• Five of the ten published techniques had no evidence behind them. One was T1003 OS Credential Dumping, attached to the debugger check's read of the PEB. In the reverse direction the report makes 10 contradicted or unsupported claims (iteration 2: 9).

Iteration 5 scored 14 found and 0 wrong.

• It gained the loader ("It operates as a DLL loader") and the mutex (the analyst's claim "The malware creates a mutex to ensure only one instance is running.").
• It lost the scheduled task: Updater and LogonTrigger appear only as rows labelled "Likely part of a mutex name or identifier", and the prose says persistence is by Startup shortcuts and Run keys. T1053.005 and the task's IOC went with it.
• The host-identifier section grew to 73 rows, so only one item was left in the appendix alone, but it gave eleven values the same wrong purpose.
• The judge's answer overran its output cap once and the compact retry closed, so the verdict was stated. The judge wrote the two C2 domains, but as LIKE patterns, which the platform then dropped as malformed, so they were not published.
• 7 of the 15 published techniques had no evidence behind them, two of them (T1003 and T1018) added by the judge with no analyst claim. The report makes 11 contradicted or unsupported claims.

Iteration 6 scored 12 found and 0 wrong.

• It lost the loader (the report calls the sample a "Trojan/Backdoor"), the mutex (the claim now reads "uses a custom mutex or identifier for its operations", with no single-instance check) and T1218.011 (only the parent T1218 was claimed). It gained T1059.003.
• No item is left in the appendix alone. The C2 response instructions came nearest yet: URLS, COMMAND, ERROR and CLEARURL are read as "Response parsing logic", three of them with the right purpose.
• The two C2 domains were published again, with a YARA rule and a Suricata DNS rule for them.
• The report's own claims improved: 7 contradicted or unsupported (iteration 5: 11), and 2 of the 13 published techniques unsupported (iteration 5: 7 of 15). No technique was published on the judge's word alone.
• The static analyst's answer was cut at its 4,096-token cap and so was its retry, so none of the questions put to it was answered and its first answer was kept.

The score has plateaued. Across the five scored runs of the default model, found has stayed at 12–15, missed at 16–20 and wrong at 0. A move of two or three found items between runs is wording in the model's answers: iteration 6 lost items for wording and gained one the same way, and the platform's handling did not cause those moves. The remaining gap is on the model's side:

• Control flow. The anti-analysis checks, the bot-ID derivation, the beacon interval, the beacon types and the command IDs need the code read. With a disassembler attached (iteration 2), the model decompiled the right functions but did not turn them into claims.
• The purposes of values it already prints. runnung, Updater, Custom_update, update_data.dat, 12345 and :wtfbbq are all in the report body, and their roles are unstated or guessed.

Item by item, with the quoted sentence and the human report section for each: iteration 6, iteration 5, iteration 4, iteration 2, iteration 2 with r2, iteration 1 and iteration 1, small model.

What changed between iterationsCopy Anchor Link

Before iteration 1. The baseline report on the reference sample found 3 items. Three changes carried iteration 1 to 12 found and 16 missed:

1. FLOSS in the triage pack. It recovered 81 decoded strings: the C2 pair, /live/, the User-Agent, the beacon format, the mutex, the install folder and file name, the scheduled task, the RC4 key, the group, the command keywords and every discovery command. The IOC group went from 7 missed to 0.
2. VirusTotal labels in the triage pack, which put the family right.
3. Measured header facts in the report: compile time, an x86-64 DLL, the export name and the four exports sharing one address.

Iteration 1 to iteration 2. Fixes for what iteration 1 showed:

• The judge is asked once more when its answer is cut at the output cap (sample A then stated its verdict with a confidence and the family).
• Citations are checked against the evidence entry that holds the quoted value, not only for the id existing.
• Decoded C2 values are published in the report's IOC section and the IOC endpoint, not only in the STIX export.
• No template detection drafts are generated on a Benign verdict.
• An invalid STIX pattern is asked about instead of exported; no run since has a validator error.
• Report sections that had been dropped (the configuration section) came back, and a host-identifier section was added for decoded values.
• Wrong ATT&CK technique names in prose are asked about.
• A cancelled job stops its model calls.
• For a slow model: the closing summary after a time cap is sized to fit the context window, and an analyst with no claims is not run again from scratch. The small model was not re-run, so this is untested on it.

Result: the User-Agent and its IOC became found and the C2 domains were published, but the new host-identifier section overran its 8,192-token budget and was missing, and the analyst claimed different techniques. The total stayed at 12 found.

Iteration 2 to iteration 3.

• The triage pack prints whole hashes instead of 16-character prefixes.
• A job is stored before it is queued, so a worker cannot pick up a job it cannot find.
• llama.cpp's own prompt and generation rates are recorded, and derived timeouts include reading the prompt.
• A report section cut at its output cap is asked once for a shorter answer.
• The capability check no longer flags negated sentences, and a sentence it flags that the retry keeps is marked where it stands.

Result: the reference run did not complete. Both attempts were stopped by the memory rule in the report stage, with the model server grown to 13.4–14.2 GB, so iteration 3 has no score. On PuTTY the analyst attached technique ids to sentences such as "does not contain any obvious persistence … mechanisms", and 13 techniques were published on a benign file.

Iteration 3 to iteration 4.

• A claim whose technique id sits on a sentence that reads as absence is asked about. The claim is never edited, and a technique the analyst keeps is still published, with a note.
• A section answer with alike rows is asked once whether they are repeats.
• The STIX report for a Benign verdict is typed threat-report instead of malware.
• Host configuration: the model server keeps 8 context checkpoints per slot instead of 32, which kept its peak at 11.9–12.3 GB.

Result: both runs completed on the first attempt, the host-identifier section reached a stored Latrodectus report (57 rows, all cited to the FLOSS entry that holds them), and the score rose to 15 found. PuTTY published no techniques, but the analyst made no absence claim with a technique id in this run, so the drop comes from the analyst's different output, not from the new check.

Iteration 4 to iteration 5.

• The judge is asked only for what it decides, so its structured answer is shorter. When the answer is cut at the output cap, the retry is told what filled it and asked for a compact answer.
• When the verdict falls back to text extraction, the indicators the judge's answer wrote in full are kept and still go through the publish rule.
• The capability check reads the analysts' claims, not the values in the report, so it no longer marks a configuration path or a sentence saying that something is absent. A claim whose sentence does not describe its technique is asked about.
• A section with repeated rows is asked about instead of only recorded.

Result: 14 found. On Latrodectus the compact retry closed and the verdict was stated, where iteration 4 had fallen back; on PuTTY the judge answered the compact request with essentially the same over-long text, and the verdict fell back on the benign control for the first time. The run also exposed new failures:

• The judge wrote the C2 domains as LIKE patterns, and the platform kept only patterns containing =, so the domains were dropped before the publish rule.
• The static analyst's answer and its retry both stopped at the analyst's 4,096-token cap, and nothing asked about the cut, so none of the questions put to it was answered.
• PuTTY's fallback STIX export was invalid (a note that referred to no object), the first invalid export of the benchmark.
• Two techniques the judge added without an analyst claim, T1003 and T1018, were still published.

Iteration 5 to iteration 6.

• A STIX pattern is kept whatever its comparison operator. A LIKE is read as a shape, every value a pattern names goes through the publish rule, and the official STIX pattern validator decides whether a pattern is valid where it is installed. A LIKE whose fixed text is a value the run holds is asked about as =.
• An analyst's answer cut at its output cap is asked once for a whole, shorter one, and the judge's retry no longer carries the cut answer.
• The fallback STIX export's note names the objects it is about, and the export lists each object once.
• A technique the judge states with no analyst claim is published as the judge's own claim, and says so.
• VirusTotal's count is written in words, not as a fraction.
• When no output cap is set, the analysts' and the judge's caps are derived from the model's context window.

Result: 12 found, within the spread of the earlier runs. Both C2 domains were published and drafted as YARA and Suricata rules, both STIX exports were valid, and the judge closed at its first attempt on Latrodectus (3,178 tokens). The analysts' cap was a stored 4,096 rather than the derived 8,192, so both analysts' answers and both retries stopped at 4,096 tokens again and the questions put to them went unanswered. On PuTTY the judge ran to its cap twice again and the verdict fell back; see the false-positive control below.

The disassembler findingCopy Anchor Link

In iteration 2 the reference sample was run a second time with the r2 disassembler attached (core.static.provider = "r2"). This shows a model gap, not a tool gap. The static analyst got 68 r2 tools and made nine decompiles down the real start-up path, from the export through the start-up routine to the API resolver (53 callers) and the string decoder (121 callers). But it spent its 39 steps before reaching the anti-analysis checks, and its closing answer was a repeated list of strings with no confidence, which the platform correctly refused to read as claims.

It recovered none of the control-flow items: the anti-analysis checks, the bot-ID derivation, the beacon interval and the command IDs. The report written without analyst claims was worse than the default run's: four "does not" sentences contradict the human report (no C2 commands, no persistence, no payloads, and runnung as a registry key), one of them scored wrong. This is a single run.

The false-positive controlCopy Anchor Link

PuTTY is a signed, benign SSH client. Every run named it Benign with the family PuTTY (at 0.99, or 0.95 in iteration 5), and no indicator was ever typed malicious. What varies is what the report says around the verdict:

• In iteration 1 both models produced the same 20 template Suricata drafts with the class trojan-activity, one of them titled as a C2 IP for 6.0.0.0, and the default model's STIX export had 3 validator errors. Since iteration 2, a Benign verdict generates no drafts.
• Iteration 3's 13 techniques came from absence sentences carrying technique ids. They included T1014 Rootkit and T1555 Credentials from Password Stores.
• Iteration 4's four malicious sentences say that the sample performs keylogging and captures clipboard data, attempts to evade detection and debuggers, employs anti-debugging and evasion techniques, and may be a repacked binary. They rest on capa rule names. The report also says "verified clean by 75/75 AV engines" where 0 of 75 engines flag it.
• Iteration 4 printed four marks on sentences its capability check doubted, and none of them is right: two sit on sentences stating that persistence APIs are absent, and two on PuTTY's configuration path /SSH/Auth/Credentials. The four real over-claims are not marked.
• Iteration 5's two malicious sentences say that the sample "exhibits behaviors associated with anti-debugging, discovery, evasion, execution, filesystem access, and keylogging" and "implements anti-debugging and evasion mechanisms". Its four marks were better placed: three are right, and the wrong one sits on a sentence restating the API catalogue's measured counts. The report again said "verified clean by 75/75 AV engines". The judge's answer ran to its output cap twice, so the verdict fell back to text for the first time on this sample, and that fallback export failed STIX validation.

Iteration 6 is a regression on this control. The verdict fell back again: the judge ran to its 8,192-token cap on both attempts, the second time with 58 STIX attack-pattern objects begun on a benign file. The static analyst began 33 claims, each with a technique id and a sentence that explains the capability away as ordinary. T1056 Input Capture, for example, sits on "capabilities associated with keylogging, which in the context of a terminal emulator, is likely for capturing terminal input, not malicious keys…", and T1055 Process Injection on "… common in malware but also in legitimate software for debugging". The answer was cut at 4,096 tokens, the retry returned no claims, and the first answer was kept. The fallback export carries every technique id that an evidence claim carries, so seven techniques from these hedged claims were published under a Benign report: T1059, T1547, T1027, T1055, T1010, T1083 and T1056. Seven further ids that appeared only in the judge's raw answer were withheld.

This is iteration 3's failure again by a different path: a Benign verdict does not stop an analyst's technique from being published. The rest of the report:

• Three malicious sentences stand unmarked: "The sample employs anti-debugging and evasion techniques to hinder analysis.", "We assess these capabilities are used for storing settings or establishing persistence mechanisms." and "The sample uses registry modification for persistence-like behavior". The capability check reads claims, and the analyst's claims back each of them.
• VirusTotal is now stated right: "zero detections among 75 engines" and "0 of 75 engines flag it as malicious".
• The STIX export is valid, and no detection draft was made.
• The fallback's missing category is printed as text: the title reads "PuTTY None analysis".

LimitationsCopy Anchor Link

• One reference sample. Its human report is family-level prose that names this hash among about 80 samples, so build-specific values are checked only for consistency.
• The other samples have no human report, and their false positives are judged against each run's own evidence.
• One run per cell, no variance measured. The static analyst's output changes shape from run to run (on Latrodectus 7, 41, 24, 11, 42 and 36 claims across the iterations), the judge overran its output cap on one sample or the other in iterations 3 to 6, and several movements between iterations are the model's rather than the tree's, such as T1053.005 appearing and disappearing. The 12–15 found range across the five scored runs is the rerun spread of this setup, not a trend.
• The model server's settings changed between iterations (prompt cache from iteration 2, checkpoint budget from iteration 4), so wall times are like for like only across iterations 4 to 6.
• In the iterations, nothing was executed (mock sandbox). The dynamic and network analysts never ran.
• The host is a laptop with an 8 GB GPU shared with other work. Every counted run up to iteration 5 finished within about 1 GB of the 6 GB memory stop, and in iteration 3 two attempts crossed it. Iteration 6 had about 1 GB more headroom than iteration 5.
• The small model's numbers come from iteration 1 only, and they reflect this host: a GPU that holds the whole model would change its time-cap outcome.
• The all-tools series is one run per page (two hosted, one scored local), on one sample. In run 4 and the local run the sample's own gates kept it inert on the guest, so no C2 contact, drop or task registration was observed.
• A beyond-human count is a count of stated, proven facts, not a measure of their value: an export name and a decompiled handler address each count one.

Score filesCopy Anchor Link