Development

What is where, which command runs which gate, and the conventions a change is expected to follow.

Repository layoutCopy Anchor Link

Maljan/
├── apps/
│   ├── api/           FastAPI app, arq worker, Alembic migrations ("maljan-api")
│   ├── docs/          these documentation pages: content/docs/, images under
│   │                  public/assets/, and the content check (npm run check)
│   └── web/           Next.js console, Playwright specs under e2e/
├── src/maljan/        the core package: agents, pipeline, analysis layers,
│                      providers, memory, reporting, core (config, container)
├── services/          stdio MCP sidecars, one server.py each
├── scripts/
│   ├── ci/            the uv.lock dependency-graph submission the workflow runs
│   ├── dev/           fetch_external.sh, the Ghidra image manager and
│   │                  run_capped.py, the capped scope for heavy local runs
│   ├── goldens/       capture scripts that write tests/fixtures/golden/, plus
│   │                  render_team_graphs.py, which draws the team-*.svg diagrams
│   └── knowledge/     builders for the data/ assets
├── tests/             unit/ api/ integration/ fixtures/
├── data/              tracked knowledge assets, loaded lazily, each with a fallback
├── docker/            Dockerfiles, the compose stack and its dev overlay
├── docs/              examples/: the importable all-tools team document
├── Makefile           every gate and every generator
└── pyproject.toml uv.lock   one uv workspace: maljan plus apps/api

The repository is one uv workspace with one lockfile. uv sync --all-extras --all-packages installs both Python packages, so import maljan and import app work from any directory without PYTHONPATH, in the venv, in the backend image and in CI alike.

Route-local React components stay in their route folder; a component two routes use moves to apps/web/src/components/. tests/unit/ mirrors src/maljan, one subdirectory per subpackage, and is where a new test starts.

Host-specific helpers (a launcher for a local llama-server, a memory guard, a restart wrapper for long evaluations) live outside the repository. For the record, the local model server the measurements in this repository were taken with ran ik_llama.cpp as llama-server -m Qwen3.6-35B-A3B-IQ3_K_R4.gguf -c 131072 -t 16 -fa on -ctk q8_0 -ctv q8_0 -ngl 999 -ot 'blk\.([1-3][0-9])\.ffn_(up|gate|down)_exps=CPU' --context-shift on --jinja on loopback port 8080.

Make targetsCopy Anchor Link

TestsCopy Anchor Link

• Python — uv run pytest tests/ -q. tests/unit/ is the default home; tests/api/ covers the FastAPI surface and tests/integration/ the flows that cross the worker, the database and object storage. The suite needs no secret in the environment: a pytest-only JWT secret is substituted, and the auth bypass is forced off so real 401 and 403 assertions still mean something.

• Goldens — scripts/goldens/capture_provider_goldens.py freezes the sandbox tools' answers per report under tests/fixtures/golden/sandbox_tools/. Most source reports (data/cape_reports/) are kept only on the machine that captured them; two are tracked, one of them a hand-written report (tests/fixtures/sandbox/cape2_synthetic.json) whose registry calls and network rows CI compares. A case whose source is absent is skipped, and the session ends with the number of such skips (on GitHub Actions also a warning annotation and a line in the job summary). MALJAN_TEST_NO_CAPE_REPORTS=1 takes that path on a machine that has the reports. A golden whose text passes 5 MB is written as <name>.json.gz and read back decompressed. The same script writes the two prompt fixtures under tests/fixtures/prompts/, which a test compares with the live prompt builder, so a prompt change comes with its recapture.

• Frontend unit — cd apps/web && npm run test:unit (vitest), alongside npx tsc --noEmit and npm run lint.

• Playwright — cd apps/web && npm run test:e2e. The suite starts its own next dev on port 3100, forces NEXT_PUBLIC_AUTH_DISABLED=false and points the client at the Next server's own origin, so it is hermetic and contacts no backend; every API call is mocked in e2e/mocks.ts. Run a named spec while developing (npx playwright test e2e/settings-configuration.spec.ts --project=chromium) rather than the whole suite, and stop any dev server you started yourself first — the browsers are memory-hungry, which is why the local worker count and the timeouts are raised deliberately.

• Replay — python scripts/replay/run.py <stored run> re-runs the platform's deterministic rules over a stored run with no model call, and --compare REF_A REF_B names every difference a rule change makes between two refs. Run it before and after a change to a check, the report builder or a publish rule; see Replaying a stored run.

The evaluation harness behind the earlier measurements is no longer part of this tree; the benchmark holds the current results.

Heavy local runs in a capped scopeCopy Anchor Link

A benchmark or observation run against the local model, a replay over many stored runs or a large test suite can fill the host's memory, and when it runs out the whole desktop session goes with it, not only the run. On a host with the systemd user manager, run such a command through scripts/dev/run_capped.py; nothing runs this way unless asked:

python scripts/dev/run_capped.py -- python scripts/replay/run.py runs/haiku7 runs/haiku8
python scripts/dev/run_capped.py --memory 8G --floor 4G -- uv run pytest tests/unit -q

It starts the command with systemd-run --user --scope in a transient scope whose unit name it prints, under a hard memory cap (--memory, default 12G, refused at or above the host's memory), no swap (MemorySwapMax=0), a task ceiling (--tasks, default 4096) and a CPU quota (--cpu, a percentage, default three quarters of the host's CPUs). --unit names the scope (default maljan-capped-<pid>-<time>); it must not name a unit that is already loaded, and such a name is refused rather than shared.

A watcher polls the scope, and acts only once it has seen the command's own process in it. On that first poll it reads the caps back from the scope's control group (memory.max, memory.swap.max, pids.max, cpu.max) and, when one was accepted but not enforced, kills the scope and exits 2. On every poll it keeps the scope's memory events and stops the scope when the host's MemAvailable falls below --floor (default 3G; a floor at or above what is available at start is refused), which holds the host against everything that runs beside the command, a local model server included; --no-watch turns the floor off. A watcher that fails says so. SIGTERM or SIGHUP to the wrapper stops the scope too. When the command ends the wrapper prints its exit status, systemd's result for a scope that failed (oom-kill, for one), whether the watcher stopped it, and the memory events at the watcher's last poll (high, max, oom, oom_kill). --dry-run prints the systemd-run line and starts nothing.

The wrapper follows the command's main process. When that process exits but leaves children running in the background, the scope stays alive after the wrapper has returned: still under its caps, but with no floor and no report. Stop it with systemctl --user stop <unit>.scope (the name is printed at start).

Where the caps cannot be enforced (no systemd-run, no user manager, a memory, pids or cpu controller not delegated to it, or no swap accounting, where systemd would accept the cap and enforce nothing) the wrapper exits 2 and says why, without running the command. Only --allow-uncapped skips that requirement: it runs the command anyway, with a line on standard error that it runs uncapped.

Native runs pin LC_CTYPECopy Anchor Link

C code folds case with the C library's tolower, by the process's LC_CTYPE. Under a Turkish or Azerbaijani locale a capital I does not fold, so the YARA engine's nocase misses every I it should match and its pe.imphash() differs. Each of the platform's own processes therefore sets LC_CTYPE, and nothing else, to C.UTF-8 (or C.utf8) when it starts, before any thread or the YARA engine loads, and keeps it only when the C library then folds I to i (core.ctype_locale.pin_character_classification). This covers the analysis worker, the maljan CLI, scripts/validation/run.py, scripts/rehearsal/run.py and the analysis and network sidecars, whose line goes to standard error because standard output is their protocol. The test session pins the same way in tests/conftest.py, and a test that changes the locale itself runs in a subprocess. Messages, times and collation stay the operator's. On a host whose locale is not UTF-8 (a legacy 8-bit one such as tr_TR.ISO-8859-9), the pin also makes UTF-8 Python's preferred encoding, which matches the image; the INFO line says so when the encoding changes. An operator's own tool server keeps the environment it was started with, and so the operator's locale: a YARA engine inside one can still mishandle I and i on such a host. The backend image already runs under LANG=C.UTF-8.

Documentation siteCopy Anchor Link

The pages are MDX files under apps/docs/content/docs/, their images are in apps/docs/public/assets/, and the files served under /examples/ are in docs/examples/. The website at malspark.com is built and published separately on Cloudflare Pages: production reads these three directories from main, previews from dev, so they stay where they are. Pages link to each other as /docs/<page>/. The sidebar's order and groups are content/docs/meta.json.

.github/workflows/docs.yml checks the content on every pull request that touches it; run the same check from apps/docs with npm ci and npm run check. It fails if a page does not compile as MDX, its frontmatter has no title or a field of the wrong type, or it uses a component it neither imports nor gets from the site (the list in scripts/check.mjs must match the site's component map); if a meta.json does not parse or lists a page that is not there; if a link to /docs/..., /assets/... or /examples/..., anchor included, leads nowhere, or a link is relative, outside those three, or points at the old GitHub Pages address; and if an address the old site served (scripts/old-addresses.mjs) no longer leads to a page. On main the workflow also publishes a redirect site to GitHub Pages, so the old address https://root0ne.github.io/Maljan/<page>/ leads to the same page under https://malspark.com/docs/, and a page that moved, such as /paper/, to the page that holds its text now.

Validation codes is generated, not written. Run uv run python scripts/validation_codes/run.py and commit the page after any of these edits:

• adding, removing or renaming a code, or the constant that names it;
• adding, moving or removing a place that raises a code or passes one on, renaming the function it is in included;
• changing a code constant's comment, the comment over a group of code constants, or the message a raise site passes;
• adding or removing a mention of a code on a docs page, or renaming or moving such a page;
• adding or changing a code position the register cannot follow.

Adding, changing or removing a test does not change the page: the tests are not listed on it, and run.py --tests prints which tests name each code. The register reads only the files git tracks, so add a new file before regenerating. tests/unit/test_the_validation_code_register_is_current.py fails when the committed page is not what the source gives, naming the command, when nothing raises a code, and when no test names one.

Continuous integrationCopy Anchor Link

.github/workflows/ci.yml runs on pushes to main and dev and on pull requests into main, dev or feat/**:

Three more workflows run beside CI. CodeQL analyses Python, TypeScript and the workflow files themselves on every push and pull request to main and dev, with the security-extended query suite, and weekly. Dependency review blocks a pull request that introduces a dependency with a known high or critical vulnerability or a copyleft licence. OpenSSF Scorecard runs on main weekly and publishes its findings to code scanning. Dependabot opens weekly update pull requests against dev for the uv workspace, the web console, the pinned GitHub Actions and the Docker base images; security updates arrive as soon as an advisory matches. Every action in a workflow is pinned to a commit with its version in a comment, which Dependabot keeps current. A dependency submission workflow posts the packages resolved in uv.lock to the dependency graph on every push to main, so Dependabot's Python alerts follow the lockfile the way the console's follow package-lock.json. Secret scanning with push protection is on for the repository, and vulnerabilities are reported through private vulnerability reporting (see SECURITY.md).

ConventionsCopy Anchor Link

• Conventional commits, one logical slice per commit.
• Feature branches start from dev and are merged into dev; main is branch-protected and only ever advanced by an explicit promotion pull request from dev.
• No question sentences in headings, comments or documentation.
• A comment explains why the code is the way it is. Process tags — dated audit identifiers, ticket numbers, "new in phase B" — do not belong in the tree; the reasoning stays, the bookkeeping goes.
• New settings need an entry in src/maljan/core/settings_annotations.py, or they appear in the console under their dotted path with no description.
• A removal is not done until grep shows zero remaining references across src apps tests scripts docs and the affected tests are updated.