Architecture

This is the map: where everything lives and how a byte becomes a finding. It links to the authoritative in-code docs rather than restating them, so there is one source of truth per fact. Read this first; then jump to the cited module.

• New contributor? Read Repository layout → The pipeline → the “where do I find X?” table.
• Touching detection? The detector format is data, not code; see the detector reference.
• Touching the scan engine? Its own header doc is the deepest map: crates/scanner/src/engine/mod.rs (“# The one flow” + “# Where each method lives”).
• Choosing between a one-shot scan, a large repository scan, the daemon, and watch? Start with Execution surfaces, then read the daemon workflow and autoroute reference.

Repository layout

Every top-level directory, one line each. Code is Rust under crates/; everything else is data, tooling, docs, or eval harness.

Internal execution planning lives in the private Santh monorepo, not in this public repository.

The crates and their layering

Dependencies point one way: core is the foundation and depends on no other KeyHog crate; cli sits on top and wires the rest together. This DAG is enforced by Cargo and must stay acyclic (domain logic never imports CLI/transport/UI).

            ┌─────────────────────────── cli ───────────────────────────┐
            │  binary, subcommands, daemon, watch, baselines, installer  │
            └───────┬───────────────┬───────────────┬───────────────────┘
                    │               │               │
              ┌─────▼─────┐   ┌─────▼─────┐          │
              │  scanner  │   │  sources  │          │
              │ detection │   │  inputs   │          │
              └──┬─────┬──┘   └──┬─────┬──┘          │
                 │     │         │     │             │
                 │     └────┬────┘     │             │
                 │      ┌───▼────┐     │             │
                 │      │verifier│     │             │
                 │      │  live  │     │             │
                 │      └───┬────┘     │             │
                 └──────────┼──────────┴─────────────┘
                        ┌───▼───┐
                        │ core  │   types · detector registry · report · dedup
                        └───────┘   · allowlists · incremental (merkle) cache

The pipeline: bytes → finding

The end-to-end flow, stage by stage, each pointing at the crate/module that owns it. The scan engine’s own header doc (engine/mod.rs) is the authoritative, method-level version of steps 2-4.

1. Acquire bytes: a source yields file-path + content chunks. crates/sources/src/ (filesystem/, git/, stdin.rs, docker/, s3/, gcs.rs, cloud/azure_blob.rs, github_org.rs, gitlab_group.rs, bitbucket_workspace.rs, hosted_git/, web/, har.rs, strings.rs, binary/).
2. Phase 1: trigger production (which detectors could fire, and where). Swappable backend: scalar CPU literal/regex, SIMD Hyperscan (engine/backend_triggered.rs, engine/scan_coalesced.rs), or the GPU batched literal region-presence route (engine/gpu_region_dispatch.rs). It produces one “which detectors may match here” bitmap per chunk. The fast prefilters (simdsieve, bigram_bloom, alphabet_filter, prefix_trie) live at crates/scanner/src/; detector-to-matcher construction lives in engine/compile.rs, compiler.rs, and compiler/.
3. Phase 2: extraction (the shared tail, identical for CPU and GPU): per-chunk confirmed → phase2 capture → generic → entropy → ML (engine/extract.rs, engine/phase2*.rs, engine/scan.rs). Decode-through (base64/hex/url/unicode/json) runs here and recurses: decode/.
4. Finish raw matches: scanner-owned suppression, confidence, and cross-chunk seam reassembly run in engine/scan_postprocess/, engine/process.rs, and engine/boundary.rs. Confidence + ML scoring live in confidence/, ml_scorer.rs, and ml_scorer/; context inference lives in context/. The per-match policy here (suppression gates · example/placeholder · checksum · confidence penalties) is governed by one invariant; see Match adjudication: one policy, one chokepoint below.
5. Verify (optional): for detectors with a [detector.verify] endpoint, turn a candidate into verified-live, behind SSRF/bogon/rate guards. verifier/.
6. Resolve and report: the CLI orchestrator applies scan-level policy and allowlists; core deduplication and reporters emit text/JSON/SARIF and support baseline comparison. crates/cli/src/orchestrator/postprocess.rs, crates/cli/src/orchestrator/reporting.rs, crates/core/src/dedup.rs, and crates/core/src/report/ own these steps.

The accelerated batch path is two-phase and coalesced: files with no phase-one hit stop before extraction, while full extraction runs only on hits. Large filesystem scans may instead use the fused reader/scanner pipeline so I/O and scanning overlap; crates/cli/src/orchestrator/dispatch.rs and dispatch/fused.rs own that execution choice. Both paths feed the same scanner and report contracts. Backend choice must change performance only, never finding semantics.

Execution surfaces

The CLI owns process-level routing. The scanner crate exposes explicit backend execution; it does not read the autoroute cache or silently choose from local hardware. This keeps library calls deterministic and makes CLI routing inspectable.

Persisted backend selection lives under crates/cli/src/orchestrator/dispatch/backend.rs and orchestrator/dispatch/backend/. Daemon transport and lifecycle live under crates/cli/src/daemon/. See the operator references for cache-miss, cold-versus-warm, and active-versus-inactive daemon behavior.

The routing package keeps measurement, proof, and persistence separate:

This separation is deliberate: persisted bytes cannot define routing policy, inspection cannot bypass cache validation, and performance evidence cannot silently weaken detection parity.

Finding identity and dedup

There is one identity contract with stage-specific keys, not interchangeable “same finding” guesses:

The required seam test is scan_windowed_overlap_dedups_end_to_end: a token placed wholly inside the 128 KiB overlap must scan as one raw match and one final reported finding.

Match adjudication: one policy, one chokepoint

Governing invariant. Whether a candidate match becomes a reported finding, and at what confidence, is a pure function of the value and its context, never of which emission path produced it. A value that is a ${} shell template, a name-name:v1 public identifier, or Config-Word-and-Word-only policy prose is not a secret no matter whether the entropy detector, the generic keyword bridge, the weak-anchor post-pass, or the hot-pattern fast path surfaced it. Phase-2 has several emission paths; they exist for speed and recall, not to each carry their own copy of policy.

The rule. Emission paths produce CandidateMatch values and typed signals; adjudicate_match owns the ordered suppression verdict. Path owners may compute context-specific facts (entropy shape, generic bridge boundaries, named detector policy), but they do not invent an untyped final drop reason:

emission paths (entropy · generic/keyword bridge · weak-anchor · hot fast path · GPU)
        │  each yields CandidateMatch { detector, span, value }
        ▼
adjudicate_match(CandidateMatch, MatchCtx)
   1. explicit/process signals
   2. generic/entropy/hot-pattern signals
   3. named-detector suppression
   4. final report-floor policy
        ▼
   Verdict::Suppressed(stage_name)  |  Verdict::Reported(confidence)

MatchCtx carries one explicit signal family at a time. The Verdict names the deciding StageId, which is what dogfood telemetry records. Shared shape policy lives under suppression::shape; path-specific callers convert its result into the matching typed signal before adjudication.

Why this shape. Candidate discovery necessarily differs by detector family, but the final vocabulary and ordering of suppression decisions must not. Typed signals preserve the context each path needs while keeping one auditable verdict pipeline and one telemetry reason per decision.

The ML model (weights.bin)

The scanner serves a Mixture-of-Experts confidence model embedded at build time (crates/scanner/src/weights.bin, include_bytes!). It is trained out-of-band by the Python pipeline in ml/:

ml/harvest_corpus.py   real labelled candidates (CredData), harvested at a LOW
                       report floor so sub-floor hard negatives are captured
        │
ml/train_classifier.py blend synthetic + real, file-grouped split (no leakage),
                       train the 42-feature MoE, gate on held-out F1 plus
                       aggregate, per-class, and per-detector real recall
        │
ml/retrain_loop.sh     one command: harvest → train → (--write) ship weights.bin
                       → (--verify) rebuild + per-detector-FP bench gate,
                       fail-closed revert on any regression

Because the model is compile-time-embedded, a new model is only observable after a rebuild, which is why --verify rebuilds before benching. The adjacent crates/scanner/src/model_card.json carries the model hash, training inputs, and gate metrics; build.rs refuses a card/weights mismatch and embeds the summary shown by keyhog --version.

Where do I find X?