How detection works

A KeyHog scan is a pipeline. Files come in one side, findings go out the other. In between, four stages:

files → [chunker] → [prefilter] → [detector match] → [post-process] → findings

Most chunks that fail the cheap prefilter stop there, which keeps full regex evaluation focused on plausible inputs. This is not an unconditional hard drop: a rejected chunk that looks encoded can enter a bounded decode-only recovery pass (recursively decoding up to a max_decode_depth, defaulting to 10), so an encoded secret is not lost merely because its plaintext anchor is absent from the original bytes.

Detection mechanisms

KeyHog does not use one universal test for “secret-like.” It composes several mechanisms, and their roles are deliberately different:

Regex, generic extraction, entropy, and decode-through therefore find different candidate classes. Named regexes and generic assignment extraction create candidates; companions, validators, BPE, shape/context policy, and confidence then confirm, reject, or score them. Verification runs only after a candidate survives detection and reporting policy.

BPE is not a replacement name for entropy: it is an independent post-candidate signal. BetterLeaks calls the approach Token Efficiency; KeyHog uses the same broad BPE idea while keeping its own detector schema, thresholds, pipeline, and behavioral evidence.

Terminology matters here: BetterLeaks’ public documentation names the feature Token Efficiency and describes BPE tokenization as a natural-language false positive filter; it does not present “BPD” as a separate score. KeyHog names its related mechanism BPE token efficiency, uses cl100k_base, measures UTF-8 bytes per token, and resolves the ceiling per detector. If “BPD” is being used informally to mean a bits/byte or bytes/token density, do not treat it as a third implemented score: Shannon entropy and BPE token efficiency are the two separate signals documented here.

Detector-owned tuning: what each setting changes

Detection policy belongs in the detector TOML whenever the choice is specific to a credential type. Scan-wide CLI/TOML values are operational overrides for controlled comparisons or a corpus-wide policy; they are not a second hidden detector definition. keyhog explain <detector-id> shows the policy declared by that detector TOML and its provenance; keyhog config --effective shows the resolved scan-wide policy.

Resolution rules

These settings do not all use one generic “last value wins” rule:

• BPE ceiling: the compiled fallback is 2.2 UTF-8 bytes per cl100k_base token. The owning detector’s bpe_max_bytes_per_token replaces that fallback. An explicitly supplied [scan].entropy_bpe_max_bytes_per_token or --entropy-bpe-max-bytes-per-token replaces every BPE-enabled entropy/generic detector ceiling; the CLI wins over the config file. bpe_enabled = false still disables the gate for that detector.
• Confidence floor: the scan floor defaults to 0.40. A detector TOML min_confidence replaces the scan floor for that detector, and an operator [detector.<id>].min_confidence replaces the detector-declared floor. Under --precision, the resolved global and per-detector floors are clamped to at least 0.85; neither source can weaken the precision preset.
• Entropy policy: omitted detector fields use 4.5 (entropy_high), 3.0 (entropy_low), 5.8 (entropy_very_high), and 4.0 (mixed_alnum_floor). Detector TOML values replace those individual fallbacks. The scan-wide entropy_threshold is deliberately not a blanket replacement for all four bands. On the phase-2 generic bridge it tightens only when it exceeds the owning detector’s high band. On the entropy scanner, a value above that high band tightens keyword and isolated candidates; a value below the keyword detector’s low band loosens that keyword path, while values between the low and high bands leave its low floor in place. The isolated path keeps its mixed-alphanumeric floor unless the scan threshold exceeds the high band. These rules preserve the different evidence carried by an assignment key, an isolated opaque token, and an unanchored generic value.

Token efficiency can carry more of the precision burden for a detector whose assignment key or regex already creates the candidate. That is the practical per-detector alternative to making Shannon entropy the decisive signal: use a permissive detector-owned entropy floor appropriate to the credential family, then let its BPE, shape, context, and confidence policy reject word-like noise. It is not equivalent to blindly replacing entropy with one global BPE number, and bpe_enabled alone never creates a candidate.

Detector-owned canonical_hex_key_material is the deliberate exception to the BPE and generic low-diversity/decode-as-data gates. Hexadecimal key bytes tokenize efficiently and use a small alphabet for the same mechanical reasons hexadecimal digests do, so the exact detector-owned keyword/length contract supplies the discriminator. Placeholder, degenerate-repeat, entropy, context, and reporting gates remain active. When ML is enabled, this exact TOML match is structural positive evidence and therefore preserves the detector heuristic floor; the model may raise its score but cannot erase a policy-proven key as if it were an unowned entropy candidate.

Scan-wide settings remain operational controls, but they do not all compose the same way. Explicit CLI values take precedence over config-file values. An explicit scan-wide BPE ceiling takes precedence over detector-local BPE ceilings so a benchmark can compare one bound consistently. entropy_threshold can tighten a detector’s high band but does not silently replace its lower detector-owned keyword band. A detector’s min_confidence replaces the global reporting floor for that detector, and [detector.<id>] min_confidence is the operator override for that one ID. For production detector tuning, put the stable value in the owning detector TOML and prove it with that detector’s positive, negative, evasion, backend-parity, and corpus contracts.

Settings, hardware, and result parity

Hardware changes execution, not detection policy. CPU, SIMD/Hyperscan, and GPU routes consume the same resolved detector/config digest. Autoroute admits a candidate only when its canonical detection identities match the reference: chunk membership, detector id/name/service/severity, hashed credential value, stored credential hash, hashed companion names and values, source, file, line, byte offset, commit, author, date, entropy, confidence, and multiplicity. Confidence, suppression, verification, deduplication, and output formatting then run through the shared policy tail. Missing or stale exact evidence is an error; calibration never relaxes a detector to make a backend look faster.

Configuration Presets

• --fast (or ScannerConfig::fast()): Disables high-FP generic entropy checks, ML, and deep decoding (max_decode_depth = 0). Maximizes throughput.
• --deep (or ScannerConfig::thorough()): Admits unanchored generic high-entropy strings, enabling deep decoding (max_decode_depth = 10), ML scoring, and entropy sweeps. Maximizes recall.
• --precision (or ScannerConfig::high_precision()): Sets min_confidence to 0.85 (HIGH_PRECISION_MIN_CONFIDENCE), keeps ML enabled, limits decoding depth (max_decode_depth = 1), and disables high-FP generic entropy checks. Maximizes precision.

Strict Backend Parity

KeyHog supports three search backends: pure Rust CPU, SIMD/Hyperscan (simd-regex), and GPU/VYRE region presence. Portable builds retain the pure-Rust trigger path without Hyperscan. keyhog calibrate-autoroute measures every eligible backend for the host/config/workload key and rejects candidates whose canonical match identity differs from the reference. It records the first real GPU dispatch plus warm trials: an ordinary process resolves against the cold-aware GPU cost, while a daemon that initialized its engines before readiness resolves against the warm GPU evidence. A missing or invalid decision is an error; automatic routing never silently substitutes another backend.

When comparing settings, record the effective config, detector digest, input identity, backend, host/accelerator identity, and complete findings, not only elapsed time or finding count. A faster run with a different result set is a detection change or parity failure, not a routing win.

Stage 1 - chunker

A file becomes one or more chunks. A chunk is {data: str, metadata: {source_type, path, line_offsets, …}}. The chunker:

• Skips obvious binaries via magic-byte sniffing (PDF, PNG, zip, …).
• Skips files matching is_default_excluded_path (node_modules, .min.js, build/, etc.).
• Splits files larger than the 1 MiB window size into overlapping ~1 MiB windows so a single giant log file doesn’t blow scratch memory. Each window carries its absolute base byte offset and base line so findings report the real file offset/line, not the per-window one. Cross-window secrets are reassembled in stage 4.
• Decodes UTF-16 BOM files into UTF-8 (PowerShell / .NET configs).

Specialized chunkers run too:

• Git history → one chunk per (commit × file × diff line)
• Docker images → one chunk per layer × file
• Web URLs → one chunk per response body / sourcemap / WASM strings
• S3 buckets → one chunk per object body
• GCS buckets → one chunk per object body
• Azure Blob containers → one chunk per blob body

Stage 2 - prefilter (the cheap pass)

Three gates, in order, each cheaper than the next:

1. Alphabet screen. A 256-bit mask of which bytes the corpus’s detectors care about. A chunk with no relevant byte becomes a prefilter miss.

2. Bigram bloom. A 4096-bit bloom filter of 2-byte sequences from detector keyword prefixes. A chunk with no overlapping bigram becomes a prefilter miss. This cheaply recognizes source that carries no relevant anchor vocabulary.

After these screens, ordinary misses stop. Decode-shaped misses instead take the bounded decode-only path described above; transformed plaintext is then attributed back to the original source.

3. Backend trigger pass. The simd-regex backend compiles the detector corpus into Hyperscan databases when the simd feature is present; cpu-fallback uses the pure-Rust trigger path. One pass returns which detector IDs have a candidate match.

   GPU-capable builds add VYRE’s region-presence literal-set backend. There is no universal model-name or byte threshold at which KeyHog silently switches to it. --backend auto requires an exact persisted calibration decision for the current binary, detector/config digest, host/device/driver, workload class, and size bucket. Calibration keeps a GPU route only when its canonical match identities equal the reference and it is the fastest eligible backend for that key.

Stage 3 - detector match

For each pattern-backed detector that the prefilter flagged, the full regex evaluates. The regex is detector.patterns[].regex in that detector’s TOML, and its configured capture group becomes the candidate credential. Generic phase-2 detector TOMLs use keyword, length, entropy, token-efficiency, and shape policy for shapeless assignments or isolated opaque values. They may also carry explicit patterns for strongly structured envelopes such as JSON "secret", "token", or "apiKey" fields; both mechanisms remain owned by the same detector TOML instead of a central compatibility detector.

A detector’s .toml carries:

• id, name, service, severity, keywords
• zero or more patterns, each with regex + group + optional description (required for service-anchored detectors; optional structured-envelope anchors for phase2-generic)
• optional companions (e.g. AWS access key needs the secret key nearby)
• optional verify block - HTTP method, URL template, auth scheme, success status

Detectors fall into two camps:

• Service-anchored. Regex requires a service-specific keyword (AWS_SECRET_ACCESS_KEY=, stripe.com/v1/, dn_ Deepnote prefix). These have HIGH precision: the keyword itself is positive evidence, not just a hint.

• Generic / entropy discovery (generic-password, entropy-api-key, entropy-token). Triggered by entropy + assignment shape only - password = "...", secret: "...", JSON { "token": "..." }. Lower precision; suppression filters do most of the work.

  Surviving candidates also pass a BPE token-efficiency gate. Shannon entropy asks how evenly bytes are distributed; token efficiency asks how readily a fixed subword vocabulary compresses the value. Dotted API names and prose can have high Shannon entropy but tokenize into a few common pieces, while opaque secrets usually require many short tokens. The mechanisms are complementary, and generic detector TOMLs may own their token-efficiency ceiling through bpe_max_bytes_per_token. Opaque API-key/secret policies use the measured 2.3 ceiling; password/passphrase policies set bpe_enabled = false because human-chosen credentials may intentionally be word-like. Disabled policies skip tokenizer work entirely rather than using a magic oversized ceiling.

The entropy-generic, entropy-password, entropy-token, and entropy-api-key IDs are output classifications for entropy-discovered findings, not four additional detector TOML files. Their candidate policy is owned by the corresponding phase-2 TOMLs selected from the assignment context: generic-secret, generic-password, generic-keyword-secret, or generic-api-key. Use keyhog explain on those owning detector IDs when tuning entropy, BPE, length, or canonical-key policy.

The split matters for the post-process stage.

Stage 4 - post-process

Even a regex match isn’t always a credential. Stage 4 filters:

• Known example fixtures (Stripe docs key, AWS docs key, RFC 7519 specimen JWT).
• Placeholder language - credentials containing YOUR_, INSERT, EXAMPLE, PLACEHOLDER, TODO, FIXME, etc.
• Shape gates.
  • Universal: punctuation_decorated_identifier - credentials starting with --, &, @, !, /, $ (CLI flags, pointers, SQL vars, shell vars, GraphQL refs) or ending in : / ! (UI labels, TypeScript non-null assertions).
  • Generic / entropy only: pure_identifier, word_separated_identifier, scheme_prefixed_uri, url_or_path_segment, contains_uuid_v4_substring. These shapes can be real credentials when paired with a service or protocol anchor, so named detector TOMLs and structural authorization detectors own those cases. A generic token=<uuid> remains an identifier; an Authorization: Bearer <uuid> value is a credential because the Bearer envelope supplies the missing evidence. Public salts and nonces are not generic secrets. A detector for a product whose field is genuinely secret despite that name must own the product syntax explicitly.
• Path-based suppressions - vendored bundles (node_modules/, wp-includes/, bower_components/), CI workflow files (where ${{ secrets.NAME }} references are syntactic, not credentials), i18n translation files, secret-scanner source files (the file IS a scanner; its regex literals shouldn’t fire on itself).
• Cross-chunk reassembly. A secret split across window boundaries gets reassembled from the tail of chunk N + the head of chunk N+1.

A finding that survives stage 4 makes it to output.

Where the speed comes from

The alphabet screen and bigram bloom reject irrelevant chunks before regex confirmation. Literal triggers narrow the active detector set, and the scanner shares confirmation, suppression, and reporting tails across CPU and GPU backends. Windowing bounds scratch space for large inputs; caches avoid repeated compiler and index work.

End-to-end throughput depends on the detector/config digest, source shape, candidate density, decoding and verification policy, cache state, CPU, GPU, driver, and storage. Use keyhog calibrate-autoroute for routing evidence on the installed host and the repository benchmark harness for reproducible cross-version measurements; do not treat a throughput number from another machine or detector corpus as a routing threshold.

Where the precision comes from

Each filter has a known-FP-cluster it was built to defuse. The Suppressions page enumerates them with examples.

What this looks like for one finding

file.env contains: AWS_SECRET_ACCESS_KEY=ev0BsFtSD7S/4VWYObxiEhME3hJBXeYzR43jgiB1

stage 1 - chunker:        emit chunk{ path: "file.env", data: "AWS_SECRET..." }
stage 2 - alphabet:       PASS (chunk has `=`, alphanumerics from the corpus)
stage 2 - bigram bloom:   PASS (`AW`, `WS`, `_S` are in the bloom)
stage 2 - simd-regex:     MATCH → triggers `aws-secret-access-key` + `generic-password`
stage 3 - regex eval:
  `aws-secret-access-key` detector pattern captures the 40-byte value
    captures `ev0BsFtSD7S/4VWYObxiEhME3hJBXeYzR43jgiB1`
  `generic-password` regex doesn't match (no `_password`/`_pwd` substring)
stage 4 - post-process:
  known-example check: no
  `looks_like_pure_identifier`: false (has digits + /)
  `looks_like_punctuation_decorated_identifier`: false
  → EMIT

That’s one finding’s life. Multiply by 10⁶ files and the throughput math is why each stage matters.