tftsr-devops_investigation/docs/architecture/adrs/ADR-004-pii-regex-aho-corasick.md

ADR-004: Regex + Aho-Corasick for PII Detection

Status: Accepted Date: 2025-Q3 Deciders: sarman

---

Context

Log files submitted for AI analysis may contain sensitive data: IP addresses, emails, bearer tokens, passwords, SSNs, credit card numbers, MAC addresses, phone numbers, and API keys. This data must be detected and redacted before any content leaves the machine via an AI API call.

Requirements:

• Fast scanning of files up to 50MB
• Multiple pattern types with different regex complexity
• Non-overlapping spans (longest match wins on overlap)
• User-controlled toggle per pattern type
• Byte-offset tracking for accurate replacement

---

Decision

Use Rust regex crate for per-pattern matching combined with aho-corasick for multi-pattern string searching. Detection runs entirely in the Rust backend on the raw log content.

---

Rationale

Alternatives considered:

Option	Pros	Cons
regex + aho-corasick (chosen)	Fast, Rust-native, no external deps, byte-offset accurate	Regex patterns need careful tuning; false positives possible
ML-based NER (spaCy, Presidio)	Higher recall for contextual PII	Requires Python runtime, large model files, not offline-friendly
Simple string matching	Extremely fast	Too many false negatives on varied formats
WASM-based detection	Runs in browser	Slower; log content in JS memory before Rust sees it

Implementation approach:

1. 12 regex patterns compiled once at startup via lazy_static!
2. Each pattern returns (start, end, replacement) tuples
3. All spans from all patterns collected into a flat Vec<PiiSpan>
4. Spans sorted by start offset
5. Overlap resolution: iterate through sorted spans, skip any span whose start is before the current end (greedy, longest match)
6. Spans stored in DB with UUID — referenced by approved flag when user confirms redaction
7. Redaction applies spans in reverse order to preserve byte offsets

Why aho-corasick for some patterns: Literal string searches (e.g., password=, api_key=, bearer ) are faster with Aho-Corasick multi-pattern matching than running individual regexes. The regex then validates the captured value portion.

---

Patterns

Pattern ID	Type	Example Match
url_credentials	URL with embedded credentials	https://user:pass@host
bearer_token	Authorization headers	Bearer eyJhbGc...
api_key	API key assignments	api_key=sk-abc123...
password	Password assignments	password=secret123
ssn	Social Security Numbers	123-45-6789
credit_card	Credit card numbers	4111 1111 1111 1111
email	Email addresses	user@example.com
mac_address	MAC addresses	AA:BB:CC:DD:EE:FF
ipv6	IPv6 addresses	2001:db8::1
ipv4	IPv4 addresses	192.168.1.1
phone	Phone numbers	+1 (555) 123-4567
hostname	FQDNs	db-prod.internal.example.com

---

Consequences

Positive:

• No runtime dependencies — detection works fully offline
• 50MB file scanned in <500ms on modern hardware
• Patterns independently togglable via pii_enabled_patterns in settings
• Byte-accurate offsets enable precise redaction without re-parsing

Negative:

• Regex-based detection has false positives (e.g., version strings matching IPv4 patterns)
• User must review and approve — not fully automatic (mitigated by UX design)
• Pattern maintenance required as new credential formats emerge
• No contextual understanding (a password in a comment vs an active credential look identical)

User safeguard: All redactions require user approval via PiiDiffViewer before the redacted log is written. The original is never sent to AI.