tftsr-devops_investigation/docs/wiki/PII-Detection.md

PII Detection

Overview

Before any text is sent to an AI provider, TFTSR scans it for personally identifiable information (PII). Users must review and approve each detected span before the redacted text is transmitted.

Detection Flow

1. Upload log file
      ↓
2. detect_pii(log_file_id)
   → Scans content with PII regex patterns (including hostname + expanded card brands)
   → Resolves overlapping matches (longest wins)
   → Returns Vec<PiiSpan> with byte offsets + replacements
      ↓
3. User reviews spans in PiiDiffViewer (before/after diff)
   → Approves or rejects each span
      ↓
4. apply_redactions(log_file_id, approved_span_ids)
   → Rewrites text with replacements (iterates spans in REVERSE order to preserve offsets)
   → Records SHA-256 hash of redacted text in audit_log
      ↓
5. Redacted text safe to send to AI

Detection Patterns

Type	Replacement	Pattern notes
UrlWithCredentials	[URL]	scheme://user:pass@host
BearerToken	[Bearer]	Case-insensitive bearer keyword + token chars
ApiKey	[ApiKey]	api_key=, apikey=, access_token= + 16+ char value
Password	[Password]	password=, passwd=, pwd= + non-whitespace value
Ssn	[SSN]	\b\d{3}-\d{2}-\d{4}\b
CreditCard	[CreditCard]	Visa/MC/Amex/Discover/JCB/Diners patterns
Email	[Email]	RFC-compliant email addresses
MacAddress	[MAC]	XX:XX:XX:XX:XX:XX and XX-XX-XX-XX-XX-XX
Ipv6	[IPv6]	Full and compressed IPv6 addresses
Ipv4	[IPv4]	Standard dotted-quad notation
PhoneNumber	[Phone]	US and international phone formats
Hostname	[Hostname]	FQDN/hostname detection for internal names
UrlCredentials	(covered by UrlWithCredentials)

Overlap Resolution

When two patterns match overlapping text, the longer match wins:

let mut filtered: Vec<PiiSpan> = Vec::new();
for span in sorted_by_start {
    if let Some(last) = filtered.last() {
        if span.start < last.end {
            // Overlap: keep the longer span
            if span.end - span.start > last.end - last.start {
                filtered.pop();
                filtered.push(span);
            }
            continue;
        }
    }
    filtered.push(span);
}

PiiSpan Struct

pub struct PiiSpan {
    pub id: String,          // UUID v7
    pub pii_type: PiiType,
    pub start: usize,        // byte offset in original text
    pub end: usize,
    pub original: String,
    pub replacement: String, // e.g., "[IPv4]"
}

Redaction Algorithm

Spans are applied in reverse order to preserve byte offsets:

let mut redacted = original.to_string();
for span in approved_spans.iter().rev() {  // reverse!
    redacted.replace_range(span.start..span.end, &span.replacement);
}

Audit Logging

Every redaction and every AI send is logged:

write_audit_event(
    &conn,
    "ai_send",                        // action
    "issue",                          // entity_type
    &issue_id,                        // entity_id
    &json!({
        "log_file_ids": [...],
        "redacted_hash": sha256_hex,  // SHA-256 of redacted text
        "provider": provider_name,
    }).to_string(),
)?;

Security Guarantees

• PII detection runs locally — original text never leaves the machine
• Only the redacted text is sent to AI providers
• The SHA-256 hash in the audit log allows integrity verification
• If redaction is skipped (no PII detected), the audit log still records the send
• Stored pii_spans.original_value metadata is cleared after redaction is finalized