The Problem Nobody Talks About
Most AI security discussions focus on what goes into the model — prompt injection, jailbreaking, data poisoning. There is an equally critical and less discussed attack surface: what comes out.
Large language models hallucinate. This is well known. What is less appreciated is that they sometimes hallucinate data that looks exactly like real personally identifiable information: social security numbers, credit card numbers, phone numbers, and email addresses that happen to belong to real people.
This is not a theoretical risk. We have documented cases where models generated valid-format SSNs, syntactically correct credit card numbers that pass Luhn checks, and real email addresses of public figures — all in response to completely benign prompts.
Why Input-Side Scanning Is Not Enough
If you are only scanning prompts for PII before they reach the model, you are solving half the problem.
The model can generate PII from its training data. Even if the input contains zero PII, the model may produce PII that it memorized during training. This is especially common for public figures, well-known organizations, and data that appeared frequently in the training corpus.
Hallucinated PII passes format validation. A hallucinated SSN like "482-93-7156" has the correct format, length, and structure. Without checking against a known-good list (which raises its own privacy concerns), you cannot distinguish it from a real one — so you must treat it as real.
Aggregation risk. A model might not produce a full identity in a single response, but across multiple responses to the same user, it might generate enough fragments (name, city, employer, approximate age) to constitute PII under regulations like GDPR and CCPA.
UMAI's POST_LLM Architecture
UMAI's POST_LLM layer sits between the model's response and your end user. Every response passes through the following pipeline.
Stage 1: Pattern Detection
We scan for 47 PII patterns across 12 categories:
- Government IDs: SSN, passport numbers, driver's license numbers (US, EU, and 8 additional country formats)
- Financial: credit card numbers, bank account numbers, routing numbers, IBAN
- Contact: email addresses, phone numbers, physical addresses
- Health: medical record numbers, health plan IDs, NPI numbers
- Digital: IP addresses, MAC addresses, device identifiers
- Custom: organization-specific patterns defined by the customer
Pattern detection runs in under 0.4ms for a typical response.
Stage 2: Contextual Analysis
Not every pattern match is PII. The string "123-45-6789" in a discussion about SSN format examples is different from the same string appearing as someone's actual SSN. Our contextual analyzer evaluates:
- Surrounding text: is this string presented as an example, a placeholder, or real data?
- Named entity association: is the detected pattern linked to a person's name?
- Prompt context: did the user ask for real data, or is this an educational discussion?
This stage reduces false positives by 67% compared to pattern-only detection.
Stage 3: Enforcement
When PII is detected, UMAI applies the organization's configured policy:
- Redact: replace the PII with a placeholder like [SSN REDACTED] and deliver the rest of the response
- Block: reject the entire response and return a safe error message
- Alert: deliver the response but log an alert for security review
- Tag: add metadata tags for downstream systems to handle
The enforcement action, the detected PII type (but not the PII itself), and a hash of the original content are recorded in the tamper-evident audit log.
Real-World Impact
One of our healthcare customers discovered that their customer-facing AI assistant was generating realistic-looking Medical Record Numbers (MRNs) when users asked questions about medical billing. The MRNs had the correct format for their EHR system, which meant they could theoretically be used to query real patient records.
UMAI's POST_LLM layer caught and redacted those hallucinated MRNs before they reached users. In the first month after deployment we detected and redacted 1,247 instances of hallucinated health-related PII.
Recommendations
- Deploy POST_LLM scanning immediately. Even if your model never receives PII in prompts, it can generate PII in responses.
- Use contextual analysis, not just regex. Pattern-only detection generates too many false positives to be useful in production.
- Configure enforcement per PII type. Not all PII is equally sensitive. A detected email address might warrant an alert, while a detected SSN should trigger an immediate block.
- Monitor your false-positive rate. We target < 0.5% false positives. If your rate is higher, your team will start ignoring alerts.
- Review the POST_LLM audit log weekly. Trends in PII detection often reveal upstream issues — like a training data leak or a misconfigured RAG pipeline — that should be fixed at the source.
UMAI's POST_LLM layer is available on all plans. Set it up in under 10 minutes at app.umaisolutions.com.