title: "Why Binary PII Detection Fails Compliance" description: "Detected/not-detected flags cannot support defensible redaction decisions. Confidence scoring transforms PII anonymization from a binary guess into an auditable compliance control." category: technical publishedAt: 2026-06-21 tags:
- confidence scoring
- PII detection
- legal discovery
- compliance
- GDPR audit readingTime: 8
Why Binary PII Detection Fails Compliance
Updated for 2026
Every PII tool faces one hard problem. The same string can be personal data in one place and not in another.
"John" in a customer file is a data subject. "John" in a history paper about John F. Kennedy is not. A nine-digit number in a medical record is a HIPAA code. The same nine digits in a product code are not.
A yes/no flag cannot handle this. It forces two bad choices: redact all strings that might be PII, or redact only sure matches. Both fail in law, where every decision must be clear and documented.
A per-entity score from 0 to 100 offers a third path. It drives tiered rules, human review queues, and full audit records.
The Limit of Yes/No Flags
Context changes the meaning of data. Two files can hold the same string. In one, it is personal data. In the other, it is not. A flag cannot show that. A number can.
With only a flag, your two options are bad. Over-redaction kills document value. Under-redaction creates legal risk. Neither holds up in court.
Legal Discovery: Why Scores Are Needed
Legal discovery has rules that make scored detection a must.
The over-redaction problem. Redacting attorney names or court citations damages the evidence. Courts have fined attorneys for over-redaction. The same case law that covers under-redaction covers this too.
The under-redaction problem. Missing real PII creates risk. That includes client privacy breaches, bar complaints, and in some places, criminal charges.
The need to explain each call. When a court asks why an item was redacted, attorneys must explain it. "The tool flagged it" is not enough. "The tool scored this at 94% as a Social Security Number. Our rule auto-redacts above 85%." That is enough.
A yes/no flag cannot give that answer. A scored tool with set rules can. See also: Defending Redactions: AI Scores in Court.
A Three-Tier Review System
The most effective setup uses three tiers based on the entity score.
Tier 1 — Auto (above 85%):
- Items that match high-certainty formats (SSN, IBAN, MRN)
- Auto-redacted with no human step
- Log records entity type, score, method, and time
- Example: "571-44-9283" at 97% as SSN — auto-redacted
Tier 2 — Human review (50–85%):
- Items that may be PII but need a judgment call
- Sent to a reviewer to accept, reject, or reclassify
- Log records entity type, score, reviewer ID, decision, and time
- Example: "John Davis" in a tech doc at 67% — reviewer confirms it is a name — redacted
Tier 3 — Suggestion only (below 50%):
- Low-certainty items shown as tips
- Not auto-redacted; reviewer may act or skip
- Log records entity type, score, and reviewer choice
- Example: "Smith" in a product doc at 42% — reviewer finds it is a firm name — not redacted
Only Tier 2 needs human work. All three tiers produce audit records.
How Scores Are Built
PII tools combine signals to produce one number per entity.
Regex patterns. An exact SSN-format match gets a high base score. A partial match gets a lower one.
Model output. Named entity models assign a probability per class. A score of 0.93 for PERSON gives a high-certainty result.
Context signals. Text around the entity adjusts the score. "My SSN is 571-44-9283" raises it. "Product code 571-44-9283" lowers it.
Ensemble rules. Systems combine regex, model, and context signals with set weights. The final number reflects all the evidence.
That number drives every threshold decision in your workflow. For more on false positives from yes/no tools, see: The False Positive Tax on PII Tools.
Insurance Claims: A Real Example
Insurance files mix clear PII — policyholder name, address, SSN — with context-dependent data: witness names, firm names, adjuster signatures.
A yes/no tool either redacts all names (wrong for firms) or misses witness names (a risk). A scored tool handles each item on its own:
- SSN with label "policyholder SSN" at 96% — auto-redacted
- Policyholder name tagged PERSON at 91% — auto-redacted
- Contractor firm tagged ORG at 78% — reviewed — reviewer rejects redaction
- Witness name tagged PERSON at 82% — reviewed — reviewer accepts
- Adjuster name tagged PERSON at 71% — reviewed — reviewer accepts (third-party data)
Each call has a numeric basis. The audit trail is full.
Building Compliance Records
For GDPR Article 5(1)(f) and the HIPAA Security Rule, scored tools generate records on their own.
Entity-level audit records capture entity type, score, decision type (auto or manual), reviewer ID, and time. These export as CSV for data authority inquiries.
Threshold records document current settings and every change. Each change includes who made it, when, and why. This shows a managed, deliberate policy.
Stats reports cover detection rates by entity type, Tier 2 review rates, and override rates. They answer a data authority asking to "show us your controls."
For HIPAA audit trail guidance, see: Explainable Redaction: HIPAA Audits.
A yes/no flag is a guess. A score is evidence.
When This Approach Has Limits
Replacing a yes/no flag with a per-entity score is the right move — it enables tiered review, defensible thresholds, and a real audit trail. But three limits apply.
A score is only as good as the detection beneath it. The number reflects regex, model, and context signals, but a missed entity gets no score at all. A residual false-negative rate sits underneath every threshold: the system can only assign 67% or 94% to text it surfaced as a candidate. Quasi-identifiers spread across a document — a date, a role, a location that together identify someone — rarely trigger a single confident entity score, so output can be pseudonymized rather than anonymized even when every flagged item is handled well. Calibrate scores against held-out data so the numbers mean what your thresholds assume.
Scores inform the legal judgment; they do not make it. A redaction decision in discovery must withstand a judge, and "the tool scored this at 94%" supports an explanation rather than supplying one. Whether a name is privileged, whether a citation is properly disclosable, whether third-party data must be protected — these are legal standards that require human or expert judgment, not a model output. The three-tier system rightly routes the ambiguous band to a reviewer. Keep that human step; a score is a defensible input to a decision, not the decision.
Thresholds are policy choices with consequences. Setting auto-redact at 85% trades false positives against false negatives, and that line determines how much PII slips through versus how much value is destroyed. The right cutoff depends on the document type, the regulator, and your risk tolerance — there is no universal correct number. A miscalibrated model can be confidently wrong, assigning a high score to a false match or a low one to real PII. Review threshold settings deliberately, log every change, and re-test when the model or the data distribution shifts.