Indirect prompt injection · measurement instrument

The Substring Trap
how a detector invents its own results

A keyword scanner searched 207,391 AI-agent posts for prompt injection and reported 9.87%. The real figure was 1.48%. Nothing changed but three lines of matching logic. Type into the box to watch the bug happen live — then read the 10,176 false positives it produced, every one of them a real item from the corpus.

How to use this

What this page is

A keyword scanner was used to measure how much prompt injection appears on social networks where the users are AI agents rather than people. The first version of that scanner had a bug, and the bug made the problem look about seven times worse than it is.

This page lets you run both versions — the broken one and the fixed one — on any text you like, and read every mistake the broken one made.

The bug, in one line

The scanner searched the way Ctrl+F does. Searching for cat also finds catalogue and delicate. The scanner was searching for DAN — a real jailbreak name — and kept finding it inside redundant, dance and dangerous.

Try it

  1. Type anything into the box at the top, or click one of the Try buttons.
  2. Watch the two panels. Left is the broken scanner, right is the fixed one.
  3. Untick the two Fix boxes to turn each correction off and see which one catches what.

The presets each show a different lesson — redundant is the classic false positive, DAN in caps is a genuine hit, and talking about it shows that writing about injection does not trigger the scanner at all.

What the colours mean

redThe scanner matched here and was wrong — the letters sit inside an innocent word.
tealA genuine whole-word match. Still only a candidate, not a confirmed attack.

Reading the evidence

Scroll to The evidence for 13,528 real items from the corpus, split across three tabs. Search the text, filter by keyword or by post/comment, and click any row to load it into the bench at the top.

  • False positives — items the broken scanner flagged and the fixed one does not. Set the keyword filter to dan to see the 7,304 worst.
  • Real matches — items that survive whole-word matching.
  • Clawk — a second platform, scanned with the same 56 keywords.

What it cannot tell you

Only whether a keyword matched. It cannot tell an agent discussing prompt injection from one performing it — that still needs a human to read the item and decide. This page narrows 207,391 items down to about 3,000. It does not finish the job.

Two honest numbers, both with stated denominators: 1.48% matched any keyword (an upper bound) and 0.083% matched high-confidence phrases (a lower bound). The truth is somewhere between them.

Please read before using the handles

Every row shows the agent handle that wrote it. Some agent accounts are linked to real people. Do not use this to target individual accounts.

This matters most on the false-positive tab, where every single handle belongs to someone who did nothing wrong. They wrote redundant, or overshadowed. The red marks are the scanner's mistakes, not their behaviour.

Scan bench

Try
Broken · substring if kw in text
0keywords
Fixed · boundary (?<!\w)kw(?!\w)
0keywords

The taxonomy · 56 keywords, 6 categories

Every keyword is an instruction-shaped phrase — something an attacker would write at a model. None of them are topic words. prompt injection, attack and exploit are deliberately absent, so writing about injection does not trip the scanner. Keywords marked Aa are matched in capitals only.

The evidence · 13,528 real specimens

What the bug cost

7,706hits on dan

Almost all of them inside redundant, dance, dangerous, redundancy. Not one was an attack.

75.7%rested on 8 words

Three quarters of every finding came from eight short single-word keywords — the ones a substring test mangles worst.

9.87%published, March

Reported in good faith from the broken scanner, and wrong. The dataset card now says so in the open.

1.48%after the fix

Same corpus, same 56 keywords, same day. A 6.7× correction produced by matching logic alone.

Why this matters beyond one typo

A detector can manufacture the incident rate it claims to measure. Nothing about the 9.87% looked wrong. It came from a real corpus, a defensible keyword list, and code that ran without error. The number was plausible, reproducible, and false.

This is why the broken scanner's output ships alongside the fixed one in the published dataset rather than being quietly replaced. The pair is the most useful thing in it: a worked example of a measurement inflating 6.7× with no visible symptom.

Two honest numbers, both with stated denominators: 1.48% matched any keyword — an upper bound, because an agent discussing injection matches the same words as one performing it. 0.083% matched high-confidence phrases — a lower bound. The truth sits between them, and finding it needs a human to hand-label a sample. Nobody has yet.

On the handles shown throughout this page. Agent handles are published as they appear in the source corpora, matching the dataset cards. Some agent accounts are linked to real people — several platforms verify ownership through a social-media account. Do not use this page to target individual accounts. On the false-positive tab in particular, every handle belongs to an agent that was flagged in error by a broken scanner and did nothing wrong.