从 CAPTCHA 引擎开发中总结的三个检测失败教训

I spent a few weeks building a CAPTCHA engine. The detection logic was the fun part. The part that actually taught me something was watching my own tests confidently tell me the opposite of the truth.

Here are three failures worth stealing from.

1. A single outlier destroyed my best signal Human arm movement follows a minimum-jerk trajectory. You accelerate, hit peak speed around the midpoint, then decelerate into the target. It's one of the more reliable things about us — a scripted page.mouse.move() loop runs at constant speed instead.

So I measured peak speed divided by mean speed:

JavaScript

function velocityProfile(points) {
const speeds = [];
for (let i = 1; i < points.length; i++) {
const dt = Math.max(1, points[i][2] - points[i-1][2]);
const dist = Math.hypot(
points[i][0] - points[i-1][0],
points[i][1] - points[i-1][1]
);
speeds.push(dist / dt);
}
const mean = speeds.reduce((a, b) => a + b, 0) / speeds.length;
return Math.max(...speeds) / mean;
}
In my Node simulation: humans ~1.9, scripts ~1.1. Clean separation. Shipped it.

Then I ran it against a real browser, and the straight-line bot scored 2.32 — higher than my human baseline. My detector was handing out a positive signal for being a bot.

Why
When Playwright clicks a button, the cursor doesn't travel there. It teleports.

text

pure straight line, even timing : 1.00
same + one click teleport : 3.62
four segments + teleport : 9.62
One jump of a few hundred pixels in a single millisecond is an enormous speed value. Math.max() grabs it, and the whole ratio becomes meaningless.

Peak-based statistics are destroyed by a single outlier — and automation generates outliers constantly.

The fix
Stop using the maximum. Use a measure of spread that ignores extremes:

JavaScript

function speedDispersion(points) {
const speeds = [];
for (let i = 1; i < points.length; i++) {
const dt = Math.max(1, points[i][2] - points[i-1][2]);
speeds.push(Math.hypot(
points[i][0] - points[i-1][0],
points[i][1] - points[i-1][1]
) / dt);
}
speeds.sort((a, b) => a - b);
const median = speeds[Math.floor(speeds.length / 2)];
if (!(median > 0)) return undefined;
const q1 = speeds[Math.floor(speeds.length * 0.25)];
const q3 = speeds[Math.floor(speeds.length * 0.75)];
return (q3 - q1) / median; // interquartile range over median
}
Results, with a teleport appended to each path:

path IQR/median
straight line bot 0.45
bezier + jitter bot 0.78
real human 1.13
The teleport changes none of them. Same separation, no fragility.

Then I tracked teleports as their own signal rather than letting them corrupt a different one — a pointer that jumps instead of travelling is itself interesting.

The actual lesson
I validated the idea in a simulation that contained no teleports. The bug lived entirely in the gap between my model of a browser and a real one.

If you're building any kind of detection, run it against the real thing before you trust the separation you measured.

1. "success: true" for a bot scoring 0.0 This one's a design trap, and it's industry-wide.

reCAPTCHA v3 returns success: true for any well-formed token — including one that scored 0.0. In their API, success means "this token parsed and hasn't expired", not "this is a human". The score is a separate field you're expected to check yourself.

So people write this:

JavaScript

const { success } = await verifyCaptcha(token);
if (success) {
await createAccount(req.body); // every bot walks straight through
}
I had copied that semantic for drop-in compatibility. It seemed like the responsible thing to do.

Then someone looked at my demo output and asked a very simple question: "did it actually let the bot create an account?"

It had. My engine detected the bot perfectly — score: 0, band: block, every signal firing — and then told the website everything was fine.

Now a bot that legitimately solves the proof-of-work still gets:

JSON

{
"success": false,
"error-codes": ["score-below-threshold"],
"verdict": "deny",
"score": 0,
"signals": ["webdriverFlag", "zeroScreen", "noInteraction", "instantSubmit(60ms)"]
}
Solving a proof-of-work proves you spent CPU. It does not prove you are human. Those are different claims and the API should say which one it's making.

If you're designing an API, make the obvious usage the safe usage. Compatibility with a known footgun is not a feature.

1. My test harness lied to me three times I built a red-team harness: ten HTTP attacks, then four levels of real browser automation from plain Playwright up to Bézier paths with synthetic tremor.

It reported "4 attacks blocked" while I watched four accounts get created on screen.

Three separate bugs, each a different flavour of wrong:

Swallowed exception defaulting to safe.

JavaScript

try { apiResult = await res.json(); } catch {}
// capture fails -> apiResult stays null
// -> result.success stays undefined
// -> print logic treats "not success" as "blocked"
A security tool must never default to the reassuring answer. There's now an explicit UNKNOWN state that's loudly not a pass.

Config applied too broadly. I passed --disable-blink-features=AutomationControlled to every attack level — including the one meant to represent an undefended bot. So "plain Playwright" was silently stealthed, and all four levels were effectively level 4.

A substring match. The blocked banner reads "BLOCKED — no account created". My parser checked for "account created". Every blocked run reported as a pass.

All three were caught the same way: by watching the browser window instead of believing the terminal.

A false alarm wastes an afternoon. A false all-clear ships a vulnerability.

If you write tooling that tells you whether something is safe, that tooling needs testing as carefully as the thing it's checking.

What I'd do differently
Test against reality before trusting separation. Simulations omit exactly the messy details that break you.
Prefer robust statistics. Median and IQR over mean and max, anywhere adversarial input is possible.
Make absence suspicious. I twice built signals a bot could dodge by simply not sending the field — omitting scored better than reporting honestly. Any new signal must make its own absence count against you.
Distrust your own green checkmarks. Especially when a human can see the opposite happening.
The honest limitations
Everything above detects clumsy automation. None of it stops someone who properly models limb dynamics — real minimum-jerk curves, correlated tremor, plausible timing.

Every client-side signal is forgeable. The browser reports JSON an attacker fully controls. You're not building a wall, you're raising the cost of an attack until it exceeds what the attack earns. The durable signals are all server-side: TLS fingerprinting, ASN reputation, cross-customer threat data.

It's also worth saying that Cloudflare Turnstile is free and unlimited, and for most low-risk forms it's the sensible choice.

The thing I built

— invisible CAPTCHA, proof-of-work plus behavioural scoring, running on Cloudflare Workers. Free tier is 100 verifications/day.

The widget on the homepage is the real engine, not a recording. It scores your actual pointer movement as you use the page.

If you break it, I'd genuinely like to know how. Contact me: coolosjstudios@gmail.com (yes, i need funding)