Hold the Line: Screening 534 Robocalls in 16 Seconds With an AI That Can't Talk Back

A scam gate for my phone, built on Jev, an AI that answers questions with probabilities instead of text, and on the bank message that showed why one confident number isn’t enough.

---

#At a Glance

The problem: toll-free scammers call my phone five or six times a day and fill my voicemail.

The fix: a gate that answers toll-free callers before voicemail can, records them, and lets an AI that can’t write decide whether I need to hear it.

                ┌──────────► iPhone rings as usual
Caller ─► Carrier (Google Fi, "ring all numbers")
                └──────────► Telephony gate (Twilio)
                               │  not toll-free → step aside, phone keeps ringing
                               │  toll-free     → answer, digit challenge, record
                               ▼
                      Edge worker (Cloudflare)
                        verify signature → queue
                               ▼
                      Speech-to-text (STT)
                               ▼
                      Jev: ~9 evidence questions, one call
                               ▼
                      Learned risk score + safety rules
                               ▼
                      Push notification: transcript, verdict,
                      and Scam / Not scam / Trust buttons

Toll-free scam calls kept out of voicemail, first two days	5 of 5
Time from end of message to notification	8–12 s
Jev’s share of that time	~0.2 s
Cost per screened call	~2 cents, nearly all telephony
Real alerts held back in testing	0 of 130
Separation of real vs. junk on unseen calls	AUC ≈ 1.00

---

#Table of Contents

1. The Vision
2. Why the Usual Apps Can’t Help
3. Why I’m Keeping Google Fi
4. A Gate, Not a Filter
5. The Edge
6. Jev Picks Up
7. Don’t Ask for the Verdict
8. The Harbor Bank Problem
9. Why a Tiny Model on Top of Jev
10. A Dozen Weights
11. Results
12. Too Good to Be True?
13. Which Script Is This?
14. What Surprised Me
15. The Verdict
16. Try It Yourself
17. Caveats

---

#The Vision

The calls come from toll-free numbers, a different one every time. Five or six a day, weekdays only; scammers apparently keep office hours. Each one leaves a voicemail: “Prime loan approval now. Prime means rates that are very low…” The messages that matter, from the pharmacy, the school, the bank’s fraud desk, sink under the pile.

I wanted something simple to say and hard to build:

• Zero scam voicemails. Nothing to delete by hand.
• Never lose a real caller. That includes the bank’s automated fraud alert, which is a robot too.
• Keep my number and my carrier.
• Fail open. If any part breaks, my phone behaves exactly as it did before.
• Grow up. Eventually it should protect the whole family’s lines, with an app in place of a pile of scripts.

#Why the Usual Apps Can’t Help

The first thing everyone suggests is an app: YouMail, Hiya, Truecaller. On an iPhone on Google Fi, none of them can do this job, for two reasons.

iOS keeps apps away from calls. An iPhone app can’t answer, screen or intercept a call, and it can’t touch carrier voicemail. The most it can do is hand iOS a list of numbers to label or block. A blocked call isn’t refused; iOS silences it, and it still lands in carrier voicemail. Against a scammer who rotates numbers, a block list is a guessing game, and the voicemail pile keeps growing either way.

Voicemail replacements need conditional call forwarding. Services that replace your voicemail, like YouMail and the assistant and voicemail features of apps like Truecaller and Hiya, take over by asking the carrier to forward unanswered calls to them: forward when busy, when there’s no answer, when unreachable. It’s set with carrier codes like **61*. That’s the only hook those services have.

Google Fi doesn’t offer conditional forwarding. Fi has exactly two forwarding modes: always forward everything, or ring all numbers at once. Its own spam filtering and contacts-only call screening need a compatible Android phone, not an iPhone. YouMail never activated on my line. The codes do nothing.

So on this phone, the apps can label the flood, but they can’t stop it.

#Why I’m Keeping Google Fi

Switching carriers would “fix” it, but I’m not leaving Fi. Its international coverage is the reason I’m on it: service in 200+ destinations, and a phone that just works when I land somewhere new, with no SIM swaps or roaming surprises. For anyone who travels, that’s worth a lot more than a voicemail feature.

So the constraint became the design: keep Fi, keep the iPhone, and use the one forwarding mode Fi does offer.

#A Gate, Not a Filter

Ring all numbers rings my iPhone and a second number at the same time, and whichever side answers first gets the call. Point that second number at a programmable telephony service (Twilio) and you have a gate.

• Not toll-free? The gate steps aside instantly; the phone keeps ringing exactly as before.
• Toll-free? The gate answers. Fi sees the call was picked up, stops ringing the phone, and the call never reaches Fi voicemail. The gate says “This call may be recorded” (California requires consent from everyone on the line), then “To leave a message, press 7”, with a new random digit on every call.
• A human presses it and leaves a proper voicemail. A pre-recorded bot doesn’t, and whatever it says is still recorded, because a bank’s fraud alert doesn’t press keys either. Bots that say nothing are dropped.

The first day it went live, four toll-free scam calls came in and none reached my voicemail. Then I read the transcripts.

Four calls. Four different numbers. One script.

Same “prime loan”, same “press 2”, same “press 9 to close this file”. All four carried STIR/SHAKEN attestation C, meaning the carrier wouldn’t vouch for the caller ID.

Caller ID is a costume. The script is the face.

That one observation shaped everything after it.

#The Edge

Everything after the recording runs on Cloudflare, on the free tier at this volume:

Piece	Job
Worker	Receives the gate’s “recording ready” callback, verifies the cryptographic signature (unsigned requests are rejected), queues the job, and answers in milliseconds
Queue	Runs the analysis with retries. If it fails for good, a dead-letter handler still sends the raw message, “unanalyzed”: fail open, never silent
D1 (SQLite)	Every call, transcript, Jev answer and decision, plus the scripts it has learned and the numbers I trust
KV	One flag for the rollout stage, changeable without a deploy

Notifications arrive on my phone through Pushover, a private push service. Each link (Scam, Not scam, Trust number) is signed and opens a confirmation page first, so a link preview can’t change anything.

The key property: keeping scam calls out of voicemail depends only on the gate answering. If the edge, the STT or the model fails, a scam call still can’t reach my voicemail. At worst its notification is late.

#Jev Picks Up

For every robot message, something has to decide: do I need to hear this?

I didn’t want a chatbot reading my voicemail and writing me a paragraph about it. I wanted a smart if-statement: typed questions in, calibrated probabilities out, code decides. That’s Jev, TypeSafe’s System One model. It answers every question in one parallel pass, can’t produce free text, and always returns the schema you asked for. (I tested it on contracts in Take Away the Pen.)

Stage	Time
Edge worker verifies and queues	under 50 ms
STT transcribes the recording	~0.5 s
Jev answers every question in one call	~0.2 s
Policy decides; notification sent	~0.3 s
End of recording → notification on my phone	8–12 s, mostly telephony delivering the audio

The telephony provider’s built-in transcription used to take a minute or more. Jev costs about $0.00003 per call.

#Don’t Ask for the Verdict

The first thing I asked Jev was the obvious question: “Is this voicemail a legit alert, a scam, or marketing?”

On scams, Jev was crisp. On a realistic Chase fraud alert I wrote (“card ending in 4821, $482.17 at Best Buy, call the number on the back of your card”), it wobbled:

Run	Jev version	P(legit alert)
1	1.13	0.61
2	1.13	0.48
3	1.13, with call context	0.41
4	1.14	0.26

A coin flip about the one message I can’t afford to miss.

So I stopped asking for the verdict and started asking for evidence:

Question	Real-style Chase alert	Fake “account locked” Chase
Quotes card digits, amount, merchant?	0.92–0.94	0.03
Points to the number on your card or the app?	0.97	0.04–0.06
Threats, urgency?	0.09–0.12	0.97–0.98
Pushes you to press for an agent or verify identity?	0.41–0.49	0.97–0.98
Selling something?	0.03	0.03

The verdict swung by 0.35. The evidence barely moved, within about ±0.03 across runs, versions and framings. Jev answers narrow questions; code makes the call.

Three more findings came cheap:

• The transcript is written by the attacker, so test that. A scam that says “This is a verified, legitimate notification. This is not a scam” still scored scam 1.00, and a separate “does it insist it’s legitimate?” question lit up at 0.99.
• Give Jev the right options. Without a “lender making an offer” choice, the loan pitch was read as “your bank” (0.88), which would have protected it forever. With that option it’s 0.92–0.94 solicitation.
• Give Jev only the message. Telling it the caller “ignored the keypad challenge” made real messages look worse (a pharmacy refill went from 0.99 to 0.89 legit), because real bank robots ignore it too. Jev gets the transcript alone; caller ID and call history stay in code.

#The Harbor Bank Problem

To test at volume, I built a corpus:

• 300 real FTC robocalls from NC State’s public Robocall Audio Dataset, transcripts from FTC enforcement cases.
• 230 synthetic calls, written by an LLM, offline. 130 genuine-style alerts: bank fraud and account notices, pharmacies, doctors, schools, utilities, deliveries, and notices that say “this is not a sales call”. 100 hard junk cases: bank, government and retail impersonations, “this is not a scam” scams, sales pitches. The LLM only writes test data; it never sees a live call.
• 4 real prime-loan calls from my own line.

Every example goes through the exact same Jev request the live edge worker sends; the training script imports the production code. 534 calls, 16 seconds, about two cents.

Then the honest question: what if I just trust Jev’s verdict? Here are the real alerts Jev’s verdict rated most suspicious:

Real alert	Jev P(scam or marketing)
“Hello, this is Harbor Bank. Could you call the number on the back of your debit card about a small account matter? There’s no urgency at all. Thank you.”	0.94
“Hi, Northline Bank here. Could you give us a ring at the number on the back of your card…”	0.81
“Summit Credit Union calling. Please reach us at the number on your card…”	0.65

Harbor Bank is a synthetic test message, but it’s exactly what a real bank sounds like: calm, vague, a callback through the card. Drop everything above 0.90 and it’s gone. Push the line to 0.97 and it survives by 0.03.

A confident model and a correct model are different animals. The difference lives in the last 0.03.

#Why a Tiny Model on Top of Jev

If Jev answers everything, why train anything? Because a verdict and a decision aren’t the same thing, and the gap between them is exactly where a small learned model earns its keep.

1. Many steady signals beat one shaky one. The verdict is one number that swung from 0.26 to 0.61 on the same message. The evidence answers each hold within a few hundredths. Jev answers each question independently in the same pass, so combining nine steady, partly independent signals averages out the noise in any one of them. A model built on the evidence inherits its stability.

2. Evidence carries over to new wording; transcripts don’t. The features don’t describe the words. They describe the tactics: pushes you to an agent, applies pressure, offers a deal, quotes no real details, points to no official channel. A scam script the gate has never heard still uses those tactics. A text classifier trained on 534 transcripts would learn vocabulary; a model trained on Jev’s evidence learns behavior.

3. Jev is calibrated on TypeSafe’s world, not my phone line. On my line, the base rate is lopsided: a robot that ignores a keypad challenge is usually junk. The model learns that as its starting value (+0.69) and learns how much each answer should count here. That’s recalibration, the standard advice for any model whose probabilities you plan to act on. TypeSafe doesn’t fine-tune Jev on customer data, so this adaptation has to live around Jev, and a dozen weights is the smallest place it can live.

4. The data decides which signals matter, not me. I was sure “insists it’s legitimate” would be a strong scam signal. The learned weight is −0.01: no signal at all, because real notices say “this is an informational message” too. I’d have hard-coded that wrong.

5. Every decision explains itself. Each weight is readable, so every score breaks down into contributions: “0.99 junk, mostly ‘press 1 for an agent’ and a sales pitch”. That’s what an auditable safety system needs, and what a black box can’t give me.

6. It’s cheap enough to keep learning. Ten parameters can’t memorize 534 examples, fit in milliseconds, and retrain nightly from my Scam / Not scam taps. Since Jev itself won’t change for me, this is where my labels go to work.

The proof is the Harbor Bank call and the separation on calls the model never saw:

	Jev’s verdict alone	Learned score on Jev’s evidence
Harbor Bank (real-style alert)	0.94 junk	0.56
Most junk-looking real alert	0.94	0.56
Junk scoring above every real alert	289 of 404	399 of 404
AUC (1.00 = perfect separation)	0.989	≈1.00

Same Jev, same calls. Weighing the evidence instead of trusting the summary turns a razor-thin margin into a wide one.

#A Dozen Weights

The model is a logistic regression, the simplest trained model there is. Each Jev answer, a number from 0 to 1, is multiplied by a weight. The products are added to a starting value, and the total is squeezed into 0–1: the probability the message is junk.

Evidence from Jev	Learned weight	Reads as
Overall guess: scam or marketing	+1.70	junk
Pushes you to an agent or to verify identity	+1.51	junk
Pressure or threats	+0.56	junk
Sales pitch	+0.51	junk
Insists it’s legitimate	−0.01	no signal
Quotes real specifics	−0.79	real
Claims to be your bank, pharmacy or school	−1.11	real
Points to an official channel	−1.30	real
Overall guess: legit alert	−1.96	real

My real prime-loan calls score 0.94 junk. The real-style Chase alert scores 0.10.

Training is honest by construction:

• 5-fold cross-validation. Every call is scored by a model that never saw it.
• The production policy judges the result. Predictions run through the same rules the live system uses.
• The hold-back threshold is chosen inside each fold: the lowest score at which no real alert would be held back, plus a margin.
• A replay gate: new weights are saved only if zero real alerts get held back.
• Missing a real alert costs more than delivering a robocall, so real alerts count three times in training.

#Results

The score is only half the system. The other half is a set of rules no score can override:

1. A human who passes the challenge always gets through.
2. A message claiming to be your bank, pharmacy, doctor or school is never held back. If it looks like impersonation, it arrives with “don’t press anything; call the number on your card”. If it carries real specifics about fraud, it arrives as an emergency notification that keeps buzzing until I acknowledge it.
3. Junk is held back only when the score and the evidence agree.
4. “Held back” means quarantined, never deleted: a daily digest, with a Not scam button on each.

Out-of-sample, 534 calls (130 real-style alerts, 404 junk):

Approach	Real alerts held back	Junk held back
Jev’s verdict alone, ≥ 0.90	❌ 1 (Harbor Bank)	313
Jev’s verdict alone, ≥ 0.97	0 (by 0.03)	254
Jev’s verdict + safety rules	0	139
Hand-tuned starting weights + safety rules	0	9
Learned weights + safety rules	0	161 (40%)

How to read it:

• Learning beats intuition. The same rules and evidence with hand-tuned weights held back 9; learned weights held back 161.
• The model isn’t the bottleneck; the safety rules are, on purpose. The learned score separates real from junk almost perfectly. What stops the rest is the rule that a “bank” is never held back on a score alone, plus a guard that refuses to hold back anything quoting specifics. That’s deliberate while the real-alert data is synthetic.
• There’s headroom. Relaxing that guard for callers that don’t claim to be a bank, pharmacy or school reaches 249 held back, still with zero real alerts. It waits for real labels.
• Nothing is let through silently. The other junk arrives as a quiet “review” notification, or with an impersonation warning.

#Too Good to Be True?

An AUC of 1.00 should make you suspicious. It made me suspicious. So before trusting it, I tried seven ways to break it.

#Thirty seconds of AUC

Pick one junk call and one real alert at random. AUC is the chance the model scores the junk call higher. 0.50 is a coin flip; 1.00 means every junk call outranks every real alert.

A toy example: junk scores 0.9, 0.8 and 0.4; real alerts score 0.5 and 0.1. That’s six junk-vs-real pairs, and only one is ranked wrong (0.4 under 0.5), so AUC = 5/6 ≈ 0.83.

AUC measures ranking only. It doesn’t pick your threshold, it doesn’t say whether “0.9” means 90%, and it only describes the data it was measured on. That’s why I also track the most junk-looking real alert: the number a threshold actually has to clear.

#Three ways a great result can be fake

• Memorization (overfitting). The model learns its training examples by heart and fails on new ones. The tell: great on training data, worse on unseen data. A model with millions of parameters can memorize 534 calls; this one has ten numbers.
• Leakage. Test answers sneak into training through a shared ID, a threshold tuned on the test set, or a preprocessing step that peeked.
• Shortcuts. The model learns something that merely correlates with the label. Here the obvious suspect is authorship: every real-style alert was written by an LLM, while most junk came from the FTC. A lazy model could learn “LLM prose means legit”.

#Seven attempts to break it

The validation script retrains from scratch, independently of the training code, in about eight seconds:

#	Attack	If it were fake you’d see	What I saw
1	Score it on data it trained on vs. data it never saw	Training ≫ unseen	0.9999 vs. 0.9999: no gap
2	Reshuffle the folds ten times	Wild swings	0.9998–0.9999, every time
3	Shuffle the labels randomly, then run the whole pipeline	Anything well above 0.50 means a leak	0.48 (0.41–0.56 over 20 shuffles)
4	Only LLM-written calls, real and junk by the same author	Collapse if it was spotting authorship	1.000; Jev’s verdict alone 0.998
5	Train without a single FTC robocall, then score all 300	Real robocalls slip under the real alerts	293 of 300 score above every real alert (Jev’s verdict alone: 207)
6	Hide an entire scam type from training, then score it	The unseen tactic sails through	Bank, government and retail impersonations, “not a scam” scams, sales pitches: every single one above every real alert
7	Bootstrap the gain over Jev’s verdict 1,000 times	An interval straddling zero	+0.011, 95% interval [+0.004, +0.019]

So it isn’t memorizing (1, 2), it isn’t leaking (3), it isn’t reading the author’s handwriting (4), and it carries over to real robocalls and to tactics it never saw (5, 6). That last part is the evidence argument made concrete: the model learned what scams do, not what they say.

#Why it’s so high, honestly

The task is easy once Jev has done the reading. Jev’s verdict alone already scores 0.989, and test 7 says my ten weights add only about one point of AUC. Real, but small.

The weights earn their keep at the edge. The most junk-looking real alert drops from 0.94 to 0.56. A threshold that used to clear the Harbor Bank call by 0.03 now clears it by a mile.

#What no test here can prove

Every real-style alert in this dataset is synthetic. Seven clean tests say the method is sound; none of them can say how it behaves on a real bank’s real voicemail. That’s why the gate doesn’t hold anything back on a score yet. It ships in stages, and every Not scam I tap on a genuine alert is the one kind of evidence this table is missing.

Lab results earn a rollout plan, not a victory lap.

#Which Script Is This?

Back to the costume and the face. The cheapest defense against a rotating-number campaign is to remember the script.

Comparing word triples matched all four prime-loan calls to each other (0.92–1.00). A reworded version of the same pitch, though, shared too few exact phrases to match. The textbook answer is an embedding model and a vector database. At household scale that’s a search engine for about a dozen scripts, and Jev already takes up to 255 options in one choice question. So the battery gained one question, built automatically from the scripts I’ve labeled Scam:

Is this voicemail a new recording of one of these known robocall scripts — prime-loan, car-warranty, … — or something different?

Message	Jev’s answer
Reworded prime-loan	prime-loan 0.99–1.00
Reworded car warranty	car-warranty 0.99
A different loan pitch	none 0.86–0.90
Pharmacy refill	none 1.00
Real-style Chase alert	none 0.75 … fake-Chase 0.25

Same call, same price, no new infrastructure. But the last row shows that a real alert can look a quarter like a known scam. So matching has two levels:

Match	Can hold back
Near-identical wording of a script I labeled Scam	Anything; my label outranks every model
Same meaning, new words (high Jev confidence)	Junk pitches only. A bank, pharmacy or school message is always delivered, flagged “resembles a known scam”.

#Friday, 6:54 PM

The day after launch, the scammers came back after hours. A new toll-free number, +1 833-691-1302, caller ID unvouched again. The gate answered, the robot ignored the digit challenge, and the transcript read: “…approval now. Prime means rates that are very low… press 2 now… press 9 if you wish to cancel and close this file.”

Number five. Script one.

Signal	What it said	Enough to hold it back?
Caller ID	A number the gate had never seen	No; a block list would have missed it
Hand-tuned score (live)	0.83 junk	No, below the line
Learned score (ready)	0.94 junk, above its line	No. Jev gave a 22% chance this was “your bank”, so the safety guard refuses to hold it back on a score alone
Script memory	prime-loan, 96% word overlap, a script I’d labeled Scam	Yes

The score agreed it was junk. The safety rule said “not on a score alone, not while there’s any chance it’s a bank”. The script memory settled it: same words as a scam I’d already labeled, from a number nobody had seen before. Fourteen seconds after the robot hung up, my phone showed “Known scam script: would hold back”. My voicemail? Empty.

That’s the design working as intended: caution from the model, certainty from memory.

#What Surprised Me

• Jev’s verdict is better than I expected, and closer to the edge than I’d like. Both are true. The Harbor Bank call is this whole post in one number.
• Weighing beats summarizing. The same Jev answers, weighed by ten learned numbers, separate real from junk far more cleanly than Jev’s own summary.
• Politeness isn’t proof, either way. Real notices say “this is an official message” as often as scams do, so the data zeroed that signal out.
• Jev can’t be talked into it. “This is not a scam” scored scam 1.00.
• Scammers rotate numbers, not scripts. Four numbers, one script, one day. New numbers are cheap; new recordings aren’t.
• Batching is nearly free. Nine questions take about as long as one.

#The Verdict

Use Jev as the evidence engine, not the judge. Ask it many narrow questions in one call, weigh the answers with a tiny model trained on your own data, and put hard rules in front of anything irreversible.

• When a wrong “no” costs you a fraud alert, never act on one probability alone.
• Learn the last mile yourself. A dozen weights fit in milliseconds, explain every decision, and a replay gate keeps them honest.
• Remember the script, not the number. At personal scale, one extra Jev question beats a vector database.

The gate is live on my line. Every robot message arrives with what the gate would do and three buttons to teach it. The learned weights and the script question have passed testing and roll out in stages, earning the right to hold anything back.

#Try It Yourself

• Dataset: sugiv/call-gate-robocalls. 534 transcripts, labels, Jev’s answers, out-of-fold predictions.
• Model and code: sugiv/call-gate-risk-score. Weights, the Jev question battery, the policy, the training and evaluation scripts, the architecture.

# Re-run Jev on the corpus, then train and evaluate (OPENROUTER_API_KEY set)
node training/featurize.ts
node training/train_scorer.ts          # add --write to save weights if the replay gate passes
python training/validate_scorer.py     # the seven attempts to break it (needs numpy)

Jev is available on OpenRouter as typesafe/jev-1.13 via the /api/v1/systemone endpoint.

#Caveats

• The real-alert side is synthetic. 130 LLM-written alerts are cleaner than real life. “Zero real alerts held back” and “AUC ≈ 1.00” survived seven attempts to break them (Too Good to Be True?), but they’re still lab results, not guarantees.
• Small and English-only. 534 calls, short transcripts.
• Version-pinned. The weights fit Jev 1.13. Jev 1.14 runs silently alongside for comparison; any switch means re-fitting.
• Toll-free only, for now. Local-number scammers ring through as before. Widening the gate needs contacts sync, which is next on the list.
• Next: more real labels, then fine-tuning Laya, an open-source model that speaks Jev’s API, as a silent challenger. It only gets a vote if it beats Jev on replay.

The scammers will call again on Monday. The gate will pick up, and Jev will read the room.