Inside the Hexr SDK: Why Grep Beats Embeddings for Agent Code, How We Built a Sub-Second Rust Detector, and the 96% Benchmark

hexr.dev · hexr.cloud · Hexr on LinkedIn

Part 1 — The Philosophy: Why Grep Beats Embeddings for Code
Part 2 — The Implementation: How We Built a Sub-Second Rust Detector
Part 3 — The Benchmark: 48/50, 173 Milliseconds, Two Published Misses
Part 4 — When Would We Ever Use Embeddings?
Part 5 — Common Objections, Answered
Part 6 — Lessons We’d Write on a Whiteboard
Part 7 — What Shipped Since May (Status Update)
Part 8 — What This Means for Enterprise Customers

“Grep works because the agent knew exactly what it was looking for. Embeddings work when it doesn’t.” — paraphrasing jxnl and Eugene Yan’s lineage of posts on the same theme.

A note on licensing: The Hexr SDK is enterprise-licensed. Source is available to licensed customers for audit and extension purposes. This blog post describes architecture decisions, principles, and shipped results — not a reproduction guide. If you’re a licensed customer, you have full access to everything described here.

Part 1 — The Philosophy: Why Grep Beats Embeddings for Code (And When It Doesn’t)

TL;DR

For code, lexical and structural matching beats neural retrieval almost every time. The reason isn’t religious — it’s that code has properties prose doesn’t: exact tokens, an unambiguous grammar, and a downstream consumer (a compiler, a security auditor, a runtime gate) that demands precision and recall, not “relevance.” So Hexr’s framework-and-pattern detection ships as YAML signatures evaluated against a Ruff AST with regex post-filters — classic iterative grep, just on a parse tree. No embeddings, no SLM, no vector DB, no GPU on the customer’s build path.

The four techniques in 60 seconds

Technique	What it indexes	What it returns	Determinism	Hardware
Iterative grep	The source text (or a parse tree)	Exact matches, often refined across passes	100%	CPU, milliseconds
ColGrep	Code chunks → ColBERT-style late-interaction embeddings	Top-K chunks ranked by token-level cosine similarity	Stochastic per index build	GPU recommended
Embedding grep (dense)	Whole files / functions → single dense vector	Top-K by vector distance	Stochastic	GPU recommended
Fuzzy match	The source text	Approximate string matches (edit distance, trigram)	Deterministic but lossy	CPU

Iterative grep and fuzzy match return what is there. The two embedding families return what is similar to what you asked — and “similar” is a learned function whose mistakes you cannot enumerate.

“Tokens carry semantics” — what that actually means

In English prose, the word bank could mean a riverbank, a financial institution, or a turn in an aircraft. The token alone tells you almost nothing; the embedding’s job is to fold neighboring context into a vector so the meaning resolves. That’s why neural retrieval shines on prose.

Code is the opposite. Consider:

from crewai import Crew, Agent, Task

The token crewai is not ambiguous. It refers to exactly one package on PyPI, version-pinnable, with a known public API. The token Crew imported from crewai is not ambiguous either — it’s a specific class with a specific constructor signature. There is no “Crew with similar meaning”; there is only crewai.Crew, the actual symbol resolved by Python’s import machinery.

The same applies to:

@app.tool — Anthropic Claude Agent SDK tool decorator. One meaning.
langchain_core.runnables.RunnablePassthrough — one symbol, one type.
tool_use in a JSON payload from messages.create — Anthropic protocol field, exact spelling.

When the token is the meaning, an embedding is a lossy re-encoding of information you already had perfectly. You get strictly less information, plus probabilistic noise, plus GPU dependency, plus an opaque failure mode. That’s the trade you’re making.

The corollary: when the token doesn’t carry the meaning — for example a free-text user query like “show me the part where we cancel the subscription if the trial ended” — embeddings genuinely help. Code is the wrong workload for them. Issue triage, doc search, and customer support tickets are the right workload.

“Recall and precision matter more than fuzzy similarity”

A code-analysis tool is graded on a confusion matrix, not on a relevance score:

	Detection should fire	Detection should NOT fire
Tool fires	True positive ✅	False positive ❌ — auditor loses trust, customer disables the rule
Tool doesn’t fire	False negative ❌ — the very thing we sell missed a real agent	True negative ✅

Embedding-based detection optimizes a third axis — cosine similarity — that is only loosely correlated with correctness. A 0.83 similarity score to “this looks like a CrewAI agent” is not an answer to the question ”is this a CrewAI agent?” It’s a soft vote, and soft votes are not what ship in a product where the output is forwarded to OPA, mTLS gating, and billing.

What we actually need:

Recall ≈ 1.0: every CrewAI agent in the codebase must be detected, or the customer’s agent runs without an SVID and the gateway will deny its tool calls. Recall failure is a Sev-1 outage.
Precision ≈ 1.0: no plain class CustomCrew business object is flagged as an agent, or the customer’s CI fails on objects that have nothing to do with AI.
Auditability: the rule that fired must be inspectable line-by-line. An embedding score isn’t.
Determinism: same source code → same detections, forever, across OS and Python version. Required for reproducible builds.

Iterative grep on the AST hits all four. Embeddings hit none of them without heavy guard rails (and even then, audit is hand-wavy).

Part 2 — The Implementation: How We Built a Sub-Second Rust Detector

Why we threw out the old Python analyzer

Hexr’s job at build time is to look at a customer’s Python source tree and answer three questions before we package anything:

Which agent framework(s) is this code using? (CrewAI? LangGraph? OpenAI Agents SDK? Or just raw openai calls?)
What agents, tools, and LLM clients are declared?
Which agentic patterns are at play? (ReAct loop? Routing? Evaluator–optimizer? Reflection?)

The original Hexr SDK answered (1) and (2) with a stack of Python importlib-based introspection plus AST visitors. It worked, but:

It executed the customer’s import graph to walk classes. That meant any side-effect import (a boto3 client, a vector DB connection) ran at build time, which is wrong.
It was slow. Several hundred milliseconds per file, not great on monorepos with thousands of .py files.
It hard-coded each framework as Python code. Adding LangGraph meant writing a new visitor class. The blast radius for one PR was the entire detector.

We wanted: no execution, deterministic, fast enough that nobody notices it runs, and frameworks added as data, not code.

The architecture

The analyzer is a three-crate Rust workspace compiled via PyO3 into a native extension that ships inside the hexr-sdk Python wheel:

hexr build  ──►  hexr_analyzer (native extension)
                      │
                      ▼
   ┌──────────────────────────────────────────┐
   │   hexr-parse     Ruff parser per file    │
   │   hexr-symbols   Scope + ImportTable     │
   │   hexr-detect    YAML packs + walker     │
   └──────────────────────────────────────────┘
                      │
                      ▼
              BuildManifest (JSON)
              { framework, agents, tools,
                llm_clients, patterns, ... }

Three small Rust crates. One pass over the AST. One function call from Python. One JSON manifest out.

Parsing — Ruff, not the CPython parser

We use Astral’s Ruff Python parser (same one behind the most popular Python linter) pinned to a specific version. It’s the fastest Python parser in active development, fully async/PEP 695 aware, and deterministic. It recursively walks the project tree, skipping dotfiles and __pycache__, and returns parsed modules.

We deliberately do not type-infer. Detection is based on the shape of the code (which classes are constructed, which decorators are applied, which dotted call paths are invoked), and a Ruff-based scope + binding table is enough.

Symbol resolution — knowing what `cw.Agent` resolves to

The classic gotcha for static framework detection is import aliasing:

import crewai as cw
researcher = cw.Agent(role="researcher", ...)

The symbol-resolution layer walks every Import and ImportFrom in a module (recursively, including inside if/try/function/class bodies) and builds an import table mapping local binding → fully-qualified module path. resolve("cw.Agent") yields "crewai.Agent". The entire reason we don’t need types — we need bindings, not types.

We also ship a factory-resolution pass that tracks patterns like client = OpenAI() → client : openai.OpenAI, and def make_client(): return OpenAI() → make_client : () -> openai.OpenAI across files. This covers ~90% of real-world factory patterns deterministically.

Detection — the meat: YAML signature packs

This is where the design pays off. Every framework Hexr knows about is a YAML file, not Rust code, not a plugin. A signature is a small DAG of AST predicates plus optional regex-on-source post-filters.

Here’s a simplified example — how the engine detects CrewAI:

schema_version: "1.0"
kind: framework
id: crewai
display_name: "CrewAI"

detect:
  imports:
    - module: crewai
      weight: 0.4
  symbol_patterns:
    - kind: class_instantiation
      class: crewai.Agent
      weight: 0.3
    - kind: class_instantiation
      class: crewai.Crew
      weight: 0.3

idioms:
  agent:
    - kind: class_instantiation
      class: crewai.Agent
      capture:
        name: "{kwarg('role')}"
  tool:
    - kind: decorator
      name: crewai.tool
      target: function

Two things are happening:

detect scores how likely this pack matches the project. Imports contribute on a per-module basis; symbol patterns add weight whenever the walker sees one of the listed AST shapes.
idioms is the structured-extraction half: when a matched shape is also an idiom, we pull captured kwargs into typed manifest entries (agents, tools, LLM clients).

The walker is a single pass per module. For every call and function/class definition, it consults the pack registry and fires matching rules. One pass over the AST handles every pack.

The 15 frameworks (shipped)

The shipping pack set covers:

ID	Display name
`adk`	Google ADK
`agno`	Agno (formerly Phidata)
`autogen`	Microsoft AutoGen
`bespoke`	Raw `openai` / `anthropic` / `google.genai` / `cohere` / `mistralai`
`claude-agent-sdk`	Anthropic Claude Agent SDK
`crewai`	CrewAI
`dspy`	DSPy
`langchain`	LangChain
`langgraph`	LangGraph
`llamaindex`	LlamaIndex
`mcp`	Model Context Protocol (Python)
`openai-agents`	OpenAI Agents SDK
`pydantic-ai`	Pydantic AI
`smolagents`	Hugging Face smolagents
`strands`	Strands Agents SDK

bespoke is intentionally low-weighted — when both raw OpenAI calls and a real framework appear, the framework wins; bespoke is the fallback that guarantees we still capture evidence on “no framework, just calls” repos.

From frameworks to patterns

Frameworks tell you which library is in use. Patterns tell you what the agent is actually doing — and that’s where Hexr’s policy and runtime get interesting (different patterns warrant different identity, isolation, and rate-limit defaults).

We followed Anthropic’s Building Effective Agents taxonomy and added ReAct (Yao et al. 2022):

Pattern	When it fires
`prompt-chain`	≥2 sequential LLM calls in a function, no loop, no parallelism
`routing`	LLM-classified value is used in a branching condition
`parallelization`	LLM calls dispatched via `asyncio.gather` / `ThreadPoolExecutor` etc.
`orchestrator-workers`	A function constructs ≥2 agents and an LLM call
`evaluator-optimizer`	Loop with ≥2 LLM calls (one grades, one improves)
`reflection`	Loop with ≥2 LLM calls and zero tool calls (self-critique)
`react`	Loop with ≥1 LLM call and ≥1 tool call (Reason → Act → Observe)

Each pattern is a YAML file. Pattern detection is deliberately a shape heuristic, not a semantic proof — confidences cap below 1.0, and the manifest consumer (Hexr’s policy engine) decides thresholds. Patterns are useful priors, not ground truth.

Customer-private signatures

A customer with a bespoke internal framework drops a YAML into ~/.hexr/signatures/:

id: framework.acme.orchestrator
display_name: Acme Orchestrator (internal)
match:
  any_of:
    - rule: import
      ast: ImportFrom
      module_matches: '^acme.orchestrator($|.)'
emits:
  attestation_class: framework.acme.orchestrator

No Hexr release required. No model retraining. No round trip with the vendor. The signature is reviewed by the customer’s own security team and lives inside their boundary.

How new packs get to customers — signed, not scraped

Hexr-shipped packs ride the wheel by default (fully offline / air-gap clean) and can be updated out of band from a Hexr-operated signed feed. Every pack is cosign keyless-signed (no long-lived Hexr signing key exists) and logged to the public Sigstore transparency log; the SDK refuses to install a pack whose signature or signer identity doesn’t verify.

This closes the obvious follow-up: “OK, the detector is deterministic — but how do I trust the rules you’re distributing?” The answer is the same answer the Linux distros give: signed, transparent, independently verifiable. Customer-private packs live outside that feed entirely — they’re the customer’s own supply chain.

When a new agent framework lands, adding coverage is a signed YAML push — not an SDK version bump, not a model retrain, not a customer-visible upgrade.

Part 3 — The Benchmark: 48/50, 173 Milliseconds, Two Published Misses

The thesis

We just shipped a framework detector that classifies any Python agent codebase into one of 15 supported frameworks plus a bespoke fallback. It hits 48 / 50 = 96.0% on a 50-fixture real-world benchmark, processes the entire corpus in ~173 ms on a single MacBook Air core, and has zero machine learning in the hot path. No embeddings. No LLM judge. No Python in the detection loop.

Why this architecture vs embeddings, on every axis

Axis	Embeddings + cosine	Rust AST + YAML packs
Latency for 50 files	2–10 s (warm)	173 ms
Memory at rest	400 MB+ (model + tokenizer)	﹤ 10 MB (compiled binary)
Determinism	Floating-point, drifts with onnxruntime version	Bit-identical across runs
Auditability	“Why did you classify X as Y?” → vibes	Every detection traces to a YAML rule with file:line
Air-gap behavior	Bundle 16–100 MB of weights	Zero downloads, ever
Failure mode	Silent confident wrongness	Loud `bespoke` + zero confidence

The last row is the one that mattered most. An embedding model that’s 70% confident in the wrong framework will silently poison the downstream OPA policy bundle, the runtime sidecar layout, and the build manifest. A grep-shaped detector that finds no signal returns bespoke with confidence 0.00 and asks for help. We will take loud ignorance over quiet wrongness every time.

The three-layer detection ladder

The shipped detector is a three-layer deterministic-then-advisory stack:

L1 — Ruff AST + YAML packs (100% deterministic, audit-grade, always on). Parses the file, builds an import table that resolves aliases, then matches each AST node against versioned YAML rules. Every detection in the build manifest traces to a rule id and a file:line. This is the only layer whose output flows into OPA policy decisions.
L2 — Rust receiver-type + factory tracking (in-process, 100% deterministic, always on). Tracks client = OpenAI() → client : openai.OpenAI, and detects factory patterns across files. Adds the ~90% of real-world factory patterns that L1-alone misses. Zero new dependencies — compiled into the same native extension.
L3 — Late-interaction suggest mode (opt-in via hexr-sdk[enterprise], advisory only, never auto-applies). hexr analyze --suggest runs a small on-CPU late-interaction encoder over call sites L1+L2 didn’t classify, and — if score and neighbour agreement both pass threshold — emits a YAML diff for the human to review. It cannot write a verdict into the build manifest. Ever.

The L1+L2 path is what produces the 96% number. L3 is the escape hatch for stuff the corpus does not yet cover; it’s why we don’t have to choose between “deterministic” and “useful on novel code.”

Four things we killed, in order

1. Python AST analyzer (killed Q1 2026, replaced with Rust)

The original SDK shipped a Python-based AST walker built on ast.NodeVisitor and libcst. It worked. It also took ~4 seconds on the same 50-fixture corpus, single-threaded. The GIL meant we couldn’t meaningfully parallelize it. And every Python minor release moved the ast module schema under us.

We rewrote it in Rust. Same files, same logic, same YAML packs — 23× faster end-to-end and a single statically-linked native extension. The Python layer is now a thin wrapper that hands bytes to Rust and gets back a typed verdict.

hexr build runs on every developer save. Four seconds vs 173 milliseconds is the difference between something you forget is running and something you reach for --skip flags to disable.

2. Hexr-SDK-lite (killed March 2026)

We had a “lite” SKU. We killed it for one reason:

Two SKUs of a developer tool means two debugging stories, two support backlogs, and roughly two times the cognitive load on every contributor.

The right answer wasn’t less SDK. The right answer was make the one SDK so light that “lite” stops being a distinguishing word. The default install is now a small wheel. The [enterprise] profile adds a late-interaction model for advisory suggestions. That’s the whole menu. There is no third tier.

3. Pyright sidecar (killed May 2026)

We briefly shipped an extras marker that pulled in pyright for the long tail of generic/TypeVar/classmethod factory patterns. It survived nine days. We reverted it for three reasons:

Node.js as a transitive runtime dep. pip install pyright downloads a Node binary on first invocation. That’s air-gap-hostile by construction. Half our enterprise pipeline is air-gapped; we cannot ship a tool that quietly tries to fetch Node from registry.npmjs.org.
Cross-process JSON marshalling on every build. The exact anti-pattern our design constraints retired, re-introduced via the back door.
Two more failure modes in the operational surface. Pyright timeouts. Node-runtime fragility on Alpine, ARM64-musl, FIPS-mode RHEL.

The replacement: our in-house Rust factory-resolution pass (catches ~90% of real-world factories deterministically) plus L3 late-interaction suggest for the long tail — and only as advisory YAML diffs the human reviews. L3 never silently mutates the symbol table. Deterministic answers go into the build manifest; probabilistic answers go into a diff a human reads.

4. SaaS-first GTM + the marketplace (killed Q4 2025)

This one is GTM, not engineering, but it shaped every engineering decision after it. We spent most of 2025 building toward a Hexr-hosted SaaS plus marketplace distribution. We killed both late last year for a reason that compounds: a SaaS-first agent-runtime story forces customers to send their agent code to us before they get any value. That’s a non-starter for the enterprise design partners we actually want.

They want a local SDK that produces an audit-grade artifact, full stop. Everything in the current release train — the deterministic detector, the Rust rewrite, the explicit choice to keep the model + index local — flows downstream from killing SaaS-first.

The benchmark numbers

Headline numbers, single MacBook Air core, release build:

48 / 50 = 96.0% per-file accuracy
~173 ms to parse + classify the whole corpus
14 / 15 supported labels at 1.00 / 1.00 / 1.00 P / R / F1
2 misses, both published below

The two misses, published

Fixture	Expected	Detected	Why
`markdown_validator.py`	`crewai`	`langchain`	This file is a crewAI-examples entry point but imports `langchain_openai.ChatOpenAI` as its LLM client and never imports `crewai` directly — it delegates that to a sibling file. The detector reads files in isolation and sees a langchain-only import surface. Honest call: in isolation, this file is langchain-flavored. We treat this as a pack-rule gap, not a detector bug.
`requests_agent.py`	`bespoke`	`<none>`	Imports only `requests`, `json`, `dataclasses`. Zero framework signals — not even the openai SDK. The detector correctly returns “no framework detected” with confidence `0.00`. Calling something “bespoke” requires some agentic signal; pure `requests`-with-a-loop is indistinguishable from any HTTP script. We publish it as a miss to be honest about the floor.

Neither miss is silently swept under the rug. Both are reproducible. Both feed the pack-tuning queue.

Per-label P / R / F1

Every framework label except langchain scored 1.00 / 1.00 / 1.00. langchain scored 0.80 / 1.00 / 0.89 because of the crewai-mislabel-as-langchain false positive. crewai scored 1.00 / 0.83 / 0.91 (same fixture, its one false negative). bespoke scored 1.00 / 0.67 / 0.80.

Eval on synthetic fixtures (per-framework correctness)

We also run a synthetic fixture set covering all 15 frameworks and all 7 patterns — each derived from the framework’s own quickstart docs and Anthropic’s pattern descriptions:

- Framework accuracy: 15/15 (100%)
- Pattern accuracy: 7/7 (100%)
- Avg parse + detect latency per fixture: 328 µs

Selected results:

Fixture	Expected	Detected	Confidence	Time (µs)
crewai_app	crewai	crewai	1.00	812
langgraph_app	langgraph	langgraph	1.00	905
openai_agents_app	openai-agents	openai-agents	1.00	736
claude_agent_app	claude-agent-sdk	claude-agent-sdk	1.00	332
strands_app	strands	strands	1.00	313
mcp_app	mcp	mcp	1.00	147
bespoke_app	bespoke	bespoke	0.35	422
routing	routing	routing	0.75	99
react	react	react	0.80	226
orchestrator_workers	orchestrator-workers	orchestrator-workers	0.80	415

The Semgrep Comparison: Same Corpus, Same Labels, Different Architecture

The obvious follow-up to publishing a 96% benchmark is: “Could you get that with Semgrep?” Semgrep is the closest open-source tool to what we’re doing — static pattern matching on Python source, YAML rule definitions, no ML in the loop. Fair question. We answered it.

What we did

We authored a best-effort Semgrep custom ruleset — 15 rules, one per framework — targeting the same import and class-instantiation patterns the Hexr YAML packs detect. We ran this ruleset against the identical 50-fixture real-world corpus used by bench_50repo.rs. Ground truth labels are the same. The rules use pattern-regex against import lines (since Semgrep’s AST-level from X import ... syntax doesn’t parse in Python mode) and a simple priority heuristic to break ties when multiple rules fire.

This is a fair-effort comparison. Semgrep is designed for security rule matching, not agent-framework classification. These rules represent what an engineer could reasonably author in a few hours. We’re not dunking on Semgrep — we’re showing where the architectural ceiling is.

Headline numbers

Tool	Accuracy	Time (50 fixtures)	ML in hot path	Air-gap clean
Hexr SDK (L1+L2)	48/50 = 96.0%	~173 ms	No	Yes
Semgrep (custom rules)	47/50 = 94.0%	~3,000 ms (batch)	No	Yes

Semgrep batch mode: ~3s wall-clock including Python startup. Per-file invocation: ~2s/file due to process spawn overhead. Hexr’s 173ms is a native Rust binary scanning all 50 files in-process.

The three Semgrep misses

Fixture	Expected	Semgrep detected	Root cause
`requests_agent.py`	`bespoke`	`<none>`	File uses raw `requests` to call an LLM API endpoint — no `openai`/`anthropic` import at all. Semgrep can only match text patterns; it cannot infer “this HTTP call targets an LLM.”
`markdown_validator.py`	`crewai`	`langchain`	CrewAI import lives in a sibling module (`crew.py`). This file only imports `langchain_openai`. Cross-file resolution failure.
`rag.py`	`smolagents`	`langchain`	Imports both `langchain` (for document loading) and `smolagents` (for the agent orchestrator). Priority heuristic picks wrong winner. Weighted-scoring failure.

Why Semgrep hits its ceiling at ~94%

The 2-point gap (94% → 96%) understates the architectural difference. On this corpus — which is deliberately “easy” (self-contained files with obvious imports) — Semgrep comes close. On real enterprise repos the gap compounds:

No symbol resolution. Semgrep matches syntactic patterns but cannot resolve import crewai as cw; cw.Agent(...) → crewai.Agent. Our hexr-symbols crate resolves all aliases before detection fires.
No cross-file analysis. Semgrep processes one file at a time with no shared symbol table. A file that imports a helper from a sibling module (which in turn imports the framework) is invisible. Miss #2 above is this case exactly.
No weighted scoring. Hexr accumulates confidence across multiple signals — imports contribute 0.4, class instantiation adds 0.3, decorators add 0.3. When both langchain and smolagents appear, Hexr’s weights know that the agent orchestrator (smolagents.CodeAgent()) outweighs a data-loading utility (langchain.text_splitter). Semgrep fires binary yes/no per pattern and has no concept of “which library is the orchestrator.”
No pattern detection. Semgrep cannot express function-level aggregates like “a loop containing ≥1 LLM call AND ≥1 tool call” (the ReAct pattern). The 7 agentic patterns Hexr detects — react, routing, parallelization, orchestrator-workers, evaluator-optimizer, reflection, prompt-chain — are architecturally beyond Semgrep’s rule language.
No bespoke fallback logic. Hexr’s weighted fallback to bespoke when only raw LLM client imports are present (with low confidence) requires the priority-and-weight system Semgrep doesn’t have.

What this means

Semgrep is an excellent security scanner. For agent-framework classification — which requires alias resolution, weighted multi-signal scoring, and function-level aggregate analysis — a purpose-built AST engine with YAML packs is the right tool. The gap isn’t about writing better Semgrep rules. It’s architectural: Semgrep’s rule language was designed for “find this security anti-pattern in a single file,” not “classify this multi-file project into one of 15 frameworks based on weighted evidence across AST shapes.”

The comparison is published with full results and the Semgrep ruleset in the SDK repo at sdk/python/_hexr_analyzer/benchmark/semgrep/ for anyone who wants to reproduce.

Part 4 — When Would We Ever Use Embeddings?

Three places, and we name them so the temptation is contained:

Internal signature authoring — a Hexr engineer asks “find code that looks like a multi-agent handoff so I can write a new pattern signature.” That’s a fuzzy human query; late-interaction retrieval over the pack corpus helps. It runs internally, never in customer builds.
hexr analyze --suggest (shipped, opt-in via pip install "hexr-sdk[enterprise]") — over the call sites the deterministic L1 + L2 layers left unclassified, the enterprise extra runs a small on-CPU late-interaction encoder and emits an advisory YAML diff the human reviews before it becomes a pack. It cannot mutate classification. It cannot write into the build manifest. It exists so the long tail of novel code turns into a reviewed pack change, not into silent misclassification. Advisory, never authoritative.
Documentation search on our docs site. Pure prose corpus. Embeddings make sense here.

Customer build path — the code that produces evidence rows and drives OPA — remains zero embeddings, forever.

Part 5 — Common Objections, Answered

“But you’ll miss novel frameworks.” We’ll miss them until someone writes a YAML — which is hours, not weeks, and customers can do it themselves. An embedding model would also miss them until retrained, and the retraining cycle is slower and more expensive than a YAML update.

“Regex is fragile.” Regex on raw source is fragile. Regex as a post-filter on a structurally-matched AST node is not — the regex only sees a small typed slice (a function name, a decorator argument), not the whole file.

“What about code generated by an LLM that doesn’t import the framework cleanly?” If the code doesn’t import the framework, it isn’t using the framework. Python is explicit about its dependencies; the import statement is ground truth. Anything that reaches crewai.Crew(...) had to import it somehow, and the AST shows the import.

“Won’t this be slow on large repos?” Ruff parses ~10 MB/s of Python source per core. The detector adds microseconds per node. We measure 300ms cold, 120ms warm/incremental on 100kLOC. No GPU. A small wheel that ships the native extension on macOS, Linux, and Windows.

Part 6 — Lessons We’d Write on a Whiteboard

If you know the shape of what you’re looking for, don’t reach for embeddings. This is the jxnl / Eugene Yan thesis stated plainly. Framework-detection has 15 known targets with crisp syntactic signatures.
Deterministic results go into the manifest. Probabilistic results go into a diff a human reads. We will not let an LLM, an encoder, or pyright silently rewrite our symbol table. Audit posture is a non-negotiable.
Air-gap compatibility is a forcing function. It killed pyright. It will kill the next “let’s just shell out to X” idea too. If a feature needs network on first invocation, it doesn’t belong in the default install path.
Latency is a UX feature. 173 ms is the difference between hexr build being something you forget is running and something you reach for --skip flags to disable.
One SKU, lean by default. Every extras-marker is a future support story. The default install has to be small enough that “lite” is not a meaningful word.
Publish the misses. A benchmark that only shows wins is marketing. A benchmark that shows 48/50 plus the two misses with their reasons is engineering.

Part 7 — What Shipped Since May (Status Update)

When we first wrote about the analyzer in May 2026, several items were on the roadmap. Here’s what landed:

Item	Status
Maturin mixed-mode build (native extension in wheel)	✅ Shipped — `pip install hexr-sdk` ships the precompiled Rust extension on macOS and Linux. No local Rust toolchain needed.
Real-world corpus benchmark	✅ Shipped — 50 fixtures across 17 upstream repos, 48/50 = 96.0%, published with the two misses and their explanations.
Scope resolution via ruff_python_semantic	✅ Shipped — aliased imports like `from openai.chat.completions import create as c; c(...)` correctly resolve and count as LLM calls.
MCP and ADK depth	✅ Shipped — both have dedicated YAML packs shipping in the wheel.
Signed pack feed	✅ Shipped — cosign keyless + Sigstore transparency log; `hexr update signatures` verifies and installs.
Hybrid pivot spec	✅ Shipped and live — the architecture described here is now the production SDK inside the Hexr hybrid platform.
Semgrep comparison	✅ Shipped — 15-rule custom ruleset benchmarked against same corpus. Semgrep 47/50 (94%); Hexr 48/50 (96%). Architectural gap documented below.

The SDK is now at version 0.5.9 and has been used in production enterprise deployments generating audit-grade evidence for HIPAA, SOC 2, and ISO 42001 compliance frameworks.

Part 8 — What This Means for Enterprise Customers

For the 96% case (the vast majority): hexr build Just Works. The detector picks the right framework, generates the right manifest, loads the right OPA bundle. No flags. No configuration. No model download.

For the 4% case: we’d rather you discover a misclassification at hexr build time (where it surfaces a clear error in the manifest) than at runtime. If you hit one, reach out — each new miss feeds directly into the pack-tuning queue, and the detector ships a new YAML rule, not a new model weight.

For the long tail: hexr analyze --suggest (enterprise license) will propose new pack entries for your human reviewers. The suggestion is advisory — it never auto-applies, never mutates the build manifest, never touches your runtime. Your team decides what becomes a rule.

What’s Next

This post covers the SDK’s static-analysis and detection layer — the build-time brain. A companion post on the Hexr platform will cover what happens after hexr build: the evidence emission pipeline, the SPIFFE-based identity layer, the hybrid control-plane / data-plane architecture, and how all of it turns agent activity into signed, auditor-readable evidence without your data ever leaving your cluster.

Stay tuned.

If you want to read more on the grep-vs-embeddings frame, start with jxnl on why grep beat embeddings in their SWE-bench agent and Eugene Yan’s “On the lost nuance of grep vs semantic”. We didn’t invent the frame. We just took it seriously for an agent compliance tool.

The Hexr SDK is distributed under an enterprise license. Licensed customers have full source access for audit, extension, and custom signature authoring. For more information: hexr.dev · hexr.cloud · Hexr on LinkedIn

Inside the Hexr SDK: Why Grep Beats Embeddings for Agent Code, How We Built a Sub-Second Rust Detector, and the 96% Benchmark

#Table of Contents

#Part 1 — The Philosophy: Why Grep Beats Embeddings for Code (And When It Doesn’t)

#TL;DR

#The four techniques in 60 seconds

#“Tokens carry semantics” — what that actually means

#“Recall and precision matter more than fuzzy similarity”

#Part 2 — The Implementation: How We Built a Sub-Second Rust Detector

#Why we threw out the old Python analyzer

#The architecture

#Parsing — Ruff, not the CPython parser

#Symbol resolution — knowing what cw.Agent resolves to

#Detection — the meat: YAML signature packs

#The 15 frameworks (shipped)

#From frameworks to patterns

#Customer-private signatures

#How new packs get to customers — signed, not scraped

#Part 3 — The Benchmark: 48/50, 173 Milliseconds, Two Published Misses

#The thesis

#Why this architecture vs embeddings, on every axis

#The three-layer detection ladder

#Four things we killed, in order

#1. Python AST analyzer (killed Q1 2026, replaced with Rust)

#2. Hexr-SDK-lite (killed March 2026)

#3. Pyright sidecar (killed May 2026)

#4. SaaS-first GTM + the marketplace (killed Q4 2025)

#The benchmark numbers

#The two misses, published

#Per-label P / R / F1

#Eval on synthetic fixtures (per-framework correctness)

#The Semgrep Comparison: Same Corpus, Same Labels, Different Architecture

#What we did

#Headline numbers

#The three Semgrep misses

#Why Semgrep hits its ceiling at ~94%

#What this means

#Part 4 — When Would We Ever Use Embeddings?

#Part 5 — Common Objections, Answered

#Part 6 — Lessons We’d Write on a Whiteboard

#Part 7 — What Shipped Since May (Status Update)

#Part 8 — What This Means for Enterprise Customers

#What’s Next

Related Posts

Table of Contents