Fine-Tuning Retrieval and Context Pruning Models for Stablecoin Regulatory Intelligence

Models: sugiv/modernbert-us-stablecoin-encoder | sugiv/stablebridge-pruner-highlighter

What is Stablebridge?

Stablebridge is a regulatory intelligence and compliance platform for the global stablecoin industry. It enables compliance teams, legal analysts, and financial institutions to search, analyze, and reason over regulatory requirements across multiple jurisdictions.

The stablecoin regulatory landscape is evolving rapidly. In the US alone, we’re tracking the GENIUS Act (Guiding and Establishing National Innovation for US Stablecoins), the Clarity for Payment Stablecoins Act, SEC framework statements, and guidance from the Federal Reserve, Treasury, FDIC, OCC, and FinCEN. Globally, frameworks like MiCA in the EU, FCA guidelines in the UK, and MAS regulations in Singapore are shaping how digital payment instruments operate.

The core problem: Regulatory documents are long, dense, and cross-referential. A compliance analyst asking “What reserve requirements apply to stablecoin issuers?” needs to find the right passages across hundreds of pages, identify the most relevant sentences, and compile a focused answer — all without missing critical requirements buried in subsections or cross-references.

Our approach: Build a specialized RAG (Retrieval-Augmented Generation) pipeline with domain-adapted models that can:

Retrieve relevant regulatory passages with domain-specific embeddings
Rerank candidates using a cross-encoder that understands legal language
Prune and highlight — compress retrieved context by 50-80% while preserving the most relevant sentences, reducing LLM costs and improving answer quality

The US Stablecoin Regulatory Dataset

For our first jurisdiction, we focused on US stablecoin regulation. We assembled a corpus from primary regulatory sources:

Congress.gov — Full text of stablecoin-related bills (GENIUS Act, Clarity for Payment Stablecoins Act)
SEC.gov — Framework statements on digital assets, staff bulletins, Chair speeches
Federal agencies — Reserve, Treasury, FDIC, OCC, and FinCEN guidance on digital assets and payment instruments

The corpus covers the key regulatory themes:

Licensing — Who can issue payment stablecoins and under what authority
Reserve requirements — What assets back stablecoins and how they must be held
Disclosure obligations — What issuers must report and how often
Risk limits — Capital adequacy, concentration limits, and redemption guarantees
Enforcement — Penalties for non-compliance, SEC jurisdiction boundaries

Silver Label Generation

Training domain-specific models requires labeled data. We used Claude Opus 4 with reasoning-first prompts to generate silver training labels — over 10,000 annotated examples where each example includes:

A regulatory query (e.g., “What penalties exist for non-compliant stablecoin issuers?“)
A document passage from the corpus
Sentence-level relevance labels (keep/drop for each sentence)
LLM reasoning explaining why each sentence is or isn’t relevant

The reasoning-first approach forces the annotation model to explain its logic before making decisions, significantly improving label quality over direct annotation.

From these base labels, we derived:

10,260 triplets (query, positive passage, hard negative) for encoder training
10,006 token-level examples for pruner training

Two Fine-Tuned Models

We trained two complementary models, each targeting a different stage of the RAG pipeline.

Model 1: Domain-Adapted Encoder (LoRA)

Model: sugiv/modernbert-us-stablecoin-encoder
Base: answerdotai/ModernBERT-base (149M parameters, 768-dim embeddings)
Method: LoRA (Low-Rank Adaptation) fine-tuning
Task: Dense retrieval — encoding queries and documents into embeddings that bring regulatory-relevant pairs closer together in vector space

Why ModernBERT? ModernBERT is a 2024-vintage encoder that incorporates architectural improvements from the GPT lineage (rotary position embeddings, Flash Attention native, unpadding) while remaining a bidirectional encoder optimized for embedding tasks. It’s more parameter-efficient and faster than BERT-large variants for our use case.

Training details:

Parameter	Value
LoRA rank	16
LoRA alpha	32
Target modules	Wqkv, Wo
Dropout	0.1
Training loss	Triplet loss (query, positive, hard negative)
Adapter size	8.77 MB
Training data	10,260 triplets

Results:

Metric	Score
NDCG@10	0.847
MRR@10	0.802
Recall@100	1.000

The key result: perfect recall at 100 candidates means the encoder never misses a relevant document in the top 100 — critical for a regulatory pipeline where missing a requirement could have legal consequences.

Loading the model:

from peft import PeftModel
from transformers import AutoModel, AutoTokenizer

base = AutoModel.from_pretrained("answerdotai/ModernBERT-base", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "sugiv/modernbert-us-stablecoin-encoder")
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base", trust_remote_code=True)

inputs = tokenizer("What are stablecoin reserve requirements?", return_tensors="pt")
embeddings = model(**inputs).last_hidden_state[:, 0, :]  # 768-dim CLS token

Model 2: Pruner + Highlighter (Custom MLP Head)

Model: sugiv/stablebridge-pruner-highlighter
Base: BAAI/bge-reranker-v2-m3 (568M parameters, frozen)
Innovation: A trainable PruningHead MLP (525K parameters) on top of the frozen reranker
Tasks: Reranking (via base model), token-level pruning and semantic highlighting (via PruningHead)

Architecture:

Input: [BOS] query [SEP] passage_tokens [SEP]
                    |
    ┌───────────────────────────────────┐
    │ BGE-reranker-v2-m3 (FROZEN)       │
    │ 568M params, 1024 hidden dim      │
    │ All weights frozen during training│
    └───────────────────────────────────┘
                    |
        ┌───────────┴───────────┐
        ▼                       ▼
  [CLS] logit              Token hidden states
  → sigmoid                (N × 1024-dim)
  = rerank score                |
                    ┌───────────────────────┐
                    │ PruningHead MLP       │
                    │ Linear(1024, 512)     │
                    │ → GELU → Dropout(0.2) │
                    │ → Linear(512, 1)      │
                    │ → Sigmoid             │
                    │ (525K trainable params)│
                    └───────────────────────┘
                                |
                    Token-level keep probabilities
                                |
                    ┌───────────────────────┐
                    │ Sentence aggregation  │
                    │ + Semantic labeling   │
                    └───────────────────────┘
                                |
          ┌─────────┬───────────┴───────────┐
          ▼         ▼                       ▼
    "highlight"   "kept"              "pruned"
    (prob ≥ 0.9)  (prob ≥ 0.6)       (prob ﹤ 0.6)

The key insight: The same forward pass through the frozen reranker produces both a relevance score (from the classifier head) and token-level hidden states (from the last transformer layer). Our PruningHead MLP transforms these hidden states into keep/drop probabilities, enabling unified reranking + pruning in a single inference call.

Training details:

Parameter	Value
Loss	BCEWithLogitsLoss
Class imbalance	105:1 (pos_weight=70)
Training data	10,006 examples
Trainable params	525K (0.09% of total model)
Batch size	4
Optimizer	AdamW

Output labels:

Label	Threshold	Meaning
`highlight`	prob ≥ 0.9	Directly answers the query — highest confidence
`kept`	prob ≥ 0.6	Supporting context worth preserving
`pruned`	prob ﹤ 0.6	Can be safely removed to reduce LLM input
`summary`	—	Compression statistics for the passage

Loading the model:

import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import hf_hub_download

class PruningHead(nn.Module):
    def __init__(self, hidden_size=1024, intermediate_size=512, dropout=0.2):
        super().__init__()
        self.classifier = nn.Sequential(
            nn.Linear(hidden_size, intermediate_size),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(intermediate_size, 1),
        )
    def forward(self, embeddings):
        return self.classifier(embeddings).squeeze(-1)

encoder = AutoModel.from_pretrained("BAAI/bge-reranker-v2-m3")
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3")
head = PruningHead()
ckpt = torch.load(
    hf_hub_download("sugiv/stablebridge-pruner-highlighter", "best.pt"),
    map_location="cpu", weights_only=False
)
head.load_state_dict(ckpt["model_state_dict"])

The RAG Pipeline

Our full pipeline chains both models through five stages:

Query: "What reserve requirements apply to stablecoin issuers?"
                    │
                    ▼
    ┌─────────────────────────────────────────┐
    │  Stage 1: Encode                        │
    │  sugiv/modernbert-us-stablecoin-encoder  │
    │  → 768-dim domain-adapted embeddings    │
    └─────────────────────────────────────────┘
                    │
                    ▼ Cosine similarity → top-k candidates
    ┌─────────────────────────────────────────┐
    │  Stage 2: Retrieve                      │
    │  Dense retrieval over corpus            │
    │  → Top 5 candidates                    │
    └─────────────────────────────────────────┘
                    │
                    ▼
    ┌─────────────────────────────────────────┐
    │  Stage 3: Rerank                        │
    │  sugiv/stablebridge-pruner-highlighter   │
    │  Cross-encoder scoring → relevance [0,1]│
    └─────────────────────────────────────────┘
                    │
                    ▼ Top-3 by rerank score
    ┌─────────────────────────────────────────┐
    │  Stage 4: Extract & Highlight           │
    │  sugiv/stablebridge-pruner-highlighter   │
    │  Token probabilities → sentence labels  │
    │  highlight | kept | pruned              │
    └─────────────────────────────────────────┘
                    │
                    ▼
    ┌─────────────────────────────────────────┐
    │  Stage 5: Compress                      │
    │  Keep only highlighted + kept sentences │
    │  → 74% average compression             │
    │  → LLM generation with focused context │
    └─────────────────────────────────────────┘

End-to-end latency: 104ms average across 5 diverse regulatory queries — encoding, retrieval, reranking, and extraction combined. That’s fast enough for interactive compliance search.

Deploying on SuperLinked’s Search Inference Engine (SIE)

To move from local inference to production-grade serving, we deployed both models on SuperLinked’s Search Inference Engine (SIE) — a platform purpose-built for small model inference (embeddings, rerankers, extractors) as opposed to large language model inference.

Why SIE?

The LLM inference ecosystem (vLLM, TensorRT-LLM, SGLang) optimizes for loading one massive model across multiple GPUs. Small model inference is the opposite problem: many models (embedders, rerankers, extractors) sharing one GPU, with fast model switching and memory management. SIE is built specifically for this.

SIE’s three primitives cover the entire small-model inference space:

Primitive	Purpose	Our Usage
`encode(model, items)`	Text → vector embeddings	Encoder LoRA for domain-adapted retrieval
`score(model, query, items)`	Cross-attention reranking	Pruner model for relevance scoring
`extract(model, items)`	Entity/structure extraction	Pruner model for semantic highlighting

Key SIE Features We Used

Multi-Model GPU Sharing: Our pipeline requires both the encoder (ModernBERT-base + LoRA) and the pruner (BGE-reranker + PruningHead) on the same GPU. SIE’s reactive LRU memory management handles this automatically — models are loaded on demand, kept in GPU memory for fast switching, and evicted only when memory pressure requires it. In our tests, we ran 10 rounds of rapid alternation between all endpoints with zero errors and zero memory growth.

LoRA Profile System: SIE serves LoRA adapters through named profiles. We register answerdotai/ModernBERT-base as the base model and define a us-regulatory profile that activates our sugiv/modernbert-us-stablecoin-encoder LoRA weights. Clients simply specify profile: "us-regulatory" — no model changes needed:

# Base embeddings (no LoRA)
base_emb = client.encode("answerdotai/ModernBERT-base", items)

# Domain-adapted embeddings (with LoRA)
lora_emb = client.encode("answerdotai/ModernBERT-base", items,
                          options={"profile": "us-regulatory"})

The same mechanism scales to multi-jurisdiction deployment — we can add eu-mica, uk-fca, sg-mas profiles without deploying separate models.

Cost-Based Batching: Regulatory documents vary wildly in length — from 2-page memos to 400-page directives. SIE batches by token count rather than request count, ensuring efficient GPU utilization regardless of input size. We tested with short (35 chars), medium (930 chars), and long (3,081 chars) documents and saw the batching system handle them efficiently with a 4.2x throughput improvement when batching 5 documents together.

Custom Adapter Framework: SIE’s built-in CrossEncoderAdapter only returns scalar rerank scores. Our pruner needs access to hidden states for the PruningHead MLP. We wrote a 659-line StablebridgePrunerAdapter that extends SIE’s adapter protocol:

score() — Reranking via the base model’s classifier head
extract() — Token-level pruning and highlighting via PruningHead, returning character-offset entities with semantic labels

The adapter loads the frozen reranker + PruningHead from HuggingFace Hub and integrates seamlessly with SIE’s batching, memory management, and API surface.

Live Performance Results

We ran a comprehensive integration test suite on a self-hosted SIE instance (RTX PRO 6000 Blackwell, 98GB VRAM):

Metric	Result	SOW Target
RAG E2E latency	104ms	﹤ 500ms
Throughput (sequential)	64 QPS	﹥ 5 QPS
Throughput (4x concurrent)	187 QPS	—
GPU memory	2.4% of 98GB	﹤ 20%
Ranking accuracy	100% correct	—
LoRA switching overhead	+6ms	—
Multi-model contention	0 errors	—
Context compression	74%	50-80%

The results demonstrate that SIE is well-suited for production deployment of domain-specific RAG pipelines with custom models.

The Encoder: Deep Dive

Our encoder fine-tuning takes answerdotai/ModernBERT-base and adapts it for regulatory text understanding via LoRA.

Why LoRA instead of full fine-tuning?

Parameter efficiency — Only 8.77 MB of adapter weights vs 149M base model parameters
Base preservation — The general-purpose encoder capabilities remain intact; LoRA adds domain knowledge without catastrophic forgetting
Serving efficiency — SIE can host the base model once and switch between multiple LoRA profiles (one per jurisdiction) with minimal memory overhead
Training speed — LoRA converges in a single epoch on our 10K triplet dataset

What the encoder learns: Through triplet training (query + relevant passage vs query + irrelevant passage), the LoRA adapter learns that regulatory concepts like “reserve requirements,” “segregation of assets,” and “1:1 backing” are semantically related when they appear in the context of stablecoin compliance — relationships that a general-purpose encoder would rank weakly.

Impact on retrieval: The NDCG@10 of 0.847 means the encoder places the most relevant passage in the top positions consistently. Combined with Recall@100 = 1.0, the pipeline never misses a relevant document in its candidate set.

The Pruner and Highlighter: Deep Dive

The pruner is our more novel contribution. Rather than training a separate model for reranking and another for context compression, we built a single architecture that does both.

The efficiency argument: In a traditional pipeline, you’d run a cross-encoder for reranking (one forward pass), then a separate model for context pruning (another forward pass). Our approach extracts both signals from a single forward pass through the BGE-reranker backbone — the classifier head gives the rerank score, and our PruningHead MLP gives token-level keep probabilities. This halves the compute cost for the reranking + pruning stages.

Why freeze the backbone? The BGE-reranker-v2-m3 is already an excellent general-purpose reranker (ranked #2 on MTEB reranking leaderboard). By freezing its 568M parameters and only training our 525K-parameter MLP head, we:

Preserve the strong general reranking capability
Reduce training to hours instead of days
Avoid the risk of catastrophic forgetting with a small domain dataset
Keep the model compatible with SIE’s existing cross-encoder infrastructure

Semantic highlighting as a product feature: Beyond LLM context compression, the token-level probabilities serve as semantic highlights for human analysts. A compliance officer reviewing a 50-page bill can immediately see which sentences are most relevant to their query — the highlight labels (probability ≥ 0.9) pinpoint the answer, while kept labels (≥ 0.6) provide supporting context.

Compression in practice: On our live test with 5 diverse regulatory queries across 12 corpus documents, the pruner achieved 74% average compression — meaning only 26% of the original text was passed to the LLM. For queries with highly focused answers (like “What reserve requirements apply?”), compression reached 62%. For broader queries, the pruner conservatively keeps more context. This adaptive behavior is exactly what we want — aggressive pruning when the answer is localized, conservative when the topic is broad.

Conclusion

Building domain-specific RAG for regulatory intelligence required solving three connected problems:

Retrieval quality — General-purpose encoders don’t understand that “1:1 backing requirement” and “full reserve maintenance” mean the same thing in stablecoin regulation. Our LoRA fine-tuned encoder (sugiv/modernbert-us-stablecoin-encoder) closes this gap with NDCG@10 of 0.847.
Context compression — Passing entire regulatory documents to an LLM is expensive and noisy. Our pruner/highlighter (sugiv/stablebridge-pruner-highlighter) compresses context by 74% on average while achieving 100% ranking accuracy on our test set.
Production serving — Two custom models need to coexist on a GPU, handle variable-length regulatory documents, and support multi-jurisdiction expansion via LoRA profiles. SIE handles all of this with 104ms end-to-end latency and 64+ QPS throughput.

The US jurisdiction is our first. The architecture is designed to scale to 8+ jurisdictions (EU MiCA, UK FCA, Singapore MAS, and more) through additional LoRA profiles on the same base encoder and jurisdiction-specific pruning heads — all sharing a single GPU through SIE’s memory management.

Links:

Encoder LoRA: sugiv/modernbert-us-stablecoin-encoder
Pruner/Highlighter: sugiv/stablebridge-pruner-highlighter
PR with full code: superlinked/sie#4

Fine-Tuning Retrieval and Context Pruning Models for Stablecoin Regulatory Intelligence

#Table of Contents

#What is Stablebridge?

#The US Stablecoin Regulatory Dataset

#Silver Label Generation

#Two Fine-Tuned Models

#Model 1: Domain-Adapted Encoder (LoRA)

#Model 2: Pruner + Highlighter (Custom MLP Head)

#The RAG Pipeline

#Deploying on SuperLinked’s Search Inference Engine (SIE)

#Why SIE?

#Key SIE Features We Used

#Live Performance Results

#The Encoder: Deep Dive

#The Pruner and Highlighter: Deep Dive

#Conclusion

Related Posts

Table of Contents