Mastering Natural Language Processing with Ensemble NLU: Combining DIET, ModernBERT, and LLM

Introduction: Combining Classic NLU and Modern Language Models for Rewriting Emails and Official Letters

Natural language responses to agentic frameworks attempting to rewrite emails or official letters present unique challenges in extracting structured information. Our project addresses these challenges through a novel ensemble approach that combines multiple Natural Language Understanding (NLU) technologies.

Our project implements a sophisticated pipeline that intelligently routes user-provided responses between specialized models based on intent classification, semantic similarity, and content validation. This hybrid architecture leverages the unique strengths of different model types:

DIET (Dual Intent and Entity Transformer): A specialized model for recognizing intents and extracting structured entities from user responses
ModernBERT: A fine-tuned semantic similarity model evaluating question-answer relevance
DeepSeek LLM: A powerful large language model for handling complex or ambiguous cases

                    ┌───────────────┐
                    │  User Query   │
                    └───────┬───────┘
                            │
                            ▼
               ┌────────────────────────┐
               │    Ensemble Router     │
               └────────────┬───────────┘
                            │
               ┌────────────┴───────────┐
               │                        │
               ▼                        ▼
     ┌──────────────────┐   ┌─────────────────────┐
     │  DIET Classifier │   │     ModernBERT      │
     └────────┬─────────┘   └──────────┬──────────┘
              │                        │
              ▼                        ▼
     ┌──────────────────┐   ┌─────────────────────┐
     │Intent/Entity     │   │Semantic Similarity  │
     │Recognition       │   │Calculation          │
     └────────┬─────────┘   └──────────┬──────────┘
              │                        │
              └────────────┬───────────┘
                           │
                           ▼
              ┌─────────────────────────┐
              │   Ensemble Decision     │
              └────────────┬────────────┘
                           │
                ┌──────────┴─────────┐
                │                    │
                ▼                    ▼
         ┌────────────┐      ┌─────────────┐
         │    NLU     │      │  DeepSeek   │
         │ Processing │      │     LLM     │
         └────────────┘      └─────────────┘

Our system delivers impressive results with an 85.71% success rate in routing decisions while maintaining robust performance across diverse natural language responses for email and letter rewriting scenarios.

DIET: Specialized Intent and Entity Extraction for Response Processing

The DIET (Dual Intent and Entity Transformer) classifier forms the foundation of our ensemble system, providing specialized natural language understanding for responses related to email and letter rewriting.

Dataset Creation and Preparation

Our carefully crafted training dataset focuses on interactions related to email and letter rewriting. The data is organized in JSON format with question-answer pairs:

{
  "question": "What's your email address?",
  "answer": "john.doe@example.com",
  "is_correct": true
}

The dataset contains thousands of carefully curated examples covering:

Email and Contact Information Responses:

Example questions: “What’s your email?”, “Where can I send you an email?”, “What’s your preferred email address?”
Valid responses: Properly formatted email addresses like ”john.doe@example.com”
Invalid responses: Text without email format, irrelevant statements

Subject Line Suggestions:

Example questions: “What subject should I use?”, “What’s a good subject for this letter?”, “What should the subject be?”
Valid responses: Properly formatted subjects like “Meeting Request: Project Update”
Invalid responses: Single words, email addresses, unrelated content

Conversational Exchanges:

Questions: “Hello there”, “How are you?”
Appropriate responses: “Hi, how can I help?”, “Good morning!”

The training data shows careful balancing across these categories to prevent model bias, with equal representation of positive (correct) and negative (incorrect) examples for robust classification.

Training Process and Configuration

The DIET model was trained using a customized Rasa pipeline, with important modifications to optimize for our specific letter and email rewriting use case:

pipeline:
- name: WhitespaceTokenizer
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- analyzer: char_wb
  max_ngram: 4
  min_ngram: 1
  name: CountVectorsFeaturizer
- constrain_similarities: true
  entity_recognition: true
  entity_recognition_confidence_threshold: 0.6
  epochs: 150
  intent_classification_confidence_threshold: 0.7
  model_confidence: softmax
  name: DIETClassifier

Notable settings include:

Entity recognition threshold of 0.6 (balanced for precision/recall)
Intent classification threshold of 0.7 (ensuring high confidence predictions)
150 training epochs for thorough model convergence

Model Evaluation and Performance

The DIET model demonstrates strong performance in intent classification, reliably identifying when users are providing emails, subject lines, or engaging in conversational exchanges. Analysis of the test results shows:

Intent Classification: High accuracy (>85%) in identifying the correct intent
Entity Extraction: Reasonably effective but room for improvement, particularly with email formats
Confidence Scoring: Reliable confidence metrics for routing decisions

The model is particularly effective at disambiguating between email provision and subject line suggestions, even when the phrasing is similar, due to the specialized training.

ModernBERT: Semantic Understanding for Email and Letter Rewriting Contexts

While DIET excels at intent classification, our ModernBERT model provides the semantic understanding layer necessary for evaluating response relevance and quality.

Custom Data Preparation for Semantic Similarity

We took a sophisticated approach to preparing training data for semantic similarity:

def analyze_data(data):
    """Analyze the training data distribution"""
    email_questions = []
    subject_questions = []
    other_questions = []
    
    for item in train_data:
        question = item["question"].lower()
        
        if "email" in question:
            email_questions.append(item)
        elif "subject" in question:
            subject_questions.append(item)
        else:
            other_questions.append(item)

This categorization allows for question-specific threshold tuning and balanced representation during training. The script creates contrast pairs for training, ensuring that the model learns to distinguish between relevant and irrelevant responses:

# Create training examples with positive and negative pairs
train_examples = []
for item in train_data:
    label = 1.0 if item.get("is_correct", True) else 0.0
    train_examples.append(InputExample(texts=[item["question"], item["answer"]], label=label))

Enhanced Training with Flash Attention

The ModernBERT training leverages several advanced techniques:

# Initialize model on CPU first to prevent OOM errors
model = SentenceTransformer("answerdotai/ModernBERT-base", device="cpu")
model.to(device)  # Now move to GPU

# Use OnlineContrastiveLoss with margin for better similarity learning
train_loss = losses.OnlineContrastiveLoss(
    model=model,
    distance_metric=losses.SiameseDistanceMetric.COSINE_DISTANCE,
    margin=0.5
)

# Training with mixed precision for performance
model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=num_epochs,
    evaluator=evaluator,
    warmup_steps=int(len(train_dataloader) * 0.1),
    use_amp=True  # Mixed precision training
)

Key enhancements include:

Flash Attention for faster transformer operations
CPU initialization followed by GPU transfer to prevent memory issues
Mixed precision training for speed improvements
OnlineContrastiveLoss for better similarity boundary learning

Question-Type Specific Thresholding

A notable innovation in our approach is the use of question-specific similarity thresholds:

# Set appropriate threshold based on question type
is_email = "email" in question.lower()
is_subject = "subject" in question.lower()
threshold = 0.85 if is_email else (0.50 if is_subject else 0.70)

These thresholds were determined through extensive testing:

Email questions: 0.85 (requiring high precision)
Subject questions: 0.65 (more flexible due to greater variability)
Default/conversational: 0.80 (balanced approach)

This adaptive thresholding accommodates the inherent differences in how semantically similar good responses are across different question types.

Custom Trained Models: Specialized Intelligence for Email and Letter Rewriting

Our training efforts resulted in two specialized models published to HuggingFace, each serving a distinct purpose in our ensemble system.

sugiv/email-processing-diet

This custom-trained DIET model specializes in intent recognition and entity extraction for email and letter rewriting. Its capabilities include:

Intent Classification: Accurately identifying:

provide_email intent when a user is giving an email address
provide_subject intent when suggesting subject lines
greeting, goodbye, and thank_you for conversational exchanges

Entity Extraction: Recognizing structured entities within text:

Identifying email entities like ”john.doe@example.com”
Extracting subject entities from longer responses

Confidence Scoring: Providing reliable confidence metrics used for routing decisions

This model serves as the first layer of intelligence in our ensemble approach, quickly identifying the user’s intent with high precision.

sugiv/email-processing-modernbert

Our fine-tuned ModernBERT model adds crucial semantic understanding capabilities:

Semantic Similarity: Determining if an answer is actually relevant to a question, beyond simple keyword matching
Context-Aware Validation: Understanding when responses appropriately address questions based on contextual meaning
Quality Assessment: Evaluating how well a response satisfies the intent of a question

This model complements the DIET classifier by providing deeper semantic analysis, particularly valuable for detecting cases where responses might superficially appear correct but lack meaningful content.

The combination of these specialized models provides several advantages over using general-purpose models:

Domain Specificity: Tailored specifically for email and letter rewriting patterns
Efficient Processing: Lower computational requirements than large language models
Explainable Decisions: Clear confidence scores and similarity metrics for transparent routing
Specialized Performance: Superior accuracy on rewriting tasks compared to general models

The Ensemble Router: Orchestrating Multiple Models for Optimal Decisions

The heart of our system is the intelligent ensemble router that coordinates multiple models to make sophisticated routing decisions.

Router Architecture

The ensemble router combines signals from multiple sources:

┌───────────────────────┐      ┌────────────────────────┐
│                       │      │                        │
│     Diet Classifier   ├─────►│   Intent & Entities    │
│                       │      │                        │
└───────────────────────┘      └──────────┬─────────────┘
                                          │
                                          ▼
┌───────────────────────┐      ┌────────────────────────┐
│                       │      │                        │
│     ModernBERT        ├─────►│   Semantic Similarity  │
│                       │      │                        │
└───────────────────────┘      └──────────┬─────────────┘
                                          │
                                          ▼
┌───────────────────────┐      ┌────────────────────────┐
│                       │      │                        │
│  Content Validators   ├─────►│   Content Validation   │
│                       │      │                        │
└───────────────────────┘      └──────────┬─────────────┘
                                          │
                                          ▼
                               ┌────────────────────────┐
                               │                        │
                               │   Weighted Ensemble    │
                               │       Decision         │
                               │                        │
                               └──────────┬─────────────┘
                                          │
                                ┌─────────┴────────────┐
                                │                      │
                                ▼                      ▼
                         ┌────────────┐        ┌─────────────┐
                         │            │        │             │
                         │    NLU     │        │  DeepSeek   │
                         │            │        │     LLM     │
                         └────────────┘        └─────────────┘

Weighted Feature Combination

The router calculates an ensemble score that combines multiple signals:

Intent Confidence: From the DIET classifier (high weight for high-confidence predictions)
Semantic Similarity: From ModernBERT (measuring relevance between question and answer)
Entity Presence: Validating presence of expected entities based on question type
Content Validation: Specific rules based on question content

This weighted combination provides a nuanced understanding beyond what any single model can achieve.

Multi-Level Classification

The ensemble score translates to different quality levels:

- "complete" (score ≥ 0.8): Answer fully addresses the question
- "sufficient" (score ≥ 0.6): Answer adequately addresses the question
- "partial" (score ≥ 0.4): Answer partially addresses the question
- "insufficient" (score < 0.4): Answer doesn't meaningfully address the question

Contextual Entity Validation

Entity checking is specific to question types, as seen in the validator implementation:

def validate_response(response):
    """Validate a response based on intent and content."""
    intent = response.get("intent", "")
    answer = response.get("answer", "")
    
    validated = response.copy()
    validated["valid"] = False
    
    # Basic validation based on intent
    if intent == "provide_email":
        # Check if answer contains @ sign and a domain
        validated["valid"] = "@" in answer and "." in answer
        
    elif intent == "provide_subject":
        # Subject should have reasonable length and not look like an email
        validated["valid"] = len(answer.split()) >= 2 and "@" not in answer
        
    elif intent in ["greeting", "goodbye", "thank_you"]:
        # More lenient validation for conversational intents
        validated["valid"] = len(answer.split()) > 0
        
    else:
        # Default to valid for unknown intents
        validated["valid"] = True
        
    return validated

This approach overrides score-based decisions for high-confidence intents (>0.95), applying content-specific validation rules for more reliable decisions.

Evaluation Results

Testing demonstrates the effectiveness of our ensemble approach, with strong results across diverse cases:

Test	Question	Answer	Similarity	Threshold	Route	Correct?
1	Email address?	john.doe@example.com	0.8963	0.85	NLU	✓
2	Subject line?	Meeting Request: Project Update	0.6570	0.70	NLU	✓
3	Hello there	Hi, how can I help?	0.7706	0.80	NLU	✓
4	Email address?	I don’t want to share	0.8721	0.85	DeepSeek	✓
5	Email address?	The weather is nice	0.8243	0.85	DeepSeek	✓
6	Subject?	Use ‘Meeting Request’	0.5225	0.70	NLU	✓
7	Subject?	My email is contact@example.org	0.6796	0.70	DeepSeek	✓

The ensemble achieves an impressive 85.71% accuracy rate in routing decisions, successfully handling complex edge cases like:

Providing irrelevant responses to email questions
Using email formats in subject line answers
Detecting contextually inappropriate responses

Implementation Details and Future Improvements

The implementation of our ensemble router demonstrates sophisticated API design and robust error handling patterns.

Key Features of the Enhanced Ensemble Approach

The ensemble router combines multiple signals from different models through an intelligently weighted system:

def calculate_ensemble_score(diet_result, modernbert_result, question, answer):
    """Calculate an ensemble score combining DIET and ModernBERT results with content checks"""
    # Extract base scores
    intent_confidence = diet_result.get("intent_confidence", 0.0)
    similarity_score = modernbert_result.get("similarity", 0.0)
    
    # Question-specific content checks
    is_email_question = "email" in question.lower()
    is_subject_question = "subject" in question.lower()
    
    # Entity presence checks
    has_email = "@" in answer and "." in answer
    reasonable_subject = len(answer.split()) >= 2 and "@" not in answer
    
    # Weight different factors based on question type
    if is_email_question:
        content_match = 1.0 if has_email else 0.2
        weight_intent = 0.4
        weight_similarity = 0.4
        weight_content = 0.2
    elif is_subject_question:
        content_match = 1.0 if reasonable_subject else 0.2
        weight_intent = 0.3
        weight_similarity = 0.5
        weight_content = 0.2
    else:
        content_match = 0.8  # Default content match weight
        weight_intent = 0.5
        weight_similarity = 0.4
        weight_content = 0.1
        
    # Calculate ensemble score
    ensemble_score = (
        weight_intent * intent_confidence +
        weight_similarity * similarity_score +
        weight_content * content_match
    )
    
    return ensemble_score

The multi-level classification system translates ensemble scores into quality levels:

def get_quality_level(ensemble_score):
    """Translate ensemble score to a quality level"""
    if ensemble_score >= 0.8:
        return "complete"
    elif ensemble_score >= 0.6:
        return "sufficient"
    elif ensemble_score >= 0.4:
        return "partial"
    else:
        return "insufficient"

The router implementation includes robust parallel API calls to both models and intelligent fallback mechanisms:

async def route_query(question, answer, timeout=10.0):
    """Route a query between NLU and DeepSeek based on intent and similarity"""
    # Make parallel calls to both services
    diet_future = asyncio.create_task(call_diet_service(question, answer))
    modernbert_future = asyncio.create_task(call_modernbert_service(question, answer))
    
    try:
        # Wait for both results with timeout
        diet_result, modernbert_result = await asyncio.gather(
            diet_future, 
            modernbert_future,
            return_exceptions=True
        )
        
        # Handle exceptions and fallback logic
        if isinstance(diet_result, Exception):
            logger.error(f"DIET service error: {diet_result}")
            diet_result = {"intent": "", "intent_confidence": 0.0, "entities": []}
            
        if isinstance(modernbert_result, Exception):
            logger.error(f"ModernBERT service error: {modernbert_result}")
            # Try local fallback for ModernBERT
            try:
                modernbert_result = await call_local_modernbert(question, answer)
            except Exception as e:
                logger.error(f"Local ModernBERT fallback error: {e}")
                modernbert_result = {"similarity": 0.0}
        
        # Calculate ensemble score and make routing decision
        ensemble_score = calculate_ensemble_score(diet_result, modernbert_result, question, answer)
        quality = get_quality_level(ensemble_score)
        
        # Apply special case handling for high-confidence intents
        result = apply_special_case_routing(diet_result, modernbert_result, question, answer, ensemble_score, quality)
        
        return result
        
    except Exception as e:
        logger.error(f"Error in route_query: {e}")
        # Default to DeepSeek in case of errors
        return {"route": "deepseek", "reason": f"Error: {str(e)}"}

Areas for Further Improvement

While our system performs well with an 85.71% success rate, several areas offer potential for improvement:

Entity Extraction: Email entity extraction still faces challenges as noted in test results. Potential improvements include:
- Adding more labeled training examples with explicit email entities
- Further reducing the entity recognition confidence threshold below 0.6
- Adding regex post-processing specifically for email formats
Validation Logic: The validation function needs refinement to properly handle edge cases.
Subject Line Similarity: Test Case 6 shows a similarity of only 0.5225 for a valid subject line. This indicates that additional subject-line focused training would be beneficial.

Conclusion: The Power of Ensemble NLU for Email and Letter Rewriting

Our NLU pipeline for processing user responses in email and letter rewriting represents a significant advancement in combining traditional NLU techniques with modern transformer models and LLMs. By leveraging an ensemble approach, we achieve several key benefits:

Improved Accuracy: 85.71% routing accuracy that exceeds what any single model approach could achieve
Explainability: Clear confidence scores and similarity metrics provide transparency into decision-making
Efficiency: Lightweight models handle most cases, reserving the computationally intensive LLM for only the necessary situations
Robustness: Multiple validation layers ensure resilience against edge cases and unexpected inputs

This hybrid approach demonstrates that combining specialized models with targeted training can outperform generic large language models for domain-specific tasks like processing responses for email and letter rewriting. The ability to route between specialized NLU and general-purpose LLM processing provides both efficiency and flexibility.

As agentic frameworks continue to evolve, systems like ours that can reliably extract structured information from natural language responses will continue to provide significant value for automated document generation and communication tasks.

References

EmailRLetter-IntentEntity-ExtractorRouter-Hybrid Repository
RunPod Serverless Deployment Code
ModernBert
Bunk, T., Varshneya, D., Vlasov, V., & Nichol, A. (2020). DIET: Lightweight Language Understanding for Dialogue Systems. arXiv preprint arXiv:2004.09936.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., … & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.

Calculus Related Posts