Chat Context Anticipation Engine

Created: 2026-01-21 Status: Design Phase Related: Reggie SMS, Chat Context, Embedding Strategy

Executive Summary

The Context Anticipation Engine is a parallel agent that monitors conversations in real-time, predicts where they're heading, and pre-fetches relevant context before the user asks for it. The goal: keep the LLM ahead of the user, not behind.

This is not just "find similar past conversations" - it's learned pattern recognition per user per topic, enabling speculative pre-fetch with high confidence.

---

1. Core Concept

1.1 The Problem

Traditional chat context:

1. User asks question
2. System searches for context
3. LLM waits for context
4. Response generated
5. Latency felt by user

1.2 The Solution

Anticipation engine running in parallel:

1. User starts talking
2. Shadow agent detects topic
3. Shadow agent predicts direction based on user's patterns
4. Pre-fetch begins immediately
5. User completes question
6. Context already loaded → instant response

USER CONVERSATION                    SHADOW AGENT (parallel)
==================                   ======================

User: "Hey Reggie"                   → Load user profile
                                     → Load recent conversation history
                                     → Ready user's common topics
                                     
Reggie: "Hi! How can I help?"        → Waiting for signal...

User: "Quick question about          → TOPIC: roster (detected)
      the roster"                    → USER PATTERN: Sarah + roster
                                       - 80% sick coverage
                                       - 10% SIL scheduling
                                       - 10% other
                                     → HIGH CONFIDENCE: Fetch sick coverage data
                                     → PRE-FETCH: Next 48hrs shifts
                                     → PRE-FETCH: Available staff + quals
                                     → BACKGROUND: SIL calendar (just in case)
                                     
User: "...someone called in          → CONFIRMED: sick coverage
      sick for tomorrow"             → Data already loaded!
                                     → Inject: shifts, available staff, quals
                                     
Reggie: [Instant response with       → Log: sarah + roster + sick_coverage
        full coverage options]       → Update pattern weights
                                     → Prediction accuracy: +1

---

2. Architecture

2.1 System Overview

┌─────────────────────────────────────────────────────────────────┐
│                     USER CONVERSATION                            │
│                                                                  │
│   User  ←──────────────────→  Reggie (Main LLM)                 │
│                                      ↑                          │
│                                      │ context injection        │
│                                      │ (each turn)              │
└──────────────────────────────────────┼──────────────────────────┘
                                       │
┌──────────────────────────────────────┼──────────────────────────┐
│              CONTEXT ANTICIPATION ENGINE                         │
│                                      │                          │
│  ┌───────────────┐    ┌──────────────┴───────────┐              │
│  │ Conversation  │    │      Context Cache       │              │
│  │   Monitor     │    │      (hot data)          │              │
│  └───────┬───────┘    └──────────────────────────┘              │
│          │                         ↑                            │
│          ↓                         │                            │
│  ┌───────────────┐    ┌────────────┴─────────────┐              │
│  │    Topic      │    │     Pre-fetch Queue      │              │
│  │  Detector     │    │                          │              │
│  └───────┬───────┘    └──────────────────────────┘              │
│          │                         ↑                            │
│          ↓                         │                            │
│  ┌───────────────┐    ┌────────────┴─────────────┐              │
│  │   Pattern     │───→│   Direction Predictor    │              │
│  │   Matcher     │    │                          │              │
│  └───────────────┘    └──────────────────────────┘              │
│          ↑                                                      │
│          │                                                      │
│  ┌───────┴───────────────────────────────────────────────┐     │
│  │              USER PATTERN DATABASE                     │     │
│  │  (learned from historical conversations)               │     │
│  └───────────────────────────────────────────────────────┘     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

2.2 Component Responsibilities

Component	Job
Conversation Monitor	Watches each turn, extracts signals
Topic Detector	Classifies conversation into topic(s)
Pattern Matcher	Looks up user's historical patterns for this topic
Direction Predictor	Predicts likely conversation directions
Pre-fetch Queue	Prioritizes and executes data fetches
Context Cache	Holds pre-fetched data ready for injection
User Pattern Database	Stores learned patterns per user per topic

---

3. User Pattern Learning

3.1 The Key Insight

Different users have different patterns for the same topic:

USER: Sarah (Team Leader)
TOPIC: Roster
─────────────────────────────────────────────────────
PATTERN ANALYSIS (from 58 historical conversations):

├── 80% (47 convos) → Sick leave coverage
│   ├── Typical flow: "roster" → "someone sick" → "need cover"
│   ├── Data needed: Next 48hrs shifts, available staff, quals
│   └── Resolution: Staff suggestions, contact details
│
├── 10% (6 convos) → SIL resident scheduling
│   ├── Typical flow: "roster" → "resident" → "appointment/visit"
│   ├── Data needed: SIL medical calendar, resident routines
│   └── Resolution: Schedule adjustments, notifications
│
├── 7% (4 convos) → Leave approvals
│   ├── Typical flow: "roster" → "leave request" → "approve?"
│   ├── Data needed: Pending requests, coverage impact
│   └── Resolution: Approval recommendation
│
└── 3% (1 convo) → Other
    └── Wait for more signal


USER: Mike (Support Worker)  
TOPIC: Roster
─────────────────────────────────────────────────────
PATTERN ANALYSIS (from 35 historical conversations):

├── 60% (21 convos) → Checking own shifts
│   ├── Typical flow: "roster" → "my shifts" → time period
│   ├── Data needed: Mike's schedule, pending changes
│   └── Resolution: Schedule display
│
├── 30% (11 convos) → Swap requests
│   ├── Typical flow: "roster" → "swap" → specific shift
│   ├── Data needed: Who's on target shifts, swap policy
│   └── Resolution: Swap options, request submission
│
└── 10% (3 convos) → Availability updates
    ├── Typical flow: "roster" → "availability" → changes
    ├── Data needed: Current availability settings
    └── Resolution: Update confirmation

3.2 Pattern Database Schema

CREATE TABLE chat.user_topic_patterns (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID NOT NULL REFERENCES admin.users(id),
  
  -- Topic classification
  topic VARCHAR(50) NOT NULL,           -- roster, payroll, leave, etc.
  
  -- Direction/intent classification  
  direction VARCHAR(100) NOT NULL,      -- sick_coverage, check_shifts, etc.
  
  -- Statistics
  occurrence_count INTEGER DEFAULT 1,
  last_occurrence TIMESTAMPTZ DEFAULT NOW(),
  first_occurrence TIMESTAMPTZ DEFAULT NOW(),
  
  -- Confidence tracking
  prediction_attempts INTEGER DEFAULT 0,
  prediction_successes INTEGER DEFAULT 0,
  
  -- What data was typically needed
  typical_data_needs JSONB DEFAULT '[]',
  
  -- Average resolution path
  typical_resolution TEXT,
  avg_turns_to_resolution NUMERIC(4,1),
  
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW(),
  
  UNIQUE(user_id, topic, direction)
);

CREATE INDEX idx_user_topic_patterns_lookup 
  ON chat.user_topic_patterns(user_id, topic);
CREATE INDEX idx_user_topic_patterns_recent 
  ON chat.user_topic_patterns(last_occurrence DESC);

3.3 Pattern Update Flow

After every conversation:

async function updatePatterns(conversation) {
  const { userId, detectedTopic, actualDirection, datUsed, turnsToResolve } = 
    analyzeConversation(conversation);
  
  await db.query(`
    INSERT INTO chat.user_topic_patterns 
      (user_id, topic, direction, occurrence_count, typical_data_needs)
    VALUES ($1, $2, $3, 1, $4)
    ON CONFLICT (user_id, topic, direction) DO UPDATE SET
      occurrence_count = user_topic_patterns.occurrence_count + 1,
      last_occurrence = NOW(),
      typical_data_needs = merge_data_needs(
        user_topic_patterns.typical_data_needs, 
        $4
      ),
      avg_turns_to_resolution = (
        user_topic_patterns.avg_turns_to_resolution * 
        user_topic_patterns.occurrence_count + $5
      ) / (user_topic_patterns.occurrence_count + 1),
      updated_at = NOW()
  `, [userId, detectedTopic, actualDirection, dataUsed, turnsToResolve]);
}

---

4. Topic Detection

4.1 Topic Taxonomy

Primary topics (detect from keywords + context):

Topic	Trigger Signals	Sub-topics
roster	roster, shift, schedule, working, on tomorrow	coverage, swaps, availability
payroll	pay, payslip, money, hours, timesheet	queries, errors, deductions
leave	leave, holiday, time off, annual, sick	requests, balance, approval
participants	[participant name], client, resident, SIL	care, incidents, schedules
vehicles	car, van, vehicle, keys, booking	bookings, issues, locations
training	training, course, certificate, qualified	requirements, bookings, expiry
incidents	incident, accident, injury, report	reporting, followup, review
hr	contract, employment, policy, procedure	queries, complaints, updates
general	(no specific topic detected)	information, chitchat

4.2 Detection Strategy

async function detectTopic(message, conversationHistory, userId) {
  // 1. Quick keyword scan
  const keywordTopics = scanKeywords(message);
  
  // 2. If ambiguous, use LLM classification
  if (keywordTopics.length !== 1) {
    const llmTopic = await classifyWithLLM(message, conversationHistory);
    return llmTopic;
  }
  
  // 3. Consider user's common topics (prior probability)
  const userTopics = await getUserTopTopics(userId);
  
  // 4. Return highest confidence topic
  return mergeConfidences(keywordTopics, userTopics);
}

---

5. Direction Prediction

5.1 Prediction Algorithm

async function predictDirection(userId, topic, conversationSignals) {
  // 1. Load user's patterns for this topic
  const patterns = await db.query(`
    SELECT direction, occurrence_count, last_occurrence,
           typical_data_needs, prediction_successes, prediction_attempts
    FROM chat.user_topic_patterns
    WHERE user_id = $1 AND topic = $2
    ORDER BY occurrence_count DESC
  `, [userId, topic]);
  
  // 2. Calculate probabilities
  const total = patterns.reduce((sum, p) => sum + p.occurrence_count, 0);
  const predictions = patterns.map(p => ({
    direction: p.direction,
    probability: p.occurrence_count / total,
    accuracy: p.prediction_attempts > 0 
      ? p.prediction_successes / p.prediction_attempts 
      : 0.5, // assume 50% if no data
    recency_boost: daysSince(p.last_occurrence) < 7 ? 0.1 : 0,
    data_needs: p.typical_data_needs
  }));
  
  // 3. Apply signal boosting (if message has specific keywords)
  for (const pred of predictions) {
    if (signalMatchesDirection(conversationSignals, pred.direction)) {
      pred.probability *= 1.5; // boost matching directions
    }
  }
  
  // 4. Normalize and return ranked predictions
  return normalizeAndRank(predictions);
}

5.2 Signal Extraction

From each message, extract signals that hint at direction:

function extractSignals(message) {
  return {
    mentions_self: /\b(my|i|me)\b/i.test(message),
    mentions_other: /\b(someone|they|staff|team)\b/i.test(message),
    mentions_time: extractTimeReferences(message),  // tomorrow, next week, etc.
    mentions_problem: /\b(sick|issue|problem|can't|unable)\b/i.test(message),
    mentions_request: /\b(can i|could i|want to|need to)\b/i.test(message),
    mentions_query: /\b(what|when|where|who|how)\b/i.test(message),
    question_mark: message.includes('?'),
    entities_mentioned: extractEntities(message)  // names, places, etc.
  };
}

---

6. Pre-fetch Strategies

6.1 Confidence-Based Actions

Confidence	Action	Example
>80%	Fetch immediately, inject proactively	Sarah + roster → sick coverage data
60-80%	Fetch in background, ready to inject on confirmation	Have it ready, wait for signal
40-60%	Prepare query, don't execute	Know what to fetch, wait
<40%	Wait for more signal	Too uncertain, don't waste resources

6.2 Pre-fetch Registry

Define what data each direction needs:

const PREFETCH_REGISTRY = {
  roster: {
    sick_coverage: {
      priority: 'immediate',
      fetches: [
        { name: 'shifts_48hrs', query: 'getShiftsNextHours(48)' },
        { name: 'available_staff', query: 'getAvailableStaff()' },
        { name: 'quals_matrix', query: 'getQualificationsMatrix()' },
        { name: 'recent_sickness', query: 'getRecentSickLeave(7days)' }
      ]
    },
    sil_scheduling: {
      priority: 'background',
      fetches: [
        { name: 'sil_calendar', query: 'getSILMedicalCalendar(14days)' },
        { name: 'resident_routines', query: 'getResidentRoutines()' },
        { name: 'sil_staff', query: 'getSILRosteredStaff(14days)' }
      ]
    },
    check_own_shifts: {
      priority: 'immediate',
      fetches: [
        { name: 'user_schedule', query: 'getUserSchedule(userId, 14days)' },
        { name: 'pending_swaps', query: 'getPendingSwaps(userId)' },
        { name: 'user_leave', query: 'getUserLeaveBalance(userId)' }
      ]
    },
    swap_request: {
      priority: 'immediate',
      fetches: [
        { name: 'user_schedule', query: 'getUserSchedule(userId, 14days)' },
        { name: 'swap_policy', query: 'getSwapPolicy()' },
        { name: 'potential_swapees', query: 'getStaffOnUserShifts(userId)' }
      ]
    }
  },
  
  payroll: {
    payslip_query: {
      priority: 'immediate',
      fetches: [
        { name: 'recent_payslips', query: 'getPayslips(userId, 3)' },
        { name: 'pay_schedule', query: 'getPaySchedule()' }
      ]
    },
    timesheet_issue: {
      priority: 'immediate',
      fetches: [
        { name: 'recent_timesheets', query: 'getTimesheets(userId, 14days)' },
        { name: 'timesheet_status', query: 'getTimesheetApprovalStatus(userId)' },
        { name: 'hr_contact', query: 'getHRContactInfo()' }
      ]
    }
  }
  // ... more topics
};

6.3 Pre-fetch Execution

async function executePrefetch(predictions, userId, contextCache) {
  for (const pred of predictions) {
    const registry = PREFETCH_REGISTRY[pred.topic]?.[pred.direction];
    if (!registry) continue;
    
    if (pred.probability > 0.8 && registry.priority === 'immediate') {
      // Fetch immediately, block if needed
      await fetchAllImmediate(registry.fetches, userId, contextCache);
    } 
    else if (pred.probability > 0.6) {
      // Fetch in background
      fetchAllBackground(registry.fetches, userId, contextCache);
    }
    else if (pred.probability > 0.4) {
      // Just prepare the queries
      contextCache.preparedQueries.push(...registry.fetches);
    }
  }
}

---

7. Context Cache

7.1 Cache Structure

class ContextCache {
  constructor(sessionId, userId) {
    this.sessionId = sessionId;
    this.userId = userId;
    this.data = new Map();          // fetched data
    this.pending = new Set();       // in-flight fetches
    this.preparedQueries = [];      // ready-to-execute queries
    this.injectionQueue = [];       // priority-ordered for injection
    this.ttl = new Map();           // expiry times
  }
  
  async get(key) {
    if (this.data.has(key)) {
      return this.data.get(key);
    }
    if (this.pending.has(key)) {
      // Wait for in-flight fetch
      return this.waitFor(key);
    }
    return null;
  }
  
  set(key, value, ttlSeconds = 300) {
    this.data.set(key, value);
    this.ttl.set(key, Date.now() + ttlSeconds * 1000);
    this.pending.delete(key);
  }
  
  getContextForInjection(maxTokens = 2000) {
    // Return highest priority items that fit in token budget
    return prioritizeAndTruncate(this.injectionQueue, maxTokens);
  }
}

7.2 Cache Lifecycle

Session Start
    │
    ├── Load user profile
    ├── Load recent conversation summaries
    └── Pre-warm common data for user's top topics
    
Each Turn
    │
    ├── Topic detected? → Trigger prediction → Pre-fetch
    ├── Direction confirmed? → Promote cache items to injection queue
    └── Turn complete → Update patterns, adjust predictions
    
Session End
    │
    ├── Analyze conversation flow
    ├── Update user_topic_patterns
    ├── Log prediction accuracy
    └── Clear cache

---

8. Integration with Embeddings

8.1 Where Embeddings Help

Use Case	How Embeddings Help
Cross-user pattern discovery	"Other users who talked about X often needed Y"
Similar conversation lookup	Find past convos with similar opening to prime predictions
Topic clustering	Group conversations to discover new patterns
Knowledge retrieval	Once topic/direction known, fetch relevant KB content

8.2 Cross-User Learning

async function findSimilarConversationStarts(currentMessage, topic) {
  // Embed the current conversation opening
  const embedding = await embed(currentMessage);
  
  // Find similar openings from OTHER users
  const similar = await db.query(`
    SELECT DISTINCT user_id, direction, COUNT(*) as occurrences
    FROM chat.conversation_logs cl
    JOIN chat.user_topic_patterns utp ON utp.user_id = cl.user_id
    WHERE cl.topic = $1
      AND cl.turn_number <= 2
      AND 1 - (cl.embedding <=> $2) > 0.8
      AND cl.user_id != $3  -- exclude current user
    GROUP BY user_id, direction
    ORDER BY occurrences DESC
    LIMIT 5
  `, [topic, embedding, currentUserId]);
  
  // Use this to boost predictions for directions we haven't seen from this user
  return similar;
}

8.3 Pattern Discovery

Monthly/weekly job to discover new patterns:

async function discoverNewPatterns() {
  // Cluster recent conversations by embedding similarity
  const clusters = await clusterConversations(last30Days);
  
  for (const cluster of clusters) {
    // Check if this represents a pattern not in our taxonomy
    const directions = cluster.conversations.map(c => c.direction);
    const uniqueDirections = new Set(directions);
    
    if (uniqueDirections.size === 1 && cluster.size > 10) {
      // Strong cluster around one direction - this is a valid pattern
      log(`Pattern confirmed: ${cluster.topic} → ${[...uniqueDirections][0]}`);
    } else if (cluster.size > 10 && !cluster.topic) {
      // Large cluster with no topic - might be new topic to add
      alert(`Potential new topic discovered: ${summarizeCluster(cluster)}`);
    }
  }
}

---

9. Accuracy Tracking

9.1 Prediction Metrics

Track per user, per topic:

CREATE TABLE chat.prediction_log (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  session_id UUID NOT NULL,
  user_id UUID NOT NULL,
  
  -- What we predicted
  topic VARCHAR(50),
  predicted_direction VARCHAR(100),
  prediction_confidence NUMERIC(3,2),
  predicted_at TIMESTAMPTZ DEFAULT NOW(),
  
  -- What actually happened
  actual_direction VARCHAR(100),
  was_correct BOOLEAN,
  confirmed_at TIMESTAMPTZ,
  
  -- Pre-fetch effectiveness
  data_prefetched JSONB,
  data_actually_used JSONB,
  prefetch_hit_rate NUMERIC(3,2),  -- % of prefetched data that was used
  
  -- Latency impact
  response_time_ms INTEGER,
  would_be_time_without_prefetch_ms INTEGER  -- estimated
);

9.2 Dashboard Metrics

Metric	Target	Description
Prediction Accuracy	>75%	How often top prediction is correct
Pre-fetch Hit Rate	>60%	How often pre-fetched data is used
Latency Improvement	>50%	Response time saved vs no pre-fetch
Coverage	>90%	% of conversations with successful topic detection

---

10. Implementation Phases

Phase 1: Foundation (Week 1-2)

• Create pattern database schema
• Build topic detector
• Implement basic pattern matching
• Build context cache

Phase 2: Learning Loop (Week 3-4)

• Implement pattern update after conversations
• Build direction predictor
• Create pre-fetch registry for top topics
• Add confidence-based fetch strategies

Phase 3: Integration (Week 5-6)

• Integrate with Reggie's turn processing
• Build context injection pipeline
• Add shadow agent monitoring
• Test with real conversations

Phase 4: Optimization (Week 7-8)

• Add cross-user learning
• Tune prediction weights
• Add embedding-based pattern discovery
• Build accuracy dashboard

Phase 5: Refinement (Ongoing)

• Expand pre-fetch registry
• Tune confidence thresholds
• Add new topics as discovered
• A/B test prediction strategies

---

11. Example: Full Flow

═══════════════════════════════════════════════════════════════════
                    COMPLETE EXAMPLE FLOW
═══════════════════════════════════════════════════════════════════

USER: Sarah (Team Leader)
SESSION START: 2026-01-21 08:15:00

[08:15:00] SHADOW AGENT:
  → Session started for Sarah
  → Loading user profile...
  → Sarah's top topics: roster (65%), participants (20%), payroll (10%)
  → Pre-warming common roster data (just in case)
  
[08:15:01] USER: "Morning Reggie!"

[08:15:01] SHADOW AGENT:
  → Greeting detected, no topic signal yet
  → Waiting...
  
[08:15:02] REGGIE: "Good morning Sarah! How can I help you today?"

[08:15:05] USER: "Got a bit of a roster problem"

[08:15:05] SHADOW AGENT:
  → TOPIC DETECTED: roster (confidence: 95%)
  → Loading Sarah's roster patterns...
  → Pattern analysis:
      - sick_coverage: 80% (47 occurrences)
      - sil_scheduling: 10% (6 occurrences)  
      - leave_approval: 7% (4 occurrences)
      - other: 3%
  → PREDICTION: sick_coverage (80% confidence)
  → ACTION: IMMEDIATE PRE-FETCH
      ✓ Fetching shifts next 48 hours...
      ✓ Fetching available staff...
      ✓ Fetching qualifications matrix...
  → BACKGROUND: SIL calendar (just in case)
  
[08:15:06] REGGIE: "Sure, what's happening with the roster?"

[08:15:06] SHADOW AGENT:
  → Pre-fetch complete (1.2 seconds)
  → Context cache loaded:
      - 12 shifts in next 48hrs
      - 8 available staff identified
      - Quals matrix ready
  → Ready for injection on next turn

[08:15:10] USER: "Dave called in sick, he was supposed to cover 
                 the afternoon shift at Parkview"

[08:15:10] SHADOW AGENT:
  → CONFIRMED: sick_coverage direction
  → Prediction was CORRECT ✓
  → Entity detected: Dave (staff), Parkview (venue)
  → Shift identified: Today afternoon @ Parkview
  → Filtering pre-fetched data...
  → INJECTING CONTEXT:
      - Shift details: 2pm-10pm, Parkview, 2 participants
      - Required quals: Manual handling, Medication
      - Available staff with quals: Jane, Mike, Tom
      - Jane: already worked 38hrs this week
      - Mike: available, 30hrs, has all quals
      - Tom: available, 25hrs, has all quals

[08:15:11] REGGIE: "I've found the shift - Dave's 2pm-10pm at 
                   Parkview today. I can see Mike and Tom are both 
                   available and have the right qualifications. 
                   Mike has done more shifts at Parkview recently. 
                   Want me to reach out to them?"

[08:15:11] SHADOW AGENT:
  → Response generated with pre-fetched context
  → Response time: 800ms (estimated 3500ms without pre-fetch)
  → Latency saved: ~75%

[08:15:15] USER: "Yes, try Mike first"

[08:15:15] SHADOW AGENT:
  → Action request detected
  → Preparing Mike contact flow...

... conversation continues ...

[08:18:00] SESSION END

[08:18:01] SHADOW AGENT:
  → Conversation analysis complete
  → Updating patterns:
      - Sarah + roster + sick_coverage: now 48 occurrences
      - Prediction accuracy: +1 success
      - Data used: shifts, available_staff, quals (all prefetched!)
      - Pre-fetch hit rate: 100%
  → Logging to prediction_log
  → Session metrics saved
  
═══════════════════════════════════════════════════════════════════

---

Changelog

Date	Change	Author
2026-01-21	Initial document created	Droid