Dialpad Integration Architecture

Real-time call tracking, local caching, YP3000 identity enrichment, and live ticker notifications.

---

Overview

The Dialpad integration provides comprehensive phone system visibility through a hybrid push/pull architecture:

• Push (Webhooks): Real-time call events for live notifications
• Pull (Cron): Periodic sync for historical data and enrichment
• Local Cache: Fast queries without API rate limits
• Identity Mining: Automatic YP3000 enrichment from call data and transcripts

---

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         DIALPAD INTEGRATION                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  PUSH (Real-time via Webhooks)          PULL (Cron Sync)            │
│  ┌─────────────────────────┐           ┌─────────────────────────┐  │
│  │ Dialpad Webhook/Zapier  │           │ Scheduled Cron Job      │  │
│  │ • call.started          │           │ • Sync last 24-48h      │  │
│  │ • call.ended            │           │ • Backfill missing      │  │
│  │ • call.missed           │           │ • Fetch transcripts     │  │
│  │ • voicemail.received    │           │ • Fetch recordings meta │  │
│  └───────────┬─────────────┘           └───────────┬─────────────┘  │
│              │                                     │                 │
│              ▼                                     ▼                 │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │                       LOCAL DATABASE                             ││
│  │                                                                  ││
│  │  comms.dialpad_calls                                       ││
│  │  ├── call_id, direction, state, duration                        ││
│  │  ├── from_number, from_name, from_identity_id                   ││
│  │  ├── to_number, to_name, to_identity_id                         ││
│  │  ├── transcript_text, recording_url                             ││
│  │  └── yp3000_enriched_at, created_at                             ││
│  │                                                                  ││
│  │  comms.dialpad_identities                                  ││
│  │  ├── phone_number (unique)                                      ││
│  │  ├── dialpad_contact_id, dialpad_name                           ││
│  │  ├── yp3000_identity_id (FK)                                    ││
│  │  └── auto_created, confidence_score                             ││
│  │                                                                  ││
│  └─────────────────────────────────────────────────────────────────┘│
│              │                                     │                 │
│              ▼                                     ▼                 │
│  ┌─────────────────────────┐           ┌─────────────────────────┐  │
│  │ LIVE: Ticker + SSE      │           │ HISTORY: Phone Log Page │  │
│  │                         │           │                         │  │
│  │ "📞 Incoming call:      │           │ • Fast DB queries       │  │
│  │  John Smith → Staff     │           │ • Full-text search      │  │
│  │  Enquiries"             │           │ • Filter by date/type   │  │
│  │                         │           │ • YP3000 enriched names │  │
│  │ Real-time broadcast     │           │ • Click for details     │  │
│  └─────────────────────────┘           └─────────────────────────┘  │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

---

Data Flow

1. Real-time Events (Push)

Dialpad Call Event
       │
       ▼ (Webhook POST)
┌──────────────────────────────────────┐
│  POST /api/v1/dialpad/webhook        │
│  ├── Validate webhook signature      │
│  ├── Parse event type                │
│  ├── Upsert to dialpad_calls         │
│  ├── Quick YP3000 lookup (cached)    │
│  ├── Broadcast SSE to clients        │
│  └── Queue enrichment job (async)    │
└──────────────────────────────────────┘
       │
       ▼
┌──────────────────────────────────────┐
│  SSE Broadcast                       │
│  ├── Header ticker animation         │
│  ├── Phone log page live update      │
│  └── Desktop notification (optional) │
└──────────────────────────────────────┘

2. Historical Sync (Pull)

Cron Job (every 15 min or nightly)
       │
       ▼
┌──────────────────────────────────────┐
│  Dialpad API: GET /call              │
│  ├── Paginate through recent calls   │
│  ├── Limit to 500 calls per run      │
│  └── Track last_synced_at cursor     │
└──────────────────────────────────────┘
       │
       ▼
┌──────────────────────────────────────┐
│  For each call:                      │
│  ├── Upsert to dialpad_calls         │
│  ├── Check dialpad_identities cache  │
│  ├── If unknown: lookup YP3000       │
│  ├── If still unknown: queue mining  │
│  └── Update enrichment timestamp     │
└──────────────────────────────────────┘

3. Identity Enrichment Pipeline

Unknown Phone Number
       │
       ▼
┌──────────────────────────────────────┐
│  Step 1: Check Local Cache           │
│  SELECT * FROM dialpad_identities    │
│  WHERE phone_number = $1             │
└──────────────────────────────────────┘
       │ (miss)
       ▼
┌──────────────────────────────────────┐
│  Step 2: YP3000 Lookup               │
│  Search by phone in yp3000_contacts  │
│  └── If found: link identity_id      │
└──────────────────────────────────────┘
       │ (miss)
       ▼
┌──────────────────────────────────────┐
│  Step 3: Dialpad Contact Check       │
│  Use Dialpad's contact.name if set   │
│  └── Store as provisional identity   │
└──────────────────────────────────────┘
       │ (miss)
       ▼
┌──────────────────────────────────────┐
│  Step 4: Transcript Mining           │
│  Parse transcript for:               │
│  ├── "Hi this is [NAME] from [ORG]"  │
│  ├── "My name is [NAME]"             │
│  ├── Email addresses mentioned       │
│  └── Create candidate identity       │
└──────────────────────────────────────┘
       │
       ▼
┌──────────────────────────────────────┐
│  Store in dialpad_identities         │
│  ├── phone_number                    │
│  ├── resolved_name                   │
│  ├── source: 'yp3000'|'dialpad'|     │
│  │           'transcript'|'manual'   │
│  ├── confidence: 0.0 - 1.0           │
│  └── yp3000_identity_id (if linked)  │
└──────────────────────────────────────┘

---

Database Schema

comms.dialpad_calls

CREATE TABLE comms.dialpad_calls (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  
  -- Dialpad identifiers
  dialpad_call_id TEXT UNIQUE NOT NULL,
  master_call_id TEXT,
  entry_point_call_id TEXT,
  
  -- Call details
  direction TEXT NOT NULL, -- 'inbound', 'outbound'
  state TEXT NOT NULL,     -- 'ringing', 'connected', 'missed', 'voicemail'
  
  -- Timestamps (stored as TIMESTAMPTZ, converted from Dialpad ms)
  date_started TIMESTAMPTZ,
  date_rang TIMESTAMPTZ,
  date_connected TIMESTAMPTZ,
  date_ended TIMESTAMPTZ,
  duration_ms INTEGER DEFAULT 0,
  
  -- From party
  from_number TEXT,
  from_name TEXT,           -- From Dialpad contact or YP3000
  from_dialpad_contact_id TEXT,
  from_yp3000_identity_id UUID REFERENCES core_identity.yp3000_identities(id),
  
  -- To party  
  to_number TEXT,
  to_name TEXT,
  to_dialpad_user_id TEXT,
  to_dialpad_target_type TEXT, -- 'user', 'department', 'callcenter'
  to_yp3000_identity_id UUID REFERENCES core_identity.yp3000_identities(id),
  
  -- Entry point (for routed calls)
  entry_point_name TEXT,    -- 'Staff Enquiries', 'Customer Service'
  entry_point_type TEXT,
  
  -- Recording & transcript
  has_recording BOOLEAN DEFAULT false,
  recording_url TEXT,
  transcript_text TEXT,
  transcript_parsed_at TIMESTAMPTZ,
  
  -- AI analysis
  ai_summary TEXT,
  ai_sentiment TEXT,        -- 'positive', 'neutral', 'negative'
  ai_topics TEXT[],
  
  -- Enrichment tracking
  yp3000_enriched_at TIMESTAMPTZ,
  transcript_mined_at TIMESTAMPTZ,
  
  -- Metadata
  raw_data JSONB,           -- Full Dialpad response for debugging
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for common queries
CREATE INDEX idx_dialpad_calls_date ON comms.dialpad_calls(date_started DESC);
CREATE INDEX idx_dialpad_calls_from ON comms.dialpad_calls(from_number);
CREATE INDEX idx_dialpad_calls_to ON comms.dialpad_calls(to_number);
CREATE INDEX idx_dialpad_calls_state ON comms.dialpad_calls(state);
CREATE INDEX idx_dialpad_calls_direction ON comms.dialpad_calls(direction);
CREATE INDEX idx_dialpad_calls_unenriched ON comms.dialpad_calls(yp3000_enriched_at) 
  WHERE yp3000_enriched_at IS NULL;

comms.dialpad_identities

CREATE TABLE comms.dialpad_identities (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  
  -- Phone number (E.164 format preferred)
  phone_number TEXT UNIQUE NOT NULL,
  phone_normalized TEXT,    -- Stripped to digits only
  
  -- Dialpad's info
  dialpad_contact_id TEXT,
  dialpad_name TEXT,
  dialpad_email TEXT,
  dialpad_company TEXT,
  
  -- YP3000 link
  yp3000_identity_id UUID REFERENCES core_identity.yp3000_identities(id),
  
  -- Resolved display info
  display_name TEXT,        -- Best name to show
  organization TEXT,
  
  -- Source tracking
  source TEXT NOT NULL,     -- 'yp3000', 'dialpad', 'transcript', 'manual'
  confidence FLOAT DEFAULT 0.5,
  
  -- For transcript-mined identities
  mined_from_call_id UUID REFERENCES comms.dialpad_calls(id),
  mined_phrases TEXT[],     -- "Hi this is Sarah from NDIS"
  
  -- Metadata
  auto_created BOOLEAN DEFAULT true,
  verified_by UUID,         -- User who verified/corrected
  verified_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_dialpad_identities_phone ON comms.dialpad_identities(phone_normalized);
CREATE INDEX idx_dialpad_identities_yp3000 ON comms.dialpad_identities(yp3000_identity_id);
CREATE INDEX idx_dialpad_identities_unlinked ON comms.dialpad_identities(yp3000_identity_id) 
  WHERE yp3000_identity_id IS NULL;

---

API Endpoints

Webhook (Real-time)

Method	Endpoint	Description
POST	/api/v1/dialpad/webhook	Receive Dialpad/Zapier events

Webhook Payload (from Zapier):

{
  "event": "call.ended",
  "call_id": "5965371398168576",
  "direction": "inbound",
  "state": "connected",
  "from_number": "+61481140711",
  "from_name": "John Smith",
  "to_number": "+61287631474",
  "to_name": "Jennifer Albert",
  "duration_ms": 245000,
  "has_recording": true,
  "timestamp": "2026-01-07T14:30:00.000Z"
}

Phone Log (Local DB)

Method	Endpoint	Description
GET	/api/v1/phone-log	List calls from local DB
GET	/api/v1/phone-log/:id	Get call details
GET	/api/v1/phone-log/:id/transcript	Get transcript (PIN protected)
GET	/api/v1/phone-log/:id/recording	Get recording URL (PIN protected)
POST	/api/v1/phone-log/:id/link-identity	Link call to YP3000 identity

Sync & Admin

Method	Endpoint	Description
POST	/api/v1/dialpad/sync	Trigger manual sync
GET	/api/v1/dialpad/sync/status	Check last sync time
POST	/api/v1/dialpad/enrich	Run YP3000 enrichment

---

Cron Jobs

1. Call Sync (Every 15 minutes)

// cron: */15 * * * *
async function syncRecentCalls() {
  const lastSync = await getLastSyncTime();
  const calls = await dialpad.getCalls({ 
    started_after: lastSync,
    limit: 200 
  });
  
  for (const call of calls) {
    await upsertCall(call);
    await quickEnrich(call); // Cached YP3000 lookup
  }
  
  await setLastSyncTime(Date.now());
}

2. Deep Enrichment (Nightly at 2am)

// cron: 0 2 * * *
async function deepEnrichment() {
  // Find calls without YP3000 links
  const unenriched = await db.query(`
    SELECT * FROM comms.dialpad_calls 
    WHERE yp3000_enriched_at IS NULL
    AND date_started > NOW() - INTERVAL '7 days'
    LIMIT 500
  `);
  
  for (const call of unenriched) {
    await enrichFromYP3000(call);
    await enrichFromTranscript(call);
  }
}

3. Transcript Mining (Nightly at 3am)

// cron: 0 3 * * *
async function mineTranscripts() {
  const calls = await db.query(`
    SELECT * FROM comms.dialpad_calls
    WHERE transcript_text IS NOT NULL
    AND transcript_mined_at IS NULL
    LIMIT 100
  `);
  
  for (const call of calls) {
    const identities = await extractIdentities(call.transcript_text);
    for (const identity of identities) {
      await createCandidateIdentity(identity, call.id);
    }
  }
}

---

Live Ticker Integration

SSE Event Format

// Broadcast to connected clients
sse.broadcast('dialpad:call', {
  type: 'call.started', // or 'call.ended', 'call.missed'
  call_id: '5965371398168576',
  direction: 'inbound',
  from: {
    number: '+61481140711',
    name: 'John Smith',        // Resolved name
    organization: 'NDIS'       // If known
  },
  to: {
    number: '+61287631474', 
    name: 'Jennifer Albert',
    department: 'Staff Enquiries'
  },
  timestamp: '2026-01-07T14:30:00.000Z'
});

Ticker Display

┌──────────────────────────────────────────────────────────────────┐
│ 📞 Incoming: John Smith (NDIS) → Staff Enquiries                 │
└──────────────────────────────────────────────────────────────────┘

---

Transcript Identity Mining

Pattern Matching

const IDENTITY_PATTERNS = [
  // "Hi this is Sarah from NDIS"
  /(?:hi|hello|hey)[\s,]+(?:this is|it's|i'm)\s+(\w+)\s+(?:from|at|with)\s+(.+?)(?:\.|,|$)/i,
  
  // "My name is John Smith"
  /my name is\s+(\w+(?:\s+\w+)?)/i,
  
  // "This is [Name] calling"
  /this is\s+(\w+(?:\s+\w+)?)\s+calling/i,
  
  // "Speaking with [Name]"
  /speaking with\s+(\w+(?:\s+\w+)?)/i,
  
  // Email mentions
  /\b([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})\b/g
];

function extractIdentities(transcript) {
  const candidates = [];
  
  for (const pattern of IDENTITY_PATTERNS) {
    const matches = transcript.matchAll(pattern);
    for (const match of matches) {
      candidates.push({
        name: match[1],
        organization: match[2] || null,
        confidence: 0.6,
        source_phrase: match[0]
      });
    }
  }
  
  return candidates;
}

LLM Enhancement (Optional)

For higher accuracy, use GPT to extract structured identity info:

async function llmExtractIdentities(transcript) {
  const response = await openai.chat({
    model: 'gpt-4.1-nano',
    messages: [{
      role: 'system',
      content: `Extract any person identities mentioned in this call transcript.
                Return JSON array: [{name, organization, role, confidence}]`
    }, {
      role: 'user', 
      content: transcript
    }]
  });
  
  return JSON.parse(response.content);
}

---

Performance Considerations

Why Local Caching?

Approach	Page Load Time	API Calls	Rate Limit Risk
Direct Dialpad API	5-30 seconds	10-20 per load	High
Local DB Cache	50-200ms	0	None

Sync Strategy

• Real-time webhook: Instant for live calls
• 15-min sync: Catches any missed webhooks
• Nightly deep sync: Full enrichment, transcript mining

Indexes

All common query patterns are indexed:

• Date range queries (date_started DESC)
• Phone number lookups (from_number, to_number)
• State/direction filters
• Unenriched records for batch processing

---

Future Enhancements

1. Voicemail Transcription - Auto-transcribe voicemails
2. Sentiment Analysis - Track caller satisfaction trends
3. Call Analytics Dashboard - Volume, duration, missed rate by department
4. Smart Routing Suggestions - ML-based call routing optimization
5. Participant Call History - Show calls on participant detail page
6. Staff Performance Metrics - Calls handled, avg duration, etc.

---

Related Documentation

• YP3000 Identity System - Identity resolution
• Ticker Announcements - Live notifications
• 05_Getting_Started_with_Agents.md - Background jobs