📧 Email System Architecture

Overview

The RABS email system is a multi-service architecture built around IMAP connectivity, background queue processing, and a message cache layer. All state is stored in the admin_mail schema on DB2. The system separates concerns across six backend services, each with a distinct responsibility, coordinated through database-backed queues and a shared message cache.

Backend Services

Six Node.js services compose the email backend:

Service	File	Purpose
IMAP Service	email-imap.js	Core IMAP operations — connect, fetch, search, stream
Cache Warmer	email-cache-warmer.js	Background cache warming for configured mailboxes
Mail Ops Worker	email-mail-ops-worker.js	Async delete, move, and flag operations via queue
Mail Sync Worker	email-mail-sync-worker.js	Background folder synchronisation
Outgoing Worker	email-outgoing-worker.js	SMTP sending via outgoing mail queue
Mailbox Service	email-mailboxes.js	Mailbox CRUD, schema validation, health tracking

All services read from and write to tables in the admin_mail schema.

API Endpoints

IMAP Routes (backend/routes_v1p/email-imap.js)

Method	Endpoint	Description
GET	/accounts	List all configured email accounts
GET	/accounts/:id/mailboxes	List mailboxes (folders) for an account
GET	/accounts/:id/messages	Fetch message list for an account/folder
GET	/accounts/:id/messages/:uid	Fetch a single message by UID
DELETE	/accounts/:id/messages/:uid	Delete a message (?background=true for async)
PATCH	/accounts/:id/messages/:uid/flags	Update message flags (read, starred, etc.)
POST	/accounts/:id/mailbox/:mailbox/resync	Force resync a specific mailbox folder
POST	/accounts/:id/send	Send an email via SMTP
GET	/cache-stats	Cache storage and hit-rate statistics
GET	/queue-stats	Mail operations queue statistics
GET	/system-mailbox-health	Health status of all configured mailboxes

Me Email Routes (me-email.js)

Method	Endpoint	Description
GET	/	Get current user's email preferences
PUT	/	Update current user's email preferences

Queue Processing Pipeline

The mail operations queue follows a linear state machine:

User action → API endpoint → admin_mail.mail_ops (status: pending)
                                      ↓
                          Worker polls every 5 seconds
                                      ↓
                            status: processing
                                      ↓
                         status: done  OR  status: error

When a user triggers a delete, move, or flag operation with ?background=true, the API inserts a row into admin_mail.mail_ops with status pending. The Mail Ops Worker polls the table at a configurable interval and processes each job sequentially, updating status to processing, then done or error upon completion.

Environment Variables

Variable	Default	Description
EMAIL_WARMER	false	Enable/disable the background cache warmer
EMAIL_OPS_INTERVAL_MS	5000	Mail Ops Worker polling interval (ms)
EMAIL_SYNC_INTERVAL_MS	5000	Mail Sync Worker polling interval (ms)

State Ownership & Database Schema

All tables reside in the admin_mail schema. They are organised into five categories by function.

Core Tables

Table	Purpose	Key Details
mailboxes	Configuration truth for all email accounts	connection_health: healthy, degraded, failed, auth_failed
mailbox_user_access	User-to-mailbox relationship mappings	Links platform users to mailbox accounts
messages	Cached message data (not authoritative)	has_full_body flag indicates whether full content is cached

Sync Tables

Table	Purpose	Key Details
mailbox_folder_state	Per-folder sync bookmarks	Tracks last_uid, last_modseq, last_sync_at

Queue Tables

Table	Purpose	State Machine
mail_ops	Delete, move, flag operations	pending → processing → done | error
mail_sync_jobs	Background sync job tracking	Managed by Mail Sync Worker
outgoing_mail	SMTP send queue	queued → sending → sent | failed

Stats Tables (Derived Views)

Table/View	Purpose
email_stats_folder_current	Per-folder message counts and sync status
email_stats_mailbox_current	Per-mailbox aggregate statistics
email_stats_queue_current	Queue depth and processing rates

Feature Tables

Table	Purpose	Status
user_email_rules	User-defined email filtering rules	Active
reggie_rules	AI-driven email rules	Future
reggie_drafts	AI-generated draft responses	Future

Legacy Tables (Deprecated)

These tables are from the original implementation and should not be used for new features:

Legacy Table	Replaced By
user_email_accounts	mailboxes + mailbox_user_access
user_email_messages	messages
user_email_prefs	mailbox_user_prefs

Truth Hierarchy

Data authority flows top-down. When conflicts arise, the higher-level source wins:

IMAP Server (authoritative)
    ↓
mailboxes (configuration truth)
    ↓
messages (cache — rebuilt from IMAP)
    ↓
Queue tables (transient processing state)
    ↓
Stats tables (derived, read-only)

Analytics Widgets

Eight dashboard widgets provide operational visibility into the email system:

#	Widget	Data Source
1	Mail Operations Queue	GET /api/v1/email/imap/queue-stats
2	Email Cache Storage	GET /api/v1/email/imap/cache-stats
3	Email Cache Storage Monitor	Derived from cache-stats over time
4	Mail Sync Jobs	admin_mail.mail_sync_jobs
5	Outgoing Mail Pipeline	admin_mail.outgoing_mail
6	Queue Alerts & Failures	Filtered view of failed/error queue entries
7	Mailbox Health & Cache	GET /api/v1/email/imap/system-mailbox-health
8	Queue Throughput (Last Hour)	Aggregated ops completed in rolling 60-minute window