How Slack Efficiently Classifies Emails at Scale with Eventual Consistency

When designing software systems, seemingly trivial feature requests often conceal immense distributed systems complexity beneath the surface. A prime example is Slack’s internal mechanism for classifying user invitations.

When a user invites a colleague or collaborator to a workspace via email, Slack needs to decide whether that invitee should be treated as an internal employee (full access to all public channels and workspace features) or an external guest (limited access restricted to specific channels, such as Slack Connect or single-channel guests).

While this looks like a problem that could be solved by a simple UI checkbox or string matching on the domain name, real-world enterprise topologies break naive implementations. To handle this at enterprise scale, Slack engineered an asynchronous, heuristic-based, and eventually consistent classification system backed by event streams and self-healing worker jobs.

---

The Product Problem: Why Simple Solutions Fail

1. UX Friction Rules Out Manual Prompts

A product-driven solution might suggest adding a checkbox to the invite dialog asking: “Is this person an internal colleague or an external contractor?”

This degrades user experience. When users invite dozens or hundreds of collaborators at once, forcing them to manually classify every address introduces cognitive overload and invites human error, potentially leaking sensitive workspace information to external parties.

2. Multi-Domain Enterprise Topologies

Another naive assumption is that all internal employees share the exact same email domain (e.g., @company.com). In enterprise environments, this assumption breaks down immediately:

• Regional Domains: Multinational corporations often segment by geography (e.g., @company.in, @company.co.uk, @company.us).
• Contractors and Interns: Companies frequently issue dedicated domains or subdomains for non-permanent staff (e.g., @contractor.company.com or @vendor-corp.com).
• Corporate Mergers & Acquisitions: Large enterprises regularly operate multiple distinct domains concurrently under one parent organization.

Slack needed an automated backend service capable of inferring whether an arbitrary email address belongs inside or outside the organization with minimal latency.

---

The Multi-Layered Heuristic Classification Pipeline

To classify an email without resorting to brittle, resource-heavy machine learning models, Slack relies on a cascading set of deterministic heuristics:

Incoming Invitation Email
          │
          ▼
┌────────────────────────────────────────┐
│ 1. Settings Context                    │ ──(Match)──► Classify Internal/External
│    Explicit admin domain allowlists    │
└────────────────────────────────────────┘
          │ (No match)
          ▼
┌────────────────────────────────────────┐
│ 2. Inviter Context                     │ ──(Match)──► Inherit Inviter's Status
│    Does invitee domain == inviter's?   │
└────────────────────────────────────────┘
          │ (No match)
          ▼
┌────────────────────────────────────────┐
│ 3. Team Context                        │ ──(Evaluate)──► Threshold Analysis
│    Aggregated domain statistics        │                 (e.g., >= 10% = Internal)
└────────────────────────────────────────┘

1. Settings Context

The system first checks explicit workspace-level configurations. Administrators can specify explicit allowlists of trusted domains or block external invites altogether. If a domain matches these settings, the service short-circuits immediately.

2. Inviter Context

If the configuration does not resolve the classification, the system inspects the inviter’s profile. If an employee with an external domain (e.g., @vendor.com) invites another individual with @vendor.com, the system infers that the invitee should share the same access level as the inviter.

3. Team Context (Statistical Aggregation)

When the first two layers yield no conclusive answer, the system inspects workspace-wide statistics. A Slack workspace can scale to hundreds of thousands or millions of users. Iterating over all workspace users in real time during an invite lookup is prohibitively slow ($O(N)$ query time).

Instead, Slack tracks pre-aggregated domain counts grouped by team, domain, and role.

---

Data Modeling: Tracking Domains by Role

Slack maintains an aggregated table representing domain distribution per workspace:

team_id	domain	role	count	date_updated
team_1	example.com	admin	3	2023-01-15 10:00:00
team_1	example.com	member	150	2023-01-15 10:00:00
team_1	example.in	member	45	2023-01-16 11:30:00
team_1	gmail.com	member	2	2023-01-17 08:15:00

Why Track Roles Alongside Domains?

A primary email domain used by an admin provides a much stronger signal of corporate ownership than a domain used solely by standard members. If an administrative account operates on @example.com, the confidence that @example.com is an internal domain increases significantly.

The Percentage Threshold Heuristic

To classify an email using team context:

1. Extract the domain from the candidate email (e.g., example.in).
2. Query the aggregated table for team_1.
3. Compute the ratio of active users carrying that domain against the total workspace population.
4. Apply a threshold (e.g., 10%):
   • If users with that domain account for $\ge 10\%$ of the team, classify as Internal.
   • If the domain accounts for $< 10\%$, classify as External.

(Note: The 10% threshold is a tunable parameter adjusted based on workspace size and behavior).

---

Event-Driven Architecture and Atomic Upserts

To keep the aggregated statistics updated without synchronously blocking user lifecycle events (sign-ups, profile updates, role elevations, deactivations), Slack uses an asynchronous, event-driven pipeline.

┌─────────────────┐
│ User Lifecycle  │
│ (Create/Update) │
└────────┬────────┘
         │ Publishes Event
         ▼
┌─────────────────┐
│  Apache Kafka   │
└────────┬────────┘
         │ Consumed by
         ▼
┌─────────────────┐       Atomic UPSERT       ┌─────────────────────┐
│ Async Worker    │ ────────────────────────► │ Aggregated Domains  │
│ Consumer        │   (Row-Level Locking)     │ Database Table      │
└─────────────────┘                           └─────────────────────┘

Atomic Upserts

When a user is created, rather than issuing a SELECT query followed by conditional application logic (IF exists UPDATE ELSE INSERT), workers issue atomic database UPSERT statements:

INSERT INTO team_domains (team_id, domain, role, count, date_updated)
VALUES ('team_1', 'example.com', 'member', 1, NOW())
ON DUPLICATE KEY UPDATE 
    count = count + 1,
    date_updated = NOW();

Handling Role Elevations

When a member is promoted to an administrator, two atomic operations occur:

1. Decrement previous role counter:

   UPDATE team_domains 
   SET count = count - 1, date_updated = NOW()
   WHERE team_id = 'team_1' AND domain = 'example.com' AND role = 'member';

2. Increment new role counter:

   INSERT INTO team_domains (team_id, domain, role, count, date_updated)
   VALUES ('team_1', 'example.com', 'admin', 1, NOW())
   ON DUPLICATE KEY UPDATE 
       count = count + 1, date_updated = NOW();

Because atomic upserts leverage internal row-level locks, multiple workers can execute concurrent updates safely without requiring coarse-grained distributed transactions.

---

The Distributed Systems Trap: Counter Drift

Despite atomic database operations, counter drift inevitably emerges in distributed pipelines due to message delivery semantics.

The At-Least-Once Delivery Problem

Most distributed message brokers (including Apache Kafka) guarantee at-least-once delivery by default. Under network partitions, consumer crashes, or rebalances, a message acknowledged by the consumer might fail to commit its offset back to the broker.

When this happens, the broker re-delivers the message:

1. Event: UserCreated(domain='example.com', role='member') is consumed.
2. Worker executes count = count + 1.
3. Network blip prevents the Kafka offset commit.
4. Broker re-delivers the exact same event to another worker.
5. Worker executes count = count + 1 a second time.

Because relative increment operations (count = count + 1) are non-idempotent, duplicate message processing silently corrupts aggregate values over time.

---

The Healer Service: Reconciling Without Lost Updates

To correct counter drift, Slack deployed a Healer Service. The healer can run periodically (e.g., daily cron), when a workspace upgrades to a paid plan, or when an unfamiliar domain is detected.

However, designing a reconciliation job under continuous write traffic introduces race conditions.

The Naive Approach: Full In-Memory Recomputation

In a naive reconciliation implementation:

1. The healer iterates over all user records in the workspace.
2. It computes the total counts per domain and role in memory.
3. It overwrites the database table with the freshly computed counts.

Time ──►
T0: Healer starts scanning users
│
├── T1: New user registers -> Worker increments count in DB
│
T2: Healer finishes scan and OVERWRITES DB table
    └── Result: The increment from T1 is completely lost!

If calculating counts across a 500,000-member workspace takes several minutes, any user creations, role updates, or deletions occurring during that calculation window will be permanently overwritten and lost.

The Robust Solution: Snapshot-Based Delta Upserts

Instead of full replacements, Slack uses a snapshot timestamp with delta upserts:

sequenceDiagram
    autonumber
    participant H as Healer Service
    participant DB as Domains Table
    participant U as Users DB
    
    H->>H: 1. Record snapshot timestamp T_start
    H->>DB: 2. Read existing counts (up to T_start)
    H->>U: 3. Compute actual counts for users where updated_at <= T_start
    H->>H: 4. Calculate drift delta = (Actual - Existing)
    H->>DB: 5. Apply relative delta via atomic UPSERT (count = count + delta)

Detailed Steps:

1. Capture Snapshot Timestamp ($T_{start}$): The healer records the precise system time it begins execution.
2. Read Current Baseline: Read current aggregated domain counts up to $T_{start}$ from the team_domains table.
3. Compute Actual Baseline: Scan the source of truth (the users database), but strictly filter records where updated_at <= T_{start}.
4. Calculate Drift ($\\Delta$): Compute the difference between actual state and aggregated state at $T_{start}$: $\Delta = \text{Count}_{\text{actual}} - \text{Count}_{\text{current}}$
5. Apply Relative Delta: Apply the delta using atomic arithmetic:

   UPDATE team_domains 
   SET count = count + :delta, date_updated = NOW()
   WHERE team_id = :team_id AND domain = :domain AND role = :role;

Why Delta Upserts Prevent Lost Updates

If new users join or change roles after $T_{start}$, normal event consumers continue processing events and issuing standard increments (count = count + 1).

Because the Healer Service applies a relative mathematical delta ($\\Delta$) calculated strictly on data prior to $T_{start}$, concurrent live updates are preserved. The system remains highly available, non-blocking, and converges to eventual consistency.

---

Key Architectural Takeaways

1. Heuristics Over Complex Models: Complex AI/ML systems are not always the optimal solution. Layered rule evaluation (Settings Context $\rightarrow$ Inviter Context $\rightarrow$ Aggregated Team Context) delivers low-latency, deterministic classifications with predictable operational overhead.
2. Isolate High-Cardinality Scans: Avoid running full-table aggregation queries on critical user paths. Maintain auxiliary summary tables updated asynchronously via event streams.
3. Plan for At-Least-Once Broker Semantics: Any system relying on non-idempotent mutations (count = count + 1) over message brokers will experience drift over time.
4. Avoid Blind State Overwrites in Reconcilers: When designing healer or reconciliation loops, full dataset overwrites introduce lost update anomalies. Using snapshot timestamps combined with relative delta updates guarantees mathematical convergence without locking production databases.