An Engineering Deep-Dive into Atlassian's April 2022 Outage

In April 2022, Atlassian suffered a critical, high-profile outage that completely took down approximately 400 paying cloud customers across major products including Jira, Confluence, and Statuspage. The incident was not caused by a sophisticated cyberattack or infrastructure scaling failure; it was triggered by a script execution error and a communication breakdown that physically purged tenant data.

Even more perplexing to observers was the recovery timeline: why would an enterprise-grade cloud software company take nearly two weeks to restore data from existing backups? This deep dive analyzes the officially released incident report to uncover the underlying systems design, architectural choices, backup mechanisms, and the multi-tenant database realities that governed the outage.

---

1. The Incident: What Happened?

During a maintenance window on Monday, April 4, 2022, Atlassian engineers executed an internal script to deactivate a legacy standalone app named Insight Asset Management (which was being migrated into core native functionality for Jira Service Management and Jira Software).

Instead of uninstalling the app, the maintenance task physically deleted the cloud sites belonging to roughly 400 customer organizations. The deletion affected every relational table, permission set, configuration, and attachment tied to those environments.

The Direct Causes

1. Communication Gap (Ambiguous Identifiers): The team coordinating the deprecation submitted a list of IDs to the operational team executing the script. Instead of passing the unique App IDs (app_id) representing the standalone extension, they mistakenly passed the Cloud Site IDs (site_id) corresponding to the customer environments.
2. Dangerous Script Dual-Modes: The script being used had two operating modes:
   • Mark for Deletion (Soft Delete): Flags records as inactive for routine decommissioning.
   • Permanently Delete (Hard Delete): Issues destructive DELETE statements or drops entities to meet strict regulatory and compliance requirements.
3. Mismatched Execution Mode: The operator executed the script with the permanently delete flag enabled, targeting the cloud site identifiers instead of the app identifiers.

[Maintenance Request] 
         │
         ├── Provided Identifiers: [site_id_1, site_id_2, ...] (Expected: app_id)
         ├── Mode: Hard Delete (Expected: Soft Delete)
         ▼
[Internal Deletion Script]
         │
         ▼
[Primary Production DB] ──▶ Hard deletes entire site schemas & multi-tenant rows

---

2. Core Concepts: Soft Deletes vs. Hard Deletes

In distributed architectures and operational tooling, handling deletions requires balancing data recovery guarantees with data privacy compliance.

Soft Deletes (Mark for Deletion)

Instead of removing a row or object from disk, a soft delete marks the record as inactive:

UPDATE tenant_sites 
SET is_deleted = TRUE, deleted_at = NOW() 
WHERE site_id = 'site_12345';

• Pros: Recovery is instantaneous. Rolling back an accidental deletion is simply an update flipping is_deleted back to FALSE.
• Cons: Queries must filter out deleted rows (WHERE is_deleted = FALSE), which degrades indexing efficiency, increases table bloat, and complicates unique constraints.

Hard Deletes (Permanent Removal)

Hard deletion removes the record entirely using DELETE FROM ... or physical data drops.

• Why is Hard Delete even an option? Global data privacy regulations such as GDPR (Right to Be Forgotten) and CCPA strictly require organizations to permanently excise personal data across transactional stores, analytical warehouses, and caches upon request. Soft-deleting records does not satisfy regulatory requirements if user identifiers remain retrievable.
• The Flaw: When an automated script combines both soft and hard delete flags without safeguards (such as dry-runs, safety checks, or confirmation prompts for large batch operations), human error can convert a minor operational maintenance task into a major catastrophe.

---

3. Replication and Backup Architecture

To understand both the survival of the data and why some customers lost up to 5 minutes of writes, we must examine Atlassian’s data resilience tiers.

flowchart LR
    User([User Write]) --> Primary[Primary DB]
    subgraph Multi-AZ High Availability
        Primary -- Synchronous Replication --> Standby[Synchronous Standby Replica]
        Standby -- Acknowledged --> Primary
    end
    Primary -- Confirmed Write --> User
    
    subgraph Data Capture & Long-Term Storage
        Primary -- CDC (e.g. 5-min batch) --> BackupDB[(Incremental Backup Store)]
        Standby -- Periodic Snapshot Dump --> S3[(Immutable Backups on S3)]
    end

Tier 1: Synchronous Standby Replicas (High Availability)

For mission-critical production systems, high availability (HA) protects against hardware crashes or localized Availability Zone (AZ) failures:

1. The application client executes an insert or update against the Primary Database.
2. The Primary applies the transaction and immediately replicates the write log to a Synchronous Standby Replica in a separate AWS Availability Zone.
3. The Standby confirms write persistence to disk.
4. The Primary returns a write acknowledgement (ACK) back to the application client.

Because replication lag is effectively zero, an automated failover (typically completing within 60 to 120 seconds) guarantees RPO = 0 (Zero Data Loss) during an infrastructure outage.

The Hard Delete Problem: Standby replication operates at the physical or logical storage layer. When a destructive DELETE statement executes on the Primary, that deletion immediately replicates to the synchronous standby. Both copies are purged in real-time.

Tier 2: Change Data Capture (CDC) and Incremental Backups

Atlassian noted in their post-mortem that while most restored customers experienced no data loss, a subset reported missing data for up to five minutes prior to the deletion.

This delay occurs due to Change Data Capture (CDC) and incremental backup schedules:

• Incremental backups batch and drain database write-ahead logs (WAL) or snapshot deltas at defined intervals (e.g., every 5 minutes) into secondary storage.
• If a customer created Jira issues 2 minutes before the deletion script wiped their site, those writes had not yet been processed by the next 5-minute CDC batch window.
• Once the primary database deleted the records, the source rows vanished, leaving a permanent 5-minute gap in the recovery timeline.

Tier 3: Immutable Snapshots on Object Storage

To survive catastrophic database events, Atlassian maintains immutable cold backups:

• Point-in-time snapshots of standby nodes are compressed and shipped to cheap, durable object storage (like Amazon S3).
• These snapshots use error-correcting codes (ECC) and write-once policies to prevent bit rot and data tampering.
• Snapshots are retained for 30 days and are regularly audited and restored to test recovery reliability.

---

4. The Multi-Tenant Architecture Trade-Off

Why did restoring from immutable S3 backups take two weeks instead of a few hours? The answer lies in multi-tenant storage partitioning.

Isolated Multi-Tenancy (Single-Tenant Silos)

In a pure siloed model, each customer receives a dedicated database instance.

Tenant A ──▶ [ Database A ] ──▶ [ Backup A ]
Tenant B ──▶ [ Database B ] ──▶ [ Backup B ]

• Recovery: If Tenant A’s database is accidentally deleted, an engineer simply spins up a new instance, restores Backup A to that specific point in time, repoints DNS/routing, and the customer is back online in under an hour.
• Drawback: Maintaining hundreds of thousands of independent relational database instances is prohibitively expensive, creates massive connection-pooling overhead, and leads to severe resource fragmentation.

Shared Multi-Tenancy (Multiplexed Data Stores)

To maintain cost efficiency and high hardware utilization, SaaS systems multiplex multiple tenants across shared database clusters:

┌─────────────────────────────────────────────────────────┐
│                     Shared Data Store                   │
│  ┌───────────────────────────────────────────────────┐  │
│  │ Table: issues                                     │  │
│  │ id  | site_id (Tenant) | issue_key | title        │  │
│  │ 101 | site_A           | PROJ-1    | Login Bug    │  │
│  │ 102 | site_B           | DEV-42    | Refactor API │  │
│  │ 103 | site_C           | OPS-12    | Patch OS     │  │
│  └───────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

Atlassian multiplexes tenant data: a single large relational data store houses thousands of rows belonging to multiple unique organizations, partitioned logically by site_id or tenant_id.

---

5. The Recovery Bottleneck: Surgical Row-Level Extraction

When the incident deleted 400 customer sites, only a small fraction of the data within any given shared data store was purged. The unaffected tenants sharing those databases continued to operate normally, reading and writing thousands of updates every minute.

sequenceDiagram
    autonumber
    participant LiveDB as Production Shared DB (Tenants A & B)
    participant S3 as Cold Backup Snapshot
    participant StagingDB as Isolated Staging DB
    
    Note over LiveDB: Tenant A deleted by script!<br/>Tenant B remains active and writes new data.
    LiveDB->>LiveDB: Tenant B adds Rows (IDs: 104, 105)
    
    Note over StagingDB: Cannot overwrite LiveDB with Snapshot!<br/>That would delete Tenant B's new writes.
    
    S3->>StagingDB: 1. Restore Snapshot (Hundreds of GBs)
    Note over StagingDB: Contains original Tenant A + old Tenant B data
    StagingDB->>StagingDB: 2. Filter & extract only Tenant A rows
    StagingDB->>StagingDB: 3. Resolve relational dependencies (FK trees)
    StagingDB->>LiveDB: 4. Surgically inject Tenant A data back into Live DB

Why Atlassian Couldn’t Just “Restore the Database”

If engineers had performed a standard full-database point-in-time restore, they would have rolled back the entire data store to its pre-deletion state. While this would recover the 400 impacted tenants, it would wipe out all new writes made by thousands of unaffected customers during the incident.

The Manual, Surgical Recovery Process

To prevent collateral data loss, Atlassian had to implement a surgical data extraction pipeline for each impacted site across multiple data stores:

1. Provision Staging Clusters: For every shared database that held affected data, spin up an isolated staging database instance.
2. Restore Heavy Snapshots: Download and restore multi-hundred-gigabyte snapshots from cold S3 storage into staging.
3. Isolate Target Tenant Records: Query the staging database to extract only the records belonging to the affected site_id.
4. Traverse Relational Dependency Graphs: Enterprise platforms like Jira and Confluence have deeply nested foreign key (FK) relationships (users, permissions, projects, issues, comments, custom fields, history logs, attachments). Data cannot simply be inserted haphazardly:
   • Parents must be inserted before children.
   • Surrogate primary keys and unique constraints must be maintained or carefully remapped.
   • Circular dependencies must be temporarily deferred or resolved.
5. Inject Data into Production: Stream the extracted rows into the live production tables while other tenants continue to perform active transactions.
6. Validate Integrity: Run automated consistency audits to ensure zero relational orphan records exist, followed by customer-specific verification.

Because this pipeline had to be designed, scripted, tested, verified, and executed across 400 separate organizations—each with distinct data topologies and varying snapshot volumes—the recovery process stretched from hours into weeks.

---

6. Architectural Lessons and Best Practices

Failure Point	Anti-Pattern	Recommended Architectural Pattern
ID Ambiguity	Generic identifiers used interchangeably across internal tools (id).	Strictly typed or prefixed identifiers (e.g., site_xyz vs. app_inst_abc) and parameter type-safety.
Script Capabilities	Combining soft and hard delete flags within a single automated routine.	Decouple destructive workflows into separate, restricted binaries requiring multi-party approval.
SaaS Multi-Tenancy	Designing backup strategies optimized exclusively for full-instance recovery.	Building automated tenant-level point-in-time recovery (PITR) tooling and tenant-scoped backup streams.
Data Purge Blast Radius	Hard deletes executed directly against primary databases without validation.	Enforcing multi-stage deletion pipelines: always soft-delete first with a mandatory 14–30 day quarantine before hard purging.

Key Takeaways

• High Availability is Not Disaster Recovery: Having multi-AZ synchronous standby replicas protects against node failures, not against human error or logical data corruption. Destructive writes replicate instantaneously.
• Design for Tenant-Level Restoration: If you multiplex multiple customers onto a single database, full-snapshot point-in-time recovery is insufficient. You must maintain either logical row-level historical backups or build automated tenant extraction tooling long before an incident occurs.
• Operational Safeguards Matter: Destructive operations, especially those handling compliance-driven hard deletions, should never accept bulk site identifiers without cross-verifying the entity type, running mandatory dry-runs, and requiring independent operational approvals.