Punchh Platform Campaign Processing Delays

Incident Report for Punchh US-West

Postmortem

Postmortem — Punchh Platform Campaign Processing Delays | April 27, 2026

What Happened On April 27, 2026 between 17:20 UTC and 18:30 UTC, Punchh platform experienced campaign processing delays caused by a compute outage at Databricks, one of our third party infrastructure providers. The outage affected compute clusters across multiple global regions simultaneously.

Detection The issue was first detected via Airbrake alerts on MS2 and DQ at 12:36 PM CDT, followed by a Databricks official status page update at 12:37 PM CDT confirming they were actively investigating. Impact on campaigns was confirmed by the engineering team shortly after.

Impact

Mass gifting campaigns failed or were delayed on MS1 and MS2
Data exports failed due to inability to spin up new clusters
Data sync delays observed across stacks
Reporting dashboards were temporarily unavailable
All guest-facing features, mobile APIs, and core platform remained fully operational throughout

Root Cause Databricks compute clusters became unresponsive or failed to start across multiple regions. This is a third party vendor issue — no Punchh code or infrastructure changes contributed to this incident.

Resolution Databricks restored service at 18:30 UTC. All impacted campaigns have been identified and are being retriggered by the campaigns team. Platform is fully operational.

What We Are Doing to Prevent Recurrence

Reviewing alerting to improve early detection of third party outages and their downstream impact on campaigns
Evaluating resilience improvements for campaign processing to handle temporary third party unavailability more gracefully

Posted Apr 27, 2026 - 12:38 PDT

Resolved

This incident has been resolved.

Summary:
Between 17:20 UTC and 18:30 UTC on April 27, 2026, Punchh platform experienced campaign processing delays due to a compute outage at one of our third party infrastructure providers (Databricks). The outage affected multiple regions globally.

Impact:
- Mass gifting campaigns experienced failures or delays during the outage window
- Data exports were delayed or failed
- Reporting dashboards were temporarily unavailable
- All guest-facing features remained fully operational throughout

Resolution:
The infrastructure provider restored service at 18:30 UTC. All impacted campaigns have been identified and will be retriggered. Platform is now fully operational.

We sincerely apologize for any inconvenience this may have caused. We are reviewing this incident to prevent similar disruptions in the future.

Posted Apr 27, 2026 - 12:30 PDT

Update

Update: Third Party Vendor Issue Identified — Monitoring

We have identified that the campaign processing delays were caused by a compute outage at one of our third party infrastructure providers. The vendor has confirmed service was restored as of 18:30 UTC on April 27, 2026.
Customer Impact:

Mass gifting campaigns experienced failures or delays during 17:20 — 18:30 UTC
Data exports and dashboards were also affected during this period
All other platform features remained fully operational throughout

Current Status:

Vendor has restored service and is actively monitoring for full stabilization
Our team is verifying that all affected campaigns and services have recovered
No action required from customers at this time

We will provide a further update once we have fully confirmed recovery on our end. We apologize for any inconvenience caused.

Posted Apr 27, 2026 - 11:55 PDT

Investigating

We are currently experiencing delays in processing campaigns. Our team is actively working to resolve the issue as quickly as possible.
Customer Impact:
• Campaigns may take longer than usual to be sent out.
• You might experience delays in receiving real-time data and updates.
• The Punchh Dashboard will show that your campaign is scheduled instead of processing or processed after than the scheduled start time
• If you're still seeing this behavior 2 hours after the scheduled start time, please submit a support case

We apologize for any inconvenience this may cause and appreciate your patience. Updates will be provided as more information becomes available. Thank you for your understanding.

Posted Apr 27, 2026 - 11:41 PDT