Incident Summary

The reporting data for 2018-06-06 hour 14 UTC was delayed, and was not available for reporting until 2018-06-07 00:50 UTC.

Scope of Impact

During the incident window, reporting data for 2018-06-06 hour 14 UTC was not available.

Timeline (in UTC)

• 2018-06-06 14:07: Incident started. Data transportation servers were decommissioned.
• 2018-06-06 17:26: Engineers were notified of incident and began work to resolve the reporting delay.
• 2018-06-07 00:50: Incident resolved. Data pipeline caught up, reporting back to normal.

Cause Analysis

Data transportation servers that had been decommissioned erroneously continued to receive reporting data. When the servers were decommissioned, the reporting data on those servers had not yet been ingested.

Resolution Steps

Our engineers resolved the issue by copying data from the decommissioned data transportation servers to active servers in another datacenter, and reprocessing the data.

Next Steps

• Increase the automated monitoring of the data transportation server builds within our data pipeline.