Billing period data unavailable in Billing and Analytics reports
Incident Report for Xandr
Postmortem

Incident Summary

On Monday, April 11 at 14:00 UTC, a release went out that caused billing_period_id and flight_id to appear as "0" in Billing reports, Network Analytics reports and Log Level Data. The null values were fixed and data was available to be repulled by 2019-04-16 at 20:18 UTC.

Scope of Impact

During the incident window, billing_period_id and flight_id were unavailable in Billing and Network Analytics reports, and in Log Level Data. Log Level Data for these fields was not recovered for the span of the incident.

Timeline (UTC)

2019-04-11 14:00:00 Incident started due to release
2019-04-16 16:30:00 Incident flagged and escalated to engineering
2019-04-16 20:18:00 Release pushed that fixed null values in new data

Cause Analysis

The incident was caused by a release causing values for "billing_period_id" and "flight_id" to populate as "0" in our data pipeline. These incorrect null values then appeared in Billing Reports, Network Analytics reports and Log Level Data.

Resolution Steps

Our engineers resolved the issue by directly correcting aggregated data used to populate Network Analytics and Billing Reports. Log level data was not recovered.

Next Steps

Institute alerts to notify engineering teams of fields incorrectly populated as "0". Improve methods for reprocessing incorrect data. A test data pipeline is currently in development; this will allow us to push and verify new releases in the test pipeline before deploying them in production.

Posted May 23, 2019 - 13:34 UTC

Resolved

All data in console reporting is now correct. Reports for the affected dates can be re-pulled. Some visualizations in "Visual Success" screens for Insertion Orders, Line Items and Campaigns may still display incorrect data for the affected dates.

Posted May 16, 2019 - 15:33 UTC
Monitoring

We have patched the issue and are monitoring our systems closely. Our engineers are reprocessing the impacted analytics and billing data, and we will notify clients upon resolution. Log Level Data will not be reprocessed.

Posted Apr 24, 2019 - 19:02 UTC
Identified

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Apr 17, 2019 - 19:57 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Analytics reports, Log Level Data, Billing reports
  • Impact(s):
    • Billing period data unavailable in Billing and Analytics reports
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Apr 17, 2019 - 14:50 UTC