On Monday, April 11 at 14:00 UTC, a release went out that caused billing_period_id and flight_id to appear as "0" in Billing reports, Network Analytics reports and Log Level Data. The null values were fixed and data was available to be repulled by 2019-04-16 at 20:18 UTC.
During the incident window, billing_period_id and flight_id were unavailable in Billing and Network Analytics reports, and in Log Level Data. Log Level Data for these fields was not recovered for the span of the incident.
2019-04-11 14:00:00 Incident started due to release
2019-04-16 16:30:00 Incident flagged and escalated to engineering
2019-04-16 20:18:00 Release pushed that fixed null values in new data
The incident was caused by a release causing values for "billing_period_id" and "flight_id" to populate as "0" in our data pipeline. These incorrect null values then appeared in Billing Reports, Network Analytics reports and Log Level Data.
Our engineers resolved the issue by directly correcting aggregated data used to populate Network Analytics and Billing Reports. Log level data was not recovered.
Institute alerts to notify engineering teams of fields incorrectly populated as "0". Improve methods for reprocessing incorrect data. A test data pipeline is currently in development; this will allow us to push and verify new releases in the test pipeline before deploying them in production.
All data in console reporting is now correct. Reports for the affected dates can be re-pulled. Some visualizations in "Visual Success" screens for Insertion Orders, Line Items and Campaigns may still display incorrect data for the affected dates.
We have patched the issue and are monitoring our systems closely. Our engineers are reprocessing the impacted analytics and billing data, and we will notify clients upon resolution. Log Level Data will not be reprocessed.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.