Unable to create publishers
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 15:56 UTC on Monday, August 5th to 14:11 UTC on Tuesday, August 6th users were unable to create new publishers due to an issue retrieving rates from the Currency API.

Scope of Impact

During the incident window, users were unable to save new publishers.

Timeline (UTC)

2019-08-05 15:56: Incident Started: Currency API release.

2019-08-06 09:21: Incident opened.

2019-08-06 09:36: Incident escalated to engineering. Code change that caused the incident identified.

2019-08-06 14:11: Incident Resolved: Currency API version updated with hotfix.

Cause Analysis

Internal maintenance work was done to migrate the Currency API to use a new central request routing service. The new service did not handle requests from the publisher creation process in the way that the Currency API expected.

Resolution Steps

Our engineers resolved the incident by updating the Currency API to better handle proper request routing protocols.

Next Steps

  • Establish clearer application ownership to ensure changes are communicated in a timely and effective manner.

  • Improve alerting on error logging for the Currency API to more quickly address issues.

  • Clean up escalation pathway for owning engineering team.

Posted Aug 12, 2019 - 12:34 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Aug 06, 2019 - 14:45 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Sell-side Console pages
  • Impact(s):
    • Unable to save/edit objects
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Aug 06, 2019 - 12:14 UTC