Market API outage
Incident Report for OpenChannel
Postmortem

We performed a major update which also required a database migration script to be run. This was the first time that we used a new database migration and versioning library. When the application code was deployed and the database migration script was run.

The downtime was caused due to the large number of updates performed by the migration script. The script essentially caused the database to deny service to other requests.

In future, all database migration scripts must be throttled as to only take a maximum of 60% of database resources.

Posted Jul 06, 2021 - 15:33 UTC

Resolved
This incident has been resolved.
Posted Jul 03, 2021 - 17:43 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 03, 2021 - 17:40 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 03, 2021 - 17:30 UTC
Update
The issue is due to our deployment of 2.4.0. We're working on rolling back the deployment.
Posted Jul 03, 2021 - 17:29 UTC
Investigating
We are currently investigating this issue.
Posted Jul 03, 2021 - 17:16 UTC
This incident affected: Market API.