The Orc delayed events dropped to zero. The team continues to monitor the behavior of Orc and the creation of new orders.
Posted May 30, 2025 - 15:47 CST
Monitoring
The integrations handler system is up and running. Operational teams are now executing mitigation steps on their end to enable users to continue processing orders.
Posted May 30, 2025 - 14:52 CST
Identified
Corrupted event found. The team is evaluating the best approach to handle it and fix the queued events.
Posted May 30, 2025 - 14:01 CST
Update
The approach of skipping events was unsuccessful. The team is now aiming to edit the corrupted event directly in the database; this action should allow the integration system to continue processing events.
Posted May 30, 2025 - 13:51 CST
Update
The engineering team continues to try to skip corrupted events. These steps are taking more time than expected. The impact remains the same: - No new orders are being received and created successfully. - No events for integrated orders (via Orc) are being received or sent successfully.
Users should receive errors when attempting to transition an order from one status to another.
Posted May 30, 2025 - 13:37 CST
Update
Current impact: New orders are not being received or created on our end. Orders updates received via integration are not being processed. Orders updates that we send via integration are also not being processed.
Posted May 30, 2025 - 13:06 CST
Investigating
Orders are not being received, and order updates are not being processed.
Posted May 30, 2025 - 12:48 CST
This incident affected: Orders Reception (Integrations) (Andino, Brasil, Chile, Mexico).