This incident has been resolved for now. We will be investigating root cause and deploying the AWS update in the next few weeks.
Posted Jun 12, 2023 - 12:54 MDT
Monitoring
It looks like data is showing up again in the Live page as well as in Route. We will continue to monitor today and will plan for another update to AWS in the coming weeks once we have identified the root cause of what happened today.
Please keep us updated throughout the day if you see any more issues.
Thank you so much for your patience as we worked through the issues. We appreciate all of your support!
Posted Jun 12, 2023 - 09:57 MDT
Update
We are still seeing very slow response times. Our database is 99% utilized and that is what is causing the slowness to continue. We are working with our database provider and should have an update soon. Thank you for your continued patience! I know this is causing a lot of confusion and frustration and we are working to get it resolved as soon as possible.
Posted Jun 12, 2023 - 07:50 MDT
Update
For Route users, you may also notice that scheduled orders are not appearing on the Schedule page. This will also be resolved once the server is fully functional. All of your scheduled orders are still scheduled.
Posted Jun 12, 2023 - 06:56 MDT
Identified
We have pointed traffic back to the old server and are standing up new instances to keep up with the traffic. I see systems slowly coming back online. I will send out another update as soon as things appear more stable.
Posted Jun 12, 2023 - 06:45 MDT
Investigating
We upgraded our AWS instance yesterday and it is causing response times to be slow or appear unresponsive this morning. We are in the process of pointing traffic back to the old server and will let you know when it's complete.
This should not be affecting the Deliver app, only the portal. And most notably Deliver Web and Route.
We are so sorry for the inconvenience this morning and should have you back up and running soon.
Posted Jun 12, 2023 - 06:26 MDT
This incident affected: Route and Customer Portal.