As for the data... Signatures, photos, comments, etc. were all updated to Eclipse so that is good news! Unfortunately, any stop or manifest that was completed this morning AFTER the system went down will not display in their associated tables in Deliver Web. We do apologize for that, and for everything that happened today. We always learn something when these types of situations happen, and those learnings make us stronger as a team moving forward. Again, we are so sorry for the inconvenience this has caused you today. Thank you for your continued partnership and your patience.
Posted Jan 08, 2024 - 16:05 MST
Monitoring
We are FINALLY back up! We have identified and fixed the issue. There should be no more disruptions to Route and Deliver Web operations moving forward. We are still investigating if there was any data lost and will send out an update as soon as we know more. You may notice the data listed in the Live view is old. This will clear out tonight. Thank you SO much for your patience and support. It's been a day.
Posted Jan 08, 2024 - 15:26 MST
Investigating
Spoke too soon. We were up for a bit and then went down again. We are working through this as best we can and right now I have no update other than my previous update was incorrect. We will update you as soon as we know more. I so apologize for the issues today. Please know we are all doing the best we can to get you up and running again.
Posted Jan 08, 2024 - 13:46 MST
Monitoring
Good news! It looks like Route and Deliver Web are back online. We did have to update the DNS so it may take a bit for those changes to propagate. The data being displayed currently in Deliver Web Live does not look current. We are in the process of determining if any data was lost and if not, how to restore. We will send you another update when we have information on that. For all of you that need to start routing for tomorrow, you should be able to do that now.
Posted Jan 08, 2024 - 12:40 MST
Identified
We are working diligently to resolve the issue. We did identify the root cause of the failure. A scheduled report that runs nightly kept failing to run and locked up our storage nodes. Storage nodes connect to the database to perform operations, such as update, delete, insert, etc. Usually this is something that resolves on its own however due to a large spike in web traffic at the same time, our load balancer struggled to catch up. This ended up causing a bottleneck and the web traffic to eventually time out. This will resolve on its own when the web traffic subsides however we are still working with AWS to get it resolved ASAP. This is something that AWS should be able to help with but it is taking longer than we thought. We will keep you updated and let you know as soon as we know more. We will also try to identify the root cause of the report failure once our system is back online.
Posted Jan 08, 2024 - 10:49 MST
Update
Just wanted to provide an update. We are still working with AWS on identifying the cause. We do not yet have an ETA on when this will be resolved. Please know we are doing everything we can to get the site back up and running ASAP. We really appreciate your continued support and patience!
Posted Jan 08, 2024 - 08:13 MST
Update
We are in communication with AWS (Amazon) to see if they can help us identify the issue.
Posted Jan 08, 2024 - 07:16 MST
Investigating
We are getting reports that Deliver Web and Route are not displaying any data this morning. Users can still load manifests and deliver as normal through the Deliver app. We are investigating ASAP and will send out updates as we have them. Sorry for the inconvenience this morning. We understand the effect to your business operations.