Resolved -
The fix of yesterday has resolved the issues. Please see the further timeline and impact below.
- Timeline
18:57 Paazl monitoring picks up issue with cache cleaning mechanism resulting in partial outages
19:00 Customers begin reporting issues with labels and webapp login
19:20 IT Teams finds the issue
19:31 IT team implements fix
19:39 First customers notice improvements
19:51 IT team confirms that fix has worked for all customers
19:52 Statuspage updated to reflect fix
- Technical impact
The cache cleaning mechanism that we rely on throughout our platform is set to evict old cache entries whenever it’s full. In this instance it did not do so. We have identified the issue in this mechanism and fixed it. The load testing that we performed in preparation for Black Friday were volume tests, performed over a specific period of time. This kind of stress test does not replicate the cache filling of a full day of volumes.
The most significant impact of this issue, related to the generation of shipping labels, and the logging into our webapp.
During the affected period, we noticed a potential issue with labels that were requested. The barcodes created may not have been pre-registered at the carrier. We advise you to verify in your carrier portal whether the created barcodes are known to them. If they are not, then please consider re-labeling those packages.
We remain confident in our preparation for this peak season and the steps we have taken to make it a success
Nov 26, 14:37 CET
Update -
We have noticed a potential issue with labels that were requested during the time period of this issue.
The barcode may not have been pre-registered at the carrier.
Please verify in your carrier portal whether the barcodes are known to them. If they are not, then please consider relabeling the packages.
Nov 25, 20:42 CET
Monitoring -
A fix has been implemented and we are monitoring the results.
Nov 25, 19:53 CET
Identified -
We are noticing issues with our Web Application. We have identified the issue and are performing corrections.
Nov 25, 19:35 CET