The fix of yesterday has resolved the issues. Please see the further timeline and impact below.
- Timeline 18:57 Paazl monitoring picks up issue with cache cleaning mechanism resulting in partial outages 19:00 Customers begin reporting issues with labels and webapp login 19:20 IT Teams finds the issue 19:31 IT team implements fix 19:39 First customers notice improvements 19:51 IT team confirms that fix has worked for all customers 19:52 Statuspage updated to reflect fix
- Technical impact The cache cleaning mechanism that we rely on throughout our platform is set to evict old cache entries whenever it’s full. In this instance it did not do so. We have identified the issue in this mechanism and fixed it. The load testing that we performed in preparation for Black Friday were volume tests, performed over a specific period of time. This kind of stress test does not replicate the cache filling of a full day of volumes.
The most significant impact of this issue, related to the generation of shipping labels, and the logging into our webapp. During the affected period, we noticed a potential issue with labels that were requested. The barcodes created may not have been pre-registered at the carrier. We advise you to verify in your carrier portal whether the created barcodes are known to them. If they are not, then please consider re-labeling those packages.
We remain confident in our preparation for this peak season and the steps we have taken to make it a success
Posted Nov 26, 2024 - 14:37 CET
Update
We have noticed a potential issue with labels that were requested during the time period of this issue. The barcode may not have been pre-registered at the carrier.
Please verify in your carrier portal whether the barcodes are known to them. If they are not, then please consider relabeling the packages.
Posted Nov 25, 2024 - 20:42 CET
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 25, 2024 - 19:53 CET
Identified
We are noticing issues with our Web Application. We have identified the issue and are performing corrections.
Posted Nov 25, 2024 - 19:35 CET
This incident affected: Paazl (Paazl Live Webservice, Paazl Live Web Application).