Thank you all for your continued engagement. We absolutely agree there is a need to have a robust alert system that tells us when data has not been refreshed. Historically IATI tools were not built with comprehensive, or at times any, tests. Over the last few years we have been increasing the number of tests we run on our existing tools, including the ones referenced in this post, whilst pushing forward with our new technical products. However, currently we don’t have a test for the old datastore that specifically checks if data has not refreshed.
We will be adding comprehensive tests to the new datastore, to try and ensure this type of break does not reoccur.
Although our recent focus has not been on the old datastore, we continue to maintain it and will add fixes when it breaks. This is with the understanding that it will be deprecated 6 months after the launch of the new datastore.
As Matt has already noticed, our senior developer has been investigating yesterday the issue with the datastore and below is a more technical explanation of what caused it and how it was resolved:
The datastore logs deleted activities in a table with a unique constraint on the column to which the external activity id is written. That id is theoretically unique and it previously had recorded nearly 700,000 unique ids without collision, presumably over its lifespan.
However, on 28th March it met its first collision, which caused the database transaction to fail against that constraint and the program to error.
Given that this is the first instance of this occurring after two thirds of a million ids it would appear to be a very rare occurrence, and there also appears to be no reason not to record that occurrence in the table. As such, we’ve remedied the situation by removing the constraint.
We do apologise that we didn’t spot the recent issue with data not refreshing sooner. As mentioned above, we will make sure we have this capability in the new Datastore.