Hi, I was looking up an project this morning for COVID in Palestine this morning and:
Thanks!
These are the 20 most recently minted DFID projects - the newest 17 of which aren’t on v1 of the datastore.
Any AIMS providers who are currently still using the v1 datastore should take note, as a few of these are COVID-19 related and most are global, and so could be relevant.
matmaxgeds you’re clearly up to speed, but ccing Mark Brough too. Do you know any other users? (Catalpa perhaps?)
Also, could you both mention countries that will be affected by this and the concurrent codelist issue so I can anticipate any issues from partners / other donors can be informed?
Edit: I’ve only included hierarchy 2 activities here, as they are the ones with the financials.
IATI devs used to have smoke tests in place, for checking possible problems with services offered.
I’ve just got these tests working again, and they report that the datastore hasn’t successfully fetched data since the end of March. I’m not exactly sure if that’s correct… The datastore doesn’t provide a reliable way to check if the parser is still running.
Hi IATI Technical Team - I am doing a training on Thursday afternoon for users in Somalia, any chance the DSv1 might be working again by then, especially as this is affecting the availability of COVID data until the DSv2 is launched? Thanks, Matt
Hi matmaxgeds
Great to know you’re doing some IATI training in Somalia.
We will look into the issue with the old Datastore immediately and get our Developer team to see what has gone on. Unfortunately it is unlikely we will have this resolved by Thursday afternoon. Can we help you in any other way to get the data you need in the interim-- happy to discuss options and run through the new Datastore with you if you’re able to use that in your training?
Hi IATI Technical Team - thanks, and looking forward to heading what the developer team say.
The backend of our aid management system in Somalia pulls data from the datastore nightly and we then access it through the AIMS system interface so can’t just swap to the upcoming datastore interface for the training (which is based on the system) - as soon as the new datastore is released, we will transition to pulling data from it and have been working on that with Josh.
Is it possible to setup some checks at your end to make sure that core infrastructure is working (I guess we will at our end too) and alert people? If it has really not been working between March and July, that is more than a quarter of a year with a core piece of IATI infrastructure offline which is pretty worrying!
I just discovered the datastore query parameter last-change__gt, which returns all data updated since the specified date. It shows data in the datastore hasn’t been updated since 28th March 2020:
http://datastore.iatistandard.org/api/1/access/activity.xml?last-change__gt=2020-03-28
I also discovered that the api returns old data. OGB updated their files, including COVID-19 activities. API returns the old file/activities.
New datastore returns incomplete activities (excludes elements that are not well declared)… opened ticket at github.
So bit stuck between old and new datastore
Hi all, the Tech Team is investigating as a priority today! We’ll update as soon as we know a bit more about what’s happened/happening. Thanks for your patience!
Hi IATI Technical Team it seems like you have cracked it as I am getting recent results again, and I see 15 new ones since yesterday - thanks!
Please can we have some sort of check in place so that we catch these earlier next time?
Thank you all for your continued engagement. We absolutely agree there is a need to have a robust alert system that tells us when data has not been refreshed. Historically IATI tools were not built with comprehensive, or at times any, tests. Over the last few years we have been increasing the number of tests we run on our existing tools, including the ones referenced in this post, whilst pushing forward with our new technical products. However, currently we don’t have a test for the old datastore that specifically checks if data has not refreshed.
We will be adding comprehensive tests to the new datastore, to try and ensure this type of break does not reoccur.
Although our recent focus has not been on the old datastore, we continue to maintain it and will add fixes when it breaks. This is with the understanding that it will be deprecated 6 months after the launch of the new datastore.
As Matt has already noticed, our senior developer has been investigating yesterday the issue with the datastore and below is a more technical explanation of what caused it and how it was resolved:
The datastore logs deleted activities in a table with a unique constraint on the column to which the external activity id is written. That id is theoretically unique and it previously had recorded nearly 700,000 unique ids without collision, presumably over its lifespan.
However, on 28th March it met its first collision, which caused the database transaction to fail against that constraint and the program to error.
Given that this is the first instance of this occurring after two thirds of a million ids it would appear to be a very rare occurrence, and there also appears to be no reason not to record that occurrence in the table. As such, we’ve remedied the situation by removing the constraint.
We do apologise that we didn’t spot the recent issue with data not refreshing sooner. As mentioned above, we will make sure we have this capability in the new Datastore.
IATI-techteam:
As such, we’ve remedied the situation by removing the constraint.
Fix sounds good to me – many thanks for digging into this and resolving it.
I notice the open source code on GitHub hasn’t changed. I guess the bit you’re referring to is just here? Please could you push the fix (and associated migration) to the open source code base.
I really recommend restarting IATI Website Tests – it was a really good initiative that previous iterations of the IATI tech team invested significant work into. I’ve sent a couple of small fixes that are enough to get the tests working again. This includes a fix to check whether data in the datastore has been refreshed.
I’ve also created a very minimal dashboard for it here:
IATI Service Monitor
Smoke tests and sanity checks for some live IATI websites and web tools.
It would be great to expand these tests to cover the new datastore and new validator, too, as well as other services like the codelist API.
Rory Scott from my side, we have 236 registered users in Somalia which uses the current datastore to access IATI data. I have also been using the datastore in Rwanda for their annual aid report, and in Palestine at the moment for COVID.
I guess it might also be impacting AMPs if they are in the middle of an import cycle Joshua Powell
Thanks for the heads up matmaxgeds – have flagged for colleagues working on IATI import in Haiti and DRC.