Current Datastore API use

Thanks for sharing this. I’m afraid that a 10-day period is unlikely to be the best judgement for both actual and potential use of the DataStore, or the impact of failing to preserve backward-compatibility. For example, Madagascar production AMP has the latest version of the IATI-AIMS import tool deployed, which allows them to use the (current) datastore directly – I’m not aware if they’ve done this yet, but they had previously been importing IATI so it is likely that they will, if they’ve not already (outside of that 10-day evaluation period). Other AMPs also have the tool deployed, but have not engaged with IATI. If those countries chose to do so in the future, they would first need to have any new changes pushed to their servers.

Bill Anderson great to get some data on the table and thanks for sharing. A small note to say that the South Korea calls are probably a developer who works for me - we have a tool that I use to figure out email addresses across multiple activities which uses the v1 datastore CSVs for their aggregations plus a few other calls - we should be done soon but it is throttled pretty heavily at the moment so it is taking us some time to iterate through the bugfixes and then all the publishers.

It’s great to see this research! Thanks to IATI Technical Team for digging into this. A few points:

1. Agree with Joshua Powell - 10 days is unlikely to provide a representative sample of current users, even aside from underestimating the number of systems that are able to use IATI data but not currently doing so.

2. On Bangladesh: “random” is not correct. As I explained to Bill Anderson , the IATI import module is supposed to check nightly for new data. This process needs to be started again. However in the meantime, donors can (and are) still choosing to import their data. Three donors have imported their data in the last month or so: the Netherlands, USAID and World Bank.

3. I also discovered a system I built a couple of years ago for the Liberian Ministry of Finance that is still importing data for Liberia and Uganda from the Datastore, and generating nice spreadsheets of data according to fiscal years.

4. It would be great to work with those countries (Rwanda, Haiti and Malawi) that have nascent IATI import systems to get them into production mode. Perhaps also Bangladesh to address the current fairly minor issue there.

5. Finally: I think this shows how hard it is to get these systems up and running sustainably at country level. We should be supporting and strengthening these existing systems. However, if the existing IATI Datastore API is discontinued, many of these systems will break. That will significantly set back efforts to promote data use at country level. It will also raise serious questions as to whether countries should invest in IATI in future.

What is the actual cost of providing backwards compatibility to these existing systems? Based on Andy’s work, it does seem that the investment on the Datastore side should be fairly modest, and certainly less than the cost of all providers needing to both update their tools and deploy those updates (to the extent this is even possible given ongoing contracts or relationships). Would that not be worth the cost to avoid losing any of these countries?

I also thought that we might need to add aidhedge to this list as they were using the DSv1 according to their website: https://story.aidhedge.org/integrating-iati-a-tech-overview-3c921aa9ded9

As mentioned in the first post, we are continuing to monitor the logs so decisions are not going to be based on 10 days only.

To expand a bit on our plans, we have listened to your requests and concerns and, in conjunction with the new Datastore’s dev team, are devising a plan to cater to all the existing user’s needs during the months that will see both Datastores running in parallel so that no action will be needed on the existing user’s side.

How this will be implemented still needs to be finalised, but once that happens it will be promptly communicated. We just need to be given some thinking space and proper time to consider all edge cases, drawbacks and release a robust solution for all.

Hope this helps a bit.

Samuele Mattiuzzo - great to hear - if there is anything (e.g. edge cases, calls that could be tweaked to make a specific solution possible etc) that you would find helpful to get input on, then please do just ask.

Great to hear, thanks Samuele Mattiuzzo

To echo Mark Brough matmaxgeds Joshua Powell and others in the Community That Can Give Technical Advice & Support In A Collaborative Way (CTCGTASIACW - I agree it’s not that catchy), do reach out early and often for our help.

Herman van Loon I believe Mark Brough means that donor staff from the Netherlands embassy imported the Netherlands IATI data using the tool built into the Bangladesh AIMS (which relies on the current datastore maintained by the IATI tech team).

Mark Brough and matmaxgeds : thanks, perfectly clear now!