Discussion

Datastore update?

matmaxgeds
matmaxgeds • 22 March 2019

Hi,

[~379] IATI Technical Team

Any chance of getting an update on the datastore progress/timeline? In particular, do you know when there will be a draft (or final) version of the API spec published?

The github tracker shows a lot of ‘todo’s’ but nothing ‘in progress’

Is there also confirmation of the date when the existing datastore will be switched off?

Thanks,

Matt

Comments (14)

matmaxgeds
matmaxgeds

sorry @siemvaessen I need to ask what ‘check!’ means i.e. that I should check all the old API routes against the new ones myself, or ‘check!’ as in ‘box ticked’ i.e. that the new API will not break any of the old ones?

Mark Brough
Mark Brough

+1 matmaxgeds and thanks @siemvaessen!

IATI Technical Team answers to these points would be very useful for planning around use of data at country level (especially “confirmation that the datastore endpoints/routes will not change for systems currently using the existing datastore”), thanks!

Also: it would be great to know what the strikethrough on “D1.3: Results should be deliverable in full” means?

Kate
Kate

Hi everyone,
Sorry for the delay. In terms of timeline the datastore will be using the IATI validator tool so the two things need to be launched together. We have a meeting next week in Holland where we are doing some face to face planning with @siemvaessen and Rolf Kleef . After that I’ll be able to share more information!

Expect more details from the IATI Technical Team after Easter.

Thanks for your patience.

matmaxgeds
matmaxgeds

Thanks for the update Kate - just to confirm, Easter is around the 20th of April I think, so it will be approximately three weeks before you will know if the new datastore will have different API routes to the current one?

Kate
Kate

Hi Matt,
I’m keen that when we give information we give full and clear information which is why we are going to wait to give you the update.
Please bare with us,

Kate

matmaxgeds
matmaxgeds

hi Kate IATI Technical Team

I am not sure that we can bare with you. Given that Siem is estimating a May launch, this could mean that the community gets just a few weeks notice that systems that currently use the datastore might stop working. I am particularly thinking about those without in-house programming support who could not respond on that kind of time-frame.

Several of these questions were inserted into the ToR a year ago and I have been asking for confirmation since last October so it doesn’t seem unreasonable to know by now for the two questions on the API endpoints, and returning the original/unstandardised XML?

@siemvaessen RE D1.3 - thanks for the info - I am guessing that there will be an equivalent to stream=true, or that the API will allow parameters e.g. start=100 or tokens or similar so that the full return can be accessed?

If the answer is ‘no’ then I guess there is some sort of a process planned i.e. identifying the affected users, discussing a timeline for the changeover? I guess there would also have been a discussion of the pro’s and con’s when the decision was taken which may include some alternative solutions for existing users that we can publicise as guidelines? Are you able to share more about that?

Mark Brough
Mark Brough

Kate thanks a lot for the reply! While I appreciate you may not have any more information for us right now, I do want to second matmaxgeds concern – this is something we have been raising for over one year and have never actually had an answer to.

My assumption was that that radio silence meant that the new Datastore would ensure that existing systems using IATI data via the Datastore would not break. For example, the Bangladesh AIMS has been importing IATI data for about 3-4 years now using the IATI Datastore. I’m not involved directly with it anymore, but I know that it is still working and frequently importing/updating data via IATI. This is great, and something that we should be encouraging, rather than breaking systems at country level and forcing them to spend money and conduct procurement to get the system working again. Probably in some cases, the country will just give up.

So I really do want to emphasise the importance of getting this right and maintaining compatibility with existing endpoints to ensure we are not quite significantly harming efforts to use the data at country level. We really do need to have stable and reliable central infrastructure if IATI is going to work.

@siemvaessen, I think forced pagination will also be a problem, because that is not the way the existing Datastore works.

UPDATE: I also just remembered that this was included as a recommendation in the technical audit:

Where possible, maintain consistency in API endpoints between the old Datastore and the new one, to avoid breaking existing applications.

Andy Lulham
Andy Lulham

If there could be some guarantee that the existing datastore will remain live for some minimum period alongside the new datastore, that would probably make life easier for everyone (users of the existing datastore; the new validator/datastore contractors; the secretariat). Then there’s no deadline day for the “big switch”, but instead a period of time in which systems using the existing datastore can make the switch to the new one.

At the risk of getting a bit technical… Current datastore routes live at:
datastore.iatistandard.org/api/1/ (where the “1” refers to version 1 of the API)

The new datastore could therefore live alongside it, at:
datastore.iatistandard.org/api/2/

It could use semantic versioning (i.e. 2.0.0), but the crucial bit is: it should co-exist with version 1, at least for some predefined period. That’s the usual way for technical services like this to work. When a new version is released, it doesn’t just replace the old version. It sits alongside it for a period of time, in order to give users time to migrate.

Versioning the API like this would also set a good precedent for future datastore upgrades.

Kate
Kate

Yes, as Andy says we can run the two side by side for a period.

As Siem mentioned the datastore has a dependancy on the IATI validator tool, next week we are all getting together to discuss timelines for both tools so we can have a launch plan. It will not be launched in May because the validator won’t be built by then.

The validation contract was only signed three weeks ago, so we’d appreciate some time to meet face to face with the vendors of these core products so that we can get all the details in order., as Andy says we can run the two side by side for a period.

As Siem mentioned the datastore has a dependancy on the IATI validator tool, next week we are all getting together to discuss timelines for both tools so we can have a launch plan. It will not be launched in May because the validator won’t be built by then.

Gele Mohamed
Gele Mohamed

Hi Kate

Thanks for your feedback.

Somalia is currently developing Aid Information Management System which is having an IATI data import feature. The UNDP support for the government in development of AIMS will end in July 2019. we hope that IATI datastore changes will not cause any fault to our system.

My colleague Matt mentioned that Somali AIMS is coping the old IATI registry and dataset and parsing all files to make them Somalia related information. he repeatedly said to me that we need to get in touch with IATI Technical Team for further discussions about the possible change of current API.

It would great that we are clear with everything ahead of time so that we keep our development smoothly.

Looking forward to hear from IATI Technical Team after Easter.

Gele,

matmaxgeds
matmaxgeds

Hi Kate , that sounds like a sensible plan.

RE technical questions, looking forward to hearing after Easter then. In the absence of any information, we will proceed on that basis that the new datastore will fully meet the specifications as outlined in the ToR. UNDP are funding our programmer in Somalia until the end of July, so any changes to the datastore (api base path, api routes, pagination, delivery of unstandardised data) that we are not aware of by the end of May means that the IATI aspects of the Somali AIMS will break when the full changeover happens, no matter how long the transition period.

matmaxgeds
matmaxgeds

@siemvaessen sounds like a plan - will get our dev to input shortly.

IATI Technical Team I would also second the call here for some kind of monitoring so we can start to work out who all the other users might be - is this kind of thing part of the data use strategy i.e. there must be a document somewhere that breaks down the data use a bit?

Mark Brough
Mark Brough

@siemvaessen – great, thanks! I will add into the document the API calls used by the Bangladesh AIMS IATI importer.

IATI Technical Team – +1 on matmaxgeds ’ suggestion. While it would make me very happy if we didn’t break things that I use / have worked on, I am also a bit concerned about breaking the tools for other users we are not aware of, or for users who are less present on these kind of forums. So I think it would be really great if we could use analytics or monitoring tools to see which other routes are popular.


Please log in or sign up to comment.