We would like to invite you to participate in a consultation on the Terms of Reference for a new IATI Data Store
In realising IATI’s vision that “transparent, good quality information on development resources and results is available and used by all stakeholder groups” it is the responsibility of the IATI secretariat to ensure a reliable aggregated flow of all data published to the standard is accessible. We need to provide a robust, timely and comprehensive data service which can be used by developers and data scientists to produce information products tailored to their specific needs.
The current datastore, developed in the early days of IATI, is not fit for purpose and needs replacing.
At a meeting of the IATI Governing Board in January this year it was agreed that it was now a priority to put a reliable and sustainable data service into place. The developer workshop held in Manchester in January had a brainstorming session on the desired functionality. This has formed the basis of these ToRs.
As our Technical Team is currently stretched (over and above its day-to-day commitments) with the consolidation and re-design of IATI’s many websites and web-delivered services, we advised the Board in its March meeting that we should consider outsourcing both the build and initial maintenance of a new datastore.
Your comments on our draft document are most welcome. If you could manage this in the next two weeks we would be very grateful.
Please add comments relating to specific text in the document itself, but add more general feedback in this thread.
@herman
On Ad 1:
I feel this is derailing the main objective of a DS ToR tbh. Stating supplier preference is something we should avoid at this stage. Lets keep this exercise very agnostic and functional.
Ad 1 is based on to either accept all data or not. Lets talk about acceptance criteria first then before talking about the many validators out there.
On Ad 2
I believe the ToR has been produced with the user need in mind (Userstories), or that’s my understanding when reviewing the ToR. I see the user stories definitely need more work, but they are in essence build on the user needs.
Perhaps as part of the RfP it should require an analysis phase with different types of users panning out user stories that may use DS, rather than speccing all the user needs upfront.
And what is a ‘Data Mart’?
On Ad 3
On Ad 3 I agree about re-using software. But re-using for the sake of re-usability does not make sense to me.
I am not too sure either about rallying around pyIATI w/o any real-life scenario’s on how pyIATI is currently being used by IATI Publishers and consumers alike. I am not aware of any platforms/tools that currently make use of pyIATI.
I also agree with the scoping the technical architecture that would allow for multiple vendors to take part in an eco-system of IATI data services, with the IATI technical team leading the technical architecture effort, oversight and overall strategy for technical development on the short and long term.
Ad 1:
The wording was too strong (‘possibly’ should be ‘e.g.’). Discussion should indeed be about requirements and acceptance criteria.
Since the consensus seems to be that that data validation is not a part of the data store functionality, this discussion is not directly relevant for the data store requirements. What is important though is to make sure that bad quality data are not fed into the data store and that we have clear criteria what constitutes bad data.
Ad 2:
A ‘data mart’ is a well known data warehouse component, which enables high volume data queries with flexibility and good response times.
Ad 3:
Agree, reuse should not be done for the sake of reuse. It should be done to avoid duplication of effort, limit throughput times and reduce costs. Especially important given the limited IATI budgets.