Thanks Bill for sharing these specs with us. Reading through the specs and all the comments already presented I see 3 crucial topics being discussed:
1 - Should the DS accept all publisher data or not? And if not, what are the acceptance criteria. Derived from this question is the question if an IATI data validator should be part of the datastore (DS) or not.
2 - Should the DS limit the users how often, how much and what they can search on? This question seems to be driven from perceived technological limitations.
3 - The requirement that all software developed for the IATI validator, should be written in pyIATI with as assumption that in the end the IATI technical team will maintain all IATI core software products.
Since the existing IATI data quality is one of major obstacles for using IATI data in practice, I think the answer on the first question should be no. This implies that the definition of the acceptance/rejection criteria should be a part of the design of the DS. It is argued by several people that data validation should not be part of the DS, to keep its functionality ‘lean & mean’.
Therefore I suggest that a separate data validator should be developed, possibly reusing the software already being developed by Rolf Kleef for a number of organizations, including the Netherlands MFA and the DFID.
Although I understand the inclination of the more technical people to limit the technical complexity of the DS, i.m.o. the user needs should drive the design process. Technology should follow (and not the other way around). We are not talking ‘big data’ here and there is a lot of experience in the market to design flexible and robust Data Marts.
When we want to have a chance to achieve any results the next 12 - 18 months, I think we should reuse as much existing software as we can. Rebuilding everything from scratch with a yet unproven pyIATI library, would preclude the reuse of any existing software components (e.g. the Data Validator software mentioned under point 1).
I think we should seriously consider if all IATI core products in the long term must be maintained solely by the IATI technical team. In a loosly coupled ecosystem of applications/software components, it is i.m.o. quite possible to have multiple vendors being responsible for the maintenance of IATI core products. The most important crucial responsibility of the IATI technical team in such a scenario, would be to define and guard the overall conceptual and technical architecture, especially with regards to the interfaces between the software components. This would have the additional benefit of limiting the resources needed from the IATI technical team, which is already overburdened as it is.