Please, no magic! The magic solutions never work. They build false expectations. And then, people change their mind and want them in blue.
… so that a sustainable tool can be developed that meets market need?
To even begin to get to this, I think we need to answer: why is the current CSV2IATI tool not to be sustained?
I can offer three reasons:
1 - It hasn’t been maintained
The main reasons to switch off CSV2IATI, seems to be:
feedback from users tells us that CSV2IATI is struggling to cope with converting files from an increasing number of IATI publishers, with a rapidly expanding volume of data. Additionally, the tool was built in such a way that it is now difficult to update it in line with changes to the IATI Standard.
Some context on this might help us:
- CSV2IATI was built five years ago by @markbrough , as a quick response to a user need to “convert data from spreadsheets”. This was - as in all best efforts - build on code already openly available
- Looking at the code activity on the CSV2IATI repo, there was a spike of activity in the initial development, and then some patching up by @bjwebb (whilst at the Secretariat) in 2014.
- CSV2IATI has had - according to these logs - no major development for three+ years. The latter commits from @hayfield & @rory_scott are about decommissioning it
So, whilst it’s fair to say that the tool presents difficulties, a reality is that it has not had any active development to address these.
(it should be noted that the minutes of the Oct 2014 Steering Committee state that it would be 2.01 compliant by December 2014).
2 - It’s been made available for free, with free support
In relation to:
Has online guidance and 1:1 support
CSV2IATI has been made available for free, with free support to publishers, by the secretariat / @IATI-techteam. To be clear: this has been great to help IATI adoption growth. But, in the context of sustainability it’s difficult for anyone else to step in and offer universal access and free support
A sustainable tool needs a team of people to support, maintain, document and enhance it. If not - and further compounded by the complexity of IATI-as-spreadsheet - then people can get things wrong, produce poor quality data, and then blame the tool.
It looks like these first two observations might be linked
3 - It might be the wrong answer to an irrelevant question
This is a bit more difficult. With our developments of the CoVE codebase, we’re actively developing how one format (csv/xslx) can be transformed to another (xml), and backwards (xml > xlsx) too. I’ve even given a brief talk at the Members’ Assembly about the value of spreadsheets to IATI publishers (yes, with cats)! So - don’t read the following as anti-spreadsheet!
However. the one-to-many dynamic of IATI means that maintaining lots of data in a single flat CSV sheet, for conversion into IATI XML, is really really hard for most humans I know. CSV2IATI presented one approach to that - but in turn added several complexities.
In reality - and I think people like @Herman & @theo.sande would agree - if you plan to publish data on hundreds / thousands of projects, then complete reliance on a “free online tool” that requires a csv file input isn’t going to be a sustainable publishing strategy.
To this end, I disagree with the specification posted by @amys, and don’t subscribe to the assessment of other solutions not meeting this. Sorry.
User story interlude: I recently worked with IFAD. They were in CSV2IATI world. This was painful. They now automatically publish IATI data out of their IT systems, and can now see how they can include a far wider range of data points. This isn’t painful. This would just not have been possible with CSV2IATI, or any other spreadsheet>XML setup.
Interestingly, however, we used CoVE to model some sample XML, in order to provide the development team with ideas around how things could look. The idea of quickly prototyping IATI, perhaps by someone non-technical, via spreadsheets, does seem to be something that can fit into the workflow of many organisations.
At Open Data Services Co-operative, we are working on CoVE, and using it with various organisations. We have both a command line and GUI version in operation. We’re finding (also via work with organisations publishing with the open contracting and 360Giving standards) that workbooks (spreadsheets with tabs!) are more intuitive to people than flat CSV files. BUT - we’re learning as we go along, and actively developing in tandem. For example, by enabling CoVE to work in the context of the Open Ag Toolchain @reidmporter mentions, we opened up a whole piece of development around ruleset testing, building on @andylolz’s fantastic work.
But, we haven’t a magic solution.
And we’re very sensitive about sustainability.