Download everything from the Registry

Amusingly, this torrent still exists, but a) it’s from 2016 and b) it’s only a snapshot, but [~379] might be happy to put another torrent online .

I imagine pyIATI has something to do this, but I’m not sure how long it takes to sort out python environments etc. I’m fully expecting Andy Lulham or Herman van Loon to have some magic for this.

Serious suggestions though:

If you don’t mind the known issues with the IATI Datastore, or the existence of some wrapper XML including query metadata, this (theoretically) should do the trick as an ultra lazy approach: http://datastore.iatistandard.org/api/1/access/activity.xml?stream=true.

But if you want the exact published xml, this is possibly still the simplest / lowest friction way of downloading all of the datasets in their raw form: https://github.com/IATI/IATI-Registry-Refresher as far as I know.

John Adams I made this proof-of-concept yesterday evening: https://andylolz.github.io/iati-data-dump/

It provides all IATI data on the registry – unfettered and unfiltered – in a single HTTP request.

It’s a big zip file (currently ~350MB) that’s auto-updated daily. It’s made using IATI Registry Refresher that Rory Scott mentions above, and is hosted on dropbox.

I’ll try and make it look a bit more presentable, but let me know if that’s useful.

Currently I download IATI XML data by using Pentaho Data Integrator on a file by file basis for a defined list of approx. 140 publishers, making use of the URL’s from the registry.

Hi,

Is there any push to have either Andy Lulham ’s IATI wonderful data dump, or something identical (with Andy’s permission, provided by the new datastore and therefore adopted as an official IATI product?

Thanks,

Matt