Discovered the hard way that the IATI dashboards can’t handle xml files larger than 50 million bytes. This is a hardcoded maximum as explained by the IATI support team.
If publishers follow the desired quest in coverage and comprehensiveness sooner or later any file size limit will turn into an obstacle. What’s large today as limit will be tight tomorrow.
Wonder if such size limit stands in ‘modern times’ and in the vision path of IATI. Looking forward to views on this…
In order to have our publishing statistics properly reflected on IATI dashboard and on the Grand Bargain dashboard, we are now advised/forced to split our data set.
Splitting data sets seems an easy fix, it is however cumbersome and not without risks. How to avoid overlap (double activity publication) and assure no gaps. Also one needs to monitor and plan a new split before running into size problems again.
Isn’t it better to adjust the coding and allow for large data files?
(Emphasis mine.) I agree with the above. But I don’t think I’m out on a limb when I say: we don’t currently have a robust datastore. Until it’s improved and provided as a reliable service, I think the filesize limit needs to stay.
Ideally, data users wouldn’t need to interact with XML unless they want to. But at the moment, for bulk data querying, I don’t think it’s avoidable.