This post provides an update on the work to migrate IATI Standard content from IATI’s old Reference site (now available here) to iatistandard.org as announced in our previous posts. Many thanks to all the feedback from the community on this project.

What changes have been made?

IATI’s website has been updated to include information on the rules and guidance of the IATI Standard. Content previously held on IATI’s old Reference site has been added to iatistandard.org, ensuring all information about the IATI Standard can be easily accessed in one place.

All IATI Standard information now on iatistandard.org

  • Technical documentation on all components of the IATI Standard (elements, attributes, rulesets, upgrades etc.) can be found in the IATI Standard section of this website. Do note, as the IATI Standard version 1 was deprecated in 2019, all version 1 pages are kept as an archive within this section.

  • All guidance material (including setting up IATI accounts, how to publish data, Standard guidance and developer documentation) can be found in the Guidance and Support section of the website.

Old IATI Reference site

  • The old IATI Reference site has been archived and will be turned off in 6 months time on 3 February 2020.
  • However, from today users who access old links to the IATI Reference site, will be redirected to the new pages on iatistandard.org.

Users should note that no information on the IATI Standard that has been migrated from the old IATI Reference site has changed, however it has been reordered and re-structured on iatistandard.org. Updates have been made to content in the Guidance section of iatistandard.org, following consultations with the community. These changes will make it easier and more streamlined for publishers and data users to find and understand information about the IATI Standard.

Additional Technical changes: IATI - GitHub

There have also been a few behind-the-scenes changes on IATI-GitHub to provide more flexibility in how the Secretariat can use and display IATI Standard content.

Changes:

  • From IATI/Extra-Documentation we have moved:

    • The updated overview pages to the IATI/Guidance repository
    • The codelist-api.rst page now sits within IATI/Developer-Documentation
    • The codelist-management.rst page sits within IATI/Upgrades
  • From IATI/Guidance we have moved:

    • The old datastore documentation to the IATI/Developer Documentation repository
    • The old introduction pages have been archived as these are now contained within Wagtail CMS as per the first website launch
    • The IATI upgrade pages to the IATI/Upgrades repository.

These changes provide the Technical Team with the required level of version control for each page type. For example, we only need one copy of each IATI standard upgrade page. These pages do not differ between the IATI versions.

Next steps

In the coming months we will continue to undertake formatting fixes and optimisations on iatistandard.org. These will happen both on the website and behind the scenes.

We’re committed to listening to any feedback so please email any questions about these changes to support@iatistandard.org.

Comments (32)

Andy Lulham
Andy Lulham

It looks as though the codelist API has moved as a result of this change. Most of the API URLs listed on that page now return a 404.

For example: previously, this URL returned the sector codelist as JSON:
https://iatistandard.org/203/codelists/downloads/clv2/json/en/Sector.json

It looks as though this has now moved here:
https://iatistandard.org/downloads/203/codelists/downloads/clv2/json/en/Sector.json

Would it be possible to add the relevant redirects? This is currently a breaking change for services that rely on the codelist API.

Also worth noting that the link to the codelist API is now missing from the codelist page (I suspect this means all links to developer docs have been removed?) In any case, it would be good to restore this link.

Thanks,

IATI Technical Team
IATI Technical Team

Hi Andy, absolutely! We will look into this today.

Please do let us know if you find any more redirects or suggestions for areas we could better link the content due to the new structure.

Andy Lulham
Andy Lulham
Image removed. IATI-techteam:

Please do let us know if you find any more redirects or suggestions for areas we could better link the content due to the new structure.

Yes, sure.

A couple I’ve noticed are:

Amy Silcock
Amy Silcock

Update about the Codelist API links from one of our devs:

Hi all,

Apologies for any interruption of service you may have experienced. If you need access urgently, please use the backup domain we set up at archivereference.iatistandard.org .When we made the migration from static HTML to Wagtail, all files referenced in the content were programmatically downloaded, uploaded to the Wagtail CMS, and made available with redirects from their old paths.

For e.g. http://iatistandard.org/201/codelists/downloads/clv1/codelist/ActivityDateType.csv Where this seems to have fallen flat is that not all needed links are actively referenced in the content (most of 1.04 and 1.05 download files for e.g.)

Additionally, what wasn’t anticipated was the need for users to see Apache2 style download folders of files for download. We will need to remove those files from the Wagtail CMS, and instead manually host them to imitate this previous behaviour.

Andy Lulham
Andy Lulham
Image removed. amys:

all files referenced in the content were programmatically downloaded

Right, yes – I see. That approach won’t work for an API, where most URLs aren’t referenced explicitly. Although… I think the codelist API had already been moved from reference.iatistandard.org to iatistandard.org?

I think there are some other problems here, too, but they’re hopefully not too painful to iron out. Here’s an example redirect path for a codelist:

I think redirect 3 is new, and it looks like it’s the problematic one – it incorrectly strips the “en” language code from the URL.

Additionally, I’m not sure where the codelists.json and mappings.json files live now?

Alex Miller
Alex Miller

Hi all,

Implementing a fix right now. You might experience some additional 500 or 404 errors while the fix is going live.

Once the fix is live, I’ll leave another update. Unfortunately some of the redirects are 301 permanent so you may have incorrect endpoints cached even after the fix is live.

Alex Miller
Alex Miller

Hi Andy Lulham and matmaxgeds

I’ve reworked the SSOT import to work around the Wagtail Document interface.

Indexed all downloads should be available here: https://iatistandard.org/reference_downloads/

And all redirects should properly point to the correct file now.

I think there are still some additional links that Amy Silcock and I will need to work together to fix throughout the RST content. But hopefully these fixes should allow your services that depend on the SSOT to continue functioning.

Please let me know if you spot anything else that needs fixing.

Alex Miller
Alex Miller

Hi shi ,

I believe not listing 1.04 and below is a content decision from the IATI BAs, but they will be able to confirm tomorrow morning.

https://iatistandard.org/reference_downloads/ will be stable, and I’ll try my best to keep redirects from old paths up to date.

As for performance, I could look into some static file caching on the NGINX side, but we could run into a cache-clearing issue for updates. reference.iatistandard.org was proxied via CloudFlare, and I’ve just turned on the proxy for iatistandard.org as well. Hopefully this provides you with a bit of a faster response in the meantime.

shi
shi

Thanks, Alex Miller , it does seem faster today.
Just thought we should mention the download time as it was a noticeable change.

We can also confirm that all codelist API redirects are working from our end and will use the new endpoints.

shi
shi

Thanks for clarifying, Amy Silcock .

It would be of use from the perspective of historical documentation and official source.

A generic ‘version 1’ link would be appreciated, thank you.

However, the changelogs for version 1.5 and below are currently showing up as error pages. Looks like the url is missing ‘/v1-upgrades/’.

shi
shi

Thanks, Alex Miller

We rely on the redirects as part of d-portal’s nightly update. Should we change to the new urls; ie. are these going to be stable?

This downtime has affected us and we haven’t been able to do a full import for the last 5 days. We’re currently checking to see if everything is still working; ie. redirects are all working.

We notice that the json downloads are much slower than they used to be before this change. Is this intentional?

We also notice that version 1.04 and below is not listed on https://iatistandard.org/en/iati-standard/ under ‘Previous versions’. Is this as designed?

Thanks!

Andy Lulham
Andy Lulham
Image removed. alex_miller:

Added some of the source from the Django Common middleware into our custom middleware to try and ensure trailing slashes are always appended for non-file URLs. It should all be fixed now.

LGTM! Thanks for fixing.

Alex Miller
Alex Miller

Cheers Andy Lulham ,

I’m not sure why certain page depths were not appending the trailing slash (/203/codelists would append, but /203/codelists/sector wouldn’t).

Added some of the source from the Django Common middleware into our custom middleware to try and ensure trailing slashes are always appended for non-file URLs. It should all be fixed now.

matmaxgeds
matmaxgeds

Thanks - the links that formerly went directly to the pdf’s now go to a page where the pdf’s can be downloaded - I am not sure I understand what is happening here - (could the links for each file not just go to the pdf files themselves?) but great that access has been restored in general

Amy Silcock
Amy Silcock

Hi Matt, I’ve redone the links so now they you can directly download the file.

Not sure what happened with the links. There was previously a broken link taking the user to the one data use research page.


Please log in or sign up to comment.