Good spot. I’ve updated the links on both these pages:
Right, yes – I see. That approach won’t work for an API, where most URLs aren’t referenced explicitly. Although… I think the codelist API had already been moved from reference.iatistandard.org to iatistandard.org?
I think there are some other problems here, too, but they’re hopefully not too painful to iron out. Here’s an example redirect path for a codelist:
- Request: http://reference.iatistandard.org/201/codelists/downloads/clv3/csv/en/ActivityDateType.csv
- Redirect 1: http://iatistandard.org/201/codelists/downloads/clv3/csv/en/ActivityDateType.csv
- Redirect 2: https://iatistandard.org/201/codelists/downloads/clv3/csv/en/ActivityDateType.csv
- Redirect 3: https://iatistandard.org/downloads/201/codelists/downloads/clv3/csv/ActivityDateType.csv
- Redirect 4: https://iatistandard.org/downloads/201/codelists/downloads/clv3/csv/activitydatetype.csv
- 404 response.
I think redirect 3 is new, and it looks like it’s the problematic one – it incorrectly strips the “en” language code from the URL.
Additionally, I’m not sure where the codelists.json and mappings.json files live now?
Great – thanks. It’s still non-trivial to reach the API docs, since the links to the individual API pages point back to datastore page again. I guess they should now be relative or absolute links to github instead.
Implementing a fix right now. You might experience some additional 500 or 404 errors while the fix is going live.
Once the fix is live, I’ll leave another update. Unfortunately some of the redirects are 301 permanent so you may have incorrect endpoints cached even after the fix is live.
I’ve reworked the SSOT import to work around the Wagtail Document interface.
Indexed all downloads should be available here: https://iatistandard.org/reference_downloads/
And all redirects should properly point to the correct file now.
I think there are still some additional links that @amys and I will need to work together to fix throughout the RST content. But hopefully these fixes should allow your services that depend on the SSOT to continue functioning.
Please let me know if you spot anything else that needs fixing.
We rely on the redirects as part of d-portal’s nightly update. Should we change to the new urls; ie. are these going to be stable?
This downtime has affected us and we haven’t been able to do a full import for the last 5 days. We’re currently checking to see if everything is still working; ie. redirects are all working.
We notice that the json downloads are much slower than they used to be before this change. Is this intentional?
We also notice that version 1.04 and below is not listed on https://iatistandard.org/en/iati-standard/ under ‘Previous versions’. Is this as designed?
Codelist API redirects are all working for me – thanks for resolving this.
I believe not listing 1.04 and below is a content decision from the IATI BAs, but they will be able to confirm tomorrow morning.
https://iatistandard.org/reference_downloads/ will be stable, and I’ll try my best to keep redirects from old paths up to date.
As for performance, I could look into some static file caching on the NGINX side, but we could run into a cache-clearing issue for updates. reference.iatistandard.org was proxied via CloudFlare, and I’ve just turned on the proxy for iatistandard.org as well. Hopefully this provides you with a bit of a faster response in the meantime.
We decided to slightly hide version 1.04 and below. Most of the IATI users and publishers don’t need it.
We could change the version 1.05 link to a generic ‘version 1’ and link here: https://iatistandard.org/en/iati-standard/upgrades/how-we-manage-the-standard/versions/ ?
If the trailing slash is omitted from a codelist page URL, the server now responds with a 403 error. For example:
I think that’s a bug. Other pages on the site don’t appear to behave that way.
Is there a better place to flag such bugs? Presumably https://github.com/IATI/IATI-Standard-Website/issues is now the best place to flag reference site issues?
That would be great, yes please Andy
Thanks, @alex_miller, it does seem faster today.
Just thought we should mention the download time as it was a noticeable change.
We can also confirm that all codelist API redirects are working from our end and will use the new endpoints.
Thanks for clarifying, @amys.
It would be of use from the perspective of historical documentation and official source.
A generic ‘version 1’ link would be appreciated, thank you.
However, the changelogs for version 1.5 and below are currently showing up as error pages. Looks like the url is missing ‘/v1-upgrades/’.
Thanks Shi, I’ll get these working.
I’ve raised an issue to get a generic version 1 link. This will require a little longer as it needs a template change.
Committing to support these new routes will serve to increase technical debt. It might be preferable to unofficially support this (i.e. redirect there), but officially recommend the URLs listed on the codelist API page. Just a thought.
I’m not sure why certain page depths were not appending the trailing slash (/203/codelists would append, but /203/codelists/sector wouldn’t).
Added some of the source from the Django Common middleware into our custom middleware to try and ensure trailing slashes are always appended for non-file URLs. It should all be fixed now.
LGTM! Thanks for fixing.
I see the dropdown arrows on the archive reference site are all broken:
This is a perennial bug – I sent a fix for it 2 years ago. I’ve just re-sent it:
@IATI-techteam - I just tried to download something from this page but the Rwanda link to the pdf was broken: https://iatistandard.org/en/news/spotlight-on-iati-data-use-rwanda/
Thanks @matmaxgeds, we’ll look into it.