Examples of well-implemented IATI extensions?

(Rory Scott) #1

Quick question to all the wonks - can anyone show me an example of a well-executed extension of IATI data the isn’t the United States? Their namespace link isn’t working: http://beta.foreignassistance.gov/assets/Organisations/DOS%20and%20USAID.xml (I’m aware this might change, but am impatient).

(Further question - is there any way of finding these activities with extensions through the datastore, dashboard, OIPA, or D-portal? My suspicion is that there isn’t, but I’d be interested to hear.)

(Hayden Field) #2

As something potentially useful that I can think of… The elements page on the dashboard includes information about non-standard namespaces.

A quick look through indicates that most of the provided namespaces resolve to 404, though it could act as somewhere useful to start from.

(Vincent van 't Westende) #3

As to the question to OIPA; We indeed don’t support namespaces and have no plans to support them in the master of OIPA.

There’s certainly advantages for namespaces when used in certain groups of publishers, but there’s too much interpretation to be done to be able to standardize namespaces into a generic application. A customized branch to address that interpretation and its specific use cases would be a better option.

As to examples, I only know of RAIN foundation and Akvo that use namespaces. Maybe @mato74 can tell you more on Akvo’s use.

(shi) #4

Hi @rory_scott - we’ve recently implemented this during the OpenAg tool workshop but didn’t have time to talk about it.

Not sure if this helps but we’ve added element search and from your link, we can do either:

http://d-portal.org/q.json?from=element&orderby=element_volume-&limit=10&element_name0_like=usg:%

or

d-portal.org/q.json?from=element,act&orderby=element_volume-&limit=100&reporting_ref_nteq=US-1&element_name0_like=usg:%

You can go deeper into elements hierarchy like so:

d-portal.org/q.json?from=element&orderby=element_volume-&limit=10&element_name0=dimension&element_name1=target&element_name2=period&element_name3=indicator

For more info, I’ll be publishing a blog post about this next week on d-portal.

ps: This is new and may likely crash or fail to work.
pps: Hyperlinks are borked due to discuss platform limiting new users to one link for first post :}

(Rory Scott) #5

Thanks everyone, this has been really helpful. I think it’s highlighted two things to me:

  1. I’m still not completely clear on how I would go about making an extension, and it doesn’t seem like there are any obvious examples to draw upon.
  2. It doesn’t look as though many (or even any?) publishers are using them correctly as of right now.

If anyone has any further insights please post them here :slight_smile:

(Marten Schoonman) #6

@VincentVW thanks for flagging this.

@rory_scott you might want to have a look at this page: https://github.com/akvo/akvo-rsr/wiki/IATI-Import-mapping what the use of these in practice look like you can find on e.g. https://rain.akvoapp.org/en/projects/ (The Rain IATI file imports in Akvo RSR that Vincent referred to).

Please let me know if you have any questions on this. Regarding suggestions for enhancing the standard in this respect please see this post: Tech Paper: Codelist Management

Last but not least; why would you get involved with these types of fields? That has everything to do with the storage of data for communication purposes. See this presentation for more context: http://www.slideshare.net/martenschoonman/using-iati-datasets-for-communication-where-can-we-improve

(Rolf Kleef) #7

I can find a couple of namespace declarations:

A namespace declaration is supposed to contain a URI, not a URL, so you’re not required to provide a web page (but it’s recommended).

So xmlns:fts="nothing" and xmlns:ns1="xml" are wrong, but the others are technically ok. Although for instance both variations of ir or fts or usg are considered to be different namespaces, even if that wasn’t intended.

I internally use my own xmlns:iati-me="http://iati.me/..." variations in pipeline processes but (try to) make sure it doesn’t show up in published files. At the end of everything, I should be able to remove every attribute and element from none-IATI namespaces, and end up with valid IATI.

If you want, you can add invoices as UBL data to your expenditure transactions: <ubl:Invoice xmlns:ubl="urn:oasis:names:specification:ubl:schema:xsd:Invoice-2" ... and then provide an interface looking for //iati-actiivity[transaction[@type="4"]/ubl:Invoice.

I think the difficulty is that we don’t specify interpretation of IATI data:

If Akvo publishes (leaving out the narrative layer):
<description type="1" akvo:type="4">Subtitle</description> <description type="1" akvo:type="5">Project plan</description>

Can I can concatenate those as <description type="1">Subtitle&#xa;Project plan</description>?

(Rory Scott) #8

Thank you @rolfkleef and @mato74 - I see what mistake I’ve made: I’ve been thinking that the reason these files are valid is that their namespace URI is also a URL which points to an actual schema extension. Not so. Even though the files which use this method will validate via the IATI Validator, they don’t actually strictly validate if you use something like xmllint -valid <file> -schema <iati-activities.xsd>:

### Validate agains the XML standards and the IATI activities schema
xmllint -valid http://iati.openaid.se/xml/189.xml -schema /IATI-Schemas/iati-activities-schema.xsd -noout
# -valid: validate this file
# -schema: fetch the xsd file from this address
# -noout: don't print the whole file to console afterwards!

######## Output ########
# http://iati.openaid.se/xml/189.xml:1: validity error : Validation failed: no DTD found!
# <iati-activities version="2.01" generated-datetime="2017-03-17T13:53:14">
#                                                                           ^
# http://iati.openaid.se/xml/189.xml validates

Similarly, when the namespace is declared at the top of the file: validator, versus terminal:

xmllint -valid http://beta.foreignassistance.gov/assets/IATI/Argentina.xml -schema http://iatistandard.org/202/schema/downloads/iati-activities-schema.xsd -noout

######## Output ########
# http://beta.foreignassistance.gov/assets/IATI/Argentina.xml:1: validity error: Validation failed: no DTD found!
# /web/IATI/usg-extension" generated-datetime="2017-03-22T23:18:00" version="2.01"
#                                                                               ^
# http://beta.foreignassistance.gov/assets/IATI/Argentina.xml validates

So the questions is, is this a problem? If not, I could happily just get around validation of an extension in this way, but it doesn’t seem ideal. For instance if I was extending a transaction to have a new date value within it, I would prefer to be able to extend the IATI schema so that when I run a validation procedure, I can tell if I’ve violated the original IATI specification or my own extension within one validation…

(Rolf Kleef) #9

Use xmllint without the -valid switch: that switch lets you specifify a DTD rather than an XSD to define the schema. Only --schema is needed, with the file to validate as input:

xmllint -schema http://iatistandard.org/202/schema/downloads/iati-activities-schema.xsd -noout http://beta.foreignassistance.gov/assets/IATI/Argentina.xml http://beta.foreignassistance.gov/assets/IATI/Argentina.xml validates

(Rory Scott) #10

Ah right, got it - it still seems a little unsatisfying to me, but I see the logic. I really appreciate your time, thanks!