Introducing the IATI Python Library

(Dale Potter) #1

Our recent post on IATI Systems Design & Architecture outlined the long-term plan to migrate core IATI products to a more resilient and sustainable technology setup. A key part of this new architecture is the development of a Python library - this library will help the IATI Technical Team, as well as tool builders and software developers to more easily work with IATI data and build end-user tools.

Purpose

The Python library will contain IATI functionality that is currently replicated across many of our software projects - as shown by the number of standalone tools in our current architecture map. Alongside our tools, there are an numerous third-party tools (including OIPA, IATI Studio, AidSight and many others) which also reproduce very similar functionality (e.g. fetching and loading files, data validation, extracting data, etc).

The IATI Technical Team will use this new library, with core IATI services refactored over time to use the library as a base for common tasks. For others, it is intended that the library will make it easier to use IATI data by removing the need to reinvent the wheel before moving onto implementing innovative functionality, performing analysis of data or simply accessing data for data visualisations.

The library will be distributed publicly using the PyPi python package index, therefore allowing software developers to install using the pip package manager.

Long-term functionality

Current plans for library components include:

  • iati.core - will allow programmatic access to SSOT constructs - i.e. the schema, codelists, rulesets

  • iati.validate - an API to validate IATI XML against the schema, codelists and rulesets. The new validator API and web user-interface will heavily use this functionality

  • iati.fetch - functionality will be included to download individual raw IATI XML datasets (i.e as a wrapper around the IATI Registry API) links to queryable data (i.e. as a wrapper around the Datastore API)

  • iati.convert - (under exploration) conversion between versions of the IATI Standard or from XML to CSV

Current development

The IATI Technical Team been working to develop the first stages of this library for a few weeks, focussing on iati.core.

Following overwhelming interest and discussion from those we have spoken to directly, we are today making this project open source: https://github.com/IATI/iati.core

Current usage examples are provided in the project readme. However, please note, at present the contents of this library is at best unstable, and much is experimental. As such, it must be expected that its contents and API will change over the short-to-medium term future. Warning sections in code docstrings help flag up some particular known stability concerns. Todo sections describe known missing or incorrect features.

At present the library (core) represents the contents of the IATI Single Source of Truth (SSOT). It is able to handle a single version of the Standard (2.02 by default, although it is possible to load any IATI XML dataset).

It is planned that different sections of the library, such as validate are split into their own repositories. They exist within this repository at present to help speed up the iteration process.

This code is provided for illustrative purposes, as we are aware that some developers are about to imminently begin work on new applications to work with IATI data. We hope that an insight into our current development will inform technology choices, encourage future adoption of the library and prevent developers from again needing to reproduce common functionality, therefore saving time.

Developer feedback and contributions welcome

Feedback, suggestions, use-case descriptions, code reviews, bug reports and so on are much appreciated - it’s far better to know of issues earlier in the development cycle. Please use the code repository’s Issues page for this.

Furthermore, if the direction of the library suggests development time can be saved on your projects, we would encourage you to use some of the time saved to be used to contribute to future development - please post below if you are at all interested in helping this library, and we can follow up direct.

1 Like
New Somali AIMS - IATI aware developers needed!
State of pyIATI?
Update on the Roadmaps for IATI Technical Team Products (August-December 2017)
Our shared understanding of what IATI is, and is not,
Roadmaps for IATI Technical Team Products
Generating a list of all IATI file URLs
(Andy Lulham) #2

Awesome! Yeah, getting it on PyPi && readthedocs would be great, but this is already super useful. Thanks!

(Herman van Loon) #3

Great idea! Will make IATI accessible to a large group of Python developers.

(Michael Rans) #4

I think that this is an excellent initiative and will be welcomed by the organisations who wish to use the IATI standard, particularly ones with greater technical capability. This is our experience with the Humanitarian Data Exchange (HDX) where we have received positive feedback from organisations.

(Michael Rans) #5

An example of such feedback on our Python library is “The HDX API Python wrapper is great for our use case and well documented.”

We have found that key to there being confidence in the library is a high level of test coverage and plenty of documentation including API docs.

(Michael Rans) #6

I think that humanitarian organisations using IATI would find it useful and time saving to be able to publish datasets directly to HDX and I would be happy to work with you on that.

I would also like to encourage that we look at interoperability in the area where they converge between IATI and the Humanitarian Exchange Language (HXL).

The above would be very helpful for that interoperability.

Sorry for multiple posts but I am only allowed one link per post!

(johnbritt) #7

Currently all IATI code is Python 2, and in general isn’t Python 3. However, we have started some Python 3 features (e.g. the Python 3 print statement) in new code to make a future transition easier.

Source: http://iatistandard.org/202/developer/code-python/