IATI Coverage ; timing and scope of next update?

(Michelle Levesque) #10

@stevieflow

I agree with your point about not trying to satisfy GB use case particularly as even the basic IATI dashboard hasn’t addressed coverage in a couple of years and cracking that nut is important. I would still point out that “total expenditure”, if not well defined or understood, could still be very misleading as a denominator in the coverage calculation. Total expenditure for an organization often includes expenditures never intended to be part of IATI publishing (administrative, normative) so without clarity or sub-elements, attributes (I don’t know the correct way to slice that) it may still lead to misleading results.

And just and FYI: the GB dashboard has not yet captured coverage at all. I have no idea if/when this will be addressed in the narrow context of the GB but we all score zero so it is a level playing field for the moment.

(Steven Flower) #11

Thanks @Michelle_IOM

The IATI documentation states:

The recommendation is that, where and when possible, the organisation’s total annual planned budget for each of the next three years is reported.

It’ll be interesting to see if there are any ranges around this definition, as you (rightly) anticipate. Once we’ve got some data in place, it could then be time to review.

A (hopefully interesting!) added bonus with the organisation standard, is that there is the ability to include a/multiple “line item” within any of the financial elements, including total-expenditure (known as expense-line). This might be what you’re thinking about / looking for, in terms of answering the question “what part of the total expenditure was on humanitarian?”…?

I can see that 15 publishers currently use expense-line in total-expenditure, so it’d be interesting if someone (or something) were to analyse this …

(Michelle Levesque) #12

IOM is one of those 15 publishers. We do include line-item info all of which ties to our audited financials. Our data doesn’t reference or even isolate humanitarian from development but it is something we could do if/when coverage statistics want to pull from the org file.

(Yohanna Loucheur) #13

@IATI-techteam, the link from this Dashboard total-expenditure element to the Standard page (presumably describing the element) returns a 404 error.

(Andy Lulham) #14

@YohannaLoucheur: fixed here.

1 Like
(Steven Flower) #15

@bill_anderson any updates on this? The number of publishers without (2014) coverage reference data now far exceeds those with. The knock on is that the coverage-adjusted score is calculated with the arbitrary 20% for many many publishers. This just looks strange, particularly to those new to IATI.

(IATI Technical Team) #16

The IATI technical team did a manual collection of reference spend for 2014 and 2015 from all IATI publishers two years ago. The process involved emailing publishers, collecting data manually and storing in a public Google sheet. This is not a sustainable method for collecting coverage data and the IATI tech team has not done reference spend collection for any new IATI publishers over the past two years. This means that the coverage data and then coverage-adjusted scores on the Summary Statistics are not up to date for all IATI publishers.

As a result, the current plan is to remove the coverage and coverage-adjusted columns from the Summary Statistics page on the Dashboard, keeping the coverage page only for reference, until the process for collecting reference spend is automated and up to date. In the meantime, we strongly encourage that all IATI publishers include in their organisation file total-expenditure.

(Michelle Levesque) #17

How about putting a statistic about which publisher includes total expenditure in the org file on the statistics page? It would highlight whose coverage could theoretically could be calculated (as imperfect as it may be) and may provide an incentive for others to include that data in their org files so we get closer to actually having an automated way of calculating coverage. Assuming people still agree that coverage is important.

1 Like
(Petya Kangalova) #18

Thanks for the comments and ideas @Michelle_IOM
We will be emailing all IATI publishers next week to inform them about the planned changes of removing coverage-adjusted score and would strongly encouraged all to use total-expenditure in their organisation file.

On including total-expenditure in the summary statistics methodology, we don’t have an ETA yet for when that will be incorporated in the publisher statistics, but would keep in mind the idea about highlighting publishers that have already included total-expenditure in their file.

1 Like
(Steven Flower) #19

Thanks @petyakangalova

Have publishers been emailed?

I assume that the methodology to use the total-expenditure would run (at least) a year “behind”. For example:

  • In 2018, coverage data would be based on 2016 expenditure
  • In 2019, coverage data would be based on 2017 expenditure
  • In 2020, coverage data would be based on 2019 expenditure

Because - it seems unrealistic for publishers to have their total-expenditure published (and all their activity data, on the 1st January each year. For many, getting this data together might take a few months. Therefore, it would seem the IATI dashboard methodology would need to accommodate this, in line with regular financial process & timescales.

Great to see this move forward, nevertheless. Thanks.

(Yohanna Loucheur) #20

(At least some) publishers have been emailed, yes.

The core message is as follows:
"In the future we plan to compare the value of the element published in your organisation file with the calculated total spend (expenditure + disbursements) for a given year published in your activity files.

To prepare for this please ensure you are publishing an organisation file which contains the element for at least all years that you have been publishing to IATI."

I’m not sure I understand this last part: “for at least all the years that you have been publishing to IATI”. This seems like a fairly tall order, and not necessarily a very useful one for publishers who have (very partial) data going back over many years.

I assume coverage will be assessed for recent years (probably a year behind, as suggested by Steven). Could the Tech Team confirm what is the proposed approach?

1 Like
(Michelle Levesque) #21

Perhaps another mini-conversation at TAG?

(Yohanna Loucheur) #22

Yes, the TAG would be a good opportunity to make sure we all understand what is being proposed.

However, it would be great if the @IATI-techteam could share more details ahead of time so we can prepare.

(Yohanna Loucheur) #23

Going over the thread again, I’m not sure we fully answered some queries.

Could the @IATI-techteam provide clarifications on the questions above - in particular regarding the period for which coverage would be calculated ie 1) what time lag (2 years, as suggested by Steven) and 2) how far back in time.

thanks

(Andy Lulham) #24

I suppose the detail around this bit is also important, since it seems the methodological complexity has been shifted to here.

(IATI Technical Team) #25

Hi Yohanna.

Thanks for the follow-up and apologies for the delay in getting back to you. Following the tech audit last August, there was no immediate action to work on the IATI dashboard and we have not looked into a new methodology for the coverage assessment.

This is currently on hold so I cannot confirm what period the coverage assessment would cover. The recommendation for the Dashboard was to revisit re-building the Dashboard as we have updated the community in our last blog: https://iatistandard.org/en/news/iati-technical-team-our-detailed-quarterly-update/

In the meantime, I would still encourage using total-expenditure in the organisation file .

(Alex Tilley) #26

Hi IATI Technical Team:

Following up on this I wanted to let you know that we have been looking into this issue at Publish What You Fund.

We looked at the total-expenditure field of the 45 publishers assessed in the 2018 Aid Transparency Index and compared these with the total of expenditures and disbursements for that year. This is what we found:

  1. Most publishers are not using the total-expenditure field to publish annual expenditure. Of the 45 publishers in the Index, 4 have published 2018 total-expenditure data, 7 have published 2017 data and 8 have published 2016 data.
  2. The figure used is often just the sum of disbursements + expenditure recorded in a publisher’s IATI activities, so does not help to provide a comparison with which to measure coverage.

In order for this proposed methodology to work we would need to encourage publishers to consistently add their data to the total-expenditure field and to have some form of verification, in order to address point 2. Linking to an annual report or official figure elsewhere is one suggestion of how this could be done.

3 Likes
(Petya Kangalova) #27

Hi Alex. Many thanks for providing this useful analysis from the work you have done.

In order for this proposed methodology to work we would need to encourage publishers to consistently add their data to the total-expenditure field and to have some form of verification, in order to address point 2.

Yes, I agree that we need to encourage more organisation to be using the total-expenditure field. I believe the ‘verification’ part will be something you will be doing for the Aid Transparency Index. One way will be to see if organisations have added a relevant document-link in their organisation file.

(Andy Lulham) #28

I suspect the “verification” part may also become relevant for the IATI Coverage methodology, as part of the intention to:

In fact, I suspect much of the complexity of the new coverage methodology has been shifted into this bullet point.

1 Like
(Alex Tilley) #29

Hi Petya - thanks for getting back to me.

To avoid confusion I suggest we focus the discussion in this thread on IATI’s coverage statistics rather than on the Aid Transparency Index. My suggestion is that the total-expenditure element should have some form of verification to address my point 2:

One way of doing this would be to require a published document such as an annual report, audited accounts or other public statement which corroborates the figure entered in the total-expenditure field. I’m not sure whether this could/should be done using the existing document-link in the org file, by adding a new code to the document-link codelist or as an additional subelement in the total-expenditure element.

It would be good to hear views from those more involved in the technical detail of the standard as to whether this is a workable proposal and what may be the best way to implement it.