Results – improve consistency of results standard (included 2.03)


(Mike Smith) #1

Title
Results – improve consistency of results standard

Standard
Activity

Schema Object

  1. iati-activities/iati-activity/result/indicator
  2. iati-activities/iati-activity/result/indicator/baseline

Type of Change

  1. Addition to Schema
  2. Addition and change to Schema

Issue
There are a number of areas of the result standard that are inconsistent (capability in one area is missing in other), making it more difficult to understand and use the standard than it could be otherwise. For example, currently:

  1. you can only flag if results are suitable for aggregation, not underlying indicators that make up the result.
  2. indicator “baselines” cannot be reported in the same way as “targets” and “actuals” (e.g. there is no way to report disaggregation’s, or use dates rather than just years etc.)

Proposal

  1. Add optional “aggregation-status” attribute (per result/@aggregation-status ) to result/indicator

(compare relevant sections of http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/ and http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result)
2) Make the following additions and changes to result/indicator/baseline :
a. Add location and dimension elements (per result/indicator/period/actual/location and result/indicator/period/actual/dimension respectively)
b. Change baseline cardinality from 0…1 to 0…* to permit disaggregations (per proposal Results – allow disaggregations of results data (included 2.03) )
c. Add iso-date attribute (type xsd:date)

(compare relevant sections of http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/baseline and eg: http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/period/actual/ )

Standards Day
Workshopped at the TAG 2017 and mentioned at the end of the Standards day as part of the results section. Although there was very little time to discuss the proposal, no criticism of the proposal was offered. Proposal has been on IATI Discuss since March 2017.

Links
• This topic is discussed here:

  1. Results: improve consistency of result standard: aggregations-status
  2. Results: improve consistency of result standard: baseline
    • Proposal 2)b. is for consistency with another v2.03 proposal (permitting disaggregation): Results – allow disaggregations of results data (included 2.03)
    • This topic addresses Principle 7 from a consultation driven by Monitoring and Evaluation experts from UK CSOs Jan – Mar 2017 – see Results: discussion space and TAG 2016/17 path. Technical suggestions were devised by technology specialists at the Nethope Athens conference March 2017. In all around 30 M&E and technical specialists were involved in this consultation and it builds on a previous consultation by Bond 2015-16 (https://www.bond.org.uk/resources/publishing-results-to-iati - also on discuss.iatistandard : Sharing Results using IATI data standard: will it improve learning and accountability? ).

Multiple result indicator baseline elements (v. 2.03)
(Herman van Loon) #2

Yes, this is definitely an improvement. It is indicator values which can be aggregated or not. Not results.


(Andy Lulham) #3

Agree with Herman – these seem like good fixes.

Is adding @iso-date to baseline done with a view to potentially removing @year in the next integer upgrade? If so, would it be possible to also add something to the guidelines discouraging the use of baseline/@year?


(Rolf Kleef) #4

I don’t think adding the attribute to the indicator solves it. Aggregation should be defined with respect to dimensions, not to facts.

  • Can I aggregate the numbers across periods?
  • Can I aggregate the numbers across activities?
  • Can I aggregate the numbers across organisations?
  • Can I aggregate the numbers across countries or locations?

There also should be documentation about what we mean with @aggregation-status="true". Aggregation is possible across all dimension for the indicator within a single publisher? (So not across organisations.)

You can say:

<result aggregation-status="true">(result A)<indicator 1></result>
<result aggregation-status="false">(result A)<indicator 2></result>

So basically it is already possible to separate indicators that can be aggregated from those that cannot. I agree this is not elegant, but more a matter of documentation and guidance at this point.


I’m also not sure about adding a single date: that seems to make sense when the baseline is more a “point in time” rather than a “year in time” or “some point in a year”.

It depends on the indicator as well: for output indicators, it seems more appropriate to add a period to the baseline?

If we want to achieve consistency between baseline and periods, we should go all the way.

Perhaps an easier way is to add a @baseline boolean flag to <period> to indicate it represents a baseline? It would only be expected to have an <actual>. With period-start and period-end the same, it offers the equivalent of the iso-date.

And to propose some syntactic sugar-type solutions: we could introduce shorthand notations in version 2.03 or 3.01:

 <baseline year="2012" value="10">
  <comment>
   <narrative>Baseline comment text</narrative>
  </comment>
 </baseline>

as equivalent to:

 <period baseline="true">
  <period-start iso-date="2012-01-01" />
  <period-end iso-date="2012-12-31" />
  <actual value="10">
   <comment>
     <narrative>Baseline comment text</narrative>
   </comment> 
  </actual>
 </period>

and likewise:

 <baseline iso-date="2012-06-02" value="10"/>

as

 <period baseline="true">
  <period-start iso-date="2012-06-02" />
  <period-end iso-date="2012-06-02" />
  <actual value="10"/>
 </period>

(Hayden Field) #5

Agree with the suggestion from @rolfkleef to add a boolean attribute to <period> rather than extending the baseline element through replication of the majority of period (this isn’t very DRY).

As such, a proposal in place of (2) in the post by @mikesmith would be:

  • Add an optional boolean iati-activities/iati-activity/result/indicator/period/@baseline
  • Deprecate use of iati-activities/iati-activity/result/indicator/baseline plus its attributes and sub-element (remove element at integer)
  • Add Rules:
    • If iati-activities/iati-activity/result/indicator/period/@baseline is present for a given indicator, iati-activities/iati-activity/result/indicator/baseline must not be present for that indicator (remove rule at integer)
    • If iati-activities/iati-activity/result/indicator/period/@baseline is true or 1 for a given indicator, iati-activities/iati-activity/result/indicator/period/actual must be present

(IATI Technical Team) #6

This topic has been included for consideration in the formal 2.03 proposal depending on further clarification relating to how baselines are handled


(Bill Anderson) #8

I think this thread is falling between discussion on M&E best practice and logical elegance.

When the results element was designed the assumption was explicitly made that each indicator (or disaggregations within the indicator) can only have one baseline value which is established at the outset of the activity. For this reason the baseline element was deliberately kept outside of the period element (which allows for targets and actuals to be reported on a period-by-period basis).

Can M&E experts comment on this?


(Mike Smith) #9

Hi @bill_anderson, thanks for this, yes there will only ever be one set of baseline data (set, meaning that actual “baseline” element in IATI would be repeated once for each disaggregation, but all referring to the same baseline set) for an indicator (note sometimes people refer to mid-lines and end-lines but think of these more as reviews, sometimes where new indicators are introduced and old retired). Sometimes you take a baseline over a period of time (number of water shortages during last year’s drought) and we were thinking this would be recorded in the comment, but I can see how the period functionality could be useful here (unless anyone can see drawbacks?).

In terms of following DRY principles and logical elegance, you could extend the aforementioned iati-activities/iati-activity/result/indicator/period/@baseline to use a codelist (maybe called Indicator Value Type) that can be one of baseline, target and actual - then you only need one set of locations, dimensions, comments in the schema for all three constructs (baseline, target and actual) thereby creating opportunity to simplify the current Results elements at integer update. It would be helpful to make guidelines to indicate that there should only ever be one set of baselines. Happy with either approaches as both achieve the same M&E requirement.


(Bob Walkden) #10

I agree with the idea of using a code to distinguish between baseline, target and actual. A baseline is just a measurement that has been chosen as a reference point for other measurements, it is not inherently different from other types - I don’t think it should be treated any differently. This has practical benefits in that it simplifies the schema considerably without loss of any capability.


(Bill Anderson) #11

My understanding of where we have got to is …

The ideal solution is:

  1. We deprecate the <baseline>, <target> and <actual> elements
  2. We add a new complex element (called <measurement> ??) which inherits the existing target/actual content.
  3. We add an attribute (called measurement\@measurementtype ??) which has values baseline, target, actual

But …

The baseline element is currently used by 117 publishers in over 150,000 activities so we are in integer terrain. Given that the next integer is most likely a few years away there are a couple of options that could be possibly be considered

  • We could implement the changes now if we could provided data users with a utility that converted data in deprecated elements into the new format.
  • We could implement an extension for the new format now that could be used in addition to the existing elements. (The integer upgrade would then merely move the extension into the main schema.)

A temporary @baseline attribute (implemented now in anticipation of the bigger integer change) would require users to look in two places for baseline data so I’m not convinced that it is a viable option.

Thoughts?


(Bob Walkden) #12

I would support the extension. It would not be enough to provide a conversion utility because existing publishers would need to change their software at very short notice to write to the new structure or to run the utility. If they have a few years’ notice they should be able to factor the integer upgrade changes into their road maps.


(Herman van Loon) #13

@bill_anderson I completely agree with you. This is not viable. The extension proposal would be detrimental for ease of data-use and data-quality.


(Pelle Aardema) #14

I think that is exactly what it is.

I really think the results area needs more insights from experience - thus time - to facilitate a proper redesign. Looking at the organisations currently using the baseline element, the majority seem to publish in accordance with the Dutch publication guidelines, hence experiences will still be ‘fresh’.

I think IATI would benefit from a proper Results Working group, which was announced at the TAG, but doesn’t seem to have materialized yet. I’d be more than happy to participate.


(Bill Anderson) #15

I think we are heading towards a position where any change to existing elements is untenable.

Which leaves us with the original proposal: to make the <baseline> sub-elements consistent with <target> and <actual>.

Can we reach consensus on that?


(Mike Smith) #16

Hi @bill_anderson, you beat me to it! Yes there’s still the original suggestion (potentially using a start and end data, unless anyone see’s drawbacks), which, as mentioned would also satisfy M&E folks, but as @hayfield points out is not DRY. I don’t think that logical elegance (DRY) is a reason not to make up for a shortfall in the standard that M&E folks have identified and asked to change (especially given its about consistency of the standard).


(John Adams) #17

I agree with @bill_anderson to make the <baseline> sub-elements consistent with <target> and <actual>.

We will also need further discussion leading up to a potential <results> redesign for a future integer update.

@pelleaardema you mentioned a TAG Results Working Group. Would someone like to volunteer to co-ordinate that? That group will need time to come up with a set of proposals that can meet consensus across a range of M&E people.


(Pelle Aardema) #18

Sounds like the most feasible solution to me.


(Pelle Aardema) #19

@bill_anderson Sounds like the most feasible solution to me

@JohnAdams I know that names were collected during the TAG. @SJohns and @rbesseling may have those? I would be happy to contribute.


(Herman van Loon) #20

Agree, maybe not perfect but more consistent than it is right now.


(Andy Lulham) #21

Yes there’s still the original suggestion (potentially using a start and end data, unless anyone see’s drawbacks)

I agree – in order to be consistent, the original proposal would need to be amended to include <period-start> and <period-end> (or similar) data. Also, “Add iso-date attribute” (2c) should be removed.