Title
Results – represent more than quantitative data

Standard
Activity

Schema Object

  1. IndicatorMeasure Codelist
  2. iati-activities/iati-activity/result/indicator/baseline
    iati-activities/iati-activity/result/indicator/period/target
    iati-activities/iati-activity/result/indicator/period/actual

Type of Change

  1. Add code to codelist
  2. Change to Schema

Issue
• Currently, only quantitative indicators (units or percentages) may be represented using the IATI standard.
Why is this a problem?: While some quantitative indicators can provide direct meaning (for example: percentage of patients that responded positively to drug X) they cannot reflect the qualitative context that is often necessary to articulate the meaning behind the values.

There are many interventions in international development where quantitative measures are not relevant or meaningful, for example in relation to policy-influencing and advocacy work, and in some forms of capacity-development and empowerment interventions. These are common areas of work for civil society organisations.

If there is not scope to report such results, CSOs will try to ‘play the game’ either by doing more of the measurable work and less of the hard-to-measure work, even if the latter is vital, or as we are already seeing, by converting qualitative results into quantitative results. Conversion of results into a quantitative format can be time-consuming and expensive, provides little or no value to the initiative, risks poor quality and redundant data, and provides little value for money.

Proposal

  1. Add a new code to the Indicator Measure codelist for Indicator attribute @measure
    Code = qualitative
    Description = This indicator is qualitative

(see http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/ and http://iatistandard.org/202/codelists/IndicatorMeasure/ for relevant sections of the standard)

  1. In order to avoid having to use a unit or a percentage, make the @value attribute for indicator/baseline, and period/target and period/actual elements optional
    (http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/result/indicator/period/target/ for example relevant section of the standard)
    This will have the additional advantage of avoiding having to record ‘no baseline available’ as a zero (0) value – no value and zero value are very different concepts in monitoring and evaluation and having to record a zero value may cause issues with data quality.

Standards Day
Workshopped at the TAG 2017 and mentioned at the end of the Standards day as part of the results section. Although there was very little time to discuss the proposal, no criticism of the proposal was offered. Proposal has been on IATI Discuss since March 2017.

Links
• This topic is discussed here: Results: represent more than quantitative data
• This topic addresses Principle 1 from a consultation driven by Monitoring and Evaluation experts from UK CSOs Jan – Mar 2017 – see Results: discussion space and TAG 2016/17 path. Technical suggestions were devised by technology specialists at the Nethope Athens conference March 2017. In all around 30 M&E and technical specialists were involved in this consultation and it builds on a previous consultation by Bond 2015-16 (https://www.bond.org.uk/resources/publishing-results-to-iati - also on discuss.iatistandard : Sharing Results using IATI data standard: will it improve learning and accountability? ).

Comments (50)

Herman van Loon
Herman van Loon

I agree that only quantitative measures are not enough, but do not agree to make the baseline, target and actual values optional. I suggest to add an ordinal scale as an extension, e.g. to the aggregation status code list. The mean reason, being that you lose a lot of analytical power when you do not value a result. I.m.o. you can always value a result by using an ordinal scale (e.g. '1 - fair, ‘2 - good’, ‘3 - very good’). This allows for e.g. surveys and expert opinions of very complex situations to be translated to a value.

See for detailed comments:

docs.google.com Results1_representMoreThanQuantitativeData

Results: represent more than quantitative data Note: This suggestion addresses Principle 1 from a consultation driven by Monitoring and Evaluation experts from UK CSOs Jan – Mar 2017 – see...

Renate Kersten
Renate Kersten

Thank you for adressing this issue, possibilities for qualitative indicators would be very welcome. Another solution would be to include a new code which is binary (in addition to percentage and unit): 1. achieved, 0. not achieved.
An ordinal scale could be a good alternative in case there is flexibility in defining the scaling, e.g. the scaling could be 1. achieved, 0. not achieved in this case (the result indicator description could then describe the result to be achieved).

Daniel Mackenzie
Daniel Mackenzie

Hi all,

Thank you to everyone for your input so far. This is a very interesting topic, and something I heard come up many times while I was working with Bond, and has been brought up by IRC staff while since I have been here.

I’m afraid I can’t add a technical solution, but I have spoken to a colleague who works with results data regularly and I can pass on her experience around capturing quantitative vs. qualitative data.

"I don’t see the purpose of ascribing a value to qualitative data. They are just fundamentally different methodologies. Part of the reason why one conducts qualitative research is to better understand constructs, beliefs, attitudes, etc. that are not easily quantifiable.

For example, we conduct focus group discussions with women to learn about the barriers they face when accessing health services. We end up with a rich qualitative dataset and we pull out common themes across the groups of women we talked to. In this case, while it’s possible to say 52% of women described XX theme, it may not make sense to describe qualitative data in these terms because the sample is not representative and the %s are therefore less important than the actual themes that emerged. Moreover, qualitative and quantitative data collection are carried out for very different reasons, so while it may be technically feasible to quantify qualitative data, this doesn’t really make sense methodologically."

In short, while it is possible to wrangle qual data into a quant measure, it is time-consuming, costly and may make the data itself meaningless. This statement suggests that the original proposal is an effective way forward.

Many thanks,

Daniel

Rolf Kleef
Rolf Kleef

I don’t think we should try to shoe-horn qualitative data into a quantitative element. There is room to add documents with qualitative results in various formats (documents, video’s, datasets, etc).

In Daniel’s example: I would also suspect that the qualitative analysis of what themes are mentioned across group discussions will lead to other, probably more quantative indicators about what you then will try to do or to achieve in the activity. Or as a means of verification whether you achieved certain other goals.

I’ve proposed a measure type “other” to cater more broadly for anything other than units or percentages. But you’d still have to provide some value (even if just “0” or “1”), to be able to do something meaningful with the data.

I’d love to see examples of what the target and actual of a qualitative indicator would look like, to be meaningful.

The qualitative methods I have seen produce more elaborate data structures such as lists, graphs, or more complex models. That’s a topic for the results working group.

Herman van Loon
Herman van Loon

Daniel Mackenzie When using qualitative results, how do you assess if you achieved your intended goals or not (e.g. in your example)? Wasn’t the current IATI definition for results designed especially to answer this question?

Rolf Kleef Yes I agree, some examples of baselines, targets and actuals of qualitative results might be very helpful for this discussion.

Anna Downie
Anna Downie

This is so important for us- particularly when it comes to reporting on progress on programmes that are focused on policy change and advocacy where qualitative data is often far more valuable than quantitative. Whilst I can see the value in having baseline, target and actual values that could be translated for qual data (eg ‘1, on track, 0, not on track’ etc), I’d still want to see them as optional to give people the freedom to show progress against the indicator in the way that makes most sense to the programme and the data.

Hayden Field
Hayden Field
  1. In order to avoid having to use a unit or a percentage, make the @value attribute for indicator/baseline, and period/target and period/actual elements optional […] This will have the additional advantage of avoiding having to record ‘no baseline available’ as a zero (0) value

The value attribute is of type xsd:string. As such, it is possible at present to provide text data as the value attribute value - it is not necessary to convert data to something numerical such as a unit or percentage.

Mike Smith
Mike Smith

Per Rolf and Herman’s request, below is an example of a qualitative indicator. It’s for a project that aims to promote safe hygiene practices through the routine immunisation programme in Nepal.
Note: I have used <indicator measure="Qualitative"> and have not included a value for eg <actual> per this proposal. Although technically per Hayden Field ’s comment the current standard requires a string “value”, using freetext like <actual value="qualitative">, <actual value="none">, <actual value=" "> or <actual value="0"> seems to provide a data quality issue that is easily avoided by making value optional (per proposal).

<result type="2">
  <title>
    <narrative>Potential for inclusion of hygiene promotion into national vaccination programme beyond pilot project</narrative>
  </title>
  <description>
    <narrative></narrative>
  </description>
  <indicator measure="Qualitative">
    <title>
      <narrative>feasibility of integrating hygiene promotion into vaccination programme under normal programme delivery circumstances</narrative>
    </title>
    <baseline year="2014">
      <comment>
        <narrative>Scoping study in 2012 demonstrated the desire of health sector to pilot initiative through identifying missed opportunity to combine hygiene promotion into national vaccination programme – see published scoping document http://washdev.iwaponline.com/content/3/3/459.</narrative>
      </comment>
    </baseline>
    <period>
      <period-start iso-date="2014-01-01"></period-start>
      <period-end iso-date="2016-04-05"></period-end>
      <actual>
        <comment>
          <narrative> recommendations copied from the document 16_04_051-Nepal vaccines FCP narrative report template April 2016 - March 2017_Final_13 May 2017_A_B_Om from the activity’s document-link:
o	Both the FCHV and health worker should be involved to run the session smoothly.
o	Some incentive for health worker to motivate them to work actively.
o	Regular supervision from district and center level for proper feedback.
o	Vaccinator should also involve in review program.
o	This program should give continuity in future and scale up in other districts also because hygiene promotion is directly/indirectly related to decrease/minimize the cases of child mortality and morbidity.
o	Number of games should be decrease so that it will takes less time to run session.</narrative>
        </comment>            
      </actual>
    </period>
    <period>
      <period-start iso-date="2016-04-05"></period-start>
      <period-end iso-date="2017-02-10"></period-end>
      <actual>
        <comment>
          <narrative> conclusion copied from review meetings document with title” Central Level Review Meeting of "Hygiene Promotion through Routine Immunization Programme in Nepal" on 10th Feb, 2017”:
            Representatives of four District/Public Health Office and FCHVs have shared that the Hygiene Promotion through Routine Immunization Programme can be implemented successfully using the existing health system. It has shown its mutual synergy to increase the positive hygiene behaviour change and increasing the coverage of routine immunization. They have also flag that the program has played a vital role in reducing the dropout and wastage rate of the vaccine. All four districts mentioned that there is no outbreak of diarrhoeal and cholera diseases in their districts after the implementation of this program.
            The integration of hygiene promotion package in Routine Immunization sessions has been well appreciated and accepted by FCHVs / health workers and the mothers. So, this review meeting has recommended for the need to continue the program in these four districts and should be scaled up in other districts.</narrative>
        </comment>
      </actual>
    </period>
    <period>
      <period-start iso-date="2014-01-01"></period-start>
      <period-end iso-date="2017-06-01"></period-end>
      <target>
        <comment>
          <narrative>fully feasible to integrate hygiene promotion into vaccination programme under normal programme delivery circumstances.</narrative>
        </comment>
      </target>
    </period>        
  </indicator>
</result>

Attempting to codify and “shoehorn” the indicator into numbers would either trivialize, be unrepresentative and undermine IATI data; or require a disproportionate amount of effort (the project only needs and uses the qualitative result, not a quantitative result) and be very complex (for example different quantitative values are relevant for each evaluation context which would require additional verification with stakeholders) – this provides poor value for money, especially given this is just one (very important) indicator of many for the project.

I’m happy to provide the source documents if needed and see related proposal Add document-link to Results indicator (included 2.03) for alternative to embedding references to documents in the freetext.

Daniel Mackenzie
Daniel Mackenzie

Apologies for the late posting. Here are two example indicators provided by a colleague at IRC which demonstrate how assigning a value to a qualitative measure can undermine data quality, creating a greater reporting burden and even have a detrimental effect on programme strategy.

"Indicator 1) National policy on sexual violence is adopted in X country.

Why use qualitative narrative to report this?

There are many nuances. First, what if the government adopts a national policy but it doesn’t cover all the issues that we find critical (like marital rape). Second, even if adopted, it may be poorly enforced, and may even have detrimental effects (it can whitewash the issues when a government can claim that the policies are in place). And if either of those cases happen, does it still count as “one” policy changed in our logframe?

In the past, IRC have reported on these indicators using qualitative narrative: IRC’s actions, and reporting on any outcomes. Sometimes we’ll conduct focus groups, for example, to get feedback from citizens on how these policies impact (or not) their daily lives. These are more nuanced account of how policies change than a number."

A quantitative measure may mask political tensions and misrepresent the situation.

Indicator 2) Women are increasingly empowered to participate in local governments.

Why use qualitative narrative to report this?

Evidence has shown that time and time again, quotas for gender in government structures do not always trickle down to women (both those in the government and the broader community) feeling more empowered to voice their opinions, needs & concerns. Nor do women in government necessarily act primarily as representatives for their entire gender rather than other segments of their local constituents. Qualitative data can help us better understand not just the numbers but also whether women can and do actively participate."

As in many cases, it would be possible to add a quantitative measure, but this would add two extra layers of reporting - 1) decided what the quant measure is 2) adding narrative report to compensate for the first.

I hope these examples contribute to the discussions in a productive way. Any questions, please let me know.

IATI Technical Team
IATI Technical Team

It is proposed that a measure “Other” is added to the IndicatorMeasure codelist now, and that further discussion on a structured solution to qualitative indicators is carried forward for the next upgrade

SJohns
SJohns

Thanks for this IATI Technical Team . However I wondered why the team had a preference for IndicatorMeasure = “Other” over IndicatorMeasure = “Qualitative” (the original proposal), and how using the word ‘Other’ adds to the usefulness of the data where the word 'Qualitative would not?

In the interests of transparency, please can you also explain the decision-making process behind this as there was clearly articulated support on this forum for the original proposal.

Bill Anderson
Bill Anderson

Can I first make it clear that IATI Technical Team hasn’t decided anything. We have proposed a way forward. The consultation is ongoing.

SJohns we made our choice from the above discussion based on a ‘lowest-common denominator’ approach in the hope of reaching consensus. If there is, in fact, consensus for the original proposal we stand corrected and will gladly revert. If there is no consensus can I suggest that the M&E community continue to work it out?

This is, in my opinion, a matter for M&E experts, not the Tech Team. We have no vested interests and will follow any consensus reached.

SJohns
SJohns

The original proposal was a consensus from a range of M&E experts and IATI users that are involved in using IATI to provide data on results. This is all clearly documented on IATI Discuss. I’m not inclined to go back to the M&E experts to ask for more evidence when they have already contributed beyond what was originally asked of them (see the posts above for examples). The burden of proof is really on the IATI technical team to justify their proposal to use ‘other’ and punt the rest of it into a future upgrade.

Although you say that the IATI technical team have not made a decision, the perception from the IATI team’s post on this thread is that they have clearly stated a preference for a particular way forward.

In order to maintain the integrity and transparency of the consultation, I would like to see the whole original proposal be put forward for consultation and for the community themselves to be allowed to refine the proposal between now and 7 July, as per the timeline. http://iatistandard.org/202/upgrades/decimal-upgrade-to-2-03/

Bill Anderson
Bill Anderson
Image removed. SJohns:

Although you say that the IATI technical team have not made a decision, the perception from the IATI team’s post on this thread is that they have clearly stated a preference for a particular way forward.

Fair comment.

At least it would have been If I hadn’t already tried to answer your concerns above.

At Standards Day in Dar I think we attempted to consolidate and improve on the governance of the standard through consensus. Personally I think it was a good step forward for the community as a whole. For the technical team it was another day of learning how best to balance the disparate interests and concerns of the community. We may not always make the right call, but let me assure you again that we do so in good faith. We have no vested interests.

It is your prerogative to question the assurances I have given you. It is mine to defend the integrity of our team.

Laoise Ní Bhriain
Laoise Ní Bhriain

I completely agree with the need to represent more than quantitative data. Most of our work (at Anti-Slavery International) aims for results that can’t meaningfully be represented by a number or percentage. Most of our projects look at the level of political interest in a range of anti-slavery measures and willingness to implement positive changes.
Partners and communities give feedback on the sincerity and quality of governments’ interactions and implementation of their recommendations into policies and laws. A number of projects report on communities’ perceived changes in gender equality and female leadership/empowerment and acceptance of that.

Echoing the points made from the IRC, narratives inform about the majority of our work far more comprehensively and meaningfully than numbers could. As the whole point is often not simply about whether something ‘is’ or ‘is not’, but it is about subjective perceptions and experiences of factors that are complex and shifting. Quantitative measures of these are often meaningless and fail by their own standard of supposed accuracy. To only have quantitative measures assumes that everything can be assessed that way which isn’t the case with our work.
As I understand it, the point of IATI is to improve transparency and inform - insisting on quantifying qualitative information seems to me to be counter to this purpose as it overly simplifies data and information to the point where integrity and meaning are lost.

I’m afraid that I can’t contribute to the technical side of the conversation about how this might be implemented in IATI but looking forward to the conversation and what comes out of it all!

Laurence Leclercq
Laurence Leclercq

Sorry as well for the late feedback. Handicap International can only agree on and further highlight the importance and the need for qualitative data to be reported on. Most of our work relates to the improvement of people with disabilities’ or vulnerable people’s quality of life and we use many qualitative indicators alongside with quantitative ones to account for the quality of our actions. Here are a few examples of these indicators:
· Satisfaction of the beneficiaries in relation to the services received.
· Improved quality of services (against a set of quality criteria)
· Increased level of understanding of people about specific issues
· Improved self-esteem and self-confidence of beneficiaries
· Improved quality of life of beneficiaries (quality for life indicators being defined according to each context).
Hope this is useful at this point of the discussion.
Best,
Laurence

Daniel Mackenzie
Daniel Mackenzie

Hi all,

Many thanks to SJohns and Bill Anderson for your comments.

I agree with Sarah that the original proposal should go through. I’d also like to emphasise that no-one is attacking the integrity of the Technical Team. We all recognise the great work that goes into the upgrades and the difficulties in mediating many differing views. It is the variety of perspectives that make it such a rich and diverse community!

The proposal should go through in its original form for the following reasons:

  • While Rolf’s suggestion for ‘Other’ marker may be seen as the lowest common denominator, there has been no support for this suggestion, on this thread or the one linked to in the post. This is not to suggest the proposal has no value in itself - indeed, it would be better than the original position! - but it has not been approved by the community in relation to the issues raised in the original proposal. I do not see how this relates to consensus.

  • A suggestion was raised by Hayden Field that value can be recorded as a string, so there is no need to make it optional. Mike Smith countered with a concern about data quality. This concern has not been addressed, and relates to Herman van Loon ’s point about analytical power.

  • Herman and Rolf Kleef requested examples of qualitative indicators that would lose meaning if they were assigned a value. Examples have been provided by three organisations from across the sector, with support for the original proposal. This counter has not been addressed.

It is confusing to those of us who put in a lot of work to answer all of the challenges why the proposal has not made it through to be discussed and refined further. Nor has our evidence been addressed. If the proposal goes through in full to the next stage we can work to refine it together.

As a separate note, we may want to consider whether consensus is an effective way to move forward. It is very easy for someone to post a negative comment without providing evidence, and a proposal stalls.

I hope these comments add positively to this fruitful discussion.

Best,

Daniel

Mike Smith
Mike Smith

Hi All. I have to admit I’m a bit confused by the overall process - I thought the in/out discussion would happen from 17/06/17 till 07/07/17 per http://iatistandard.org/202/upgrades/decimal-upgrade-to-2-03/ and the technical team undergoes consultation with the community? It seems on some topics this is the case, but others, such as this one, there’s already been a proposal made? I’m not sure how it was arrived at, specifically:

Re: 1) from original proposal (the need for “qualitative” IndicatorMeasure):

  • Herman van Loon rasied a concern saying you lose analytical power and that you can always value a result by using an ordinal scale
  • Renate Kersten , Daniel Mackenzie , Anna Downie , Laoise Ní Bhriain , Laurence Leclercq agreed with the proposal
  • Herman van Loon and Rolf Kleef challenged for qualitative example to discuss
  • Mike Smith countered with an example where it’s not possible to convert to quantitative in a meaningful way and is therefore essential
    No-one has responded to say that qualitative is not essential

Re: 2) from original proposal (“value” should be optional)

  • Hayden Field pointed out a string is currently required so no need to put “0”
  • Mike Smith countered this will cause a data quality issue (eg more difficult to analyse etc.)
    No-one has responded to say that data quality is not an issue for IATI

Given there have been no response to the counters it seems reasonable to assume that the concerns have been addressed and that at this time the original proposals should go through, not that they should be dropped/amended?

Thankfully I see that Bill Anderson has clarified that nothing has been decided yet (and I think this only happens on the 28/07/17 per http://iatistandard.org/202/upgrades/decimal-upgrade-to-2-03/)? How do I tell what the current position is for the proposals - is there a topic/document I’ve missed somewhere?

IATI Technical Team
IATI Technical Team

In response to the above discussion and concerns it is proposed that Mike Smith ’s original submission is adopted. Namely:

  • Add “9 - Qualitative” to the IndicatorMeasure codelist
  • Make the following value attributes optional
  • result/indicator/baseline/@value
  • result/indicator/period/target/@value
  • result/indicator/period/actual/@value

These changes have been included on the summary table

Bart Stevens
Bart Stevens

Maybe a bit late, but I would still like to add my 2 cents from a measurement- technical point of view:

Qualitative measurements are fundamentally different from value-based measurements and ordinal measurements. As other commenters rightly remarked, we shouldn’t force people to reduce everything to values. However, you can’t in all circumstances reduce qualitative measurements to an ordinal scale either.

Ordinal scales are a breed on their own. Should we wish to integrate ordinal scales in the activity standard then it’s not sufficient to have tags for target and actual values. You need additional provisions to be able to register the list of codes/values and their respective explanation (text).

In some cases you can make a summary of qualitative information by using a nominal scale rather than an ordinal one. This means you can use values but one value is not better than the other.

About targets: in some cases you can have a target even when you use qualitative information in the sense that you describe a desired situation. But in other cases this may not be possible. Qualitative information is also useful to describe unexpected but very important effects, in which case you don’t have a target.

Finally, there is more than just numbers or (qualitative) text. Working on project management software I identified 17 different kinds of indicators so far and that is certainly not the end of it.

IATI Technical Team
IATI Technical Team

There will be some consultation calls in early July for any 2.03 proposals where people would like to discuss them further - if you would like to discuss this proposal on one of the calls please ‘Like’ this IATI tech team post by end of Mon 26 June - you can do this by clicking the heart symbol to the bottom right hand side of this message.

Further details on the calls are available in the ‘How to participate’ topic.

IATI Technical Team
IATI Technical Team

This proposal will be discussed on a consultation call on Results - Monday 3 July, 2pm (BST), 1 hour

To join this call, use this link from your computer, tablet or smartphone https://global.gotomeeting.com/join/469747029
You can also dial in using your phone.
United Kingdom: +44 330 221 0088
United States: +1 (646) 749-3129
Access Code: 469-747-029

Please 'like' this post if you plan on joining this call (click the heart symbol to the bottom right of this message)

IATI Technical Team
IATI Technical Team

Notes from consultation calls w/c 3rd July

Outcome:
The proposal was reviewed by those on the call and there was no objection from the group.

Herman van Loon
Herman van Loon

Back from vacation I read through all comments resulting in the decision to add this proposal to 2.03. I still have severe doubts that this is the correct way forward.

The example given by Mike Smith does i.m.o. clearly shows that an ordinal value could easily be defined to measure the success of reaching the goal of the intended intervention:
0 - not achieved (it is not feasible to integrate hygiene promotion into the vaccination program)
1 - partly achieved (it is partly feasible to integrate hygiene promotion into the vaccination program)
2 - fully achieved (it is fully feasible to integrate hygiene promotion into the vaccination program)

Adding this type of ordinal values, would enable answering very interesting questions, especially when multiple organizations and many activities are involved in the implementation of such a programme eg:

  • which programme improvement interventions are actually successful?
  • which programme intervention prove unsuccessful and what can we learn from it?
  • in which countries is such an intervention successful?

From the viewpoint of a single activity, it does not matter very much if an result value is not defined. From the viewpoint of portfolio management when you are dealing with many activities implemented by many organizations, you lose a lot of analytical power by allowing qualitative results only. I.m.o. you should always be able to define a measure for success. If you can’t, than what is the point of doing the activity at all?

IATI Technical Team
IATI Technical Team

Herman van Loon Thank you for your comments. During the consultation call the proposal about using ordinal scale was mentioned but no views were expressed either way (for or against).
The consultation call was attended by 8 representatives from different organisations, including Mike Smith and other members who have previously engaged with the results agenda.

We welcome further discussion on this proposal.

Bill Anderson
Bill Anderson

Herman van Loon in my view the decision to classify qualitative results as not/partly/fully achieved is one for the M&E community to agree upon. Are there similar approaches used in other systems? (Mike Smith et al?)

Also, I’m not sure that having a meaningful value of ‘0’ is helpful for data users needing to distinguish between different meanings for zero and null.

Herman van Loon
Herman van Loon

@bill_Anderson The real point I tried to make was that the whole design of the results framework in IATI is geared towards publishing MEASURABLE results. Therefore we have baseline, target and actual values which enable the publisher to express in howfar the results of the activity are being achieved in a certain timeframe.

Therefore the value is mandatory.

If you want to publish qualitative results, the results are by definition not measurable. Therefore I think qualitative results should be published as a linked document as suggested by Rolf Kleef , using a document category ‘Qualitative results’.

The other part of the discussion was that what is seen as a qualitative result, can often be expressed on an ordinal scale (see example above). This type of results does belong i.m.o. the IATI results framework. In that case, as always, the value is mandatory.

So in summary: I suggest to add the ordinal scale as a measure type , thus allowing a lot more flexibility in results reporting (no need for objective quantification). True qualitative results can easily be accommodated with the current link-document functionality. Hope this makes sense.

Bill Anderson
Bill Anderson

Yes, Herman van Loon that does make sense. To make sure I do understand …

  • A value should be added to the IndicatorMeasure codelist that is associated with a rule that the related @value can only be not/partly/fully achieved. (Naming this ‘Qualitative’ is perhaps misleading as it is defining a specific way of measuring a qualitative condition)
  • The proposal to accept qualitative results where the ‘measure’ is explained in text, not in the @value should not be accepted.
Herman van Loon
Herman van Loon
  1. I would suggest to add and value in the IndicatorMeasure codelist “Ordinal” (see WiKi). The rule is that when you use this measure type the @value must be a positive integer. In this way also other ordinal scales can be accommodated (see the example in the WiKi on ordinal data). The meaning of the ordinal values can be described in the indicator narrative.

This setup would allow great flexibility to model all kind of results, eg responses in surveys, questionnaires, etc.

  1. Yes definitely: measures should not be in text. Text measures can not be summarized, aggregated or otherwise analyzed thus making the use of results-data very difficult if not impossible.
Erik Hesseling
Erik Hesseling

As stated earlier by my colleague, the Netherlands Enterprise Ageny is in favor of having the opportunity to represent qualitative indicators using the IATI standard.

Bill Anderson
Bill Anderson

Erik Hesseling I think there is consensus to include qualitative indicators. The outstanding issue is whether to make @value optional and/or assign an ordinal value.

Herman van Loon
Herman van Loon

Bart Stevens made the interesting point that there are also nominal scales, in which one value is not better than the other. That kind of ‘result’ could also be expressed by adding a measure type ‘nominal’. I have the feeling though that this does not represent an actual result. But I might be wrong here: an typical example showing that a nominal scale can represent a valid result would be helpful in this discussion.

Anna Downie
Anna Downie

I’ve just re-read this thread following Bill’s email asking us to review the discussion as I’m disappointed this hasn’t reached consensus (as it’s a big challenge in our work). To summarise:

The IATI Technical Team put forward the original submission (which was based on a consultation with M&E people in a wide variety of organisations):

Add “9 - Qualitative” to the IndicatorMeasure codelist
Make the following value attributes optional
result/indicator/baseline/@value
result/indicator/period/target/@value
result/indicator/period/actual/@value

At the consultation call the proposal received no objections.

I may have missed something someone said in the thread, but I think the subsequent comments all agreed with the proposal with the exception of Herman van Loon (sorry to single you out!).

If I’ve understood correctly, the points of disagreement are views on whether:

  1. All results can be translated meaningfully and usefully into a quantitative scale of some kind
  2. If they can’t, qualitative results should only be communicated through a document link

This may have already happened, but would a call between Herman van Loon and the original proposers of the change, such as Mike Smith be helpful to talk directly about the point of disagreement and see if we can find a way forward?

Bill Anderson
Bill Anderson

Anna Downie I agree that another call - involving all interested parties reviewing all outstanding results issues - is a very good idea.

We can draft an agenda (summarising where we are on all outstanding issues) and a doodle poll to agree on a date.

If I may be so bold I think it would also be extremely helpful if the interested NGO communities in the UK and Netherlands could also speak with a more collaborative voice on some of these issues.

Mike Smith
Mike Smith

(apologies for my lack of responsiveness)

Without forcing a change in M&E practice and/or distorting/shoehorning data (incorporating Herman van Loon ’s view about allowing ordinal scales, Bart Stevens ’ about nominal scales, and the M&E and Bart Stevens ’ view about allowing qualitative data) can we agree to:

Add three codes to the IndicatorMeasure codelist:

  • Nominal - the indicator is measured as a quantitative nominal scale.
  • Ordinal - the indicator is measured as a quantitative ordinal scale.
  • Qualitative - the indicator is qualitative

and make the following value attributes optional

  • result/indicator/baseline/@value
  • result/indicator/period/target/@value
  • result/indicator/period/actual/@value

Using only document-link to record qualitative actuals (per Add document-link to Results indicator (included 2.03) ) would also require @value to be optional, but suggest “Qualitative” could be used for IndicatorMeasure rather than adding a fourth code to the codelist.

Mark Brough
Mark Brough

TL;DR:
+1 on using narrative rather than <document-link>
+1 on making @value optional

Image removed. bwalkden:

the narrative can contain a summary that is conveniently displayed by visualisation tools, and people are not obliged to follow links (sometimes over slow and difficult connections) to arbitrary document types which are not easily amenable to further analysis / use by query & other tools

Fully agree with this. It would be much easier to use such data.

I have a lot of time for the goal of making all the data easy to process and use, for assessing programmatically whether projects have succeeded (in the sense that they have met their intended goals). However, on the question of whether the IATI Standard should require qualitative results to be quantified – my strong suggestion is not. I think we should be guided by the M&E community (and we have received some pretty clear feedback from them).

There are clearly good reasons for not quantifying particular results (I guess particularly when looking at complex change, where pre-planned quantitative indicators can just lead to isomorphic mimicry; and where qualitative results may provide a richer depth and more accurate picture than quantitative ones).

Secondly, the IATI Standard shouldn’t be trying to force changes in the way that monitoring and evaluation proceeds especially when M&E experts are telling us that this is not a good idea.

Finally, from a practical perspective, many systems that have some good results data do not have that data available in ways that would allow results to be quantified. Requiring them to be quantified is likely to lead to less useful data (e.g. just putting “unquantifiable” as the value).

Herman van Loon
Herman van Loon

Anna Downie Bob Walkden Yes, you are right. these are the two major points. The discussion is not about whether or not IATI should support qualitative results for a single activity, but about how qualitative results can be modelled in IATI without complicating data-use. When you want to do an statistical analysis over multiple activities, then you really need non-qualitative results.

Mike Smith Thanks for your comments. I think we could have an agreement

Image removed. mikesmith:

Add three codes to the IndicatorMeasure codelist:

Nominal - the indicator is measured as a quantitative nominal scale.
Ordinal - the indicator is measured as a quantitative ordinal scale.
Qualitative - the indicator is qualitative

and make the following value attributes optional

result/indicator/baseline/@value

result/indicator/period/target/@value

result/indicator/period/actual/@value

if we add 2 rules for the IATI guidelines:
1 - The @value may only be optional if the indicator measure is 'Qualitative’
2 - The @value must be a valid number for all non-qualitative measures.

Does this make sense?

Bob Walkden
Bob Walkden

Actually, I’d go slightly further than my previous reply and suggest that for qualitative results the @value must be omitted (not even an option), and for quantitative results it must be present. However I’d be perfectly content with Herman’s suggestion rather than delay further.

Mike Smith
Mike Smith

Agree! To pull it all together I think its:

Add three codes to the IndicatorMeasure codelist:

  • Nominal - the indicator is measured as a quantitative nominal scale.
  • Ordinal - the indicator is measured as a quantitative ordinal scale.
  • Qualitative - the indicator is qualitative

and make the following value attributes optional

  • result/indicator/baseline/@value
  • result/indicator/period/target/@value
  • result/indicator/period/actual/@value

add 3 rules for the IATI guidelines:
1 - the @value must be omitted for qualitative measures
2 - The @value must be included for non-qualitative measures
3 - The @value must be a valid number for all non-qualitative measures.

Bob Walkden
Bob Walkden

I said in an earlier contribution that just because something can be quantified, it doesn’t mean it’s desirable or practical to quantify it. Qualitative research is a standard part of participatory studies and is widely used in the sector, so by including it we are not doing anything that is outside the scope of the subject matter. The M&E people asked for this to be included in the standard, and therefore in my view the ability to report qualitative results should be included – I can certainly see no reason not to. This implies that for qualitative results the @value attributes must be optional; however, if for some reason they remain mandatory then the associated documentation should state that the numbers in the attribute are meaningless by definition.

The question then remains of how to implement it. The choices seem to be between a narrative field and a document link. Technically speaking I have a slight preference for a narrative, and indeed I think the best approach might be to allow both (at least one being mandatory), so that the narrative can contain a summary that is conveniently displayed by visualisation tools, and people are not obliged to follow links (sometimes over slow and difficult connections) to arbitrary document types which are not easily amenable to further analysis / use by query & other tools.

Bob Walkden
Business Architect
Save the Children UK


Please log in or sign up to comment.