Continuing the discussion from Bright ideas for the TAG:
I agree it is extremely important that aid agencies report their activity results, and that everyone can access these reports through IATI. I am in awe of the analytical grasp that Herb_Caudill has brought to bear on this. But it seems over-ambitious in the forseeable future to think in terms of a way in which indicators can be rigorously defined machine-readably for all activities.
A key point is that different activities - and within those, different objectives - are differently amenable to standardization. It may not be too difficult to count the number of children to whom your project has helped give a basic education, and it is certainly useful to be able to aggregate that number with others in local and national statistics. But not all objectives are like that.
Let’s say your objective is to build an all-weather road from Amule to Buba. A simple way of reporting the achievement of that objective in an aggregatable way is to say you built 100km of road. But the validity of any aggregation along those lines will be dubious because a road from Amule to Buba is a unique matter, entirely different in character from a road between Calakal and Dor. Measuring things like volume of traffic or value of merchandise does not absolve you from that because any assessment of the impact of the project has to take into account a host of other costs and benefits. Unless you follow the path of calculating everything into a single number using a full social cost-benefit analysis, you are left with a hopeless number of interpretable dimensions. Social cost-benefit analysis is less popular than it once was, and for good reason. In reality it hides many doubtful and politically controversial questions behind a facade of economic expertise.
Or take another example: one which you use in your exposition, Herb: training journalists. What matters in the result is not merely the number of journalists trained, and how they may be disaggregated by gender, ethnic group and so on. It is important to know how thorough the training was (a day or a year? Full-time or part-time etc), and about the features of the curriculum being used. It would be extremely complex and difficult to resolve these matters into machine-readable indicators, and the effort to do so would have unwanted side-effects. Not only would it increase the number of headaches in the publishing agency and deter it from using IATI at all; it might even end up forcing them to design their projects to fit in with the standardized indicators (including packages of standardized curricula) on offer. (I know you will say they can define their own indicators, Herb, but suspect they might feel it easier - or be pressurized - to take indicators off the peg.)
In the end, why do we want to aggregate results? Mainly for purposes of centralized planning and policy discussions. In some cases (like the provision of basic education) it is quite legitimate. In others - like roads - the planners and policy-makers really should not be shielding themselves from the social and political complexities behind pseudo-objective numbers. And in others again - like training journalists - it may be better not to involve central planners at all.
So what would my recommendations be for the TAG?
Remember that the essence of IATI is not necessarily to provide aggregatable or machine-analyzable data. The more fundamental purpose is to provide accessible information. For citizens, this may mean being able to find, understand, and follow up on the individual activities which are going to impact them, or which have impacted them, or which their government has funded or allowed to happen in their country.
There may be a case for providing some facility for using standardized indicators, but this should not be seen as something generalizable to every reported activity.
Being new to this space, I do not understand how the IATI standard has come to support a family of tags called “result” rather than “objective”. The IATI standard was supposed to be forward-looking (presumably in contrast to the OECD CRS). “Result” seems very backward-looking. Citizens and recipient-country governments need an easier way of knowing what aid agencies are planning to do and what they are in the middle of doing. Recording obejctives at the beginning also strengthens post-hoc accountability by allowing clear comparison of promises with, well, claimed results.
I do not think it should be stipulated that a “result” (or an “objective”) needs to be measurable. This job is done by indicators.
Apart from allowing and encouraging the reporting of pre-completion objectives, do not make the standard more complex. The priority should be to make donors and other publishers actually use those tag fields for reporting objectives, results and - above all - links to useful documents.