Hi everyone
I wanted to share something David Megginson & Nick Imboden encountered via importing IATI data into the Financial Tracking System (FTS).
This might is an issue that some may have hit before. It’d be great to hear from others who regularly ingest IATI data into their projects, to understand if this is a headache, or not, or whether you have a workaround. Thinking of Herman van Loon Mark Brough Taryn Davis [~379] matmaxgeds and others…
Scenario
With FTS, we want to import IATI activities, to construct Flows. A (obvious) vital cog to this is the IATI transaction.
We also know that IATI publishers will be adding transactions to their activities over time - which is expected behavior.
So -
- On first import an activity has two transactions.
- When the activity is updated, a month later, two more transactions have been added by the publisher. The activity now has four transactions.
- When we import the updated activity, we know it’s some data we’ve encountered before, because the
activity-identifier
is present. That’s helpful! - However, the transactions have no identifier. Our import now gives us six transactions (the first two, plus the new four)!
I can picture people frowning at this! Of course, our system and data import routines could have some extra process to generate and store a hash of the data, for example - or even check the transactions dates and values etc etc. We understand that, but the point is: do we think it efficient to shift this overhead from the publisher to data users, each of whom would have to work out the problem and compute a workaround?
A solution
The IATI transaction
element does actually include a @ref attribute, which could be use to declare an identifier for a transaction. For example:
<transaction ref="1">
<transaction ref="2">
<transaction ref="3">
<transaction ref="4">
Or, more verbosely:
<transaction ref="EX-AMPLE-ORG-ACTIVITY100-transaction1">
<transaction ref="EX-AMPLE-ORG-ACTIVITY100-transaction2">
<transaction ref="EX-AMPLE-ORG-ACTIVITY100-transaction3">
<transaction ref="EX-AMPLE-ORG-ACTIVITY100-transaction4">
Or even:
<transaction ref="oranges">
<transaction ref="apples">
<transaction ref="pears">
<transaction ref="cashews">
(OK, that last example isn’t advisable, but Reid Porter might like it)
For our import to FTS, we know this solution actually works! When publishers include a @ref, it acts as an identifier we can check, to understand if this is a transaction we’ve seen before.
We are aware of scenarios such as transactions being edited, or data being added to a transaction (a receiver-org activity-identifier, for example), but would appreciate we try and stick to the main point here: should we strive to include a @ref that can act as a transaction identifier within any activity, to help data users?
It’d be great to hear from others.
Hi Steven Flower thanks for raising this isssue. Here is the methodology we used in Bangladesh for handling updates. We use a similar process to Taryn Davis ’ approach in AMP. We first see whether the transaction already exists in the AIMS, considering a transaction as unique based on the transaction date, the transaction type and the transaction value (see the code here). We stop import if there is data in the AIMS that is not in IATI, on the basis that a user may have entered data into the AIMS that they wouldn’t then want overwritten by IATI (because the effect would be to either double-count funds or to remove existing information).
This seems to have worked in practice (because it is unlikely to happen that a transaction with identical values for (date, type, value) is retrospectively added on the same date as an existing transaction). Having said that, I think I agree with Bill Anderson that wiping all the transactions and then inserting new ones is the safest way of proceeding. I guess it would just make things slower, though possibly not that much slower, I’m not sure.
It agree it would be nice to have unique references for transactions and perhaps we can move that way over time, but in the interim we will need techniques like this to continue to handle this kind of issue.