Changes to the updating of dashboard publisher statistics


(Bill Anderson) #1

The data on the IATI dashboard, including the Publisher Statistics, are currently updated on a daily basis. This process now takes more than 24 hours, and the overlap is resulting in instability and unpredictable outcomes.

The long-term solution to this will involve an overhaul of the underlying data architecture. In the short-term the update will now take place every three days. As from the 1st February 2018 it will be run shorty after midnight on the following days each month:

  • 1st, 4th, 7th, 10th, 13th, 16th, 19th, 22nd, 25th. (and the 28th for months with 31 days)

Timeliness calculations

This has a potential impact on the way timeliness is assessed for those publishers who refresh their data once, and only once, a month. A publisher update made on the last day of the month will only be detected and recorded in the following month. Publishers who perform a monthly refresh of data between the 28th and 31st of January will not be counted for January only. This will not affect their overall ‘monthly’ assessment (which looks for updates in 7 or more of the past 12 full months AND data updated at least once in last 2 full months)


(Michelle Levesque) #2

Dear Bill,

Do you know if the dashboard will be updated for this new schedule? It still says the dashboard is updated monthly?

image

Kind Regards,
Michelle
IOM


(Andy Lulham) #3

Good point, @Michelle_IOM.

I’ve created an issue on github: https://github.com/IATI/IATI-Dashboard/issues/470

…and have submitted a fix: https://github.com/IATI/IATI-Dashboard/pull/471

It turns out the dashboard mentions the ‘nightly update’ a lot. I’ve replaced all the occurrences I could find, but there may be more.


(Andy Lulham) #4

I’ve just sent a fix that reduces the processing time significantly. Benchmarking suggests it’s faster by about 18 hours: https://github.com/IATI/IATI-Dashboard/pull/495


(Rory Scott) #5

Nice! Thanks Andy - I’ve been playing around with the dashboard recently, and this will be really helpful :pray: