Registry API Schema Error Since October '16 Update


(Robert Burnett) #1

I’ve encountered a schema error (“key already exists”) stemming from the “Extras” hash when attempting to create or update datasets via the Registry API. My code had been working for these actions prior to the recent API update in October.

Folks from IATI support (and CKAN) have been notified and are looking into it; however in the meantime, I wondered if anyone else in the API user community uses the “extras” hash when adding or updating datasets, and if so whether you have encountered any similar hiccups. If anyone is using that section and it’s working, I’d love to hear from you!

Regards -
Robert


(Herman van Loon) #2

I encountered some strange behaviour if the registry as well. The organisation file of publisher ‘achest’ has an incorrect filetype ‘activity’ in the json response from the registry. Unclear if this is a registry problem or an data entry problem. Imo the registry should check on the actual file type of the supplied XML, before setting the filetype attribute.

Regards
Herman


(Steven Hicks) #3

I too am encountering the error: There is a schema field with the same name regarding the ‘extras’.

Since some of the data I need to update is in this area (activity_count and data_updated), it is kind of important that the API actually works.

Is there any traction on this issue?


(Robert Burnett) #4

Hi Steven,
I managed to figure this one out a few months ago after looking at the registry source code and changelog. There was a change in how the ‘extras’ were handled.

There had been attributes we had to submit within an “extras” hash because the registry did not autoconvert them. Many (at least all of the ones I’ve been using) are now accounted for, so rather than this:

{
  filetype: "activity",
  owner_org: "ia_nam",
  ...,
  extras: {
    iati_version: "2.1",
    data_updated: "2018-01-01",
    activity_count: 5
  }
}

we now have to reposition those extras because they are now accounted for in the schema by auto-conversion. So adding them explicitly creates a conflict. So, now I do:

{
  filetype: "activity",
  owner_org: "ia_nam",
  ...,
  iati_version: "2.1",
  data_updated: "2018-01-01",
  activity_count: 5
}

without specifying extras, as they are now explicitly auto-converted by the registry.

The above works great if you’re adding brand new packages. However, I found out that updating existing packages that were submitted the old way (with an “extras” hash in our code) may fail if you just patch the fields like the CKAN docs recommend: running package_search or package_show and then calling package_update on the result with new data. The problem is that the old extras hang around, so what I did was make sure to add a line to delete the “extras” hash from my package_show/search result before patching with package_update. Then there’s no conflict to throw the schema error.

Happy to send a code snippet if needed. Hope this helps!


(Robert Burnett) #5

For reference, if you like: https://github.com/ViderumGlobal/ckanext-iati/blob/9fd3ce35441c0416e5876ef8b785acd22a6ef52e/ckanext/iati/plugins.py

check out the method “_modify_package_schema” to see all the fields that are explicitly converted now.


(Steven Hicks) #6

Robert - your suggestion worked perfectly and saved me from digging through the code. Thanks so much!


(Robert Burnett) #7

Fantastic – glad to hear it!