API improvements (following javascript use)

[dread]

Notes on the CKAN API from Aron:

• Server returns text/html for errors even when "Accept: application/json" header is set.
• DELETE package request requires a Content-Length? Should be 405 Method Not Allowed?
• Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content.
• How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work.
• Tags return package names rather than ids when querying GET /tag/{id}
• Doesn't return rendered_text property on package update.
• Relationships use "object" key rather than id or package_id.

[dread]

I'm working through these on branch defect-1214-api-improvements.

"Server returns text/html for errors" - now fixed in cset:0a533a8da3df

[dread]

Aron, I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: "Should be 405 Method Not Allowed?" DELETE should be allowed. Or is that a response you're getting? David

[dread]

"Tag returned as a JSON object when updating but as a string when requesting." I'm not sure what you mean.

​http://test.ckan.net/api/2/rest/package/reference-example gives tags: "tags": ["country-uk", "example-package"],

Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix. ​http://test.ckan.net/api/2/rest/tag/country-uk

[dread]

Fixed tag entity returning package names for apiv2 - it now gives IDs.

The problem mentioned about "Extras" content - I'm not sure what's meant here. Package extras are just strings. You might choose to store a package name in an extra, but there is no magic to convert it to a package id should you retrieve the package via apiv2. Is this just a misunderstanding or have I missed your point?

[dread]

Deleting an extra using 'null' works for me:

$ curl -d '{"name":"dtest2", "extras":{"1":"1", "2":"2", "3":"3"}}' http://test.ckan.net/api/rest/package -H "Authorization: tester" 
...
$ curl -d '{"name":"dtest2", "extras":{"1":"1", "2":"2", "3":null}}' http://test.ckan.net/api/rest/package -H "Authorization: tester" 
{"maintainer": null, "name": "dtest2", "relationships_as_subject": [], "author": null, "url": null, "relationships_as_object": [], "notes": null, "title": "dtest2", "maintainer_email": null, "revision_timestamp": "2011-07-07T12:57:18.454890", "author_email": null, "state": "active", "version": null, "groups": [], "license_id": null, "revision_id": "f0ff31c0-027b-49ce-9daf-94a73d96a913", "tags": [], "id": "fdeeb287-2783-4aac-9fc7-a6717e54e22f", "resources": [], "extras": [{"state": "active", "value": "\"1\"", "revision_timestamp": "2011-07-07T12:57:18.454890", "package_id": "fdeeb287-2783-4aac-9fc7-a6717e54e22f", "key": "1", "revision_id": "f0ff31c0-027b-49ce-9daf-94a73d96a913", "id": "d1937073-7bfc-48c5-b6ff-b00d90b451ae"}, {"state": "active", "value": "\"2\"", "revision_timestamp": "2011-07-07T12:57:18.454890", "package_id": "fdeeb287-2783-4aac-9fc7-a6717e54e22f", "key": "2", "revision_id": "f0ff31c0-027b-49ce-9daf-94a73d96a913", "id": "8147886f-9769-440c-8c35-b7d6a2f46de7"}]}

[dread]

Ok I see the problem with extras. When you create a package you get back 'simple' extras:

$ curl -d '{"name":"dtest3", "extras":{"1":"1"}, "tags":["australia", "barbeque"]}' http://test.ckan.net/api/rest/package -H "Authorization: tester" --trace log
{"maintainer": null, "maintainer_email": null, "id": "33d75910-ae5e-45b0-a154-d00022555a43", "metadata_created": "2007-04-10T21:19:38", "relationships": [], "metadata_modified": "2011-07-07T13:06:06.228718", "author": null, "author_email": null, "state": "active", "version": null, "license_id": null, "resources": [], "tags": ["australia", "barbeque"], "groups": [], "name": "dtest3", "license": null, "notes_rendered": "", "url": null, "ckan_url": "http://test.ckan.net/package/dtest3", "notes": null, "title": "dtest3", "ratings_average": null, "extras": {"1": "1"}, "ratings_count": 0, "revision_id": "31cb22d4-3452-4adc-b572-cd173aea7d

i.e. "extras": {"1": "1"} and that's what you get if you GET the package:

$ curl http://test.ckan.net/api/rest/package/dtest3
{"maintainer": null, "maintainer_email": null, "id": "33d75910-ae5e-45b0-a154-d00022555a43", "metadata_created": "2007-04-10T21:19:38", "relationships": [], "metadata_modified": "2011-07-07T13:06:06.228718", "author": null, "author_email": null, "state": "active", "version": null, "license_id": null, "resources": [], "tags": ["australia", "barbeque"], "groups": [], "name": "dtest3", "license": null, "notes_rendered": "", "url": null, "ckan_url": "http://test.ckan.net/package/dtest3", "notes": null, "title": "dtest3", "ratings_average": null, "extras": {"1": "1"}, "ratings_count": 0, "revision_id": "31cb22d4-3452-4adc-b572-cd173aea7d11"}

but if you then edit it you get them expressed much more verbosely:

$ curl -d '{"name":"dtest3", "extras":{"1":"2"}, "tags":["australia", "barbeque"]}' http://test.ckan.net/api/rest/package/dtest3 -H "Authorization: tester" --trace log
{"maintainer": null, "name": "dtest3", "relationships_as_subject": [], "author": null, "url": null, "relationships_as_object": [], "notes": null, "title": "dtest3", "maintainer_email": null, "revision_timestamp": "2011-07-07T13:06:06.228718", "author_email": null, "state": "active", "version": null, "groups": [], "license_id": null, "revision_id": "31cb22d4-3452-4adc-b572-cd173aea7d11", "tags": [{"revision_timestamp": "2011-07-07T13:06:06.228718", "state": "active", "id": "4aa0e776-ac2a-4a8b-82ba-d80237d35596", "name": "australia"}, {"revision_timestamp": "2011-07-07T13:06:06.228718", "state": "active", "id": "e890354d-e170-44c1-94b1-f0d8b38a49fc", "name": "barbeque"}], "id": "33d75910-ae5e-45b0-a154-d00022555a43", "resources": [], "extras": [{"state": "active", "value": "\"2\"", "revision_timestamp": "2011-07-07T13:21:02.220368", "package_id": "33d75910-ae5e-45b0-a154-d00022555a43", "key": "1", "revision_id": "61703821-a2f1-41f8-8f39-71130c5e6c1b", "id": "87d7d4fc-7eb7-4760-b199-e460ce505632"}]}

I'm going to ask David Raznick about what's best here. It would be good to make them uniform.

[aron]

Replying to dread:

Aron, I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: "Should be 405 Method Not Allowed?" DELETE should be allowed. Or is that a response you're getting? David

If I send a HEAD request to the server eg.

curl -i -XHEAD http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf

I get the following in the Access-Control-Allow-Methods header

Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS

I've just realised that the "Access-Control" headers are for cross origin browser requests so I think you can ignore my comments about the correct response being 405 Method Not Allowed but perhaps DELETE should be added to the Access-Control-Allow-Methods list?

[aron]

Replying to dread:

"Tag returned as a JSON object when updating but as a string when requesting." I'm not sure what you mean.

​http://test.ckan.net/api/2/rest/package/reference-example gives tags: "tags": ["country-uk", "example-package"],

I'm getting back objects for each tag in the response body from a PUT request. For example "country-uk" would be something like.

{ 
        "id" : "600dc72e-6127-4704-b801-bee00474ec0c",
        "name" : "country-uk",
        "revision_timestamp" : "2011-07-06T09:09:21.578034",
        "state" : "active"
}

Hopefully this gist <​https://gist.github.com/97447d0b28bf52f3e06b> will illustrate the issue.

Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix. ​http://test.ckan.net/api/2/rest/tag/country-uk

That was the fifth point on the list.

[aron]

Replying to dread:

Deleting an extra using 'null' works for me:

Hmm, it works for me too when there are other keys remaining in the "extras" object. However I can't seem to delete the last one. Take the following example.

curl -i -XPUT -d'{"extras": {"Tester": null}}' -H"X-CKAN-Type: application/json" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf

Gives:

{ "author" : "",
  "author_email" : "",
  "extras" : [ { "id" : "88cc50ca-9e29-4499-a542-09d364f5f64f",
        "key" : "Tester",
        "package_id" : "ec9cb930-d15f-441f-a1e1-36f4d5df19bf",
        "revision_id" : "e5c3ca9c-1dae-4a86-837f-b8c19ac31964",
        "revision_timestamp" : "2011-07-07T16:01:05.696025",
        "state" : "active",
        "value" : "\"Test Value\""
      } ],
  "groups" : [  ],
  "id" : "ec9cb930-d15f-441f-a1e1-36f4d5df19bf",
  "license_id" : "",
  "maintainer" : "aron",
  "maintainer_email" : "",
  "name" : "my-test-package",
  "notes" : "Heading\r\n===\r\nThis _is_ some text",
  "relationships_as_object" : [  ],
  "relationships_as_subject" : [  ],
  "resources" : [  ],
  "revision_id" : "688d33fb-5629-4ab3-9f59-a649fc7caa00",
  "revision_timestamp" : "2011-07-06T10:37:50.182894",
  "state" : "active",
  "tags" : [ { "id" : "600dc72e-6127-4704-b801-bee00474ec0c",
        "name" : "test-tag",
        "revision_timestamp" : "2011-07-06T09:09:21.578034",
        "state" : "active"
      } ],
  "title" : "My Test Package",
  "url" : "",
  "version" : ""
}

[aron]

Another edge case that popped up while verifying the above issues. A PUT request without a body throws a 500 Internal Sever Error

curl -i -XPUT  -H"X-CKAN-API-KEY: tester" -H"Content-Type: application/json" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf

[aron]

One more for today. Please do let me know if I should be filing these as separate tickets?

For some reason performing a search query for packages with an underscore "_ " as a query string key fails to return any results.

curl "http://test.ckan.net/api/2/search/package?q=osm"

Gives me 4 results.

curl "http://test.ckan.net/api/2/search/package?q=osm&_=1310056826904"

Gives me none.

The underscore is generally used by JavaScript? libraries as a way of bypassing the browser cache when making JSONP calls. It's easily worked around but is odd none the less.

[dread]

Fixes have gone into the branch for "Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content." and "Doesn't return rendered_text property on package update.".

I've added DELETE to the Access-Control-Allow-Methods on the server, so you can try that now.

I've added a failing test for the problem of deleting the last extra. Am discussing with David Raznick how to fix that best.

I think the only other ones left to address are:

• Relationships use "object" key rather than id or package_id.
• underscore in search parameters

[dread]

I believe "A PUT request without a body throws a 500 Internal Sever Error" was fixed yesterday in the code. (Not deployed to test.ckan.net yet though)

[dread]

Relationships use "object" key rather than id or package_id.

I'm not totally clear on this one. The URLs for relationship editing can be package ids or names. When you GET a relationship packages are referred to in IDs for v2 API. When you create or update a relationship it returns package IDs for v2 API. Are you referring to the relationships listed in the response when you PUT a package? (If so that would be fixed by my previous fix.)

[dread]

underscore in search parameters

I think its trying to filter search results by values of an extra field called "_", even though this probably doesn't occur. If it is not too ugly, I suggest using one of your other ideas for solving the cache issue. I suggest you bring it up with Rufus or the ckan-dev list to get opinions.

[dread]

I've put the fixes onto test.ckan.net - repoen the ticket if there are any issues.

[aron]

The groups property in the package resource is always empty (or is for all packages I have viewed).

The following group lists one package.

http://test.ckan.net/api/2/rest/group/0ac963e7-ba29-49bc-83c8-98f8c1991649

But when viewing the package the groups array is empty.

http://test.ckan.net/api/2/rest/package/758c26d4-5949-4347-8b4d-023374146d94

[dread]

This appears to have been fixed by version 1.4.3.