Particular characters in JSON cause exception creating package on API

[florian.marienfeld@…]

From Florian:

I am having trouble with the characters "=&;".
if they occur e.g. in the title of a package, I get a 400-Bad-Request
when I try to register a package by POSTing to ckan/rest/api/package

However, when I PUT the same package as an edit to
ckan/rest/api/package/existing_package, it works and the title shows
correctly on the front end and API.

Reproduced

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &"}' -H "Authorization:tester" 
HTTP/1.0 400 Bad Request
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 14:09:31 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 87

"Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)"

[dread]

The default Content-Type is "application/x-www-form-urlencoded" (that is what curl sends, and what CKAN defaults to). Therefore these characters need url encoding ("=&;" -> "%3D%26%3B"). Here is an example of it working:

(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test7", "title": "Test2 %3D%26%3B"}' -H "Authorization:tester"
HTTP/1.0 201 Created
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 16:36:32 GMT
Pragma: no-cache
Cache-Control: no-cache
Location: http://localhost:5000/api/rest/package/5866a0f5-905a-44ff-91c3-c1489374a73e
Content-Type: application/json;charset=utf-8
Content-Length: 571

{"maintainer": null, "maintainer_email": null, "id": "5866a0f5-905a-44ff-91c3-c1489374a73e", "metadata_created": "2011-09-06T14:06:40.735488", "relationships": [], "metadata_modified": "2011-09-06T16:36:32.543461", "author": null, "author_email": null, "state": "active", "version": null, "license_id": null, "resources": [], "tags": [], "groups": [], "name": "test7", "license": null, "notes_rendered": "", "url": null, "notes": null, "title": "Test2 =&;", "ratings_average": null, "extras": {}, "ratings_count": 0, "revision_id": "92ee0b2e-c5e4-47b9-b00c-b8e29662b950"}

Alternatively you can set the content type to "application/json" and you don't need to URL encode. e.g.:

curl -i http://localhost:5000/api/rest/package -d '{"name": "test5", "title": "Test &"}' -H "Authorization:tester" -H "Content-Type: application/json;charset=utf-8"

To help work out these problems with encoding in the future, I've improved the error message to show the decoded JSON string. cset:55273629a552 which is headed for CKAN 1.5 release.