API POST barfs on interesting Content-Type headers

[dread]

When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: "Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'""

The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header.

This affects all versions of CKAN.

This is due to webob/requests.py:1248 being pretty basic.

[dread]

Unfortunately with a few hours hacking I couldn't find a way to get this version of WebOb? to cope with this header.

The line I think it goes wrong is ctype = env.get('CONTENT_TYPE', 'application/x-www-form-urlencoded') in request.py, and the commented line above looks right, but wipes the request data for some reason. I tried intercepting the bad header and deleting it, but I couldn't do that before WebOb? processed it, and couldn't persuade WebOb? to reprocess it once the header was edited.

I did write a failing test though if that is useful to someone in the future:

diff --git a/ckan/tests/functional/api/test_api.py b/ckan/tests/functional/api/t
index 539d184..b4cc4ce 100644
--- a/ckan/tests/functional/api/test_api.py
+++ b/ckan/tests/functional/api/test_api.py
@@ -49,6 +49,18 @@ class TestApi3(Api3TestCase, ApiTestCase):
         assert_in('Bad request - JSON Error: No request body data',
                   res.body)
 
+    def test_content_type_headers_can_be_sent(self):
+        '''#2942 Webob can only cope with very specific Content-Type header
+        values, so this tests that a work-around is in place.
+
+        This particular header value is the default for 'requests' 0.14.
+        '''
+        offset = self.offset('/action/package_search')
+        params = '%s=1' % json.dumps({'q': 'russian'})
+        headers = {'Content-Type': '; charset=utf-8'}
+        res = self.app.post(offset, params=params, headers=headers,
+                            status=[200])
+

There is a clear error to the user:

>>> requests.post('http://datahub.io/api/action/package_list', data='{}').content
'"Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'"'

The advice to users of requests would be to ensure they specify the content-type as follows:

>>> requests.post('http://datahub.io/api/action/package_list', data='{}', headers={'content-type': 'application/x-www-form-urlencoded'}).content

[dread]

>>> requests.post('http://datahub.io/api/action/package_list', data='{}', headers={'content-type': 'application/x-www-form-urlencoded'}).content

Or even better:

>>> requests.post('http://datahub.io/api/action/package_list', data='{}', headers={'content-type': 'application/json'}).content