Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25.

"when logging into ​http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."

n the manual it says an API key kan be created via ​http://test.ckan.net/user/me /. However, when I try the corresponding ​http://register.data.overheid.nl/user/me, I get a 404 error (not found).

With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes.

Pawel knows about this so David Read, Pawel and I need to find time to discuss it.

1. Register User with an email address with a unicode char (e.g. u'\u044e')
2. View the User in the UI (/user/) or with 'user_show' Action API

Exception:

Module ckan.controllers.user:98 in read
<<          try:
                   user_dict = get_action('user_show')(context,data_dict)
               except NotFound:
                   h.redirect_to(controller='user', action='login', id=None)
>>  user_dict = get_action('user_show')(context,data_dict)
Module ckan.logic.action.get:488 in user_show
<<      check_access('user_show',context, data_dict)
       
           user_dict = user_dictize(user_obj,context)
       
           if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name):
>>  user_dict = user_dictize(user_obj,context)
Module ckan.lib.dictization.model_dictize:189 in user_dictize
<<      
           result_dict['display_name'] = user.display_name
           result_dict['email_hash'] = user.email_hash
           result_dict['number_of_edits'] = user.number_of_edits()
           result_dict['number_administered_packages'] = user.number_administered_packages()
>>  result_dict['email_hash'] = user.email_hash
Module ckan.model.user:59 in email_hash
<<          if self.email:
                   e = self.email.strip().lower()
               return hashlib.md5(e).hexdigest()
               
           def get_reference_preferred_for_uri(self):
>>  return hashlib.md5(e).hexdigest()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128)

Three JS bugs have appeared:

• The slug editor is not updating the text field again. [Fixed; commit:fadda17383]

• The View Dataset page will stop executing javascript if there are no resources. [Fixed; commit:e54b089f6e]

• The Add-Dataset and Add-Group URL editor shows the wrong URL for datasets and groups. Broken in this commit:

​https://github.com/okfn/ckan/commit/5a01e67 by 'map.explicit=true'. JohnG: Can you look into this? [Fixed; commit:88e88380f6]

e.g.

curl http://localhost:5000/api/action/package_create -d '{"name":"test5", "tags":[{"name": "tag6"}, {"name": "tag6"}]}' -H 'Authorization:tester'

Also broken in Web UI (adding a package).

Affects API and Web UI in 1.5 and I think all versions from 1.4.1

Sean spotted that some strings need internationalising, such as flash messages. Generally all strings that appear on the web front-end should be internationalised.

Particular places that need i18n:

• ckan/controllers/admin
• ckan/lib/base

Note: there are some exceptions, such as in i18n.py, very obscure error messages and stuff that only appears on the API.

NB: there is a cost in making a string internationalisable (all our volunteers have to translate it), so we should not be too zealous. We should also look at the i18n/ckan.pot to see if any of the existing strings can be reused.

You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank.

Should assert that one of these are set up.

Most of the different pagination links are broken and point to wrong locations. e.g:

• Dataset listings in group page:

​http://test.ckan.net/group/bibliographic

• Tag letters in tag listing:

​http://test.ckan.net/tag

• User listing:

​http://test.ckan.net/user

• Revision listing:

​http://test.ckan.net/revision

This is most probably related with:

​https://github.com/okfn/ckan/commit/1772a5c

This is what caused the exception:

curl http://localhost:5000/api/action/package_search -d='{"q": "osm"}'

The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary.

e.g. ​http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358 gives:

This is the current revision of this dataset, as edited at 2011-07-12 12:16.

but it should say the date 2011-07-12 (which is the data being displayed).

The problem is looking at PackageRevision?, when it might be a tag or extra.

Richard Cyganiak: I often see a problem where half the system thinks that I'm logged in, while the other half insists I'm not. Logout then login again may fix it.

David Read: I think I've seen it several times that the "My Account" link is given (suggesting you're logged in) but when you click it it asks you to log in, which means you weren't.

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: ​http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.

Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.

Superceded by #2234

Super ticket: #1508

Etherpad: ​http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved ​https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

Both in 1.5.1b:

​http://thedatahub.org/api/search/dataset?groups=lodcloud&title=

and 1.5.2a (current master):

​http://test.ckan.net/api/search/dataset?groups=lodcloud&title=

Although the error message in 1.5.2a is more verbose:

"Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.ParseException?: Cannot parse 'groups:lodcloud title:': Encountered \"<EOF>\" at line 1, column 22. Was expecting one of: \"(\" ... \"*\" ... <QUOTED> ... <TERM> ... <PREFIXTERM> ... <WILDTERM> ... \"[\" ... \"{\" ... <NUMBER> ..."

Some parameter validation before sending it to Solr should do the trick

metadata_created seems to be returning the date of the earliest revision (for all packages), rather than the earliest revision for a specific package.

Looks like uploading a file with foreign characters fails due to encoding reasons.

URL: http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:123 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckanext.storage.controller:2 in auth_form
Module ckan.lib.jsonp:26 in jsonpify
<<      Very much modelled after pylons.decorators.jsonify .
           """
           data = func(*args, **kwargs)
           return to_jsonp(data)
>>  data = func(*args, **kwargs)
Module ckanext.storage.controller:301 in auth_form
<<          method = 'POST'
               authorize(method, bucket, label, c.userobj, self.ofs)
               data = self._get_form_data(label)
               return data
>>  authorize(method, bucket, label, c.userobj, self.ofs)
Module ckanext.storage.controller:79 in authorize
<<      if method != 'GET':
               # do not allow overwriting
               if ofs.exists(bucket, key):
                   abort(409)
               # now check user stuff
>>  if ofs.exists(bucket, key):
Module ofs.remote.botostore:53 in exists
<<          if bucket is None: 
                   return False
               return (label is None) or (label in bucket)
           
           def claim_bucket(self, bucket):
>>  return (label is None) or (label in bucket)
Module boto.s3.bucket:87 in __contains__
<<      def __contains__(self, key_name):
              return not (self.get_key(key_name) is None)
       
           def startElement(self, name, attrs, connection):
>>  return not (self.get_key(key_name) is None)
Module boto.s3.bucket:144 in get_key
<<          response = self.connection.make_request('HEAD', self.name, key_name,
                                                       headers=headers,
                                                       query_args=query_args)
               # Allow any success status (2xx) - for example this lets us
               # support Range gets, which return status 206:
>>  query_args=query_args)
Module boto.s3.connection:388 in make_request
<<          if isinstance(key, Key):
                   key = key.name
               path = self.calling_format.build_path_base(bucket, key)
               boto.log.debug('path=%s' % path)
               auth_path = self.calling_format.build_auth_path(bucket, key)
>>  path = self.calling_format.build_path_base(bucket, key)
Module boto.s3.connection:88 in build_path_base
<<      def build_path_base(self, bucket, key=''):
               return '/%s' % urllib.quote(key)
       
       class SubdomainCallingFormat(_CallingFormat):
>>  return '/%s' % urllib.quote(key)
Module urllib:1222 in quote
<<              safe_map[c] = (c in safe) and c or ('%%%02X' % i)
               _safemaps[cachekey] = safe_map
           res = map(safe_map.__getitem__, s)
           return ''.join(res)
>>  res = map(safe_map.__getitem__, s)
KeyError: u'\xed'
CGI Variables
AUTH_TYPE	'cookie'
CONTENT_TYPE	'; charset=utf-8'
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
HTTP_ACCEPT_ENCODING	'gzip,deflate,sdch'
HTTP_ACCEPT_LANGUAGE	'en-US,en;q=0.8'
HTTP_CACHE_CONTROL	'max-age=259200'
HTTP_CONNECTION	'keep-alive'
HTTP_COOKIE	'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; ckan_user=elenaibp; ckan_display_name="Elena Mondo"; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; __utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10.1324298316; __utmc=119670349; __utmz=119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'
HTTP_HOST	'thedatahub.org'
HTTP_REFERER	'http://thedatahub.org/dataset/edit/budget-library-czeck-republic'
HTTP_USER_AGENT	'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7'
HTTP_VIA	'1.1 localhost (squid/3.0.STABLE19)'
HTTP_X_FORWARDED_FOR	'87.114.74.190'
HTTP_X_REQUESTED_WITH	'XMLHttpRequest'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
REMOTE_ADDR	'193.34.146.142'
REMOTE_PORT	'55419'
REMOTE_USER	u'elenaibp'
REMOTE_USER_DATA	'userid_type:unicode'
REMOTE_USER_TOKENS	['']
REQUEST_METHOD	'GET'
REQUEST_URI	'/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
SCRIPT_URL	'/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf'
SERVER_ADDR	'193.34.146.146'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'thedatahub.org'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.0'
SERVER_SIGNATURE	'<address>Apache/2.2.14 (Ubuntu) Server at thedatahub.org Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.14 (Ubuntu)'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x7f22601c7dd0>
beaker.cache	<beaker.cache.CacheManager object at 0x7f22601c7b50>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f22601c7a90>>
beaker.session	{'_accessed_time': 1324298703.071357, '_creation_time': 1324293077.4139669}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	'ckan.net'
mod_wsgi.reload_mechanism	'1'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 8)
paste.cookies	(<SimpleCookie: __utma='119670349.1809834699.1323782464.1324293066.1324298316.4' __utmb='119670349.3.10.1324298316' __utmc='119670349' __utmz='119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode' ckan_apikey='decd48b1-49ee-4250-bff4-98ccca9c02a5' ckan_display_name='Elena Mondo' ckan_user='elenaibp' hide_welcome_message='1' thedatahub_net='27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e'>, 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; ckan_user=elenaibp; ckan_display_name="Elena Mondo"; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; _ _utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10...)|utmcmd=(none)')
paste.registry	<paste.registry.Registry object at 0x7f226194df50>
paste.throw_errors	True
pylons.action_method	<bound method StorageAPIController.auth_form of <ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990>>
pylons.controller	<ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x7f2261daddd0>
pylons.routes_dict	{'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'}
repoze.who.identity	<repoze.who identity (hidden, dict-like) at 139785645747120>
repoze.who.logger	<logging.Logger instance at 0x7f225e23c098>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 139785625065680>, 'friendlyform': <FriendlyFormPlugin 139785618095248>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f2260874c10>, 'auth_tkt': <AuthTktCookiePlugin 139785625065808>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f2260874c90>}
routes.route	<routes.route.Route object at 0x7f22601a1090>
routes.url	<routes.util.URLGenerator object at 0x7f2261dadf50>
webob._parsed_query_vars	(GET([]), '')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multiprocess'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f2261da9af8>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0x7f2261dadf50>, {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'})

1. Viewing a dataset, hit 'Edit'
2. Click on the "Basic Information" tab (note: URL has suffix #section-basic-information
3. Click 'Save'
4. URL still has suffix #section-basic-information

Affects 1.5, 1.5.1, 1.5.2a

A test case is currently failing for me on master:

ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py", line 342, in test_user_create_unicode
    res = res.follow()
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 603, in follow
    return self.test_app.get(location, **kw)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 208, in get
    return self.do_request(req, status=status)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 389, in do_request
    **req.environ)
  File "/usr/lib/pymodules/python2.6/paste/wsgilib.py", line 343, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/lint.py", line 170, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/usr/lib/pymodules/python2.6/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 69, in __call__
    auth_ids = self.authenticate(environ, classification, ids)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 201, in authenticate
    userid = plugin.authenticate(environ, identity)
  File "/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py", line 29, in authenticate
    if user.validate_password(identity.get('password')):
  File "/home/seanh/pyenv/src/ckan/ckan/model/user.py", line 113, in validate_password
    hashed_pass = sha1(password_8bit + self.password[:40])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)

On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab.

Original ticket #48

This was seen on datacatalogs. When we upgraded it 1.5 to 1.5.1 we saw some permissions being reset so that it could be spammed. Anonymous and logged in users were given anon_editor and editor permissions. I don't know what the previous config was. Permissions for sysadmins remain unaffected.

When viewing a dataset, the "Resources" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu.

​http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f

Chrome/Linux?

This CSV resource used to preview but now the format appears to be unsupported: "We are unable to preview this type of resource: x-osdata-csv"

​http://thedatahub.org/dataset/lbhf-spending-2010/resource/9661abbd-2816-4d58-8b20-3cb0eb770c69

This is used as an example by the OpenSpending? team all the time.

Mounting CKAN at a non-root URL was made to work properly here: #1659

Unfortunately OpenID doesn't play nicely and would require some work to get working.

On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect.

This is a known issue: ​https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04)

It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently).

To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function.

This behaviour is now deprecated however.

The SOLR url, user and password defined in the CKAN config file are ignored and the default SOLR url is used.

This causes:

• "0 datasets" displayed on the home page
• Dataset searches result in 0 results and a small message "There was an error while searching". (Nothing about it in the logs)

To reproduce

This bug is only visible if your SOLR instance is not at the default place. To quickly reproduce this problem, setup your machine as a SOLR multicore instance and run: "paster db clean && paster create-test-data && paster serve development.ini". It quits with the error: "solr.core.SolrException?: HTTP code=400, reason=Missing solr core name in path"

Code affected

• Does not affect any released versions (1.5.1 is clean).
• Introduced to master: ​https://github.com/okfn/ckan/commit/627382e6ff575146da221efc50321a41152d8b0f by Ross

I get this warning when viewing the home page:

event.layerX and event.layerY are broken and deprecated in WebKit. They will be removed from the engine in the near future.

CKAN: latest master Browser: Chrome Ubuntu

Minor, but worth checking?

If you use tag autocompletion, it automatically adds a comma after the tag. This is fine if you are going to add another tag, but not fine if it is the last one. If you don't manually remove the final comma than you get form validation error "Tag string: Tag "" length is less than minimum 2"

Affects only version 1.5.2a.

If the current page URL has unicode in it (e.g. u'/tag/biocombust\xedveis') then I'm not sure how why it isn't more encoded, but we are seeing links such as http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251 which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. ​http://127.0.0.1:5000/dataset/new?__cache=37713707

When you are logged in, you cannot login as a different user before first logging out. (This is a restricting in repoze.who, which gives priority to an existing recognised identity over another one you supply.)

Yet, the CKAN login form appears when you try to do something you're not allowed to, even if you are logged in already. You login with a different identity and nothing happens - you remain as the first identity - this is incorrect.

(It is also a little confusing to be sent to the log-in page, although we do have a flash message explaining, and you might want to relogin as a superuser if you can.)

Whilst in this area, it would be worth fixing the problem for developers where you are logged in, do "paster db clean" and then try to log in - AuthTkt? recognises your old cookie but the User object doesn't exist, so you get an error logging in. In this instance we should tell people to log-out.

On the homepage click "francais" and then "English". The flash message reads "Le langage a été fixé à: français" when it should say "The language is now: English".

Apparently it's working when developers run it locally but not on the demo site.

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.

Nothing happens if you set resources=[] or resources=null.

Major UX gotcha reported by several people is that DataStore? is disabled by default and must be enabled for DataStorer? to run and import data.

Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want.

There are a couple of flash messages in ckan/controllers/user.py that show up in English, even if another language is in use, and even though the strings are marked for translation with gettext and have translations in the po and mo files:

"[USER NAME] is now logged in"

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

Can only add one dataset at a time now (not sure when this broke but was working ~2-3m ago).

Assigning to seanh for review and estimation. I'd rate this as quite annoying but not absolutely critical (hence assigned priority).

Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients).

Fix this by ensuring that error messsages on API get passed through correctly.

Minor because affect is annoying but not terminal

Appears current default search in CKAN ORs terms rather than ANDing them (i.e. adding more terms increasing number of items found rather than reducing it).

Not sure when this crept in or if it has been there for a long time.

e.g. 'Cabinet Office' returns 0 results, whereas 'Cabinet' returns 'Cabinet Office'.

/api/2/util/authorizationgroup/autocomplete?q=Cabinet%20Office

This feature only seems to be used by DGU at the moment (I think)

If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init.

Can't you just modify the clean and init functions to run without extensions enabled?

The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket.

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

There are a few errors or unclarities in the source install doc - see

​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html ​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html