Just a tip for developers regarding translations:

• When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: ​https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

Json decoding error on search, due to date in resources.

JavaScript? needs to be able to insert html templates into the document.

There are three common solutions to templating at the moment.

1. Insert the html into <script> tags at the bottom of the page and pull them in using the DOM and innerHTML. This is a very simple solution but not very portable.
2. Request the HTML snippets as plain text/html using XHR. This adds an extra HTTP request per template but is very clean. It would also be possible to inline the HTML into the JS at the minification stage for production.
3. Include the HTML in JavaScript? strings (horrible to maintain and bloats the js)

It makes sense to keep these with the other templates so that we can take advantage of localisation where possible. This indicates 1 or 2.

Its a pain to see the difference between branches as there are normally a lot of po file transaction. Make the default be see that they have changed without actually show the diffs themselves.

It's not possible to use umlauts (like "CKAN Österreich") in the title / text.

It's in the tab bar (maybe not the right place for a button? but I wanted it to be next to the follower tab and count) and doesn't like very much like a button

It's handy to be able to create users using the cli (e.g. dgu migration)

It would be useful to be able to manipulate packages using the paster command-line interface. In particular listing, reading and deleting packages. Seeing which packages were deleted too.

It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself.

Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric.

Questions:

• Core, or Extension, or Self-hosted?

It still contains the old build_search_index_synchronously which no longer works

It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly.

It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues.

Steps to test:

1. Modify a dataset on datadotgc.ca, redirects to CKAN

2. On save, redirects to ​http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to ​http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100

3. You can see that the Dataset has not updated correctly. Run a check on the API v1 - ​http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present

4. Check the v2 of the API - ​http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present.

5. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either.

It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict.

It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method.

It is useful for extensions to add or change items that go into the search index. Add an extension point for this.

It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is not found on searching.

Full text from Pascal:

we encountered a Problem concerning accessing Arrays/Lists.

curl -XGET 'http://ckan.net/api/rest/package/hbz_unioncatalog'

will get you amongst others:

 "extras": {"publishingInstitution":
"[u'http://lobid.org/organisation/DE-605',
u'http://lobid.org/organisation/DE-290',
u'http://lobid.org/organisation/DE-38M',
u'http://lobid.org/organisation/DE-98',
u'http://lobid.org/organisation/DE-38',
u'http://lobid.org/organisation/DE-Kn41',
u'http://lobid.org/organisation/DE-82',
u'http://lobid.org/organisation/DE-107',
u'http://lobid.org/organisation/DE-929',
u'http://lobid.org/organisation/DE-Zw1',
u'http://lobid.org/organisation/DE-832']"}

but if I try to query this:

wget
'http://ckan.net/api/search/package?q=lobid&publishingInstitution="http://lobid.org/organisation/DE-605"'

I get only two packages, among the package "hbz_unioncatalog" is
missing. (These two packages have only one value for
"publishingInstitution").

The "extra/publishingInstitution"-Array was uploaded through a "curl
-XPUT ...
 "extras": {
   "publishingInstitution":[
     "http://lobid.org/organisation/DE-605",
     "http://lobid.org/organisation/DE-290",
     "http://lobid.org/organisation/DE-38M",
     "http://lobid.org/organisation/DE-98",
     "http://lobid.org/organisation/DE-38",
     "http://lobid.org/organisation/DE-Kn41",
     "http://lobid.org/organisation/DE-82",
     "http://lobid.org/organisation/DE-107",
     "http://lobid.org/organisation/DE-929",
     "http://lobid.org/organisation/DE-Zw1",
     "http://lobid.org/organisation/DE-832"
   ]
   },
...

It is possible to create a group or a dataset called new, or edit, which will then be inaccessible as it is blocked by the existing routes with those names.

We should maintain a list of invalid names in the verification that happens behind the scenes in the UI.

It is necessary to add a helper method to Group for adding members to that group so that

• nobody needs to work with the Member model
• It handles existing membership and existing membership with a different capacity

It doesn't pass extras_as_string

It appears that the command

$ paster rights add russianfan admin warandpeace

has no effect, even though

$ paster rights remove russianfan admin warandpeace

works fine. This may be specific to something I've done, could someone confirm?

If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added.

Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ...

Is this done I know checkboxes are a pain but it looks a bit out of place compared to the other items.

Is testing complete and ready for release?

Investigate exception occured occasionally in last couple of days on ckan.net:

WebApp Error: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'key' 					

URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:73 in __call__
<<          # available in environ['pylons.routes_dict']    
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.revision:119 in diff
<<          c.revision_to = model.Session.query(model.Revision).get(
                   request.params.getone('diff'))
               diff = pkg.diff(c.revision_to, c.revision_from)
               c.diff = diff.items()
               c.diff.sort()
>>  diff = pkg.diff(c.revision_to, c.revision_from)
Module ckan.model.package:340 in diff
<<                              display_id = to_obj_rev.tag.name
                               elif obj_class.__name__ == 'PackageExtra':
                                   display_id = to_obj_rev.key
                               else:
                                   display_id = related_obj_id[:4]
>>  display_id = to_obj_rev.key
AttributeError: 'NoneType' object has no attribute 'key'
CGI Variables
DOCUMENT_ROOT	'/htdocs'
GATEWAY_INTERFACE	'CGI/1.1'
HTTP_ACCEPT	'*/*'
HTTP_ACCEPT_ENCODING	'gzip'
HTTP_ACCEPT_LANGUAGE	'zh-cn,zh-tw'
HTTP_CONNECTION	'close'
HTTP_HOST	'ckan.net'
HTTP_USER_AGENT	'Baiduspider+(+http://www.baidu.com/search/spider.htm)'
PATH	'/usr/local/bin:/usr/bin:/bin'
PATH_INFO	'/revision/diff/dbtune-audioscrobbler'
PATH_TRANSLATED	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler'
QUERY_STRING	'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
REMOTE_ADDR	'123.125.66.32'
REMOTE_PORT	'63767'
REQUEST_METHOD	'GET'
REQUEST_URI	'/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c'
SCRIPT_FILENAME	'/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py'
SCRIPT_URI	'http://ckan.net/revision/diff/dbtune-audioscrobbler'
SCRIPT_URL	'/revision/diff/dbtune-audioscrobbler'
SERVER_ADDR	'10.226.226.118'
SERVER_ADMIN	'[no address given]'
SERVER_NAME	'ckan.net'
SERVER_PORT	'80'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SIGNATURE	'<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80</address>\n'
SERVER_SOFTWARE	'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2'
WSGI Variables
application	<beaker.middleware.CacheMiddleware object at 0x9f603ec>
beaker.cache	<beaker.cache.CacheManager object at 0x9f6042c>
beaker.get_session	<bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x9f602ac>>
beaker.session	{'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281}
mod_wsgi.application_group	'ckan.net|'
mod_wsgi.callable_object	'application'
mod_wsgi.listener_host	''
mod_wsgi.listener_port	'80'
mod_wsgi.process_group	''
mod_wsgi.reload_mechanism	'0'
mod_wsgi.script_reloading	'1'
mod_wsgi.version	(2, 5)
paste.cookies	(<SimpleCookie: >, '')
paste.parsed_querystring	([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
paste.registry	<paste.registry.Registry object at 0x104552ec>
paste.throw_errors	True
pylons.action_method	<bound method RevisionController.diff of <ckan.controllers.revision.RevisionController object at 0xfb17aec>>
pylons.controller	<ckan.controllers.revision.RevisionController object at 0xfb17aec>
pylons.environ_config	{'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons	<pylons.util.PylonsContext object at 0x10286d4c>
pylons.routes_dict	{'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}
repoze.who.logger	<logging.Logger instance at 0xa16e0cc>
repoze.who.plugins	{'openid': <OpenIdIdentificationPlugin 167584972>, 'auth_tkt': <AuthTktCookiePlugin 169253516>}
routes.route	<routes.route.Route object at 0x9f3690c>
routes.url	<routes.util.URLGenerator object at 0xfd8d7cc>
webob._parsed_query_vars	(GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c')
webob.adhoc_attrs	{'language': 'en-us'}
wsgi process	'Multi process AND threads (?)'
wsgi.file_wrapper	<built-in method file_wrapper of mod_wsgi.Adapter object at 0x103a5bf0>
wsgiorg.routing_args	(<routes.util.URLGenerator object at 0xfd8d7cc>, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'})

Intermittent failure on ckan default branch. Looks like brittle test.

======================================================================
FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py", line 250, in test_02_metadata_created_and_modified
    assert out == exp, (out, exp)
AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938))

​http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio

Integrating ckanext-admin into core has thrown up a number of problems:

• Look & feel does not match the rest of the site
• Tests are not passing
• On the trash page, clicking 'undelete' triggers a purge
• Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org)
• Trash page can contain nested form tags in certain cases (breaking test harness & form redirection)

Integrate new logo from ​http://wiki.okfn.org/ckan/logo into site:

• In title

Also can do favicon (separate ticket:48)

Cost: 1h

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

​http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

​http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

Instead of alphabetically as we do currently, alternatively by most viewed datasets

for ​http://thedatahub.org/dataset

Inline with general naming policy going forward.

IndexError? exception in genshi.filters.i18n keeps popping up daily.

e.g.

...
Module ckan.controllers.authorization_group:28 in index
<<              items_per_page=20
               )
               return render('authorization_group/index.html')
       
           def read(self, id):
>>  return render('authorization_group/index.html')
Module ckan.lib.base:74 in render
<<      
           return cached_template(template_name, render_template, cache_key=cache_key, 
                                  cache_type=cache_type, cache_expire=cache_expire)
                                  #, ns_options=('method'), method=method)
>>  cache_type=cache_type, cache_expire=cache_expire)
Module pylons.templating:249 in cached_template
<<          return content
           else:
               return render_func()
>>  return render_func()
Module ckan.lib.base:63 in render_template
<<              stream = item.filter(stream)
               
               return literal(stream.render(method=method, encoding=None))
           
           if 'Pragma' in response.headers:
>>  return literal(stream.render(method=method, encoding=None))
Module genshi.core:183 in render
<<              method = self.serializer or 'xml'
               generator = self.serialize(method=method, **kwargs)
               return encode(generator, method=method, encoding=encoding, out=out)
       
           def select(self, path, namespaces=None, variables=None):
>>  return encode(generator, method=method, encoding=encoding, out=out)
Module genshi.output:57 in encode
<<          _encode = lambda string: string
           if out is None:
               return _encode(''.join(list(iterator)))
           for chunk in iterator:
               out.write(_encode(chunk))
>>  return _encode(''.join(list(iterator)))
Module genshi.output:339 in __call__
<<          for filter_ in self.filters:
                   stream = filter_(stream)
               for kind, data, pos in stream:
                   cached = cache_get((kind, data))
                   if cached is not None:
>>  for kind, data, pos in stream:
Module genshi.output:670 in __call__
<<          _gen_prefix = _gen_prefix().next
       
               for kind, data, pos in stream:
                   output = cache_get((kind, data))
                   if output is not None:
>>  for kind, data, pos in stream:
Module genshi.output:771 in __call__
<<          push_text = textbuf.append
               pop_text = textbuf.pop
               for kind, data, pos in chain(stream, [(None, None, None)]):
       
                   if kind is TEXT:
>>  for kind, data, pos in chain(stream, [(None, None, None)]):
Module genshi.output:586 in __call__
<<      def __call__(self, stream):
               prev = (None, None, None)
               for ev in stream:
                   if prev[0] is START:
                       if ev[0] is END:
>>  for ev in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module ckanext.googleanalytics.plugin:62 in download_adder
<<              (downloaded %s times)</span>'''
                   count = None
                   for mark, (kind, data, pos) in stream:
                       if mark and kind == START:
                           href = data[1].get('href')
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1175 in __call__
<<          """
               callable_value = hasattr(self.value, '__call__')
               for mark, (kind, data, pos) in stream:
                   if mark is ENTER:
                       if callable_value:
>>  for mark, (kind, data, pos) in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.core:288 in _ensure
<<      # unchanged
           yield event
           for event in stream:
               yield event
>>  for event in stream:
Module genshi.filters.transform:686 in _unmark
<<      def _unmark(self, stream):
               for mark, event in stream:
                   kind = event[0]
                   if not (kind is None or kind is ATTR or kind is BREAK):
>>  for mark, event in stream:
Module genshi.filters.transform:1145 in __call__
<<          :param stream: The marked event stream to filter
               """
               for mark, event in stream:
                   yield mark, event
                   if mark is ENTER:
>>  for mark, event in stream:
Module genshi.filters.transform:714 in __call__
<<          stream = iter(stream)
               next = stream.next
               for mark, event in stream:
                   if mark is None:
                       yield mark, event
>>  for mark, event in stream:
Module genshi.filters.transform:682 in _mark
<<      def _mark(self, stream):
               for event in stream:
                   yield OUTSIDE, event
>>  for event in stream:
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:618 in _include
<<                      tmpl = self.loader.load(href, relative_to=event[2][0],
                                                   cls=cls or self.__class__)
                           for event in tmpl.generate(ctxt, **vars):
                               yield event
                       except TemplateNotFound:
>>  for event in tmpl.generate(ctxt, **vars):
Module genshi.template.base:605 in _include
<<          from genshi.template.loader import TemplateNotFound
       
               for event in stream:
                   if event[0] is INCLUDE:
                       href, cls, fallback = event[1]
>>  for event in stream:
Module genshi.template.markup:378 in _match
<<                      for event in self._match(self._flatten(template, ctxt,
                                                                  **vars),
                                                    ctxt, start=idx + 1, **vars):
                               yield event
>>  ctxt, start=idx + 1, **vars):
Module genshi.template.markup:327 in _match
<<                      break
       
               for event in stream:
       
                   # We (currently) only care about start and end events for matching
>>  for event in stream:
Module genshi.template.base:545 in _flatten
<<          while 1:
                   for kind, data, pos in stream:
       
                       if kind is START and data[1]:
>>  for kind, data, pos in stream:
Module genshi.filters.i18n:177 in _generate
<<                  msgbuf.append(*previous)
                       previous = None
                   for event in msgbuf.translate(gettext(msgbuf.format())):
                       yield event
                   if previous:
>>  for event in msgbuf.translate(gettext(msgbuf.format())):
Module genshi.filters.i18n:1029 in translate
<<                      )
       
               parts = parse_msg(string)
               parts_counter = {}
               for order, string in parts:
>>  parts = parse_msg(string)
Module genshi.filters.i18n:1143 in parse_msg
<<      if string:
               parts.append((stack[-1], string))
       
           return parts
>>  parts.append((stack[-1], string))
IndexError: list index out of range

In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information.

ENV=datacatalos.org, with CKAN 1.4.2a

In this commit ​https://github.com/okfn/ckan/commit/1772a5c John Glover set map.explicit=True in ckan/config/routing.py.

The reason this was done was to avoid links collecting parameters. e.g. if you were on page /dataset/{id}/resource/{resource_id} then by default all the links on that page generated by url_for (Routes) would include the id and resource_id parameters as well. To avoid this, you had to go through all the links and add id=None and resource_id=None to the url_for parameters.

When map.explicit was changed to True, the value of the controller, action, id and any parameters were no longer automatically carried over into the generated links for the page. So previously links within the same controller didn't need to specify the controller (for example), but now they did. So when we did this we also had to fix up links that weren't explicit:

• ​https://github.com/okfn/ckan/commit/88e88380f61965f58fb105095e76a78673ce3e2c
• #1501 Pagination links
• ​https://github.com/okfn/ckan/commit/52d71102d9b07693ea240d4a66e8adb43e515414

John made the config change on 5/11/2011 which was merged to master ​https://github.com/okfn/ckan/commit/5a01e67 21/11/2011. The related fixes mentioned were in within the same week. This all went into release 1.5.1. The requirement

In the presentation of data packages, the link in "The information on this page and the downloads / resources are also available using the datapkg command line utility." (​http://knowledgeforge.net/ckan/doc/datapkg/install.html) ends up at the home page of ckan.org

Perhaps link it to ​https://bitbucket.org/okfn/datapkg/overview for now, to save some searching for people who want to find out more?

In the footer: Change 'Powered by CityData?' to Powered by DataSuite? (with link to ckan.org/datasuite)

Links in left hand side of footer should change to a link to ckan.org, link to OKFN, link to ckan.org/datasuite/about and link to Pricing Page (on ckan.org .. will be /datasuite/solutions)

In create dataset form, prefilled tag examples should include one example with a space, e.g. "mental health"

In the Web interface and API, an administrator can set the 'state' field of a group to 'deleted' but this change is ignored in the save/submit and it doesn't get deleted.

In setup.py, do we still need:

('templates/.txt', 'genshi', {

'template_class': 'genshi.template:TextTemplate'

}),

in message_extractors? Doesn't like there are any txt files in there anymore.

In password reset, it gets confused if you have two similar users. This is because with the string the user provides, it searches several fields, not just name but also fullname and email address, allowing you to search for these. But only name is unique.

Specific problem: Ira searches for "Irina" then it finds both: <User name=irina fullname=Irina [email protected] ] and <User name=shevski fullname=Ira email=> (I think)

Maybe need to choose which field it searches?

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

In downloading packages we need to download resources and we need a ResourceDownloader? object for this. these should be pluggable so that we can add support for different types of resources.

In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCIITRANSLIT).

In Firefox:

• Edit | Preferences | Content | Languages - put 'de' at top language
• Clear cookies
• Load thedatahub.org
• it should be in German, but it's in the default language - English.

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

Import metadata from both sources into PDEU via the Harvesting framework but by scraping their respective catalogue pages.

• ​http://data.wien.gv.at/katalog/
• ​http://opendata.paris.fr/opendata/jsp/site/Portal.jsp

Implementation:

​http://s031.okserver.org:2375/dataset/new_metadata

Implement the necessary backend changes to build profile for publishers using the group refactor.

See ​http://ckan.okfnpad.org/publisherprofiles

Subtickets are at:

• Publisher profile analysis #1670 [2d]
• Publisher profile sysadmin authz #1671 [1d]
• Publisher profile user authz #1672 [1d]
• Test auth logic hooks for publisher profile #1674 [3d]
• Publisher admin authz #1675 [3d]
• Publisher editor authz #1676 [2d]
• Add approval status field to groups #1673 [1d]

Implement proxy caching for performance gains

• Analysis of best solution and what we can deliver in a reasonable time [2d]

• Checks for presence of cookie 'auth_tkt' and appropriate setup for media [2d]

• Remove session requirements for user who isn't logged in [3d]

• Document the configuration and setup so that it can be easily modified and replicated [1d]

If you've got edit permission on an authzgroup, then you can change its name to be the same as another existing authzgroup.

This causes some strange UI effects at worst, and probably causes worse problems somewhere else.

Is there any reason why changing the names of existing authzgroups should be allowed? And if so, name collisions should presumably be guarded against in both the name-changing and creation functions

If you use tag autocompletion, it automatically adds a comma after the tag. This is fine if you are going to add another tag, but not fine if it is the last one. If you don't manually remove the final comma than you get form validation error "Tag string: Tag "" length is less than minimum 2"

Affects only version 1.5.2a.

If you switch to Portugese and try to add a resource on dataset edit it will fail. This is because there is a string translation in js_strings.html that has quotes in it. When this is inserted into the file this causes a js exception which then prevents any further js processing.

Specifically:

CKAN.Strings.youHaveUnsavedChanges = "Você tem alterações não salvas. Certifique-se de ter clicado "Salvar Alterações" abaixo antes de sair desta página.";
Uncaught SyntaxError: Unexpected identifier

To fix is simple i imagine: need to escape " correctly in js_strings.html (or translations used).

I have temporarily patched this live in order to allow the hackday here to continue.

If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error.

e.g. ​http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk

returns this exception:

URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:50 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:52 in search
<<                  collection=query,
                       page=request.params.get('page', 1),
                       items_per_page=50
                   )
                   # filter out ranks from the query result
>>  items_per_page=50
Module webhelpers.paginate:333 in __init__
<<              self.item_count = item_count
               else:
                   self.item_count = len(self.collection)
       
               # Compute the number of the first and last available page
>>  self.item_count = len(self.collection)
Module webhelpers.paginate:204 in __len__
<<      def __len__(self):
               return self.obj.count()
       
       # Since the items on a page are mainly a list we subclass the "list" type
>>  return self.obj.count()
Module sqlalchemy.orm.query:1094 in count
<<              q = q.params(params)
               q = q._legacy_select_kwargs(**kwargs)
               return q._count()
       
           def _count(self):
>>  return q._count()
Module sqlalchemy.orm.query:1103 in _count
<<          """
               return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
       
           def _col_aggregate(self, col, func, nested_cols=None):
>>  return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
Module sqlalchemy.orm.query:1125 in _col_aggregate
<<          if self._autoflush and not self._populate_existing:
                   self.session._autoflush()
               return self.session.scalar(s, params=self._params, mapper=self.mapper)
       
           def compile(self):
>>  return self.session.scalar(s, params=self._params, mapper=self.mapper)
Module sqlalchemy.orm.session:635 in scalar
<<          engine = self.get_bind(mapper, clause=clause, instance=instance)
       
               return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
       
           def close(self):
>>  return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
Module sqlalchemy.engine.base:834 in scalar
<<          """
       
               return self.execute(object, *multiparams, **params).scalar()
       
           def statement_compiler(self, statement, **kwargs):
>>  return self.execute(object, *multiparams, **params).scalar()
Module sqlalchemy.engine.base:844 in execute
<<          for c in type(object).__mro__:
                   if c in Connection.executors:
                       return Connection.executors[c](self, object, multiparams, params)
               else:
                   raise exceptions.InvalidRequestError("Unexecutable object type: " + str(type(object)))
>>  return Connection.executors[c](self, object, multiparams, params)
Module sqlalchemy.engine.base:895 in execute_clauseelement
<<          else:
                   keys = None
               return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
       
           def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None):
>>  return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
Module sqlalchemy.engine.base:907 in _execute_compiled
<<          context.pre_execution()
               self.__execute_raw(context)
               context.post_execution()
               self._autocommit(context)
>>  self.__execute_raw(context)
Module sqlalchemy.engine.base:916 in __execute_raw
<<              self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context)
               else:
                   self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
       
           def _execute_ddl(self, ddl, params, multiparams):
>>  self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
Module sqlalchemy.engine.base:958 in _cursor_execute
<<              self.engine.logger.info(repr(parameters))
               try:
                   self.dialect.do_execute(cursor, statement, parameters, context=context)
               except Exception, e:
                   self._handle_dbapi_exception(e, statement, parameters, cursor)
>>  self.dialect.do_execute(cursor, statement, parameters, context=context)
Module sqlalchemy.engine.default:133 in do_execute
<<      def do_execute(self, cursor, statement, parameters, context=None):
               cursor.execute(statement, parameters)
       
           def is_disconnect(self, e):
>>  cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128)