{22} Trac tickets (2647 matches)

When a package's name is changed, references to the old name (in the WUI and REST) are redirected to the new name (assuming the old name is not being reused by another package).

Also put in redirect from old locations (see examples with plurals in routing.py)

Estimated cost: 20m

Actual cost: 40m

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.

diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
 
         if not general_terms and \

This is caused by editing a package with resources over the REST interface.

Exception:

URL: http://de.ckan.net/api/rest/package/destatis
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:51 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:145 in update
<<                  fs = ckan.forms.group_fs_combined
                   fs = fs.bind(entity, data=request_fa_dict)
                   validation = fs.validate_on_edit(entity.name, entity.id)
                   if not validation:
                       response.status_int = 409
>>  validation = fs.validate_on_edit(entity.name, entity.id)
Module ckan.forms.package:385 in validate_on_edit
<<              temp_name = orig_pkg_name
                   self.data['Package-%s-name' % record_id] = u'something_unique'
               validation = self.validate()
               if temp_name:
                   # restore it
>>  validation = self.validate()
Module formalchemy.forms:175 in validate
<<          if self.readonly:
                   raise Exception('Cannot validate a read-only FieldSet')
               return AbstractFieldSet.validate(self)
       
           def sync(self):
>>  return AbstractFieldSet.validate(self)
Module formalchemy.forms:104 in validate
<<          success = True
               for field in self.render_fields.itervalues():
                   success = field._validate() and success
               # run this _after_ the field validators, since each field validator
               # resets its error list. we want to allow the global validator to add
>>  success = field._validate() and success
Module formalchemy.fields:763 in _validate
<<          try:
                   value = self._deserialize()
               except validators.ValidationError, e:
                   self.errors.append(e)
>>  value = self._deserialize()
Module formalchemy.fields:1354 in _deserialize
<<          if self.is_composite_foreign_key:
                   return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize()))
               return self.renderer.deserialize()
>>  return self.renderer.deserialize()
Module formalchemy.fields:183 in deserialize
<<          if self.field.is_collection:
                   return [self._deserialize(subdata) for subdata in self._serialized_value()]
               return self._deserialize(self._serialized_value())
       
           def _deserialize(self, data):
>>  return self._deserialize(self._serialized_value())
Module formalchemy.fields:167 in _serialized_value
<<          if self.field.is_collection:
                   return self._params.getall(self.name)
               return self._params.getone(self.name)
       
           def deserialize(self):
>>  return self._params.getone(self.name)
Module formalchemy.base:63 in getone
<<          if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage):
                   return v
               return v[0]
           def getall(self, key):
               v = dict.get(self, key)
>>  return v[0]
TypeError: 'int' object is unsubscriptable

Thanks to Friedrich for spotting this.

In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add.

Ticket is split off from ticket:221

See also ticket:155 (duplicate).

Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages.

Motivation: each package's Atom feed is mentioned in the <link rel="alternate" type="application/atom+xml" ...> tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it.

The Repo History (/revision) page has a visual link the feed.

PS It would be nice to display the feed icon in both these places too.

Intended outcome:

See resource url validation error when previewing package with a resources that has no url.

Recurring Error:

URL: http://127.0.0.1:5000/package/new
File 'pyenv-ckan/lib/python2.5/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/repoze/who/middleware.py', line 107 in __call__
  app_iter = app(environ, wrapper.wrap_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Routes-1.10.3-py2.5.egg/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File 'pyenv-ckan/src/ckan/ckan/lib/base.py', line 50 in __call__
  return WSGIController.__call__(self, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 216 in new
  clear_session=True)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 357 in _render_edit_form
  edit_form_html = fs.render()
File 'pyenv-ckan/src/ckan/ckan/forms/builder.py', line 13 in render
  return render(self.form_template)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 533 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_fields.html', line 52 in <Expression u'h.literal(field.render())'>
  ${h.literal(field.render())}
File 'pyenv-ckan/lib/python2.5/site-packages/FormAlchemy-1.3.4-py2.5.egg/formalchemy/fields.py', line 1217 in render
  return self.renderer.render(**opts)
File 'pyenv-ckan/src/ckan/ckan/forms/common.py', line 311 in render
  return render('package/form_resources')
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 510 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_resources.html', line 14 in <Expression u"res['id'] if res else ''">
  <td><input name="${c.id}-${i}-id" type="hidden" value="${res['id'] if res else ''}" /></td>
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 335 in lookup_item
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 405 in undefined
UndefinedError: {'url': u'a', 'hash': u'a', 'description': u'a', 'format': u'a'} has no member named "id"

How to reproduce:

1. Go to package edit form, enter a package resource but without a URL.
2. Click preview.

Need to be able to build a distribution. Need:

• new 'build' command
• specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution.

Extracted from ticket:216.

Dubious of its merit.

Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site).

• Also change label to: Edit summary (Briefly describe the changes you have made)
• Remove: you can markdown formatting here.
• Move author: if you have not signed in smaller and closer (like markdown instructions are nwo).
• Change commit -> save
• Remove "please save" just have the bullet points

On ckan.net there are a few packages which cause a 403 exception when you try to rate them: ​http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError?: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating? objects. This race condition needs to be solved and these particular rating objects repaired.

Bug report regarding getting revisions:

Getting revisons by ID (on the latest ID) GET "​http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"

Returns a whole load of stuff (it's also quite slow - about 10 seconds)

I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API?

GET "​http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45" Returns the empty string - I'd expect an empty array ie []

GET "​http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45" Returns an internal server error 500 - I think it should probably be "bad Request" 400 (the date is invalid)

If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error.

e.g. ​http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk

returns this exception:

URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:50 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:52 in search
<<                  collection=query,
                       page=request.params.get('page', 1),
                       items_per_page=50
                   )
                   # filter out ranks from the query result
>>  items_per_page=50
Module webhelpers.paginate:333 in __init__
<<              self.item_count = item_count
               else:
                   self.item_count = len(self.collection)
       
               # Compute the number of the first and last available page
>>  self.item_count = len(self.collection)
Module webhelpers.paginate:204 in __len__
<<      def __len__(self):
               return self.obj.count()
       
       # Since the items on a page are mainly a list we subclass the "list" type
>>  return self.obj.count()
Module sqlalchemy.orm.query:1094 in count
<<              q = q.params(params)
               q = q._legacy_select_kwargs(**kwargs)
               return q._count()
       
           def _count(self):
>>  return q._count()
Module sqlalchemy.orm.query:1103 in _count
<<          """
               return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
       
           def _col_aggregate(self, col, func, nested_cols=None):
>>  return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
Module sqlalchemy.orm.query:1125 in _col_aggregate
<<          if self._autoflush and not self._populate_existing:
                   self.session._autoflush()
               return self.session.scalar(s, params=self._params, mapper=self.mapper)
       
           def compile(self):
>>  return self.session.scalar(s, params=self._params, mapper=self.mapper)
Module sqlalchemy.orm.session:635 in scalar
<<          engine = self.get_bind(mapper, clause=clause, instance=instance)
       
               return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
       
           def close(self):
>>  return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
Module sqlalchemy.engine.base:834 in scalar
<<          """
       
               return self.execute(object, *multiparams, **params).scalar()
       
           def statement_compiler(self, statement, **kwargs):
>>  return self.execute(object, *multiparams, **params).scalar()
Module sqlalchemy.engine.base:844 in execute
<<          for c in type(object).__mro__:
                   if c in Connection.executors:
                       return Connection.executors[c](self, object, multiparams, params)
               else:
                   raise exceptions.InvalidRequestError("Unexecutable object type: " + str(type(object)))
>>  return Connection.executors[c](self, object, multiparams, params)
Module sqlalchemy.engine.base:895 in execute_clauseelement
<<          else:
                   keys = None
               return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
       
           def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None):
>>  return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
Module sqlalchemy.engine.base:907 in _execute_compiled
<<          context.pre_execution()
               self.__execute_raw(context)
               context.post_execution()
               self._autocommit(context)
>>  self.__execute_raw(context)
Module sqlalchemy.engine.base:916 in __execute_raw
<<              self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context)
               else:
                   self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
       
           def _execute_ddl(self, ddl, params, multiparams):
>>  self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
Module sqlalchemy.engine.base:958 in _cursor_execute
<<              self.engine.logger.info(repr(parameters))
               try:
                   self.dialect.do_execute(cursor, statement, parameters, context=context)
               except Exception, e:
                   self._handle_dbapi_exception(e, statement, parameters, cursor)
>>  self.dialect.do_execute(cursor, statement, parameters, context=context)
Module sqlalchemy.engine.default:133 in do_execute
<<      def do_execute(self, cursor, statement, parameters, context=None):
               cursor.execute(statement, parameters)
       
           def is_disconnect(self, e):
>>  cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128)

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

​http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

• Validation of urls as part of form entry or data loading
  • Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
• This code should support analysis of existing data so we can go through existing database and find invalid urls
  • Also useful to have this so we can do out of band validation

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

• ckan.site_title config variable
• set this on g in app_globals.py e.g.
  • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
• use in head title and in main site title/logo section (use it as alt on logo image)
• Also all other pages (e.g. index, about) which talk about CKAN
  • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

• Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
• Probably yes, but not part of this ticket.

Use cases

• Register for package changes
• Register for all revisions
• Notified of a package change
• Notified of a revision
• Deregistration
• Configuration of port in pylons config

Design

• Default port: 5672 (standard for AMQP)
• Exchange name: 'ckan'
• Exchange type: topic exchange (most flexible)
• Routing keys: (see below)

Routing detail

Routing key format: "OBJ_TYPE" (NB tags should be identified by their name, not ID)

Example routing keys

• 'package' - Package edited/created
• 'resource' - Resource edited/created
• 'revision' - Any change
• 'db.clean'
• 'db.rebuild'

Example queue bindings that clients may use:

• * - no filtering - client receives all notifications
• package - only changes to packages
• revision - all revisions
• db - all database operations

Versioning

Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps?

Documentation

• How to use
• simple example of an external client?

Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord.

Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers.

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

• Python: ckanclient
• Perl: luke closs wrote something
• PHP: drupal library?
• ...?

The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error:

DateConvertError?: Could not read date as ISO format "%a, %d %b %Y %H:%M:%S %Z". Date provided: "Mon, 04 Jan 2010 09:30:00 BST"

It turns out that it only recognises local names of timezones.

The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems.

Use case: new package

1. An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package.

2. User: clicks 'new package'.

3. CKAN presents the package/new form to the user.

4. (After a couple of previews) User: clicks 'commit'.

5. Notification message goes from CKAN to the front-end detailing the new package.

6. The user is redirected back to the front-end web page displaying the list of packages, which contains the new one.

The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6).

In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages?

Use case: CKAN imports packages

1. CKAN administrator runs a script that adds 100 new packages into CKAN.

2. CKAN sends notification message to front-end to report the new packages/revisions.

3. Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages.

4. The front-end queries CKAN for each new package one-by-one.

5. A new user request to the front-end will include the info about the new packages.

The package addition could be achieved in 1 revision, 100 revisions or some compromise:

• If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'.

• If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN.

This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages.

As a

third-party interface to a CKAN instance

I want to

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.

Design

1. The 'return URL' is passed as a parameter to CKAN.
2. CKAN substitutes the package name into the return URL.

Example

Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data

target = _blank

Cost: 30m

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID.

As a

SOLR instance

I want to

keep my search index of CKAN packages up-to-date

Implementation

• Using asynchronous event notifications
• Running in a separate process to CKAN

Sort out how we deal with dashes versus underscores in package names.

Sean Burlington wrote:

I recently came across a documentation issue

​http://knowledgeforge.net/ckan/doc/ckan/api.html

---

The key should be passed in the API request header: Header Example value HTTP_AUTHORIZATION fde34a3c-b716-4c39-8dc4-881ba115c6d4

---

But the header name is actually just 'AUTHORIZATION'

For example to set it in PHP

curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION: xxxxx-xxxx-xxxx-xxxx-xxxxxxx'));

Storage represents something you can upload to (download is done in ticket:406).

• storage sub package with defined interface (maybe just use OFS)
• Only support uploading single files for the moment (though can do multiple files).
  • Relationship to build (ticket:306)
  • No support for syncing e.g. hg/git repos -- that's up to you as a user
• Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication.

Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column.

If you change a resource then you not only get a PackageNotification?, but also a ResourceNofication?.

Runs on the commandline and prints out notifications.

Requested by DGU.

Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU.

Requested by DGU.

When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{'status': 404, 'message': 'package somename not found'}

When errors are encountered when creating a package:

{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

Not authorised:

{'status': 503, 'message': 'You are not authorised to create this package.'}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.

please ignore