{22} Trac tickets (2647 matches)

Results (301 - 400 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#301 enhancement rgrp assigned Package discussion pages

A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package.

1272301033000000 1340632055000000
#302 enhancement johnbywater dread v1.0 closed fixed Advertise a package's feed visually

Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages.

Motivation: each package's Atom feed is mentioned in the <link rel="alternate" type="application/atom+xml" ...> tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it.

The Repo History (/revision) page has a visual link the feed.

PS It would be nice to display the feed icon in both these places too.

1272370397000000 1272453821000000
#303 defect dread dread v1.0 closed fixed Package history & diff & feed to show changes to tags, extras and resources

Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a PackageResource?. This doesn't include changes to tags, extras and resources. These need to be added.

There is the complication that the 'diff' function doesn't currently work beyond PackageResource?. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now.

Rufus suggests alongside the existing methods:

pkg.all_revisions()
pkg.diff()

we have ones which also include the related objects (tags, extras, resources):

pkg.all_revisions_full()
pkg.diff_full()

You could have a function which returned these related objects. Ideas:

revobjlist = [ self, resources, extras ]
list of tuples: (revobj, changed_revobjs)
list of tuples: (revobj, {objtype: [changed_revobjs]})
tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...}
1272370659000000 1272989728000000
#304 defect johnbywater johnbywater v1.0 closed fixed Previewing package with resource with missing url is broken

Intended outcome:

See resource url validation error when previewing package with a resources that has no url.

Recurring Error:

URL: http://127.0.0.1:5000/package/new
File 'pyenv-ckan/lib/python2.5/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/repoze/who/middleware.py', line 107 in __call__
  app_iter = app(environ, wrapper.wrap_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Routes-1.10.3-py2.5.egg/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File 'pyenv-ckan/src/ckan/ckan/lib/base.py', line 50 in __call__
  return WSGIController.__call__(self, environ, start_response)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 216 in new
  clear_session=True)
File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 357 in _render_edit_form
  edit_form_html = fs.render()
File 'pyenv-ckan/src/ckan/ckan/forms/builder.py', line 13 in render
  return render(self.form_template)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 533 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_fields.html', line 52 in <Expression u'h.literal(field.render())'>
  ${h.literal(field.render())}
File 'pyenv-ckan/lib/python2.5/site-packages/FormAlchemy-1.3.4-py2.5.egg/formalchemy/fields.py', line 1217 in render
  return self.renderer.render(**opts)
File 'pyenv-ckan/src/ckan/ckan/forms/common.py', line 311 in render
  return render('package/form_resources')
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render
  format=format, namespace=kargs, **cache_args)
File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render
  **options)
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 510 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate
File 'pyenv-ckan/src/ckan/ckan/templates/package/form_resources.html', line 14 in <Expression u"res['id'] if res else ''">
  <td><input name="${c.id}-${i}-id" type="hidden" value="${res['id'] if res else ''}" /></td>
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 335 in lookup_item
File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 405 in undefined
UndefinedError: {'url': u'a', 'hash': u'a', 'description': u'a', 'format': u'a'} has no member named "id"

How to reproduce:

  1. Go to package edit form, enter a package resource but without a URL.
  2. Click preview.
1272379550000000 1272447296000000
#305 defect johnbywater johnbywater v1.0 closed fixed Search result pagination is broken

Expect to page through results.

Only page 1 is shown, all other pages fail to display.

Reproduce by searching for something common and browsing to the second page.

1272468229000000 1272994804000000
#306 enhancement rgrp rgrp closed duplicate datapkg build command

Need to be able to build a distribution. Need:

  • new 'build' command
  • specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution.
1272474212000000 1318181194000000
#307 defect rgrp rgrp datapkg-0.7 closed fixed datapkg upload command

Support uploading a distribution somewhere.

  • New cli command: upload
  • Pluggable uploader framework. Suggest initial support

Depends on ticket:306

1272474224000000 1288027815000000
#308 enhancement rgrp rgrp closed duplicate Autocomplete package names & tags in package search

Extracted from ticket:216.

Dubious of its merit.

1273050549000000 1275302577000000
#309 defect pudo dread closed fixed Tag list shows unused tags

Deleted tags are visible on the /tag page and are searchable.

Delete tags are ones that were in use once and then discarded, so this includes ones which are not used on any packages any more, including misspelling, mistakes, and deprecated tags. It would makes sense to remove them.

Example tag that is viewed at /tag and searchable: http://ckan.net/tag/2rand[0

1273247648000000 1280743432000000
#310 defect dread rgrp v1.1 closed fixed Commit message box looks wrong in edit page since edit style overhaul

Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site).

  • Also change label to: Edit summary (Briefly describe the changes you have made)
  • Remove: you can markdown formatting here.
  • Move author: if you have not signed in smaller and closer (like markdown instructions are nwo).
  • Change commit -> save
  • Remove "please save" just have the bullet points
1273348714000000 1279300525000000
#311 defect dread rgrp v1.1 closed fixed Reordering of package resources can lead to integri

Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have:

[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "package_resource_revision_pkey", referer: http://ckan.net/package/edit/cofog
[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30]  'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog
1273348968000000 1274282065000000
#312 defect rgrp dread closed invalid Race condition creating PackageRating

On ckan.net there are a few packages which cause a 403 exception when you try to rate them: http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError?: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating? objects. This race condition needs to be solved and these particular rating objects repaired.

1273482785000000 1311176173000000
#313 enhancement rgrp dread v1.1 closed fixed Allow packages to be specified by IDs in REST interface

If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID.

It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface.

1273606248000000 1275404524000000
#314 defect johnbywater johnbywater closed fixed Bugs getting revisions from the REST API

Bug report regarding getting revisions:

Getting revisons by ID (on the latest ID) GET "http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"

Returns a whole load of stuff (it's also quite slow - about 10 seconds)

I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API?

GET "http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45" Returns the empty string - I'd expect an empty array ie []

GET "http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45" Returns an internal server error 500 - I think it should probably be "bad Request" 400 (the date is invalid)

1273743755000000 1276523983000000
#315 defect dread rgrp v1.1 closed fixed Improvements and fixes to csv dump
  1. Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html
  1. Issues with package resource serialization into csv table.

<quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>

1274100887000000 1275846764000000
#316 defect rgrp dread closed fixed Search URL escaping

If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error.

e.g. http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk

returns this exception:

URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:50 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:52 in search
<<                  collection=query,
                       page=request.params.get('page', 1),
                       items_per_page=50
                   )
                   # filter out ranks from the query result
>>  items_per_page=50
Module webhelpers.paginate:333 in __init__
<<              self.item_count = item_count
               else:
                   self.item_count = len(self.collection)
       
               # Compute the number of the first and last available page
>>  self.item_count = len(self.collection)
Module webhelpers.paginate:204 in __len__
<<      def __len__(self):
               return self.obj.count()
       
       # Since the items on a page are mainly a list we subclass the "list" type
>>  return self.obj.count()
Module sqlalchemy.orm.query:1094 in count
<<              q = q.params(params)
               q = q._legacy_select_kwargs(**kwargs)
               return q._count()
       
           def _count(self):
>>  return q._count()
Module sqlalchemy.orm.query:1103 in _count
<<          """
               return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
       
           def _col_aggregate(self, col, func, nested_cols=None):
>>  return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
Module sqlalchemy.orm.query:1125 in _col_aggregate
<<          if self._autoflush and not self._populate_existing:
                   self.session._autoflush()
               return self.session.scalar(s, params=self._params, mapper=self.mapper)
       
           def compile(self):
>>  return self.session.scalar(s, params=self._params, mapper=self.mapper)
Module sqlalchemy.orm.session:635 in scalar
<<          engine = self.get_bind(mapper, clause=clause, instance=instance)
       
               return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
       
           def close(self):
>>  return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
Module sqlalchemy.engine.base:834 in scalar
<<          """
       
               return self.execute(object, *multiparams, **params).scalar()
       
           def statement_compiler(self, statement, **kwargs):
>>  return self.execute(object, *multiparams, **params).scalar()
Module sqlalchemy.engine.base:844 in execute
<<          for c in type(object).__mro__:
                   if c in Connection.executors:
                       return Connection.executors[c](self, object, multiparams, params)
               else:
                   raise exceptions.InvalidRequestError("Unexecutable object type: " + str(type(object)))
>>  return Connection.executors[c](self, object, multiparams, params)
Module sqlalchemy.engine.base:895 in execute_clauseelement
<<          else:
                   keys = None
               return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
       
           def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None):
>>  return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
Module sqlalchemy.engine.base:907 in _execute_compiled
<<          context.pre_execution()
               self.__execute_raw(context)
               context.post_execution()
               self._autocommit(context)
>>  self.__execute_raw(context)
Module sqlalchemy.engine.base:916 in __execute_raw
<<              self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context)
               else:
                   self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
       
           def _execute_ddl(self, ddl, params, multiparams):
>>  self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
Module sqlalchemy.engine.base:958 in _cursor_execute
<<              self.engine.logger.info(repr(parameters))
               try:
                   self.dialect.do_execute(cursor, statement, parameters, context=context)
               except Exception, e:
                   self._handle_dbapi_exception(e, statement, parameters, cursor)
>>  self.dialect.do_execute(cursor, statement, parameters, context=context)
Module sqlalchemy.engine.default:133 in do_execute
<<      def do_execute(self, cursor, statement, parameters, context=None):
               cursor.execute(statement, parameters)
       
           def is_disconnect(self, e):
>>  cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128)
1274265928000000 1291831177000000
#317 enhancement pudo rgrp v1.1 closed fixed Make search pluggable

Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR).

Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable.

1274295418000000 1279286041000000
#318 defect johnglover wwaites ckan-sprint-2011-10-28 closed fixed Insufficient validation of resource URIs

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

  • Validation of urls as part of form entry or data loading
    • Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
  • This code should support analysis of existing data so we can go through existing database and find invalid urls
    • Also useful to have this so we can do out of band validation
1274302723000000 1349778662000000
#319 defect dread dread closed fixed Local Authority license misnamed in migration script 18

Migration script 18 converts license "OKD Compliant::Local Authority Copyright with data.gov.uk rights" into "uklocalauthority-withrights" when it should be "localauth-withrights", according to the license service SoS 2. This causes clients difficulties looking it up.

1274349714000000 1274366882000000
#320 enhancement dread rgrp v1.1 closed fixed site_title configuration variable which is used in template

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

  • ckan.site_title config variable
  • set this on g in app_globals.py e.g.
    • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
  • use in head title and in main site title/logo section (use it as alt on logo image)
  • Also all other pages (e.g. index, about) which talk about CKAN
    • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

  • Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
  • Probably yes, but not part of this ticket.
1274378967000000 1279130535000000
#321 enhancement thejimmyg johnbywater closed duplicate Delegate authentication to Drupal

When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model.

The Drupal front-end shall have:

  1. Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER.
  1. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions.
  1. Access denied page - fixed location, static resource, gently

indicates what has happened, and how to ask for permission.

The CKAN slave edit page shall:

  1. Try to detect a Drupal session key (passed as cookie or as request param).
  1. Redirect to Drupal login page if no session key.
  1. Check authorisation if session key is found.
  1. Redirect to access denied page if session key not authorised.
  1. Present the Package edit page.
  1. Reject unauthenticated or unauthorised edit submissions.
  1. Snag invalid edit submissions from authenticated and authorised users.
  1. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end.
1274705234000000 1291831399000000
#322 enhancement dread dread v1.1 closed fixed Client interface for Notification Service

Use cases

  • Register for package changes
  • Register for all revisions
  • Notified of a package change
  • Notified of a revision
  • Deregistration
  • Configuration of port in pylons config

Design

  • Default port: 5672 (standard for AMQP)
  • Exchange name: 'ckan'
  • Exchange type: topic exchange (most flexible)
  • Routing keys: (see below)

Routing detail

Routing key format: "OBJ_TYPE" (NB tags should be identified by their name, not ID)

Example routing keys

  • 'package' - Package edited/created
  • 'resource' - Resource edited/created
  • 'revision' - Any change
  • 'db.clean'
  • 'db.rebuild'

Example queue bindings that clients may use:

  • * - no filtering - client receives all notifications
  • package - only changes to packages
  • revision - all revisions
  • db - all database operations

Versioning

Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps?

Documentation

  • How to use
  • simple example of an external client?
1274720042000000 1277722821000000
#323 enhancement dread dread v1.1 closed fixed Notification message

Which events to notify on

Listed by domain object, these are the notification message 'change types' that will be sent:

  • Package
  • PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:

  • db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

Ignored domain objects

These parts of the domain model will not carry notifications as no use case has been identified for them:

  • Revision
  • Group
  • Tag
  • Rating
  • User - list of users is sensitive info
  • Relationships - complicated
  • Authz - complicated and sensitive info
  • License - change of a license's metadata is a question for the 'license service'

Message format

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload.

1274723333000000 1278578841000000
#324 enhancement dread dread v1.1 closed fixed Search indexing using notifications

Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead.

The indexing shall run in a separate shell/process, managed by supervisord.

1274723483000000 1278599927000000
#325 enhancement dread dread v1.1 closed fixed Event push notification

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

  1. An external search engine needing to (re)index a package. (interest: Package)
  2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
  3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
  4. Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)

Context

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

Design

Split-off into two tickets:

  • Notification message - ticket:323
    • Which events to notify on
    • Message format
  • Interface for Notifier Service - ticket:322

Testing

To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

1274723512000000 1278599979000000
#326 task dread dread v1.1 closed fixed Centralise importation of json library

Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers.

1274784223000000 1274789296000000
#327 defect pudo pudo closed wontfix Create a web hook worker for CKAN

This will be useful as an example consumer of the new queue notifications (#325).

  • A user can register any URL to be notified upon an event.
  • Possible extensions: GET with ID only vs. POST with serialized object
  • E-Mail notifications are also hooks, essentially.

We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)

1274807361000000 1296467361000000
#328 enhancement dread rgrp v1.1 closed fixed Mention code libraries in API documentation

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

  • Python: ckanclient
  • Perl: luke closs wrote something
  • PHP: drupal library?
  • ...?
1274988106000000 1275318745000000
#329 defect rgrp dread v1.1 closed fixed Bad dates cause exception on Gov form

Reproduction

Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message.

Affects all versions of CKAN.

Why it's happening

The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught.

1275060617000000 1275079189000000
#330 defect dread dread closed fixed getdata/ons timezone not recognised on non-British servers

The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error:

DateConvertError?: Could not read date as ISO format "%a, %d %b %Y %H:%M:%S %Z". Date provided: "Mon, 04 Jan 2010 09:30:00 BST"

It turns out that it only recognises local names of timezones.

The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems.

1275300271000000 1275303122000000
#331 enhancement rgrp ckan-backlog new Timezone of CKAN timestamps should be configurable

Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either.

Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone.

1275302440000000 1339774701000000
#332 defect pudo rgrp closed fixed Autocomplete tag on click

During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box.

1275302887000000 1280743320000000
#333 enhancement dread v1.1 closed wontfix CKAN front end requirements for package notifications

Use case: new package

  1. An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package.
  1. User: clicks 'new package'.
  1. CKAN presents the package/new form to the user.
  1. (After a couple of previews) User: clicks 'commit'.
  1. Notification message goes from CKAN to the front-end detailing the new package.
  1. The user is redirected back to the front-end web page displaying the list of packages, which contains the new one.

The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6).

In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages?

Use case: CKAN imports packages

  1. CKAN administrator runs a script that adds 100 new packages into CKAN.
  1. CKAN sends notification message to front-end to report the new packages/revisions.
  1. Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages.
  1. The front-end queries CKAN for each new package one-by-one.
  1. A new user request to the front-end will include the info about the new packages.

The package addition could be achieved in 1 revision, 100 revisions or some compromise:

  • If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'.
  • If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN.

This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages.

1275324042000000 1275407987000000
#334 defect pudo dread closed fixed Wrong link for package feed icon

This is a problem with the package page e.g. http://ckan.net/package/open-election-data-project . The feed icon links to [1] and the text next to it "Subscribe" links to [2]. Surely these should be the same? The second link seems to right one to me.

1275407445000000 1280743667000000
#335 enhancement dread dread v1.1 closed fixed Post-package-edit redirect to configurable URL

As a

third-party interface to a CKAN instance

I want to

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.

Design

  1. The 'return URL' is passed as a parameter to CKAN.
  2. CKAN substitutes the package name into the return URL.

Example

Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data

1275408834000000 1276179605000000
#336 defect dread donovanhide v1.1 closed fixed Resource Search API

As a

CKAN client such as ScraperWiki

I want to

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

Proposed implementation

Add resource search API at:

/api/search/resource

AND resource added to model API at:

api/rest/resource

(see ticket:358)

Functional differences from the ScraperWiki suggested patch:

  • URL is not normalised

  • URLs are not grouped
  • All fields of the resource object are returned, not just the URL
  • Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend 'http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

Example search

POST

{"url": "scraperwiki.com/", "all_fields": 1}

to: /api/2/search/resource

returns JSON:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe

Search Parameters

Key:  q
Description: Search all resource fields for the value

Key: url / description / format / 
Description: Search particular field for the value

Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

Original request

Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?

Donovan

1275411765000000 1279373842000000
#337 enhancement johnbywater rgrp v1.1 closed fixed Download links for resources should open in new window

target = _blank

Cost: 30m

1275422151000000 1279300972000000
#338 story johnbywater johnbywater v1.1 closed Reference groups by ID in addition to name, since group names can change 1275901137000000 1280446480000000
#339 defect johnbywater closed fixed Remove download_url from package data format in API Version 2, but maintain the old data format in the old version of the interface (API Version 1) 1275901499000000 1278074705000000
#340 enhancement dread dread v1.1 closed fixed Web UI theme easier to configure

An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.

http://wiki.okfn.org/ckan/doc/theme

1276009729000000 1328807317000000
#341 enhancement dread closed fixed Web UI accepts package IDs in URLs

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

1276162400000000 1277483030000000
#342 enhancement dread dread closed fixed JSONP parameter in API

As a

CKAN client using JQuery

I want to

call the CKAN API and instead of receiving back JSON I get JSONP. i.e. "%s(%s)" % (callback, json_content)

Suggested implementation

All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide:

http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch

Test

import re import unittest

def test_jsonp_callback():

response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com&callback=jsoncallback') match = re.match('jsoncallback\(.*\);',response) self.assertTrue(match)

response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com') match = re.match('jsoncallback\(.*\);',response) self.assertFalse(match)

I think the point needs to be made that JSONP only works for GET requests and not POST/PUT/DELETE, so there needs to be a check for that in the _finish_ok method.

(thanks to Donovan Hide for test)

1276166426000000 1276278485000000
#343 defect johnbywater dread closed fixed Packages referred by ID in API

When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID.

1276180179000000 1278066420000000
#344 defect dread closed fixed REST Create package with incorrect format gives 500 error

PUT to /rest/api/package of {"name": "name", "resources": ["someurl.com/data"] } (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message.

Exception:

Module ckan.controllers.rest:154 in create
<<              if register == 'package' and not subregister:
                       fs = ckan.forms.get_standard_fieldset()
                       request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
                       fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session)
                   elif register == 'package' and subregister in model.PackageRelationship.get_all_types():
>>  request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data)
Module ckan.forms.package_dict:88 in edit_package_dict
<<                      for res_dict in value:
                               res_dict_str = {}
                               for key, value in res_dict.items():
                                   res_dict_str[str(key)] = value
                               resources.append(res_dict_str)
>>  for key, value in res_dict.items():
AttributeError: 'unicode' object has no attribute 'items'
1276341172000000 1277477712000000
#345 requirement dread johnbywater closed invalid metastable, stable & ultrastable branches shall be automatically built and tested 1276523083000000 1291831615000000
#346 defect dread johnbywater ckan-v1.3 closed wontfix Revision search API (response data format and documentation issue)

Whilst going through the API docs for the revision search API, it was noticed that the "Gdu" SoS doc doesn't match up. It returns revision IDs (perhaps this is useful to note in the spec?) so the format is probably not 'limitedstring'. Also, they appear to be ordered youngest first, not oldest as stated.

And in the revision model, it refers to 'simplestring' which it doesn't define - I guess the names should be 'limitedstring'?

Could this be checked out?

1276523517000000 1296477510000000
#347 enhancement johnbywater johnbywater closed fixed The system shall present the package edit form in an API 1276523692000000 1277820440000000
#348 enhancement johnbywater johnbywater closed fixed The system shall accept package edit form submissions in an API 1276523793000000 1277820496000000
#349 defect johnbywater johnbywater ckan-v1.2 closed invalid The system shall support authorised users to edit packages on data.gov.uk 1276523851000000 1282908974000000
#350 enhancement dread ckan-backlog reopened Search engine optimisation

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (http://www.ckan.net/package/ascoe -> http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

1276594541000000 1339774690000000
#351 enhancement dread ckan-backlog new Homepage: list new, updated and 'hot' packages

Have a simpler list of exciting data, as opposed to the big revision list.

For example:

Hot data
===========

New packages: package1, package2, package3
Updated resources: package1, package2, package3
Popular packages: 
1276595816000000 1339774677000000
#352 enhancement dread dread closed wontfix Package notification worker - sends XML-RPC

As an

external front-end

I want to

be notified (by XML-RPC) about package creations and updates.

Implementation

  1. A message queue worker waits for package update notifications
  2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI.

Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter.

Config - in the CKAN config will be:

  • URI to callback to
  • API version to use (version 2 gives packages referred by ID not name)
1276597996000000 1286375870000000
#353 defect dread closed fixed SOLR search indexing

As a

SOLR instance

I want to

keep my search index of CKAN packages up-to-date

Implementation

  • Using asynchronous event notifications
  • Running in a separate process to CKAN
1277123480000000 1280756399000000
#354 defect johnbywater johnbywater closed invalid Collect together requirements and top-level design for user/package 'groups'

Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff.

http://knowledgeforge.net/ckan/trac/wiki/AccessControl

Do we add these into user-role table somehow or new table? To present this to team

1277131335000000 1282908983000000
#355 defect rgrp rgrp closed fixed Dashes versus underscores in package names

Sort out how we deal with dashes versus underscores in package names.

1277221996000000 1311177552000000
#356 enhancement rgrp v1.1 closed fixed Search box in at top of page (UI)

A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

  • It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration).
1277235411000000 1278931830000000
#357 defect johnbywater johnbywater closed fixed Fix API documentation, to correct misleading HTTP_AUTHORIZATION statement.

Sean Burlington wrote:

I recently came across a documentation issue

http://knowledgeforge.net/ckan/doc/ckan/api.html


The key should be passed in the API request header: Header Example value HTTP_AUTHORIZATION fde34a3c-b716-4c39-8dc4-881ba115c6d4


But the header name is actually just 'AUTHORIZATION'

For example to set it in PHP

curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION: xxxxx-xxxx-xxxx-xxxx-xxxxxxx'));

1277459886000000 1277461466000000
#358 enhancement rgrp dread ckan-v1.5 closed duplicate Resources in REST API

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Authorization

  1. Have it generic (ie. not per resource) and use an action/role on system
  2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
  3. Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:

if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)
1277483282000000 1310128782000000
#359 requirement pudo rgrp datapkg-0.7 closed fixed Create Storage with pluggable extensions

Storage represents something you can upload to (download is done in ticket:406).

  • storage sub package with defined interface (maybe just use OFS)
  • Only support uploading single files for the moment (though can do multiple files).
    • Relationship to build (ticket:306)
    • No support for syncing e.g. hg/git repos -- that's up to you as a user
  • Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication.

1277750654000000 1291135692000000
#360 defect rgrp rgrp datapkg-0.7 closed fixed New sqlite-based DB index

In 0.5 and 0.6 have been more oriented to getting file indexes working to support the convenience of using packages on disk directly. However we now need to go back to having a better central/default index and the best model for this is a simple db index using sqlite (which is now supported in the standard library).

NB: this is a retrospective ticket (been working on this for a couple of months)

1277803174000000 1288004891000000
#361 enhancement rgrp rgrp datapkg-0.7 closed fixed Allow specification of the default index in configuration 1277803305000000 1291135756000000
#362 defect dread dread closed invalid Ratings should not be created with a GET

Background

In the Web UI, when you rate a package it simply links to something like:

http://ckan.net/package/rate/mke-liquor-licenses?rating=3

This creates a GET request.

This is bad because:

  • Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
  • There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
  • Best practise for web requests is for GET to be a read-only request.
1278925451000000 1311176564000000
#363 defect kindly dread ckan-backlog closed wontfix Blank revisions

Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column.

1278947772000000 1310125872000000
#364 defect dread ckan-v1.3 closed fixed Search for 'statistic' returns nothing

On ckan.net there are plenty of packages (and indeed their tags) with the word 'statistic' in them, but no packages turn up when you search for it:

http://ckan.net/package/search?q=statistic&search=Search+Packages+%C2%BB

(Using Postgres full text search)

1278949620000000 1291637291000000
#365 enhancement dread dread closed fixed ResourceNotifications

If you change a resource then you not only get a PackageNotification?, but also a ResourceNofication?.

1279037411000000 1279300621000000
#366 enhancement pudo dread ckan-v1.4 closed fixed A flash message says why you are redirected to the 'user login' page

You get redirected for a number of different reasons - often confuses me let alone the average user!

Cost: 1h

1279128058000000 1300212171000000
#367 enhancement dread dread closed fixed Notfication monitor

Runs on the commandline and prints out notifications.

1279303310000000 1279303693000000
#368 defect anonymous closed wontfix 500 Server error when creating package

I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response:

"publish: New Zealand Coastline (new-zealand-coastline)" opening connection to nz.ckan.net... opened <- "POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n" <- "{\"name\":\"new-zealand-coastline\",\"title\":\"New Zealand Coastline\"}" -> "HTTP/1.1 500 Internal Server Error\r\n" -> "Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n" -> "Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n" -> "Vary: Accept-Encoding\r\n" -> "Content-Type: text/html; charset=iso-8859-1\r\n" -> "Cache-Control: proxy-revalidate\r\n" -> "Content-Length: 640\r\n" -> "Connection: close\r\n" -> "\r\n" reading 640 bytes... -> "<!DOCTYPE HTML PUBLIC \"-IETFDTD HTML 2.0EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80</address>\n</body></html>\n" read 640 bytes Conn close

The server should not respond with a 500 error if the package was created.

1279622462000000 1291831811000000
#369 enhancement shudson@… ckan-backlog new "Package Listing Key" should appear on Tag results

Currently there's a nice legend titled "Package Listing Key" that appears in right side of "Browse Packages" results. The same key should show on other search results like when searching for a tag.

1279821634000000 1339774666000000
#370 enhancement shudson@… ckan-backlog new Use better email encryption for author_email and maintainer_email

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

1279821819000000 1339774649000000
#371 requirement nils.toedtmann johnbywater ckan-v1.4 closed fixed The system shall monitor QoS against SLA

Requested by DGU.

1279885622000000 1300217820000000
#372 bug johnbywater johnbywater ckan-v1.2 closed Fix system limits on CKAN for DGU

Set limits in /etc/security/limits.conf so that we can always ssh in at least. Requested by DGU.

1279885752000000 1281522535000000
#373 requirement dread johnbywater closed fixed Sort out gov daily script

Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU.

1279885887000000 1286376176000000
#374 task johnbywater johnbywater v1.1 closed fixed Design servicization of CKAN API 1280223118000000 1280398501000000
#375 task johnbywater johnbywater v1.1 closed fixed Integrate Forms API into Drupal

Requested by DGU.

1280223168000000 1280405812000000
#376 task johnbywater johnbywater v1.1 closed fixed Implement servicization of CKAN API

Requested by DGU.

1280223226000000 1281000755000000
#377 enhancement glen@… ckan-future closed wontfix API Should return JSON in all cases

When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{'status': 404, 'message': 'package somename not found'}

When errors are encountered when creating a package:

{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

Not authorised:

{'status': 503, 'message': 'You are not authorised to create this package.'}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.

1280268158000000 1338206349000000
#378 story johnbywater johnbywater v1.1 closed fixed My user story #1 1280325186000000 1280326706000000
#379 task johnbywater johnbywater v1.1 closed fixed My dev task #1 1280325277000000 1280326496000000
#380 task johnbywater johnbywater v1.1 closed fixed My dev task #2 1280325304000000 1280326595000000
#381 defect johnbywater closed invalid Test defect

please ignore

1280345937000000 1294410466000000
#382 story johnbywater johnbywater v1.1 closed Measure quality of service parameters

As a service administrator, I want to measure responsiveness, throughput, and availability of a CKAN service.

1280346974000000 1280854608000000
#383 task johnbywater johnbywater v1.1 closed fixed Create middleware feature to generate QoS throughput log 1280347595000000 1280854578000000
#384 story johnbywater johnbywater v1.1 closed Send alert when QoS measurements break expectation 1280347841000000 1280496812000000
#385 task johnbywater johnbywater v1.1 closed fixed Send email if QoS measurement breaks QoS criteria 1280355527000000 1280496127000000
#386 task johnbywater johnbywater v1.1 closed fixed Set limits in /etc/security/limits.conf 1280356594000000 1281088994000000
#387 task johnbywater johnbywater v1.1 closed fixed Fix failing tests in search sub-system trigged by recent changes 1280357804000000 1280446047000000
#388 task johnbywater johnbywater v1.1 closed fixed Reply to "two projects" question from RS 1280395503000000 1280405288000000
#389 task johnbywater johnbywater v1.1 closed fixed Investigate timezone issue raised by SB 1280395554000000 1280422414000000
#390 task dread johnbywater ckan-v1.2 closed fixed Remove a package for SB 1280418625000000 1282214629000000
#391 task johnbywater johnbywater v1.1 closed fixed Create monitoring feature to generate QoS responsiveness log 1280446714000000 1280854592000000
#392 task johnbywater johnbywater v1.1 closed fixed Create monitoring feature to generate QoS availability log 1280446769000000 1280496140000000
#393 task johnbywater johnbywater v1.1 closed fixed Create aggregation feature to aggregate QoS log 1280446811000000 1280496962000000
#394 task johnbywater johnbywater closed duplicate Fix munin on DGU (?) 1280485351000000 1294407189000000
#395 task pudo ckan-v1.3 closed duplicate Set up profiling to analyze performance issues

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

1280824739000000 1294417538000000
#396 task johnbywater johnbywater closed fixed Integrate Forms API into Drupal 1280854226000000 1282324934000000
#397 task johnbywater johnbywater v1.1 closed fixed Implement QoS monitoring for DGU 1280854935000000 1281000745000000
#398 task johnbywater johnbywater ckan-v1.3 closed fixed Create Public API documentation for DGU (draft) 1281000828000000 1294407372000000
#399 task dread johnbywater ckan-v1.2 closed fixed Import BIS data for SD 1281000890000000 1282294177000000
#400 task johnbywater johnbywater ckan-v1.2 closed fixed Configure DGU Public API with TSO

Have TSO configure data.ov.uk/api/catalogue. Have TSO register catalogue.data.gov.uk.

1281000956000000 1288003690000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Note: See TracReports for help on using and creating reports.