{22} Trac tickets (2647 matches)

doc/api.rst needs to cover the new Revision REST interface.

This is caused by editing a package with resources over the REST interface.

Exception:

URL: http://de.ckan.net/api/rest/package/destatis
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:51 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:145 in update
<<                  fs = ckan.forms.group_fs_combined
                   fs = fs.bind(entity, data=request_fa_dict)
                   validation = fs.validate_on_edit(entity.name, entity.id)
                   if not validation:
                       response.status_int = 409
>>  validation = fs.validate_on_edit(entity.name, entity.id)
Module ckan.forms.package:385 in validate_on_edit
<<              temp_name = orig_pkg_name
                   self.data['Package-%s-name' % record_id] = u'something_unique'
               validation = self.validate()
               if temp_name:
                   # restore it
>>  validation = self.validate()
Module formalchemy.forms:175 in validate
<<          if self.readonly:
                   raise Exception('Cannot validate a read-only FieldSet')
               return AbstractFieldSet.validate(self)
       
           def sync(self):
>>  return AbstractFieldSet.validate(self)
Module formalchemy.forms:104 in validate
<<          success = True
               for field in self.render_fields.itervalues():
                   success = field._validate() and success
               # run this _after_ the field validators, since each field validator
               # resets its error list. we want to allow the global validator to add
>>  success = field._validate() and success
Module formalchemy.fields:763 in _validate
<<          try:
                   value = self._deserialize()
               except validators.ValidationError, e:
                   self.errors.append(e)
>>  value = self._deserialize()
Module formalchemy.fields:1354 in _deserialize
<<          if self.is_composite_foreign_key:
                   return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize()))
               return self.renderer.deserialize()
>>  return self.renderer.deserialize()
Module formalchemy.fields:183 in deserialize
<<          if self.field.is_collection:
                   return [self._deserialize(subdata) for subdata in self._serialized_value()]
               return self._deserialize(self._serialized_value())
       
           def _deserialize(self, data):
>>  return self._deserialize(self._serialized_value())
Module formalchemy.fields:167 in _serialized_value
<<          if self.field.is_collection:
                   return self._params.getall(self.name)
               return self._params.getone(self.name)
       
           def deserialize(self):
>>  return self._params.getone(self.name)
Module formalchemy.base:63 in getone
<<          if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage):
                   return v
               return v[0]
           def getall(self, key):
               v = dict.get(self, key)
>>  return v[0]
TypeError: 'int' object is unsubscriptable

Thanks to Friedrich for spotting this.

Failing test: ckan.tests.models.test_resource.TestResourceLifecycle?.test_03_reorder_resources

Not clear how visible this is to the user.

Related to ticket:292

Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a PackageResource?. This doesn't include changes to tags, extras and resources. These need to be added.

There is the complication that the 'diff' function doesn't currently work beyond PackageResource?. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now.

Rufus suggests alongside the existing methods:

pkg.all_revisions()
pkg.diff()

we have ones which also include the related objects (tags, extras, resources):

pkg.all_revisions_full()
pkg.diff_full()

You could have a function which returned these related objects. Ideas:

revobjlist = [ self, resources, extras ]
list of tuples: (revobj, changed_revobjs)
list of tuples: (revobj, {objtype: [changed_revobjs]})
tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...}

Expect to page through results.

Only page 1 is shown, all other pages fail to display.

Reproduce by searching for something common and browsing to the second page.

Deleted tags are visible on the /tag page and are searchable.

Delete tags are ones that were in use once and then discarded, so this includes ones which are not used on any packages any more, including misspelling, mistakes, and deprecated tags. It would makes sense to remove them.

Example tag that is viewed at /tag and searchable: ​http://ckan.net/tag/2rand[0

Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have:

[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "package_resource_revision_pkey", referer: http://ckan.net/package/edit/cofog
[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30]  'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog

Bug report regarding getting revisions:

Getting revisons by ID (on the latest ID) GET "​http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"

Returns a whole load of stuff (it's also quite slow - about 10 seconds)

I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API?

GET "​http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45" Returns the empty string - I'd expect an empty array ie []

GET "​http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45" Returns an internal server error 500 - I think it should probably be "bad Request" 400 (the date is invalid)

If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error.

e.g. ​http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk

returns this exception:

URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:50 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:52 in search
<<                  collection=query,
                       page=request.params.get('page', 1),
                       items_per_page=50
                   )
                   # filter out ranks from the query result
>>  items_per_page=50
Module webhelpers.paginate:333 in __init__
<<              self.item_count = item_count
               else:
                   self.item_count = len(self.collection)
       
               # Compute the number of the first and last available page
>>  self.item_count = len(self.collection)
Module webhelpers.paginate:204 in __len__
<<      def __len__(self):
               return self.obj.count()
       
       # Since the items on a page are mainly a list we subclass the "list" type
>>  return self.obj.count()
Module sqlalchemy.orm.query:1094 in count
<<              q = q.params(params)
               q = q._legacy_select_kwargs(**kwargs)
               return q._count()
       
           def _count(self):
>>  return q._count()
Module sqlalchemy.orm.query:1103 in _count
<<          """
               return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
       
           def _col_aggregate(self, col, func, nested_cols=None):
>>  return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
Module sqlalchemy.orm.query:1125 in _col_aggregate
<<          if self._autoflush and not self._populate_existing:
                   self.session._autoflush()
               return self.session.scalar(s, params=self._params, mapper=self.mapper)
       
           def compile(self):
>>  return self.session.scalar(s, params=self._params, mapper=self.mapper)
Module sqlalchemy.orm.session:635 in scalar
<<          engine = self.get_bind(mapper, clause=clause, instance=instance)
       
               return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
       
           def close(self):
>>  return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
Module sqlalchemy.engine.base:834 in scalar
<<          """
       
               return self.execute(object, *multiparams, **params).scalar()
       
           def statement_compiler(self, statement, **kwargs):
>>  return self.execute(object, *multiparams, **params).scalar()
Module sqlalchemy.engine.base:844 in execute
<<          for c in type(object).__mro__:
                   if c in Connection.executors:
                       return Connection.executors[c](self, object, multiparams, params)
               else:
                   raise exceptions.InvalidRequestError("Unexecutable object type: " + str(type(object)))
>>  return Connection.executors[c](self, object, multiparams, params)
Module sqlalchemy.engine.base:895 in execute_clauseelement
<<          else:
                   keys = None
               return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
       
           def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None):
>>  return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
Module sqlalchemy.engine.base:907 in _execute_compiled
<<          context.pre_execution()
               self.__execute_raw(context)
               context.post_execution()
               self._autocommit(context)
>>  self.__execute_raw(context)
Module sqlalchemy.engine.base:916 in __execute_raw
<<              self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context)
               else:
                   self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
       
           def _execute_ddl(self, ddl, params, multiparams):
>>  self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
Module sqlalchemy.engine.base:958 in _cursor_execute
<<              self.engine.logger.info(repr(parameters))
               try:
                   self.dialect.do_execute(cursor, statement, parameters, context=context)
               except Exception, e:
                   self._handle_dbapi_exception(e, statement, parameters, cursor)
>>  self.dialect.do_execute(cursor, statement, parameters, context=context)
Module sqlalchemy.engine.default:133 in do_execute
<<      def do_execute(self, cursor, statement, parameters, context=None):
               cursor.execute(statement, parameters)
       
           def is_disconnect(self, e):
>>  cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128)

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

​http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

• Validation of urls as part of form entry or data loading
  • Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
• This code should support analysis of existing data so we can go through existing database and find invalid urls
  • Also useful to have this so we can do out of band validation

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

• ckan.site_title config variable
• set this on g in app_globals.py e.g.
  • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
• use in head title and in main site title/logo section (use it as alt on logo image)
• Also all other pages (e.g. index, about) which talk about CKAN
  • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

• Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
• Probably yes, but not part of this ticket.

Which events to notify on

Listed by domain object, these are the notification message 'change types' that will be sent:

• Package
• PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:

• db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

Ignored domain objects

These parts of the domain model will not carry notifications as no use case has been identified for them:

• Revision
• Group
• Tag
• Rating
• User - list of users is sensitive info
• Relationships - complicated
• Authz - complicated and sensitive info
• License - change of a license's metadata is a question for the 'license service'

Message format

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload.

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

1. An external search engine needing to (re)index a package. (interest: Package)
2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
4. Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)

Context

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

Design

Split-off into two tickets:

• Notification message - ticket:323
  • Which events to notify on
  • Message format
• Interface for Notifier Service - ticket:322

Testing

To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

Related

• Run CKAN search indexing using this system - ticket:324
• Run SOLR indexing using this system - ticket:353
• Web hooks for notifications - ticket:327

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

• Python: ckanclient
• Perl: luke closs wrote something
• PHP: drupal library?
• ...?

The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error:

DateConvertError?: Could not read date as ISO format "%a, %d %b %Y %H:%M:%S %Z". Date provided: "Mon, 04 Jan 2010 09:30:00 BST"

It turns out that it only recognises local names of timezones.

The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems.

This is a problem with the package page e.g. ​http://ckan.net/package/open-election-data-project . The feed icon links to [1] and the text next to it "Subscribe" links to [2]. Surely these should be the same? The second link seems to right one to me.

• [1] ​http://ckan.net/package/open-election-data-project?format=atom
• [2] ​http://ckan.net/package/history/open-election-data-project?format=atom&days=7

As a

CKAN client such as ScraperWiki

I want to

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

Proposed implementation

Add resource search API at:

/api/search/resource

AND resource added to model API at:

api/rest/resource

(see ticket:358)

Functional differences from the ScraperWiki suggested patch:

• URL is not normalised

• URLs are not grouped

• All fields of the resource object are returned, not just the URL

• Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend '​http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

Example search

POST

{"url": "scraperwiki.com/", "all_fields": 1}

to: /api/2/search/resource

returns JSON:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe

Search Parameters

Key:  q
Description: Search all resource fields for the value

Key: url / description / format / 
Description: Search particular field for the value

Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

Original request

Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?

Donovan

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID.

As a

SOLR instance

I want to

keep my search index of CKAN packages up-to-date

Implementation

• Using asynchronous event notifications
• Running in a separate process to CKAN

A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

• It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration).

Storage represents something you can upload to (download is done in ticket:406).

• storage sub package with defined interface (maybe just use OFS)
• Only support uploading single files for the moment (though can do multiple files).
  • Relationship to build (ticket:306)
  • No support for syncing e.g. hg/git repos -- that's up to you as a user
• Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication.

If you change a resource then you not only get a PackageNotification?, but also a ResourceNofication?.

Runs on the commandline and prints out notifications.

Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU.

Requested by DGU.

We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app)

Allow using different indexes, including ckan, file, dcat, ...

Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download.

• Also rename InstallCommand? -> DownloadCommand?

And we should support interactive retrieving of resources ...

Implementation Ideas

for resources in pkg.resources:
    # datapkg/hg 
    load_storage_handler(resource.format)

• (Load cli commands from setuptools entry point name datapkg.cli)
• Document how other can write new commands
• Refactor base Command class to support this pluggability (if necessary)

• Need to figure out cache invalidation for CKAN
• Evaluate against an internal cache (i.e. memcached)

Ask Paul for a new machine for testing. Then one for varnish-live and one for varnish-test.

Requirements:

• cater naturally for multiple packages with multiple resources
• international encoding
• large number of packages

End users want to get hold of a JSON & CSV dump of the DGU data.