(Related to ticket:169 - Package derivations)

A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.)

e.g. london-traffic-visualisation depends on road-map

'Dependency' relationship is:

• directional
• many:many
• stateful

'dependency' table columns:

• id (primary key)
• dependent (foreign key)
• dependency (foreign key)

Further tickets:

• WUI - package view - have list of dependencies (do not need to list packages which depend on this one)
• WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package')
• REST api - expose reading and writing 'depends on' property.

Issues

• How do we deal with dependency at a particular version?

With up to date statistics such as:

• total packages
• total tags
• total users
• most active users
• last update
• etc.

Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184

Create /stats/ page displaying main statistics, e.g:

• Most highly rated packages
• Most edited packages
• Largest groups
• Top tags (by packages)
• Package addition rate
• Users with most packages

Related to ticket:181 - Stats side-bar

Details

• Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display)
• For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html

(Follows on from ticket:107). We want to provide facility for users to automatically upload material.

• Create an upload page
• Store the material on archive.org using s3 - ​http://www.archive.org/help/abouts3.txt
• Main issue is how we cache large datasets ...

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

See how we did it already for other things.

Note: on ckan.net older PackageRevision?.id might not be identical to Package.id but this may need sorting at this point.

Current write method just creates a bare bones package and does not write any of the available metadata other than name.

p:wui 2009-09-09 validate email address when creating an enquiry

Native login scenario: Complete login form, require email address. Any session is resumed.

Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff?.

See CKAN or Microfacts for usage of repose with openid.

Package history page package/history/mypackagename

• "Revisions" is redundant
• "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
• Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.

This needs to be thought out ...

Broken out from ticket:216

​http://ckan.net/group/publicdomain has:

"There is 16 package in this group. There are 16 packages in this group."

Problem here is that foreign key then becomes "complicated" (composite).

• Could also deprecate continuity_id field in favour of the basic foreign key on ie

Search of package name and title (and other fields) using regular expressions.

Current example use-case: Wanting to specify packages with title beginning with 'B'.

Issues:

• Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields?

Implementation:

• Postgres reg ex searching detailed here:

​http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE

Need a way to mark packages as duplicate.

Ideas/Questions?:

• Could work off a tag (meta-duplicate or somesuch)
• What happens if we want to delete the package (to we leave a place-holder, do we merge content?)

If the notes field could use a WYSIWYG editor with word cleanup this would really help users who may well be pasting in text from Word (and I guess may have been approved or written by someone else)

having auto-complete on tags would both make the system easier to use - and reduce the risk of synonymous tags being created inadvertently.

Does the department drop down options list interact with user permissions - so that users who only have rights to one department only see that option ?

Similarly for licenses - can this reduce to the set of allowed options for this user/

What level of validation is there on the fields?

I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group.

• This would allow people to embed feed widgets in other places

This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type)

Cost: 0.5h

from ckan.model.types import JsonType

forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer

Use case

As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion.

Suggested solution

In the package view side-bar, there is a note: "To have this package completely removed, contact the [ca.ckan.net administrators admin@…]."

Other solutions

A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill.

c.f. ​http://en.wikipedia.org/wiki/Deleting_an_article

On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.

diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
 
         if not general_terms and \

A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package.

Extracted from ticket:216.

Dubious of its merit.

On ckan.net there are a few packages which cause a 403 exception when you try to rate them: ​http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError?: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating? objects. This race condition needs to be solved and these particular rating objects repaired.

If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error.

e.g. ​http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk

returns this exception:

URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:50 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.package:52 in search
<<                  collection=query,
                       page=request.params.get('page', 1),
                       items_per_page=50
                   )
                   # filter out ranks from the query result
>>  items_per_page=50
Module webhelpers.paginate:333 in __init__
<<              self.item_count = item_count
               else:
                   self.item_count = len(self.collection)
       
               # Compute the number of the first and last available page
>>  self.item_count = len(self.collection)
Module webhelpers.paginate:204 in __len__
<<      def __len__(self):
               return self.obj.count()
       
       # Since the items on a page are mainly a list we subclass the "list" type
>>  return self.obj.count()
Module sqlalchemy.orm.query:1094 in count
<<              q = q.params(params)
               q = q._legacy_select_kwargs(**kwargs)
               return q._count()
       
           def _count(self):
>>  return q._count()
Module sqlalchemy.orm.query:1103 in _count
<<          """
               return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
       
           def _col_aggregate(self, col, func, nested_cols=None):
>>  return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key))
Module sqlalchemy.orm.query:1125 in _col_aggregate
<<          if self._autoflush and not self._populate_existing:
                   self.session._autoflush()
               return self.session.scalar(s, params=self._params, mapper=self.mapper)
       
           def compile(self):
>>  return self.session.scalar(s, params=self._params, mapper=self.mapper)
Module sqlalchemy.orm.session:635 in scalar
<<          engine = self.get_bind(mapper, clause=clause, instance=instance)
       
               return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
       
           def close(self):
>>  return self.__connection(engine, close_with_result=True).scalar(clause, params or {})
Module sqlalchemy.engine.base:834 in scalar
<<          """
       
               return self.execute(object, *multiparams, **params).scalar()
       
           def statement_compiler(self, statement, **kwargs):
>>  return self.execute(object, *multiparams, **params).scalar()
Module sqlalchemy.engine.base:844 in execute
<<          for c in type(object).__mro__:
                   if c in Connection.executors:
                       return Connection.executors[c](self, object, multiparams, params)
               else:
                   raise exceptions.InvalidRequestError("Unexecutable object type: " + str(type(object)))
>>  return Connection.executors[c](self, object, multiparams, params)
Module sqlalchemy.engine.base:895 in execute_clauseelement
<<          else:
                   keys = None
               return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
       
           def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None):
>>  return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
Module sqlalchemy.engine.base:907 in _execute_compiled
<<          context.pre_execution()
               self.__execute_raw(context)
               context.post_execution()
               self._autocommit(context)
>>  self.__execute_raw(context)
Module sqlalchemy.engine.base:916 in __execute_raw
<<              self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context)
               else:
                   self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
       
           def _execute_ddl(self, ddl, params, multiparams):
>>  self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
Module sqlalchemy.engine.base:958 in _cursor_execute
<<              self.engine.logger.info(repr(parameters))
               try:
                   self.dialect.do_execute(cursor, statement, parameters, context=context)
               except Exception, e:
                   self._handle_dbapi_exception(e, statement, parameters, cursor)
>>  self.dialect.do_execute(cursor, statement, parameters, context=context)
Module sqlalchemy.engine.default:133 in do_execute
<<      def do_execute(self, cursor, statement, parameters, context=None):
               cursor.execute(statement, parameters)
       
           def is_disconnect(self, e):
>>  cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128)

When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model.

The Drupal front-end shall have:

1. Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER.

2. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions.

3. Access denied page - fixed location, static resource, gently

indicates what has happened, and how to ask for permission.

The CKAN slave edit page shall:

1. Try to detect a Drupal session key (passed as cookie or as request param).

2. Redirect to Drupal login page if no session key.

3. Check authorisation if session key is found.

4. Redirect to access denied page if session key not authorised.

5. Present the Package edit page.

6. Reject unauthenticated or unauthorised edit submissions.

7. Snag invalid edit submissions from authenticated and authorised users.

8. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end.

The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error:

DateConvertError?: Could not read date as ISO format "%a, %d %b %Y %H:%M:%S %Z". Date provided: "Mon, 04 Jan 2010 09:30:00 BST"

It turns out that it only recognises local names of timezones.

The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems.

This is a problem with the package page e.g. ​http://ckan.net/package/open-election-data-project . The feed icon links to [1] and the text next to it "Subscribe" links to [2]. Surely these should be the same? The second link seems to right one to me.

• [1] ​http://ckan.net/package/open-election-data-project?format=atom
• [2] ​http://ckan.net/package/history/open-election-data-project?format=atom&days=7

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID.

As a

SOLR instance

I want to

keep my search index of CKAN packages up-to-date

Implementation

• Using asynchronous event notifications
• Running in a separate process to CKAN

Sort out how we deal with dashes versus underscores in package names.

Background

In the Web UI, when you rate a package it simply links to something like:

​http://ckan.net/package/rate/mke-liquor-licenses?rating=3

This creates a GET request.

This is bad because:

• Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
• There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
• Best practise for web requests is for GET to be a read-only request.

Runs on the commandline and prints out notifications.

Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU.

We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app)

As a

user browsing a CKAN package which has a resource in a bad format

I want to

be alerted to the possibility of scraping it on scraperwiki

Implementation

This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it
Image: (Scraperwiki dumper truck)
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can ​request this data is scraped or even ​write the scraper yourself.

Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism.

​http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py

This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed.

Using use cases for security etc. Draw diagram to satisfy use cases. Communicate to Paul and therefore TSO.

• Create test data on visible ckan
• Discuss with JF

Follow up with NS and RP.