To enable you to easily track down what is taking all the time when you make a request.

Map licenses: given ​http://creativecommons.org/licenses/by/2.5/{countrycode}/ as a license value through the API, convert it to ​http://www.opendefinition.org/licenses/cc-by . Should also work if given ​http://www.opendefinition.org/licenses/cc-by . NOTE: the API may send through a list of licenses; in this situation, take the head of the list.

We need to separate the CLI part of CreateTestData? from the bit which creates test fixtures. This is because of the problem with imports and logging. Otherwise we have to do unpythonic things to get logging to work when deriving from CreateTestData? from an extension such as ecportal / dgu. I've only introduced logging into CreateTestData? but I think it is a good thing to have. This change doesn't cause breakages

Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms.

An AJAX call is made every N seconds, and the auto-saved blob is updated.

note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave.

Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

• separate pages for each section of the form
• validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
• no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

• with the javascript approach it would be harder to test the workflow.
• with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure.
• the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
• the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
• the multi-page wizard wouldn't preclude a javascript-tabbing edit-form.

User story #037 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData? etc and will be used by DGU.

Need to investigate alternative payment gateways

These two tests:

ckan.tests.lib.test_helpers.TestHelpers?.test_parse_rfc_2822_gmt_case

ckan.tests.lib.test_helpers.TestHelpers?.test_parse_rfc_2822_with_offset

have been failing in certain timezones. In the UK they pass, if you're in Germany they'll fail.

The function being tested uses datetime.datetime.fromtimestamp() which uses the local time, but the tests compare the results against hardcoded strings, so the tests cannot work in different timezones.

Tom Kralidis kindly told us on IRC [1] that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration.

[1] ​http://irclogs.okfn.org/ckan/2012-04-25.txt 17:16

Make sure the related item count is available on all dataset pages (inc. follow) when it is enabled.

Super is [1649]

We need to provide customisable mappings between CKAN models and RDF. This needs to be easily configurable for different clients and should take into account objects related to the model being mapped.

Current contenders for impl are:

1. ordf.org's Object Description Mapper (which uses OWL)

2. RDFAlchemy, although this looks like it may result in fixed mappings

3. A custom scheme with a simple json file

4. A template based approach using Genshi templates?

Approach is to get a mapping complete of the attributes directly in the package class, and then provide mappings for related objects (tags/resources etc).

We'll then need to be able to publish the RDF to a triplestore.

See ​http://ckan.okfnpad.org/rdfmapping for further analysis.

I won't be needed for this during the current sprint, closing.

e.g. 'Cabinet Office' returns 0 results, whereas 'Cabinet' returns 'Cabinet Office'.

/api/2/util/authorizationgroup/autocomplete?q=Cabinet%20Office

This feature only seems to be used by DGU at the moment (I think)

Page views and Resources clicks need to be tracked.

User Stories

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

• I want to see the traffic breakdown by country to my site ...
• Ditto for browser type, language, etc etc
• I want to see it graphed over time ...

Implementation Details

1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display

• Config option ckan.status.enabled = False (by default)

Storing Data

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

Displaying Data

• Helper functions / dictize:
  • Helper function: h.stats_get(object_type, id, stats_type)
    • h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
  • Change to dictize
• Location in the default theme (do we show for example in search results too!)
• Support for ranking by most popular in search?

Tracking Data

• Our own solution (just write to site_tracking)
• Google analytics (plus extension for retrieving data) <- would need a refactor
• Piwiki

Own Solution

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

• Make javascript to make request to ckan to store clicks and page views.
• Add middleware so these requests do not go through pylons and just store data quickly.

Add a way to share datasets and see mentions of datasets on twitter (and elsewhere)

As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it).

• Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc.
  • Will need a config option for relevant API key e.g. ckan.sharethis.apikey
• Location: Put this at top of sidebar on dataset view (a better suggsetion?)
• Should show how many mentions / shares there are. (very important!)

Discussion

• (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension?
• What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1.
• Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?)

A test case is currently failing for me on master:

ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py", line 342, in test_user_create_unicode
    res = res.follow()
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 603, in follow
    return self.test_app.get(location, **kw)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 208, in get
    return self.do_request(req, status=status)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 389, in do_request
    **req.environ)
  File "/usr/lib/pymodules/python2.6/paste/wsgilib.py", line 343, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/lint.py", line 170, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/usr/lib/pymodules/python2.6/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 69, in __call__
    auth_ids = self.authenticate(environ, classification, ids)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 201, in authenticate
    userid = plugin.authenticate(environ, identity)
  File "/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py", line 29, in authenticate
    if user.validate_password(identity.get('password')):
  File "/home/seanh/pyenv/src/ckan/ckan/model/user.py", line 113, in validate_password
    hashed_pass = sha1(password_8bit + self.password[:40])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)

Add logic functions for getting the activity stream for a dataset in JSON and rendered HTML formats, add HTML activity stream into dataset page.

This overlaps with Sean's follow support (for activity streams) and so will be dealt with there.

Finalised metadata model requires several changes to our existing model:

• new extra: alternative title
• new extra: identifier
• new extra: interop level
• new extra: version description
• new extra: dataset status
• dataset type: change to fixed vocab
• resource types: prefix with documentation/distribution/etc for clarity, add new resource types
• rename update frequency to accrual periodicity
• add new fixed vocab field: language coverage
• resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model.

If you switch to Portugese and try to add a resource on dataset edit it will fail. This is because there is a string translation in js_strings.html that has quotes in it. When this is inserted into the file this causes a js exception which then prevents any further js processing.

Specifically:

CKAN.Strings.youHaveUnsavedChanges = "Você tem alterações não salvas. Certifique-se de ter clicado "Salvar Alterações" abaixo antes de sair desta página.";
Uncaught SyntaxError: Unexpected identifier

To fix is simple i imagine: need to escape " correctly in js_strings.html (or translations used).

I have temporarily patched this live in order to allow the hackday here to continue.

Seanh and icmurray both touched on the html snippets in facets.html. So there's a merge issue. icmurray to merge changes in the multi-lingual branch prior to merging into master.

Allow a user to embed the data viewer in their webpage.

• [x] Embedding widget [1d]

• [x] The embed page [1-2d]

• [x] Documentation and Announce [0.5d]

Depends upon:

• [x] Recline improvements ?d

All the above, plus user stories are fleshed out in [1]

[1] ​http://ckan.okfnpad.org/feature-2283-data-viewer-v3?

Can only add one dataset at a time now (not sure when this broke but was working ~2-3m ago).

Assigning to seanh for review and estimation. I'd rate this as quite annoying but not absolutely critical (hence assigned priority).

Move ECPortal rdf dump paster command into core so that it can be run on thedatahub for virtuoso install.

Groups should be able to specify a URL to an image file for a group logo, the image will be shown on the group page.

Actually uploading an image file (rather than giving a URL) and cropping and resizing the file will be done later, for now just give a URL. (later ticket)

Further customisation of group pages (CSS, templates) also for later. (not sure if we really want this, but in a later ticket if so)

See trace below.

As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert
    converter(key, converted_data, errors, context)
  File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)

• Create Jenkins task for packaging CKAN
• ... and publishing to s3://apt.ckan.org

Make the search results pagination styled by Bootstrap.

Needs putting into a <ul> structure. Current page and '..' need become links that are styled disabled.

Wanted by DGU.

Major UX gotcha reported by several people is that DataStore? is disabled by default and must be enabled for DataStorer? to run and import data.

Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want.

Provide help to users in the WUI, with this linking to a detailed user guide.

• User guide (suggest at help.thedatahub.org (or possibly help.ckan.org)
• Snippets in the WUI with links to full user guide.
  • Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis
• General info about publishing/working with/etc data (where we don't duplicate elsewhere)

Est: (?) 5-10d

Topics

See ​http://ckan.okfnpad.org/datahub-help

• Step-by-step for adding a dataset including tips
  • Reuse: ​http://datapatterns.org/datahub.html
  • Quick overview of relevant parts of domain model (dataset and resource)
• API (perhaps link to main docs)
• Command line interface and dpm
• Using groups

Why do this

We already have wiki.ckan.org and docs.ckan.org. Why do this?

Answer for docs:

• This is administrative guide and focused on general CKAN software.
• Hard to provide completely generic help.
• Integratable with the DataHub? site

Answer for wiki:

• No spam
• General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful.
• Integratable with the DataHub?

Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach.

• probably needs a new converter, as needs to be usable via API as 'published_by'.

Don't provide a field in the form or spreadsheet import for this, just use dataset.metadata_modified.

Why?

• We already use quite a bit of boostrap
• We already plan to use bootstrap JS
• Bootstrap is excellent and pretty light

Some recent changes caused things to break in ckanext-ecportal:

• change from blueprint to bootstrap
• other style / css changes (eg: tables, resource view notes field ID, etc. Forms are also broken, but that will be addressed in #2262)
• change to going through db_to_form_schema on package_show

...but only if it can be done reasonably quickly.