{22} Trac tickets (2647 matches)

Allow OpenID to be disabled as a login option.

Currently (CKAN 1.6) there is a config option openid_enabled which stops the login option being displayed, but:

• In the User edit form you can still add your OpenID
• The repoze.who.openid middleware is still required - can it be avoided?

Allow a user to embed the data viewer in their webpage.

• [x] Embedding widget [1d]

• [x] The embed page [1-2d]

• [x] Documentation and Announce [0.5d]

Depends upon:

• [x] Recline improvements ?d

All the above, plus user stories are fleshed out in [1]

[1] ​http://ckan.okfnpad.org/feature-2283-data-viewer-v3?

Allow choice of language on ckan instance.

• Configuration variable listing language choices (could try doing this automatically but recommend against this)
• Switcher option in UI (perhaps on user profile page)
• Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]

2. The user is provided the link to the file as currently.

3. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6

• It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.

4. The archiver will continue to move the file into permanent storage as per current practice.

Tickets

• #1608 Merge storage into Core [5d]
• #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
• #1687: Deploy to datahub [1d]
• #1681: Core storage documentation [1d?]
  • Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
  • External: here's how to configure it and here's what you get ... (goes in main sphinx docs)

Allow using different indexes, including ckan, file, dcat, ...

Allowing people to use the full range of markdown results in extremely messy output across the site. I'd suggest limiting support to only a subset of common use cases.

• Allow all inline elements, this allows bold, italic, code and links.
• Allow lists.

Disallow

• Horizontal Rules
• Headings
• Block quote and code (this may turn out to be useful and so could be included)

This way you get markdowns paragraph handling and a few inline styles without breaking the entire layout of the page.

Here's the full syntax if anyone is interested ​http://daringfireball.net/projects/markdown/syntax

Already have the basic capacity using *_extras paths for templates and public directory (see ​http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: ​http://packages.python.org/Flask-Themes/

Also configure hmgckan, hmgapi, test-hmg and dev-hmg machines.

Also put in redirect from old locations (see examples with plurals in routing.py)

Estimated cost: 20m

Actual cost: 40m

Among other things this will allow the tests to run much quicker.

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

• Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
• At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
  • If I am logged in
    • See a form for "Add to do" at the top of the todo section and can add one straight away
    • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

• Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.

Implementation

The Todo form

• One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
• Add a description
• Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

• GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1

• support limit (?)

/api/2/todo/category -> return list of todo categories

• No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

• Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...

An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.

​http://wiki.okfn.org/ckan/doc/theme

Analysis using pg_top reveals orphaed processes:

​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/460#comment:6

This followed switching ckan instances on the dgu machine. Do these hang around forever clogging up a db server? Can switching between instances be smoothed to avoid this?

And associated page to browse group.

Apparently auto-complete in Organizations may not work. Check and fix.

See ​https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5

Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised?

Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit & offset).

Archiver is included in some places just for access to download(), we should move this to lib in core.

As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). Decision: use follow

Implementation

Interface

Become a follower:

• Follow button on packages (if already watching say 'unfollow')

Package-related changes:

• Show number of followers on a package
• List followers of a package at /package/{name}/followers
  • On a separate page

a package User-related changes:

• List followed packages
  • Either on user's page on a separate 'following' page. (NB: called 'following')
• Does watching involve notifications (by email)
  • Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
• [Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page

Nitty-Gritty

• Want to do this in ajax-y manner
• API endpoint: /api/2/follower
• Store data in a new follower table

API

/api/2/follower

follow => PUT / POST
{
   user_id
   object_type
   object_id
}

If this is submitted by a user with user.id != user_id => error (401)

unfollow => DELETE

/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

Table

Called 'follower'

user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]

Random Extras

• What about following users as well

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

• Use the existing support for this feature in google analytics or piwiki
• For google analytics see:
  • Integrate to record: ​http://www.google.com/support/analytics/bin/answer.py?answer=55529
  • Accessing analytics data via API (for showing download counts): ​http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
• Limitations only record downloads by browsers with js turned on.
  • This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

Old Spec (do it in CKAN)

• Record info of form: resource id (or url?), timestamp
• Do this via javascript capturing of onclick event talking to an api
• API: /api/resource/{id}/download
  • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

• Do we record ip addresses (to handle de-botting etc)?
• Do we count preview clicks as well?

As a service administrator, I want to measure responsiveness, throughput, and availability of a CKAN service.

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

• ckan.site_title config variable
• set this on g in app_globals.py e.g.
  • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
• use in head title and in main site title/logo section (use it as alt on logo image)
• Also all other pages (e.g. index, about) which talk about CKAN
  • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

• Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
• Probably yes, but not part of this ticket.

As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually.

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

1. Need a 'data proxy' - see ticket:698 and ticket:888
2. Preview using javascript in WUI - do this in an extension ​http://bitbucket.org/okfn/ckanext-dataapi

As a user I want to view a package at a given revision:

• When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
• package history page should provide links to these pages

Cost: 2h

As a user i want to see a package at "version" X (NB: not revision X).

• When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
• Implementation:
  • Find revision for this version (search revision history for when version field was last 0.7)
  • Show pacakge at that revision (as in ticket:103)
• On history page also shows versions in list of revisions associated with the package

As designed by Sam Smith -> ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/ Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same.

As part of #1574 we decided that it would be better for webstore (and future external services) to be able to authenticate simply with CKAN-Core.

Currently webstore access the CKAN database to obtain the key for the user but it would be better if this connection was not so tightly bound and that webstore used an API as any other external service might.

Need to discuss further with dread

As part of merging Organisations into core, it is necessary that we clarify the capacity field with which the users/datasets are added as members to the group 'subclass'.

Rather than the capacity being an opaque string that implies auth but doesn't clearly specify it, we will use role names where roles are defined in the database - with a clearly defined set of standard roles. The Role table is expected to have simply a string name/representation and acts as a container for permissions.

Each permission is a string of the form object.action (such as package.add, group.delete) of which several are expected to be associated with a role. This means the permission table will contain a string and a reference to the role.

This work will require UI changes to the screens allowing users to be added to a group/organisation so that the list of available roles is available to add those users.

[x] Model for Role and Permission

[ ] Logic layer changes for managing roles/permissions etc.

[ ] Determine default roles, perhaps just admin/editor/viewer

[ ] Fix the auth layer to use the permissions/roles - may be better implemented as another ticket.

As per #2650 (on master/1.8 now)

If the resource has a description it isn't rendered as markdown ..

Ask Paul for a new machine for testing. Then one for varnish-live and one for varnish-test.

Assumes 8 is done:

8. Tool for validation an IATI XML record (referenced by a URL) for compliance with the IATI specification

• Outside of scope of registry
• Multiple validators with different levels of rigour

At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

• Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
• Disable the Download button unless the resource is stored locally.

At the paster db init command when installing ckan from source I get the error "Import Error: cannot import name UnicodeMultiDict?" (happens with both ckan 1.4.2 and today's latest bleeding edge code, on Ubuntu 10.04.3).

UnicodeMultiDict? has been removed in a recent version of python-webob, and the pip install ... lucid_missing.txt causes a too-new version of python-webob to be installed into ckan's virtualenv (the new webob gets installed as a dependency of formalchemy).

I manually did pip uninstall webob and then ran paster db init again and it worked.

At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)

At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio.

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)

At the moment the user has to figure out to login with providers -- this is not always obvious (e.g. google).

We could improve this with a small bit of javascript. As an example see: ​http://standalone.demo.civicrm.org/standalone/

Not sure exactly what library that is using (or if bespoke) so alternatives include (NB: we already use jquery):

• ​http://code.google.com/p/openid-realselector/
• ​http://code.google.com/p/openid-selector/

In addition to implementing this we also need to test logging in with main providers: google, wordpress -- as we have had reports of it not working with e.g. wordpress (not sure if this testing can be automated really -- best hope would be selenium I think -- so OK if done by hand).

Cost: 3h (2h UI), (1h testing)

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

• #191 Searching by modification date
• #193 Searching by time-related field
• #301 Package discussion pages
• #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
• #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
• Other improvements as per my word doc.

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:

• Default this in solr (no need to touch code) but fragile and affects everything ...
• Do it in code and default to score
• Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls.

This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance.

NB: We should also agree on a maximum request latency.

TODO: Read up on all those QoS tickets to avoid overlapping efforts.

At: ​http://ckan.net/revision you can see: "Next »" on pager This doesn't seem to be a problem with earlier (1.0.1) version.

Auth Proposal

Use case: We'd like an authenticated and authorized Drupal user to be able to edit/delete packages using the API

To do this a user would need a CKAN API key so we need some way of Drupal telling CKAN who a user is so that they can get their API key.

Proposed Implementation

A user visits the CKAN API key page to get their key. Because CKAN is at catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal cookies.

If there is no DRXtrArgs or DRXtrArgs2 cookie, we know the user isn't signed in so we redirect them to Drupal to sign in.

WISHLIST: It would be really nice if we could pass the URL to redirect back to Drupal so tha Druapl can send them back to the CKAN API key page

Either way, they get back to CKAN and now the cookies exist. When the first HTTP request header is sent, CKAN will read the Drupal session ID and then call a Drupal API, server to server.

TODO: Implement an API on the drupal server which accepts a Drupal session ID as an argument and returns the username and credentials, but only if the request if from the CKAN server (perhaps we specify an API key in the request)

If the session is valid CKAN will set its own auth cookie and show them the page with the API key. CKAN only considers a user signed in if both the CKAN cookie *and* Drupal session ID are present. If at any time they sign out of Drupal the Drupal session disappears so they will be signed out of CKAN too.

Now the user has an API key they can use the standard CKAN command line API tools.

The API key is the only thing the user will need to use the CKAN API. This poses a problem. What if a user is removed from Drupal but still has a CKAN API key?

Two solutions:

1. API keys will be set to only be valid for the length of a Drupal session so that a user will only be able to use the write API for a little time after credentials are revoked

2. CKAN provides an API to allow Drupal to revoke keys

Option 1. seems easier to me. If we choose this we will write a get_api_key command line tool so that Drupal user can write things like this:

::

datapkg list_packages --api_key get_api_key

The get_api_key function will prompt for username and password and then perform the steps necessary to get an API key.

Authorization Groups:

• Rename UserObjectRole? to SubjectObjectRole?
• Create database migration to do the rename in DB
• Make sure all the tests are passing again

(Doing this first seems more stable since we first want to "re-gain" exisiting functionality)

• Create authz.py and model tests for authorization_group-based auth
  • tests: functional: create fixtures, do actual requests

lib-tests: test for membership addition, removal, group-right additions affecting all members, group-right removal, group rights affecting non-members.

# * Adapt model to get these passing

• Integration into existing controllers should be negligible but may want ot have tests
  • do not want that. would mean design is broken
• Need to create a new controller for editing authz groups (will have same permissions as any other object -- Reader, Editor, Administrator)

Awesome to have the captcha implementation and running on thedatahub.org! Here are some further suggestions concerning the config options, before this goes into a release:

• Document "ckan.recaptcha.publickey" and "ckan.recaptcha.privatekey" in doc/configuration.rst
• Which one config option defines whether recaptcha is enabled? There is disagreement between the template and the lib. Maybe better to have a third option for this. When setting the syntax for this, I can see us wanting to use captcha for other operations, like package edit, package create, group create etc. so perhaps we'd want to be able to turn each one on/off separately, using the name of the logic action. e.g. ckan.recaptcha.enable = user_create, group_create
• We got this exception last night from the template:

  Module ?:37 in <Expression u'g.recaptcha_publickey'>
  <<          <dd><input type="password" name="password2" value="" /></dd>
         
                 <dd py:if="g.recaptcha_publickey">
                   <script type="text/javascript"
                     src="http://www.google.com/recaptcha/api/challenge?k=${g.recaptcha_publickey}">
  >>  <dd py:if="g.recaptcha_publickey">
  Module genshi.template.eval:324 in lookup_attr
  <<                      val = obj[key]
                         except (KeyError, TypeError):
                             val = cls.undefined(key, owner=obj)
                 return val
  >>  val = cls.undefined(key, owner=obj)
  Module genshi.template.eval:410 in undefined
  <<          """Raise an ``UndefinedError`` immediately."""
                 __traceback_hide__ = True
                 raise UndefinedError(key, owner=owner)
  >>  raise UndefinedError(key, owner=owner)
  UndefinedError: <ckan.lib.app_globals.Globals object at 0xb8f1182c> has no member named "recaptcha_publickey"
  

I'm not sure why - might be something to do with the maintenance on the server, but worth double checking, since we don't want exceptions, say if one or other config variable is not defined.

Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package.

Based on number of views.

Related to ticket:143.

Basic description of multilingual features intended to be supported.

Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support:

1. Cases where multiple objects changed in a revision but only want to revert 1 (low priority)
2. Want to revert but have reversion as a new revision of that object.

Seems low priority at present.

Cost: ?

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.