It would be useful to be able to manipulate packages using the paster command-line interface. In particular listing, reading and deleting packages. Seeing which packages were deleted too.

Most of the different pagination links are broken and point to wrong locations. e.g:

• Dataset listings in group page:

​http://test.ckan.net/group/bibliographic

• Tag letters in tag listing:

​http://test.ckan.net/tag

• User listing:

​http://test.ckan.net/user

• Revision listing:

​http://test.ckan.net/revision

This is most probably related with:

​https://github.com/okfn/ckan/commit/1772a5c

Reasons: more people familiar with Git, bigger Open Source community on Github.

Keeping track of the move here: ​https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&hl=en_GB#gid=0

This query caused ckan to except because ckan/controllers/api.py doesn't catch SearchError? and SearchQueryError?:

curl http://localhost:5000/api/action/package_search -d '{"sort": "metadata_modified"}'

This ticket is about finalizing, then deploying (to ​http://thedatahub.org/), and finally announcing some of recent major new features such as:

• Deploy wordpresser extension for thedatahub (so we can start blogging) - #1533 - WONTFIX
• (Deploy new Resource and Dataset pages (invite comments))
• Get thedatahub geo-ified i.e. ckanext-spatial - #1601 (?)
  • Any additional dev that needs to be done? (e.g. geo in search ...?)
• (Deploy / re-enable ckanext-googleanalytics on thedatahub - #1451 (more stuff #1566))
  • Rename the extension to ckanext-analytics?
• Custom forms (and use of 'dataset types') - Ticket (??)
• (Get QA deployed and running - #1582)

It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc.

Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.

Richard Cyganiak: I often see a problem where half the system thinks that I'm logged in, while the other half insists I'm not. Logout then login again may fix it.

David Read: I think I've seen it several times that the "My Account" link is given (suggesting you're logged in) but when you click it it asks you to log in, which means you weren't.

We want to add "activity streams" (chronological lists of activities) to CKAN.

Activity streams standard: ​http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): ​https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: ​http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: ​http://datahub.pudo.org/pudo

Tasks

Strikethrough means the task is done.

• New domain objects Activity and ActivityStream? [2d] #1298

• Migrate existing data into activity streams model [3.5d] #1510

• Generate activities when objects are saved [4d] #1298

• Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511

• API call for getting a user's public activity stream as JSON [0.5d] #1495

• API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494

• Render a user's activity stream on her user page [3d] #1496 (needs review/page design work)

• Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.

• Logic functions for getting activity streams for packages, groups, and users.

• API calls for getting activity streams of packages, groups, and users as JSON.

• Rendering activity streams for packages and groups etc. into their pages. (done but disabled pending page design work)

• API calls for inserting events into activity streams. (Still need to handle activity details in the API, ticketed).

• Conform to activitystrea.ms standard

• Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.

• Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams

• Email notifications

• Extension point for extensions to register themselves as renderers for different types of activity

• Deploy to UAT [0.5d]

• Docs and announce [1d]

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: ​http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.

Makes more sense to only have one comprehensive stats/analytics extension, so when people are looking to add a stats extension they won't have to add two which may be confusing (is one an old version of the other? why do I have to add two? what's the difference? etc)

User-wise we need a way to display our stats with google analytics in the same place

Various UX and UI improvements to groups.

Etherpad: ​http://ckan.okfnpad.org/feature-1521-groups-ux

Improved groups

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as ​http://ie.ckan.net/ could be turned into a simple group on the ​Data Hub without losing their separate identity.

• Group homepages: Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.
• Subdomains for groups: Enable subdomains such as ie.thedatahub.org as aliases for thedatahub.org/group/ireland
• Search within the group: Have a big search box that searches just within the group on the group page, with a look like on ​http://thedatahub.org/dataset
• Have an Add a dataset to this group link on the group page, perhaps in the sidebar.
• Navigation within the group: Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from ​http://thedatahub.org/dataset
• Group documentation: Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.
  • The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.
  • Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines About, Frequently Asked Questions and How to get involved, we'd then get three links in the sidebar that point to datahub.org/group/ireland/about, datahub.org/group/ireland/frequently-asked-questions, datahub.org/group/ireland/how-to-get-involved
    • RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

Additional

• Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)
• Better group index page (sort by number of dataset, allow searching ...) - #1403
  • Add search to groups: ​http://thedatahub.org/group
  • Way to filter by activity, number of datasets
• #1597 Tag search, filtered by group
  • RP: not worth it in terms of benefit/cost IMO

Steps

• "Add Dataset" button on action bar
  • Make it a dropdown? -> New Dataset / Existing Dataset
  • Redirect Add Existing -> Edit
  • Check permissions
• Expandable "about this group" box
• Implement simplest search-this-group box
  • turn the list of datasets into a dynamic list of search results
    • pagination
  • Matching set of facets in the sidebar (matches 'real' search page)
    • Deleteable, displayed facets

Nice-To-Have

• Get rid of the old group_package_show action (confirm with team)
• Tabbed "about this group" box (extract headers?)
• Scrollable group members widget in the sidebar

Form refactor.

New create/edit package form wizard for DGU. See attached spec.

Form refactor. New create/edit package form wizard for DGU.

Etherpad: ​http://ckan.okfnpad.org/dgu-package-form

Tasks:

• #1526 : Simple one-page form.
• #1527 : Tab-ify the form using JS. No validation between steps.
• #1541 : Setup server for the DGU form-refactor.
• #1537 : Package create form wizard
• #1538 : Store partially-filled in package form data as a draft.
• #1539 : Package forms auto-save

How the rest of the work is carried out is up for discussion, as it depends on the approach taken. Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages. ​Google doc outlining design(s)

• use javascript to selectively hide/show parts of the form
• there's no validation between steps at this stage. It's still a "big save button at the end".

Currently when you log in it says "logged-in". Most sites show your user-name and this is helpful when you have more than one account or human using the computer.

The new members table needs a way so you can add arbitrary domain objects against them.

We need to extend the group schema to accept types, and instead of just being able to add packages to groups add members with their capacities that associated with different table rows.

4d

Superceded by #2234

Super ticket: #1508

Etherpad: ​http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved ​https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

• separate pages for each section of the form
• validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
• no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

• with the javascript approach it would be harder to test the workflow.
• with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure.
• the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
• the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
• the multi-page wizard wouldn't preclude a javascript-tabbing edit-form.

Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms.

An AJAX call is made every N seconds, and the auto-saved blob is updated.

note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave.

To enable us to show DGU work in progress, for feedback.

CKAN is pulling a number of resources from external locations. This causes problems when connectivity is limited and you have to work locally. Maybe some of them cold be moved to CKAN source to avoid external requests.

Quick search:

./ckan/templates/layout_base.html:            <img src="http://assets.okfn.org/images/logo/okf_logo_white_and_green_tiny.png" id="footer-okf-logo" />
./ckan/templates/layout_base.html:            <a href="http://opendefinition.org/"><img alt="This Content and Data is Open" src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" style="border: none ; margin-bottom: -4px;"/></a>
./ckan/templates/package/resource_read.html:                <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/package/read.html:          <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /></a>
./ckan/templates/_util.html:                    <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/_util.html:                  <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      this.$dialog.html('<h2>Loading results...</h2><img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:          self.setMessage('Uploading file ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      self.setMessage('Checking upload permissions ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
Binary file ./ckan/lib/app_globals.pyc matches
./ckan/lib/app_globals.py:                                  'http://assets.okfn.org/p/ckan/img/ckan.ico')
./ckan/config/deployment.ini_tmpl:ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

• Modify the query being sent to Solr

def before_search(search_params):

    return search_params

• Modify the results returned (after_search)

def after_search(search_results, search_params):

    return search_results

Please see this pad for implementation details:

​http://ckan.okfnpad.org/search

It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself.

Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric.

Questions:

• Core, or Extension, or Self-hosted?

(instead of using the prompt) - needed for deployment

Polish off comments extension dev and test. estimate 2 days.

• (Deploy it)
• Integrate with stats plugin (now that is in core) - #1101
• Blog about it #1581
• (Other tickets re upgrading for v1.5.1 - done but where are tickets?)

Is this ready for release? What needs to be done?

Last week I imported ca.ckan.net datasets into thedatahub.org, but the tags seem to have lost their dashes, underscores and dots.

We should allow full text search within databases for individual databases, although we may want to wait for any pending ports to postgres.

• deploy celery
• deploy QA and archiver tasks
• write up a blog post announcing QA on thedatahub

(details embargoed until 31/1/2012)

Super: #1594

From Richard Cyganiak on the CKAN discuss list:

According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license.

CKAN 1.5 installations need to be upgrade to 1.5.1.

Working methodically through the CKANs on ​http://datacatalogs.org/group/ckan :

At 1.5:

• ckan-brazil ​http://br.ckan.net/ s057 - DONE
• ckan_ireland ​http://ie.ckan.net/ s057 - DONE
• cz-ckan-net ​http://cz.ckan.net/ s057 - DONE
• datakilder-no ​http://no.ckan.net/ s057 - DONE
• data_gov_uk ​http://catalogue.data.gov.uk/
• datacatalogs-org ​http://datacatalogs.org/ s054 - DONE

Other versions:

• berlin-open-data ​http://daten.berlin.de/ CKAN hidden
• ckan-italia ​http://it.ckan.net/ 1.3.3b
• ckan-poland ​http://pl.ckan.net/ Down
• data-norge-no ​http://data.norge.no/ CKAN hidden
• datadotgc-ca ​http://datadotgc.ca/ ​http://ca.ckan.net/ 1.4.2a
• datagm ​http://www.datagm.org.uk/ 1.3.3
• datagov-ru ​http://datagov.ru/ CKAN hidden
• datospublicos-org ​http://datospublicos.org/ 1.4.2
• iati-registry ​http://iatiregistry.org/ 1.5.1c
• katalog-offene-daten-oesterreich ​http://offener.datenkatalog.at/ 1.3.3b
• nosdonnees_fr ​http://www.nosdonnees.fr/ 1.3.1a
• opengov-fi ​http://opengov.fi/ CKAN hidden
• opengov_es ​http://opengov.es/ 1.2
• publicdata-eu ​http://publicdata.eu/ 1.4.2a
• the-data-hub ​http://thedatahub.org/ 1.5.1
• helsinki_finland ​http://hri.fi/ CKAN hidden

Now deleted from datacatalogs.org:

• ckan-belgium ​http://be.ckan.net/ Migrated
• si-ckan-net ​http://si.ckan.net/ migrated
• gr-ckan-net ​http://gr.ckan.net/ Migrated
• hu-ckan-net ​http://hu.ckan.net/ Migrated
• ckan-lithuania ​http://lt.ckan.net/ Migrated

Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:

http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*

On this case a message "Subscribe to these search results" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)

Other tickets related:

• #1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.

Or at least it appears that way from revision log which displays all Group datasets against the changelog, see ​http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217 -- not apparent from that page but see my user page: ​http://thedatahub.org/user/rufuspollock

This may turn out to be a UX bug rather than a real bug ...

Assigning to kindly for review and comment.

Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups.

• Simple group count widget in JS for embedding in wordpress and elsewhere
  • Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box.
• All you need to do is do a dataset query over the API e.g. ​http://thedatahub.org/api/search/dataset?groups=economics and then embed in some html!

Estimate: 30m (for someone who knows their jquery).

Need thoughts from Adria and James. RP thoughts

• Integration into WUI search: #1469
• JS Editor for adding/editing spatial attribute: #1818
• Theme/template hook for adding the map e.g. on dataset page (or elsewhere)
• Clear Documentation
• Deployment
• Announce post
• (?) Map page for stats section showing location of datasets

Extras (probably separate):

• csw support ...

Super ticket: #1745

Ability to build up search query using a nice javascript-y interface.

• Add facets by selecting attribute and adding -> search facet options in dropdown -> added to search (with 'x' to remove -- as we currently do).
  • (a bit like the data.hri.fi)
• Some improvements to css
  • See ​http://eutr.ep.io/
• Improvements to faceting
  • Ability to configure faceting and number of items to show (?)
• Pure JS search implementation to make it easy to reuse across site

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: ​https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

• 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1
• 6. Publisher Registration Improvements MUST PHASE 1
• 3. Themes/Taxonomy? as well as tags WONT PHASE 2
• 32. Browse by Tags PHASE 1 MUST
• 12. Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1
• 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]
• 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]
• 22. Mixed Licenses MUST PHASE 2

Plus, brought forward from last sprint:

• Publisher hierarchy support
• Improved dataset creation tools (form) - #1525
• Edit/delete for harvested datasets

Overview of tasks: see the design doc.

From super Storage changes - #1574 - and ​http://ckan.okfnpad.org/newstorage we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore

Analysis

When I upload a file to CKAN:

• End up with file in permanent storage
• IF file is ot type ... csv,xls,xlsx,sqlite,.sql
  • End up with new db in webstore
    • Where? {username}/{resource-id}/...
      • If a single table: name it after the file name (appropriately slugified)
    • A resource *always* corresponds to a 'database' in webstore ...
    • In Data Explorer have "Sheets" tab ...
• Resource url = /dataset/{x}/resource/{y}/link -> cached_url ...

As part of #1574 we decided that it would be better for webstore (and future external services) to be able to authenticate simply with CKAN-Core.

Currently webstore access the CKAN database to obtain the key for the user but it would be better if this connection was not so tightly bound and that webstore used an API as any other external service might.

Need to discuss further with dread

1. Viewing a dataset, hit 'Edit'
2. Click on the "Basic Information" tab (note: URL has suffix #section-basic-information
3. Click 'Save'
4. URL still has suffix #section-basic-information

Affects 1.5, 1.5.1, 1.5.2a

This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*.

We would like this work to facilitate testing within VMs as part of our new build infrastructure.

It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment.

[Child of #1526]

Contact/FOI information is derived from Publisher/User?, but can be altered for each dataset.

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

[Child of #1526]

Create the interface used to retrieve publisher and authz information from the publisher hierarchy.

A test case is currently failing for me on master:

ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py", line 342, in test_user_create_unicode
    res = res.follow()
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 603, in follow
    return self.test_app.get(location, **kw)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 208, in get
    return self.do_request(req, status=status)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 389, in do_request
    **req.environ)
  File "/usr/lib/pymodules/python2.6/paste/wsgilib.py", line 343, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/lint.py", line 170, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/usr/lib/pymodules/python2.6/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 69, in __call__
    auth_ids = self.authenticate(environ, classification, ids)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 201, in authenticate
    userid = plugin.authenticate(environ, identity)
  File "/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py", line 29, in authenticate
    if user.validate_password(identity.get('password')):
  File "/home/seanh/pyenv/src/ckan/ckan/model/user.py", line 113, in validate_password
    hashed_pass = sha1(password_8bit + self.password[:40])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)

The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now.

Dump = paster db simple-dump-csv/json

reasoning

The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them.

Admins get the full database anyway in the backup pg_dump.

We only included them in the user dump because it was designed before use of state=deleted was established.

Time estimate: 2h

This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well.

On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab.

Original ticket #48

This was seen on datacatalogs. When we upgraded it 1.5 to 1.5.1 we saw some permissions being reset so that it could be spammed. Anonymous and logged in users were given anon_editor and editor permissions. I don't know what the previous config was. Permissions for sysadmins remain unaffected.