CKAN: Ticket Query

#1530: URL field help text in wrong place

lucychambers — Wed, 07 Dec 2011 12:46:24 GMT

Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.

#1605: [super] Multilingual support in CKAN

rgrp — Mon, 02 Jan 2012 03:02:47 GMT

Multi-language:

dataset and resource metadata (and other objects such as groups?)
- Options for implementation in http://ckan.okfnpad.org/multilingual
Field values in taxonomy (e.g. country names - Eurovoc)
- #1665 Research into Eurovoc
- Display Taxonomy in different languages
Field values not in taxonomy (e.g. title & description)
- Use extra fields e.g. _i18n_title_fr = Le data.
- (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)
EC extension (templates, form)
- Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?

#1734: [super] Updates on harvesting for UKLP

amercader — Tue, 31 Jan 2012 11:14:34 GMT

Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources.

#1726 Update harvesting model
#1727 Add "current" field to harvesting objects
#1728 Update UKLP harvesters
#1733 Delete (inactivate) harvest sources

At some point we also want to support a publisher model for the harvesting:

#1731 Publisher support for harvesting

#2332: Fixes for v1.7 release

rgrp — Sat, 28 Apr 2012 20:15:16 GMT

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

~~Related Item has comments on index page (they should not show up here -- why is this happening!)~~
~~Related item should be to left of history (history could in fact move to RHS)~~
- What about losing icons since we duplicate for related item
Related item plus sign is very poorly aligned within box
- Add related items should be in drop down like for resources (?) to make it easy to add items
- Hover should show a title that gives some information about what related are (not at all clear)
Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

#2331 - search should add not or
~~Search results should be similar across group pages and normal search results (suggest adopting group page style)~~
- Search results resource link should link to resource page rather than url for that resource
- Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

~~Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07~~
- Ensure still using read-only (this seems to have been lost in upgrade)
Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

(? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
- Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
- Do #2322 ...
Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
Data viewer embed url includes all the resource info

Dataset page:

Put home page link somewhere more prominent (e.g. the top of the sidebar)
Put author more prominent (at top right)
(? - new work) List CKAN owners at top right in sidebar
Make resources stand out more
Rename resources tab and heading to Data & Resources

Dataset edit and create:

Corrections to dataset creation form for v1.7 - #2317
Better auto-complete for groups on dataset edit/new page - #2319
- Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
Description / notes text area is not wide enough.
- Should also be larger on edit page
Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
- A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
- Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

Do not show no description in no description (just show dataset info)

Theme:

~~Fix spacing above page title (i.e. dataset title etc)~~
~~Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer~~
~~Fix subnav width on resources pages (does not extend all the way to the right like it should)~~
~~Fix spacing of top menu and search box (do not look right)~~
Group pages:
- view page: image / logo is really poorly locate and sized, see e.g. http://datahub.io/group/wikimedia
- ~~view page: admin heading on RHS sidebar is wrongly indented~~

Miscellaneous:

Ensure that we out-of-the-box config for CKAN has (#2388):
- Only logged in users able to create
- Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

~~Social extension~~
~~Analytics~~

#1397: [super] Resource archiving

rgrp — Thu, 13 Oct 2011 18:13:13 GMT

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)
CELERY_IMPORTS = celeryimports

Work Items

Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
Generate archiving request on resource url change [0.25d][0.25d] - #1399
Make site user account.
Make entry point system for celery config
Archiver daemon #891
1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
Write to resource and task status table.[1d][0.75d]
[Required?] Make archived data available in WUI - #892
Documentation - #1400

#1607: [super] Data.gov.uk Maintenance Refactor and UKLP Development Support

dread — Mon, 02 Jan 2012 11:22:28 GMT

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

7a. ~~Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1~~
6. ~~Publisher Registration Improvements MUST PHASE 1~~
3. ~~Themes/Taxonomy? as well as tags WONT PHASE 2~~
32. ~~Browse by Tags PHASE 1 MUST~~
12. ~~Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1~~
13. ~~Provider labelling MUST PHASE 1 [UKLP #14 MUST]~~
17. ~~Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]~~
22. ~~Mixed Licenses MUST PHASE 2~~

Plus, brought forward from last sprint:

~~Publisher hierarchy support~~
~~Improved dataset creation tools (form) - #1525~~
~~Edit/delete for harvested datasets~~

Overview of tasks: see the design doc.

#1566: [super] Finalize Google analytics extension

jilly mathews — Mon, 19 Dec 2011 11:12:48 GMT

~~(Deploy it)~~
Integrate with stats plugin (now that is in core) - #1101
Blog about it #1581
(Other tickets re upgrading for v1.5.1 - done but where are tickets?)

#1465: Upgrade the harvester to support publishers properly

thejimmyg — Tue, 15 Nov 2011 16:52:03 GMT

At the moment most of the harvesting functionality is only available to sysadmins. This is fine for most cases but going forward we will need to grant each publisher access to their own sources, harvest objects and harvest object errors. This will probably be done based on the groups refactor.

One more pressing concern is giving DGU publishers the ability to make calls to /harvest/object/id. We want them to be able to see documents that faild validation and also to see *all* errors in each document.

#1496: Render a user's activity stream on her user page

seanh — Mon, 28 Nov 2011 15:53:08 GMT

#1515: [super] Activity Streams

seanh — Mon, 05 Dec 2011 17:10:48 GMT

We want to add "activity streams" (chronological lists of activities) to CKAN.

Activity streams standard: http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: http://datahub.pudo.org/pudo

Tasks

~~Strikethrough~~ means the task is done.

~~New domain objects Activity and ActivityStream? [2d] #1298~~

Migrate existing data into activity streams model [3.5d] #1510

~~Generate activities when objects are saved [4d] #1298~~

~~Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511~~

~~API call for getting a user's public activity stream as JSON [0.5d] #1495~~

~~API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494~~

~~Render a user's activity stream on her user page [3d] #1496~~ (needs review/page design work)

~~Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.~~

~~Logic functions for getting activity streams for packages, groups, and users.~~

~~API calls for getting activity streams of packages, groups, and users~~ as JSON.

~~Rendering activity streams for packages and groups etc. into their pages.~~ (done but disabled pending page design work)

~~API calls for inserting events into activity streams.~~ (Still need to handle activity details in the API, ticketed).

Conform to activitystrea.ms standard

Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.

Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams

Email notifications

Extension point for extensions to register themselves as renderers for different types of activity

Deploy to UAT [0.5d]

Docs and announce [1d]

#1574: [super] Storage changes including automated save to Webstore

ross — Mon, 19 Dec 2011 17:27:13 GMT

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]

The user is provided the link to the file as currently.

The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6

It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.

The archiver will continue to move the file into permanent storage as per current practice.

Tickets

#1608 Merge storage into Core [5d]
#1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
#1687: Deploy to datahub [1d]
#1681: Core storage documentation [1d?]
- Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
- External: here's how to configure it and here's what you get ... (goes in main sphinx docs)

#1601: [super] Well integrated Geo-spatial support including search

rgrp — Fri, 30 Dec 2011 15:43:41 GMT

Need thoughts from Adria and James. RP thoughts

Integration into WUI search: #1469
JS Editor for adding/editing spatial attribute: #1818
Theme/template hook for adding the map e.g. on dataset page (or elsewhere)
Clear Documentation
Deployment
Announce post
(?) Map page for stats section showing location of datasets

Extras (probably separate):

csw support ...

#1669: [super] Publisher profile

ross — Tue, 17 Jan 2012 12:14:33 GMT

Implement the necessary backend changes to build profile for publishers using the group refactor.

See http://ckan.okfnpad.org/publisherprofiles

Subtickets are at:

Publisher profile analysis #1670 [2d]
Publisher profile sysadmin authz #1671 [1d]
Publisher profile user authz #1672 [1d]
Test auth logic hooks for publisher profile #1674 [3d]
Publisher admin authz #1675 [3d]
Publisher editor authz #1676 [2d]
Add approval status field to groups #1673 [1d]

#2230: Tidy up of search facets code duplication

seanh — Wed, 14 Mar 2012 16:54:45 GMT

Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point):

The package_search() function is adding the search facets to the search results twice with two different data structures, with keys "facets" and "new_facets". It should be reduced to just the new facets (with the key changed to "facets").

Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets).

The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure.

In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets.

In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead.

Anywhere that "new_facets" appears it will have to be changed to "facets" (e.g. in the ckanext/multilingual/plugin.py.

This is the merge commit that introduced the duplication: https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d

This is the earlier commit where the search facets were refactored: https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c

#2317: Corrections to dataset creation form for v1.7

rgrp — Sat, 21 Apr 2012 15:27:41 GMT

Add home page url
Remove group option (seen it confuse several users) - perhaps replace with tags (?)
Check form looks ok in portugese (seems to be a problem with css)

#1594: [super] QA Improvements

johnglover — Wed, 21 Dec 2011 17:47:00 GMT

Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement.

Integrate QA into frontend - 1d - #1583
QA report improvements - 2.5d - #1584
QA - Check license as part of 5 star rating - #1587
QA - Give SPARQL endpoints a 4 star rating - #1588
QA - Give 5 star rating to datasets with link metadata - #1589

Main pad: http://ckan.okfnpad.org/qa

Some QA discussion (relating to Celery in particular) at http://ckan.okfnpad.org/queue

#1404: Search Page UI improvements

zephod — Mon, 17 Oct 2011 10:36:58 GMT

[Refactored] :: Follows on from #1506 UX changes.

Declutter the sidebar. No yellow box.
Facets to go on the left, rather than the right. More logical flow.
Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
- Update that look-and-feel to include the new resource icons created in #1506

#1510: Migrate existing data into activity streams database tables

seanh — Mon, 05 Dec 2011 14:05:01 GMT

See '### Task 2' in this etherpad: http://ckan.okfnpad.org/notifications

I have no idea yet about how to do this or how long it might take me.

#1595: Editing of a Group leads to recording a change against all associated datasets

rgrp — Wed, 21 Dec 2011 22:01:15 GMT

Or at least it appears that way from revision log which displays all Group datasets against the changelog, see http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217 -- not apparent from that page but see my user page: http://thedatahub.org/user/rufuspollock

This may turn out to be a UX bug rather than a real bug ...

Assigning to kindly for review and comment.

#1600: [super] User guide with integrated help in the WUI

rgrp — Fri, 30 Dec 2011 15:10:24 GMT

Provide help to users in the WUI, with this linking to a detailed user guide.

User guide (suggest at help.thedatahub.org (or possibly help.ckan.org)
Snippets in the WUI with links to full user guide.
- Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis
General info about publishing/working with/etc data (where we don't duplicate elsewhere)

Est: (?) 5-10d

Topics

See http://ckan.okfnpad.org/datahub-help

Step-by-step for adding a dataset including tips
- Reuse: http://datapatterns.org/datahub.html
- Quick overview of relevant parts of domain model (dataset and resource)
API (perhaps link to main docs)
Command line interface and dpm
Using groups

Why do this

We already have wiki.ckan.org and docs.ckan.org. Why do this?

Answer for docs:

This is administrative guide and focused on general CKAN software.
Hard to provide completely generic help.
Integratable with the DataHub? site

Answer for wiki:

No spam
General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful.
Integratable with the DataHub?

#1632: Render a group's activity stream on its' page

seanh — Wed, 11 Jan 2012 17:49:59 GMT

Add logic functions for getting the activity stream for a group in JSON and rendered HTML formats, add HTML activity stream into group page.

#1633: Render a dataset's activity stream on its' page

seanh — Wed, 11 Jan 2012 17:51:38 GMT

Add logic functions for getting the activity stream for a dataset in JSON and rendered HTML formats, add HTML activity stream into dataset page.

#1636: Purge items from activity streams when object (package etc.) purged from CKAN

seanh — Wed, 11 Jan 2012 17:59:14 GMT

#1692: Add image attribute to Dataset and Group

rgrp — Sat, 21 Jan 2012 01:19:21 GMT

Add image attribute for Dataset and Group which would be shown in prominent place.

For dataset this could be a relevant image and for Group this would be more of a logo.

I think this has very high value from a UX point of view.

Issues

Do we need to support uploading and storing the image (or do users store elsewhere)
- Could use something like transloadit service
- Use our own storage facility with a reserved directory
Do we need to do resizing etc?
- Maybe. Perhaps can prefer this in reasonably small size and then just size down in css for thumbnails etc.

#1763: Add some simple model-level tests for activity streams

seanh — Mon, 06 Feb 2012 13:51:25 GMT

#1764: Move activity streams API tests into tests/logic

seanh — Mon, 06 Feb 2012 13:54:59 GMT

Which is where the other API tests have now been moved.

#1784: Links to renamed objects break

seanh — Tue, 07 Feb 2012 11:19:15 GMT

The HTML templates for some activity stream activities link to objects (packages, resources, etc.) by their names, e.g. /dataset/foo, but if the object has since been renamed then the link will be broken.

Possible solutions are to use the object ID to find its current name at rendering time and link to that, or link to the object by ID rather than by name.

Linking by ID is both simpler to implement and better. Links by ID are ugly, but they are permanent and don't break when an object is renamed. We should be using links by ID everywhere on CKAN, so that when third-party sites copy-paste our URLs and link to us, their links don't break later.

Fixing this means fixing the dataset_link(), resource_link(), group_link() etc. functions in ckan/lib/helpers.py which will change the links everywhere in CKAN where those functions are used to link to objects (which should be everywhere where we link to an object).

If we want to have nice looking by-name URLs and what them to be permanent, then we need to change CKAN so that objects cannot be renamed.

#2288: Fix i18n of strings on auth group add/edit pages

seanh — Thu, 12 Apr 2012 18:23:23 GMT

There are a couple of strings on the authorization group add/edit form that are not being translated:

"Name: Please enter a value" (error message when you try to add an auth group with no name)

"User name"

"Save" button

The form seems to be built using FormBuilder?. See ckan/forms/authorizationgroup.py, builder.set_displayed_fields()

#1586: 'Sort by' field for package search

dread — Wed, 21 Dec 2011 14:59:48 GMT

We default to sort packages search results by "search rank", but we can change it to be "name", "title", "metadata_modified" date, "metadata_created" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this.

Follows on from #191.

#2201: Add citation info to Dataset and Resource page

rgrp — Sat, 03 Mar 2012 08:58:00 GMT

Suggest:

%title. %author. Retrieved %date. %site_title.

For resource: %title = %dataset_title. %resource_name.

#1603: Search query builder

rgrp — Fri, 30 Dec 2011 18:06:04 GMT

Super ticket: #1745

Ability to build up search query using a nice javascript-y interface.

Add facets by selecting attribute and adding -> search facet options in dropdown -> added to search (with 'x' to remove -- as we currently do).
- (a bit like the data.hri.fi)
Some improvements to css
- See http://eutr.ep.io/
Improvements to faceting
- Ability to configure faceting and number of items to show (?)
Pure JS search implementation to make it easy to reuse across site

#1634: Allow users to follow/unfollow activity streams of other users, datasets and groups

seanh — Wed, 11 Jan 2012 17:55:42 GMT

This is a fairly big feature to add. Analysis: http://ckan.okfnpad.org/27

This ticket is related to #1635 (email notifications for activity streams) which is about managing a list of activity streams for which the user receives email notifications. There are decisions to be made about how the two will work together.