{22} Trac tickets (2647 matches)

As A

Site User

I Want To

Create/Edit/Update/Delete? tags associated with my package

Notes

• Tags are service-wide and not package specific
• Tags should not allow spaces (but allow everything else e.g. ':')
• web interface details
  • should provide a text box where users can add space separated tags (a la delicious).
  • tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background)
  • would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious

Part of super-ticket #1506.

Follows on from #826.

We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis.

Child tickets

• #1198 Publisher hierarchy
• #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
• #1004 Group creation instructions missing
• #1099 Strange interactions between two browsers while playing with authz groups
• #1115 can have two authzgroups with the same name
• #1133 command line rights manipulation doesn't work
• #1138 minor navigations behave inconsistently

Old ticket description:

1. Change name of AuthzGroup? to UserGroup? to reflect what it is for

2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
   • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
   • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

3. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

This extension should be in core now -- functionality is basic and very useful.

Copy across from here: ​http://bitbucket.org/okfn/ckanext-admin

Suggest this no longer even be an extension.

Subticket of #1294.

• Nice way to edit all resource attributes
• Nice way to add new resources
  • Including file upload (cf #878).
    • Cf work in ckanjs (this can be a straight integration).

• Major issue with spam signup on thedatahub.org
• Suggest using recaptcha

Est cost: 1d

Awesome to have the captcha implementation and running on thedatahub.org! Here are some further suggestions concerning the config options, before this goes into a release:

• Document "ckan.recaptcha.publickey" and "ckan.recaptcha.privatekey" in doc/configuration.rst
• Which one config option defines whether recaptcha is enabled? There is disagreement between the template and the lib. Maybe better to have a third option for this. When setting the syntax for this, I can see us wanting to use captcha for other operations, like package edit, package create, group create etc. so perhaps we'd want to be able to turn each one on/off separately, using the name of the logic action. e.g. ckan.recaptcha.enable = user_create, group_create
• We got this exception last night from the template:

  Module ?:37 in <Expression u'g.recaptcha_publickey'>
  <<          <dd><input type="password" name="password2" value="" /></dd>
         
                 <dd py:if="g.recaptcha_publickey">
                   <script type="text/javascript"
                     src="http://www.google.com/recaptcha/api/challenge?k=${g.recaptcha_publickey}">
  >>  <dd py:if="g.recaptcha_publickey">
  Module genshi.template.eval:324 in lookup_attr
  <<                      val = obj[key]
                         except (KeyError, TypeError):
                             val = cls.undefined(key, owner=obj)
                 return val
  >>  val = cls.undefined(key, owner=obj)
  Module genshi.template.eval:410 in undefined
  <<          """Raise an ``UndefinedError`` immediately."""
                 __traceback_hide__ = True
                 raise UndefinedError(key, owner=owner)
  >>  raise UndefinedError(key, owner=owner)
  UndefinedError: <ckan.lib.app_globals.Globals object at 0xb8f1182c> has no member named "recaptcha_publickey"
  

I'm not sure why - might be something to do with the maintenance on the server, but worth double checking, since we don't want exceptions, say if one or other config variable is not defined.

Tweaking UIs and polishing the work done in #1294.

## Edit Page ##

• [Done; cset:49e0b059b6f5] Hide upload if storage not enabled
• [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
• [Done; cset:03904fc50cbf] Move edit form tabs to the left
• [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
• [Done] We used to autocomplete the format field - restore this functionality
  • Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
• [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
• [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
  • NB: reveal_instructions.js already disabled on edit page
  • [RP] have now done this on create -- think we can leave on edit
• [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
• [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view
• [Done; cset:d3595143e506] Need ability to delete resources
  • No thought yet given to uploaded resources. Long term re-think probable.
• [Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text)

## General ##

• [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
  • More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
• [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
• [Done] Move script stuff to footer of page

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

• application.js
• templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs.

Suggest we write a script that automatedly generates reference documentation for the config from the source.

May be obsoleted by #277 (some config in db)

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

• [dataset/edit]: Deletions are confirmed with a dialog box
• [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
• [dataset/edit]: Rearrange the delete resources mechanism;
  • Use a little minus icon on the far left, followed by a pencil icon for editing.
  • The 'delete resource' button can become 'hide'
  • Deletions are confirmed with a dialog box
• [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
  • Delete the little green plus sign on the right;
  • Try a flash banner instead (more general purpose & beautiful)
• [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
• [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
• [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
• [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
• [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
• [groups/add]: Hide unimportant fields
• [navigation] Add 3D bevel (@see github) to current tab highlight.
• [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see ​http://groupspaces.com/a/group/
• [groups/view]: Show description and owner. To match dataset section.
• [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
• [groups]: Entire section to use pretty buttons please.
• [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

• [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
• [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
• ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
• [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
• [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
• [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)

Integrating ckanext-admin into core has thrown up a number of problems:

• Look & feel does not match the rest of the site
• Tests are not passing
• On the trash page, clicking 'undelete' triggers a purge
• Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org)
• Trash page can contain nested form tags in certain cases (breaking test harness & form redirection)

eg. ​http://thedatahub.org/dataset/us-gov-tarp

[This Paper][my-paper] is very interesting.

[my-paper]: ​http://somelink.com/paper

Write a test to check this markdown feature works - if not, use external markdown lib rather than webhelpers.markdown. Probably need to add markdown lib to list of requirements.

Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain.

Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column.

Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible).

The forthcoming groups refactor will probably have some bearing on this task.

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

[Reopened: Additional] Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history.

Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering).

• Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?)

Super ticket: #1032

• Locate at: /dataset/{dataset}/resource/{id}

See: ​http://wiki.ckan.org/Dataset_View_Page

Implemented in branch feature-1445-resource-view.

Still to do:

• Add inline data explorer to page

Historical Parent Tickets

• Parent-ticket: #1450
• Parent-ticket: #1032

Step Zero

• COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

• Force format field to be lowercase
  • Modify backend to serve it up in that format
• Subticket #978: Add extra fields to resources in Web UI
• Fix display of large datasets (​http://test.ckan.net/dataset/europeana-lod)
• When submitting forms with errors, open the correct resource pane.
• No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
• Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

• Subticket #1648: Clarify edit terminology
• Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
• Fix (i.e. allow / show) name editing on dataset edit.
  • Fix the slugify so it does not overwrite if i change title after fixing name.
  • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
• Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

• Fix animation glitch in expanding description
• Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
• Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

• "Upload or link data files, APIs and other materials related to your dataset."
• Include the description markdown editor again (keep it small)
• Licence should maybe be restored
  • "(Don't worry if you don't know what your licence is)"
• Show the log message field
  • Beautify
• Pre-populate the log message field?
• ? Show Home Page field when creating dataset [WONTFIX don't push it]
• ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

• Improvements to user page UX - 1d - #1396
• Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

• Take before and after screenshots

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: ​http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.

Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in.

This was suggested by Richard Cyganiak: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html

Tasks:

• On dataset view page, add star rating after each resource view url. Probably similar to current format label after url.
• Add star rating to resource view page, along with a way of displaying the reason for the rating and tips to improve. Should be unobtrusive.
• Fix the table at ​http://thedatahub.org/qa/dataset/broken_resource_links - URLs should be truncated to a fixed not-too-long length and the other fields line up (eg by using a table)~
• Handle 404s by showing that 'we think this resource is 404ing'.
• Handle the absence of QA data on a resource (currently nothing is displayed. Maybe that's fine.
• Do we need to integrate the 'reason for rating' string into the UI?

Next Steps:

• Use db dump of thedatahub.org for final testing.

When viewing a dataset, the "Resources" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu.

​http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f

Chrome/Linux?

Super ticket #1744

If you edit a dataset, simply add a tag and click on "Save" then you get the message "You have unsaved changed. Make sure to click 'Save Changes' below before leaving this page." inserted whilst the submission takes place. (This is because the click moves focus away from the tag field). This (reasonably common) edge-case could usefully be squashed.

Also note typo: "unsaved changed"

• Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
• Icons for *types* of resources in resource list
• List of 'Owners' (administrators) at top right of sidebar (a la group page)
  • Much prefer owner to administrator ...

Why?

• We already use quite a bit of boostrap
• We already plan to use bootstrap JS
• Bootstrap is excellent and pretty light

When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state.

Repro:

1. Go to /dataset/new
2. Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size...
3. Add another resource, doing the same thing: make one of the fields invalid.
4. Try to save the dataset.
5. The entered resource information will be lost, and a js error "Uncaught Error: Can't add the same model to a set twice,: backbone.js:586" will be thrown.

Related to: ticket:90 Link to RDF version of CKAN data and ticket:163 (regularly convert CKAN data to RDF).

Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net.

There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests.

Introducing caching has to be done carefully and should be done in a standards compliant manner.

General strategy

• Where possible, cache output within the pylons app (beaker).
• Facilitate external caching in an end-user's web browser or a caching proxy
• Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes).
• When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data.

Tasks

These tasks should be broken into sub-tickets:

• caching of parts of templates that are expensive to render (package list, tag list, group list)
• caching of entire output using beaker particularly for API read operations.
• need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches.
• cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: ​http://www.mnot.net/cache_docs/#CACHE-CONTROL
  • Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well
  • Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner

Future Work

• Investigate ckanclient library maintaining a local cache as a web browser would
• Investigate using a CDN like Google Storage or Amazon for serving cached data.

To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: ​http://www.ibm.com/developerworks/web/library/wa-rails2/

In the first place we just need to apply this to:

• You are logged in as
• Package/group edit/authz tabs

Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them.

Perhaps 80% of the users of the UKLII may be using ArcGIS.

ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example:

B30E6D9F       2873 Sat Dec  4 14:05:40  MAILER-DAEMON
(delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out)
                                         [email protected]

both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null

Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

• master ticket #852

...

handle various dgu extras, configure apache for autonegotiation, etc.

master ticket

• #884 implement harvest job delete command for an easier life
• #885 move csw code into dedicated ckanext plugin
• #886 move forms api into dgu extension module
• #887 move harvesting code to use generic harvesting module

doesn't belong in ckan core

second part of #908

need to make some changes for the links to semantic.ckan.net. it should use ​http://semantic.ckan.net/record/<package_id> now

append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table) to taste (or just leave off the suffix and let content negotiation take care of it)

the base url is changed, but it now uses id not name.

see for example:

• ​http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33
• ​http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33.ttl

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

If a user enters a url with no protocol (eg: www.data.gov.uk) in the 'url' field of a dataset, a broken link is created on the view page.

CKAN should probably either put in a ​http:// if no protocol is given, or validation should fail.

To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

• Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

• Write Middleware + update url_for to take account of the language. [2d]

• Document the language setup, and how to replicate it. [1d]

language chooser should look like the ec one

not all tests run and they take too long :(

Page views and Resources clicks need to be tracked.

User Stories

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

• I want to see the traffic breakdown by country to my site ...
• Ditto for browser type, language, etc etc
• I want to see it graphed over time ...

Implementation Details

1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display

• Config option ckan.status.enabled = False (by default)

Storing Data

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

Displaying Data

• Helper functions / dictize:
  • Helper function: h.stats_get(object_type, id, stats_type)
    • h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
  • Change to dictize
• Location in the default theme (do we show for example in search results too!)
• Support for ranking by most popular in search?

Tracking Data

• Our own solution (just write to site_tracking)
• Google analytics (plus extension for retrieving data) <- would need a refactor
• Piwiki

Own Solution

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

• Make javascript to make request to ckan to store clicks and page views.
• Add middleware so these requests do not go through pylons and just store data quickly.