{22} Trac tickets (2647 matches)

Various UX and UI improvements to groups.

Etherpad: ​http://ckan.okfnpad.org/feature-1521-groups-ux

Improved groups

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as ​http://ie.ckan.net/ could be turned into a simple group on the ​Data Hub without losing their separate identity.

• Group homepages: Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.
• Subdomains for groups: Enable subdomains such as ie.thedatahub.org as aliases for thedatahub.org/group/ireland
• Search within the group: Have a big search box that searches just within the group on the group page, with a look like on ​http://thedatahub.org/dataset
• Have an Add a dataset to this group link on the group page, perhaps in the sidebar.
• Navigation within the group: Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from ​http://thedatahub.org/dataset
• Group documentation: Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.
  • The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.
  • Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines About, Frequently Asked Questions and How to get involved, we'd then get three links in the sidebar that point to datahub.org/group/ireland/about, datahub.org/group/ireland/frequently-asked-questions, datahub.org/group/ireland/how-to-get-involved
    • RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

Additional

• Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)
• Better group index page (sort by number of dataset, allow searching ...) - #1403
  • Add search to groups: ​http://thedatahub.org/group
  • Way to filter by activity, number of datasets
• #1597 Tag search, filtered by group
  • RP: not worth it in terms of benefit/cost IMO

Steps

• "Add Dataset" button on action bar
  • Make it a dropdown? -> New Dataset / Existing Dataset
  • Redirect Add Existing -> Edit
  • Check permissions
• Expandable "about this group" box
• Implement simplest search-this-group box
  • turn the list of datasets into a dynamic list of search results
    • pagination
  • Matching set of facets in the sidebar (matches 'real' search page)
    • Deleteable, displayed facets

Nice-To-Have

• Get rid of the old group_package_show action (confirm with team)
• Tabbed "about this group" box (extract headers?)
• Scrollable group members widget in the sidebar

Re-create behaviour of old form without using form-alchemy. And unit test.

• [X] write high-level functional tests for the form. Creation / edition / validation.

Original estimate: 6 days (3 days x 2)

Time spent: 3 days

Time remaining: 0 days

• [X] write the html

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 0 days

• [X] modify the existing validation schema

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 1 days

• [X] split resources into types: individual, time-series, and additional.

Original estimate: 2 days

Time spent: 2.5 days

Time remaining: 0 days

• [X] contact information is derived from Publisher, but can be altered for each dataset. #1617

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

(Waiting on publisher integration)

• [X] foi contact information is derived from user, but can be altered for each dataset. #1617

Original estimate: 1 days

Time spent: 0 days

Time remaining: 1 days

(Waiting on publisher integration)

• [X] provide stubs for the theme and sub-themes #1618

Original estimate: 1 days

Time spent: 0 days

Time remaining: 0 days

• [X] provide stubs for the publisher hierarchy #1619

Original estimate: 2 days

Time spent: 0.5 days

Time remaining: 1.5 days

Note - the groups refactor is essentially ready to be used. And this would be a good test case for it.

• [X] don't allow package-type selection when editing a package

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (#1620)

Original estimate: 0.5 days

Time spent: 0 days

Time remaining: 0 days

Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in.

This was suggested by Richard Cyganiak: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html

Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.

The log-in page says "Login using Open ID" and gives instructions for signing up. YET this is only available to users who've already added openid to their account. If you have not done this and then sign-in via OpenID (which is successful from the OpenID end) then you are told "Login failed. Bad username or password." in a flash message.

Proposed solution (i don't know if this is possible):

• When you log-in for the first time via OpenID, it doesn't actually log you in in CKAN. It just sends you to the 'Create User' page with the OpenID field pre-filled, and puts up a flash message "This OpenID account is not yet registered on thedatahub. Please complete your details.". This allows you to complete the registration and logs you in, and allows you to log-in directly with OpenID in the future.

Alternatives:

• Just change the error message to be 'You need to register in CKAN first. Quote your OpenID in the registration form to use it in future.'
• Remove OpenID altogether

Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

• separate pages for each section of the form
• validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
• no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

• with the javascript approach it would be harder to test the workflow.
• with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure.
• the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
• the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
• the multi-page wizard wouldn't preclude a javascript-tabbing edit-form.

To enable us to show DGU work in progress, for feedback.

metadata_created seems to be returning the date of the earliest revision (for all packages), rather than the earliest revision for a specific package.

It's hard to understand what's going on with the functional tests because there is so much going on in inherited classes. Would be better to be explicit in constructors, even if it involves cut & paste.

(instead of using the prompt) - needed for deployment

Polish off comments extension dev and test. estimate 2 days.

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]

2. The user is provided the link to the file as currently.

3. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6

• It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.

4. The archiver will continue to move the file into permanent storage as per current practice.

Tickets

• #1608 Merge storage into Core [5d]
• #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
• #1687: Deploy to datahub [1d]
• #1681: Core storage documentation [1d?]
  • Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
  • External: here's how to configure it and here's what you get ... (goes in main sphinx docs)

• Move into core (keeping as an extension) and default enable
• Do we need tests for controller? (No: already there)
• Documentation (?) - e.g. of basic config options
• Remove lib/stats.py which is still in core
  • dread moved it to extension in ​https://github.com/okfn/ckanext-stats/commit/293b2544737d8ab7610884bd5e999ee37f1ab813 but did not remove from core
• Deprecation notice on ckanext-stats repo and update extension registry

Est: 0.5d.

Questions:

• Why do this?
  • tiny extension with few dependencies - and really nice to have out of the box
  • trial for doing this on larger scale
• Do we keep as extension (even if in core)?
  • Ans: Yes, keep as extension because:
    • Already set up that way
    • Cleaner
    • Easier to disable / enable
• How do we integrate with main them (e.g. have stats link)
  • Ans: not sure (this is part of more general issue of how we update theme for varying changes elsewhere). Best answer is to have some widgetization in theme.
• Hide ratings section (at least until we reinstate ratings #1598)
  • Ans: no, let's not bother (and having ratings there encourages us to do #1598 and/or find out whether people are interested in ratings)

• Write up: task status table in ​http://wiki.ckan.org/Domain_Model
• Write up: ​http://wiki.ckan.org/Data_Quality to reflect current state of play (move all old material to "Discussion / Talk" page)
• Ping ckan-discuss re new Data Quality page

Tasks:

• On dataset view page, add star rating after each resource view url. Probably similar to current format label after url.
• Add star rating to resource view page, along with a way of displaying the reason for the rating and tips to improve. Should be unobtrusive.
• Fix the table at ​http://thedatahub.org/qa/dataset/broken_resource_links - URLs should be truncated to a fixed not-too-long length and the other fields line up (eg by using a table)~
• Handle 404s by showing that 'we think this resource is 404ing'.
• Handle the absence of QA data on a resource (currently nothing is displayed. Maybe that's fine.
• Do we need to integrate the 'reason for rating' string into the UI?

Next Steps:

• Use db dump of thedatahub.org for final testing.

Super: #1594

From Richard Cyganiak on the CKAN discuss list:

According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license.

CKAN 1.5 installations need to be upgrade to 1.5.1.

Working methodically through the CKANs on ​http://datacatalogs.org/group/ckan :

At 1.5:

• ckan-brazil ​http://br.ckan.net/ s057 - DONE
• ckan_ireland ​http://ie.ckan.net/ s057 - DONE
• cz-ckan-net ​http://cz.ckan.net/ s057 - DONE
• datakilder-no ​http://no.ckan.net/ s057 - DONE
• data_gov_uk ​http://catalogue.data.gov.uk/
• datacatalogs-org ​http://datacatalogs.org/ s054 - DONE

Other versions:

• berlin-open-data ​http://daten.berlin.de/ CKAN hidden
• ckan-italia ​http://it.ckan.net/ 1.3.3b
• ckan-poland ​http://pl.ckan.net/ Down
• data-norge-no ​http://data.norge.no/ CKAN hidden
• datadotgc-ca ​http://datadotgc.ca/ ​http://ca.ckan.net/ 1.4.2a
• datagm ​http://www.datagm.org.uk/ 1.3.3
• datagov-ru ​http://datagov.ru/ CKAN hidden
• datospublicos-org ​http://datospublicos.org/ 1.4.2
• iati-registry ​http://iatiregistry.org/ 1.5.1c
• katalog-offene-daten-oesterreich ​http://offener.datenkatalog.at/ 1.3.3b
• nosdonnees_fr ​http://www.nosdonnees.fr/ 1.3.1a
• opengov-fi ​http://opengov.fi/ CKAN hidden
• opengov_es ​http://opengov.es/ 1.2
• publicdata-eu ​http://publicdata.eu/ 1.4.2a
• the-data-hub ​http://thedatahub.org/ 1.5.1
• helsinki_finland ​http://hri.fi/ CKAN hidden

Now deleted from datacatalogs.org:

• ckan-belgium ​http://be.ckan.net/ Migrated
• si-ckan-net ​http://si.ckan.net/ migrated
• gr-ckan-net ​http://gr.ckan.net/ Migrated
• hu-ckan-net ​http://hu.ckan.net/ Migrated
• ckan-lithuania ​http://lt.ckan.net/ Migrated

Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:

http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*

On this case a message "Subscribe to these search results" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)

Other tickets related:

• #1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.

Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups.

• Simple group count widget in JS for embedding in wordpress and elsewhere
  • Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box.
• All you need to do is do a dataset query over the API e.g. ​http://thedatahub.org/api/search/dataset?groups=economics and then embed in some html!

Estimate: 30m (for someone who knows their jquery).

We already have first pass of Data Preview that was done as part of #1357. This is about v2 of this. Two main components:

• A new version of the separate Data Explorer library (now called Recline Data Explorer) - #1446 Est: 10-15d
  • Why:
    • Cleaner, backbone based solution.
    • Provides editing support if wanted.
    • Collaboration with Max Ogden
• Integrate this into CKAN (into resource view page) - see #1630 Est: 1d
• Improvements to Data Viewer / Previewer system in CKAN itself - Est: 2-3d
  • (Update Data Preview-er (View-er) to support images ...)
  • (WONTFIX here - (?) Update Data Preview-er to support geodata - #1151)

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: ​https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

• 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1
• 6. Publisher Registration Improvements MUST PHASE 1
• 3. Themes/Taxonomy? as well as tags WONT PHASE 2
• 32. Browse by Tags PHASE 1 MUST
• 12. Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1
• 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]
• 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]
• 22. Mixed Licenses MUST PHASE 2

Plus, brought forward from last sprint:

• Publisher hierarchy support
• Improved dataset creation tools (form) - #1525
• Edit/delete for harvested datasets

Overview of tasks: see the design doc.

From super Storage changes - #1574 - and ​http://ckan.okfnpad.org/newstorage we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore

Analysis

When I upload a file to CKAN:

• End up with file in permanent storage
• IF file is ot type ... csv,xls,xlsx,sqlite,.sql
  • End up with new db in webstore
    • Where? {username}/{resource-id}/...
      • If a single table: name it after the file name (appropriately slugified)
    • A resource *always* corresponds to a 'database' in webstore ...
    • In Data Explorer have "Sheets" tab ...
• Resource url = /dataset/{x}/resource/{y}/link -> cached_url ...

Its a pain to see the difference between branches as there are normally a lot of po file transaction. Make the default be see that they have changed without actually show the diffs themselves.

[Child of #1526]

Contact/FOI information is derived from Publisher/User?, but can be altered for each dataset.

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

[Child of #1526]

Create the interface used to retrieve publisher and authz information from the publisher hierarchy.

A test case is currently failing for me on master:

ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py", line 342, in test_user_create_unicode
    res = res.follow()
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 603, in follow
    return self.test_app.get(location, **kw)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 208, in get
    return self.do_request(req, status=status)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 389, in do_request
    **req.environ)
  File "/usr/lib/pymodules/python2.6/paste/wsgilib.py", line 343, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/lint.py", line 170, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/usr/lib/pymodules/python2.6/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 69, in __call__
    auth_ids = self.authenticate(environ, classification, ids)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 201, in authenticate
    userid = plugin.authenticate(environ, identity)
  File "/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py", line 29, in authenticate
    if user.validate_password(identity.get('password')):
  File "/home/seanh/pyenv/src/ckan/ckan/model/user.py", line 113, in validate_password
    hashed_pass = sha1(password_8bit + self.password[:40])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)

The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now.

Dump = paster db simple-dump-csv/json

reasoning

The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them.

Admins get the full database anyway in the backup pg_dump.

We only included them in the user dump because it was designed before use of state=deleted was established.

Time estimate: 2h

This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well.

On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab.

Original ticket #48

Super ticket: #1602

Steps:

• Include (or git submodule) data explorer into vendor
• Update existing js integration code
• Fix css and any bugs

Est: 1d

Actual: 1d

(and add test cases for it)

Set-up publicdata.eu to harvest datasets at rs.ckan.net (Serbian community CKAN).

Theme the DGU form.

Doesn't need to be an exact replica of DGU, but just enough to show it's possible.

• Make sure that the extension works as it should and isn't broken on current releases [3d]

• Make it work with the EC schema, including extras [2d]

To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

• Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

• Write Middleware + update url_for to take account of the language. [2d]

• Document the language setup, and how to replicate it. [1d]

Time estimate: 2d

• Fix logs (apache, ckan, harvest): rotate, set suitable levels
• Fix harvesting jobs: supervisord for gather consumer, cron job
• Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

The create_activity() logic function already exists, but validation and authorization need to be added before it's hooked up to an API call.

How big is it? How are we going to store it? etc.

Implement the necessary backend changes to build profile for publishers using the group refactor.

See ​http://ckan.okfnpad.org/publisherprofiles

Subtickets are at:

• Publisher profile analysis #1670 [2d]
• Publisher profile sysadmin authz #1671 [1d]
• Publisher profile user authz #1672 [1d]
• Test auth logic hooks for publisher profile #1674 [3d]
• Publisher admin authz #1675 [3d]
• Publisher editor authz #1676 [2d]
• Add approval status field to groups #1673 [1d]

Super #1669

Implement authz for sysadmins in publisher profile

Super #1669

As groups need to be authorised in some cases, particularly publisher profiles, we need a field to denote how this is managed.

We should add an 'approval_status' field that will be used to denote its status. By default this will be "approved" but this will allow us to override it with a "pending" or "denied" status. The latter should also change the main status of the Group.

Super #1669

Authz for the publisher admin (who has permission to administer the publisher group themselves).

Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour.

On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect.

This is a known issue: ​https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04)

It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently).

We need an API for changing the authorization (permissions) on a model object.

This was neglected in #1253 because of various changes to authz were looming. But six months later it still hasn't been done.

Suggest this is just moving _add_user_object_role from ckan/lib/base.py to the logic layer, but no doubt it will be more complicated.

Estimate: 3 days

We should try and move register_pluggable_behaviour into environment and out of routing as it is causing import issues when using simple search.

These functions already take a map and so they should be fine any time after the call to make_map

Suggested by Richard Cyganiak in ​http://lists.okfn.org/pipermail/ckan-discuss/2012-January/001977.html

• new relationship type links_to / linked_from
  • Maybe with extra 'count' property?
• creating a new relationship should be post to the relationship register, not the source entity.
• add missing doc about creating a new relationship
• 'Model Formats' section seems to be missed out of the docs since release 1.4.2. See: ​http://readthedocs.org/docs/ckan/en/ckan-1.4.2/api/version2.html#model-formats

Add drupal-like "taxonomies" to CKAN.

Etherpad with user stories, feature list, design and implementation discussion:

​http://ckan.okfnpad.org/22

Branch where this is being developed is feature-1698-tag-taxonomies:

​https://github.com/okfn/ckan/compare/master...feature-1698-tag-taxonomies

Tickets related to this have ​keyword taxonomies.

Document deployment on tenforce test server.