CKAN: Ticket Query

#819: Tag autocompletion widget broken

cygri — Wed, 17 Nov 2010 14:28:41 GMT

The widget for tag autocompletion is broken in various ways.

For example, if I edit a package that is tagged "music", and just tab through the form fields to get to the Author field, then the widget changes to "industrial-music" as I tab through it.

Or if the tag is "foo bar" and I hit alt-right to jump to the end of the text field (with the intent of adding a third tag), then the contents change to "foo barbecue".

#925: Change the search box icon to remove the down arrow

dread — Mon, 24 Jan 2011 11:13:13 GMT

Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed?

#1004: Group creation instructions missing

dread — Thu, 24 Feb 2011 14:52:24 GMT

Need instructions on the group page to tell people they need to login to create a group. Someone must have deleted this.

#1139: Create CKAN Theme Gallery

lucychambers — Tue, 17 May 2011 15:24:19 GMT

Take screenshots of existing ckan instances esp those mentioned <http://wiki.ckan.net/Theming> and put on flickr in ckan or ckan-theme group so we can create a gallery ... (both to illustrate theming but also to show ckan instances that are around -- could add to http://wiki.ckan.net/Instances)

#1141: [super] Moderated Edits User Interface

johnglover — Wed, 18 May 2011 12:16:43 GMT

Proposer: John Glover
Seconder: James Gardner

Abstract

We are trying to achieve these goals:

To get people involved with making edits to CKAN metadata.
To have an ownership model as to who can moderate and validate these changes
To not put too huge a burden on these owners.

This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made "official".

There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface.

The Problem

We require the following functionality:

Allow a group of changes to be stored as a new revision.
Allow a linear stack of "community" revisions.
Provide a way for the editor and moderator to compare previous revisions to the current one.
When a moderator approves a change it creates a new revision flagged "moderated" (this is analogous to a merge commit)
Provide a way for the editor and moderator comment on revisions if necessary.

Extra features:

Need a way to summarise the changes (as part of the preview perhaps)
Sysadmin needs to purge a revision completely

Specification

UI/UX

UI Mockup:

Revisions:

Revisions are per package rather than per field.
Internally CKAN has separate revisions for resources, extras and package metadata. From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision.

On the Edit page:

We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?).
The values displayed in the form are by default populated from the latest revision (whether community or moderated)
Under each field is a "shadow", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too.
When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it
If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named "Replace fields with values from this revision" under the revision list. You click this, a warning pops up and then you say "Yes". You then select the moderated revision again.
We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this.
In addition to package comments, each revision will have a revision log (analogous to a commit message).

Technical Details

This CREP will result in a new CKAN extension.
It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised.
This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use.
We will then use suitable test data to mimic these API calls until CREP0002 is ready.

Why do it this way

This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX.

Implementation plan

Deliverables

A new CKAN extension, consisting of:

Code: Python, HTML, CSS, Javascript
Unit tests
Localization
Documentation

Participants

John Glover to do it.

Progress

John has implemented the bulk of this UI. Just some things to tidy up before it is complete:

Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates
history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API)

I've split these two off into a new ticket #1604.

Related Progress

The Todo extension is written and available at: https://bitbucket.org/johnglover/ckanext-todo.

In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done.

#1153: Update CKAN wiki front page

lucychambers — Mon, 23 May 2011 12:53:31 GMT

Update CKAN wiki front page - a la OpenSpending?: http://wiki.openspending.org/Main_Page

Sections should relate to different types of people using the site:

Developers, Users etc..

#1251: Rename wiki.ckan.net to wiki.ckan.org

rgrp — Fri, 29 Jul 2011 18:45:47 GMT

Inline with general naming policy going forward.

#1280: fix sqlalchemy so that it works with postgres 9.0

kindly — Tue, 16 Aug 2011 16:44:50 GMT

After the ckan migrate process, sqlalcehmy reflect does not work and causes an error when reflecting indexes. A bug report needs to be put into sqlalchemy.

#1283: Deleted packages shouldn't be searchable or browsable

dread — Tue, 23 Aug 2011 15:14:41 GMT

A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else.

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076.

#1290: Better error when blank database

dread — Thu, 25 Aug 2011 09:24:15 GMT

When installing CKAN, when doing "paster serve development.ini", lots of users encounter the error for every request:

ProgrammingError: (ProgrammingError) relation "user" does not exist

This is because the database tables have not been created - they have forgotten or missed the "paster db init" step.

Can we provide a better error to say that the database is not initialised yet?

Implementation options

At the start of every request we reflect the database tables and check they are there. This is rather expensive!

Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it.

#1305: SMTP config for thedatahub.org and IATI

amercader — Fri, 02 Sep 2011 09:44:17 GMT

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},)

As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org

#1339: Issues / question re navl and data conversion

rgrp — Tue, 13 Sep 2011 20:50:22 GMT

I ran into a bug with the size field on resources.

It would not accept an empty value from form (IMO this clearly equates to null/None)
This could be fixed via using ignore_empty instead of ignore_missing
However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
similar issues could arise around other fields (such as last_modified ...)
- cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works.

#1388: etags caching on home page

dread — Wed, 12 Oct 2011 14:24:03 GMT

Needs to update on:

language - It's broken on thedatahub.org if you change to German and then click on the CKAN icon top-left to go to thedatahub.org again.

AND

whether signed in or not - #1373

BUT not latest revisions (which is what it was)

Or get rid of etag caching on this page and others?

#1413: Ask users to 'add email address' when logged in

shevski — Mon, 24 Oct 2011 10:45:08 GMT

I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates

Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard

DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation

#1528: Gravatar next to My Account link on all pages

zephod — Tue, 06 Dec 2011 18:58:51 GMT

Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in.

This was suggested by Richard Cyganiak: http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html

#1551: CKAN auth for webstore changes

ross — Fri, 16 Dec 2011 15:39:26 GMT

Webstore should use auth api ( #1550 ) for authenticating users accessing webstore rather than talking directly to the CKAN database. We also need it to suppose /user/ urls and /userid/ for accessing databases.

#1579: Full text search in Webstore

ross — Tue, 20 Dec 2011 12:54:58 GMT

We should allow full text search within databases for individual databases, although we may want to wait for any pending ports to postgres.

#1648: Clarify that additional info = extra fields + add guidance

shevski — Mon, 16 Jan 2012 00:47:23 GMT

Super ticket: #1506

Need to decide which term to use and then have the same for editing as well as viewing a dataset.

In creating/editing a dataset, want more explanation about adding extra fields (probably as a tooltip or similar).. i.e. that this let's you add extra custom metadata such as 'location: uk' which is then searchable etc

#1801: No links to password reset

dread — Thu, 16 Feb 2012 15:14:50 GMT

You can reset your password (#1186) but you have to know the URI (/user/reset) - there is no link to this page!

#1812: Restrict editing rights/permissions based on publisher

johnglover — Mon, 20 Feb 2012 14:24:49 GMT

see how this currently works with DGU first

#2323: Auto-complete in organizations

ross — Tue, 24 Apr 2012 13:32:34 GMT

Apparently auto-complete in Organizations may not work. Check and fix.

See https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5

#129: Secure db access by channelling query generation through authz module

dread — Fri, 25 Sep 2009 13:42:16 GMT

Controllers and templates should not access db objects directly - they should do all access via authz module giving username. They are handed by a query that has already been filtered by the packages they are authorized to read.

(Additional idea to be discussed: When they request a package object, they are handed an copy of the db object - disconnected from the database - so it the db object can't be changed.)

A couple of tests can be reenabled when this is done: ckan.tests.functional.test_authz.TestUsage?.test_admin_list_deleted ckan.tests.functional.test_authz.TestUsage?.test_search_deleted

#952: CKAN should run under nginx/uswgi

pudo — Thu, 03 Feb 2011 10:54:58 GMT

second part of #908

#1132: test_authz doesn't run

johnlawrenceaspden — Mon, 09 May 2011 18:48:43 GMT

Trying to run the tests in test_authz.py with

$ nosetests --ckan ckan/tests/functional/test_authz.py results in no tests being run:

Ran 0 tests in 0.840s

OK (SKIP=3)

#1330: Deprecate / Remove test_authz.py

rgrp — Tue, 13 Sep 2011 07:32:09 GMT

test_authz.py appears to test in great detail some very specific additional authz (related to total site lock-down it seems -- introduced I think for hri project).

I think there are simpler ways to get total site lockdown (use external auth!) and this test is slow and delicate (e.g. depends on specific words in templates). Suggest removing. If we don't remove we should at least refactor tests for access to certain pages to use a proper method of testing (e.g. agreed html comments in each page) rather than being depending on the presence of absence of specific wording.

#363: Blank revisions

dread — Mon, 12 Jul 2010 15:16:12 GMT

Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column.

#482: API Rate Limiting

johnbywater — Sat, 21 Aug 2010 20:32:36 GMT

#1197: Add JavaScript guide for CKAN

timmcnamara — Thu, 23 Jun 2011 03:50:32 GMT

A new library, guiders.js, has been open sourced that seems to be great at unobtrusively introducing new users to features of websites.

The library drives Optimizely's website. The GitHub repo is here: https://github.com/jeff-optimizely/Guiders-JS

Some examples:

#1231: [super] Management Information Reporting

thejimmyg — Wed, 20 Jul 2011 14:58:39 GMT

Child tickets:

#1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

#1269: User view page shows the packages followed by viewing user not user being viewed

rgrp — Tue, 09 Aug 2011 16:16:38 GMT

This overlaps with Sean's follow support (for activity streams) and so will be dealt with there.

#1289: Remove 'relationships'

dread — Wed, 24 Aug 2011 17:21:42 GMT

Abstract

Package Relationships have not taken off in the 18 months we've had them in the API. There are some issues with them and we need to spend more time improving them or consider getting rid of them.

The Problem

Original use cases are expressed here: #253 Here are comments about how we could handle these specific examples better:

groups of packages - maybe better with a custom tag?
fragment resources - soon to be covered by 'kind' resource field #957

3&5. derived resource - better to have some sort of resource relationship perhaps?

linked resource - again better to have some sort of resource relationship perhaps?

Outstanding issues needing serious effort to fix:

#256 Editing them in Web UI (not done yet)
#1288 Package edit/creation can't include 'relationships' field

Specification

Remove relationships from model, API, tests, Web UI. Data migration to remove from db.

Why do it this way

Getting frustrated having problems with the code, when it's not used much. Often asked about what it's for, but rarely used. Seems an overly complicated design.

Backwards Compatibility

n/a

Implementation plan

Deliverables

See Specification

Risks and mitigations

Risk: a customer suddenly wants this, and the new ways to relate resources are not in place yet.

Mitigation: discuss this decision thoroughly to make sure we are confident the use cases are not important. Discuss with team, ckan-discuss and specifically the LOD people who have some related packages on thedatahub.org.

Participants

David Read

Progress

Not yet.

#1306: Tests for the logic layer

rgrp — Sun, 04 Sep 2011 16:21:07 GMT

AFAICT there are no tests for the logic layer at the moment. I imagine this is an issue (if it is not please explain and close as wontfix).

#1533: Make sure ckanext-wordpresser works on CKAN 1.5.1

amercader — Wed, 07 Dec 2011 17:07:40 GMT

Superceded by #2234

Super ticket: #1508

Etherpad: http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

#1549: [super] Short link tool

ross — Fri, 16 Dec 2011 12:03:18 GMT

It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself.

Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric.

Questions:

Core, or Extension, or Self-hosted?

#1685: Move archiver download function to core

ross — Thu, 19 Jan 2012 14:42:50 GMT

Archiver is included in some places just for access to download(), we should move this to lib in core.

#1746: Activity streams pagination

seanh — Sun, 05 Feb 2012 12:54:48 GMT

Currently user, package and group activity streams only return the most recent 15 activities, even though all activities are kept in the db. Do we want to add pagination - to both the API and the HTML pages - to support retrieving older activities?

#1810: Store list of Eurovoc tag names and IDs

johnglover — Mon, 20 Feb 2012 14:03:32 GMT

Tag list not be added to CKAN core, what is the best way to do this in an extension?
Does the list need to be updated via API/WUI?

#1811: Translate Eurovoc tags from ID to dataset language

johnglover — Mon, 20 Feb 2012 14:21:24 GMT

#691: Package Relationships

johnbywater — Mon, 11 Oct 2010 18:45:35 GMT

#1780: Api to add translations.

kindly — Mon, 06 Feb 2012 17:21:48 GMT

Add api to translation to the term_translation table.

add to logic 2d
test 2d

CKAN: Ticket Query

#819: Tag autocompletion widget broken

#925: Change the search box icon to remove the down arrow

#1004: Group creation instructions missing

#1139: Create CKAN Theme Gallery

#1141: [super] Moderated Edits User Interface

Abstract

The Problem

Specification

UI/UX

Technical Details

Why do it this way

Implementation plan

Deliverables

Participants

Progress

Related Progress

See also

#1153: Update CKAN wiki front page

#1251: Rename wiki.ckan.net to wiki.ckan.org

#1280: fix sqlalchemy so that it works with postgres 9.0

#1283: Deleted packages shouldn't be searchable or browsable

#1290: Better error when blank database

Implementation options

#1305: SMTP config for thedatahub.org and IATI

#1339: Issues / question re navl and data conversion

#1388: etags caching on home page

#1413: Ask users to 'add email address' when logged in

#1528: Gravatar next to My Account link on all pages

#1551: CKAN auth for webstore changes

#1579: Full text search in Webstore

#1648: Clarify that additional info = extra fields + add guidance

#1801: No links to password reset

#1812: Restrict editing rights/permissions based on publisher

#2323: Auto-complete in organizations

#129: Secure db access by channelling query generation through authz module

#952: CKAN should run under nginx/uswgi

#1132: test_authz doesn't run

#1330: Deprecate / Remove test_authz.py

#363: Blank revisions

#482: API Rate Limiting

#1197: Add JavaScript guide for CKAN

#1231: [super] Management Information Reporting

#1269: User view page shows the packages followed by viewing user not user being viewed

#1289: Remove 'relationships'

Abstract

The Problem

Specification

Why do it this way

Backwards Compatibility

Implementation plan

Deliverables

Risks and mitigations

Participants

Progress

#1306: Tests for the logic layer

#1533: Make sure ckanext-wordpresser works on CKAN 1.5.1

#1549: [super] Short link tool

#1685: Move archiver download function to core

#1746: Activity streams pagination

#1810: Store list of Eurovoc tag names and IDs

#1811: Translate Eurovoc tags from ID to dataset language

#691: Package Relationships

#1780: Api to add translations.