{22} Trac tickets (2647 matches)

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: ​http://docs.python.org/dev/howto/doanddont.html

• Lists of entities should be full URLs, rather than just the names
• Discoverability - /api/v3/rest should list the entity types that can be listed

This could be v3 of the RESTful interface.

lxml is used in CKAN core in two places:

• ckan/lib/helpers.py:239 Parsing Markdown and ensuring it is valid XML (i.e. tags close nicely so the HTML of the rest of the CKAN page is not disruptable).
• reading the SOLR config XML to determine what version it is.

Both of these could be done with xml.dom built into Python and would save us a sizeable and slightly problematic dependency.

lxml is needed by these extensions: ckanext-inspire, ckanext-harvest, ckanext-csw & ckanext-wordpresser, so we'd have to add this to their requirements.

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar.

Just like we display download counts we should display view counts.

• cache admins etc
• aim to move check_access so independent of logic function
• general spruce up

When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar.

SOLR has a feature for finding documents similar to a particular document, called 'More Like This'.

We would like this for DGU.

Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support.

For example, on the user index page right now there are time period strings like "less than 1 month" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly).

ISO date format everywhere might be a good idea.

Need to consider local timezone issues.

Python standard library and Babel may have useful helpers for this.

Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?

DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how.

Start a Java library for interacting with the CKAN Action API.

Currently requires support for Groups, Tags, Resources, and Search.

An example app would be very useful.

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

The .ini file needs cleaning up in assorted ways which Toby will expand upon.

Requirements

To provide a freemium service it is necessary to be able to provide differing levels of functionality based on the type of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks.

Initial implementation should focus on limiting access to datastore disk space.

Interface

These changes are currently only for the data hub and should be kept as much as possible within the data hub extension.

User Stories

• As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited).

• As a system administrator I don't expect to need to manage the levels of users or the features that this applies to.

Tasks

[ ] Clarification of requirements/analysis

[ ] Tests

[ ] Code

[ ] Model

[ ] API

[ ] UI

[ ] Documentation

Estimates

Get ckan working on heroku

Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

• 'ResourceExtras?' for extra metadata on resources
• Makes 'Resource' into first class object (and make available in WUI)?

Depends

• #877 (Upload in WUI) - should be part of that extension

For example:

• name and title should have a higher weight than the notes field

Add tables described in ​http://ckan.okfnpad.org/notifications

Sub ticket of #1294

• Initial screen just title, name, description (notes), license (and tags?).
• State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff.
  • Flash some instructions? e.g. that resources are missing?
• Name slug should by default look uneditable (cf wordpress way to do the slug)

Subticket of #1294.

Can integrate from working system on ckanjs.

Only happens if the user does not update the password. Most probably introduced during #1229.

• Major issue with spam signup on thedatahub.org
• Suggest using recaptcha

Est cost: 1d

e.g. this particular revision on ckan.net: ​http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error.

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

• 'with' statement requires from __future__ import with_statement
• import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.

Resource and Tag search is currently still using the old ckan query parser which is unnecessary. Refactor this out and move to the logic layer.

Various things in templates can be tidied up after work has finished on #1294:

• remove styling in page in group/index.html (put in css main)
  • ditto for package/search.html
  • ditto for home/index.html
• Forms.css - to merge into style.css; remove explicit imports elsewhere
  • @import at the top of style.css @import url(forms.css)
• Make a note: Success/failure controllers clash CSS with success/failure flash banners

The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

• Fix the duplicate form
• Strip the preview functionality from the front-end, replacing it with js editor
• Strip the preview functionality from the back-end and test

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?

It is very annoying that you have to wrap links in <...> and most people don't bother. Extend h.markdown in code and markdown in js to fix this.

(We should also support 'package' links etc in javascript?)

*ADDITIONAL*

• Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
• Move the markdown api to /util/markdown

Json decoding error on search, due to date in resources.

We have an all new and much improved data preview system developed as part of QA in July. Need to integrate this.

• Is pure javascript
• Will have this in CKAN core but considered to be a plugin (so need datapreview in ckan.plugins list to work -- will enable this by default)
• Will depend on the ​http://wiki.ckan.org/DataExplorer

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id

ckan.js can be rearchitected to avoid instantiating showdown.js in the root namespace. That way, layout_base.html doesn't need to import the script (because we don't use the showdown-dependant parts of ckan.js).

When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface

​https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64

At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package.

One possible solution would be to set the package id in the data dict received by package_create (if it's not defined):

from ckan.model.types import make_uuid
if not 'id' in data or not data['id']:
   data['id'] = make_uuid()

Integrating ckanext-admin into core has thrown up a number of problems:

• Look & feel does not match the rest of the site
• Tests are not passing
• On the trash page, clicking 'undelete' triggers a purge
• Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org)
• Trash page can contain nested form tags in certain cases (breaking test harness & form redirection)

Superticket: #1343

This avoids difficulties when they forget their password.

• (?) Email validation (navl does not have one!)
• Require email confirmation to be activated (?) (Answer: no, not at present -- see discussion below)

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

• [dataset/edit]: Deletions are confirmed with a dialog box
• [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
• [dataset/edit]: Rearrange the delete resources mechanism;
  • Use a little minus icon on the far left, followed by a pencil icon for editing.
  • The 'delete resource' button can become 'hide'
  • Deletions are confirmed with a dialog box
• [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
  • Delete the little green plus sign on the right;
  • Try a flash banner instead (more general purpose & beautiful)
• [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
• [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
• [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
• [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
• [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
• [groups/add]: Hide unimportant fields
• [navigation] Add 3D bevel (@see github) to current tab highlight.
• [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see ​http://groupspaces.com/a/group/
• [groups/view]: Show description and owner. To match dataset section.
• [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
• [groups]: Entire section to use pretty buttons please.
• [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

• [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
• [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
• ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
• [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
• [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
• [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)

e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English.

It is not currently possible to assign or remove groups by adding or removing their group name from the "groups" list on the REST API. This should be allowed, as it is easier than first adding a package and then editing each group it must belong to.

Superticket: #1343

Creation of accounts from OpenID causes large number of problems:

• No guarantee of other profile info
• Poor username (just from openid)

We therefore will:

1. Require creation of user account via register (no auto-creation of accounts via openid)
   • sidebar of register page should point to login page rather than openid signin

2. Permit association of an openid with an account ...
   • How: paste in your openid url (requires change to edit form to have this)
     • won't work for google with their weird openid urls
   • [future] Nicer way is to have login via openid while logged in (which association then happening in background)

3. Migration (for 1+2):
   • (?) Generate decent usernames for all existing accounts
   • As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution.
     • This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See #1534

4. [optional] Change display_name to always be username
   • Change My Account to Username at top right of all pages (once we know usernames are short ...)
     • could do this straight away by truncating long usernames (e.g. truncate at 20 chars ...)
     • Also should we lose the icon?

Superticket: #1397

Generate archiving request on resource url change by implementing IResourceUrlChange and sending tasks to celery. [0.25d][0.25d]

Will be part of of ckanext-archiver extension.

Process:

1. A resource is added to CKAN
2. IResourceCreate event generated
3. IF: resource url points to ckan storage or falls within some other set of exclusion conditions then END else continue
4. Generate a Archiver.Update task with resource.id

In Firefox:

• Edit | Preferences | Content | Languages - put 'de' at top language
• Clear cookies
• Load thedatahub.org
• it should be in German, but it's in the default language - English.

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

​http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

• Validation of urls as part of form entry or data loading
  • Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
• This code should support analysis of existing data so we can go through existing database and find invalid urls
  • Also useful to have this so we can do out of band validation

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

• Testing for dataproxy
  • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
• Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
• Google docs style row/column selections
• Use the swiss library - ​https://bitbucket.org/okfn/swiss
  • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
• Handle redirects for content-length?
• Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

• Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

1. Visit ​http://ckan.net/package/kivele2010
2. Click on [preview] for any of the resources

The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems.

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.