CKAN: Ticket Query

#2477: Add extensions section to readthedocs

shevski — Wed, 30 May 2012 15:54:04 GMT

Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to a dedicated section of docs.ckan.org

#2341: Fix Jenkins issue when testing branches that contain model changes

seanh — Tue, 01 May 2012 12:46:36 GMT

#2342: Fix Jenkins issue when testing branches with different solr schema versions

seanh — Tue, 01 May 2012 12:47:25 GMT

#2466: Fix spam handling on trac

seanh — Mon, 28 May 2012 13:51:43 GMT

#2478: Get NeverNotifyUpdaterPlugin installed on trac.ckan.org

seanh — Wed, 30 May 2012 16:52:14 GMT

#2362: Improve plugin documentaion, including examples.

kindly — Tue, 01 May 2012 16:13:11 GMT

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

#2430: Look into marking tickets as dependent on other tickets in trac

seanh — Fri, 25 May 2012 11:58:06 GMT

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

seanh — Wed, 13 Jun 2012 09:07:22 GMT

there's a setting for this

#1785: Replace 'Revisions' page with site-wide activity stream

seanh — Tue, 07 Feb 2012 16:34:18 GMT

Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?

#2431: Trac git integration

seanh — Fri, 25 May 2012 11:58:48 GMT

#2475: Upgrade trac.ckan.org to trac 0.12.3

seanh — Tue, 29 May 2012 18:46:16 GMT

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog

#2370: Add examples of all config settings in default config file

seanh — Thu, 03 May 2012 10:50:37 GMT

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

#2203: Data Viewer support for PDF

rgrp — Sun, 04 Mar 2012 11:24:53 GMT

Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf
propose we use pdf.js - https://github.com/mozilla/pdf.js
We could also try iframing for browsers with native support but this seems more complex

#2520: Document undocumented config options

seanh — Wed, 13 Jun 2012 11:52:48 GMT

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

#2334: Improved data import

rgrp — Sat, 28 Apr 2012 22:21:02 GMT

Placeholder for thinking about this ...

Allow users to configure info for CSV import (e.g. separators, quotes etc)

#2405: Improvements to data viewer embed

rgrp — Mon, 21 May 2012 06:59:25 GMT

Decent powered by icon for the Embed for the DataHub?? (make this text customizable and allow html ...)
- Current icon and text looks really bad
Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
Data viewer embed url includes all the resource info (not needed -- just url and backend info ...)
Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org

#2473: Make datstorer store field ordering in _meta field

kindly — Tue, 29 May 2012 12:11:32 GMT

Recline views should have a default table order, being the same as the csv that was imported.

#2619: Omit private datasets from public activity streams

seanh — Thu, 28 Jun 2012 11:49:00 GMT

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

#250: RDF link in Atom feed

dread — Thu, 18 Feb 2010 15:41:35 GMT

Add link to RDF representation of a package in our Atom feed.

#235: Resource format normalization and detection

dread — Mon, 18 Jan 2010 15:13:24 GMT

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
Collect and map surface forms from all CKANs
Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
Add heuristics for format auto-detections:
- Map well-known file extensions
- Recognize obvious magic (Zip, Tar)
- Peek into Zipfile/Tarfiles?
Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
See also: #816 (Autocomplete for the resource format field)

#2337: Sort out ini file confusion in docs

seanh — Mon, 30 Apr 2012 14:57:56 GMT

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

#2294: Unauthorized action on Data API gets 302 rather than 401 or similar

rgrp — Sat, 14 Apr 2012 19:35:04 GMT

Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients).

Fix this by ensuring that error messsages on API get passed through correctly.

Minor because affect is annoying but not terminal

#2335: Unicode characters don't work in CKAN ini files

seanh — Mon, 30 Apr 2012 12:42:06 GMT

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

#2736: Archiver fixes

ross — Tue, 24 Jul 2012 09:13:57 GMT

We need to check the archiver to make sure it works as we would expect after the changes in #2732:

Receive notification of a file that needs processing
Check the file's DB entry and upload the file to the configured remote service
Update the URL of the file (both in the file's DB entry and the resource)
Once we're sure the file is safely archived, we should mark the file as being archived

so that we know it can safely be deleted at some point.

Maybe this could/should be implemented as a service rather than a celery task?

#2519: Dataproxy (converters) enhancement

ross — Wed, 13 Jun 2012 10:55:55 GMT

Whilst the error messages returned from the dataproxy are informative, they aren't particularly human readable. It isn't clear to the user whether they should retry, or whether the format of their data is causing the problem etc...

Would be lovely if the error messages returned contained a human readable version of the error.

#2577: Dataproxy raises 500 when it can't get CSV

ross — Wed, 20 Jun 2012 15:45:24 GMT

When given a url parameter for a file it cannot fetch, the app raises a 500 instead of something useful.

Better error reporting in the JSON would be useful

#2575: Licenses have i18n issues (was Inconsistent language on license dropdown)

amercader — Tue, 19 Jun 2012 17:10:45 GMT

License titles at least for some of them need translating. This wants to be done in a more efficient and simple manner than currently exists.

the original problem was:

The license dropdown on the dataset form shows titles in Swedish by default. Changing the language seems to default to English, but after refreshing the page a couple of times it shows a wrong language again.

Thanks to pudo for reporting.

#2629: Move ckanext-examplevocabs into master and document

seanh — Fri, 06 Jul 2012 14:55:55 GMT

#2668: Permission for sysadmin to read a deleted publisher

dread — Fri, 13 Jul 2012 09:24:13 GMT

As a sysadmin I should be able to view deleted groups/publishers (publisher profile) but I can't.

In addition, there don't seem to be tests for permissions of deleted publishers.

#2632: check allowed helpers exist

toby — Mon, 09 Jul 2012 09:56:51 GMT

in lib helpers make sure the allowed function lists functions exist or throw an error.

#2536: cleanup activity streams

toby — Fri, 15 Jun 2012 13:02:03 GMT

#2371: fanstatic integration

toby — Thu, 03 May 2012 12:00:11 GMT

aims

allow templates to specify needed css/js

allow split js files to be bundled (including plugin supplied if possible - stage 2)

serve minified in production/ non-minified in dev

possibly look at doing the minification via paster - maybe as different ticket

#2671: fix allow_partial_update

toby — Fri, 13 Jul 2012 12:46:58 GMT

need to be able to delete stuff explicitly

#2407: 'Access denied' message is unhelpful

markw — Mon, 21 May 2012 15:01:40 GMT

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

#2773: About page needs improving

markw — Mon, 30 Jul 2012 11:13:15 GMT

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

#2625: Add i18n strings from non-core but supported extensions to ckan.pot file

seanh — Mon, 02 Jul 2012 13:48:23 GMT

Have to decide which non-core extensions are going to be supported first.

#2298: Add sort-by controls to the search results page

icmurray — Mon, 16 Apr 2012 09:39:22 GMT

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

What should the sortBy controls/widget look like?
Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

#2829: Archiver fails on 403 http response

toby — Thu, 09 Aug 2012 10:34:44 GMT

Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

$ paster archiver update -c ../ckan/development.ini
2012-08-09 11:01:37,636 INFO  [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources)
2012-08-09 11:01:37,671 INFO  [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13
2012-08-09 11:01:37,900 INFO  [ckan.lib.base]  /api/action/package_show render time 0.043 seconds
Traceback (most recent call last):
  File "/home/toby/okfn/pyenv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command
    response = app.post(api_url + '/package_show', data)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post
    expect_errors=expect_errors)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request
    return self.do_request(req, status=status)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request
    self._check_status(status, res)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status
    res.body))
paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show)
{"help": "Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n\n    :rtype: dictionary\n\n    ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}}

#2480: Better message when dataset has no resources

markw — Thu, 31 May 2012 08:31:33 GMT

If a dataset has no resources the resources list currently says '(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'

#2679: Change default behaviour of TemplateController.view to 404.

icmurray — Mon, 16 Jul 2012 10:55:33 GMT

The current behaviour of TemplateController?.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want. In general, it leads to:

500s when attempting to access a normal template (eg - http://datahub.io/importer/preview)
A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template).

Solution:

Change the controller to 404
Ensure there's a way for existing ckan instances to override that behaviour should they need it.

#2415: Confusing message on attempting to register new account with upper case

markw — Wed, 23 May 2012 13:19:49 GMT

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

#2838: Context variables accepted by action functions need to be documented

seanh — Fri, 10 Aug 2012 20:35:11 GMT

I was doing this:

context = {'model': base.model, 'session': base.model.Session,
                    'user': toolkit.c.user or toolkit.c.author,
                    'extras_as_string': True}
group_dict = logic.get_action('group_show')(context,
                    {'id': group_id})

in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass 'extras_as_string': True in the context. I don't think this or other context variables are documented anywhere.

#2529: DataHub (or CKAN) widgets

rgrp — Fri, 15 Jun 2012 08:47:29 GMT

Simple widgets in pure JS. For example:

Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
Embeddable list of top X (5) datasets for a given query
Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. http://okfnlabs.org/ckanjs/

#2530: DataHub purge fails on some revisions

rgrp — Fri, 15 Jun 2012 08:54:58 GMT

See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like:

Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError?) update or delete on table "revision" violates foreign key constraint "group_extra_revision_revision_id_fkey" on table "group_extra_revision" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table "group_extra_revision". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}

#2291: DataStorer incorrectly attempts to store JSON in DataStore

rgrp — Sat, 14 Apr 2012 19:19:07 GMT

See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

#1745: Dataset search UX improvements as of Jan 2012

rgrp — Thu, 02 Feb 2012 23:22:21 GMT

Changes to make search both more exploratory and more satisfying to use

Search query build - #1603
- Ability to add new facet fields "live"
- That is add fields which then contain faceted options (a bit like data.hri.fi)
(??) Autocomplete / drop down on search (i.e. search while you type)
- Dubious about value / cost ratio here

Probably would involve to pure JS and HTML implementation.

Implementation

Probably require

API changes to expose solr style API directly #1737

#2733: Datastore logic functions

johnglover — Mon, 23 Jul 2012 15:54:46 GMT

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

datastore_create
datastore_delete
datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}

The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
rows: can be empty so that you can just set the fields
fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

Any error results in total failure!! For now pass back the actual error.
Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

http://www.postgresql.org/docs/9.1/static/datatype.html

http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

#2582: Do not hide notes / readme on dataset pages

rgrp — Thu, 21 Jun 2012 20:59:00 GMT

Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?)

Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket.

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

nils.toedtmann — Fri, 01 Jun 2012 11:47:09 GMT

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

research ways to limit requests-per-process with the different web servers (see below for a start);
explain these safeguards in the CKAN documentation and encourage users to apply them;
consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

Apache:
- Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
- (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
nginx/supervisord: to be researched.

#2406: Errors in source install doc

markw — Mon, 21 May 2012 10:11:33 GMT

There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

#2585: Escape solr control characters in search queries, add advanced search screen

seanh — Fri, 22 Jun 2012 10:26:13 GMT

Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds".

So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?

#2426: Get rid of CKAN's formalchemy dependency

seanh — Fri, 25 May 2012 11:47:42 GMT

#2350: Hooks in package controller for validating form data

icmurray — Tue, 01 May 2012 14:16:01 GMT

When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController? subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

[ ] Add hooks
[ ] Use in ckanext-example

#2333: Introduce new resource type datastore

rgrp — Sat, 28 Apr 2012 22:16:00 GMT

This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly:

New Option: DataStore? Table in add a resource form
URL field is disabled (or points at /api/data or something rather pointless)
Download button is disabled
- Better would be to find a way to serialize an ES table or stream it ...

#2320: Introduce new state: archived

rgrp — Mon, 23 Apr 2012 20:12:29 GMT

Some datasets become obsolete: e.g. http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.

#2885: Labels (editor, admin) not translated in authorization pages for datasets or groups

seanh — Wed, 22 Aug 2012 08:49:11 GMT

#2324: Link API docs to relevant version not just latest

amercader — Wed, 25 Apr 2012 10:58:52 GMT

The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading

#2328: Link to Catalog API for each Dataset, Group and Tag

rgrp — Thu, 26 Apr 2012 12:15:01 GMT

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

#2479: Meanings of Author and Maintainer fields are unclear

markw — Thu, 31 May 2012 08:28:18 GMT

CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank.

The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest:

(1) Rename the fields 'Owner' and 'Maintainer'.

(2) Change the explanatory text for the relevant fields:

Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.'

(3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile.

#2483: Non-local resources should not have Download links

markw — Thu, 31 May 2012 12:52:14 GMT

At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
Disable the Download button unless the resource is stored locally.

#2745: Password reset returns an exception if the key parameter is missing

amercader — Tue, 24 Jul 2012 16:05:31 GMT

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

seanh — Fri, 25 May 2012 11:48:27 GMT

#2603: Remove deprecated 'fields' parameter from resource_search

icmurray — Tue, 26 Jun 2012 17:10:01 GMT

The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result.

#2621: Remove the deprecated 'fields' parameter from tag_search and tag_autocomplete

icmurray — Thu, 28 Jun 2012 16:22:49 GMT

This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict. See #2439

In a future release of CKAN (probably 1.9) it can be removed.

Internal uses of it were removed in #2439, but there are tests that still use it.

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#2889: Support JSON values in extras, when returning JSON from API

seanh — Fri, 24 Aug 2012 12:35:37 GMT

For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects.

#2321: Use resource names/titles in URLs

rgrp — Tue, 24 Apr 2012 03:00:12 GMT

Would make them much nicer but requires:

Ensure uniqueness (or deal with ambiguity)
(?) Slug/names versus titles
- Aside: what is currently called name should probably be title

#2572: clean up stats plugin

toby — Tue, 19 Jun 2012 11:24:14 GMT

attempt to disengage the stats plugin from core as much as possible

#2257: cleanup template vars

toby — Fri, 23 Mar 2012 14:35:07 GMT

look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc

#2543: facet.sort is not available in the package_search action

icmurray — Mon, 18 Jun 2012 09:55:35 GMT

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

#2758: file storage gives error if config not available but no useful user information

toby — Thu, 26 Jul 2012 07:28:29 GMT

We get an error which should be improved the actual problem is this but not passed to user

⇝ KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

#2663: h.resource_display_name needs love

toby — Wed, 11 Jul 2012 14:42:26 GMT

This function is shit and needs cleaning up and a doc string

description is markdown and should be treated properly

either we should truncate all or leave it to the templates but work universally

url if no name / desc this is in demo-theme branch

#2708: limit extra data for package/group show

toby — Wed, 18 Jul 2012 14:40:20 GMT

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

#2579: move sort_by functions into lib.helpers

toby — Thu, 21 Jun 2012 12:29:58 GMT

make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc

#2243: Fix ckanext-example

seanh — Mon, 19 Mar 2012 15:58:30 GMT