CKAN: Ticket Query

#1562: Finish Geo Spatial

jilly mathews — Mon, 19 Dec 2011 10:56:33 GMT

Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients.

#1557: Complete Webstore Preview Extension

jilly mathews — Mon, 19 Dec 2011 10:40:53 GMT

Finish any work out standing on web store preview extension to be able to package and release.

Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted.

#1558: Publisher Tools

jilly mathews — Mon, 19 Dec 2011 10:46:13 GMT

Summarise final set of requirements for this and finish development and test. Estimated 10 working days.

#1560: Follow extension

jilly mathews — Mon, 19 Dec 2011 10:51:19 GMT

Estimate 2 days to finish dev and test.

David can you add any info needed here?

#1561: To do extension

jilly mathews — Mon, 19 Dec 2011 10:52:52 GMT

Can we finish this ready for release on data hub and CKAN Hosted.

#1564: Structured Data (Data API)

jilly mathews — Mon, 19 Dec 2011 11:07:14 GMT

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.

#1567: Finish QA extension

jilly mathews — Mon, 19 Dec 2011 11:19:59 GMT

Requires change to celeryd. Estimated 4 weeks.

#1569: Wordpressser

jilly mathews — Mon, 19 Dec 2011 11:27:36 GMT

How much effort will this be to be ready to use?

#1572: Meta data Harvester

jilly mathews — Mon, 19 Dec 2011 11:35:09 GMT

Need to write custom harvesters for each client. Is it worth having one for data hub?

#1573: Apps and Ideas

jilly mathews — Mon, 19 Dec 2011 11:36:33 GMT

Estimate 2 weeks for someone to finish and test.

#1565: Admin dashboard finished?

jilly mathews — Mon, 19 Dec 2011 11:11:32 GMT

Is testing complete and ready for release?

#2938: Explain bundling in doc/resources.rst

dominik — Mon, 24 Sep 2012 12:13:55 GMT

Since it makes sense to bundle resources into one js file that are commonly used together, there should be some explanations on how separate js scripts are merged into one file for production.

Explanations could include how groups are handled or different resource files.

#1815: Reenable Sparql endpoint on publicdata.eu

amercader — Mon, 20 Feb 2012 17:08:43 GMT

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

#2382: Investigate options for basic geocoding

amercader — Mon, 14 May 2012 17:39:20 GMT

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

#2407: 'Access denied' message is unhelpful

markw — Mon, 21 May 2012 15:01:40 GMT

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#3023: New methods on IPackageController to provide access to the data_dict

amercader — Thu, 22 Nov 2012 17:00:57 GMT

Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it

#3025: Add requests to core requirements

amercader — Tue, 27 Nov 2012 21:35:24 GMT

Because yes please

#2961: Preview plugin endpoint

dominik — Mon, 08 Oct 2012 11:39:27 GMT

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

We can say that you can add your own previews
Every preview would have to define it's own dependencies
External development (community creates new extensions?)

Questions to answer:

Specification of the interface
Precedence of previews for the same data types
Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

#3013: common-error-messages is unreadable

dominik — Sat, 10 Nov 2012 13:18:25 GMT

Since the update of the doc theme, the page became unreadable.

http://docs.ckan.org/en/latest/common-error-messages.html

#250: RDF link in Atom feed

dread — Thu, 18 Feb 2010 15:41:35 GMT

Add link to RDF representation of a package in our Atom feed.

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

nils.toedtmann — Fri, 01 Jun 2012 11:47:09 GMT

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

research ways to limit requests-per-process with the different web servers (see below for a start);
explain these safeguards in the CKAN documentation and encourage users to apply them;
consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

Apache:
- Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
- (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
nginx/supervisord: to be researched.

#2709: Atom feeds are undocumented

markw — Wed, 18 Jul 2012 15:11:50 GMT

There doesn't seem to be any documentation yet for Atom feeds.

#2784: model dictize sensitive data

icmurray — Wed, 01 Aug 2012 09:51:25 GMT

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

#1203: Moderated edits: html code shows as "changed" although it is not

rolf — Tue, 28 Jun 2011 15:29:30 GMT

I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML.

The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &#8212 and in the rendered table that is an —

#2829: Archiver fails on 403 http response

toby — Thu, 09 Aug 2012 10:34:44 GMT

Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

$ paster archiver update -c ../ckan/development.ini
2012-08-09 11:01:37,636 INFO  [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources)
2012-08-09 11:01:37,671 INFO  [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13
2012-08-09 11:01:37,900 INFO  [ckan.lib.base]  /api/action/package_show render time 0.043 seconds
Traceback (most recent call last):
  File "/home/toby/okfn/pyenv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command
    response = app.post(api_url + '/package_show', data)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post
    expect_errors=expect_errors)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request
    return self.do_request(req, status=status)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request
    self._check_status(status, res)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status
    res.body))
paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show)
{"help": "Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n\n    :rtype: dictionary\n\n    ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}}

#2697: create dataset validation

shevski — Wed, 18 Jul 2012 14:00:35 GMT

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

#2917: Organization admins can delete themselves

ross — Wed, 12 Sep 2012 12:10:43 GMT

Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end.

#3016: CKAN 2.0 template tweaks

johnmartin — Tue, 13 Nov 2012 13:30:17 GMT

Just a ticket to keep track of a few suggested template changes.

#3018: Load more in activity streams

johnmartin — Wed, 14 Nov 2012 13:34:11 GMT

Activity streams should be able to load more than 15 items within them. Suggest the default amount of loading is around 30 and then click to load more.

#3021: Logout doesn't work without JS

johnmartin — Tue, 20 Nov 2012 11:19:10 GMT

Essentially, the functionality should be as follows:

Add logout link that has .js-hide attached to it within the header that isn't hidden within a dropdown

See http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS

#979: Edit Resource extras in the API

dread — Fri, 11 Feb 2011 13:09:37 GMT

Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API.

#1077: Move to simpler vdm system

rgrp — Fri, 08 Apr 2011 23:14:24 GMT

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset
for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

Easier support for pending state and similar behaviour
No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

Migration is required
More difficult to query revision history.
- Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
Performance (?)
- Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
  - Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

#1240: [super] API v4

rgrp — Sun, 24 Jul 2011 16:41:00 GMT

(Just creating this ticket as somewhere to keep notes)

Decide on REST api versus action API
- Do we want to support both?
Tidying
- Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
- extras merged into normal field list in package
- Get rid of /rest/ so just have api/v1/package
- Get rid of separation of search api from 'rest' api
  - Propose that GET on REST index is search e.g. /package/?q=...
    - This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
Resource read/write in API (separate from package)
- Does this need authorization work?
user/account API - read/write
Remove autocomplete -- can just use search
- Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

#1255: Drupal consistancy checks.

kindly — Mon, 01 Aug 2011 17:32:48 GMT

Make a robust way to make sure the drupal database is consistent with the ckan data.

#1341: Delete spam users from ckan

kindly — Wed, 14 Sep 2011 10:10:34 GMT

Spam users where added to thedatahub and we need to clean them.

#1652: How we intergrate with Drupal Multiligual?

ross — Mon, 16 Jan 2012 10:31:34 GMT

Analysis of how/where we can integrate with Drupal Multilingual

Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/...

Check with Ian for other issues.

#2291: DataStorer incorrectly attempts to store JSON in DataStore

rgrp — Sat, 14 Apr 2012 19:19:07 GMT

See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

#2334: Improved data import

rgrp — Sat, 28 Apr 2012 22:21:02 GMT

Placeholder for thinking about this ...

Allow users to configure info for CSV import (e.g. separators, quotes etc)

#2406: Errors in source install doc

markw — Mon, 21 May 2012 10:11:33 GMT

There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

#2415: Confusing message on attempting to register new account with upper case

markw — Wed, 23 May 2012 13:19:49 GMT

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

#2426: Get rid of CKAN's formalchemy dependency

seanh — Fri, 25 May 2012 11:47:42 GMT

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

seanh — Fri, 25 May 2012 11:48:27 GMT

#2473: Make datstorer store field ordering in _meta field

kindly — Tue, 29 May 2012 12:11:32 GMT

Recline views should have a default table order, being the same as the csv that was imported.

#2524: If there are no translation files for selected language fall back to default lang.

kindly — Wed, 13 Jun 2012 17:37:28 GMT

If a user selects a language there are no mo files for then an error is raised. Revert to default language instead.

#2548: Object ownership for groups/package

ross — Mon, 18 Jun 2012 11:02:11 GMT

Requirements

We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'.

At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner.

More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature.

Interface

None

User Stories

None

Tasks

[ ] Analysis/Clarification?

[ ] Tests

[ ] Migration

[ ] Code/Schema? changes

[ ] Documentation

Estimates

#2708: limit extra data for package/group show

toby — Wed, 18 Jul 2012 14:40:20 GMT

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

#2729: searching for tags:[tag] works but tag:[tag] doesn't

shevski — Sun, 22 Jul 2012 09:58:29 GMT

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

#2877: Bugs with datastore v2

rgrp — Sat, 18 Aug 2012 00:33:22 GMT

In progress

~~[major] q does not seem to work reliably. e.g. using the setup from this gist https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with "q=de" either)~~
- ~~q=second does work ...~~
[major] q does not work with 2 values (see below)
Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT
- Also weird fact that limit is returned but as as as string - should it not be an integer
Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...)

Multiple query values

Try a query such as: "second UK" and you will get 500 error:

http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK

ProgrammingError: (ProgrammingError) syntax error in tsquery: "second UK" 'select "_id", "id", "date", "x", "y", "z", "country", "title", "lat", "lon", count(*) over() as "_full_count"\n from "4f1299ab-a100-4e5f-ba81-e6d234a2f3bd" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',)

Suggestions

Filter support: should think in more detail about this (may want to follow recline style)

Simple filters in query parameters would be nice too ...

#3027: solr for 2.0

kindly — Tue, 04 Dec 2012 02:22:09 GMT

change mm support solr 3 and 4 add *_date field

#2796: Need a datahub one-pager

ross — Thu, 02 Aug 2012 16:28:36 GMT

A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data.

#2773: About page needs improving

markw — Mon, 30 Jul 2012 11:13:15 GMT

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

#1009: Improvements to user accounts sytem

rgrp — Fri, 25 Feb 2011 12:13:11 GMT

Forgot password (email a new password)
Confirm email
Do not show register page if you are logged in (redirect to home page)
ticket:1010 - listing of users.
- Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)

#1384: CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages

shevski — Wed, 12 Oct 2011 10:07:57 GMT

Most articles still refer and link to ckan.net, wiki.ckan.net and to packages

#1429: Provide DOIs for datasets in a CKAN instance

rgrp — Sun, 30 Oct 2011 12:21:45 GMT

DOI = digital object identifier = http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

Probably implement as extension rather than core

#1458: Support previewing kml files in data viewer

rgrp — Thu, 10 Nov 2011 14:48:08 GMT

Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map

#1459: Featured Dataset feature

rgrp — Sat, 12 Nov 2011 15:50:12 GMT

Provide way to mark a dataset as featured. Featured database show up on the front page.

TODO: detail this more.

#1571: [super] Issues Extension

jilly mathews — Mon, 19 Dec 2011 11:30:25 GMT

"Proper" issues extension.

Etherpad: http://ckan.okfnpad.org/issues
Code: https://github.com/okfn/ckanext-issues

Overview

This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues)
- Current extension does work but lacks polish
- Should we polish or do significant improvements?
Functionality - see etherpad

Steps

~~(Rename todo(s) -> issue(s) in existing extension)~~

#2874: Clean up bin directory

rgrp — Fri, 17 Aug 2012 08:01:48 GMT

Full of obsolete material

#1096: [super] CKAN Hosted

pudo — Tue, 19 Apr 2011 17:44:22 GMT

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

introduce model.Site and c.site
- site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
Subdomain detector to activate sites.
use site in Authorizer instead of System, have a NullSite? for global things
allow cross-site search
packages are in a list of sites, m:n rather than 1:n
- list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

#1635: Email notifications (e.g. for activity streams)

seanh — Wed, 11 Jan 2012 17:56:27 GMT

CKAN should be able to send email notifications to users.

Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users.

One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well.

Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about.

Analysis here: http://ckan.okfnpad.org/27

#1667: Add an extension point for rendering activity streams

seanh — Tue, 17 Jan 2012 10:29:11 GMT

Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types.

#1748: Make activity streams conform to http://activitystrea.ms/ standard

seanh — Sun, 05 Feb 2012 14:06:24 GMT

At the very least we should make our JSON output conform to their spec:

http://activitystrea.ms/

#1778: Replace classmethod's with module-level functions

seanh — Mon, 06 Feb 2012 15:53:34 GMT

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: http://docs.python.org/dev/howto/doanddont.html

#1785: Replace 'Revisions' page with site-wide activity stream

seanh — Tue, 07 Feb 2012 16:34:18 GMT

Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?

#2200: Add vocabulary_id option to tag_show() logic action function

seanh — Wed, 29 Feb 2012 16:41:18 GMT

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

#2335: Unicode characters don't work in CKAN ini files

seanh — Mon, 30 Apr 2012 12:42:06 GMT

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

#2337: Sort out ini file confusion in docs

seanh — Mon, 30 Apr 2012 14:57:56 GMT

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

#2370: Add examples of all config settings in default config file

seanh — Thu, 03 May 2012 10:50:37 GMT

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

#2475: Upgrade trac.ckan.org to trac 0.12.3

seanh — Tue, 29 May 2012 18:46:16 GMT

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog

#2477: Add extensions section to readthedocs

shevski — Wed, 30 May 2012 15:54:04 GMT

Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to a dedicated section of docs.ckan.org

#2478: Get NeverNotifyUpdaterPlugin installed on trac.ckan.org

seanh — Wed, 30 May 2012 16:52:14 GMT

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

seanh — Wed, 13 Jun 2012 09:07:22 GMT

there's a setting for this

#2520: Document undocumented config options

seanh — Wed, 13 Jun 2012 11:52:48 GMT

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

#2537: Test and document ckanbuild

seanh — Fri, 15 Jun 2012 15:48:48 GMT

https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

#2538: Add multiple-instance support to ckanbuild

seanh — Fri, 15 Jun 2012 15:51:39 GMT

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

#2539: Investigate the existing ckan debian package for ckanbuild

seanh — Fri, 15 Jun 2012 15:54:21 GMT

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

#2540: Implement a way of upgrading ckan sites using ckanbuild

seanh — Fri, 15 Jun 2012 15:55:40 GMT

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

#2541: Add non-core extensions to ckanbuild

seanh — Fri, 15 Jun 2012 15:57:06 GMT

We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.

#2542: Create jenkins job to run ckanbuild, and run tests

seanh — Fri, 15 Jun 2012 15:58:08 GMT

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.

#2619: Omit private datasets from public activity streams

seanh — Thu, 28 Jun 2012 11:49:00 GMT

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

#2771: Documentation and examples for IDatasetForm and IGroupForm

seanh — Fri, 27 Jul 2012 12:30:38 GMT

Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date.

#2815: db_to_form_package_schema() strips tracking summary, isopen

seanh — Wed, 08 Aug 2012 16:47:07 GMT

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

#2870: 1.8 tag_list not defined

ross — Thu, 16 Aug 2012 10:58:52 GMT

From: <thedatahub.org@…> Date: Thu, Aug 16, 2012 at 3:20 AM Subject: WebApp? Error: <class 'genshi.template.eval.UndefinedError?'>: "tag_list" not defined To: ckan-sysadmin@…, kindly@…

URL: http://ckan.net/dataset/uk-postboxes Module weberror.errormiddleware:162 in call << traceback_supplement = Supplement, self, environ

sr_checker = ResponseStartChecker?(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker)

except:

app_iter = self.application(environ, sr_checker)

Module beaker.middleware:73 in call << self.cache_manager)

environ[self.environ_key] = self.cache_manager return self.app(environ, start_response)

return self.app(environ, start_response)

Module beaker.middleware:155 in call << headers.append(('Set-cookie', cookie))

return start_response(status, headers, exc_info)

return self.wrap_app(environ, session_start_response)

def _get_session(self):

return self.wrap_app(environ, session_start_response)

Module routes.middleware:131 in call << r'\1', oldpath)

response = self.app(environ, start_response)

# Wrapped in try as in rare cases the attribute will be gone already

response = self.app(environ, start_response)

Module pylons.wsgiapp:125 in call <<

controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response)

if 'paste.testing_variables' in environ and hasattr(response,

response = self.dispatch(controller, environ, start_response)

Module pylons.wsgiapp:324 in dispatch << if log_debug:

log.debug("Calling controller class with WSGI interface")

return controller(environ, start_response)

def load_test_env(self, environ):

return controller(environ, start_response)

Module ckan.lib.base:239 in call << try:

res = WSGIController.call(self, environ, start_response)

finally:

model.Session.remove()

res = WSGIController.call(self, environ, start_response)

Module pylons.controllers.core:221 in call << return response(environ, self.start_response)

response = self._dispatch_call() if not start_response_called:

self.start_response = start_response

response = self._dispatch_call()

Module pylons.controllers.core:172 in _dispatch_call << req.environpylons.action_method? = func

response = self._inspect_call(func)

else:

if log_debug:

response = self._inspect_call(func)

Module pylons.controllers.core:107 in _inspect_call << func.name, args)

try:

result = self._perform_call(func, args)

except HTTPException, httpe:

if log_debug:

result = self._perform_call(func, args)

Module pylons.controllers.core:60 in _perform_call << """Hide the traceback for everything above this method"""

traceback_hide = 'before_and_this' return func(args)

def _inspect_call(self, func):

return func(args)

Module ckan.controllers.package:322 in read << template = template[:template.index('.') + 1] + format

return render(template, loader_class=loader)

def comments(self, id):

return render(template, loader_class=loader)

Module ckan.lib.base:153 in render << try:

return cached_template(template_name, render_template,

loader_class=loader_class)

except ckan.exceptions.CkanUrlException?, e:

raise

loader_class=loader_class)

Module pylons.templating:249 in cached_template << return content

else:

return render_func()

return render_func()

Module ckan.lib.base:102 in render_template << if loader_class == NewTextTemplate?:

return literal(stream.render(method="text", encoding=None))

return literal(stream.render(method=method, encoding=None,

return literal(stream.render(method="text", encoding =None))

Module genshi.core:183 in render << method = self.serializer or 'xml'

generator = self.serialize(method=method, kwargs) return encode(generator, method=method, encoding=encoding, out=out)

def select(self, path, namespaces=None, variables=None):

return encode(generator, method=method, encoding=encoding, out =out)

Module genshi.output:57 in encode << _encode = lambda string: string

if out is None:

return _encode(.join(list(iterator)))

for chunk in iterator:

out.write(_encode(chunk))

return _encode(.join(list(iterator)))

Module genshi.output:569 in call << def call(self, stream):

strip_markup = self.strip_markup for event in stream:

if event[0] is TEXT:

data = event[1]

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module ckanext.googleanalytics.plugin:93 in download_adder << [downloaded %s times]</span>

count = None for mark, (kind, data, pos) in stream:

if mark and kind == START:

href = data[1].get('href')

for mark, (kind, data, pos) in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1175 in call << """

callable_value = hasattr(self.value, 'call') for mark, (kind, data, pos) in stream:

if mark is ENTER:

if callable_value:

for mark, (kind, data, pos) in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.core:288 in _ensure << # unchanged

yield event for event in stream:

yield event

for event in stream:

Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream):

for mark, event in stream:

kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK):

for mark, event in stream:

Module genshi.filters.transform:1145 in call << :param stream: The marked event stream to filter

""" for mark, event in stream:

yield mark, event if mark is ENTER:

for mark, event in stream:

Module genshi.filters.transform:714 in call << stream = iter(stream)

next = stream.next for mark, event in stream:

if mark is None:

yield mark, event

for mark, event in stream:

Module genshi.filters.transform:682 in _mark << def _mark(self, stream):

for event in stream:

yield OUTSIDE, event

for event in stream:

Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound?

for event in stream:

if event[0] is INCLUDE:

href, cls, fallback = event[1]

for event in stream:

Module genshi.template.base:565 in _flatten << elif kind is EXPR:

result = _eval_expr(data, ctxt, vars) if result is not None:

# First check for a string, otherwise the iterable test

result = _eval_expr(data, ctxt, vars)

Module genshi.template.base:277 in _eval_expr << if vars:

ctxt.push(vars)

retval = expr.evaluate(ctxt) if vars:

ctxt.pop()

retval = expr.evaluate(ctxt)

Module genshi.template.eval:178 in evaluate << traceback_hide = 'before_and_this'

_globals = self._globals(data) return eval(self.code, _globals, {'data': data})

return eval(self.code, _globals, {'data': data})

Module ?:51 in <Expression u"tag_list(c.pkg_dict.get('tags', ))"> << <li py:if="c.pkg_dict.get('tags')" class="sidebar-section">

<h3>Tags</h3> ${tag_list(c.pkg_dict.get('tags', ))}

</li>

${tag_list(c.pkg_dict.get('tags', ))}

Module genshi.template.eval:309 in lookup_name << val = BUILTINS.get(name, val)

if val is UNDEFINED:

val = cls.undefined(name)

return val

val = cls.undefined(name)

Module genshi.template.eval:410 in undefined << """Raise an UndefinedError? immediately."""

traceback_hide = True raise UndefinedError?(key, owner=owner)

raise UndefinedError?(key, owner=owner)

UndefinedError?: "tag_list" not defined CGI Variables CKAN_CURRENT_URL '/dataset/uk-postboxes' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,text/plain,text/xml' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'vi,en-us;q=0.7,en;q=0.3' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'coccoc/1.0 ()' PATH_INFO '/dataset/uk-postboxes' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/dataset/uk-postboxes' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '47065' REQUEST_METHOD 'GET' REQUEST_URI '/dataset/uk-postboxes' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '<address>Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80</address>\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application <beaker.middleware.CacheMiddleware? object at 0x7f5312a51650> beaker.cache <beaker.cache.CacheManager? object at 0x7f5312a51790> beaker.get_session <bound method SessionMiddleware?._get_session of <beaker.middleware.SessionMiddleware? object at 0x7f5312a516d0>> beaker.session {'_accessed_time': 1345083636.079552, '_creation_time': 1345083636.079552} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script mod_wsgi.input_chunked '0' mod_wsgi.listener_host mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (<SimpleCookie: >, ) paste.registry <paste.registry.Registry object at 0x7f52e47477d0> paste.throw_errors True pylons.action_method <bound method PackageController?.read of <ckan.controllers.package.PackageController? object at 0x7f52e477b810>> pylons.controller <ckan.controllers.package.PackageController? object at 0x7f52e477b810> pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons <pylons.util.PylonsContext? object at 0x7f52e477be90> pylons.routes_dict {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'} repoze.who.logger <logging.Logger object at 0x7f53127b2b50> repoze.who.plugins {'openid': <OpenIdIdentificationPlugin? 139994769488336>, 'friendlyform': <FriendlyFormPlugin? 139994769488528>, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator? object at 0x7f5312d893d0>, 'auth_tkt': <AuthTktCookiePlugin? 139994771858704>, 'ckan.lib.authenticator:OpenIDAuthenticator': <ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350>} routes.route <routes.route.Route object at 0x7f5312955a50> routes.url <routes.util.URLGenerator object at 0x7f52eb8c9090> webob._parsed_query_vars (GET([]), ) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f5308983a80> wsgi.version (1, 1) wsgiorg.routing_args (<routes.util.URLGenerator object at 0x7f52eb8c9090>, {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'})

#3005: Add following/unfollowing of groups: model, API, frontend, tests

seanh — Tue, 23 Oct 2012 17:35:44 GMT

#3028: Feature: dashboard activity stream filtering

seanh — Mon, 10 Dec 2012 12:07:24 GMT

https://github.com/okfn/ckanext-pdeu/issues/13

#3029: JSONP parameter scuppers Search in API

dread — Tue, 11 Dec 2012 15:00:35 GMT

http://datahub.io/api/2/search/package?jsonp=jsonpcallback&q=canada returns

{"count": 0, "results": []}
I believe this worked in CKAN 1.4 or 1.5, but it is broken on 1.7.1, 1.8 and whatever demo.ckan.org is running. I suspect the jsonpcallback parameter is getting sent to SOLR.

This bug prevents using javascript on another site to search CKAN (although hopefully the action API would work).

#2547: ODS Initial data sets

ross — Mon, 18 Jun 2012 10:55:06 GMT

Requirements

The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS.

May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps.

Interface

None

User Stories

As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed.
As a bizdev person I would like to be able to demonstrate how ODS works with real datasets.
Tasks

[ ] Identify relevant sources for datasets

[ ] Pick datasets

[ ] Set them up for import

Estimates

#2590: Publisher dashboard

ross — Mon, 25 Jun 2012 10:03:37 GMT

Need proper user stories but ...

Publisher admins/editors may need a more useful group read page showing things like:

The current search
Recent activity
People within the group
Followers
Others?

#2699: workflow for associating datasets with groups

shevski — Wed, 18 Jul 2012 14:02:56 GMT

needs review & speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group

#2702: Future Javascript wishlist for demo

shevski — Wed, 18 Jul 2012 14:07:55 GMT

tooltip on popular datasets with number of views facets to update automatically creating a dataset without reloading page between steps hover on licences information autocomplete on search terms group filtering social share buttons in lightboxes dataset counts on homepage

#2814: Demo: upload file behaves oddly

markw — Wed, 08 Aug 2012 10:06:00 GMT

Uploading a file behaves counter-intuitively (I would suggest wrongly).
When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen:

the pathname of that file is filled in the box;
nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc).
Instead of this,

the file is immediately and, seemingly, irrevocably uploaded;
the box is filled with a mysterious URL (https://commondatastorage.googleapis.com/ckan-demo ...)
the mysterious URL is so long I can't even see the final element, which is my file name.

#2956: Allow for resource editing in CKAN 2.0

johnmartin — Wed, 03 Oct 2012 15:11:40 GMT

On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually.

#1287: NAVL validation errors - Junk fields should be listed explicitly

dread — Wed, 24 Aug 2011 16:25:02 GMT

When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message:

{"__junk": ["The input field __junk was not expected."]}
It should mention the actual key which is not expected. e.g.

{"relationships": ["The input field 'relationships' was not expected."]}
Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that.

#235: Resource format normalization and detection

dread — Mon, 18 Jan 2010 15:13:24 GMT

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
Collect and map surface forms from all CKANs
Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
Add heuristics for format auto-detections:
Map well-known file extensions
Recognize obvious magic (Zip, Tar)
Peek into Zipfile/Tarfiles?
Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
See also: #816 (Autocomplete for the resource format field)

#1069: Stub datasets (request for datasets)

rgrp — Fri, 01 Apr 2011 14:08:39 GMT

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

Agree a new dedicated tag. e.g. todo.does-not-exist
Related

This ticket has clear links to http://getthedata.org/

#2244: clean up auth

toby — Mon, 19 Mar 2012 17:22:51 GMT

cache admins etc
aim to move check_access so independent of logic function
general spruce up

#2257: cleanup template vars

toby — Fri, 23 Mar 2012 14:35:07 GMT

look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc

#2282: log api calls for users

toby — Tue, 10 Apr 2012 16:18:45 GMT

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

#2411: .ini file is a mess

markw — Tue, 22 May 2012 11:41:40 GMT

The .ini file needs cleaning up in assorted ways which Toby will expand upon.