CKAN: Ticket Query

#763: Read-only mode - Setup

dread — Tue, 26 Oct 2010 11:11:46 GMT

Admin configures entering read-only mode in one of two places:

CKAN config file (e.g. ckan.ini)
environment variable from Apache config

Once enabled, no writes can occur to the database (including user ratings and other usage stats).

#948: Highlight (to a sysadmin) which packages are deleted

dread — Wed, 02 Feb 2011 11:32:49 GMT

When a customer logs in as a sysadmin then he/she see all packages, including deleted and pending ones. These are hidden to the average user, but the sysadmin has no idea of this until he clicks on the package and sees at the bottom 'state: deleted'.

It should be more obvious than that on the search view - an icon, message or crossed-out name to packages are deleted.

#1285: Errors cause emails

dread — Tue, 23 Aug 2011 16:29:04 GMT

Currently a sysadmin gets an email when an exception is not caught. But there are occasions when we DO want to catch an exception so we can fail nicely for the user, but the sysadmin STILL gets an email to know to fix something. e.g. if there is an exception when search indexes a package. You want to catch the exception and still run any other notify calls.

#1366: Search inside extra fields

dread — Thu, 29 Sep 2011 10:09:52 GMT

SOLR search doesn't support searching for part of an extra field, but it does for other fields.

i.e. title="One Two Three" matches q=one AND q=title:one and geographic_coverage="England Scotland" matches q=England BUT NOT q=geographic_coverage:England

This problem emerged when we went to SOLR in #1275 (CKAN 1.5a). Tests were skipped.

This is could be a problem for DGU and maybe elsewhere.

#1458: Support previewing kml files in data viewer

rgrp — Thu, 10 Nov 2011 14:48:08 GMT

Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map

#1749: Allow creating activity details through API

seanh — Sun, 05 Feb 2012 18:10:06 GMT

Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts.

#1778: Replace classmethod's with module-level functions

seanh — Mon, 06 Feb 2012 15:53:34 GMT

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: http://docs.python.org/dev/howto/doanddont.html

#2234: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates

seanh — Fri, 16 Mar 2012 12:59:15 GMT

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

Feed items should probably be cached somewhere

Suggested

The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

Caching is pretty crucial and should probably obey the ttl of the feed.

#2348: Java client library for CKAN

ross — Tue, 01 May 2012 13:46:20 GMT

Start a Java library for interacting with the CKAN Action API.

Currently requires support for Groups, Tags, Resources, and Search.

An example app would be very useful.

#2378: Extract metadata directly from resources that contain it.

ross — Thu, 10 May 2012 11:58:08 GMT

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

#2404: Auto-screenshot for related items

rgrp — Mon, 21 May 2012 06:55:47 GMT

As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere

Implementation: (?) Use phantomjs or something like http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct.

#2552: Controlling access to features

ross — Mon, 18 Jun 2012 11:26:10 GMT

Requirements

To provide a freemium service it is necessary to be able to provide differing levels of functionality based on the type of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks.

Initial implementation should focus on limiting access to datastore disk space.

Interface

These changes are currently only for the data hub and should be kept as much as possible within the data hub extension.

User Stories

As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited).

As a system administrator I don't expect to need to manage the levels of users or the features that this applies to.

Tasks

[ ] Clarification of requirements/analysis

[ ] Tests

[ ] Code

[ ] Model

[ ] API

[ ] UI

[ ] Documentation

Estimates

#84: Revert support on versioned objects

rgrp — Thu, 23 Jul 2009 08:59:03 GMT

Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support:

Cases where multiple objects changed in a revision but only want to revert 1 (low priority)
Want to revert but have reversion as a new revision of that object.

Seems low priority at present.

Cost: ?

#377: API Should return JSON in all cases

glen@… — Tue, 27 Jul 2010 22:02:38 GMT

When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{'status': 404, 'message': 'package somename not found'}

When errors are encountered when creating a package:

{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

Not authorised:

{'status': 503, 'message': 'You are not authorised to create this package.'}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.

#1444: Language detection defaults to en_us?

dread — Fri, 04 Nov 2011 18:50:34 GMT

Using curl you always get English, even if ckan.locale_default=de. Find out why. (1.5b)

Maybe we should disable locale detection for this release if lots of people's browsers don't have it set correctly (e.g. chinese with us internet explorer)

#1563: Finish Data Storage

jilly mathews — Mon, 19 Dec 2011 10:59:06 GMT

Unsure what needs to happen here. Need to list outstanding tasks and implement.

#1568: Moderated Edits

jilly mathews — Mon, 19 Dec 2011 11:22:56 GMT

Can this be released as a standard CKAN feature?

#1570: Integrated file Storage

jilly mathews — Mon, 19 Dec 2011 11:29:02 GMT

Is this ready for release? What needs to be done?

#1664: Smarter group activity streams

seanh — Tue, 17 Jan 2012 10:20:54 GMT

We already have simple activity streams for groups that have new group or group updated events (see #1631). Smarter/more useful group activity streams would also contain the events for things related to the group: users, datasets, other groups, etc.

This should just mean using a more complex SQL query in the group_activity_list() logic function.

Things might get tricky when a user or dataset joins a group, this may show up as two events (the user or dataset was modified, and the group was modified). We may want to have two separate events for this (one to go in the user or dataset activity stream and one for the group), or maybe we just want one 'user/dataset joined group' event.

#1796: Get rid of lxml dependency

dread — Tue, 14 Feb 2012 18:34:00 GMT

lxml is used in CKAN core in two places:

ckan/lib/helpers.py:239 Parsing Markdown and ensuring it is valid XML (i.e. tags close nicely so the HTML of the rest of the CKAN page is not disruptable).
reading the SOLR config XML to determine what version it is.

Both of these could be done with xml.dom built into Python and would save us a sizeable and slightly problematic dependency.

lxml is needed by these extensions: ckanext-inspire, ckanext-harvest, ckanext-csw & ckanext-wordpresser, so we'd have to add this to their requirements.

#2238: Deploy some test vocabs and publishers on test.ckan.net, check tutorial-style instructions for these

seanh — Mon, 19 Mar 2012 13:55:11 GMT

#2312: Analysis of how datasets could belong to users instead of Groups

ross — Tue, 17 Apr 2012 11:56:10 GMT

DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how.

#2544: facet.sort is not available in the package_search action

icmurray — Mon, 18 Jun 2012 09:57:01 GMT

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

#1650: Add new language/translation of fields for datasets.

ross — Mon, 16 Jan 2012 10:23:12 GMT

Datasets should have a field that specifies the language used in the dataset, this should be a two character ISO 639-1 code (see http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes for list) . Also make sure all tests and migrations are resolved. [2d]

Determine the best approach to having 'translation_of' references to show which dataset the current one is a translation of. This may be a translation of field, or a 'copied from' field or another form of package relationship. [2d]

#989: Extending the model from plugins

pudo — Mon, 14 Feb 2011 13:22:04 GMT

We need to support extending the model from plugins. This could involve:

Adding a plugin hook to extend the mapper
Adding an upgrade hook for plugin schema migrations
Documenting how this is to be done
Find a way to avoid conflicts

#1096: [super] CKAN Hosted

pudo — Tue, 19 Apr 2011 17:44:22 GMT

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

introduce model.Site and c.site
- site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
Subdomain detector to activate sites.
use site in Authorizer instead of System, have a NullSite? for global things
allow cross-site search
packages are in a list of sites, m:n rather than 1:n
- list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

#1165: Add multi-site support to ckan

nils.toedtmann — Thu, 26 May 2011 12:41:07 GMT

Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3).

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "<VirtualHost?>", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:

ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
In a global ini file, there are directives mapping servernames to ini files.

In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc.

#1235: [super] Search Improvements

thejimmyg — Wed, 20 Jul 2011 17:24:01 GMT

Child tickets:

#234 UI Review - Autocomplete package names & tags in search
#193 Searching by time-related field
#191 Searching by modification date
#905 Unable to search with accented characters in package names
#906 Ability to search without accents for accented words
#924 Search box has no search button

Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two.

#1438: Action API - parameter discovery/checking

dread — Tue, 01 Nov 2011 15:38:10 GMT

Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error.

Currently they are listed in the docs, extracted from the code manually.

So you could GET /action/api/package_list to receive not only the help text, but a list of arguments.

And if you send an extra or missing argument then an intelligent error message can be returned.

implementation

How about some sort of decorator on the action function:

@logic_params(id, offset, limit)
def get_package_list(context, data_dict):
    ...

This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function?

I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code.

Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict.

#1457: Bug with DataNL instance

jilly.mathews — Thu, 10 Nov 2011 11:37:21 GMT

"when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."

n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding http://register.data.overheid.nl/user/me, I get a 404 error (not found).

#1557: Complete Webstore Preview Extension

jilly mathews — Mon, 19 Dec 2011 10:40:53 GMT

Finish any work out standing on web store preview extension to be able to package and release.

Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted.

#1558: Publisher Tools

jilly mathews — Mon, 19 Dec 2011 10:46:13 GMT

Summarise final set of requirements for this and finish development and test. Estimated 10 working days.

#1560: Follow extension

jilly mathews — Mon, 19 Dec 2011 10:51:19 GMT

Estimate 2 days to finish dev and test.

David can you add any info needed here?

#1561: To do extension

jilly mathews — Mon, 19 Dec 2011 10:52:52 GMT

Can we finish this ready for release on data hub and CKAN Hosted.

#1562: Finish Geo Spatial

jilly mathews — Mon, 19 Dec 2011 10:56:33 GMT

Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients.

#1564: Structured Data (Data API)

jilly mathews — Mon, 19 Dec 2011 11:07:14 GMT

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.

#1565: Admin dashboard finished?

jilly mathews — Mon, 19 Dec 2011 11:11:32 GMT

Is testing complete and ready for release?

#1567: Finish QA extension

jilly mathews — Mon, 19 Dec 2011 11:19:59 GMT

Requires change to celeryd. Estimated 4 weeks.

#1569: Wordpressser

jilly mathews — Mon, 19 Dec 2011 11:27:36 GMT

How much effort will this be to be ready to use?

#1572: Meta data Harvester

jilly mathews — Mon, 19 Dec 2011 11:35:09 GMT

Need to write custom harvesters for each client. Is it worth having one for data hub?

#1573: Apps and Ideas

jilly mathews — Mon, 19 Dec 2011 11:36:33 GMT

Estimate 2 weeks for someone to finish and test.

#1581: Blog post about Google Analytics extension for CKAN

johnglover — Tue, 20 Dec 2011 17:40:00 GMT

The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post.

Can ping John Glover in January for any details required.

Key link is: http://thedatahub.org/analytics/dataset/top though this should probably move to be under stats (e.g. http://thedatahub.org/stats/usage)

#1596: Refactor authz roles

dread — Thu, 22 Dec 2011 10:31:28 GMT

Suggestions from rgrp:

Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

(This ticket is split off from #1065)

#1667: Add an extension point for rendering activity streams

seanh — Tue, 17 Jan 2012 10:29:11 GMT

Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types.

#1740: Get rid of `from module import ...`

seanh — Wed, 01 Feb 2012 11:06:09 GMT

It's really bad to do from module import * and CKAN has a lot of them. I suggest a three-pronged approach:

Don't add any more of them.

When you're programming if you see an easy opportunity to remove one then do so.

At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things.

We should also get rid of most or all of the from module import foo and from module import foo, bar statements.

I think the right thing to do is just import module and then use module.foo in your code. But if you find yourself doing module.foo.bar then you may have a code smell.

See: http://docs.python.org/dev/howto/doanddont.html

#1748: Make activity streams conform to http://activitystrea.ms/ standard

seanh — Sun, 05 Feb 2012 14:06:24 GMT

At the very least we should make our JSON output conform to their spec:

http://activitystrea.ms/

#1782: Improvements to datasets page UX (Feb 2012)

rgrp — Tue, 07 Feb 2012 10:10:41 GMT

Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
~~Icons for *types* of resources in resource list~~
List of 'Owners' (administrators) at top right of sidebar (a la group page)
- Much prefer owner to administrator ...

#1787: [super] Improve RESTful API

dread — Wed, 08 Feb 2012 11:54:42 GMT

Lists of entities should be full URLs, rather than just the names
Discoverability - /api/v3/rest should list the entity types that can be listed

This could be v3 of the RESTful interface.

#1790: Click to delete tags, rather than have all existing tags in the tag text box

dread — Fri, 10 Feb 2012 15:44:34 GMT

From Pablo:

Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box.

#1800: Tidy up _list() and _search() functions in ckan/logic/action/get.py

seanh — Thu, 16 Feb 2012 15:12:09 GMT

For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality.

#1818: Spatial metadata editor

amercader — Mon, 20 Feb 2012 18:52:21 GMT

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

Integration with the form sections.
Allow some degree of configuration (default extent, limit geometry types...)

[1] http://openlayers.org/dev/examples/draw-feature.html [2] http://openlayers.org/dev/examples/modify-feature.html

#1824: Add vocabulary pages

seanh — Tue, 21 Feb 2012 17:24:49 GMT

For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar.

#2200: Add vocabulary_id option to tag_show() logic action function

seanh — Wed, 29 Feb 2012 16:41:18 GMT

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

#2235: Group drop down on dataset edit should use chosen and sort groups by name

rgrp — Fri, 16 Mar 2012 14:15:57 GMT

#2244: clean up auth

toby — Mon, 19 Mar 2012 17:22:51 GMT

cache admins etc
aim to move check_access so independent of logic function
general spruce up

#2258: Customizable contributor agreement

rgrp — Mon, 26 Mar 2012 08:45:49 GMT

Customize text at bottom of forms
Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)

#2265: 'More Like This' for a dataset

dread — Tue, 27 Mar 2012 16:20:20 GMT

When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar.

SOLR has a feature for finding documents similar to a particular document, called 'More Like This'.

We would like this for DGU.

#2282: log api calls for users

toby — Tue, 10 Apr 2012 16:18:45 GMT

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

#2287: Consistent datetime and time period strings localization

seanh — Thu, 12 Apr 2012 18:08:43 GMT

Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support.

For example, on the user index page right now there are time period strings like "less than 1 month" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly).

ISO date format everywhere might be a good idea.

Need to consider local timezone issues.

Python standard library and Babel may have useful helpers for this.

Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?

#2301: Replace old-style string formatting with format() method (at least in strings marked for translation)

seanh — Mon, 16 Apr 2012 12:27:44 GMT

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

#2326: dataset permalinks

toby — Wed, 25 Apr 2012 16:39:06 GMT

create permalinks

eg \dataset\~a5-DQ\nice-name

~XXXXX defines the permalink the nice-name is ignored

packages table needs a new field for this data

existing datasets need updating

new datasets need the identifier creating

routing needs updating

needs to work for all urls like edit

Once this is implemented we can decide when we will use permalinks vs the normal urls.

#2382: Investigate options for basic geocoding

amercader — Mon, 14 May 2012 17:39:20 GMT

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

#2411: .ini file is a mess

markw — Tue, 22 May 2012 11:41:40 GMT

The .ini file needs cleaning up in assorted ways which Toby will expand upon.

#2573: package_search does not allow solr's per-field facet parameters

icmurray — Tue, 19 Jun 2012 13:27:19 GMT

Solr allows its facet parameters to be specified on a per-field basis, eg. facet.limit applies to all facet fields, but solr allows it to be overriden for a specific field, eg. facet.tags.limit.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See ckan.lib.search.query.VALID_SOLR_PARAMETERS.

#2810: heroku ckan support

kindly — Tue, 07 Aug 2012 18:40:58 GMT

Get ckan working on heroku

#1815: Reenable Sparql endpoint on publicdata.eu

amercader — Mon, 20 Feb 2012 17:08:43 GMT

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

#2202: Display page view count on dataset and resource pages

rgrp — Sat, 03 Mar 2012 09:04:15 GMT

Just like we display download counts we should display view counts.