Currently you can specify the capacity in the API search, this should be stripped and default to public for now.

Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json”

See the following Basecamp thread for UI examples and discussion.

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056

Currently use old "bullet-point" style. Should change to now standard style used on e.g. package search results page.

Cost: 0.5h

Currently this page:

​http://docs.ckan.org/en/ckan-1.8/datastore.html

has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient.

Suggest moving the google presentation into the documentation proper, so it works with grep etc.

Currently there is this page:

​http://docs.ckan.org/en/latest/data-viewer.html

which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it.

Separately there is this page: ​http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api.

I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be:

CKAN has builtin previews of data resources on resource pages, enabled by default.

Images, Google Documents, and web page resources will be loaded into embedded iframes for preview.

Text-like files will be displayed raw.

CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer.

Additionally, you can enable CKAN's DataStore?, requires you to install ElasticSearch? and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data.

Does having DataStore? enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline,

You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd.

Currently the stats plugin has a template and templates_legacy directory. We need to ensure that the templates in templates_legacy are loaded.

Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package.

I'm requesting the ability to diff against a "blank slate" initial state of a project, so I can see the content of the first project commit.

Not sure if this is a vdm feature, so I'm putting this ticket in against ckan.

Currently the default authz for a package is hard-coded to:

 <PackageRole user="visitor" role="editor" context="Package">,
 <PackageRole user="logged_in" role="editor" context="Package">,

This should be configurable in the config, so that you can have a more locked down instance etc.

Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits?

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

Currently stages of the same colour blur together, these could do with a nice divider.

Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

• Common template code (checkbox template)
• Logic code (or just common code) for wiring into authz system
• Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup.

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

Currently it is possible (v0.11) to have a PackageResource? object without a url. This should be impossible to do (either at creation time or via modification).

Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?

Currently if you try to refresh (i.e. schedule the harvester to re-run) an inactive source (one who's state is 'withdrawn' - e.g. ​http://s031.okserver.org:2375/harvest/3feea629-32d8-4431-9957-9f31d32864f1)

you get the error message: "An error occurred: [Can not create jobs on inactive sources]"

This could be better and not red, e.g. and info box (maybe yellow) saying that "Cannot re-run harvesting on inactive sources. Please first change the status to 'active'"

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....

Currently for some reason I'm not getting the errors/data dict in the package/new.html template.

Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.

Currently can run as a celery task launched by paster command.

Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource.

Currently autocomplete method on package controller. This method should be in API (like other autocomplete methods).

Will need to update client code (just forms atm I think).

Currently all data is only available via the web interface which makes it difficult to obtain in bulk. Data should be provided in other more convenient, machine usable forms:

• DB Dump
• JSON for each package page (maybe Atom as well)

Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a PackageResource?. This doesn't include changes to tags, extras and resources. These need to be added.

There is the complication that the 'diff' function doesn't currently work beyond PackageResource?. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now.

Rufus suggests alongside the existing methods:

pkg.all_revisions()
pkg.diff()

we have ones which also include the related objects (tags, extras, resources):

pkg.all_revisions_full()
pkg.diff_full()

You could have a function which returned these related objects. Ideas:

revobjlist = [ self, resources, extras ]
list of tuples: (revobj, changed_revobjs)
list of tuples: (revobj, {objtype: [changed_revobjs]})
tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...}

Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?)

Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket.

Current extensions documentation needs some work: ​http://docs.ckan.org/en/latest/plugins.html

• Queue extension section may now be out of date (?)
• Think about how it integrates with ​https://github.com/okfn/ckanext-example (especially tutorial and example extension)
• Document all plugin points (auto-extract from CKAN source??)

Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have:

[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "package_resource_revision_pkey", referer: http://ckan.net/package/edit/cofog
[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30]  'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog

Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

• 'ResourceExtras?' for extra metadata on resources
• Makes 'Resource' into first class object (and make available in WUI)?

Depends

• #877 (Upload in WUI) - should be part of that extension

Create plugin system using <​http://pypi.python.org/pypi/pyutilib.component.core/>

Specification in CEP-003 (browser:doc/cep/cep-003.txt) and <​http://okfnpad.org/ckan-plugins>

Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!)

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

• Purge revisions (or sets of revisions) and purge objects #1076
• Set roles for users #1075
• Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
  • WONTFIX
• Overview of activity
  • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

​https://bitbucket.org/okfn/ckanext-admin

Tickets

• ticket:1031 - user autocomplete api in ckan

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system

Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'.

Create a group, create some datasets and add them to the group, then view the group's page with a sort-order specified in the URL, e.g. /group/foo?q=&sort=title_string+asc, no datasets are displayed even though there are datasets in the group that match the search query. Remove the sort order from the URL and the datasets will appear.

Could be done by:

• using the solr distributed search
  • difficult set up
• merging result sets from apis
  • make sure that results can be merged properly (score, facets, ...)

Cost: 1h

Cost: 0.5h (?)

Conjecture this went missing in cset:a35db862a841

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

Cost - 2 days

Search interface has new options to filter and sort the results by the date the package has been last modified in ckan. Search options are included in both Web UI and Search API.

The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package.

Example search parameters:

Core CKAN currently has a dependency on LXML and it would be nice if we could remove it (although this *may* be indirectly through genshi).

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

Controllers and templates should not access db objects directly - they should do all access via authz module giving username. They are handed by a query that has already been filtered by the packages they are authorized to read.

(Additional idea to be discussed: When they request a package object, they are handed an copy of the db object - disconnected from the database - so it the db object can't be changed.)

A couple of tests can be reenabled when this is done: ckan.tests.functional.test_authz.TestUsage?.test_admin_list_deleted ckan.tests.functional.test_authz.TestUsage?.test_search_deleted

Consolidate the extra spatial as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features.

Consolidate js. Remove cruft.

• Move stuff to vendor
  • Have all vendor libs local (?)
  • jquery, jquery-ui etc
• Consolidate autocomplete to use jquery-ui
  • package autocomplete (groups)
  • tag autocomplete (package)
  • user autocomplete (authz group create)
    • turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
• Move forms.css into main
• Remove tag cloud from front page (can be example going forward)
• Remove bookmarklet (since in wiki at: ​http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
• Move name slug generation into standard js and clean up

Possible:

1. JS unit tests for the given pages ...
2. Option to use external site to serve js and css (e.g. CDN)

Configure buildbot to use postgres db which has been moved to eu5.okfn.org

pudo said: Could you perhaps also comment on ​http://knowledgeforge.net/okfn/tasks/ticket/466 with a hint on how to update this properly? I only edited /home/buildslave/okfn/full/build/buildandsmoke/buildandsmoke.ini which was the only occurence of the DB credentials I could find there.

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

Common requirements for running CKAN behind a (e.g Wordpress or Drupal) front-end:

1. Unrestricted total read-only access to catalogue API for general public (e.g. voluntary organisation).
   • monitored by API key
   • not monitored by API key
2. Restricted total read-write access to catalogue API for authorized clients (e.g. front-end system, bulk upload clients).
   • restricted by CKAN access controller
   • restricted by HTTP Auth
   • restricted by IP address
3. Restricted total read-write access to catalogue Web UI for authorized users (e.g. site admins).
   • restricted by CKAN access controller
   • restricted by HTTP Auth
4. Restricted partial read-write access to catalogue Web UI for authorized users (e.g. group admins).
   • restricted by CKAN access controller
   • restricted by HTTP Auth

Clicking on the blue buttons or images on related page ​http://s031.okserver.org:2375/dataset/gold-prices/related should take you to a new tab instead of navigating you away

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: ​http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.

Child tickets:

• #234 UI Review - Autocomplete package names & tags in search
• #193 Searching by time-related field
• #191 Searching by modification date
• #905 Unable to search with accented characters in package names
• #906 Ability to search without accents for accented words
• #924 Search box has no search button

Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two.

Child tickets:

• #1107 Move package autocomplete from package controller and move to API
• #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

• ​http://wiki.apache.org/solr/SolJSON
• ​http://www.flickr.com/services/api/response.json.html

Child tickets:

• #1041 Start Using the CKAN Wiki for Tutorial-style documentation
• #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): ​http://trac.ckan.org/ticket/927

• CKAN 1-page overview (for enterprise and for data hackers)
• Administrator's Guide (including install)
• Extensions Guide
• Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: ​http://wiki.ckan.net/Documentation_Plans

Minor Items