CKAN: Ticket Query

#834: Searching in CKAN

Alexander — Fri, 26 Nov 2010 11:06:04 GMT

Hello.

I've installed stable CKAN v1.1 from PyPI.

I can't find any docs about using CKAN API in order to query packages.

Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible?

Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient:

result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1})

As result nothing found.

Thanks.

#668: Caching issues on API v1

Colin Calnan — Fri, 01 Oct 2010 17:19:02 GMT

It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues.

Steps to test:

Modify a dataset on datadotgc.ca, redirects to CKAN

On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100

You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present

Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present.

Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either.

#822: Docs: Configuration as site-available instead of site-enabled for apache

Thu, 18 Nov 2010 19:05:03 GMT

The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net

#904: Review CKAN documentation

Stiivi — Wed, 12 Jan 2011 11:43:30 GMT

What's bad at the moment?

lack of documentation e.g. config (very poorly documented)
too many sources of documentation
no common theming

Sources:

http://ckan.org/
http://packages.python.org/ckan
http://wiki.okfn.org/ckan/ - user
Deprecate: http://knowledgeforge.net/ckan/doc/ckan/

Resulting meta-ticket with things to do: ticket:927

#905: Unable to search with accented characters in package names

Stiivi — Thu, 13 Jan 2011 17:18:40 GMT

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

psql DB with UTF-8 tables

psql search backend

ckan version: 1.3.1b

Example:

http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

#906: Ability to search without accents for accented words

Stiivi — Thu, 13 Jan 2011 17:26:28 GMT

In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCIITRANSLIT).

#699: Paging search results re-sets Open Only and Downloadable Only flags

TimDavies — Fri, 15 Oct 2010 11:54:58 GMT

When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings.

#951: Can't add a package to group

adrian.pohl@… — Thu, 03 Feb 2011 09:54:46 GMT

I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page.

#1246: Search results on ckan.net are mistakenly all 'open'

adrian.pohl@… — Thu, 28 Jul 2011 14:29:13 GMT

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package

#1064: Remove Workers from ckanext-queue

amercader — Tue, 29 Mar 2011 16:58:11 GMT

The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message). This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly.

#1160: Havest CSW servers on PDEU

amercader — Thu, 26 May 2011 10:54:31 GMT

It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include:

Compose Endpoint List (1d)
Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc: https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0

#1161: Build a simplified theme for PDEU

amercader — Thu, 26 May 2011 11:03:55 GMT

We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue.

This ticket relates to work on the PDEU theme only!

#1162: Define a common extra for storing the geographic extent

amercader — Thu, 26 May 2011 11:18:24 GMT

We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion: http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html

#1164: Cloropleth Map of European Data Availability for PDEU

amercader — Thu, 26 May 2011 11:20:24 GMT

A nice map in the homepage showing the availability of data across Europe

#1166: Less stringent filters for displaying the WMS preview

amercader — Thu, 26 May 2011 15:10:46 GMT

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

#1173: Offer a DCat representation for packages in the API

amercader — Wed, 01 Jun 2011 14:47:19 GMT

/api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf.

#1253: Authz refactor

amercader — Mon, 01 Aug 2011 11:51:31 GMT

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

#1272: Store spatial extents provided in the package form

amercader — Mon, 15 Aug 2011 12:36:27 GMT

Consolidate the extra spatial as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features.

#1279: Refactor or deprecate Authorization Groups

amercader — Mon, 15 Aug 2011 13:36:07 GMT

Authorization Groups are going to be deprecated in 1.8

It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets #1229 and #1253)

#1301: Give access to extensions from the old formalchemy controllers

amercader — Thu, 01 Sep 2011 13:14:48 GMT

This was removed during the auth refactoring, but it's needed in the old controllers

#1303: User can not login again if he has edited his profile

amercader — Thu, 01 Sep 2011 15:04:32 GMT

Only happens if the user does not update the password. Most probably introduced during #1229.

#1304: Show more descriptive error messages when sending emails

amercader — Thu, 01 Sep 2011 16:06:46 GMT

Sending emails is failing in some cases, and the error message just says 'Could not send reset link'.

#1305: SMTP config for thedatahub.org and IATI

amercader — Fri, 02 Sep 2011 09:44:17 GMT

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},)

As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org

#1329: i18n poor in some templates - missing i18n tags

amercader — Mon, 12 Sep 2011 10:44:48 GMT

Just a tip for developers regarding translations:

When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

#1333: Move form converters to ckan core

amercader — Tue, 13 Sep 2011 10:21:08 GMT

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

#1334: "AttributeError: 'NoneType' object has no attribute 'name'" exception when logging in

amercader — Tue, 13 Sep 2011 10:25:08 GMT

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

#1337: Group extras are JSON endoded in the REST API responses

amercader — Tue, 13 Sep 2011 14:38:47 GMT

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}

#1338: Exception when trying to use a custom form schema from an extension

amercader — Tue, 13 Sep 2011 14:46:57 GMT

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'

#1342: Home controller does not capture SearchError exceptions

amercader — Wed, 14 Sep 2011 11:53:47 GMT

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting http://localhost:5000/dataset).

Steps to reproduce:

Change the solr_url parameter in the ini file to something non-existant
Visit http://localhost:5000

#1354: Pass the context object to plugins implementing the IPackageController and IGroupController

amercader — Sun, 25 Sep 2011 13:01:20 GMT

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

#1356: Can not recreate a deleted extra

amercader — Mon, 26 Sep 2011 10:49:40 GMT

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

#1375: Define new package id before sending it to the IPackageController plugins

amercader — Thu, 06 Oct 2011 14:14:01 GMT

When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface

https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64

At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package.

One possible solution would be to set the package id in the data dict received by package_create (if it's not defined):

from ckan.model.types import make_uuid
if not 'id' in data or not data['id']:
   data['id'] = make_uuid()

#1430: Documents get mixed between SOLR cores

amercader — Mon, 31 Oct 2011 13:34:36 GMT

On some occasions (apparently random), the documents indexed in a specific SOLR core get mixed with different site_ids.

E.g: We look for all documents in the testing.iatiregistry.org core, faceted by site_id. We would expect all documents to have site_id = iati_testing, but some of them have site_id = iatiregistry.org

http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=*:*&fq=+state:active&facet=true&facet.field=site_id

<lst name="facet_fields">
<lst name="site_id">
<int name="iati_testing">265</int>
<int name="iatiregistry.org">255</int>
</lst>
</lst>

If we compare one of the records which disappeared from the "iati_testing" site_id in both the production and testing SOLR cores of the server, the records are exactly the same, including the indexed_ts property:

http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

Note that the response from the URLs shown may vary, as the testing site could have been reindexed.

#1456: Use resource description instead of name if both are present

amercader — Wed, 09 Nov 2011 18:16:59 GMT

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

#1470: Check user name in the profile form

amercader — Wed, 16 Nov 2011 12:22:23 GMT

#1490: Standardize output from package listings coming from the logic layer

amercader — Fri, 25 Nov 2011 12:47:19 GMT

Right now, the two logic functions that return a list of packages (package_search [1] and group_packages_list [2])use custom functions to generate the output dict. That's suboptimal because:

The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts.
Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension.

In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the "isopen" property, needed by the template renderer.

[1] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685 [2] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442

#1498: Support multiple SOLR schema versions

amercader — Tue, 29 Nov 2011 16:47:44 GMT

At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema. This process will involve:

Better documentation on how to deploy SOLR, as single instance or multicore
A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL.
(If possible) Checks in CKAN core that deal with different schema versions.

#1501: Pagination links broken

amercader — Wed, 30 Nov 2011 09:46:37 GMT

Most of the different pagination links are broken and point to wrong locations. e.g:

Dataset listings in group page:

http://test.ckan.net/group/bibliographic

Tag letters in tag listing:

http://test.ckan.net/tag

User listing:

http://test.ckan.net/user

Revision listing:

http://test.ckan.net/revision

This is most probably related with:

https://github.com/okfn/ckan/commit/1772a5c

#1516: Update SOLR schema after 1.5.1 release and set up multicore SOLR instances if needed

amercader — Mon, 05 Dec 2011 18:07:11 GMT

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
- Update CKAN
- Update SOLR schema (symlink to suitable version in CKAN source)
- Rebuild search index

Single SOLR instance used by multiple CKAN sites.
- Update CKAN source
- Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
- Update solr_url on each of the CKAN sites with the suitable core
- Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

Same machine as the CKAN site:
- ~~test.ckan.net~~
- ~~iati.test.ckan.net~~
- data.gov.uk/ catalogue.data.gov.uk (confirm)

~~s046.okserver.org~~
- ~~iatiregistry.org~~

~~s052.okserver.org~~
- ~~datacatalogs.org~~

~~s004.okserver.org / eu4.okfn.org / solr.okfn.org~~
- thedatahub.org (=www.ckan.net)
- datagm.org.uk
- ~~publicdata.eu~~
- hri.fi (dev.fvh.fi?)
- it.ckan.net
- ie.ckan.net
- cz.ckan.net
- register.data.overheid.nl (nl.ckan.net?)
- no.ckan.net / datakilder.no
- br.ckan.net
- colorado.ckan.net
- at.ckan.net

TODO: which SOLR server are using these instances?

data.norge.no
nederland.ckan.net
lt.ckan.net
pl.ckan.net
datadotmontreal.ca/
ca.ckan.net / datadotgc.com

#1533: Make sure ckanext-wordpresser works on CKAN 1.5.1

amercader — Wed, 07 Dec 2011 17:07:40 GMT

Superceded by #2234

Super ticket: #1508

Etherpad: http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

#1540: Search API returns an error if empty parameters are provided

amercader — Thu, 08 Dec 2011 15:49:48 GMT

Both in 1.5.1b:

http://thedatahub.org/api/search/dataset?groups=lodcloud&title=

and 1.5.2a (current master):

http://test.ckan.net/api/search/dataset?groups=lodcloud&title=

Although the error message in 1.5.2a is more verbose:

"Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.ParseException?: Cannot parse 'groups:lodcloud title:': Encountered \"<EOF>\" at line 1, column 22. Was expecting one of: \"(\" ... \"*\" ... <QUOTED> ... <TERM> ... <PREFIXTERM> ... <WILDTERM> ... \"[\" ... \"{\" ... <NUMBER> ..."

Some parameter validation before sending it to Solr should do the trick

#1543: Pagination links in the dataset listings don't keep the current filters

amercader — Fri, 09 Dec 2011 14:57:03 GMT

E.g. Pagination links on this page don't include groups=lodcloud http://thedatahub.org/dataset?groups=lodcloud

Not sure if related to #1501 (probably not)

#1545: Remove external asset dependencies

amercader — Mon, 12 Dec 2011 15:10:35 GMT

CKAN is pulling a number of resources from external locations. This causes problems when connectivity is limited and you have to work locally. Maybe some of them cold be moved to CKAN source to avoid external requests.

Quick search:

./ckan/templates/layout_base.html:            <img src="http://assets.okfn.org/images/logo/okf_logo_white_and_green_tiny.png" id="footer-okf-logo" />
./ckan/templates/layout_base.html:            <a href="http://opendefinition.org/"><img alt="This Content and Data is Open" src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" style="border: none ; margin-bottom: -4px;"/></a>
./ckan/templates/package/resource_read.html:                <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/package/read.html:          <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /></a>
./ckan/templates/_util.html:                    <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/_util.html:                  <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      this.$dialog.html('<h2>Loading results...</h2><img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:          self.setMessage('Uploading file ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      self.setMessage('Checking upload permissions ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
Binary file ./ckan/lib/app_globals.pyc matches
./ckan/lib/app_globals.py:                                  'http://assets.okfn.org/p/ckan/img/ckan.ico')
./ckan/config/deployment.ini_tmpl:ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico

#1547: Add search methods (before and after) to the IPackageController interface

amercader — Tue, 13 Dec 2011 12:23:16 GMT

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

Modify the query being sent to Solr

def before_search(search_params):
    return search_params

Modify the results returned (after_search)

def after_search(search_results, search_params):
    return search_results

Please see this pad for implementation details:

http://ckan.okfnpad.org/search

#1590: Create customized feeds for the IATI Registry

amercader — Wed, 21 Dec 2011 17:02:45 GMT

We need a way to track changes on the registry (datasets edited or updated), globally and on a per country/publisher/etc. basis. RSS and Atom feeds are really popular, and after closing #191 and #1498 crating them from the search results should be fairly easy.

The following URLs are pretty self-explanatory:

http://localhost:5000/feed/registry.rss
http://localhost:5000/feed/country/AF.rss
http://localhost:5000/feed/publisher/worldbank.rss
http://localhost:5000/feed/organisation_type/10.rss

As we need to implement custom wrappers for countries, publishers, etc, we might as well offer a fully customizable feed, e.g.:

http://localhost:5000/feed/custom.rss?q=activity_count:[* TO 100]
http://localhost:5000/feed/custom.rss?publishertype=primary_source

Apart from the actual feeds, there will be a small amount of work at the template level to add the links to the suitable pages (and maybe a generic page showing all available feeds)

#1592: Add metadata_modified and metadata_created to package_dictize output

amercader — Wed, 21 Dec 2011 17:35:09 GMT

The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level.

#1593: [super] Create feeds from search results

amercader — Wed, 21 Dec 2011 17:38:35 GMT

Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:

http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*

On this case a message "Subscribe to these search results" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)

Other tickets related:

#1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.

#1616: Catch exceptions when rebuilding the search index

amercader — Fri, 06 Jan 2012 10:11:09 GMT

Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop.

#1641: ckanext-archiver: Content-length header not reliable to check if resource has been modified

amercader — Thu, 12 Jan 2012 13:53:40 GMT

The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated [1].

The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.:

$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/xml
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly
X-AspNetMvc-Version: 3.0
content-disposition: attachment; filename=AFG_IATI_12012012.xml
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2012 12:36:43 GMT

Also worth noting that requests, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.:

$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 12:12:46 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: application/xml
curl -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 11:56:23 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Content-Length: 2686720
Vary: Accept-Encoding
Content-Type: application/xml

All this can lead to some resources never getting updated, and of course the size property of the resource not being set.

As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it).

[1] https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72

#1654: [super] Update Publicdata.eu to the latest CKAN stable version

amercader — Mon, 16 Jan 2012 10:44:07 GMT

Tasks include:

#1813 Update ckanext-pdeu (4d)
#1814 Update harvesters (2d)
#1649 Verify ckanext-rdf works with latest CKAN (3d)
#1815 Reenable Sparql endpoint (?)
#1816 Update ckanext-apps (2-3d)

#1655: Setup issues on s025 (Publicdata.eu)

amercader — Mon, 16 Jan 2012 10:47:24 GMT

Time estimate: 2d

Fix logs (apache, ckan, harvest): rotate, set suitable levels
Fix harvesting jobs: supervisord for gather consumer, cron job
Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

#1677: Make synchronous search the default behaviour

amercader — Tue, 17 Jan 2012 13:40:04 GMT

Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour.

#1678: Make synchronous search the default behaviour

amercader — Tue, 17 Jan 2012 13:40:04 GMT

#1726: Update harvesting model

amercader — Tue, 31 Jan 2012 10:52:07 GMT

We need:

"current" column in harvest_object
"harvest_source_id" column in harvest_object, foreign key to harvest_source
"title" column in harvest_source

The extension should check on startup if the db needs updating and run the necessary scripts.

#1727: Add "current" property to harvest objects

amercader — Tue, 31 Jan 2012 10:54:14 GMT

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

Model update (See #1726)
Update queries for getting harvest sources' datasets
Update ckanext-csw queries for getting the objects
Update inspire and ckan harvesters
Document new field

#1728: Update UKLP harvesters

amercader — Tue, 31 Jan 2012 11:00:10 GMT

Three small changes:

Set Resource type to None instead of 'Undefined'
Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:

    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

#1731: Publisher support for harvesting

amercader — Tue, 31 Jan 2012 11:05:19 GMT

Estimate: 4-5d

Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources.

We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model)

Tasks include:

Auth checks on the different controller actions (and the command line interface?)
Modify the index page to just show sources the logged user has permissions on.

(Talk with Ross about the specific functions to use)

Create/edit form to show a dropdown with the user available publishers.

#1733: Delete (inactivate) harvest sources

amercader — Tue, 31 Jan 2012 11:11:27 GMT

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

Expose this on the web interface (only for sysadmins, see #1731)
Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

Publisher should create a new record with new identifier and same/different content as applicable.
If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
You won't get the state where a package is updated in its deleted state
If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

#1734: [super] Updates on harvesting for UKLP

amercader — Tue, 31 Jan 2012 11:14:34 GMT

Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources.

#1726 Update harvesting model
#1727 Add "current" field to harvesting objects
#1728 Update UKLP harvesters
#1733 Delete (inactivate) harvest sources

At some point we also want to support a publisher model for the harvesting:

#1731 Publisher support for harvesting

#1813: Upadate ckanext-pdeu to work with CKAN 1.6

amercader — Mon, 20 Feb 2012 16:58:48 GMT

Tasks include:

Estimate 4d

Update requirements (git)
Search facets error
Error on dataset read (extras in context)
Url generator for rdf
Make templates work with latest theme
Use dicts in templates (isopen())

#1814: Check publicadata.eu harvesters

amercader — Mon, 20 Feb 2012 17:03:28 GMT

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

#1816: Update ckanext-apps to work with CKAN 1.5.1 - 1.6

amercader — Mon, 20 Feb 2012 17:11:04 GMT

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

#2231: Base harvester broken on 1.6.1

amercader — Thu, 15 Mar 2012 12:44:59 GMT

On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1]

Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected

ValidationError: {'Id': u'The input field id was not expected.'}

We need a way to either:

Be more flexible accepting schemas on the logic functions
Modify the harvester plugin to implement the necessary interface

[1] https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115

[2] https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38

#2232: Unicode Exception when rebuilding the search index

amercader — Thu, 15 Mar 2012 13:03:02 GMT

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.

#2252: UKLP harvesters returning errors when importing due to schema changes

amercader — Wed, 21 Mar 2012 12:45:07 GMT

See trace below.

As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert
    converter(key, converted_data, errors, context)
  File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)

#2263: IATI: Get "Record updated" field from the registry

amercader — Mon, 26 Mar 2012 12:49:05 GMT

Don't provide a field in the form or spreadsheet import for this, just use dataset.metadata_modified.

#2271: IATI: Some datasets not being indexed

amercader — Mon, 02 Apr 2012 14:04:29 GMT

When importing a CSV file for a certain publisher, records don't come up on the search index. After some investigation this has tunrn out to be caused by the records being deleted. We need to investigate if this was caused by the import or just made by the users.

#2358: Finish off Release 1.7

amercader — Tue, 01 May 2012 15:43:50 GMT

Various stuff related with release 1.7:

Review patches sent to the release branch
Regularly update TDH with latest changes
write changelog, announce email, etc
...

#2359: Investigate pycsw integration

amercader — Tue, 01 May 2012 15:48:19 GMT

Tom Kralidis kindly told us on IRC [1] that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration.

[1] http://irclogs.okfn.org/ckan/2012-04-25.txt 17:16

#2360: Finish INSPIRE support proposal

amercader — Tue, 01 May 2012 15:50:09 GMT

I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on #2359

#2361: Install ckanext-googleanalytics and ckanext-stats on PDEU

amercader — Tue, 01 May 2012 15:51:16 GMT

ditto

#2366: Remove Authorization Groups

amercader — Tue, 01 May 2012 17:16:03 GMT

1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8.

#2388: Update "out of the box" settings for CKAN

amercader — Tue, 15 May 2012 15:21:09 GMT

Only logged in users able to create
Creation permissions have only creators able to update a dataset
Synchronous search is enabled (#1678)

#2389: More robust handling of DataExplorer errors

amercader — Tue, 15 May 2012 15:24:07 GMT

US-0a

Do not show failed preview with red-bar (esp when just empty datastore ...)

Data Viewer checks if datastore is empty nad falls back to ther options if emtpy
DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

Loading/showing error reports

#2400: Password reset fails

amercader — Wed, 16 May 2012 17:16:54 GMT

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

#2467: Fix stats extension frontend

amercader — Mon, 28 May 2012 13:53:50 GMT

After moving to Bootstrap the tables on the frontend look broken.

#2468: Finish off SlickGrid based Recline view

amercader — Mon, 28 May 2012 13:56:15 GMT

Please see these GitHub? issues:

https://github.com/okfn/recline/issues/130

https://github.com/okfn/recline/issues/131

#2505: Docs improvements for 1.7.1

amercader — Fri, 08 Jun 2012 14:42:13 GMT

There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1.

Add an introduction to harvesting support
Add an introduction to CSW support
Better upgrade docs: Some users suggested improvements to the upgrade docs on the list [1]. Also we should mention how the upgrade would work for point point releases
Source install: see #2406

[1] http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html

#2575: Licenses have i18n issues (was Inconsistent language on license dropdown)

amercader — Tue, 19 Jun 2012 17:10:45 GMT

License titles at least for some of them need translating. This wants to be done in a more efficient and simple manner than currently exists.

the original problem was:

The license dropdown on the dataset form shows titles in Swedish by default. Changing the language seems to default to English, but after refreshing the page a couple of times it shows a wrong language again.

Thanks to pudo for reporting.

#2604: Add Requests as requirement for tests

amercader — Wed, 27 Jun 2012 09:43:49 GMT

Requests (http://python-requests.org) is really useful to deal with http requests and saves a lot of time.

I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage.

#2639: Update recline version for 1.8 release

amercader — Mon, 09 Jul 2012 14:56:49 GMT

New recline version has significant improvements like the SlickGrid? based grid view, it'd be good to have it on 1.8.

#2640: Update recline on the demo site

amercader — Mon, 09 Jul 2012 14:58:31 GMT

Depends on #2639

#2642: Adapt harvest frontend to new theme

amercader — Mon, 09 Jul 2012 15:03:16 GMT

#2841: Slug generation fails on demo dev

amercader — Mon, 13 Aug 2012 11:00:53 GMT

On the add dataset form, slug generation fails with JS error:

Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning'    basic-form.js:5

#3022: setup_template_variables method of IDatasetForm never called

amercader — Thu, 22 Nov 2012 16:45:43 GMT

On the package controller the package_type is not passed to the lookup function, so the setup_template_variables defined on the extensions is never called

#1192: Convert CKAN Sphinx docs into admin/reference manual

annapowellsmith — Fri, 17 Jun 2011 15:29:53 GMT

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

index.rst - copy some stuff from README.txt, keep short
README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
[NEW] install.rst - new file on installation
[NEW] theming.rst - move over from wiki
api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
design.rst - REMOVE (can copy some over if you can be bothered to http://wiki.ckan.net/Vision (all philosophical stuff should go!))
loaders.rst: move to wiki.ckan.net/Using_Loaders
feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
importer.rst: remove (don't even copy)
forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) http://wiki.ckan.net/Customizing_Forms
forms-integration.rst: remove
model.rst: leave but move to reference (only for core devs)
load-testing.rst -> move to http://wiki.ckan.net/Load_Testing
distributed.rst -> remove
admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
deb.html -> go into reference (query james gardner on list about moving this to wiki?)

#368: 500 Server error when creating package

anonymous — Tue, 20 Jul 2010 10:41:02 GMT

I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response:

"publish: New Zealand Coastline (new-zealand-coastline)" opening connection to nz.ckan.net... opened <- "POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n" <- "{\"name\":\"new-zealand-coastline\",\"title\":\"New Zealand Coastline\"}" -> "HTTP/1.1 500 Internal Server Error\r\n" -> "Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n" -> "Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n" -> "Vary: Accept-Encoding\r\n" -> "Content-Type: text/html; charset=iso-8859-1\r\n" -> "Cache-Control: proxy-revalidate\r\n" -> "Content-Length: 640\r\n" -> "Connection: close\r\n" -> "\r\n" reading 640 bytes... -> "<!DOCTYPE HTML PUBLIC \"-IETFDTD HTML 2.0EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80</address>\n</body></html>\n" read 640 bytes Conn close

The server should not respond with a 500 error if the package was created.

#839: Package page redesign

anonymous — Thu, 02 Dec 2010 14:19:39 GMT

as per http://ckan.org/wiki/UIRedesignPackage

#2354: Front end performance improvements [super]

aron.carroll — Tue, 01 May 2012 15:10:42 GMT

Try and increase the page load times for the front end.

Now

Always include scripts at the bottom
Include CSS at the top
Load fonts with JS
Concatenate and minify JavaScript? files
Create a build script to load dependancies seperately
Enable Gzip compression on resources
Enable future cache headers on resources

Support serving assets from a CDN
Look into optimising the CSS - Remove large portions of unused CSS
Remove extreaneous libraries, can jQuery UI be replaced with bootstrap?

Tools

Webkit Inspector Audits
YSlow Firebug Plugin - http://yslow.org/
Speed Tracer - https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl

Continuous Integration

YSlow can be hooked up to Jenkins via Phantom JS

#2356: Localise JavaScript strings

aron.carroll — Tue, 01 May 2012 15:28:12 GMT

#2367: Style the analytics view counts on datasets

aron.carroll — Tue, 01 May 2012 17:19:49 GMT

Currently an eye and a number, needs to look a little nicer.

#2372: Create frontend style guide

aron.carroll — Thu, 03 May 2012 18:50:43 GMT

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

#2375: Create new base theme

aron.carroll — Tue, 08 May 2012 09:23:29 GMT

As designed by Sam Smith -> https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/ Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same.

#2417: Clean up output for dataset search results

aron.carroll — Wed, 23 May 2012 17:13:33 GMT

Currently due to the data structure the search result filters are output incorrectly. Repeating the facet with each tag.

See http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

It outputs:

Tags: fibre Tags: terrestrial

It should be:

Tags: fibre terrestrial

#2418: Facets disappear once selected

aron.carroll — Wed, 23 May 2012 17:15:30 GMT

Currently facets in the new demo site disappear from the sidebar once they are selected.

http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

Instead they should be marked with an "active" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works.