CKAN: Ticket Query

#1503: Move from Bitbucket to Github

markbrough — Thu, 01 Dec 2011 13:11:24 GMT

Reasons: more people familiar with Git, bigger Open Source community on Github.

Keeping track of the move here: https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&hl=en_GB#gid=0

#1002: Remove changeset model code

rgrp — Wed, 23 Feb 2011 19:36:13 GMT

This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed).

#1124: push apt package python-ckanext-solr into our debian repository

nils.toedtmann — Wed, 04 May 2011 19:36:33 GMT

python-ckanext-solr is already available in http://apt-alpha.ckan.org/datanl-dev, but not yet in http://apt-alpha.ckan.org/debian (that is why we had to [pip-install it for DataGM). Please push into main repo.

#1415: Comments on current status of ckan deb packages

nils.toedtmann — Mon, 24 Oct 2011 11:51:09 GMT

This is a scratch pad ticket with some comments on the current status of our ckan deb packages. I know that some of it is the deb packaging roadmap anyway, please forgive me if i mention them here again.

Rufus and me re-deployed some community ckan instances onto s022 (see http://trac.okfn.org/ticket/926). We followed the documentation http://docs.ckan.org/en/latest/install-from-package.html

Deb package version number: the version of the deb package is "python-ckan 1309471251~149be76faabc+lucid-1", and it's hard to guess from there that it contains a ckan 1.4.2a
When is 1.4.3/1.5.x expected as deb package?
There was a bug in the DB upgrade script /usr/share/pyshared/ckan/migration/versions/029_version_groups.py (line 150) which looks like it was fixed 1.4.1==>1.4.2 but was nevertheless present in this deb package.
The current script /usr/bin/ckan-std-install
- does not set the Apache ServerName? according to the $INSTANCE variable
- automatically configures a ckan extension named after $INSTANCE
- depends on local postgres
- could be replaced with "/usr/bin/ckan-deploy --name=ckan-std --domain=ckan-std.localhost (see next point)
(i think this is exactly James' plan): have more generic deployment script /usr/bin/ckan-deploy as part of python-ckan which takes arguments like
- --domain=cc.ckan.net
- --aliases=$list-of-domains
- --name=cc (defaults to "domain")
- --no-db (does not configure a DB)
- --sql-alchemy=$DB_CONFIG_STRING (also runs "paster --plugin ckan db upgrade --config")
- --extension $list-of-extesions
- ...

#1445: Resource View page in WUI

rgrp — Mon, 07 Nov 2011 11:24:09 GMT

Super ticket: #1032

Locate at: /dataset/{dataset}/resource/{id}

See: http://wiki.ckan.org/Dataset_View_Page

Implemented in branch feature-1445-resource-view.

Still to do:

Add inline data explorer to page

#1450: Simplify dataset view page

johnglover — Mon, 07 Nov 2011 14:56:28 GMT

See http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal

Create a collapsible section for the README
Reduce the number of things on the page
Tidy up groups and tags in the sidebar
Resource view becomes a list of search results

(additional)

Resource navbar item links to first resource
Resource navbar has a dropdown menu of all resources in the dataset
Edit and Authz navbar items should be in a separate logical group to the right

Very rough summary: create more info section and move stuff out (relayout page e.g. README, tags etc)

#1453: Flexible tag names

icmurray — Tue, 08 Nov 2011 17:04:10 GMT

Allowing more flexible tag names:

allowing spaces
allow capital letters (search is case in-sensitive)
allow all punctuation except for commas and double-quotes '"'
allow unicode
commas delimit tag names in the package create/edit form

Effects:

package creation/edit form.
/tag/{tagname} uri
search action
api controller (search/package-create/edit)
web controller (search/package-create/edit)
search api documentation
autocomplete for tag names

#1478: change package_group to members table

kindly — Tue, 22 Nov 2011 11:06:13 GMT

This will comprise of the following steps.

model migration.
simplify revisioning
Remove any sqlalchemy relations from groups.
change logic functions

estimate 2d

#1480: Munge functions in util

dread — Tue, 22 Nov 2011 12:43:57 GMT

munge_title_to_name, munge_tag v. useful for client software importing things into CKAN. I want it for ONS.

#1487: Fix group ordering on homepage

kindly — Thu, 24 Nov 2011 00:24:40 GMT

ordering on homepage by name instead of group count

#1490: Standardize output from package listings coming from the logic layer

amercader — Fri, 25 Nov 2011 12:47:19 GMT

Right now, the two logic functions that return a list of packages (package_search [1] and group_packages_list [2])use custom functions to generate the output dict. That's suboptimal because:

The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts.
Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension.

In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the "isopen" property, needed by the template renderer.

[1] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685 [2] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442

#1497: Suppress benign warnings

dread — Tue, 29 Nov 2011 12:27:20 GMT

We really shouldn't have lots of benign stuff printed on stderr when we run every command. It confuses users and hides important stuff.

#1498: Support multiple SOLR schema versions

amercader — Tue, 29 Nov 2011 16:47:44 GMT

At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema. This process will involve:

Better documentation on how to deploy SOLR, as single instance or multicore
A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL.
(If possible) Checks in CKAN core that deal with different schema versions.

#1499: list/read/delete packages on the command-line

dread — Tue, 29 Nov 2011 18:34:16 GMT

It would be useful to be able to manipulate packages using the paster command-line interface. In particular listing, reading and deleting packages. Seeing which packages were deleted too.

#1455: Search results when 'all_fields' don't include 'extra' fields

dread — Wed, 09 Nov 2011 17:04:25 GMT

When you do a search like this:

http://thedatahub.org/api/search/package?q=tauberer+census&all_fields=1

the "extra" fields (e.g. "triples", "shortname") get missed off the results. The docs say it should be a "full record" and I don't see any reason why this is missed off.

This is a problem because search all_fields is the only way for clients and front-ends to get packages in bulk. They end up (like lodcloud) doing thousands of requests to get packages individually.

The full record is:

http://thedatahub.org/api/rest/dataset/2000-us-census-rdf
{"count": 1, "results": [{"res_description": ["Download", "XML Sitemap", "SPARQL enpdoint", "Example (RDF/XML)"], "name": "2000-us-census-rdf", "license": "Non-OKD Compliant::Creative Commons Non-Commercial (Any)", "author": "Joshua Tauberer", "author_email": "http://razor.occams.info/", "ckan_url": "http://thedatahub.org/dataset/2000-us-census-rdf", "notes": "2000 U.S. Census converted into over a billion RDF triples.\n\nPopulation statistics at various geographic levels, from the U.S. as a whole, down through states, counties, sub-counties (roughly, cities and incorporated towns)\n\nNotes: also found in the of SPARQL Endpoints.\n\nFrom home page:\n\n> * For the detailed Census statistics, you'll have to download the raw Census data files from the Census Bureau, my Perl script and the patch file below and run it yourself because the files are too big for me to offer as a download!\n> \n> * The data and scripts can be reused under Creative Commons Attribution-NonCommercial-ShareAlike.\n", "entity_type": "package", "site_id": "www.ckan.net", "download_url": "http://www.rdfabout.com/demo/census/", "indexed_ts": "2011-11-01T12:52:36.034Z", "url": "http://www.rdfabout.com/demo/census/", "state": "active", "title": "2000 U.S. Census in RDF (rdfabout.com)", "groups": ["lod", "lodcloud"], "res_format": ["", "meta/sitemap", "api/sparql", "example/rdf+xml"], "license_id": "cc-nc", "revision_id": "fcbad0de-79ea-41bd-8e01-eb832a05b732", "res_url": ["http://www.rdfabout.com/demo/census/", "http://www.rdfabout.com/sitemap.xml", "http://www.rdfabout.com/sparql", "http://www.rdfabout.com/rdf/usgov/geo/us/ny"], "id": "551ec435-f198-4d52-9b56-ec0b0be6aec9", "tags": ["census", "data", "demographics", "deref-vocab", "format-dc", "format-geonames", "format-politico", "format-rdf", "geographic", "linkeddata", "lod", "lodcloud.nolinks", "no-license-metadata", "no-provenance-metadata", "no-vocab-mappings", "population", "published-by-third-party", "rdf", "statistics", "us"]}]}

#1468: Update the search index for individual packages

thejimmyg — Tue, 15 Nov 2011 16:56:35 GMT

When I delete a package from the pylons shell I need to remove that package's entry from the Solr search index. That doesn't happen at the moment.

It would be useful if the search-index rebuild command could take a list of package IDs to rebuild.

#1479: Can't edit a user with a unicode email address

dread — Tue, 22 Nov 2011 11:14:46 GMT

Register User with an email address with a unicode char (e.g. u'\u044e')
View the User in the UI (/user/) or with 'user_show' Action API

Exception:

Module ckan.controllers.user:98 in read
<<          try:
                   user_dict = get_action('user_show')(context,data_dict)
               except NotFound:
                   h.redirect_to(controller='user', action='login', id=None)
>>  user_dict = get_action('user_show')(context,data_dict)
Module ckan.logic.action.get:488 in user_show
<<      check_access('user_show',context, data_dict)
           user_dict = user_dictize(user_obj,context)
           if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name):
>>  user_dict = user_dictize(user_obj,context)
Module ckan.lib.dictization.model_dictize:189 in user_dictize
<<
           result_dict['display_name'] = user.display_name
           result_dict['email_hash'] = user.email_hash
           result_dict['number_of_edits'] = user.number_of_edits()
           result_dict['number_administered_packages'] = user.number_administered_packages()
>>  result_dict['email_hash'] = user.email_hash
Module ckan.model.user:59 in email_hash
<<          if self.email:
                   e = self.email.strip().lower()
               return hashlib.md5(e).hexdigest()
           def get_reference_preferred_for_uri(self):
>>  return hashlib.md5(e).hexdigest()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128)

#1481: License not shown if no resources

dread — Tue, 22 Nov 2011 17:34:26 GMT

The license is not displayed any more, unless there are resources e.g. http://test.ckan.net/dataset/dtest#section-basic-information is OGL, but that's not shown.

#1482: Minor JS bugs

zephod — Tue, 22 Nov 2011 19:44:51 GMT

Three JS bugs have appeared:

The slug editor is not updating the text field again. [Fixed; commit:fadda17383]

The View Dataset page will stop executing javascript if there are no resources. [Fixed; commit:e54b089f6e]

The Add-Dataset and Add-Group URL editor shows the wrong URL for datasets and groups. Broken in this commit:

https://github.com/okfn/ckan/commit/5a01e67 by 'map.explicit=true'. JohnG: Can you look into this? [Fixed; commit:88e88380f6]

#1484: Creating package with duplicate tags causes exception

dread — Wed, 23 Nov 2011 12:40:46 GMT

e.g.

curl http://localhost:5000/api/action/package_create -d '{"name":"test5", "tags":[{"name": "tag6"}, {"name": "tag6"}]}' -H 'Authorization:tester'

Also broken in Web UI (adding a package).

Affects API and Web UI in 1.5 and I think all versions from 1.4.1

#1493: 'search-index rebuild/clear' doesn't work if no ckan.site_id

dread — Mon, 28 Nov 2011 12:47:02 GMT

You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank.

Should assert that one of these are set up.

#1500: Slug API calls tidy up

dread — Tue, 29 Nov 2011 20:58:05 GMT

API call documented but don't exist any more: dataset create_slug
API call not documented but exists: is_slug_valid
API call not documented, badly sited in API and using custom code in controller (not logic layer): /package/autocomplete

#1501: Pagination links broken

amercader — Wed, 30 Nov 2011 09:46:37 GMT

Most of the different pagination links are broken and point to wrong locations. e.g:

Dataset listings in group page:

http://test.ckan.net/group/bibliographic

Tag letters in tag listing:

http://test.ckan.net/tag

User listing:

http://test.ckan.net/user

Revision listing:

http://test.ckan.net/revision

This is most probably related with:

https://github.com/okfn/ckan/commit/1772a5c

#1502: Group package list is ordered by revision timestamp instead of alphabetically

johnglover — Wed, 30 Nov 2011 19:11:52 GMT

#1504: Action API given blank data causes exception

dread — Thu, 01 Dec 2011 16:25:48 GMT

This is what caused the exception:

curl http://localhost:5000/api/action/package_search -d='{"q": "osm"}'

The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary.

#1505: SearchError and SearchQueryError cause exception in Action API

dread — Thu, 01 Dec 2011 17:02:48 GMT

This query caused ckan to except because ckan/controllers/api.py doesn't catch SearchError? and SearchQueryError?:

curl http://localhost:5000/api/action/package_search -d '{"sort": "metadata_modified"}'

#1517: Accessing resource view sometimes auto-downloads the resource

zephod — Mon, 05 Dec 2011 18:27:27 GMT

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.