CKAN: Ticket Query

#3028: Feature: dashboard activity stream filtering

seanh — Mon, 10 Dec 2012 12:07:24 GMT

https://github.com/okfn/ckanext-pdeu/issues/13

#3020: Update CKAN coding standards

seanh — Fri, 16 Nov 2012 20:11:51 GMT

#3019: Cannot delete dataset extras

seanh — Wed, 14 Nov 2012 18:44:38 GMT

Deleting extras in the web interface is broken

#3014: Crash when deleting a non-empty vocabulary

seanh — Tue, 13 Nov 2012 10:50:08 GMT

From Knud Möller:

when I try to delete a non-empty tag vocabulary via the API (through HTTP), I get an internal server error. Checking the logs, this turns out to be a consistency error raised by sqlalchemy:

Error - <class 'sqlalchemy.exc.IntegrityError?'>: (IntegrityError?) update or delete on table "vocabulary" violates foreign key constraint "tag_vocabulary_id_fkey" on table "tag" DETAIL: Key (id)=(21421955-7560-467c-af30-9f790b73e6ae) is still referenced from table "tag".

'DELETE FROM vocabulary WHERE vocabulary.id = %(id)s' {'id': u'21421955-7560-467c-af30-9f790b73e6ae'}

URL: http://33.33.33.10:5000/api/action/vocabulary_delete

The error makes sense, but I'm wondering if it would be useful to extend the API to also allow the deletion of non-empty vocabularies, possibly via a parameter (not sure what best practice in API design is). At the very least, it would be cool if the error message coming back in the response had more information in it.

#3010: Pin images don't appear in data explorer

seanh — Wed, 31 Oct 2012 17:45:06 GMT

When minified files are in use (set debug = false in ini file) the pin images for points don't show up in the data explorer map view when viewing a resource with geolocation data.

set debug = true will fix the problem by using unminified files instead but also turns on a bunch of debug stuff you wouldn't want on a production site.

middleware.py, around line 73, is where the decision to turn minified files on or off based on the debug setting is made. This can be hacked on a production site to run with unminified files but otherwise run on production mode thereby getting around the issue with the pins. But that doesn't fix the underlying bug.

#3007: Adding a dataset creates multiple activities

seanh — Mon, 29 Oct 2012 13:15:07 GMT

Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity.

#3006: Update translations from Transifex

seanh — Mon, 29 Oct 2012 10:22:24 GMT

There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too.

#3005: Add following/unfollowing of groups: model, API, frontend, tests

seanh — Tue, 23 Oct 2012 17:35:44 GMT

#3001: Multilingual plugin crashes CKAN on add dataset when some languages are default

seanh — Thu, 18 Oct 2012 16:50:48 GMT

Enable the multilingual plugins:

ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag

and set your default language to one not supported by the multilingual plugin, e.g.

ckan.locale_default = cs_CZ

now run CKAN and try to add a dataset:

File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index

text_field_items+ default_lang?.extend(all_terms)

KeyError?: 'text_cs_CZ'

It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not.

A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes.

I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0

#3000: Add number of followers to dataset pages

seanh — Wed, 17 Oct 2012 14:36:00 GMT

User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend).

#2999: Add activity streams to organization pages

seanh — Wed, 17 Oct 2012 14:34:17 GMT

Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations.

#2998: Add activity streams to group pages

seanh — Wed, 17 Oct 2012 14:33:14 GMT

Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups.

#2997: Add activity streams to dataset pages

seanh — Wed, 17 Oct 2012 14:31:46 GMT

Add an activity stream tab to dataset pages, like we have on user profile pages. Dataset activity streams are already implemented in the backend.

#2996: Fix behaviour of View Profile button in user popover when already on user's profile

seanh — Wed, 17 Oct 2012 14:30:38 GMT

The user popovers that appear when you hover the mouse over a user in an activity stream include a View Profile button that takes you to that user's profile page. If you are already on that user's profile page then it simply reloads the same page, but it sends you to the Datasets tab if you were on another tab.

Possible solutions:

Don't show button if already on user's page?

Button just makes popover disappear, if already on user's page?

Button does reload the page, but reloads the same tab (datasets, followers, activity stream) that you were on.

Marking this low priority

#2995: Popovers for resources, groups, organizations, tags...

seanh — Wed, 17 Oct 2012 14:27:21 GMT

If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

It would be nice to add similar popovers for other types of object in CKAN, e.g. resources, groups, organizations, tags...

#2994: Add dataset and user popovers throughout the site

seanh — Wed, 17 Oct 2012 14:26:04 GMT

If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button.

These popovers should appear wherever datasets or users are listed, throughout the site.

#2993: "logged_in" and "visitor" show in user list at /users

seanh — Wed, 17 Oct 2012 09:42:02 GMT

#2991: Add blocks to header.html template

seanh — Tue, 16 Oct 2012 11:10:20 GMT

Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are.

#2990: Fix descriptions of groups and organizations on /groups and /organizations pages

seanh — Mon, 15 Oct 2012 11:04:51 GMT

#2989: "Add dataset to organization" should auto-select the organization

seanh — Mon, 15 Oct 2012 11:04:12 GMT

'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected.

I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the "Add dataset to group" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently.

#2988: UI functional tests for CKAN 2.0

seanh — Mon, 15 Oct 2012 11:02:15 GMT

Afaik the new CKAN 2.0 frontend has no functional tests

#2987: Remove all direct calls to logic action and auth functions

seanh — Mon, 15 Oct 2012 10:51:27 GMT

From Toby

#2986: Make lib/base.py more like it was originally intended

seanh — Mon, 15 Oct 2012 10:50:45 GMT

From Toby: make lib/base.py be more like it was originally intended eg defines stuff that is shared eg _ but maybe try to include stuff like render/get_action if the circular import stuff can be solved - which should be doable if hard and likely needs an extra file or two for the externals _ etc

#2985: Make lib/helpers.py more template-specific

seanh — Mon, 15 Oct 2012 10:49:48 GMT

From Toby: make lib/helpers more template specific and less generally shared with none template uses (some functions are shared but many shouldn't be) - again help avoid circular imports

#2984: Auto-populate context with user, model, session, etc.

seanh — Mon, 15 Oct 2012 10:48:21 GMT

From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action()

#2983: Refactor lib/base.py to remove circular import issues

seanh — Mon, 15 Oct 2012 10:47:24 GMT

From Toby: c) refactor lib/base.py to remove the circular import issues (render functions to lib/render.py - would fix much of that)

#2982: Move functionality from controllers into template helpers

seanh — Mon, 15 Oct 2012 10:46:42 GMT

From Toby: strip more functionality from controllers and add via template helpers eg activity streams

#2981: Remove config from jinja templates

seanh — Mon, 15 Oct 2012 10:46:00 GMT

From Toby: remove the config from jinja2 templates get all such stuff via g. (before 2.0)

#2979: Requesting membership to groups and organizations

seanh — Mon, 15 Oct 2012 10:44:04 GMT

From Toby: The requesting membership user stories and implementation remain needed

#2978: Tests for permissions for organizations and groups

seanh — Mon, 15 Oct 2012 10:43:26 GMT

From Toby: checking the logic around the who can do what from the user stories - via some tests may be a good approach

#2977: Fix user autocomplete on group and organization member pages

seanh — Mon, 15 Oct 2012 10:42:29 GMT

From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it

#2976: Polish group and organization member pages

seanh — Mon, 15 Oct 2012 10:41:49 GMT

From Toby: A little polish to the member pages and probably a little hardening of the controller and logic actions

#2975: Tests for auth functions and new actions

seanh — Mon, 15 Oct 2012 10:41:00 GMT

From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions

#2974: General of all auth functions

seanh — Mon, 15 Oct 2012 10:40:12 GMT

From Toby:

A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc.

#2973: Move new_authz.py into logic/auth/init.py

seanh — Mon, 15 Oct 2012 10:39:05 GMT

Makes sense to keep all auth stuff under logic/auth.

Also decide which functions from new_authz.pu should be marked private with _.

Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py

#2972: Remove any imports of authz.py and delete file

seanh — Mon, 15 Oct 2012 10:37:22 GMT

it is no longer used

#2971: "Are ytou sure you want to delete this member?" should say which member

seanh — Mon, 15 Oct 2012 10:27:20 GMT

when deleting members from groups and orgs

#2970: Organization and group member links use id not name

seanh — Mon, 15 Oct 2012 10:22:11 GMT

e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar

#2969: Group members page 500s

seanh — Mon, 15 Oct 2012 10:20:54 GMT

The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s

#2968: Anyone can access organization members page

seanh — Mon, 15 Oct 2012 10:19:15 GMT

The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all

#2967: Organization members edit page reloads after demoting self

seanh — Mon, 15 Oct 2012 10:17:07 GMT

Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page.

#2966: 'Add' button text is wrong when editing organization members

seanh — Mon, 15 Oct 2012 10:15:49 GMT

e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe

#2964: Last organization admin can remove herself

seanh — Mon, 15 Oct 2012 10:14:18 GMT

If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it.

Last admin should not be able to remove or demote herself.

Also applies to groups.

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

seanh — Thu, 04 Oct 2012 09:29:36 GMT

e.g. 2012_08(주요국가).xls

I tested in CKAN 2.0 and it seemed to work fine there.

#2936: Updating a group via the API clears its packages

seanh — Wed, 19 Sep 2012 09:47:46 GMT

If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work.

Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent

#2933: Update theming docs

seanh — Mon, 17 Sep 2012 15:21:12 GMT

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

#2932: Add docstring to system_info.py

seanh — Mon, 17 Sep 2012 14:50:29 GMT

What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table.

#2931: Better docstring for app_globals.py

seanh — Mon, 17 Sep 2012 14:13:14 GMT

The application's Globals object is not very informative.

#2929: Remove is_fallback() from I*Form extension interfaces?

seanh — Thu, 13 Sep 2012 16:09:57 GMT

If a plugin wants to take over the default package type for example, it can return dataset? from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed?

Same for IGroupForm and IOrganizationForm

#2928: Run CKAN tests with example_i*form extensions enabled

seanh — Thu, 13 Sep 2012 16:05:34 GMT

Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process.

#2927: Test new package/group/organization type URLs support

seanh — Thu, 13 Sep 2012 16:02:33 GMT

IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet.

#2926: I*Form tests for per-type templates

seanh — Thu, 13 Sep 2012 16:01:21 GMT

Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods).

The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions.

#2925: Remove trans mangle paster command?

seanh — Thu, 13 Sep 2012 10:06:08 GMT

Is trans mangle really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place.

#2924: Better docs for trans js command, and add to release process

seanh — Thu, 13 Sep 2012 10:04:31 GMT

Add a better docstring to for trans js command explaining what it does and why, and how to use it.

Also add this command to the CKAN Release Process as it's needed there

#2923: Change regularise -> regularize

seanh — Thu, 13 Sep 2012 10:03:02 GMT

The function is called regularise_html(), can't remember what file it's in.

#2922: Better docstring for CKANInternationalizationExtension

seanh — Thu, 13 Sep 2012 10:01:47 GMT

I'm unsure about what's going on here. As I understand it, when we run python setup.py extract_messages it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files.

But then we have the parse() method of CkanInternationalizationExtension? also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice?

My guess is that CkanInternationalizationExtension? is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output?

Maybe CkanInternationalizationExtension? needs a better docstring saying what it does?

#2921: Add docstring to top of lib/extract.py file

seanh — Thu, 13 Sep 2012 10:00:07 GMT

I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious.

Also a couple of other small fixes:

jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export.

jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future.

#2920: Genshi text template message extractor no longer needed?

seanh — Thu, 13 Sep 2012 09:57:29 GMT

In setup.py, do we still need:

('templates/.txt', 'genshi', {

'template_class': 'genshi.template:TextTemplate'

}),

in message_extractors? Doesn't like there are any txt files in there anymore.

#2919: Remove IController from docs

seanh — Wed, 12 Sep 2012 14:47:33 GMT

Still in docs but seems gone from code

#2915: Refactor form_to_db_schema_options()

seanh — Wed, 12 Sep 2012 10:37:49 GMT

Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)?

Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay.

The same thing goes for db_to_form_schema_options().

Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?

#2914: Add form_to_db_schema_options() to IDatasetForm and IGroupForm

seanh — Wed, 12 Sep 2012 10:29:55 GMT

This optional method is supported when creating or updating groups and packages and is used in DefaultGroupForm? and DefaultPackageForm?, but seems to be missing from the interfaces. Should be added with docstring.

#2913: IGroupForm has package_form() method

seanh — Tue, 11 Sep 2012 14:45:39 GMT

Should be group_form() I think

#2912: ckanext/organizations: Neither public or private initially selected when adding dataset

seanh — Mon, 10 Sep 2012 09:57:56 GMT

When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected.

#2900: snippets/package_list.html template crashes on dataset that has no 'notes'

seanh — Thu, 30 Aug 2012 14:00:07 GMT

Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages.

Fix in the template? Or in the schema?

The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well

#2897: db_to_form_package_schema() crashes historical dataset page

seanh — Thu, 30 Aug 2012 12:41:35 GMT

If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes.

Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like:

schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp':

[validators.ignore_missing, unicode]})

#2896: resource_create crashes on ValidationErrors with KeyError: 'resources'

seanh — Wed, 29 Aug 2012 16:29:14 GMT

ckan/logic/action/create.py:resource_create():

try:

pkg_dict = _get_action('package_update')(context, pkg_dict)

except ValidationError?, e:

errors = e.error_dictresources?[-1] raise ValidationError?(errors)

This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold.

#2894: Empty lists get stripped from package dicst in package_show()

seanh — Tue, 28 Aug 2012 12:40:34 GMT

If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active).

If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value.

Suspect this also effects packages with no tags, no groups, etc.

#2893: Tags missing from form_to_db_package_schema()

seanh — Tue, 28 Aug 2012 12:11:04 GMT

With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this:

[ckan.controllers.api] Validation error: "{'type': 'Validation Error', 'junk': [u'The input field junk was not expected.']}"

They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active).

In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema.

#2892: IDatasetForm's form_to_db_schema() breaks API?

seanh — Tue, 28 Aug 2012 11:42:05 GMT

It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict.

It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method.

#2891: Update or remove DataStore google presentation

seanh — Tue, 28 Aug 2012 10:23:43 GMT

Currently this page:

http://docs.ckan.org/en/ckan-1.8/datastore.html

has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient.

Suggest moving the google presentation into the documentation proper, so it works with grep etc.

#2890: Collect data previews and data store docs in one chapter

seanh — Tue, 28 Aug 2012 10:20:36 GMT

Currently there is this page:

http://docs.ckan.org/en/latest/data-viewer.html

which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it.

Separately there is this page: http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api.

I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be:

CKAN has builtin previews of data resources on resource pages, enabled by default.

Images, Google Documents, and web page resources will be loaded into embedded iframes for preview.

Text-like files will be displayed raw.

CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer.

Additionally, you can enable CKAN's DataStore?, requires you to install ElasticSearch? and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data.

Does having DataStore? enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline,

You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd.

#2889: Support JSON values in extras, when returning JSON from API

seanh — Fri, 24 Aug 2012 12:35:37 GMT

For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects.

#2887: "Welcome to CKAN!" on front page is untranslatable

seanh — Wed, 22 Aug 2012 09:17:59 GMT

ckan/templates/home/index.html contains:

<h1 class="page_heading">Welcome to ${g.site_title}!</h1>

which means that just "Welcome to" ends up in the ckan.po files for translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence.

#2885: Labels (editor, admin) not translated in authorization pages for datasets or groups

seanh — Wed, 22 Aug 2012 08:49:11 GMT

#2884: Lock icon shown next to unknown licenses

seanh — Tue, 21 Aug 2012 16:01:13 GMT

Set the license of a dataset to blank (the first option) or to 'License Not Specified' or 'Other (Non-Commercial)' and a lock icon appears next to the license on the dataset read page. Shouldn't this be a question mark or something?

#2883: Add high level dev overview to 'For CKAN Developers' section of docs

seanh — Tue, 21 Aug 2012 10:52:35 GMT

Here's a draft: https://gist.github.com/3414107

#2882: Remove User Stories Overview from docs?

seanh — Tue, 21 Aug 2012 09:18:43 GMT

Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation?

http://docs.ckan.org/en/latest/user-stories.html

#2881: Remove or improve 'Domain Model' docs page

seanh — Tue, 21 Aug 2012 09:05:17 GMT

http://docs.ckan.org/en/ckan-1.8/domain-model.html

Currently not very useful. Also these pages:

http://docs.ckan.org/en/latest/domain-model-dataset.html

http://docs.ckan.org/en/latest/domain-model-resource.html

http://docs.ckan.org/en/latest/domain-model-task-status.html

This just seems very incomplete to me (surely there is more to the domain model the dataset, resource and task status?) and very likely to not get updated as changes are mode to the code. Would anyone seriously refer to this rather than just looking at the source code?

If we want to have something along these lines in the docs, I suggest adding docstrings to packages, modules, classes and methods and then using autodoc to pull them into sphinx.

#2846: Organizations allows you to set an organization as the parent organization of itself

seanh — Mon, 13 Aug 2012 12:22:16 GMT

that doesn't make any sense

#2838: Context variables accepted by action functions need to be documented

seanh — Fri, 10 Aug 2012 20:35:11 GMT

I was doing this:

context = {'model': base.model, 'session': base.model.Session,
                    'user': toolkit.c.user or toolkit.c.author,
                    'extras_as_string': True}
group_dict = logic.get_action('group_show')(context,
                    {'id': group_id})

in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass 'extras_as_string': True in the context. I don't think this or other context variables are documented anywhere.

#2815: db_to_form_package_schema() strips tracking summary, isopen

seanh — Wed, 08 Aug 2012 16:47:07 GMT

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

#2771: Documentation and examples for IDatasetForm and IGroupForm

seanh — Fri, 27 Jul 2012 12:30:38 GMT

Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date.

#2763: Multilingual tests failing

seanh — Thu, 26 Jul 2012 11:56:59 GMT

test_multilingual_plugin.TestDatasetTermTranslation?.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation?.test_group_read_translation both failing for me on master

#2762: test_related.py crashes

seanh — Thu, 26 Jul 2012 11:55:53 GMT

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

#2761: Document all the errors you can get when setting up filestore, and how to fix them

seanh — Thu, 26 Jul 2012 11:36:06 GMT

Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev

#2683: Add no-cache header to _tracking API call's response to make sure it doesn't get cached

seanh — Mon, 16 Jul 2012 13:49:37 GMT

#2625: Add i18n strings from non-core but supported extensions to ckan.pot file

seanh — Mon, 02 Jul 2012 13:48:23 GMT

Have to decide which non-core extensions are going to be supported first.

#2622: Login fails in Opera 12

seanh — Thu, 28 Jun 2012 16:56:42 GMT

Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

#2619: Omit private datasets from public activity streams

seanh — Thu, 28 Jun 2012 11:49:00 GMT

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

#2607: 'Upload a file' appears on resource form when storage not enabled

seanh — Wed, 27 Jun 2012 13:30:08 GMT

if the user tries to upload a file they will get "Failed to get credentials for storage upload. Upload cannot proceed"

Maybe add a test for it this time, this bug has appeared and reappeared before

#2585: Escape solr control characters in search queries, add advanced search screen

seanh — Fri, 22 Jun 2012 10:26:13 GMT

Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds".

So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?

#2542: Create jenkins job to run ckanbuild, and run tests

seanh — Fri, 15 Jun 2012 15:58:08 GMT

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.

#2541: Add non-core extensions to ckanbuild

seanh — Fri, 15 Jun 2012 15:57:06 GMT

We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.

#2540: Implement a way of upgrading ckan sites using ckanbuild

seanh — Fri, 15 Jun 2012 15:55:40 GMT

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

#2539: Investigate the existing ckan debian package for ckanbuild

seanh — Fri, 15 Jun 2012 15:54:21 GMT

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

#2538: Add multiple-instance support to ckanbuild

seanh — Fri, 15 Jun 2012 15:51:39 GMT

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

#2537: Test and document ckanbuild

seanh — Fri, 15 Jun 2012 15:48:48 GMT

https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

#2520: Document undocumented config options

seanh — Wed, 13 Jun 2012 11:52:48 GMT

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

seanh — Wed, 13 Jun 2012 09:07:22 GMT

there's a setting for this

#2508: Make it possible to run CKAN tests for each language

seanh — Mon, 11 Jun 2012 10:42:15 GMT

Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language.

This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc.

In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files.