CKAN: Ticket Query

#1785: Replace 'Revisions' page with site-wide activity stream

seanh — Tue, 07 Feb 2012 16:34:18 GMT

Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?

#2336: Move Jenkins' install script into ckan core so it can be versioned

seanh — Mon, 30 Apr 2012 13:35:14 GMT

#2341: Fix Jenkins issue when testing branches that contain model changes

seanh — Tue, 01 May 2012 12:46:36 GMT

#2342: Fix Jenkins issue when testing branches with different solr schema versions

seanh — Tue, 01 May 2012 12:47:25 GMT

#2430: Look into marking tickets as dependent on other tickets in trac

seanh — Fri, 25 May 2012 11:58:06 GMT

#2431: Trac git integration

seanh — Fri, 25 May 2012 11:58:48 GMT

#2466: Fix spam handling on trac

seanh — Mon, 28 May 2012 13:51:43 GMT

#2475: Upgrade trac.ckan.org to trac 0.12.3

seanh — Tue, 29 May 2012 18:46:16 GMT

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

http://trac.edgewall.org/wiki/ChangeLog

#2478: Get NeverNotifyUpdaterPlugin installed on trac.ckan.org

seanh — Wed, 30 May 2012 16:52:14 GMT

#2516: Make 'Assign to:' field on trac.ckan.org into a dropdown list

seanh — Wed, 13 Jun 2012 09:07:22 GMT

there's a setting for this

#2537: Test and document ckanbuild

seanh — Fri, 15 Jun 2012 15:48:48 GMT

https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

#2538: Add multiple-instance support to ckanbuild

seanh — Fri, 15 Jun 2012 15:51:39 GMT

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

#2539: Investigate the existing ckan debian package for ckanbuild

seanh — Fri, 15 Jun 2012 15:54:21 GMT

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

#2540: Implement a way of upgrading ckan sites using ckanbuild

seanh — Fri, 15 Jun 2012 15:55:40 GMT

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

#2541: Add non-core extensions to ckanbuild

seanh — Fri, 15 Jun 2012 15:57:06 GMT

We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.

#2542: Create jenkins job to run ckanbuild, and run tests

seanh — Fri, 15 Jun 2012 15:58:08 GMT

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.

#1749: Allow creating activity details through API

seanh — Sun, 05 Feb 2012 18:10:06 GMT

Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts.

#1778: Replace classmethod's with module-level functions

seanh — Mon, 06 Feb 2012 15:53:34 GMT

In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the get() classmethods in the model/ classes.

Doing it with module functions lets us type module.function() instead of module.Class.method().

In many cases we're importing classes directly with from module import Class and then doing Class.method(), but from module import foo is bad style: http://docs.python.org/dev/howto/doanddont.html

#2234: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates

seanh — Fri, 16 Mar 2012 12:59:15 GMT

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

Feed items should probably be cached somewhere

Suggested

The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

Caching is pretty crucial and should probably obey the ttl of the feed.

#2335: Unicode characters don't work in CKAN ini files

seanh — Mon, 30 Apr 2012 12:42:06 GMT

You get a UnicodeDecodeError?, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example.

#2337: Sort out ini file confusion in docs

seanh — Mon, 30 Apr 2012 14:57:56 GMT

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

#2370: Add examples of all config settings in default config file

seanh — Thu, 03 May 2012 10:50:37 GMT

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

#2520: Document undocumented config options

seanh — Wed, 13 Jun 2012 11:52:48 GMT

There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either:

ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled

There are also some options that are in the default deployment.ini even though they're deprecated:

ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously

See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html

It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358

#2607: 'Upload a file' appears on resource form when storage not enabled

seanh — Wed, 27 Jun 2012 13:30:08 GMT

if the user tries to upload a file they will get "Failed to get credentials for storage upload. Upload cannot proceed"

Maybe add a test for it this time, this bug has appeared and reappeared before

#2619: Omit private datasets from public activity streams

seanh — Thu, 28 Jun 2012 11:49:00 GMT

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

#1635: Email notifications (e.g. for activity streams)

seanh — Wed, 11 Jan 2012 17:56:27 GMT

CKAN should be able to send email notifications to users.

Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users.

One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well.

Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about.

Analysis here: http://ckan.okfnpad.org/27

#1667: Add an extension point for rendering activity streams

seanh — Tue, 17 Jan 2012 10:29:11 GMT

Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML.

There needs to be an extension point where extensions can register their own rendering functions for particular activity types.

#1740: Get rid of `from module import ...`

seanh — Wed, 01 Feb 2012 11:06:09 GMT

It's really bad to do from module import * and CKAN has a lot of them. I suggest a three-pronged approach:

Don't add any more of them.

When you're programming if you see an easy opportunity to remove one then do so.

At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things.

We should also get rid of most or all of the from module import foo and from module import foo, bar statements.

I think the right thing to do is just import module and then use module.foo in your code. But if you find yourself doing module.foo.bar then you may have a code smell.

See: http://docs.python.org/dev/howto/doanddont.html

#1747: Expire old activities

seanh — Sun, 05 Feb 2012 12:56:29 GMT

Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits?

#1748: Make activity streams conform to http://activitystrea.ms/ standard

seanh — Sun, 05 Feb 2012 14:06:24 GMT

At the very least we should make our JSON output conform to their spec:

http://activitystrea.ms/

#1750: Move ckan/lib/activity.py into the model

seanh — Sun, 05 Feb 2012 18:18:08 GMT

Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers.

#1789: Implement a tag_update() logic action function

seanh — Thu, 09 Feb 2012 16:36:53 GMT

So users can rename a tag and/or move it between vocabularies using the API.

Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all.

What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary).

#1800: Tidy up _list() and _search() functions in ckan/logic/action/get.py

seanh — Thu, 16 Feb 2012 15:12:09 GMT

For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality.

#1824: Add vocabulary pages

seanh — Tue, 21 Feb 2012 17:24:49 GMT

For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar.

#2200: Add vocabulary_id option to tag_show() logic action function

seanh — Wed, 29 Feb 2012 16:41:18 GMT

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

#2287: Consistent datetime and time period strings localization

seanh — Thu, 12 Apr 2012 18:08:43 GMT

Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support.

For example, on the user index page right now there are time period strings like "less than 1 month" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly).

ISO date format everywhere might be a good idea.

Need to consider local timezone issues.

Python standard library and Babel may have useful helpers for this.

Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?

#2301: Replace old-style string formatting with format() method (at least in strings marked for translation)

seanh — Mon, 16 Apr 2012 12:27:44 GMT

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

#2426: Get rid of CKAN's formalchemy dependency

seanh — Fri, 25 May 2012 11:47:42 GMT

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

seanh — Fri, 25 May 2012 11:48:27 GMT

#2486: Should be able to use . in dataset names

seanh — Sat, 02 Jun 2012 16:46:23 GMT

#2508: Make it possible to run CKAN tests for each language

seanh — Mon, 11 Jun 2012 10:42:15 GMT

Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language.

This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc.

In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files.

#2585: Escape solr control characters in search queries, add advanced search screen

seanh — Fri, 22 Jun 2012 10:26:13 GMT

Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds".

So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?

#2622: Login fails in Opera 12

seanh — Thu, 28 Jun 2012 16:56:42 GMT

Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

#2625: Add i18n strings from non-core but supported extensions to ckan.pot file

seanh — Mon, 02 Jul 2012 13:48:23 GMT

Have to decide which non-core extensions are going to be supported first.

#2683: Add no-cache header to _tracking API call's response to make sure it doesn't get cached

seanh — Mon, 16 Jul 2012 13:49:37 GMT

#2761: Document all the errors you can get when setting up filestore, and how to fix them

seanh — Thu, 26 Jul 2012 11:36:06 GMT

Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev

#2762: test_related.py crashes

seanh — Thu, 26 Jul 2012 11:55:53 GMT

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

#2763: Multilingual tests failing

seanh — Thu, 26 Jul 2012 11:56:59 GMT

test_multilingual_plugin.TestDatasetTermTranslation?.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation?.test_group_read_translation both failing for me on master

#2771: Documentation and examples for IDatasetForm and IGroupForm

seanh — Fri, 27 Jul 2012 12:30:38 GMT

Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date.

#2815: db_to_form_package_schema() strips tracking summary, isopen

seanh — Wed, 08 Aug 2012 16:47:07 GMT

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

#2838: Context variables accepted by action functions need to be documented

seanh — Fri, 10 Aug 2012 20:35:11 GMT

I was doing this:

context = {'model': base.model, 'session': base.model.Session,
                    'user': toolkit.c.user or toolkit.c.author,
                    'extras_as_string': True}
group_dict = logic.get_action('group_show')(context,
                    {'id': group_id})

in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass 'extras_as_string': True in the context. I don't think this or other context variables are documented anywhere.

#2846: Organizations allows you to set an organization as the parent organization of itself

seanh — Mon, 13 Aug 2012 12:22:16 GMT

that doesn't make any sense

#2881: Remove or improve 'Domain Model' docs page

seanh — Tue, 21 Aug 2012 09:05:17 GMT

http://docs.ckan.org/en/ckan-1.8/domain-model.html

Currently not very useful. Also these pages:

http://docs.ckan.org/en/latest/domain-model-dataset.html

http://docs.ckan.org/en/latest/domain-model-resource.html

http://docs.ckan.org/en/latest/domain-model-task-status.html

This just seems very incomplete to me (surely there is more to the domain model the dataset, resource and task status?) and very likely to not get updated as changes are mode to the code. Would anyone seriously refer to this rather than just looking at the source code?

If we want to have something along these lines in the docs, I suggest adding docstrings to packages, modules, classes and methods and then using autodoc to pull them into sphinx.

#2882: Remove User Stories Overview from docs?

seanh — Tue, 21 Aug 2012 09:18:43 GMT

Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation?

http://docs.ckan.org/en/latest/user-stories.html

#2883: Add high level dev overview to 'For CKAN Developers' section of docs

seanh — Tue, 21 Aug 2012 10:52:35 GMT

Here's a draft: https://gist.github.com/3414107

#2884: Lock icon shown next to unknown licenses

seanh — Tue, 21 Aug 2012 16:01:13 GMT

Set the license of a dataset to blank (the first option) or to 'License Not Specified' or 'Other (Non-Commercial)' and a lock icon appears next to the license on the dataset read page. Shouldn't this be a question mark or something?

#2885: Labels (editor, admin) not translated in authorization pages for datasets or groups

seanh — Wed, 22 Aug 2012 08:49:11 GMT

#2887: "Welcome to CKAN!" on front page is untranslatable

seanh — Wed, 22 Aug 2012 09:17:59 GMT

ckan/templates/home/index.html contains:

<h1 class="page_heading">Welcome to ${g.site_title}!</h1>

which means that just "Welcome to" ends up in the ckan.po files for translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence.

#2889: Support JSON values in extras, when returning JSON from API

seanh — Fri, 24 Aug 2012 12:35:37 GMT

For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as:

/api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10

/api/rest/dataset/warandpeace

/api/action/package_show?id=warandpeace

The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects.

#2890: Collect data previews and data store docs in one chapter

seanh — Tue, 28 Aug 2012 10:20:36 GMT

Currently there is this page:

http://docs.ckan.org/en/latest/data-viewer.html

which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it.

Separately there is this page: http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api.

I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be:

CKAN has builtin previews of data resources on resource pages, enabled by default.

Images, Google Documents, and web page resources will be loaded into embedded iframes for preview.

Text-like files will be displayed raw.

CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer.

Additionally, you can enable CKAN's DataStore?, requires you to install ElasticSearch? and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data.

Does having DataStore? enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline,

You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd.

#2891: Update or remove DataStore google presentation

seanh — Tue, 28 Aug 2012 10:23:43 GMT

Currently this page:

http://docs.ckan.org/en/ckan-1.8/datastore.html

has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient.

Suggest moving the google presentation into the documentation proper, so it works with grep etc.

#2892: IDatasetForm's form_to_db_schema() breaks API?

seanh — Tue, 28 Aug 2012 11:42:05 GMT

It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict.

It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method.

#2893: Tags missing from form_to_db_package_schema()

seanh — Tue, 28 Aug 2012 12:11:04 GMT

With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this:

[ckan.controllers.api] Validation error: "{'type': 'Validation Error', 'junk': [u'The input field junk was not expected.']}"

They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active).

In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema.

#2894: Empty lists get stripped from package dicst in package_show()

seanh — Tue, 28 Aug 2012 12:40:34 GMT

If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active).

If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value.

Suspect this also effects packages with no tags, no groups, etc.

#2896: resource_create crashes on ValidationErrors with KeyError: 'resources'

seanh — Wed, 29 Aug 2012 16:29:14 GMT

ckan/logic/action/create.py:resource_create():

try:

pkg_dict = _get_action('package_update')(context, pkg_dict)

except ValidationError?, e:

errors = e.error_dictresources?[-1] raise ValidationError?(errors)

This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold.

#2897: db_to_form_package_schema() crashes historical dataset page

seanh — Thu, 30 Aug 2012 12:41:35 GMT

If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes.

Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like:

schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp':

[validators.ignore_missing, unicode]})

#2900: snippets/package_list.html template crashes on dataset that has no 'notes'

seanh — Thu, 30 Aug 2012 14:00:07 GMT

Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages.

Fix in the template? Or in the schema?

The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well

#2912: ckanext/organizations: Neither public or private initially selected when adding dataset

seanh — Mon, 10 Sep 2012 09:57:56 GMT

When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected.

#2913: IGroupForm has package_form() method

seanh — Tue, 11 Sep 2012 14:45:39 GMT

Should be group_form() I think

#2914: Add form_to_db_schema_options() to IDatasetForm and IGroupForm

seanh — Wed, 12 Sep 2012 10:29:55 GMT

This optional method is supported when creating or updating groups and packages and is used in DefaultGroupForm? and DefaultPackageForm?, but seems to be missing from the interfaces. Should be added with docstring.

#2915: Refactor form_to_db_schema_options()

seanh — Wed, 12 Sep 2012 10:37:49 GMT

Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)?

Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay.

The same thing goes for db_to_form_schema_options().

Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?

#2919: Remove IController from docs

seanh — Wed, 12 Sep 2012 14:47:33 GMT

Still in docs but seems gone from code

#2920: Genshi text template message extractor no longer needed?

seanh — Thu, 13 Sep 2012 09:57:29 GMT

In setup.py, do we still need:

('templates/.txt', 'genshi', {

'template_class': 'genshi.template:TextTemplate'

}),

in message_extractors? Doesn't like there are any txt files in there anymore.

#2921: Add docstring to top of lib/extract.py file

seanh — Thu, 13 Sep 2012 10:00:07 GMT

I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious.

Also a couple of other small fixes:

jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export.

jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future.

#2922: Better docstring for CKANInternationalizationExtension

seanh — Thu, 13 Sep 2012 10:01:47 GMT

I'm unsure about what's going on here. As I understand it, when we run python setup.py extract_messages it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files.

But then we have the parse() method of CkanInternationalizationExtension? also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice?

My guess is that CkanInternationalizationExtension? is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output?

Maybe CkanInternationalizationExtension? needs a better docstring saying what it does?

#2923: Change regularise -> regularize

seanh — Thu, 13 Sep 2012 10:03:02 GMT

The function is called regularise_html(), can't remember what file it's in.

#2924: Better docs for trans js command, and add to release process

seanh — Thu, 13 Sep 2012 10:04:31 GMT

Add a better docstring to for trans js command explaining what it does and why, and how to use it.

Also add this command to the CKAN Release Process as it's needed there

#2925: Remove trans mangle paster command?

seanh — Thu, 13 Sep 2012 10:06:08 GMT

Is trans mangle really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place.

#2926: I*Form tests for per-type templates

seanh — Thu, 13 Sep 2012 16:01:21 GMT

Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods).

The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions.

#2927: Test new package/group/organization type URLs support

seanh — Thu, 13 Sep 2012 16:02:33 GMT

IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet.

#2928: Run CKAN tests with example_i*form extensions enabled

seanh — Thu, 13 Sep 2012 16:05:34 GMT

Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process.

#2929: Remove is_fallback() from I*Form extension interfaces?

seanh — Thu, 13 Sep 2012 16:09:57 GMT

If a plugin wants to take over the default package type for example, it can return dataset? from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed?

Same for IGroupForm and IOrganizationForm

#2931: Better docstring for app_globals.py

seanh — Mon, 17 Sep 2012 14:13:14 GMT

The application's Globals object is not very informative.

#2932: Add docstring to system_info.py

seanh — Mon, 17 Sep 2012 14:50:29 GMT

What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table.

#2933: Update theming docs

seanh — Mon, 17 Sep 2012 15:21:12 GMT

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

#2936: Updating a group via the API clears its packages

seanh — Wed, 19 Sep 2012 09:47:46 GMT

If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work.

Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent

#2958: Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8

seanh — Thu, 04 Oct 2012 09:29:36 GMT

e.g. 2012_08(주요국가).xls

I tested in CKAN 2.0 and it seemed to work fine there.

#2964: Last organization admin can remove herself

seanh — Mon, 15 Oct 2012 10:14:18 GMT

If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it.

Last admin should not be able to remove or demote herself.

Also applies to groups.

#2966: 'Add' button text is wrong when editing organization members

seanh — Mon, 15 Oct 2012 10:15:49 GMT

e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe

#2967: Organization members edit page reloads after demoting self

seanh — Mon, 15 Oct 2012 10:17:07 GMT

Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page.

#2968: Anyone can access organization members page

seanh — Mon, 15 Oct 2012 10:19:15 GMT

The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all

#2969: Group members page 500s

seanh — Mon, 15 Oct 2012 10:20:54 GMT

The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s

#2970: Organization and group member links use id not name

seanh — Mon, 15 Oct 2012 10:22:11 GMT

e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar

#2971: "Are ytou sure you want to delete this member?" should say which member

seanh — Mon, 15 Oct 2012 10:27:20 GMT

when deleting members from groups and orgs

#2972: Remove any imports of authz.py and delete file

seanh — Mon, 15 Oct 2012 10:37:22 GMT

it is no longer used

#2973: Move new_authz.py into logic/auth/init.py

seanh — Mon, 15 Oct 2012 10:39:05 GMT

Makes sense to keep all auth stuff under logic/auth.

Also decide which functions from new_authz.pu should be marked private with _.

Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py

#2974: General of all auth functions

seanh — Mon, 15 Oct 2012 10:40:12 GMT

From Toby:

A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc.

#2975: Tests for auth functions and new actions

seanh — Mon, 15 Oct 2012 10:41:00 GMT

From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions

#2976: Polish group and organization member pages

seanh — Mon, 15 Oct 2012 10:41:49 GMT

From Toby: A little polish to the member pages and probably a little hardening of the controller and logic actions

#2977: Fix user autocomplete on group and organization member pages

seanh — Mon, 15 Oct 2012 10:42:29 GMT

From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it