We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided?

2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object
2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object
2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object
2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object
2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object
2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object

We default to sort packages search results by "search rank", but we can change it to be "name", "title", "metadata_modified" date, "metadata_created" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this.

Follows on from #191.

We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head.

Remove them and update any client systems.

Cost: 20m

We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible.

Spam:

• Package creation
• Package edit
• User creation
• Group creation

Systems to consider:

• Automatic
  • captcha
  • bayesian scoring
• Sysadmin
  • a tool to quickly analyse and remove packages, edits and users
• Helpful users
  • button to click indicating spam, leading to any/all of:
    • add TODO item (which emails package admin and sysadmins)
    • quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it

General thoughts:

• We should learn from the wikipedia tools, as they are the experts
• This is a serious problem that is only going to get worse and is a threat to the whole site.

We already have simple activity streams for groups that have new group or group updated events (see #1631). Smarter/more useful group activity streams would also contain the events for things related to the group: users, datasets, other groups, etc.

This should just mean using a more complex SQL query in the group_activity_list() logic function.

Things might get tricky when a user or dataset joins a group, this may show up as two events (the user or dataset was modified, and the group was modified). We may want to have two separate events for this (one to go in the user or dataset activity stream and one for the group), or maybe we just want one 'user/dataset joined group' event.

We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list.

• Add tags_create() function that calls the existing tag_create() many times?
• Also tags_delete()?
• Or just let tag_create() and tag_delete() accept a list?
• Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.)

We "just" need a schematron.validate(metadata) method. :-)

WUI:

• View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...)

WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes.

• Convert to wp twentyten html
• Switch css to be based off twentyten css (and perhaps rework somewhat)

Various things in templates can be tidied up after work has finished on #1294:

• remove styling in page in group/index.html (put in css main)
  • ditto for package/search.html
  • ditto for home/index.html
• Forms.css - to merge into style.css; remove explicit imports elsewhere
  • @import at the top of style.css @import url(forms.css)
• Make a note: Success/failure controllers clash CSS with success/failure flash banners

Various bugs I've been encountering:

• Autocomplete of tag names no longer works (no longer works on ​http://ckan.net/). Appears to be due to no longer have a routes for apiv2 (i'm seeing failing calls to: ​http://ckan.net/apiv2/package/autocomplete?callback=callback&incomplete=a)
• Incorrect url generated for API in page footer (e.g. ​http://ckan.net/rest/get_api) due to use of old 'rest' rather than new 'api'

Latter issue was masked by existence of 'default' routes:

   map.connect('/{controller}', action='index')
   map.connect('/:controller/{action}')
   map.connect('/{controller}/{action}/{id}')

Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap.

In addition I think we should be cautious about 'default' routes in core such as:

    map.connect('/api/rest/:register', controller='api', action='list',
        conditions=dict(method=['GET'])
        )

As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map).

Validation error is incorrect arguments, so should be a "400 Bad Request" response. "409 Conflict" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors.

Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms.

An AJAX call is made every N seconds, and the auto-saved blob is updated.

note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave.

Using datapkg a user can:

• list packages on CKAN
• get info about package on CKAN
• register and update a package on CKAN

Using ckanapi

Using REST interface you can list packages and groups without authorization being checked.

Can be fixed using more advanced query to check authz.

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

• We can say that you can add your own previews
• Every preview would have to define it's own dependencies
• External development (community creates new extensions?)

Questions to answer:

• Specification of the interface
• Precedence of previews for the same data types
• Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: ​http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

User story #035 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

User object should have a "created" attribute initialized to current datetime.

Require a db migration but o/w very simple.

Cost: 1.5h

User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error.

Need to improve filtering for anchors in markdown.

<a href="http://xxxsex.com>nasty/website

Also check this related exception:

Module ckan.controllers.user:59 in read
<<          c.is_myself = user.name == c.user
               c.api_key = user.apikey
               c.about_formatted = self._format_about(user.about)
               revisions_q = model.Session.query(model.Revision
                       ).filter_by(author=user.name)
>>  c.about_formatted = self._format_about(user.about)
Module ckan.controllers.user:167 in _format_about
<<      def _format_about(self, about):
               about_formatted = ckan.misc.MarkdownFormat().to_html(about)
               return genshi.HTML(about_formatted) 
       
           def _get_form_password(self):
>>  return genshi.HTML(about_formatted)
WebApp Error: <class 'genshi.input.ParseError'>: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3

Use story: As a user of the datasore I want to be able to put the resource id without the dataset name in a url and see the resource page.

Proposed redirect:

/resource/{res-id} -> dataset/foo-bar/resource/{res-id}

Use jquery chosen?

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

Use an IGroupForm plugin to add a custom metadata field to groups using convert_to_extras() and convert_from_extras(), when calling group show the value comes back quoted, e.g. '"my_value"'

Should add tests to example_igroupform and others that setting and getting the custom fields works through the action API.

Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options.

Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first

Update report queries with more recent versions (below are report queries shipped with trac 0.11.7):

NB: Report 1 has already been done.

id | query

1 |

: SELECT p.value AS color, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY CAST(p.value AS integer), milestone, t.type, time :

2 |

: SELECT p.value AS color, : version AS group, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time :

3 |

: SELECT p.value AS color,

: 'Milestone '

milestone AS group,

: id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time :

4 |

: : SELECT p.value AS color, : owner AS group, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time :

5 |

: SELECT p.value AS color, : owner AS group, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : description AS _description_, : changetime AS _changetime, reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time :

6 |

: SELECT p.value AS color, : t.milestone AS group, : (CASE status : WHEN 'closed' THEN 'color: #777; background: #ddd; border-color: #ccc;' : ELSE : (CASE owner WHEN $USER THEN 'font-weight: bold' END) : END) AS style, : id AS ticket, summary, component, status, : resolution,version, t.type AS type, priority, owner, : changetime AS modified, : time AS _time,reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), : (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC :

7 |

: SELECT p.value AS color, : (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS group, : id AS ticket, summary, component, version, milestone, : t.type AS type, priority, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE t.status <> 'closed' AND owner = $USER : ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time :

8 |

: SELECT p.value AS color, : (CASE owner : WHEN $USER THEN 'My Tickets' : ELSE 'Active Tickets' : END) AS group, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (COALESCE(owner, ) = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time

Unsure what needs to happen here. Need to list outstanding tasks and implement.

Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)

US-0a

Do not show failed preview with red-bar (esp when just empty datastore ...)

• Data Viewer checks if datastore is empty nad falls back to ther options if emtpy
• DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

Loading/showing error reports

Trying to run the tests in test_authz.py with

$ nosetests --ckan ckan/tests/functional/test_authz.py results in no tests being run:

Ran 0 tests in 0.840s

OK (SKIP=3)

Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

Try and minimise the amount of different caching methodologies, especially for api calls.

Tried to refresh the datagm harvest source to get error message:

"An error occurred: [There already is an unrun job for this source]"

This is unclear and not sure why it created an error. We should have something informative such as "A harvest has already been scheduled for this source"

Do we have a date/time for when this has been scheduled that we can display?

Tool that checks which packages have not been indexed.

Required for DGU: ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940

To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

• Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

• Write Middleware + update url_for to take account of the language. [2d]

• Document the language setup, and how to replicate it. [1d]

To reproduce:

1. Create a package with two tags the same: "bulk bulk"
2. Click 'save'
3. 500 ERROR - 'Server Error'

To reproduce:

1. paster db init
2. paster create-test-data
3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
4. curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
5. paster search-index rebuild
6. curl http://localhost:5000/api/search/package?groups=tilo results in 1 result

To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function.

This behaviour is now deprecated however.

To enable us to show DGU work in progress, for feedback.

To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form.

(Something to consider for the new form mechanism)

To allow navigation between sections to still work, also potentially rephrase to avoid confusion that you have to save every time to add a tag or resource and can't say add description, tag and resource in one go

Time estimate: 2d

• Fix logs (apache, ckan, harvest): rotate, set suitable levels
• Fix harvesting jobs: supervisord for gather consumer, cron job
• Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

Three small changes:

• Set Resource type to None instead of 'Undefined'
• Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

• Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:

    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

Thread: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

1. User installs Refine and CKAN extension for refine
2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
3. They edit the dataset on Refine
4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

1. User visits a package on CKAN.net (or another CKAN instance)
2. There is a button on the page "View and edit this dataset in Google Refine"
3. Click button -- ask them if they have Google refine installed
   • Yes: instructions for loading dataset into refine
   • No: load dataset in hosted version of google refine (we could run this)
4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type)

Cost: 0.5h

from ckan.model.types import JsonType

forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer

This will comprise of the following steps.

• model migration.
• simplify revisioning
• Remove any sqlalchemy relations from groups.
• change logic functions

estimate 2d