​http://s031.okserver.org:2375/dataset/resource-markdown/resource_edit/ba1b094d-697c-47c4-a72a-6c2557da5409

should have back to resource as a minimum

be good to do this in phase 3

This is what we need the JSON to look like. Taken from the Jed documentation (​http://slexaxton.github.com/Jed/).

{

      // The empty string key is used as the configuration
      // block for each domain
      "" : {
        // Domain name
        "domain" : "the_domain",

        // Language code
        "lang" : "en",

        // Plural form function for language
        "plural_forms" : "nplurals=2; plural=(n != 1);"
      },

      // Other keys in a domain contain arrays as values
      // that map to the translations for that key.
      // Gettext suggests that you use english as your keys
      // in case the key isn't found, and it can just pass
      // the value directly through.
      // Note: by convention, the 0-index location of the translations
      // is never accessed. It's just a thing, I guess.
      "a key" : [ null, "the translation", "the plural translations", ... ],

      // The plural form string is converted into a function
      // and the value that's passed into the gettext call
      // is passed into the plural form function. It's result
      // (plus 1) is the index that the translation comes from.

      // We're using sprintf interpolation on our keys so we can
      // then sub in the _actual_ values into the result.
      "%d key" : [ null, "%d key", "%d keys" ],

      // Contexts are keys that are just prefixed with a context string
      // with a unicode \u0004 as the delimiter.
      // You can use it for anything. Usually it's just for being content aware
      // in some way (e.g. male vs. female, product vs. category)
      "context\u0004%d key": [ null, "context %d key", "context %d keys" ]
    }

This looks like an appropriate python library to do the formatting. ​http://pypi.python.org/pypi/pojson/ but I'll leave to your discretion.

Finally an API endpoint that returns the appropriate request would save us making requests to the i18n directory.

/api/i18n/{locale}

For example if the current url is ​http://localhost:5000/pl/dataset then the client would request:

http://localhost:5000/api/i18n/pl

This should return the Content-Type: application/json and a 200 with an appropriate JSON body (this can be the language requested or an appropriate fallback). I think that if the fallback is the default language (in most cases English) then we just return a 204 No Content to save data.

on resource pages (e.g. ​http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360), same as old ckan for continuity and to highlight!

If you make your browser window short - say half screen height on a page with delete eg ​http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada

the dialog box is partially off the screen - it should try to stay on screen

Is this done I know checkboxes are a pain but it looks a bit out of place compared to the other items.

On the add dataset form, slug generation fails with JS error:

Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning'    basic-form.js:5

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

• We can say that you can add your own previews
• Every preview would have to define it's own dependencies
• External development (community creates new extensions?)

Questions to answer:

• Specification of the interface
• Precedence of previews for the same data types
• Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: ​http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

Requires us to url encode the tag names when displaying them ...

• Number of package page visits on ckan (can we get this straight from google analytics)
• Number of times url or download url is used - now ticket:937 (Record download stats for resoures)

How do we do this?

• Google analytics will miss a lot of this usage (and how do we get that data out anyway)
• Could use javascript but again misses usage.
• One option is to redirect link but that is kind of nasty (but may be only option ...)

As a User or Visitor

I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable.

Rating out of 5. One user or IP, one vote.

package_rating table:

package | user | rating annakarenina | joe.bloggs.openid.com | 4 annakarenina | 154.2.5.47 | 1

Get rid of modes in the RESTful API. Do json stuff directly in controllers.

• package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff.

Details of how to go about this here <​http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/>

Cost: 2h

Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons).

Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Attached is the ruby script used by Talis for conversion

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

1. Package needs a groups property (backref Group.packages).
2. Put groups in the search api.
3. Display them on the package, but not editable. (Only editable from the group page.)

Follow up to ticket:105 (Groups).

Cost: 1d

Create in the model basic operation of Access Control.

roles table

name | context | action
-----------------------
admin| package | edit
admin| package | edit-permissions
admin| package | read
editor| package | update
editor| package | read
reader| package | read
This data is set-up on db init and will have no interface.

user-roles table:

username | context_type | objectid | role
rgrp     | system  | n/a | admin
visitor  | package | * | reader
bob      | package | geonames | admin
visitor  | package | geonames | editor
visitor  | package | geonames | reader
john     | group | ukgov | admin
dread    | group | ukgov | editor
visitor  | group | ukgov | reader
This data will be added when someone is given permissions for the system, a package or a group.

Pseudo code:

class Package

def is_allowed(name, action):

is_allowed(name, action, context=self)

class Group

def is_allowed(name, action):

is_allowed(name, action, context=self)

def is_allowed(name, action, context=None): name: string - a username or IP for 'visitor'

action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system)

Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package.

Take out import of paginate in setup.py. Use paginate in webhelpers instead. Make changes to take account of any i/f changes.

Look for all places where model is accessed and check authorization is checked.

Document holes (and, as necessary, suggestions for fixes) as new tickets. Likely areas that need looking at:

• search i/f
• package WUI commit

Write holes are obviously much more significant to us than read holes.

Split out from ticket:43

• group form: description field for needs to be a text area
• group form: make title and name field wider
• package view: need to move tags and license above notes

As a

User

I want to

Create a new group via the WUI

Details

• locate at /group/new/
• You must be logged in to create a group
• Group creator is automatically given role of group admin
• Edit screen is same as edit screen ...

Write up search api documentation and put it in a template that shows up at api/search/.

Cost: 2h

• parse like data4nr code

Follow up tickets: ticket:149 and ticket:150

(instead of sqlalchemy interface, to facilitate general usage of the loader)

• use ckanclient

Follows on from ticket:148 and ticket:149

Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows:

Key           Value            delete
[ country   ] [ uk, spain   ]  [x]

When presenting the form, existing pairs and 3 blank pairs are provided.

When saving the data, keys that are not changed use the existing PackageExtra object (perhaps with new value). Deleted keys put the PackageExtra state to deleted. For new keys, only create a new PackageExtra if there isn't a state=deleted one to resurrect.

Validators ensure no repeated keys.

Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously)

Use a bit of javascript to add more dropdowns.

Suggested by Jonathan Gray

• Ensure you can add/change/delete extra fields from packages using the REST API.
• Update API docs to cover this.

On Package get, ensure separation of extra attributes from other attributes.

What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone

• Delete field if field value is None

Option 2 - user must supply all fields

• Delete field if field absent

Prefer option 1. Document this.

Follows on from ticket:152

Suggest form is: key:value (with optional space after ':'.

Keys would be for an object, i.e. 'package', 'tag', 'group'.

Becomes a link.

This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace.

Cost: 4h

Rather than raising a validation error automatically lower-case tags on submission.

Cost: 0.5h

Only first tag is updated.

(Related to ticket:169 - Package derivations)

A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.)

e.g. london-traffic-visualisation depends on road-map

'Dependency' relationship is:

• directional
• many:many
• stateful

'dependency' table columns:

• id (primary key)
• dependent (foreign key)
• dependency (foreign key)

Further tickets:

• WUI - package view - have list of dependencies (do not need to list packages which depend on this one)
• WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package')
• REST api - expose reading and writing 'depends on' property.

Issues

• How do we deal with dependency at a particular version?

Create /stats/ page displaying main statistics, e.g:

• Most highly rated packages
• Most edited packages
• Largest groups
• Top tags (by packages)
• Package addition rate
• Users with most packages

Related to ticket:181 - Stats side-bar

Details

• Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display)
• For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py
  • ​http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html

Provide RDF version of a package at the RDF URI: ​http://ckan.net/package/rdf/32000-naples-florida-businesses-kml

See email on OKFN help:

For the information to be Linked Data, the following URI (from the api.talis.com SPARQL call) should be resolvable to RDF, or an RDFa enhanced HTML page.

In two places in the WUI

• list of groups
• group page in title e.g. ukgov (5)

Cost: 1h

Remove the current dependency on Yahoo YUI toolkit for doing tag autocompletion on package edit forms.

The (KForge-derived) behaviour files for CKAN can easily be switched to the new jQuery-based ones: see ticket:212. Once this is done, it would make sense to implement the tag-selection code with jQuery.

Relevant pages include:

• ​http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/
• e.g. ​http://ckan.net/package/edit/acawiki

fabric script to automate ckan deployment

New field for resources - hash of the resource file.

• CKAN itself will not calculate the hash value - user just pastes it in.
• Display text field in resource table.
• Requires migration script.

Questions for the field's value:

1. Which hash to use? Restrict to python hashlib and other major hash libraries.

2. Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.

Integrate new test upgrade into testmigrate stuff.

New "migrate.ini" config file programmed with the db to be used for migrate tests and a param saying where the db dump is.

Need to do i18n of forms (not in templates but in code). Should consult with tonfa about this.

New domain object: PackageRelationship? (revisioned)

Attributes:

• subject (Package reference)
• object (Package reference)
• type (string)
• comment

Relationship type values: depends_on (dependency_of) derives_from (has_derivation) child_of (parent_of)

Relationship type is stored as a string, as given in the first column. The reverse relationship (bracketed) is given just for display purposes only.

Subject / Object - as in a sentence: "Dracos is derived from OSM" is <subject> <predicate> <object>. We'll use 'type' instead of 'predicate' as that is a more familiar word.

API:

• Appear in package listing Example: 'relationships': [{'is_dependency_of':'osm', comments:'Since version 0.2'}, {'is_parent_of':'bobs_maps'}]

Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted).

Using normal license field does not cause breakage but has no effect.

(Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install).

    ourjson = '''{
   "maintainer" : "Applications Division",
   "version" : "1999-01-01",
   "license_id" : 9,
   "name" : "geogratiswaterfraction1km_v100",
   "author_email" : "...",
   "author" : "Dissemination Branch",
   "tags" : [
      "GeoGratis",
      "Canada",
      "GCMD",
      "Hydrosphere",
      "Ground Water",
      "Surface Water",
      "Hydropattern",
      "Earth Science",
      "Land Surface",
      "Land Cover",
      "Landscape"
   ],
   "extras" : {
      "License" : "http://geogratis.ca/geogratis/en/licence.jsp",
      "Ministry" : "Natural Resources Canada",
      "Level of Government" : "Federal"
   },
   "maintainer_email" : "[email protected]",
   "notes" : "This product is a raster coverage",
   "url" : "http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67",
   "title" : "Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada"
}
'''
    def test_15_random(self):
        offset = '/api/rest/package'
        import urllib
        tmpjson = simplejson.dumps(simplejson.loads(self.ourjson))
        postparams = urllib.urlencode({tmpjson: 1})
        res = self.app.post(offset, params=postparams, status=[200],
                extra_environ=self.extra_environ)
        model.Session.remove()

This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type)

Cost: 0.5h

from ckan.model.types import JsonType

forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer

If the key to an extra field has a space in it then various form code doesn't work. Either need to disallow spaces or cope with them.

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

When you edit a package, all of its resources are deleted and a fresh set created. This causes all history of them to be lost.

Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a PackageResource?. This doesn't include changes to tags, extras and resources. These need to be added.

There is the complication that the 'diff' function doesn't currently work beyond PackageResource?. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now.

Rufus suggests alongside the existing methods:

pkg.all_revisions()
pkg.diff()

we have ones which also include the related objects (tags, extras, resources):

pkg.all_revisions_full()
pkg.diff_full()

You could have a function which returned these related objects. Ideas:

revobjlist = [ self, resources, extras ]
list of tuples: (revobj, changed_revobjs)
list of tuples: (revobj, {objtype: [changed_revobjs]})
tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...}