Using WUI or REST interface you can search packages and groups without authorization being checked.

On the REST interface you can also read all the attributes of the packages using the 'all-fields' option.

Can be fixed using more advanced query to check authz.

• Generic text box for markdown about the user 'About'

Model's user table reflects these:

• 'about' attribute

Suggested fields:

• id, name (username/profile name), openid, email, password, extras (JsonType?)
  • + current items: apikey, about, created)
  • current contents of name is usually openid and should migrate to openid field (could match based on starting with ​http://)
• May also want a fullname field (or perhaps that can

just go in extras - may be better to have its own field as useful to search on this )

Related changes (probably separate tickets):

• Improve user home page to include this additional information (and have it be editable)
• Use user 'name' (username) for home page links (rather than current user/<id>)
• Where we show lists of revisions, each revision author is now a link to the author's user page.

Related to ticket:136 and ticket:138.

Based on number of views.

Related to ticket:143.

Follows on from ticket:148. Sister to ticket:150.

Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows:

Key           Value            delete
[ country   ] [ uk, spain   ]  [x]

When presenting the form, existing pairs and 3 blank pairs are provided.

When saving the data, keys that are not changed use the existing PackageExtra object (perhaps with new value). Deleted keys put the PackageExtra state to deleted. For new keys, only create a new PackageExtra if there isn't a state=deleted one to resurrect.

Validators ensure no repeated keys.

Give guidance on what to put in 'author' and 'maintainer' fields (e.g. name? URL?)

Suggestion from Jonathan Gray

Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed.

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

Because people might try and do "tags:gov sweden" when "tag:gov sweden" makes more sense (sweden is searched for in all fields in these cases).

But in REST i/f search allow "tag"="gov" OR "tags"="gov sweden" in parameters.

Only first tag is updated.

A 'Derived' relationship can be applied from one package to another.

e.g. sussex-demography is derived from census-2001

'Derived' relationship is:

• directional
• many:many
• stateful

'derived' table columns:

• id (primary key)
• source_package (foreign key)
• result_package (foreign key)
• description (markdown text)

Further tickets:

• WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y
• WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate
• REST if - expose reading and writing this property

Use vdm diff functionality (see ticket:82) to implement diffing of packages between versions in WUI.

In package history you should be able to select to revisions and show the diff.

Estimated cost: 12h

'created' attribute on User, Group and Rating

datetime object: default=datetime.datetime.now()

See example on revision

Requires migrate script

This is docs for users of a CKAN service (e.g. ckan.net), as opposed to someone installing / administering the software. (The latter has docs on knowledgeforge.

Editability - wiki?

Make parts of the package form hidden by default, to make it look simpler. Users are invited to click a button to reveal the less important fields.

Fields hidden by default: version, author, author_email, maintainer, maintainer_email

Cost 7 days

When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API.

The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user.

Example at bottom of package page:

Leave a comment:

Comments:

Implementation details:

Comments table is with columns:

Cost - 2 days

The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09).

Example part of package:

Date: 5/10/09
or
Date: 1/1/09 – 1/4/09

rel=nofollow or robots.txt ?

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

• Just do JSON & CSV for now.
• Put this in CLI.
• Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

Check:

• high numbers of simultaneous users
• incomplete transactions
• etc.

In two places in the WUI

• list of groups
• group page in title e.g. ukgov (5)

Cost: 1h

Importer function is exposed to logged in users.

Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'.

• Basic is as it is currently
• Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords.

All fields should be in step with the government custom input form.

For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text.

Estimate:

• Set hover names on navigation links.
• Do we really need to display "About CKAN" everywhere? Just on home page.
• Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.

• allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them <label>s.
• cut 'only's -- use word 'filter' instead
• doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package.
• autocomplete package names & tags

• openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
• maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
• results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
• maybe links should go on the full titles? Get rid of package names.
• Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
• same comments as search listing (ticket:217)

Package page package/mypackagename

• Design
  • In general: a huge amount of visual and typographic noise: "Package:" is redundant.
  • "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
  • Openness/downloadability icons: need help, as discussed earlier
  • "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
    • Rating - alternative to GET on set_rating.
  • "Metadata:" technospeak, probably redundant.
  • <li> bullets add nothing.
  • Title, Version, URL can probably go in one line.
  • Resources can be made more prominent.
  • Author, author email and Maintainer, maintainer email can be merged.
  • Groups, tags in little boxes?
  • Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
  • Don't show extras unless there are any.

Package history page package/history/mypackagename

• "Revisions" is redundant
• "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
• Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.

Groups landing page group

• Yeesh! What's up with the typography for "Just login..."?
• Truncated description along with group name.
• Explanation text about what groups are for (RHS)
• Another case of <li> bullets adding nothing.

Repository History revision

• What is this page called? "Recent Changes", "Repository History", "/revision": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar?
  • Will change page title on /revision/ to Revision History. Will not change route for the time being.
• needs copyediting.
• Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.
• Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say "Older"/"More recent" rather than "Previous"/"Next". (wontfix: now have text saying we are showing most recent changes)
• Table layout is pretty ugly (yes, I'm aware this is my fault).
  • wontfix - nothing better at the moment
• Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail?
• Can we link to an author page? Yes!
• Atom feed should have a feed icon~~
• Why are we adding another <link rel="alternate"> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page.

New field for resources - hash of the resource file.

• CKAN itself will not calculate the hash value - user just pastes it in.
• Display text field in resource table.
• Requires migration script.

Questions for the field's value:

1. Which hash to use? Restrict to python hashlib and other major hash libraries.

2. Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (​https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

• Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
• Collect and map surface forms from all CKANs
• Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
• Add heuristics for format auto-detections:
  • Map well-known file extensions
  • Recognize obvious magic (Zip, Tar)
  • Peek into Zipfile/Tarfiles?
• Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
• See also: #816 (Autocomplete for the resource format field)

4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.

e.g. Text on ckan.net home page:

"All material available under an /licenseopen license"

Also seen on the package edit page: "/user/loginClick here to sign in"

I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime.

Add link to RDF representation of a package in our Atom feed.

New domain object: PackageRelationship? (revisioned)

Attributes:

• subject (Package reference)
• object (Package reference)
• type (string)
• comment

Relationship type values: depends_on (dependency_of) derives_from (has_derivation) child_of (parent_of)

Relationship type is stored as a string, as given in the first column. The reverse relationship (bracketed) is given just for display purposes only.

Subject / Object - as in a sentence: "Dracos is derived from OSM" is <subject> <predicate> <object>. We'll use 'type' instead of 'predicate' as that is a more familiar word.

WUI:

• Editable as part of package or separately? (e.g. like authz)
• Do we normalize to only one type name of the pair?
• Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?)

Add uuids into package in REST interface.

Use case

As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package.

Implementation

Add into the Atom feed:

1. a package is a new one
2. the resources have been changed
3. the last_updated field has been changed

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

• [namespace]: ? only if KeyValues? (for settings this would then always be settings)
• key
• label
• value (json)
• type (e.g. date and to specify in advance what type should be)
• description
• tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

• /ckan-admin/settings
• Show label, plus description plus text field

Depends

• Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost.

So for this case we should add the package's license to the list displayed.

Use case

As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion.

Suggested solution

In the package view side-bar, there is a note: "To have this package completely removed, contact the [ca.ckan.net administrators admin@…]."

Other solutions

A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill.

c.f. ​http://en.wikipedia.org/wiki/Deleting_an_article

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.

diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
 
         if not general_terms and \

This is caused by editing a package with resources over the REST interface.

Exception:

URL: http://de.ckan.net/api/rest/package/destatis
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:51 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:145 in update
<<                  fs = ckan.forms.group_fs_combined
                   fs = fs.bind(entity, data=request_fa_dict)
                   validation = fs.validate_on_edit(entity.name, entity.id)
                   if not validation:
                       response.status_int = 409
>>  validation = fs.validate_on_edit(entity.name, entity.id)
Module ckan.forms.package:385 in validate_on_edit
<<              temp_name = orig_pkg_name
                   self.data['Package-%s-name' % record_id] = u'something_unique'
               validation = self.validate()
               if temp_name:
                   # restore it
>>  validation = self.validate()
Module formalchemy.forms:175 in validate
<<          if self.readonly:
                   raise Exception('Cannot validate a read-only FieldSet')
               return AbstractFieldSet.validate(self)
       
           def sync(self):
>>  return AbstractFieldSet.validate(self)
Module formalchemy.forms:104 in validate
<<          success = True
               for field in self.render_fields.itervalues():
                   success = field._validate() and success
               # run this _after_ the field validators, since each field validator
               # resets its error list. we want to allow the global validator to add
>>  success = field._validate() and success
Module formalchemy.fields:763 in _validate
<<          try:
                   value = self._deserialize()
               except validators.ValidationError, e:
                   self.errors.append(e)
>>  value = self._deserialize()
Module formalchemy.fields:1354 in _deserialize
<<          if self.is_composite_foreign_key:
                   return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize()))
               return self.renderer.deserialize()
>>  return self.renderer.deserialize()
Module formalchemy.fields:183 in deserialize
<<          if self.field.is_collection:
                   return [self._deserialize(subdata) for subdata in self._serialized_value()]
               return self._deserialize(self._serialized_value())
       
           def _deserialize(self, data):
>>  return self._deserialize(self._serialized_value())
Module formalchemy.fields:167 in _serialized_value
<<          if self.field.is_collection:
                   return self._params.getall(self.name)
               return self._params.getone(self.name)
       
           def deserialize(self):
>>  return self._params.getone(self.name)
Module formalchemy.base:63 in getone
<<          if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage):
                   return v
               return v[0]
           def getall(self, key):
               v = dict.get(self, key)
>>  return v[0]
TypeError: 'int' object is unsubscriptable

Thanks to Friedrich for spotting this.