For example:

• name and title should have a higher weight than the notes field

Add tables described in ​http://ckan.okfnpad.org/notifications

Sub ticket of #1294

• Initial screen just title, name, description (notes), license (and tags?).
• State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff.
  • Flash some instructions? e.g. that resources are missing?
• Name slug should by default look uneditable (cf wordpress way to do the slug)

Subticket of #1294.

Can integrate from working system on ckanjs.

Only happens if the user does not update the password. Most probably introduced during #1229.

• Major issue with spam signup on thedatahub.org
• Suggest using recaptcha

Est cost: 1d

e.g. this particular revision on ckan.net: ​http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c (most others have not been approved) Causes 500 error.

We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in:

• 'with' statement requires from __future__ import with_statement
• import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5.

When the API returns any response it should always retunr JSON. For example a 404 response should return something like

{'status': 404, 'message': 'package somename not found'}

When errors are encountered when creating a package:

{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be
blank', 'description':'license code must be an integer']}

Not authorised:

{'status': 503, 'message': 'You are not authorised to create this package.'}

The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created.

When a customer logs in as a sysadmin then he/she see all packages, including deleted and pending ones. These are hidden to the average user, but the sysadmin has no idea of this until he clicks on the package and sees at the bottom 'state: deleted'.

It should be more obvious than that on the search view - an icon, message or crossed-out name to packages are deleted.

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

• introduce model.Site and c.site
  • site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
• Subdomain detector to activate sites.
• use site in Authorizer instead of System, have a NullSite? for global things
• allow cross-site search
• packages are in a list of sites, m:n rather than 1:n
  • list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

Child tickets:

• #234 UI Review - Autocomplete package names & tags in search
• #193 Searching by time-related field
• #191 Searching by modification date
• #905 Unable to search with accented characters in package names
• #906 Ability to search without accents for accented words
• #924 Search box has no search button

Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two.

SOLR search doesn't support searching for part of an extra field, but it does for other fields.

i.e. title="One Two Three" matches q=one AND q=title:one and geographic_coverage="England Scotland" matches q=England BUT NOT q=geographic_coverage:England

This problem emerged when we went to SOLR in #1275 (CKAN 1.5a). Tests were skipped.

This is could be a problem for DGU and maybe elsewhere.

Using curl you always get English, even if ckan.locale_default=de. Find out why. (1.5b)

Maybe we should disable locale detection for this release if lots of people's browsers don't have it set correctly (e.g. chinese with us internet explorer)

Super ticket: #1151 (viewing geo data)

E.g. preview of ​http://thedatahub.org/dataset/louisvillecrime should bring up a map

Summarise final set of requirements for this and finish development and test. Estimated 10 working days.

Can we finish this ready for release on data hub and CKAN Hosted.

Unsure what needs to happen here. Need to list outstanding tasks and implement.

Is testing complete and ready for release?

Can this be released as a standard CKAN feature?

Is this ready for release? What needs to be done?

Estimate 2 weeks for someone to finish and test.

Suggestions from rgrp:

• Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
• Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
• Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

(This ticket is split off from #1065)

We already have simple activity streams for groups that have new group or group updated events (see #1631). Smarter/more useful group activity streams would also contain the events for things related to the group: users, datasets, other groups, etc.

This should just mean using a more complex SQL query in the group_activity_list() logic function.

Things might get tricky when a user or dataset joins a group, this may show up as two events (the user or dataset was modified, and the group was modified). We may want to have two separate events for this (one to go in the user or dataset activity stream and one for the group), or maybe we just want one 'user/dataset joined group' event.

It's really bad to do from module import * and CKAN has a lot of them. I suggest a three-pronged approach:

1. Don't add any more of them.

2. When you're programming if you see an easy opportunity to remove one then do so.

3. At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things.

We should also get rid of most or all of the from module import foo and from module import foo, bar statements.

I think the right thing to do is just import module and then use module.foo in your code. But if you find yourself doing module.foo.bar then you may have a code smell.

See: ​http://docs.python.org/dev/howto/doanddont.html

Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts.

• Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?]
• Icons for *types* of resources in resource list
• List of 'Owners' (administrators) at top right of sidebar (a la group page)
  • Much prefer owner to administrator ...

From Pablo:

Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box.

For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed.

Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality.

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

• Integration with the form sections.
• Allow some degree of configuration (default extent, limit geometry types...)

[1] ​http://openlayers.org/dev/examples/draw-feature.html [2] ​http://openlayers.org/dev/examples/modify-feature.html

Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag.

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

• Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

• Feed items should probably be cached somewhere

Suggested

• The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

• For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

• Caching is pretty crucial and should probably obey the ttl of the feed.

• Customize text at bottom of forms
• Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

​http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

create permalinks

eg \dataset\~a5-DQ\nice-name

~XXXXX defines the permalink the nice-name is ignored

• packages table needs a new field for this data

• existing datasets need updating

• new datasets need the identifier creating

• routing needs updating

• needs to work for all urls like edit

Once this is implemented we can decide when we will use permalinks vs the normal urls.

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere

Implementation: (?) Use phantomjs or something like ​http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct.

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: ​http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

Solr allows its facet parameters to be specified on a per-field basis, eg. facet.limit applies to all facet fields, but solr allows it to be overriden for a specific field, eg. facet.tags.limit.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See ckan.lib.search.query.VALID_SOLR_PARAMETERS.

Install and document (step-by-step) CKAN (and dependencies, and extensions) onto VM on s031.okserver.org which has CentOS 6.2 installed. Paying particular attention to the fact that we won't have root access on the staging nor production servers, and so a) anything that requires root access must be documented carefully; and b) we want to be able to perform maintenance without being root.

Maintenance

We want to have non-root access to:

• CKAN logs; (relevant) apache logs; qa process' logs; rdf cron logs
• restarting CKAN
• restarting qa processes
• changing/disabling rdf cronjob
• our database (through sql)
• restarting solr

CKAN and Dependencies

• CKAN installed from source
• Required plugins: synchronous_search ecportal ecportal_form organizations multilingual_dataset multilingual_group multilingual_tag qa

Extra configuration

• uploads
• analytics (toby)
• qa
• rdf cron
• caching
• datastore

Things to remember

• Use ckanext/multilingual/solr/schema.xml and stopword files
• Use nginx as proxy.

Add caching config to ecportal-server-setup repository.

Server setup scripts need to have two distinct steps:

1) Download all files that aren't in RHEL repositories, create RPMs where necessary (python virtualenv).

2) Install from local files.

Have a news section (suggest as a sidebar item).

News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org.

Details:

• Suggest pulling via rss or similar.
• Will want to cache this ...

Cost: 4h?

Overview

Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

Examples

1. There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.

2. In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)

3. OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.

4. IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?

5. Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.

See more examples discussed here: ​http://trac.ckan.org/ticket/253

Implementation

This is split into four tickets:

• Model: ticket:254
• Read in WUI: ticket:255
• Edit in WUI: ticket:256
• API: ticket:257

No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies).

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

• [namespace]: ? only if KeyValues? (for settings this would then always be settings)
• key
• label
• value (json)
• type (e.g. date and to specify in advance what type should be)
• description
• tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

• /ckan-admin/settings
• Show label, plus description plus text field

Depends

• Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

Need to research what can easily be done to improve CKAN packages in the search rankings.

Comments from Glen Barnes:

We've been pretty successful at SEO without even really trying (see ​http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (​http://www.ckan.net/package/ascoe -> ​http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations.

Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column.

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

​http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.