Should describe implementing IDatasetForm, subclassing controllers is deprecated

Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop.

Polish off comments extension dev and test. estimate 2 days.

The logic test did not have init. This caused lots of tests to fail because there were mock extensions that ran automatically in them. Fix plugin system so this can work.

Apparently it's working when developers run it locally but not on the demo site.

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

• On the resources tab, there's a "Check Resources" button which, when clicked makes an ajax request with the list of URLs entered by the user.

• The server checka each link for errors and header information about the linked resource. (Using ckanext/archiver/tasks.py:link_checker()).

• The server returns a list of dicts (json), containing information about the linked resource, and the client uses that to:

• populate the format field of each resource

The (guessed) 'file_extension' populates the 'format' field. If it's 'htm' or 'html', then we assume it's a listing page, and so don't populate the format field with 'htm' or 'html'.

• provide feedback if a URL appears to be invalid

If the URL doesn't appear to be a URL at all, or returns a HTTP error, or times-out, then URL field is highlighted in red to indicate it's a bad URL. A tooltip shows the error message to the user.

• [Optional] provide feedback if a URL appears to point to "Additional Information" - ie the Resource should be entered under "Additional Resources", rather than "Timeseries" or "Individual" datasets.

Analysis and further description on etherpad: ​http://ckan.okfnpad.org/dgu-package-form? [Section I]

[Child of #1526]

Provide the interface for the themes and subthemes hierarchy.

IGroupForm should allow overriding of non-edit templates so that it can specify the index +/or search +/or view.

User names and descriptions are not translated.

It is possible to create a group or a dataset called new, or edit, which will then be inaccessible as it is blocked by the existing routes with those names.

We should maintain a list of invalid names in the verification that happens behind the scenes in the UI.

It is necessary to add a helper method to Group for adding members to that group so that

• nobody needs to work with the Member model
• It handles existing membership and existing membership with a different capacity

Main tasks:

• Integrated seamlessly into CKAN (authorization etc)
• Based off ElasticSearch?

Etherpad: ​http://ckan.okfnpad.org/webstore

Tasks

• New API point (config + controller) [0.75d]
  • /api/data/{resource-id}
• Authorization [0.5d]
• Documentation [0.25d]
• Upgrade data viewer to use ES [0.5d]
• Convert webstorer to new setup - ​http://github.com/okfn/ckanext-webstorer [1d] - #1753
  • See below
• "Create DataStore? Resource" in Resource creation (??)
  • Just creates a resource with blank url and datastore enabled (webstore_url non null)
• Tick box to enable DataStore? in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)
  • Also need to use this info in DataStore? extension?
• Create nice instructions on Resource page about using the DataStore?
• Rename webstore -> datastore
• Deploy [0.5d]

Details

Layout on ES

elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)

Request path

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: ​https://github.com/okfn/elastic-proxy

Questions

1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy?
   • Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
2. How do we decide when webstore is activated
   • Ans: see previous.
3. Do we (and how do we) allow for resources which are just for webstores
   • Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ...
4. How does webstorer work ...
   • CSV import - NS: make sure we add e.g. rownum or id = rownum ...
   • Separate mini-service (webstorer.thedatahub.org ...)
5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy?)
   • Ans: do not bother at present. Focus on Webstore route.
6. Where does geodata support come in (in Recline or separately)
   • This is a separate issue

User Story 1

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

1. Allow normal upload process
2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
3. Update the resource in some way (set webstore_url)

User Stories

User Story 2

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)

User Story 3

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

All data is that is translatable should be translated when viewed.

This includes tag names will be ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language.

Related to the dataset view extension points for translation, recently added by kindly/=.

Get buildbot to:

• build the deb packages
• install them into a fresh virtual machine
• run smoke tests on the installed ckan

Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.

This extension point needs to combine.

• Making sure the correct data get put in search index.
• Make the search queries are modified to weight multilingual search correctly.
• View pages, including search results are translated.
• Bring together all search schema modifications.

Data viewer / previewer for a resource that has a KML file, or others, such as GeoRSS and GeoJSON, we should provide a map.

• Support for KML files: #1458

This is non-trivial for external files as we need a way to jsonify. For files stored locally this is more of a recline issue (and will require a bit of work to either guess columns or allow user to specify them).

For EC Portal need a page (or HTML snippet embedded in the front page?) listing recently added or updated datasets.

• Could be generated from the activity streams

• Should it list datasets only, or other activities as well?

• May be related to #1785 (Replace 'Revisions' page with site-wide activity stream)

As a

user browsing a CKAN package which has a resource in a bad format

I want to

be alerted to the possibility of scraping it on scraperwiki

Implementation

This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it
Image: (Scraperwiki dumper truck)
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can ​request this data is scraped or even ​write the scraper yourself.

Historical Parent Tickets

• Parent-ticket: #1450
• Parent-ticket: #1032

Step Zero

• COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

• Force format field to be lowercase
  • Modify backend to serve it up in that format
• Subticket #978: Add extra fields to resources in Web UI
• Fix display of large datasets (​http://test.ckan.net/dataset/europeana-lod)
• When submitting forms with errors, open the correct resource pane.
• No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
• Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

• Subticket #1648: Clarify edit terminology
• Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
• Fix (i.e. allow / show) name editing on dataset edit.
  • Fix the slugify so it does not overwrite if i change title after fixing name.
  • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
• Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

• Fix animation glitch in expanding description
• Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
• Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

• "Upload or link data files, APIs and other materials related to your dataset."
• Include the description markdown editor again (keep it small)
• Licence should maybe be restored
  • "(Don't worry if you don't know what your licence is)"
• Show the log message field
  • Beautify
• Pre-populate the log message field?
• ? Show Home Page field when creating dataset [WONTFIX don't push it]
• ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

• Improvements to user page UX - 1d - #1396
• Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

• Take before and after screenshots

Super ticket: #1506

Need to decide which term to use and then have the same for editing as well as viewing a dataset.

In creating/editing a dataset, want more explanation about adding extra fields (probably as a tooltip or similar).. i.e. that this let's you add extra custom metadata such as 'location: uk' which is then searchable etc

Some request exceptions are currently not being caught (see the celery log on thedatahub for examples)

Super ticket: #1506

• Show edit summary form again, at least on edit pages (and prefill it for create to something sensible like: Creating dataset {title} {name}) [Done #1506]
• Fix drop down for resource types in resource add/edit to limit to fixed set of options (as per ​http://wiki.ckan.org/Domain_Model/Resource)
  • See also ​http://lists.okfn.org/pipermail/ckan-discuss/2011-December/001860.html
• Allow reordering of resources (or: and this is somewhat more complex but possibly simpler: allow editing of a resource_order field and we use that to sort resources (this is more like wordpress page_order -- these two options could become the same if we just hijack the existing position field on the join table and get rid of ordereredlist extension)) [DONE #1506]
• Radically simplify set of licenses [DONE #1359]
• "Unsaved changes" warning briefly when you click "Save" #1735
• "Unsaved changed" typo on dataset edit page

e.g. ​http://189.9.137.65/dados/ clicking on Deutsch link is http://189.9.137.65/dados/locale?locale=de&return_to=%2Fdados%2F&hash=1dc17c315c419df850da0dd3599eefa9da76fbeb and redirect goes to http://189.9.137.65/dados/dados/?__cache=97995106 so /dados/dados/ when it should be /dados.

Affects CKAN 1.6b only (not yet released).

Tasks include:

Estimate 4d

• Update requirements (git)
• Search facets error
• Error on dataset read (extras in context)
• Url generator for rdf
• Make templates work with latest theme
• Use dicts in templates (isopen())

1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)

2. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).

3. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.

4. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu

5. Blue search button should be displayed in line with the search bar, not underneath

6. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name

It is useful for extensions to add or change items that go into the search index. Add an extension point for this.

Make sure accented characters are normalized when indexed and when searched for.

Suggested by Richard Cyganiak in ​http://lists.okfn.org/pipermail/ckan-discuss/2012-January/001977.html

• new relationship type links_to / linked_from
  • Maybe with extra 'count' property?
• creating a new relationship should be post to the relationship register, not the source entity.
• add missing doc about creating a new relationship
• 'Model Formats' section seems to be missed out of the docs since release 1.4.2. See: ​http://readthedocs.org/docs/ckan/en/ckan-1.4.2/api/version2.html#model-formats

A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else.

(But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.)

Same in the Web interface and API.

This ticket ties in with #948 highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can #1076.

We need to be able to translate:

• EC extension (templates, form)
  • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
• Field values in taxonomy (e.g. country names - Eurovoc)
  • #1665 Research into Eurovoc
  • Display Taxonomy in different languages
• Field values not in taxonomy (e.g. title & description)
  • Use extra fields e.g. _i18n_title_fr = Le data.
  • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)

Timescale to complete this - Mar/Apr?.

Related work to interface with:

• Eurovoc - Sean
• i18n in search index - Adria

Cost - 2 days

Search interface has new options to filter and sort the results by the date the package has been last modified in ckan. Search options are included in both Web UI and Search API.

The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package.

Example search parameters:

Sean spotted that some strings need internationalising, such as flash messages. Generally all strings that appear on the web front-end should be internationalised.

Particular places that need i18n:

• ckan/controllers/admin
• ckan/lib/base

Note: there are some exceptions, such as in i18n.py, very obscure error messages and stuff that only appears on the API.

NB: there is a cost in making a string internationalisable (all our volunteers have to translate it), so we should not be too zealous. We should also look at the i18n/ckan.pot to see if any of the existing strings can be reused.

I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates

Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard

DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation

See ​http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal

• Create a collapsible section for the README
• Reduce the number of things on the page
• Tidy up groups and tags in the sidebar
• Resource view becomes a list of search results

(additional)

• Resource navbar item links to first resource
• Resource navbar has a dropdown menu of all resources in the dataset
• Edit and Authz navbar items should be in a separate logical group to the right

Very rough summary: create more info section and move stuff out (relayout page e.g. README, tags etc)

As a user I want to view a package at a given revision:

• When I visit /package/osm@yyyy-mm-dd or /package/osm@rrrrrrr I should be shown package at date yyyy-mm-dd or revision rrrrrrr.
• viewing this page, it should say it is the old revision, just like wikipedia
• package history page should provide links to these pages

On the homepage click "francais" and then "English". The flash message reads "Le langage a été fixé à: français" when it should say "The language is now: English".

To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

• Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

• Write Middleware + update url_for to take account of the language. [2d]

• Document the language setup, and how to replicate it. [1d]

eg 404 page

create a 404 error via a bad url - translation links go to /document/error not the actual bad url

We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list.

• Add tags_create() function that calls the existing tag_create() many times?
• Also tags_delete()?
• Or just let tag_create() and tag_delete() accept a list?
• Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.)

How big is it? How are we going to store it? etc.