Super ticket: #1506

e.g. ​http://thedatahub.org/user The users with Gravatar have their nice user icons, but the majority of users (without Gravatars) have 'broken link' symbols. Same problem wherever users are shown, such as the dataset history pages.

Broken on 1.5.2b - currently on thedatahub.org.

We need:

• "current" column in harvest_object
• "harvest_source_id" column in harvest_object, foreign key to harvest_source
• "title" column in harvest_source

The extension should check on startup if the db needs updating and run the necessary scripts.

It is useful for extensions to add or change items that go into the search index. Add an extension point for this.

When an extra is added to or removed from a dataset or is updated, an activity should go in the dataset's and user's activity streams.

Major UX grip and incredibly simple to do. May also want to have basic autocomplete support within the list (trivial to do as well I believe).

(Talk to RP about this re how we define lists of licenses).

Add Vocabulary domain class, add logic functions for creating, updating, listing, getting, deleting vocabularies, add tests.

Part of #1698

See the etherpad for notes: ​http://ckan.okfnpad.org/22

ckan/logic/action/get.py:tag_list() Currently (I think) just returns all tags in the CKAN instance that the user has access to. Needs an optional argument to get only tags from a particular vocabulary? When called with no argument, should only return free tags? Also add an optional dataset argument, so you can get tags for a given dataset and/or vocabulary?

ckan/logic/action/get.py:tag_autocomplete() Add optional argument to get only tags for a given vocab? When called with no argument, should get only free tags?

ckan/logic/action/get.py:tag_search() Again, optional arguments for restricting by vocabulary and/or dataset.

Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation.

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

• Expose this on the web interface (only for sysadmins, see #1731)
• Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

• Publisher should create a new record with new identifier and same/different content as applicable.
• If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
• You won't get the state where a package is updated in its deleted state
• If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown.

Tasks include:

• Fix up data_dict and templates so that the pkg objects do not get used in the templates. This is to assure that all data passed is modifiable. (2d) (complete)
• Add extension point to Ipackage controller (1d)
• Test (1d)

• [x] hide black stars on the data-view page
• [x] Edit on view form (if you have the right permission)
• [x] RES_FORMAT ~> Resource Format
• [x] Map Based Search -> UK Location Map Based Search
• [x] Note at the top right expalining that these are seach filters (if they are) -> this requires "add a dataset" and "other access" to be moved to a different place (just some whitespace is fine)
• [x] Display publisher and provider (UKLP records only).
• [x] analysis of filter by OGL / non-OGL
• [x] provide /dump on the build server

The publisher index page should show a treeview of the publishers within the system. Probably behind a tab.

Only sysadmin can access create publisher form.

History on user profile should only be visible to a user who is a member of at least one group.

Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique.

Need to test:

• API call to create new vocab
• API call to add tags with a given vocab
• API call to update tags
• Dummy plugin to test that updating tags works via WUI

Add API calls for adding one or more tags to and removing one or more tags from a vocabulary, without affecting the other tags in that vocabulary and without having to pass the full list of the vocabulary's tags.

A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination.

If the current page URL has unicode in it (e.g. u'/tag/biocombust\xedveis') then I'm not sure how why it isn't more encoded, but we are seeing links such as http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251 which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. ​http://127.0.0.1:5000/dataset/new?__cache=37713707

Part of super-ticket #1506.

Follows on from #826.

We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis.

• Create page on ​http://wiki.ckan.org/ about setting up a queue (this will merge into core docs when this is stable) - having a queue will now become a generic requirement for a bunch of extensions
• README with install instructions in ckanext-archiver
• Create an entry in ​http://wiki.ckan.org/Extensions
• Write a blog post

Historical Parent Tickets

• Parent-ticket: #1450
• Parent-ticket: #1032

Step Zero

• COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

• Force format field to be lowercase
  • Modify backend to serve it up in that format
• Subticket #978: Add extra fields to resources in Web UI
• Fix display of large datasets (​http://test.ckan.net/dataset/europeana-lod)
• When submitting forms with errors, open the correct resource pane.
• No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
• Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

• Subticket #1648: Clarify edit terminology
• Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
• Fix (i.e. allow / show) name editing on dataset edit.
  • Fix the slugify so it does not overwrite if i change title after fixing name.
  • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
• Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

• Fix animation glitch in expanding description
• Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
• Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

• "Upload or link data files, APIs and other materials related to your dataset."
• Include the description markdown editor again (keep it small)
• Licence should maybe be restored
  • "(Don't worry if you don't know what your licence is)"
• Show the log message field
  • Beautify
• Pre-populate the log message field?
• ? Show Home Page field when creating dataset [WONTFIX don't push it]
• ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

• Improvements to user page UX - 1d - #1396
• Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

• Take before and after screenshots

To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

• Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

• Write Middleware + update url_for to take account of the language. [2d]

• Document the language setup, and how to replicate it. [1d]

Super ticket: #1506

• Show edit summary form again, at least on edit pages (and prefill it for create to something sensible like: Creating dataset {title} {name}) [Done #1506]
• Fix drop down for resource types in resource add/edit to limit to fixed set of options (as per ​http://wiki.ckan.org/Domain_Model/Resource)
  • See also ​http://lists.okfn.org/pipermail/ckan-discuss/2011-December/001860.html
• Allow reordering of resources (or: and this is somewhat more complex but possibly simpler: allow editing of a resource_order field and we use that to sort resources (this is more like wordpress page_order -- these two options could become the same if we just hijack the existing position field on the join table and get rid of ordereredlist extension)) [DONE #1506]
• Radically simplify set of licenses [DONE #1359]
• "Unsaved changes" warning briefly when you click "Save" #1735
• "Unsaved changed" typo on dataset edit page

On publisher view page - hide administrators from public view - only logged in admins of group should see them.

From Antonia Acuno, via Ira:

It appears that the current layout for the datahub does not work in IE7, which would mean a problem for us as most gov uses it and is one of those we have to support...any views?

Tested in several gov departments computers, so a common enough issue.

​http://imgur.com/ip4xa

From Rufus: We haven't officially supported IE7 so far but can put engineering effort into this if absolutely needed. I note that IE 7 is now 3y out of date and is likely a security risk. Obviously this is something to ask Antonio about.

Main tasks:

• Integrated seamlessly into CKAN (authorization etc)
• Based off ElasticSearch?

Etherpad: ​http://ckan.okfnpad.org/webstore

Tasks

• New API point (config + controller) [0.75d]
  • /api/data/{resource-id}
• Authorization [0.5d]
• Documentation [0.25d]
• Upgrade data viewer to use ES [0.5d]
• Convert webstorer to new setup - ​http://github.com/okfn/ckanext-webstorer [1d] - #1753
  • See below
• "Create DataStore? Resource" in Resource creation (??)
  • Just creates a resource with blank url and datastore enabled (webstore_url non null)
• Tick box to enable DataStore? in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)
  • Also need to use this info in DataStore? extension?
• Create nice instructions on Resource page about using the DataStore?
• Rename webstore -> datastore
• Deploy [0.5d]

Details

Layout on ES

elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)

Request path

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: ​https://github.com/okfn/elastic-proxy

Questions

1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy?
   • Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
2. How do we decide when webstore is activated
   • Ans: see previous.
3. Do we (and how do we) allow for resources which are just for webstores
   • Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ...
4. How does webstorer work ...
   • CSV import - NS: make sure we add e.g. rownum or id = rownum ...
   • Separate mini-service (webstorer.thedatahub.org ...)
5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy?)
   • Ans: do not bother at present. Focus on Webstore route.
6. Where does geodata support come in (in Recline or separately)
   • This is a separate issue

User Story 1

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

1. Allow normal upload process
2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
3. Update the resource in some way (set webstore_url)

User Stories

User Story 2

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)

User Story 3

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)

See how possible it is to run CKAN on a Windows server.

Plan:

• Get CKAN code and dependencies installed on Windows
  • ideally as similar to standard package install as possible
  • virtualenv/pip should work
  • try Cygwin to help
  • SOLR and Postgres could potentially run on other servers if necessary.
• Try to run on Apache
• See if there are any issues

language chooser should look like the ec one

Some request exceptions are currently not being caught (see the celery log on thedatahub for examples)

• write importer for publisher info (currently in JSON format), add to ckanext-ecportal
• add group translations to translations table
• run on test server

Dismax query across languages. Titles and selected languages, weighted more highly.

Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.

On the homepage click "francais" and then "English". The flash message reads "Le langage a été fixé à: français" when it should say "The language is now: English".

• Make sure that the extension works as it should and isn't broken on current releases [3d]

• Make it work with the EC schema, including extras [2d]

• On the resources tab, there's a "Check Resources" button which, when clicked makes an ajax request with the list of URLs entered by the user.

• The server checka each link for errors and header information about the linked resource. (Using ckanext/archiver/tasks.py:link_checker()).

• The server returns a list of dicts (json), containing information about the linked resource, and the client uses that to:

• populate the format field of each resource

The (guessed) 'file_extension' populates the 'format' field. If it's 'htm' or 'html', then we assume it's a listing page, and so don't populate the format field with 'htm' or 'html'.

• provide feedback if a URL appears to be invalid

If the URL doesn't appear to be a URL at all, or returns a HTTP error, or times-out, then URL field is highlighted in red to indicate it's a bad URL. A tooltip shows the error message to the user.

• [Optional] provide feedback if a URL appears to point to "Additional Information" - ie the Resource should be entered under "Additional Resources", rather than "Timeseries" or "Individual" datasets.

Analysis and further description on etherpad: ​http://ckan.okfnpad.org/dgu-package-form? [Section I]

It is possible to create a group or a dataset called new, or edit, which will then be inaccessible as it is blocked by the existing routes with those names.

We should maintain a list of invalid names in the verification that happens behind the scenes in the UI.

All data is that is translatable should be translated when viewed.

This includes tag names will be ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language.

Related to the dataset view extension points for translation, recently added by kindly/=.

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

Super is [1649]

We need to provide customisable mappings between CKAN models and RDF. This needs to be easily configurable for different clients and should take into account objects related to the model being mapped.

Current contenders for impl are:

1. ordf.org's Object Description Mapper (which uses OWL)

2. RDFAlchemy, although this looks like it may result in fixed mappings

3. A custom scheme with a simple json file

4. A template based approach using Genshi templates?

Approach is to get a mapping complete of the attributes directly in the package class, and then provide mappings for related objects (tags/resources etc).

We'll then need to be able to publish the RDF to a triplestore.

See ​http://ckan.okfnpad.org/rdfmapping for further analysis.

Most of the get_group/add_member style calls should be handled, or at least provided via logic layer.

Don't need to break existing code, but do need to provide access via logic layer