Enable ckanext's to add new vocabularies to default_package_schema(), use the new vocabularies in templates for dataset view, create, edit pages, helper function for extensions to create a vocabulary schema for a given vocabulary name (must match a vocab already in the db) and add it to default package schema.

The SOLR url, user and password defined in the CKAN config file are ignored and the default SOLR url is used.

This causes:

• "0 datasets" displayed on the home page
• Dataset searches result in 0 results and a small message "There was an error while searching". (Nothing about it in the logs)

To reproduce

This bug is only visible if your SOLR instance is not at the default place. To quickly reproduce this problem, setup your machine as a SOLR multicore instance and run: "paster db clean && paster create-test-data && paster serve development.ini". It quits with the error: "solr.core.SolrException?: HTTP code=400, reason=Missing solr core name in path"

Code affected

• Does not affect any released versions (1.5.1 is clean).
• Introduced to master: ​https://github.com/okfn/ckan/commit/627382e6ff575146da221efc50321a41152d8b0f by Ross

We want to make email announcements to users of thedatahub.org.

The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list.

Different ways to do it:

• 'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough?
• CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default).

Something more advanced in the future could be:

• CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle.
• Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine.
• CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach.

CKAN provides the /data page page of data.gov.uk

Breakdown of tasks:

• [x] Analysis / refinement of spec.
• [x] Log-in / register as publisher

(Waiting on publisher form)

• [x] Population of "browse by publisher"

list groups ordered by most datasets (Waiting on publisher integration)

• [ ] Browse by nation
• [ ] Featured datasets

Now a possible integration point with drupal

• [X] Tag cloud
• [X] Developers section

Dataset read page for DGU

Breakdown:

• [X] Fix missing title
• [X] Links to the social media stuff
  • "Share your app"
  • "Share your idea"
  • "Request new data"
• [ ] Presentation of additional information should be broken into sections

Content is there. Don't do anything until theming/design work is undertaken.

• [ ] "More like this": links to similar datasets.

Dropping

• [N] Stars (I think these are being removed)
• [ ] Comments

Possible integration point with Drupal. Needs it own ticket.

• [X] Give feedback to department
• [X] Tidy up the QA stars

To allow navigation between sections to still work, also potentially rephrase to avoid confusion that you have to save every time to add a tag or resource and can't say add description, tag and resource in one go

Super ticket: #1506

e.g. ​http://thedatahub.org/user The users with Gravatar have their nice user icons, but the majority of users (without Gravatars) have 'broken link' symbols. Same problem wherever users are shown, such as the dataset history pages.

Broken on 1.5.2b - currently on thedatahub.org.

In the tag-taxonomies branch tags now (optionally) belong to vocabularies and where tags previously had unique names, a CKAN instance can now have two tags with the same name as long as they belong to different vocabularies. The tests for the tags model should probably be updated to reflect this.

Other tests might also need to be updated.

If you use tag autocompletion, it automatically adds a comma after the tag. This is fine if you are going to add another tag, but not fine if it is the last one. If you don't manually remove the final comma than you get form validation error "Tag string: Tag "" length is less than minimum 2"

Affects only version 1.5.2a.

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

• Model update (See #1726)
• Update queries for getting harvest sources' datasets
• Update ckanext-csw queries for getting the objects
• Update inspire and ckan harvesters
• Document new field

Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation.

Estimate: 4-5d

Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources.

We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model)

Tasks include:

• Auth checks on the different controller actions (and the command line interface?)
• Modify the index page to just show sources the logged user has permissions on.

(Talk with Ross about the specific functions to use)

• Create/edit form to show a dropdown with the user available publishers.

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

• Expose this on the web interface (only for sysadmins, see #1731)
• Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

• Publisher should create a new record with new identifier and same/different content as applicable.
• If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
• You won't get the state where a package is updated in its deleted state
• If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

Super ticket #1744

If you edit a dataset, simply add a tag and click on "Save" then you get the message "You have unsaved changed. Make sure to click 'Save Changes' below before leaving this page." inserted whilst the submission takes place. (This is because the click moves focus away from the tag field). This (reasonably common) edge-case could usefully be squashed.

Also note typo: "unsaved changed"

Super ticket: #1745

• Convert /api/rest/dataset to be search query (i.e. take arguments in ?....)
• Directly expose solr though may want to override limit. See ​https://github.com/okfn/openspending/blob/master/openspending/ui/controllers/api.py#L48

Required for some improvements to UX (such as autocomplete and better search).

Test failures in master due to tom and rufus not running tests. 0.3d.

It is possible to create a group or a dataset called new, or edit, which will then be inaccessible as it is blocked by the existing routes with those names.

We should maintain a list of invalid names in the verification that happens behind the scenes in the UI.

Super ticket: #1506

• Show edit summary form again, at least on edit pages (and prefill it for create to something sensible like: Creating dataset {title} {name}) [Done #1506]
• Fix drop down for resource types in resource add/edit to limit to fixed set of options (as per ​http://wiki.ckan.org/Domain_Model/Resource)
  • See also ​http://lists.okfn.org/pipermail/ckan-discuss/2011-December/001860.html
• Allow reordering of resources (or: and this is somewhat more complex but possibly simpler: allow editing of a resource_order field and we use that to sort resources (this is more like wordpress page_order -- these two options could become the same if we just hijack the existing position field on the join table and get rid of ordereredlist extension)) [DONE #1506]
• Radically simplify set of licenses [DONE #1359]
• "Unsaved changes" warning briefly when you click "Save" #1735
• "Unsaved changed" typo on dataset edit page

A collection of improvements to the dataset creation/edition form (feedback form DGU).

• [x] validation error formatting
• [x] geograohic coverage: add local authority
• [x] resource format field
• [x] resource format auto-complete
• [x] renamed data tab -> files tab
• [x] integrate Adris'a INSPIRE changeset, and migration scripts as part of build
• [x] remove "discontinued" from the update frequency
• [x] disable the disabling of save
• [x] Remove ability to edit url on the edit-form (and remove "url not available")

• Upgrade to new DataStore? system
• Better support for unicode etc
• ? Rename to DataStorer?
• Deploy

The publisher index page should show an alpha paged list of publishers.

On publisher view page - hide administrators from public view - only logged in admins of group should see them.

Only sysadmin can access create publisher form.

History on user profile should only be visible to a user who is a member of at least one group.

Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique.

Need to test:

• API call to create new vocab
• API call to add tags with a given vocab
• API call to update tags
• Dummy plugin to test that updating tags works via WUI

We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list.

• Add tags_create() function that calls the existing tag_create() many times?
• Also tags_delete()?
• Or just let tag_create() and tag_delete() accept a list?
• Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.)

Basic description of multilingual features intended to be supported.

Add api to translation to the term_translation table.

• add to logic 2d
• test 2d

If the current page URL has unicode in it (e.g. u'/tag/biocombust\xedveis') then I'm not sure how why it isn't more encoded, but we are seeing links such as http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251 which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().

If I have a relationship between two packages, and then I edit any property of one of them (Author email, for example), the relationship will automatically be lost.

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. ​http://127.0.0.1:5000/dataset/new?__cache=37713707

Changes to publishers for customer

The Package model should have a approval_status as the Group model does. See migration 049.

Main tasks:

• Integrated seamlessly into CKAN (authorization etc)
• Based off ElasticSearch?

Etherpad: ​http://ckan.okfnpad.org/webstore

Tasks

• New API point (config + controller) [0.75d]
  • /api/data/{resource-id}
• Authorization [0.5d]
• Documentation [0.25d]
• Upgrade data viewer to use ES [0.5d]
• Convert webstorer to new setup - ​http://github.com/okfn/ckanext-webstorer [1d] - #1753
  • See below
• "Create DataStore? Resource" in Resource creation (??)
  • Just creates a resource with blank url and datastore enabled (webstore_url non null)
• Tick box to enable DataStore? in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)
  • Also need to use this info in DataStore? extension?
• Create nice instructions on Resource page about using the DataStore?
• Rename webstore -> datastore
• Deploy [0.5d]

Details

Layout on ES

elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)

Request path

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: ​https://github.com/okfn/elastic-proxy

Questions

1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy?
   • Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
2. How do we decide when webstore is activated
   • Ans: see previous.
3. Do we (and how do we) allow for resources which are just for webstores
   • Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ...
4. How does webstorer work ...
   • CSV import - NS: make sure we add e.g. rownum or id = rownum ...
   • Separate mini-service (webstorer.thedatahub.org ...)
5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy?)
   • Ans: do not bother at present. Focus on Webstore route.
6. Where does geodata support come in (in Recline or separately)
   • This is a separate issue

User Story 1

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

1. Allow normal upload process
2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
3. Update the resource in some way (set webstore_url)

User Stories

User Story 2

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)

User Story 3

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)

When you are logged in, you cannot login as a different user before first logging out. (This is a restricting in repoze.who, which gives priority to an existing recognised identity over another one you supply.)

Yet, the CKAN login form appears when you try to do something you're not allowed to, even if you are logged in already. You login with a different identity and nothing happens - you remain as the first identity - this is incorrect.

(It is also a little confusing to be sent to the log-in page, although we do have a flash message explaining, and you might want to relogin as a superuser if you can.)

Whilst in this area, it would be worth fixing the problem for developers where you are logged in, do "paster db clean" and then try to log in - AuthTkt? recognises your old cookie but the User object doesn't exist, so you get an error logging in. In this instance we should tell people to log-out.

See how possible it is to run CKAN on a Windows server.

Plan:

• Get CKAN code and dependencies installed on Windows
  • ideally as similar to standard package install as possible
  • virtualenv/pip should work
  • try Cygwin to help
  • SOLR and Postgres could potentially run on other servers if necessary.
• Try to run on Apache
• See if there are any issues

e.g. ​http://189.9.137.65/dados/ clicking on Deutsch link is http://189.9.137.65/dados/locale?locale=de&return_to=%2Fdados%2F&hash=1dc17c315c419df850da0dd3599eefa9da76fbeb and redirect goes to http://189.9.137.65/dados/dados/?__cache=97995106 so /dados/dados/ when it should be /dados.

Affects CKAN 1.6b only (not yet released).

language chooser should look like the ec one

All data is that is translatable should be translated when viewed.

This includes tag names will be ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language.

Related to the dataset view extension points for translation, recently added by kindly/=.

• Tag list not be added to CKAN core, what is the best way to do this in an extension?
• Does the list need to be updated via API/WUI?

• see how this currently works with DGU first

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

• write importer for publisher info (currently in JSON format), add to ckanext-ecportal
• add group translations to translations table
• run on test server

All translated fields and vocabularies need to be added to search index in the correct fields.