From #2204

not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...

### Comments

In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ...

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

• Switch to the latest minifed CSS + JS
• QA-sweep the site, especially dataset/edit

There are a couple of strings on the authorization group add/edit form that are not being translated:

"Name: Please enter a value" (error message when you try to add an auth group with no name)

"User name"

"Save" button

The form seems to be built using FormBuilder?. See ckan/forms/authorizationgroup.py, builder.set_displayed_fields()

The strings Tags, Res_format and Groups on e.g. the package index page should be marked for translation. These strings have been refactored in the feature-1821-multilingual-extension branch, so wait until that branch has been merged into master.

Can only add one dataset at a time now (not sure when this broke but was working ~2-3m ago).

Assigning to seanh for review and estimation. I'd rate this as quite annoying but not absolutely critical (hence assigned priority).

Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients).

Fix this by ensuring that error messsages on API get passed through correctly.

Minor because affect is annoying but not terminal

What needs doing, and how long will it take?

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

• What should the sortBy controls/widget look like?
• Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] ​https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

Needs re-testing after the ODS re-theme. ====

Simple image upload, text changes and css updates for a single instance. All done on a single configuration/admin page

See ​http://ckan.okfnpad.org/simple-themes for user stories

[x] Initial version will only take an image_url rather than allowing an upload.

[x] New page with form for title, tagline, image loader etc.

[x] Allow the user to paste in values for the CSS, do we really want a colour swatch?

[x] Generate local CSS override which should *somehow* override colors in the original CSS.

[x] Allow user to provide text for an about page.

User story #037 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

Finalised metadata model requires several changes to our existing model:

• new extra: alternative title
• new extra: identifier
• new extra: interop level
• new extra: version description
• new extra: dataset status
• dataset type: change to fixed vocab
• resource types: prefix with documentation/distribution/etc for clarity, add new resource types
• rename update frequency to accrual periodicity
• add new fixed vocab field: language coverage
• resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model.

Seanh and icmurray both touched on the html snippets in facets.html. So there's a merge issue. icmurray to merge changes in the multi-lingual branch prior to merging into master.

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how.

These two tests:

ckan.tests.lib.test_helpers.TestHelpers?.test_parse_rfc_2822_gmt_case

ckan.tests.lib.test_helpers.TestHelpers?.test_parse_rfc_2822_with_offset

have been failing in certain timezones. In the UK they pass, if you're in Germany they'll fail.

The function being tested uses datetime.datetime.fromtimestamp() which uses the local time, but the tests compare the results against hardcoded strings, so the tests cannot work in different timezones.

add resource_status_show action function to see the state of celery tasks.

Need to get information from both out task status table and the celery_taskmeta table.

If you switch to Portugese and try to add a resource on dataset edit it will fail. This is because there is a string translation in js_strings.html that has quotes in it. When this is inserted into the file this causes a js exception which then prevents any further js processing.

Specifically:

CKAN.Strings.youHaveUnsavedChanges = "Você tem alterações não salvas. Certifique-se de ter clicado "Salvar Alterações" abaixo antes de sair desta página.";
Uncaught SyntaxError: Unexpected identifier

To fix is simple i imagine: need to escape " correctly in js_strings.html (or translations used).

I have temporarily patched this live in order to allow the hackday here to continue.

Some datasets become obsolete: e.g. ​http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.

Raw

/raw or /download (latter may be better)

/dataset/{id}/resource/{resource-id}/raw

=> redirects to resource url (download)

API

/dataset/{id}/resource/{resource-id}/api[/....]

=> either redirects or directly serves /api/data/{resource-id}

Viewer (Explorer?)

Do data viewer but without anything else showing (full-screen?). this would not be just the data explorer but other cases too (including iframe etc ...)

/dataset/{id}/resource/{resource-id}/viewer

Resource (pre)-viewer without anything else (perhaps include resource title)

Assigning for kindly for review.

These would take about 20m to implement and would be *very* convenient from my experience of using the data API.

Implementation at enhancement-2322-convenience-urls

The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading

create permalinks

eg \dataset\~a5-DQ\nice-name

~XXXXX defines the permalink the nice-name is ignored

• packages table needs a new field for this data

• existing datasets need updating

• new datasets need the identifier creating

• routing needs updating

• needs to work for all urls like edit

Once this is implemented we can decide when we will use permalinks vs the normal urls.

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

Whitelist any GETable api actions, and optionally construct the query from url params rather than body.

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

• Related Item has comments on index page (they should not show up here -- why is this happening!)
• Related item should be to left of history (history could in fact move to RHS)
  • What about losing icons since we duplicate for related item
• Related item plus sign is very poorly aligned within box
  • Add related items should be in drop down like for resources (?) to make it easy to add items
  • Hover should show a title that gives some information about what related are (not at all clear)
• Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

• #2331 - search should add not or
• Search results should be similar across group pages and normal search results (suggest adopting group page style)
  • Search results resource link should link to resource page rather than url for that resource
  • Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

• Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) ​https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07
  • Ensure still using read-only (this seems to have been lost in upgrade)
• Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

• (? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
  • Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
  • Do #2322 ...
• Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
• Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
• Data viewer embed url includes all the resource info

Dataset page:

• Put home page link somewhere more prominent (e.g. the top of the sidebar)
• Put author more prominent (at top right)
• (? - new work) List CKAN owners at top right in sidebar
• Make resources stand out more
• Rename resources tab and heading to Data & Resources

Dataset edit and create:

• Corrections to dataset creation form for v1.7 - #2317
• Better auto-complete for groups on dataset edit/new page - #2319
  • Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
• Description / notes text area is not wide enough.
  • Should also be larger on edit page
• Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
• Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
  • A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
  • Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

• Do not show no description in no description (just show dataset info)

Theme:

• Fix spacing above page title (i.e. dataset title etc)
• Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer
• Fix subnav width on resources pages (does not extend all the way to the right like it should)
• Fix spacing of top menu and search box (do not look right)
• Group pages:
  • view page: image / logo is really poorly locate and sized, see e.g. ​http://datahub.io/group/wikimedia
  • view page: admin heading on RHS sidebar is wrongly indented

Miscellaneous:

• Ensure that we out-of-the-box config for CKAN has (#2388):
  • Only logged in users able to create
  • Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

• Social extension
• Analytics

Placeholder for thinking about this ...

• Allow users to configure info for CSV import (e.g. separators, quotes etc)

I won't be needed for this during the current sprint, closing.

After checking out a commit to test from CKAN's GitHub?, Jenkins runs a script that creates a new virtual environment and installs CKAN and its dependencies into it, and does some other necessary tasks. Jenkins then runs the tests in this virtualenv.

The install script may have to change from one commit to the next as CKAN's install instructions change, so it would be good if the script was shipped in CKAN core. That way Jenkins will run different versions of the script depending on which commit it's testing and if the tests fail because the script is wrong then that's actually a bug that needs to be fixed in CKAN core. Also the CKAN install instructions could be simplified a lot by just having users run this script instead of doing each step by hand.

Changes:

• tidy up activity feed on home page or replace with search results (by modified date)
• move ckan menu to a left sidebar (see Sam's template) (not doing this now)
• update templates with changes from master
• use local image in language dropdown (ecportal.js)
• update dataset read template

To discuss:

• should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping.

Start a Java library for interacting with the CKAN Action API.

Currently requires support for Groups, Tags, Resources, and Search.

An example app would be very useful.

When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController? subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

• [ ] Add hooks
• [ ] Use in ckanext-example

Try and increase the page load times for the front end.

Now

• Always include scripts at the bottom
• Include CSS at the top
• Load fonts with JS
• Concatenate and minify JavaScript? files
• Create a build script to load dependancies seperately
• Enable Gzip compression on resources
• Enable future cache headers on resources

Next

• Support serving assets from a CDN
• Look into optimising the CSS - Remove large portions of unused CSS
• Remove extreaneous libraries, can jQuery UI be replaced with bootstrap?

Tools

• Webkit Inspector Audits
• YSlow Firebug Plugin - ​http://yslow.org/
• Speed Tracer - ​https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl

Continuous Integration

YSlow can be hooked up to Jenkins via Phantom JS

Various stuff related with release 1.7:

• Review patches sent to the release branch
• Regularly update TDH with latest changes
• write changelog, announce email, etc
• ...

I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on #2359

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

Need to investigate alternative payment gateways

1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8.

Solr install instructions install the wrong version

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

Currently in the logic function tag_search you can specify limit and offset, but no count is returned. Therefore pagination is not possible for tag results. This is desired though.

This needs to be fixed in the CKAN templates ("no-sidebar" should not be marked for translation) but for 1.7 just fixed it in the translation files.

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

Provide a new test install of DataGM based on the ODS demo site being completed through June.

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)