The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now.

Dump = paster db simple-dump-csv/json

reasoning

The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them.

Admins get the full database anyway in the backup pg_dump.

We only included them in the user dump because it was designed before use of state=deleted was established.

Time estimate: 2h

thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data.

This was seen on datacatalogs. When we upgraded it 1.5 to 1.5.1 we saw some permissions being reset so that it could be spammed. Anonymous and logged in users were given anon_editor and editor permissions. I don't know what the previous config was. Permissions for sysadmins remain unaffected.

We need to be able to translate:

• EC extension (templates, form)
  • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
• Field values in taxonomy (e.g. country names - Eurovoc)
  • #1665 Research into Eurovoc
  • Display Taxonomy in different languages
• Field values not in taxonomy (e.g. title & description)
  • Use extra fields e.g. _i18n_title_fr = Le data.
  • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)

Timescale to complete this - Mar/Apr?.

Related work to interface with:

• Eurovoc - Sean
• i18n in search index - Adria

We need an API for changing the authorization (permissions) on a model object.

This was neglected in #1253 because of various changes to authz were looming. But six months later it still hasn't been done.

Suggest this is just moving _add_user_object_role from ckan/lib/base.py to the logic layer, but no doubt it will be more complicated.

Estimate: 3 days

The SOLR url, user and password defined in the CKAN config file are ignored and the default SOLR url is used.

This causes:

• "0 datasets" displayed on the home page
• Dataset searches result in 0 results and a small message "There was an error while searching". (Nothing about it in the logs)

To reproduce

This bug is only visible if your SOLR instance is not at the default place. To quickly reproduce this problem, setup your machine as a SOLR multicore instance and run: "paster db clean && paster create-test-data && paster serve development.ini". It quits with the error: "solr.core.SolrException?: HTTP code=400, reason=Missing solr core name in path"

Code affected

• Does not affect any released versions (1.5.1 is clean).
• Introduced to master: ​https://github.com/okfn/ckan/commit/627382e6ff575146da221efc50321a41152d8b0f by Ross

We want to make email announcements to users of thedatahub.org.

The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list.

Different ways to do it:

• 'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough?
• CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default).

Something more advanced in the future could be:

• CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle.
• Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine.
• CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach.

If the current page URL has unicode in it (e.g. u'/tag/biocombust\xedveis') then I'm not sure how why it isn't more encoded, but we are seeing links such as http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251 which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. ​http://127.0.0.1:5000/dataset/new?__cache=37713707

See how possible it is to run CKAN on a Windows server.

Plan:

• Get CKAN code and dependencies installed on Windows
  • ideally as similar to standard package install as possible
  • virtualenv/pip should work
  • try Cygwin to help
  • SOLR and Postgres could potentially run on other servers if necessary.
• Try to run on Apache
• See if there are any issues

Allow OpenID to be disabled as a login option.

Currently (CKAN 1.6) there is a config option openid_enabled which stops the login option being displayed, but:

• In the User edit form you can still add your OpenID
• The repoze.who.openid middleware is still required - can it be avoided?

Make the search results pagination styled by Bootstrap.

Needs putting into a <ul> structure. Current page and '..' need become links that are styled disabled.

Wanted by DGU.

Also need to fix bug where 'Other' is never highlighted.

The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData? etc and will be used by DGU.

We need to separate the CLI part of CreateTestData? from the bit which creates test fixtures. This is because of the problem with imports and logging. Otherwise we have to do unpythonic things to get logging to work when deriving from CreateTestData? from an extension such as ecportal / dgu. I've only introduced logging into CreateTestData? but I think it is a good thing to have. This change doesn't cause breakages

When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: "Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'""

The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header.

This affects all versions of CKAN.

This is due to webob/requests.py:1248 being pretty basic.

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

• We can say that you can add your own previews
• Every preview would have to define it's own dependencies
• External development (community creates new extensions?)

Questions to answer:

• Specification of the interface
• Precedence of previews for the same data types
• Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: ​http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

When you install ckanext-store (Storage extension) then CKAN displays 'Upload a file' button in the dataset/resource edit tab.

If you are not logged in, then this feature doesn't work, but there is nothing to indicate this. The button should be disabled.

Furthermore, it lets you think you are starting the upload process, the spinner spins, it says 'Uploading...' and stays like that for a long while.

This is a problem CKAN 1.4.3(ish) - 1.5.1 (current version) when you install the current ckanext-storage.

Try and increase the page load times for the front end.

Now

• Always include scripts at the bottom
• Include CSS at the top
• Load fonts with JS
• Concatenate and minify JavaScript? files
• Create a build script to load dependancies seperately
• Enable Gzip compression on resources
• Enable future cache headers on resources

Next

• Support serving assets from a CDN
• Look into optimising the CSS - Remove large portions of unused CSS
• Remove extreaneous libraries, can jQuery UI be replaced with bootstrap?

Tools

• Webkit Inspector Audits
• YSlow Firebug Plugin - ​http://yslow.org/
• Speed Tracer - ​https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl

Continuous Integration

YSlow can be hooked up to Jenkins via Phantom JS

Should minify and concatenate JavaScript? and CSS as part of #2354

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

Currently facets in the new demo site disappear from the sidebar once they are selected.

​http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

Instead they should be marked with an "active" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works.

if pills cross onto 2 lines they split which they shouldn't to see just add some tags/formats till it breaks

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments

Implementation:

​http://s031.okserver.org:2375/dataset/adur_district_spending

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments

Implementation:

​http://s031.okserver.org:2375/dataset/adur_district_spending/related

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

​http://s031.okserver.org:2375/group

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

​http://s031.okserver.org:2375/group/new

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

​http://s031.okserver.org:2375/user/edit

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments

Implementation:

​http://s031.okserver.org:2375/en/user/reset

Percieved benefits of switching to Jinja 2

• Block inheritance for base templates which is much simpler to understand.
• Large number of developers (and designers) are familiar with it due to its similarities with the Django templates.
• Should be much faster at parsing and rendering templates.
• Has localisation support.
• Good documentation both on it's website and on sites like Stack Overflow.

We should convert a couple of existing templates to test the switchover and then get peoples feedback.

JavaScript? needs to be able to insert html templates into the document.

There are three common solutions to templating at the moment.

1. Insert the html into <script> tags at the bottom of the page and pull them in using the DOM and innerHTML. This is a very simple solution but not very portable.
2. Request the HTML snippets as plain text/html using XHR. This adds an extra HTTP request per template but is very clean. It would also be possible to inline the HTML into the JS at the minification stage for production.
3. Include the HTML in JavaScript? strings (horrible to maintain and bloats the js)

It makes sense to keep these with the other templates so that we can take advantage of localisation where possible. This indicates 1 or 2.

They've become inconsistent as the design has changed.

• Remove white stripe
• Make text sizes consistent
• Make white space around text boxes consistent

• Needs tags
• Needs groups
• Needs slug
• Licenses links to something
• Needs tooltips (Ira to write some tooltips)

Allowing them to be displayed in many sizes, colours and resolutions.

​http://www.zurb.com/playground/foundation-icons

Share this (twitter/fb) links open in new tab (no JS) or nice lightbox of some description in demo

So that they don't interfere with buttons such as here: ​http://s031.okserver.org:2375/en/dataset/afterfibre/resource/5d19f9fa-c5a4-426c-84de-c624d6e8c3b3

assuming you have datahub data

​http://localhost:5000/en/dataset/2008_facility_ghg_emissions_by_province_territory/resource/28650645-8878-4826-9405-f20b9dfa4c12

also the breadcrumb becomes overly long too

I see X where none should be

In the footer: Change 'Powered by CityData?' to Powered by DataSuite? (with link to ckan.org/datasuite)

Links in left hand side of footer should change to a link to ckan.org, link to OKFN, link to ckan.org/datasuite/about and link to Pricing Page (on ckan.org .. will be /datasuite/solutions)

In create dataset form, prefilled tag examples should include one example with a space, e.g. "mental health"

Change 'Groups' to 'Publishers' everywhere

Homepage copy/images (Ira to supply?)

needs to move to a better home

better ubuntu install instructions plus test they work

apt-get install nodejs

integrate with paster

​http://localhost:5000/dataset/edit/2010-nigeria-budget

the edit url button appears too high up

make it as a snippet calling thing like aron would love

in 2375-demo-theme-development

templates/package/confirm_delete.html needs to be made to look nice note some writing in black

delete button needs nice home on dataset/edit

you can add js support add ?confirm=yes to url on dataset button to force delete

Please note delete is not yet implemented - but will be in the next few hours

There is a new template for editing resources outside of the add dataset logic

this is the template it just throws some crap onto the page at the moment - we need to create a form - the form will end up with the usual data/edit stuff but for now you can use the resource to populate the form and I can then change it to use data - I just want to know which fields we'll have

templates/package/resource_edit.html

when done reassign to me and I'll get it functioning

see ​http://s031.okserver.org:2375/dataset/gold-prices/related/edit/46ba1174-3666-486e-b822-b5b45ffbb703

when editing a record, button at bottom right should change to save instead of create