Add generator=ckan tag in head of templates to identify site as generated by CKAN

• code review with D.R. for ckanext-ecportal
• merge and test all multilingual code
• fix server issues: routing, Git access to CKAN repository

Bugs to fix:

• rename groups to publishers
• update home page layout (recent updates along side publishers)
• don't add 'en' to main menu links if english
• resource edit page, fix missing images
• change dataset 'settings' to 'edit'
• translate 'published by' drop-down list
• fix JS on 'add a dataset page'
• prepopulate 'published_by' when clicking 'add a dataset' from group
• chosen broken (dataset edit) if in different language

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

From #2204

not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...

### Comments

In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ...

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

• Switch to the latest minifed CSS + JS
• QA-sweep the site, especially dataset/edit

See ​http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

What needs doing, and how long will it take?

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

• What should the sortBy controls/widget look like?
• Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] ​https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

Needs re-testing after the ODS re-theme. ====

Simple image upload, text changes and css updates for a single instance. All done on a single configuration/admin page

See ​http://ckan.okfnpad.org/simple-themes for user stories

[x] Initial version will only take an image_url rather than allowing an upload.

[x] New page with form for title, tagline, image loader etc.

[x] Allow the user to paste in values for the CSS, do we really want a colour swatch?

[x] Generate local CSS override which should *somehow* override colors in the original CSS.

[x] Allow user to provide text for an about page.

User story #037 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

Finalised metadata model requires several changes to our existing model:

• new extra: alternative title
• new extra: identifier
• new extra: interop level
• new extra: version description
• new extra: dataset status
• dataset type: change to fixed vocab
• resource types: prefix with documentation/distribution/etc for clarity, add new resource types
• rename update frequency to accrual periodicity
• add new fixed vocab field: language coverage
• resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model.

Seanh and icmurray both touched on the html snippets in facets.html. So there's a merge issue. icmurray to merge changes in the multi-lingual branch prior to merging into master.

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how.

Change login cookie from a default expiry of 50 years to 2 years. You can also uncheck a 'remember me' checkbox on the login form for the cookie to just last the session.

Background conversation on ckan-dev:

DR: I wonder if anyone objects to the expiry of the login cookie to be changed from 50 years to 2 years? 50 years might be appropriate for thedatahub.org, but for government sites it seems (to me) to be too lax.

Toby: is this the repoze.who cookie? If so that seems sensible to me.

Rufus: Definitely agree. I would also like to see introduction of a standard "remember me" checkbox (set to true by default). At the moment a login lasts forever (until you logout) automatically.

• Add home page url
• Remove group option (seen it confuse several users) - perhaps replace with tags (?)
• Check form looks ok in portugese (seems to be a problem with css)

Use jquery chosen?

Would make them much nicer but requires:

• Ensure uniqueness (or deal with ambiguity)
• (?) Slug/names versus titles
  • Aside: what is currently called name should probably be title

Apparently auto-complete in Organizations may not work. Check and fix.

See ​https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5

Also need to fix bug where 'Other' is never highlighted.

• Add Ascii folding filter to text fields.
• Add capacity field for public, private access.
• Add title_string so you can sort alphabetically on title.
• Fields related to analytics, access and view counts.
• Add data_dict field for the whole package_dict.

Actual implementation of data_dict adding will be added later.

E.g. on dataset page, on revision page, on user page and on search results.

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

• Related Item has comments on index page (they should not show up here -- why is this happening!)
• Related item should be to left of history (history could in fact move to RHS)
  • What about losing icons since we duplicate for related item
• Related item plus sign is very poorly aligned within box
  • Add related items should be in drop down like for resources (?) to make it easy to add items
  • Hover should show a title that gives some information about what related are (not at all clear)
• Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

• #2331 - search should add not or
• Search results should be similar across group pages and normal search results (suggest adopting group page style)
  • Search results resource link should link to resource page rather than url for that resource
  • Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

• Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) ​https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07
  • Ensure still using read-only (this seems to have been lost in upgrade)
• Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

• (? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
  • Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
  • Do #2322 ...
• Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
• Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
• Data viewer embed url includes all the resource info

Dataset page:

• Put home page link somewhere more prominent (e.g. the top of the sidebar)
• Put author more prominent (at top right)
• (? - new work) List CKAN owners at top right in sidebar
• Make resources stand out more
• Rename resources tab and heading to Data & Resources

Dataset edit and create:

• Corrections to dataset creation form for v1.7 - #2317
• Better auto-complete for groups on dataset edit/new page - #2319
  • Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
• Description / notes text area is not wide enough.
  • Should also be larger on edit page
• Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
• Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
  • A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
  • Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

• Do not show no description in no description (just show dataset info)

Theme:

• Fix spacing above page title (i.e. dataset title etc)
• Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer
• Fix subnav width on resources pages (does not extend all the way to the right like it should)
• Fix spacing of top menu and search box (do not look right)
• Group pages:
  • view page: image / logo is really poorly locate and sized, see e.g. ​http://datahub.io/group/wikimedia
  • view page: admin heading on RHS sidebar is wrongly indented

Miscellaneous:

• Ensure that we out-of-the-box config for CKAN has (#2388):
  • Only logged in users able to create
  • Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

• Social extension
• Analytics

Placeholder for thinking about this ...

• Allow users to configure info for CSV import (e.g. separators, quotes etc)

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

• convert non-free fields to vocab fields (eg: interoperability_level)
• store values as JSON files in the ecportal repo
• translate output on dataset view page
• add tests

Some additions to the related stuff functionality to make it more useful post the 1.7 release.

[x] Edit item (after checking auth)

[x] My related things on my user page

[x] Show user created related things on the users' page with links direct to the related page of that dataset. {0.5d}

[x] Related stuff usage counts { 1d }

Record the number of times the link for a related item is clicked. This is likely to mean that the link fires to call CKAN and then redirects to the actual URL.

[x] Un-attached related items {0.25d}

Allow related items to be created without being attached to datasets. This may mean that the dataset field is optional in the API.

[ ] Dashboard for related

A dashboard for related items, site-wide (not package specific) and orderable by date, views, {rating}, popularity of the dataset. Would be useful (possibly) for site-admin to mark one or more as 'featured'

[x] Dashboard + Controller (Related) {1d}

[x] Ordering at logic layer (date, views, rating, popularity of package, featured) {1d}

[x] date [x] views [x] featured

[x] 'Featured' related item with mechanism to fetch it. {0.5d}

[ ] UI within dashboard for sysadmins to check/uncheck feature status {0.5d}

[x] Changing auth for related.

Currently auth is based on deletion being possible by dataset owner and the related owner but it may be that a lot of the auth for related items should be configured to have more restrictive auth.

[x] Implement auth changes for publisher auth profile. { 0.5d }

[x] General issues

[x] Serverside validation of URLs

[x] Activity updates

[x] Make optional

[x] Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

[ ] UI Changes

[ ] General tidy up, fix form etc (icon for menu bar)

[ ] Highlight featured items py:if="related.featured"

[ ] Show either view count or popularity ${ related.view_count }

[ ] A user's related items should be on their home page

[ ] Related item plus sign is very poorly aligned within box

[ ] Add related items should be in drop down like for resources (?) to make it easy to add items

[ ] Hover should show a title that gives some information about what related are (not at all clear)

"Apps, Ideas, Visualization and other material related to this dataset"

[x] Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. "Apps, Ideas, Visualization and other material related to this dataset") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!)

• Went with Apps & Ideas (for now)

[x] Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like ​http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says

• What it does
• How to enable (and perhaps includes a nice-screenshot!)
• How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...)
• Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...)

[ ] Clearly mark the old extension as deprecated from v1.7 forward: ​https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7)

[x] Redeploy Sparql endpoint

Done at linkeddata.openlinksw.com/sparql

[x] Make sure it is available at datahub.io/sparql or semantic.datahub.io/sparql

[x] Generate daily dumps

[x] Make dumps available via web

[x] Notify Hugh once running

[x] Announce to LD guys

[x] Think about how we can apply this to publicdata.eu (can we do the same?) ... (being done elsewhere)

Install and document (step-by-step) CKAN (and dependencies, and extensions) onto VM on s031.okserver.org which has CentOS 6.2 installed. Paying particular attention to the fact that we won't have root access on the staging nor production servers, and so a) anything that requires root access must be documented carefully; and b) we want to be able to perform maintenance without being root.

Maintenance

We want to have non-root access to:

• CKAN logs; (relevant) apache logs; qa process' logs; rdf cron logs
• restarting CKAN
• restarting qa processes
• changing/disabling rdf cronjob
• our database (through sql)
• restarting solr

CKAN and Dependencies

• CKAN installed from source
• Required plugins: synchronous_search ecportal ecportal_form organizations multilingual_dataset multilingual_group multilingual_tag qa

Extra configuration

• uploads
• analytics (toby)
• qa
• rdf cron
• caching
• datastore

Things to remember

• Use ckanext/multilingual/solr/schema.xml and stopword files
• Use nginx as proxy.

Provide some provisional user stories for the CMS/Blogging feature. Should include some of the functionality from wordpresser but be more generic.

• Do an analysis of why upgrades are problematic
• What can we do better
• Decide what to implement

Should minify and concatenate JavaScript? and CSS as part of #2354

ditto

Need better documentation on best practices in making page loads faster for non logged in users.

What would as a multisite CKAN look like?

This is really part of the work around turnkey/hosted CKAN

Currently an eye and a number, needs to look a little nicer.

aims

• allow templates to specify needed css/js

• allow split js files to be bundled (including plugin supplied if possible - stage 2)

• serve minified in production/ non-minified in dev

• possibly look at doing the minification via paster - maybe as different ticket

As designed by Sam Smith -> ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/ Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same.

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

Provide a new test install of DataGM based on the ODS demo site being completed through June.

The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData? etc and will be used by DGU.