DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how.

Change login cookie from a default expiry of 50 years to 2 years. You can also uncheck a 'remember me' checkbox on the login form for the cookie to just last the session.

Background conversation on ckan-dev:

DR: I wonder if anyone objects to the expiry of the login cookie to be changed from 50 years to 2 years? 50 years might be appropriate for thedatahub.org, but for government sites it seems (to me) to be too lax.

Toby: is this the repoze.who cookie? If so that seems sensible to me.

Rufus: Definitely agree. I would also like to see introduction of a standard "remember me" checkbox (set to true by default). At the moment a login lasts forever (until you logout) automatically.

• Add home page url
• Remove group option (seen it confuse several users) - perhaps replace with tags (?)
• Check form looks ok in portugese (seems to be a problem with css)

Use jquery chosen?

Would make them much nicer but requires:

• Ensure uniqueness (or deal with ambiguity)
• (?) Slug/names versus titles
  • Aside: what is currently called name should probably be title

Apparently auto-complete in Organizations may not work. Check and fix.

See ​https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5

Also need to fix bug where 'Other' is never highlighted.

• Add Ascii folding filter to text fields.
• Add capacity field for public, private access.
• Add title_string so you can sort alphabetically on title.
• Fields related to analytics, access and view counts.
• Add data_dict field for the whole package_dict.

Actual implementation of data_dict adding will be added later.

E.g. on dataset page, on revision page, on user page and on search results.

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

• Related Item has comments on index page (they should not show up here -- why is this happening!)
• Related item should be to left of history (history could in fact move to RHS)
  • What about losing icons since we duplicate for related item
• Related item plus sign is very poorly aligned within box
  • Add related items should be in drop down like for resources (?) to make it easy to add items
  • Hover should show a title that gives some information about what related are (not at all clear)
• Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

• #2331 - search should add not or
• Search results should be similar across group pages and normal search results (suggest adopting group page style)
  • Search results resource link should link to resource page rather than url for that resource
  • Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

• Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) ​https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07
  • Ensure still using read-only (this seems to have been lost in upgrade)
• Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

• (? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
  • Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
  • Do #2322 ...
• Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
• Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
• Data viewer embed url includes all the resource info

Dataset page:

• Put home page link somewhere more prominent (e.g. the top of the sidebar)
• Put author more prominent (at top right)
• (? - new work) List CKAN owners at top right in sidebar
• Make resources stand out more
• Rename resources tab and heading to Data & Resources

Dataset edit and create:

• Corrections to dataset creation form for v1.7 - #2317
• Better auto-complete for groups on dataset edit/new page - #2319
  • Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
• Description / notes text area is not wide enough.
  • Should also be larger on edit page
• Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
• Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
  • A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
  • Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

• Do not show no description in no description (just show dataset info)

Theme:

• Fix spacing above page title (i.e. dataset title etc)
• Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer
• Fix subnav width on resources pages (does not extend all the way to the right like it should)
• Fix spacing of top menu and search box (do not look right)
• Group pages:
  • view page: image / logo is really poorly locate and sized, see e.g. ​http://datahub.io/group/wikimedia
  • view page: admin heading on RHS sidebar is wrongly indented

Miscellaneous:

• Ensure that we out-of-the-box config for CKAN has (#2388):
  • Only logged in users able to create
  • Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

• Social extension
• Analytics

Placeholder for thinking about this ...

• Allow users to configure info for CSV import (e.g. separators, quotes etc)

For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same.

Either fix the docs or fix CKAN to use same ini file names in package and source installs/

• convert non-free fields to vocab fields (eg: interoperability_level)
• store values as JSON files in the ecportal repo
• translate output on dataset view page
• add tests

Some additions to the related stuff functionality to make it more useful post the 1.7 release.

[x] Edit item (after checking auth)

[x] My related things on my user page

[x] Show user created related things on the users' page with links direct to the related page of that dataset. {0.5d}

[x] Related stuff usage counts { 1d }

Record the number of times the link for a related item is clicked. This is likely to mean that the link fires to call CKAN and then redirects to the actual URL.

[x] Un-attached related items {0.25d}

Allow related items to be created without being attached to datasets. This may mean that the dataset field is optional in the API.

[ ] Dashboard for related

A dashboard for related items, site-wide (not package specific) and orderable by date, views, {rating}, popularity of the dataset. Would be useful (possibly) for site-admin to mark one or more as 'featured'

[x] Dashboard + Controller (Related) {1d}

[x] Ordering at logic layer (date, views, rating, popularity of package, featured) {1d}

[x] date [x] views [x] featured

[x] 'Featured' related item with mechanism to fetch it. {0.5d}

[ ] UI within dashboard for sysadmins to check/uncheck feature status {0.5d}

[x] Changing auth for related.

Currently auth is based on deletion being possible by dataset owner and the related owner but it may be that a lot of the auth for related items should be configured to have more restrictive auth.

[x] Implement auth changes for publisher auth profile. { 0.5d }

[x] General issues

[x] Serverside validation of URLs

[x] Activity updates

[x] Make optional

[x] Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

[ ] UI Changes

[ ] General tidy up, fix form etc (icon for menu bar)

[ ] Highlight featured items py:if="related.featured"

[ ] Show either view count or popularity ${ related.view_count }

[ ] A user's related items should be on their home page

[ ] Related item plus sign is very poorly aligned within box

[ ] Add related items should be in drop down like for resources (?) to make it easy to add items

[ ] Hover should show a title that gives some information about what related are (not at all clear)

"Apps, Ideas, Visualization and other material related to this dataset"

[x] Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. "Apps, Ideas, Visualization and other material related to this dataset") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!)

• Went with Apps & Ideas (for now)

[x] Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like ​http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says

• What it does
• How to enable (and perhaps includes a nice-screenshot!)
• How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...)
• Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...)

[ ] Clearly mark the old extension as deprecated from v1.7 forward: ​https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7)

[x] Redeploy Sparql endpoint

Done at linkeddata.openlinksw.com/sparql

[x] Make sure it is available at datahub.io/sparql or semantic.datahub.io/sparql

[x] Generate daily dumps

[x] Make dumps available via web

[x] Notify Hugh once running

[x] Announce to LD guys

[x] Think about how we can apply this to publicdata.eu (can we do the same?) ... (being done elsewhere)

Install and document (step-by-step) CKAN (and dependencies, and extensions) onto VM on s031.okserver.org which has CentOS 6.2 installed. Paying particular attention to the fact that we won't have root access on the staging nor production servers, and so a) anything that requires root access must be documented carefully; and b) we want to be able to perform maintenance without being root.

Maintenance

We want to have non-root access to:

• CKAN logs; (relevant) apache logs; qa process' logs; rdf cron logs
• restarting CKAN
• restarting qa processes
• changing/disabling rdf cronjob
• our database (through sql)
• restarting solr

CKAN and Dependencies

• CKAN installed from source
• Required plugins: synchronous_search ecportal ecportal_form organizations multilingual_dataset multilingual_group multilingual_tag qa

Extra configuration

• uploads
• analytics (toby)
• qa
• rdf cron
• caching
• datastore

Things to remember

• Use ckanext/multilingual/solr/schema.xml and stopword files
• Use nginx as proxy.

Provide some provisional user stories for the CMS/Blogging feature. Should include some of the functionality from wordpresser but be more generic.

• Do an analysis of why upgrades are problematic
• What can we do better
• Decide what to implement

Should minify and concatenate JavaScript? and CSS as part of #2354

ditto

Need better documentation on best practices in making page loads faster for non logged in users.

What would as a multisite CKAN look like?

This is really part of the work around turnkey/hosted CKAN

Currently an eye and a number, needs to look a little nicer.

aims

• allow templates to specify needed css/js

• allow split js files to be bundled (including plugin supplied if possible - stage 2)

• serve minified in production/ non-minified in dev

• possibly look at doing the minification via paster - maybe as different ticket

As designed by Sam Smith -> ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/ Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same.

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

Provide a new test install of DataGM based on the ODS demo site being completed through June.

The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData? etc and will be used by DGU.

• Only logged in users able to create
• Creation permissions have only creators able to update a dataset
• Synchronous search is enabled (#1678)

An extra field on datasets called "metadata_language". The language that the meta-data is in. Drop-down list needs to be translatable. Default value is "English".

We need to separate the CLI part of CreateTestData? from the bit which creates test fixtures. This is because of the problem with imports and logging. Otherwise we have to do unpythonic things to get logging to work when deriving from CreateTestData? from an extension such as ecportal / dgu. I've only introduced logging into CreateTestData? but I think it is a good thing to have. This change doesn't cause breakages

Search results are slow as we do many package_dictize. Store the package_dict in the search index so that we do not have to hit the database for each of them.

As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere

Implementation: (?) Use phantomjs or something like ​http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct.

The settings > extras page for a dataset allows extra fields to be added, but only four can be added a time in a single atomic update. It would be helpful to have an 'add another' button on the form that can be used any number of times.

Taken from ​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002218.html

A few months ago, the RDF provided when dereferencing a dataset's URI was very useful. It described the datasets using many of the "right" RDF vocabularies (dcterms, void, etc).

The RDF that is returned now no longer has these nice descriptions. Instead, it seems to be a mindless transcription of the CKAN record.

For example, the older descriptions gave:

<​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>

void:subset [

void:target <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <​http://thedatahub.org/dataset/geonames-semantic-web> ; void:triples 50 ; a void:Linkset

], [

void:target <​http://thedatahub.org/dataset/dbpedia>, <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ; void:triples 52 ; a void:Linkset

], [

void:target <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <​http://thedatahub.org/dataset/govtrack> ; void:triples 52 ; a void:Linkset

] ; void:triples 130005 ;

but now it's simply:

<​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>

dct:relation [

rdf:value "130005" ; rdfs:label "triples"

], [

rdf:value "52" ; rdfs:label "links:dbpedia"

], [

rdf:value "50" ; rdfs:label "links:geonames-semantic-web"

], [

rdf:value "52" ; rdfs:label "links:govtrack"

] ;

My old copies of the dataset descriptions mentioned a "GoLD CKAN 1.0" that was credited for the nicer modeling.

Where did that go? Can we get it back?

Thanks, Tim Lebo

<​http://semantic.ckan.net/record/91d2c0de-75a4-4bb6-b260-bc2946e1be8b>

dc:source <​http://ckan.net/> ; opmv:wasGeneratedBy [

opmv:used <​http://ckan.net/api/rest/package/farmers-markets-geographic-data-united-states>, <​http://ckan.net/api/rest/revision/9c2545b8-ebf1-42bf-ac92-633ff5ceba3f> ; opmv:wasControlledBy [

a foaf:Agent ; foaf:name "GoLD CKAN 1.0"

] ; opmv:wasPerformedAt [

a time:Instant ; time:inXSDDateTime "2012-01-31T05:34:32Z"^^xsd:dateTime

] ; a opmv:Process

] ; a dcat:CatalogRecord ; dcat:dataset <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ;

…

Take revelvant parts of autoneg ( ​https://github.com/wwaites/autoneg.git ) so that we can remove the dependency (just for two functions) and thereby remove flup as well.

Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json”

See the following Basecamp thread for UI examples and discussion.

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056

Currently facets in the new demo site disappear from the sidebar once they are selected.

​http://s031.okserver.org:2375/dataset?tags=fibre&tags=terrestrial&q=Africa

Instead they should be marked with an "active" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works.

create a depreciated decorator to wrap functions we intend to remove

logs a warning maybe check that 'depreciated' is included in the docstring too

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments

Implementation:

​http://s031.okserver.org:2375