filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters)

pass some stuff through the context

​http://localhost:5000/ckan-admin/authz

you need to be an admin

paster sysadmin add <username> -c dev.ini

like #2741 we need other admin links

you may want to make the inputs use your form macros - I've just done a basic port

also we will need to get all the tables done but looks like we need to get auth groups working for that

We need a way to pass in schemas to the logic layer to deal with edge cases.

to use on demos instead of the ckan logo & tag line.

Design here: ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381 (First, text only one at the bottom)

add resource_status_show action function to see the state of celery tasks.

Need to get information from both out task status table and the celery_taskmeta table.

with add button on dataset page and in 'data & resource' section when there's no data

In the footer as well as more clearly & directly on the About page

the stages snippet needs to be inside <form> so I can make some links buttons - eg stage 1 -> stage 2 I want to save the dataset not just move to add resource

if you can just move it within the form html leaving everything else the same - I can then add buttons and pass back for styling

disqus??

Suggest making this consistent with ​http://s031.okserver.org:2375/dataset/adur_district_spending/related by adding same button in top right of resource pages

if not logged in then go to add dataset url the language is lost

check if this is a ckan 1.7 issue

If someone who is not logged in clicks on "Upload Data", they are presented with

    Not authorized to upload files.

This is in emboldened red text. That is, the system is telling a user that they're doing something illegitimate, even though they're actually following an invitation from the main menu.

I propose two options for resolving this:

1. only display the "Upload Data" tab if someone is authorised to upload data
2. if someone isn't logged int, provide a nice welcome message thanking them for wanting to upload data, but we only allow registered users to do so

ENV: thedatahub.org, CKAN 1.4.2b

The api currently uses the default validation schemas regardless of the form you specified on the front end. There should be a way to post through the api using the same validation rules.

Main tasks:

• Integrated seamlessly into CKAN (authorization etc)
• Based off ElasticSearch?

Etherpad: ​http://ckan.okfnpad.org/webstore

Tasks

• New API point (config + controller) [0.75d]
  • /api/data/{resource-id}
• Authorization [0.5d]
• Documentation [0.25d]
• Upgrade data viewer to use ES [0.5d]
• Convert webstorer to new setup - ​http://github.com/okfn/ckanext-webstorer [1d] - #1753
  • See below
• "Create DataStore? Resource" in Resource creation (??)
  • Just creates a resource with blank url and datastore enabled (webstore_url non null)
• Tick box to enable DataStore? in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)
  • Also need to use this info in DataStore? extension?
• Create nice instructions on Resource page about using the DataStore?
• Rename webstore -> datastore
• Deploy [0.5d]

Details

Layout on ES

elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)

Request path

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: ​https://github.com/okfn/elastic-proxy

Questions

1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy?
   • Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
2. How do we decide when webstore is activated
   • Ans: see previous.
3. Do we (and how do we) allow for resources which are just for webstores
   • Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ...
4. How does webstorer work ...
   • CSV import - NS: make sure we add e.g. rownum or id = rownum ...
   • Separate mini-service (webstorer.thedatahub.org ...)
5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy?)
   • Ans: do not bother at present. Focus on Webstore route.
6. Where does geodata support come in (in Recline or separately)
   • This is a separate issue

User Story 1

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

1. Allow normal upload process
2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
3. Update the resource in some way (set webstore_url)

User Stories

User Story 2

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)

User Story 3

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)

Provide help to users in the WUI, with this linking to a detailed user guide.

• User guide (suggest at help.thedatahub.org (or possibly help.ckan.org)
• Snippets in the WUI with links to full user guide.
  • Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis
• General info about publishing/working with/etc data (where we don't duplicate elsewhere)

Est: (?) 5-10d

Topics

See ​http://ckan.okfnpad.org/datahub-help

• Step-by-step for adding a dataset including tips
  • Reuse: ​http://datapatterns.org/datahub.html
  • Quick overview of relevant parts of domain model (dataset and resource)
• API (perhaps link to main docs)
• Command line interface and dpm
• Using groups

Why do this

We already have wiki.ckan.org and docs.ckan.org. Why do this?

Answer for docs:

• This is administrative guide and focused on general CKAN software.
• Hard to provide completely generic help.
• Integratable with the DataHub? site

Answer for wiki:

• No spam
• General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful.
• Integratable with the DataHub?

Re-create behaviour of old form without using form-alchemy. And unit test.

• [X] write high-level functional tests for the form. Creation / edition / validation.

Original estimate: 6 days (3 days x 2)

Time spent: 3 days

Time remaining: 0 days

• [X] write the html

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 0 days

• [X] modify the existing validation schema

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 1 days

• [X] split resources into types: individual, time-series, and additional.

Original estimate: 2 days

Time spent: 2.5 days

Time remaining: 0 days

• [X] contact information is derived from Publisher, but can be altered for each dataset. #1617

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

(Waiting on publisher integration)

• [X] foi contact information is derived from user, but can be altered for each dataset. #1617

Original estimate: 1 days

Time spent: 0 days

Time remaining: 1 days

(Waiting on publisher integration)

• [X] provide stubs for the theme and sub-themes #1618

Original estimate: 1 days

Time spent: 0 days

Time remaining: 0 days

• [X] provide stubs for the publisher hierarchy #1619

Original estimate: 2 days

Time spent: 0.5 days

Time remaining: 1.5 days

Note - the groups refactor is essentially ready to be used. And this would be a good test case for it.

• [X] don't allow package-type selection when editing a package

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (#1620)

Original estimate: 0.5 days

Time spent: 0 days

Time remaining: 0 days

Historical Parent Tickets

• Parent-ticket: #1450
• Parent-ticket: #1032

Step Zero

• COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

• Force format field to be lowercase
  • Modify backend to serve it up in that format
• Subticket #978: Add extra fields to resources in Web UI
• Fix display of large datasets (​http://test.ckan.net/dataset/europeana-lod)
• When submitting forms with errors, open the correct resource pane.
• No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
• Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

• Subticket #1648: Clarify edit terminology
• Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
• Fix (i.e. allow / show) name editing on dataset edit.
  • Fix the slugify so it does not overwrite if i change title after fixing name.
  • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
• Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

• Fix animation glitch in expanding description
• Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
• Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

• "Upload or link data files, APIs and other materials related to your dataset."
• Include the description markdown editor again (keep it small)
• Licence should maybe be restored
  • "(Don't worry if you don't know what your licence is)"
• Show the log message field
  • Beautify
• Pre-populate the log message field?
• ? Show Home Page field when creating dataset [WONTFIX don't push it]
• ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

• Improvements to user page UX - 1d - #1396
• Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

• Take before and after screenshots

Add drupal-like "taxonomies" to CKAN.

Etherpad with user stories, feature list, design and implementation discussion:

​http://ckan.okfnpad.org/22

Branch where this is being developed is feature-1698-tag-taxonomies:

​https://github.com/okfn/ckan/compare/master...feature-1698-tag-taxonomies

Tickets related to this have ​keyword taxonomies.

Add a way to share datasets and see mentions of datasets on twitter (and elsewhere)

As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it).

• Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc.
  • Will need a config option for relevant API key e.g. ckan.sharethis.apikey
• Location: Put this at top of sidebar on dataset view (a better suggsetion?)
• Should show how many mentions / shares there are. (very important!)

Discussion

• (Tom and Ira and ...): do we really want this, versus e.g. proper follow extension?
• What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1.
• Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?)

1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)

2. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).

3. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.

4. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu

5. Blue search button should be displayed in line with the search bar, not underneath

6. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

• New resource group table. #956
• New kind field in resource. #957
• UI for new kind field. #958
• Resources in REST API ticket:358
• Resources in WUI #1445
• Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

• ​http://ckan.okfnpad.org/resources - discussion etherpad
  • Contains commented version of James Gardner's main email: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
• UseCasesResources

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: ​http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

• Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports

Work Items

• Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
• Generate archiving request on resource url change [0.25d][0.25d] - #1399
• Make site user account.
• Make entry point system for celery config
• Archiver daemon #891
  1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
  2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
• Write to resource and task status table.[1d][0.75d]
• [Required?] Make archived data available in WUI - #892
• Documentation - #1400

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

1. Generating (or just filling) a form template with 'form data' (and errors)
2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

• #926 - Pick a simpler form framework
• #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
  • #1079 Refactor API to use new logic layer and dictization
  • #1078 Refactor WUI controllers and forms to use logic layer
  • cf existing dumper and importer code
  • This will fix #662
• [not ticketed yet] - validation layer (should work on serialized objects?)
• #662 - Can't put entity that is returned by posting to package register (Defect)
• #972 - Merge 'extras' into main package dict rather than having separate key
• #1035 - Form impressions are given IDs
• #810 - Move "add packages" field up in group form (easier to do this once forms are done)

Documentation is key to the success of projects!

Move material into main docs from wiki

• ​http://wiki.ckan.org/Writing_asynchronous_tasks
• ​http://wiki.ckan.org/Domain_Model
• ​http://wiki.ckan.org/Deployment

Create a User Guide

Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. ​http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/)

• Publishing data
• Authorization and workflows
  • Publisher Profile (Workflow)

Break into sections

Suggest something like:

# General
* Intro
* Conceptual Overview
* What CKAN helps you do (http://ckan.org/)
* FAQ ? (or point to the wiki)

# For Administrators

* Installation and Setup
  * Configuration options
* Customization - Theming etc
* Authorization and Workflows
* Storage

# For Users (Publishers, Data Wranglers, etc etc)
* Walkthrough of publishing a dataset
* Storage
* API (see Developers section)

# CKAN Developers

* Domain Model
* API
* Extensions
* i18n

CKAN Developers
* Buildbot

Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement.

• Integrate QA into frontend - 1d - #1583
• QA report improvements - 2.5d - #1584
• QA - Check license as part of 5 star rating - #1587
• QA - Give SPARQL endpoints a 4 star rating - #1588
• QA - Give 5 star rating to datasets with link metadata - #1589

Main pad: ​http://ckan.okfnpad.org/qa

Some QA discussion (relating to Celery in particular) at ​http://ckan.okfnpad.org/queue

Largely an integration of work from ckanjs

• #1295 Simplify package create form
• Improvements to edit form
  • Split edit form into sections
  • Remove preview from edit form (introduce inline preview on description/notes field)
• #1296 Improved resource adding/editing on dataset/page page
• #878 File upload integrated into resource creation
• #1297 In-place editing of notes field for existing packages

Proposer: John Glover
Seconder: James Gardner

Abstract

We are trying to achieve these goals:

• To get people involved with making edits to CKAN metadata.
• To have an ownership model as to who can moderate and validate these changes
• To not put too huge a burden on these owners.

This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made "official".

There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface.

The Problem

We require the following functionality:

• Allow a group of changes to be stored as a new revision.
• Allow a linear stack of "community" revisions.
• Provide a way for the editor and moderator to compare previous revisions to the current one.
• When a moderator approves a change it creates a new revision flagged "moderated" (this is analogous to a merge commit)
• Provide a way for the editor and moderator comment on revisions if necessary.

Extra features:

• Need a way to summarise the changes (as part of the preview perhaps)
• Sysadmin needs to purge a revision completely

Specification

UI/UX

UI Mockup:

Revisions:

• Revisions are per package rather than per field.
• Internally CKAN has separate revisions for resources, extras and package metadata. From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision.

On the Edit page:

• We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?).
• The values displayed in the form are by default populated from the latest revision (whether community or moderated)
• Under each field is a "shadow", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too.
• When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it
• If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named "Replace fields with values from this revision" under the revision list. You click this, a warning pops up and then you say "Yes". You then select the moderated revision again.
• We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this.
• In addition to package comments, each revision will have a revision log (analogous to a commit message).

Technical Details

• This CREP will result in a new CKAN extension.
• It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised.
• This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use.
• We will then use suitable test data to mimic these API calls until CREP0002 is ready.

Why do it this way

This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX.

Implementation plan

Deliverables

A new CKAN extension, consisting of:

• Code: Python, HTML, CSS, Javascript
• Unit tests
• Localization
• Documentation

Participants

John Glover to do it.

Progress

John has implemented the bulk of this UI. Just some things to tidy up before it is complete:

• Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates
• history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API)

I've split these two off into a new ticket #1604.

Related Progress

The Todo extension is written and available at: ​https://bitbucket.org/johnglover/ckanext-todo.

In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done.

See also

#1129 Backend work

Child tickets:

• #1041 Start Using the CKAN Wiki for Tutorial-style documentation
• #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): ​http://trac.ckan.org/ticket/927

• CKAN 1-page overview (for enterprise and for data hackers)
• Administrator's Guide (including install)
• Extensions Guide
• Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: ​http://wiki.ckan.net/Documentation_Plans

Minor Items

Child tickets:

• #1194 "Welcome back" message for newly registered user
• #1202 Links to datapkg utility don't lead to info about it
• #925 Change the search box icon to remove the down arrow
• #923 Search box doesn't work in leaderboard page in stats extension
• #1034 Flash message cached
• #737 Markdown syntax summary page
• #811 Extra field editing form layout breaks when there are long field names

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

• #191 Searching by modification date
• #193 Searching by time-related field
• #301 Package discussion pages
• #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
• #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
• Other improvements as per my word doc.

Various UX and UI improvements to groups.

Etherpad: ​http://ckan.okfnpad.org/feature-1521-groups-ux

Improved groups

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as ​http://ie.ckan.net/ could be turned into a simple group on the ​Data Hub without losing their separate identity.

• Group homepages: Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.
• Subdomains for groups: Enable subdomains such as ie.thedatahub.org as aliases for thedatahub.org/group/ireland
• Search within the group: Have a big search box that searches just within the group on the group page, with a look like on ​http://thedatahub.org/dataset
• Have an Add a dataset to this group link on the group page, perhaps in the sidebar.
• Navigation within the group: Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from ​http://thedatahub.org/dataset
• Group documentation: Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.
  • The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.
  • Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines About, Frequently Asked Questions and How to get involved, we'd then get three links in the sidebar that point to datahub.org/group/ireland/about, datahub.org/group/ireland/frequently-asked-questions, datahub.org/group/ireland/how-to-get-involved
    • RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

Additional

• Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)
• Better group index page (sort by number of dataset, allow searching ...) - #1403
  • Add search to groups: ​http://thedatahub.org/group
  • Way to filter by activity, number of datasets
• #1597 Tag search, filtered by group
  • RP: not worth it in terms of benefit/cost IMO

Steps

• "Add Dataset" button on action bar
  • Make it a dropdown? -> New Dataset / Existing Dataset
  • Redirect Add Existing -> Edit
  • Check permissions
• Expandable "about this group" box
• Implement simplest search-this-group box
  • turn the list of datasets into a dynamic list of search results
    • pagination
  • Matching set of facets in the sidebar (matches 'real' search page)
    • Deleteable, displayed facets

Nice-To-Have

• Get rid of the old group_package_show action (confirm with team)
• Tabbed "about this group" box (extract headers?)
• Scrollable group members widget in the sidebar

• (Deploy it)
• Integrate with stats plugin (now that is in core) - #1101
• Blog about it #1581
• (Other tickets re upgrading for v1.5.1 - done but where are tickets?)

3 (4) use cases: (With and without access to auth keys.)

1. upload form on ckan (by default local system has auth key for storage)
2. datapkg upload (client) (by default without key) (plumbing done)
3. background job archiving downloads (by queue client)

Most important are:

1. Upload form on ckan.net (with access to keys)
2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

• #826 - Resource 'extra' fields
• #358 - Resources in REST API
• #235 - Resource format normalization and detection

User upload via Web UI

• #877 - File upload in WUI
• #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

• #853 - plumbing for credential-less rest upload
• #879 - Storage auth API

Also related:

• #405 - Retrieval options for package resources

More Info

• Upload/storage discussion: ​http://ckan.okfnpad.org/storage
  • For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: ​https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

• 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1
• 6. Publisher Registration Improvements MUST PHASE 1
• 3. Themes/Taxonomy? as well as tags WONT PHASE 2
• 32. Browse by Tags PHASE 1 MUST
• 12. Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1
• 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]
• 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]
• 22. Mixed Licenses MUST PHASE 2

Plus, brought forward from last sprint:

• Publisher hierarchy support
• Improved dataset creation tools (form) - #1525
• Edit/delete for harvested datasets

Overview of tasks: see the design doc.

• Basic geo map
• Facets
• Filtering
• Pivot table support (?)
• Export data (?)
• Save data query and share
• More and better graphing
• Embedding! - /resource/../embed

User stories and analysis: ​http://ckan.okfnpad.org/feature-2283-data-viewer-v3

Form refactor. New create/edit package form wizard for DGU.

Etherpad: ​http://ckan.okfnpad.org/dgu-package-form

Tasks:

• #1526 : Simple one-page form.
• #1527 : Tab-ify the form using JS. No validation between steps.
• #1541 : Setup server for the DGU form-refactor.
• #1537 : Package create form wizard
• #1538 : Store partially-filled in package form data as a draft.
• #1539 : Package forms auto-save

How the rest of the work is carried out is up for discussion, as it depends on the approach taken. Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages. ​Google doc outlining design(s)

Child tickets

• #1198 Publisher hierarchy
• #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
• #1004 Group creation instructions missing
• #1099 Strange interactions between two browsers while playing with authz groups
• #1115 can have two authzgroups with the same name
• #1133 command line rights manipulation doesn't work
• #1138 minor navigations behave inconsistently

Old ticket description:

1. Change name of AuthzGroup? to UserGroup? to reflect what it is for

2. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
   • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
   • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)

3. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

• introduce model.Site and c.site
  • site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
• Subdomain detector to activate sites.
• use site in Authorizer instead of System, have a NullSite? for global things
• allow cross-site search
• packages are in a list of sites, m:n rather than 1:n
  • list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.

We want to add "activity streams" (chronological lists of activities) to CKAN.

Activity streams standard: ​http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): ​https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: ​http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: ​http://datahub.pudo.org/pudo

Tasks

Strikethrough means the task is done.

• New domain objects Activity and ActivityStream? [2d] #1298

• Migrate existing data into activity streams model [3.5d] #1510

• Generate activities when objects are saved [4d] #1298

• Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511

• API call for getting a user's public activity stream as JSON [0.5d] #1495

• API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494

• Render a user's activity stream on her user page [3d] #1496 (needs review/page design work)

• Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.

• Logic functions for getting activity streams for packages, groups, and users.

• API calls for getting activity streams of packages, groups, and users as JSON.

• Rendering activity streams for packages and groups etc. into their pages. (done but disabled pending page design work)

• API calls for inserting events into activity streams. (Still need to handle activity details in the API, ticketed).

• Conform to activitystrea.ms standard

• Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.

• Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams

• Email notifications

• Extension point for extensions to register themselves as renderers for different types of activity

• Deploy to UAT [0.5d]

• Docs and announce [1d]

(Just creating this ticket as somewhere to keep notes)

• Decide on REST api versus action API
  • Do we want to support both?
• Tidying
  • Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
  • extras merged into normal field list in package
  • Get rid of /rest/ so just have api/v1/package
  • Get rid of separation of search api from 'rest' api
    • Propose that GET on REST index is search e.g. /package/?q=...
      • This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
• Resource read/write in API (separate from package)
  • Does this need authorization work?
• user/account API - read/write
• Remove autocomplete -- can just use search
  • Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups.

• Simple group count widget in JS for embedding in wordpress and elsewhere
  • Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box.
• All you need to do is do a dataset query over the API e.g. ​http://thedatahub.org/api/search/dataset?groups=economics and then embed in some html!

Estimate: 30m (for someone who knows their jquery).

This is the implementation of the final draft in

​http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

• model migration. (done)
• simplify revisioning (done)
• Remove any sqlalchemy releations from groups. (done)
• change logic functions (done)

Single type per group (#1531)

• migration add type to group table. (done) 1d

Add capacity table (#1522)

• model migration. (done)
• add to logic functions.

Allow other objects to be added to groups. (#1531)

• change logic functions (done)

Adapt thedatahub.org to use new model.

• move auth groups over to groups/members
• change user_object roles to use groups? keep user object roles for time being.
• make create update package/group put the correct items in the group/members/capacity tables.
• work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

• adapt auth functions to use auth.
• adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

• #1279 Deprecate Authorization Groups
• #1198 Publisher hierarchy

If you try to GET a dataset, purge it then POST it again, it fails checking authz for the group:

Module ckan.controllers.api:278 in create
<<                  (register, subregister))
               try:
                   response_data = action(context, data_dict)
                   location = None
                   if "id" in data_dict:
>>  response_data = action(context, data_dict)
Module ckan.logic.action.create:240 in package_create_rest
<<      api = context.get('api_version') or '1'
       
           check_access('package_create_rest', context, data_dict)
       
           dictized_package = package_api_to_dict(data_dict, context)
>>  check_access('package_create_rest', context, data_dict)
Module ckan.logic:129 in check_access
<<          #    log.debug('Valid API key needed to make changes')
               #    raise NotAuthorized
               logic_authorization = new_authz.is_authorized(action, context, data_dict)
               if not logic_authorization['success']:
                   msg = logic_authorization.get('msg','')
>>  logic_authorization = new_authz.is_authorized(action, context, data_dict)
Module ckan.new_authz:16 in is_authorized
<<      auth_function = _get_auth_function(action)
           if auth_function:
               return auth_function(context, data_dict)
           else:
               raise ValueError(_('Authorization function not found: %s' % action))
>>  return auth_function(context, data_dict)
Module ckan.logic.auth.create:115 in package_create_rest
<<          return {'success': False, 'msg': _('Valid API key needed to create a package')}
       
           return package_create(context, data_dict)
       
       def group_create_rest(context, data_dict):
>>  return package_create(context, data_dict)
Module ckan.logic.auth.create:15 in package_create
<<      else:
               
               check2 = check_group_auth(context,data_dict)
               if not check2:
                   return {'success': False, 'msg': _('User %s not authorized to edit these groups') % str(user)}
>>  check2 = check_group_auth(context,data_dict)
Module ckan.logic.auth.create:90 in check_group_auth
<<      groups = set()
           for group_dict in group_dicts:
               id = group_dict.get('id')
               if not id:
                   continue
>>  id = group_dict.get('id')
AttributeError: 'unicode' object has no attribute 'get'

(using RESTful Model API)

The reason is the POST is expecting the group to be expressed as a dict, not the list of names that GET returns.

see ​http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873

what XLS previews do we support?

The flags in the dropdown on details pages are not right.

You can see the error on ​http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0.

Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)