See ​http://ckan.okfnpad.org/search for details/discussion

When paster db clean is run, the search index should also be cleared.

Take the resource archiver (currently part of ckanext-qa) and make it into a standalone extension that works as a celery task.

Currently can run as a celery task launched by paster command.

Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource.

Should list resources in the form of search results rather than as a table.

• css class: resource-url-analytics
• Assign to the link tag (a) everywhere we want to count (dataset, resource view) - 0.25d
• Display counts in same place ...

And deploy on ​http://thedatahub.org/ - 0.25d

Possible: Also move analytics extension into core (decided not to).

• Write up: task status table in ​http://wiki.ckan.org/Domain_Model
• Write up: ​http://wiki.ckan.org/Data_Quality to reflect current state of play (move all old material to "Discussion / Talk" page)
• Ping ckan-discuss re new Data Quality page

• deploy celery
• deploy QA and archiver tasks
• write up a blog post announcing QA on thedatahub

Super: #1594

• qa/{username}
• qa/{groupname}
• paginate QA results
• search / filter QA results
• list organisation report by default, but can disable via config option (done)
• UX tidy up of report pages - hide border if no sidebar, etc

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type "api/sparql". A few more are recorded with the format "SPARQL". I suggest that datasets with such resources should also be considered for the fourth star.

Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement.

• Integrate QA into frontend - 1d - #1583
• QA report improvements - 2.5d - #1584
• QA - Check license as part of 5 star rating - #1587
• QA - Give SPARQL endpoints a 4 star rating - #1588
• QA - Give 5 star rating to datasets with link metadata - #1589

Main pad: ​http://ckan.okfnpad.org/qa

Some QA discussion (relating to Celery in particular) at ​http://ckan.okfnpad.org/queue

To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function.

This behaviour is now deprecated however.

Part of #1698

Need to test:

• API call to create new vocab
• API call to add tags with a given vocab
• API call to update tags
• Dummy plugin to test that updating tags works via WUI

• Tag list not be added to CKAN core, what is the best way to do this in an extension?
• Does the list need to be updated via API/WUI?

• see how this currently works with DGU first

Should describe implementing IDatasetForm, subclassing controllers is deprecated

• code review with D.R. for ckanext-ecportal
• merge and test all multilingual code
• fix server issues: routing, Git access to CKAN repository

Bugs to fix:

• rename groups to publishers
• update home page layout (recent updates along side publishers)
• don't add 'en' to main menu links if english
• resource edit page, fix missing images
• change dataset 'settings' to 'edit'
• translate 'published by' drop-down list
• fix JS on 'add a dataset page'
• prepopulate 'published_by' when clicking 'add a dataset' from group
• chosen broken (dataset edit) if in different language

I won't be needed for this during the current sprint, closing.

Changes:

• tidy up activity feed on home page or replace with search results (by modified date)
• move ckan menu to a left sidebar (see Sam's template) (not doing this now)
• update templates with changes from master
• use local image in language dropdown (ecportal.js)
• update dataset read template

To discuss:

• should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping.

Check that lists of strings are being correctly translated. See ​https://github.com/okfn/ckan/commit/f1d68c3d2d4d25a0c0f8a89a68940643fc19b156

Reported on IRC by 'floapps'.

• when posting a group update with {packages: [package-1, package-2, etc.]}, getting ValidationError?: {'Packages': {'name': Missing value?}}, although package objects did contain a name field
• same API call worked with organization and publisher profile disabled
• also worked if added a form_to_db_schema_options function which used default_update_group_schema for api updates instead of default_group_schema
• tried on both 1.7 and 1.7.1b

Looks like some of the style has been updated/changed on the group read page but the changes haven't been applied to the organizations read page.

• branch from 1.7.1
• apply CKAN patches that are current part of installation instructions
• create i18n patch so custom i18n directories can be specified via a config option

EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested.

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

• datastore_create
• datastore_delete
• datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}

• The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
• rows: can be empty so that you can just set the fields
• fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

• Any error results in total failure!! For now pass back the actual error.
• Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: ​https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

​http://www.postgresql.org/docs/9.1/static/datatype.html

​http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

As a

Visitor

I want to

Login to the service

So that

I am now an authenticated user and may perform the actions that only authenticated users may perform

Notes

• Login by providing username and password
• Login screen should also include a link or information about registration
• If username do not match (or the username does not exist) a login failure is noted and the visitor is re-presented with the login screen as well as information on how to register

As a

Visitor

I want to

Browse a list of packages resulting from a search or browse request (see other use cases)

So that

I can select one of the packages to view in more detail (-> viewing an individual package ticket:6)

Notes

• When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list
• The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable

See Update a release (ticket 14) for more information on what should be shown.

As a

Authenticated User

I want to

Create/register a package

So that

The package will be listed in the system and visible to users

Notes

• Full package registration process may be broken down into:
  1. Creating a package by provision of minimal attributes:
     1. name: unique and web-usable
     2. auto-generation of default values such as:
        1. date-registered (today's date)
        2. first owner (one owner should be created corresponding to the authenticated user performing this action)
     3. (?) auto-generate a default release
  2. updating the created package -> ../UseCase12

As a

Authenticated user

I want to

Delete a package

So that

So that the package is no longer displayed for visitors of the site

Notes

• The permissions for this action are the same as for updating the package details (see ticket:10)
• When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator

As a

Authenticated user

I want to

Update a release

So that

The release details are amended. Attributes of a release are:

• name (may include '.' and '-'
• nickname
• date
• description

Notes

• permissions: as for updating a package (ticket:10)

See ticket:16 (update a file)

As A

Administrator

I want to

Purge a deleted package

Notes

• this need only be possible via the admin interface