{22} Trac tickets (2647 matches)

Three JS bugs have appeared:

• The slug editor is not updating the text field again. [Fixed; commit:fadda17383]

• The View Dataset page will stop executing javascript if there are no resources. [Fixed; commit:e54b089f6e]

• The Add-Dataset and Add-Group URL editor shows the wrong URL for datasets and groups. Broken in this commit:

​https://github.com/okfn/ckan/commit/5a01e67 by 'map.explicit=true'. JohnG: Can you look into this? [Fixed; commit:88e88380f6]

Most of the different pagination links are broken and point to wrong locations. e.g:

• Dataset listings in group page:

​http://test.ckan.net/group/bibliographic

• Tag letters in tag listing:

​http://test.ckan.net/tag

• User listing:

​http://test.ckan.net/user

• Revision listing:

​http://test.ckan.net/revision

This is most probably related with:

​https://github.com/okfn/ckan/commit/1772a5c

Makes more sense to only have one comprehensive stats/analytics extension, so when people are looking to add a stats extension they won't have to add two which may be confusing (is one an old version of the other? why do I have to add two? what's the difference? etc)

User-wise we need a way to display our stats with google analytics in the same place

metadata_created seems to be returning the date of the earliest revision (for all packages), rather than the earliest revision for a specific package.

• deploy celery
• deploy QA and archiver tasks
• write up a blog post announcing QA on thedatahub

Super: #1594

From Richard Cyganiak on the CKAN discuss list:

According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license.

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field.

• Update CKAN on DGU UAT to 1.5.1
• Deploy Celery
• Deploy QA extension

To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function.

This behaviour is now deprecated however.

Document deployment on tenforce test server.

Eurostat datasets mainly.

Part of #1698

Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation.

Initially we'll just add all tags from all vocabularies into the existing tags search facet (this should happen automatically). Later we want to enable templates to use separate search facets for different vocabularies.

Also need to make sure that search terms are matched against the texts of the tag names themselves (should happen already) and maybe also against the texts of the vocabulary names?

Need to test:

• API call to create new vocab
• API call to add tags with a given vocab
• API call to update tags
• Dummy plugin to test that updating tags works via WUI

• Tag list not be added to CKAN core, what is the best way to do this in an extension?
• Does the list need to be updated via API/WUI?

• see how this currently works with DGU first

Should describe implementing IDatasetForm, subclassing controllers is deprecated

• code review with D.R. for ckanext-ecportal
• merge and test all multilingual code
• fix server issues: routing, Git access to CKAN repository

Bugs to fix:

• rename groups to publishers
• update home page layout (recent updates along side publishers)
• don't add 'en' to main menu links if english
• resource edit page, fix missing images
• change dataset 'settings' to 'edit'
• translate 'published by' drop-down list
• fix JS on 'add a dataset page'
• prepopulate 'published_by' when clicking 'add a dataset' from group
• chosen broken (dataset edit) if in different language

Groups should be able to specify a URL to an image file for a group logo, the image will be shown on the group page.

Actually uploading an image file (rather than giving a URL) and cropping and resizing the file will be done later, for now just give a URL. (later ticket)

Further customisation of group pages (CSS, templates) also for later. (not sure if we really want this, but in a later ticket if so)

I won't be needed for this during the current sprint, closing.

Changes:

• tidy up activity feed on home page or replace with search results (by modified date)
• move ckan menu to a left sidebar (see Sam's template) (not doing this now)
• update templates with changes from master
• use local image in language dropdown (ecportal.js)
• update dataset read template

To discuss:

• should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping.

Quite a few of the current test dataset resources have their format incorrectly specified as text/html. Attempting to preview these results in the browser trying to automatically download the resource, so for now we will not preview this format.

When clicking 'add dataset' on a publisher/group page, the new dataset form should select the appropriate group.

• created repo for final list of instructions: ​http://github.com/okfn/ecportal-server-setup

Requirements

Would be useful if we could extend the Google Analytics extension to push track events to GA so that we can determine how the system is being used, and where we can optimise the workflow.

Assuming it isn't configured to be off, track events should be added to some key action buttons so that the information on what users are doing is logged for analysis.

Interface

This could possibly just be a list of JS calls to $('x').click(_gaq.push('track'...))

User Stories

• As a developer I would like to know which components of the system are used the most so I can streamline the processes.

• As a support developer I would like to know which parts add complexity to the system for minimal gain and could possibly be removed, de-prioritised.

For which metrics, as many of the following as possible that are achievable without changing the templates. I think for now it is enough to just record counts of these pages/actions.

• Any group link on homepage

• Clicking on username (to go to profile)

• In user profile, clicking on Edit Profile

• Clicking Save Changes on Edit Profile page

• Clicking on any dataset link on User Profile page

• Any of the group links on /group

• Clicking any of the right hand sidebar tags on /group/X

• Visiting /group/history/X

• Compare Button on /group/history/X

• Compare Button on /dataset/history/X

• Tags on right hand sidebar of /dataset/X

• Download button on any /dataset/X/resource/* page

• Data API button on any /dataset/X/resource/* page

Tasks

[ ] Analysis/Clarification?

[ ] Code/Config? changes

[ ] Documentation

Estimates

EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested.

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

• datastore_create
• datastore_delete
• datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}

• The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
• rows: can be empty so that you can just set the fields
• fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

• Any error results in total failure!! For now pass back the actual error.
• Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: ​https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

​http://www.postgresql.org/docs/9.1/static/datatype.html

​http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

Fixes and updates to Recline / Datapreview for CKAN 2.0.

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

• Purge revisions (or sets of revisions) and purge objects #1076
• Set roles for users #1075
• Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
  • WONTFIX
• Overview of activity
  • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

​https://bitbucket.org/okfn/ckanext-admin

Tickets

• ticket:1031 - user autocomplete api in ckan

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system

Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users).

• API location: /api/util/user/lookup?q=querystr&limit=10
• Return json objects containing {id: ..., name: ..., fullname: ...}
• Put in a module called controllers/apiv2/user.py

Roles on System object are important because admin role on system equates to being a 'sysadmin' (i.e. able to do anything).

• Make users sysadmin (either as separate action or as part of editing roles on system object)
• /authz subpage for editing roles on system object
  • Add and update user roles
  • Add and update authz group roles
  • List actions associated to roles at top of page (extra points for checkbox table with editability)
• Document on ​http://wiki.ckan.net/Authorization what roles on System object 'mean' esp sysadmin role on System

Related Tickets

• super ticket: #833
• authz lib improvement and authztool refactor #1050

I've found that if I make an authorization group I sometimes can't remove myself from it. I've no idea why. I can add and remove other users. I'll investigate, just making a note of it here.

This is a low priority for the demo site.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments

Implementation:

​http://s031.okserver.org:2375/dataset/history/adur_district_spending

This includes the tag page as well for now.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments

Implementation:

​http://s031.okserver.org:2375/en/tag

Tags should be implemented in two sections.

• Custom predefined tags using vocabularies
• Provide suggestions, auto completion then fallback to free text

Currently stages of the same colour blur together, these could do with a nice divider.

Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now

Need to add JS (presumably?) to the related app box on dataset page: ​http://s031.okserver.org:2375/dataset/gold-prices to allow user to scroll through when multiple related items exist

when editing a dataset (e.g. here: ​http://s031.okserver.org:2375/en/dataset/edit/afghanistan-election-data) add sidebar list of resources for user to click & browse through taking you to the relevant resource edit pages

it would be nice if this didn't fall over when the dir is removed maybe a linux only issue?

​http://localhost:5000/ckan-admin

you need to be an admin

paster sysadmin add <username> -c dev.ini

we need to get links to other admin pages etc - I think this will want to be flexible so extensions can add extra links etc - but that can come later

Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource?

Ira, what are your thoughts?