{22} Trac tickets (2647 matches)

It is too easy to make a mistake with the migrations at the moment. A more systematic way of doing them is needed and this needs to be documented.

This will involve running a sample of real requests synchronously against real data.

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

add revisioning to harvested document table

see summary.

We need a way to pass in schemas to the logic layer to deal with edge cases.

We need to see what areas of ckan are slow.

Proposer: David Raznick

Abstract.

We are trying to achieve these goals.

• To get people involved with making edits to CKAN metadata.
• To have an ownership model as to who can moderate and validate these changes
• To not put too huge a burden on these owners.

In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it. The moderator should be able to look at a list of changes and accept the ones that

This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case.

The Problem

We need the following to be possible.

• Storing revision of objects that are not the current active one.
• A way of the user viewing past revisions.
• Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package.
• A robust way of doing this in the face of database schema changes.
• Make sure database queries are quick.

Solutions.

1. Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted.

Pros

• Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it. This will just need to be persisted in some way.
• Fast retrieval.
• Potential to store a branching revision tree of changes.

Cons

• No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes.
• Will only work for the particular objects we decide to store these changes for.
• Stores a lot of repeated information

2. Write specialized queries for every read of the database looking only at the revision tables.

This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'.

Pros

• No specialized storage required
• Only need to change queries when schema changes
• Can be made to work easily for other objects

Cons

• Slower query time on read, as even looking at the last active package will need to do a fairly complicated query.

Implementation details.

1.

A new table with columns id, user, package_id, timestamp, revision_id, parent_id, dictized_package. revision_id should be null unless it is actually persisted to the database. parent_id is the id that this package_dict was changed from.

We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower.

Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change. This re-caching will need to be redone for every change we make to dictized representation of a package.

2.

Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. We also need to way to get what the dictized representation of the package was like at any point of its revision history. This querying is non-trivial in sql.

Participants

David Raznick to do it.

Progress.

Decided to go with option 2. However we will change the revisioning system to be like the schema attached. This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries.

The better and more normalized version of a revisioning system is outlined ​https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&authkey=CJfd8vsB. We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact.

Revision tables need expired_id to make querying history AND pending changes more efficient.

This involves making a session extension and a large table migration.

This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.

Simplify the logic in the atom feed an make all feeds use logic layer to return lists.

Sub-tasks / tickets:

• #1335 Action API - access to any logic layer function
• Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
• Add package_create_validate & package_update_validate logic functions
• Allow extensions to provide logic layer functions (IActions)

Make a robust way to make sure the drupal database is consistent with the ckan data.

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

Add tables described in ​http://ckan.okfnpad.org/notifications

Migrate tables for activity streams

Spam users where added to thedatahub and we need to clean them.

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id

Regression found by updating datacatalogs.

Give site user system administrative rights.

navl tests are being skipped unskip them!

This is the implementation of the final draft in

​http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

• model migration. (done)
• simplify revisioning (done)
• Remove any sqlalchemy releations from groups. (done)
• change logic functions (done)

Single type per group (#1531)

• migration add type to group table. (done) 1d

Add capacity table (#1522)

• model migration. (done)
• add to logic functions.

Allow other objects to be added to groups. (#1531)

• change logic functions (done)

Adapt thedatahub.org to use new model.

• move auth groups over to groups/members
• change user_object roles to use groups? keep user object roles for time being.
• make create update package/group put the correct items in the group/members/capacity tables.
• work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

• adapt auth functions to use auth.
• adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

• #1279 Deprecate Authorization Groups
• #1198 Publisher hierarchy

We want to be able to change form depending on package type or group type.

This is dependent on a type field being added to the Package and the Group.

Need to add capacities to member tables.

Group show that lists packages is slow due to not using query in pagination.

Need to setup server in tenforce test server.

Make sure accented characters are normalized when indexed and when searched for.

The prototype portal gave us a list of classifiers that are expected for the initial launch. Add then to ckanext-ecportal.

Need to make solr schema work for many languages. Get stopwords and choose correct analysis for each.

It is useful for extensions to add or change items that go into the search index. Add an extension point for this.

Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown.

Tasks include:

• Fix up data_dict and templates so that the pkg objects do not get used in the templates. This is to assure that all data passed is modifiable. (2d) (complete)
• Add extension point to Ipackage controller (1d)
• Test (1d)

A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination.

Add api to translation to the term_translation table.

• add to logic 2d
• test 1d

Search index code needs to use package_dictize so that it can get the new vocablurary information and is more consistant with the rest of the system. This is in preperation for their translation being put in the search indexer.

This extension point needs to combine.

• Making sure the correct data get put in search index.
• Make the search queries are modified to weight multilingual search correctly.
• View pages, including search results are translated.
• Bring together all search schema modifications.

Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.

There is a disrepency between what we put in and get out when using a custom form/api. Narrow this gap by making sure package_show shows the correct serilization.

Need google spreadsheet app to work with the new datastore. This should simplify it a lot.

Page views and Resources clicks need to be tracked.

User Stories

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

• I want to see the traffic breakdown by country to my site ...
• Ditto for browser type, language, etc etc
• I want to see it graphed over time ...

Implementation Details

1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display

• Config option ckan.status.enabled = False (by default)

Storing Data

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

Displaying Data

• Helper functions / dictize:
  • Helper function: h.stats_get(object_type, id, stats_type)
    • h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
  • Change to dictize
• Location in the default theme (do we show for example in search results too!)
• Support for ranking by most popular in search?

Tracking Data

• Our own solution (just write to site_tracking)
• Google analytics (plus extension for retrieving data) <- would need a refactor
• Piwiki

Own Solution

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

• Make javascript to make request to ckan to store clicks and page views.
• Add middleware so these requests do not go through pylons and just store data quickly.

add resource_status_show action function to see the state of celery tasks.

Need to get information from both out task status table and the celery_taskmeta table.

Improve documentation for plugins and add examples. Could use ckanext-example as a base.

Search results are slow as we do many package_dictize. Store the package_dict in the search index so that we do not have to hit the database for each of them.

Datastore_url enabled checks are useless.

We should probably change this to datastore_url = 'active' and only update the flag to when a attempt at a post is made.

This will need to be removed from the form at some point.

If a user selects a language there are no mo files for then an error is raised. Revert to default language instead.

Try and put datapreview in Iframe.

Write CKAN FAQ (Basis can be: ​http://wiki.ckan.net/FAQ)

Post preliminary questions on:

​http://notebook.okfn.org/

Take screenshots of existing ckan instances esp those mentioned <​http://wiki.ckan.net/Theming> and put on flickr in ckan or ckan-theme group so we can create a gallery ... (both to illustrate theming but also to show ckan instances that are around -- could add to ​http://wiki.ckan.net/Instances)

Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.

• Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: ​http://thedatahub.org/dataset/edit/iati-registry
• Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in.