{22} Trac tickets (2647 matches)

Results (1801 - 1900 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1104 defect dread johnlawrenceaspden ckan-v1.4-sprint-7 closed fixed create-test-data doesn't index the packages it creates

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

1303494635000000 1303920791000000
#1112 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Allow searching for all packages
GET api/search/package?q=

returns all packages. This is so you can filter them e.g. by openness, which is not currently possible.

1304007852000000 1304358643000000
#1114 enhancement dread dread ckan-v1.4-sprint-7 closed fixed CLI for viewing search index of a package

To see what lexemes are generated for debug purposes.

1304078353000000 1304085484000000
#1116 defect dread dread ckan-v1.4-sprint-7 closed fixed api search loses boolean q options

filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters)

1304086823000000 1304358664000000
#1119 enhancement rgrp rgrp ckan-v1.4-sprint-7 closed fixed Fully functional storage extension with file upload

Previous work in #877 and #879 + #853 (storage API). In this ticket:

  • Improve authorization
  • Establish convention for laying out files on disk
  • Add documentation
  • Fix bugs with #879 (does not currently work -- boto may have changed)
1304094382000000 1305037201000000
#1126 enhancement dread dread ckan-v1.4-sprint-7 closed fixed Exceptions arising from error page

I'm not completely clear what the use case is for loading the error page in this way, but somehow original_request is None and that creates an unnecessary exception with the logic refactor.

http://ckan.net/error/document?__cache=39020485
...
Module ckan.controllers.error:29 in document
<<          original_response = request.environ.get('pylons.original_response')
               # Bypass error template for API operations.
               if original_request.path.startswith('/api'):
                   return original_response.body
               # Otherwise, decorate original response with error template.
>>  if original_request.path.startswith('/api'):
AttributeError: 'NoneType' object has no attribute 'path'
1304586683000000 1304587078000000
#1131 defect dread dread ckan-v1.4-sprint-7 closed fixed Search param validation exception not caught

Example request:

http://nl.ckan.net/api/2/search/package?q=delft&order_by=&offset=&limit=&tags=

Gives 500 error:

<type 'exceptions.ValueError'>: invalid literal for int() with base 10: ''
1304942023000000 1305537897000000
#103 enhancement kindly rgrp ckan-v1.5 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
  • package history page should provide links to these pages

Cost: 2h

1251289897000000 1311180850000000
#358 enhancement rgrp dread ckan-v1.5 closed duplicate Resources in REST API

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Authorization

  1. Have it generic (ie. not per resource) and use an action/role on system
  2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
  3. Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:

if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)
1277483282000000 1310128782000000
#559 defect pudo pudo ckan-v1.5 closed invalid Fix package lookup on group edit form

Lookup is needed to add a package but won't show up at some times.

Originally reported by Anja Jentzsch, re questions.

1283542776000000 1310127694000000
#764 enhancement thejimmyg dread ckan-v1.5 closed duplicate Read-only CKAN Web UI

Whilse using CKAN web interface, you are not tempted to edit stuff:

  • You know at all times this CKAN is read-only
  • All editing facilities are still seen but greyed-out with an indication why it is.
1288091619000000 1310133334000000
#810 enhancement pudo pudo ckan-v1.5 closed wontfix Move "add packages" field up in group form

Subticket of forms super ticket #961

1289992287000000 1310128477000000
#852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving

3 (4) use cases: (With and without access to auth keys.)

  1. upload form on ckan (by default local system has auth key for storage)
  2. datapkg upload (client) (by default without key) (plumbing done)
  3. background job archiving downloads (by queue client)

Most important are:

  1. Upload form on ckan.net (with access to keys)
  2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

  • #826 - Resource 'extra' fields
  • #358 - Resources in REST API
  • #235 - Resource format normalization and detection

User upload via Web UI

  • #877 - File upload in WUI
  • #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

  • #853 - plumbing for credential-less rest upload
  • #879 - Storage auth API

Also related:

  • #405 - Retrieval options for package resources

More Info

1291722993000000 1315821628000000
#910 enhancement pudo pudo ckan-v1.5 closed invalid Refactor CKANrdf to support extras

Extra fields should be represented in some manner.

1295266026000000 1310128544000000
#912 defect pudo pudo ckan-v1.5 closed wontfix Rethink result row representation in dcat-tools/rdfsolr
  • Should license go in the bottom line?
  • Formats should be styled as in CKAN
1295266299000000 1306774876000000
#913 enhancement pudo pudo ckan-v1.5 closed wontfix Handle licenses in dcat-tools
  • Create a list of all valid predicates where licenses are stored.
  • Create a license normalization registry in GSpread
  • Fill spreadsheet from OKD/OSI lists.
1295266407000000 1306774962000000
#941 enhancement thejimmyg rgrp ckan-v1.5 closed wontfix Submit apps or ideas for apps related to datasets (extension)

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

1296403099000000 1310134339000000
#954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3

Child tickets:

  • #1107 Move package autocomplete from package controller and move to API
  • #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

1296811899000000 1320142744000000
#958 enhancement kindly ckan-v1.5 closed duplicate ui for new kind field

Ui for related new field in resource (#957)

1296822896000000 1320664462000000
#961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

  1. Generating (or just filling) a form template with 'form data' (and errors)
  2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
  3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

  • #926 - Pick a simpler form framework
  • #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
    • #1079 Refactor API to use new logic layer and dictization
    • #1078 Refactor WUI controllers and forms to use logic layer
    • cf existing dumper and importer code
    • This will fix #662
  • [not ticketed yet] - validation layer (should work on serialized objects?)
  • #662 - Can't put entity that is returned by posting to package register (Defect)
  • #972 - Merge 'extras' into main package dict rather than having separate key
  • #1035 - Form impressions are given IDs
  • #810 - Move "add packages" field up in group form (easier to do this once forms are done)
1297069849000000 1310126100000000
#972 enhancement sebbacon rgrp ckan-v1.5 closed wontfix Merge 'extras' into main package dict rather than having separate key

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

  • Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

1297332282000000 1310134129000000
#1053 defect dread dread ckan-v1.5 closed fixed Deletion in Model API

Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted.

There is no way to delete objects at the moment - changes to 'state' are ignored in the API.

Do we need an alternative way to purge objects in the API?

1300790039000000 1310126546000000
#1070 enhancement rgrp rgrp ckan-v1.5 closed fixed Plan a new domain model and layer architecture for CKAN

See http://wiki.ckan.net/Domain_Model especially section on v2.

  • New domain model is planned but not yet finally agreed.
  • Layer architecture is complete and implemented
1301910940000000 1310117129000000
#1074 enhancement rgrp rgrp ckan-v1.5 closed fixed Refactor authz web user interface to have common code and templating

Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

  • Common template code (checkbox template)
  • Logic code (or just common code) for wiring into authz system
  • Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup.

1302271586000000 1314303581000000
#1078 enhancement kindly rgrp ckan-v1.5 closed fixed Refactors WUI controllers and forms to use logic layer
  • Deserialize forms to new dict format.
  • Replace controllers/forms to use dictization.
1302509347000000 1305828973000000
#1094 enhancement thejimmyg thejimmyg ckan-v1.5 closed duplicate [super] Refactor the Auth System

Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

  • We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.
  • Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

  1. The first thing the is_authorized method is a hook to a plugin

which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system (or partially)

  1. is_authorized method currently takes (username, action, object)

but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

  1. Rename these two classes to better reflect what they are
  1. Rename the Editor role to PriveledgeUser? since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

Read-only CKAN Web UI

(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

  • You know at all times this CKAN is read-only
  • All editing facilities are still seen but greyed-out with an indication why it is.
1303117973000000 1311173649000000
#1129 CREP kindly ckan-v1.5 closed fixed CREP0002: Moderated Edits

Proposer: David Raznick

Abstract.

We are trying to achieve these goals.

  • To get people involved with making edits to CKAN metadata.
  • To have an ownership model as to who can moderate and validate these changes
  • To not put too huge a burden on these owners.

In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it. The moderator should be able to look at a list of changes and accept the ones that

This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case.

The Problem

We need the following to be possible.

  • Storing revision of objects that are not the current active one.
  • A way of the user viewing past revisions.
  • Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package.
  • A robust way of doing this in the face of database schema changes.
  • Make sure database queries are quick.

Solutions.

  1. Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted.

Pros

  • Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it. This will just need to be persisted in some way.
  • Fast retrieval.
  • Potential to store a branching revision tree of changes.

Cons

  • No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes.
  • Will only work for the particular objects we decide to store these changes for.
  • Stores a lot of repeated information
  1. Write specialized queries for every read of the database looking only at the revision tables.

This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'.

Pros

  • No specialized storage required
  • Only need to change queries when schema changes
  • Can be made to work easily for other objects

Cons

  • Slower query time on read, as even looking at the last active package will need to do a fairly complicated query.

Implementation details.

1.

A new table with columns id, user, package_id, timestamp, revision_id, parent_id, dictized_package. revision_id should be null unless it is actually persisted to the database. parent_id is the id that this package_dict was changed from.

We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower.

Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change. This re-caching will need to be redone for every change we make to dictized representation of a package.

2.

Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. We also need to way to get what the dictized representation of the package was like at any point of its revision history. This querying is non-trivial in sql.

Participants

David Raznick to do it.

Progress.

Decided to go with option 2. However we will change the revisioning system to be like the schema attached. This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries.

The better and more normalized version of a revisioning system is outlined https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&authkey=CJfd8vsB. We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact.

1304851498000000 1325268100000000
#1140 defect dread ckan-v1.5 closed fixed Adding the package to the group is not search indexed

To reproduce:

  1. paster db init
  2. paster create-test-data
  3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
  4. curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
  5. paster search-index rebuild
  6. curl http://localhost:5000/api/search/package?groups=tilo results in 1 result
1305718290000000 1312537257000000
#1142 enhancement annapowellsmith rgrp ckan-v1.5 closed fixed [super] Major Overhaul and Extension of CKAN Documentation

Child tickets:

  • #1041 Start Using the CKAN Wiki for Tutorial-style documentation
  • #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

  • CKAN 1-page overview (for enterprise and for data hackers)
  • Administrator's Guide (including install)
  • Extensions Guide
  • Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

Minor Items

1305727452000000 1313775665000000
#1190 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] CREP 0004 Data API and Data Processing System

For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point):

http://wiki.ckan.org/CEP0004

We can distinguish 3 modules that are needed:

  1. "Webstore": A datastore with dataapi - #1208

Suggestion is this would be sqlite based with a simple sql based API. http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

  1. Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398
  1. A data processing system which utilizes this datastore. One could

get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432

More info

1308227611000000 1323270522000000
#1211 enhancement kindly kindly ckan-v1.5 closed fixed [super] Drupal integration

Sub-tasks / tickets:

  • #1335 Action API - access to any logic layer function
  • Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
  • Add package_create_validate & package_update_validate logic functions
  • Allow extensions to provide logic layer functions (IActions)
1309958442000000 1315948703000000
#1245 task rgrp ckan-v1.5 closed fixed Reorganise ckan.org pages

Here is Rufus' plan for ckan.org reorganisation from http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say "in addition to our clients..."
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog

Tasks:

  • add tagline in header - "the data portal software"
  • make and add glossy PDF
  • change "data hub" to "data portal" passim (if only using one term)
  • add [email protected]
  • image banner on home page - redo screenshots without menus
1311861708000000 1317422640000000
#1249 enhancement pudo rgrp ckan-v1.5 closed fixed Exclude script tag from extraction for i18n

Currently have this script section put in for i18n. It shouldn't be.

<script type="text/javascript">
//<![CDATA[
(function($){
    $.fn.ajaxCreateSlug = function(name, url) {
        var title = this;
        var updater = {
            init: function(title, name) {
                // Add a new element where the validity of the package name can be displayed
                this.name_field = name;
                this.title_field = title;
                this.name_field.parent().append('<div id="package_name_valid_msg"></div>');
                this.title_field.blur(this.title_change_handler())
                this.title_field.keyup(this.title_change_handler())
                this.name_field.keyup(this.name_change_handler());
                this.name_field.blur(this.name_blur_handler());
                this.url = url;
            },
....
1311958536000000 1311959356000000
#1250 enhancement pudo rgrp ckan-v1.5 closed fixed Search results should be sorted by score rather than alphabetical

At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases.

Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard).

Options:

  • Default this in solr (no need to touch code) but fragile and affects everything ...
  • Do it in code and default to score
  • Do it in code and have alphabetical (on name or title?) when no criteria otherwise score

Aside: may also wish to support search in query api but that is for later!

1311964752000000 1312370201000000
#1252 enhancement rgrp rgrp ckan-v1.5 closed fixed Deprecate ckan.net in favour of thedatahub.org

Deprecate ckan.net in favour of thedatahub.org

  • Announce
  • Create thedatahub.org and have it run same data as ckan.net
  • Announce closing down of ckan.net with 302
  • Implement 302 permanent

Also redirect wiki.ckan.net to wiki.ckan.org

1311965339000000 1318160334000000
#1282 enhancement rgrp rgrp ckan-v1.5 closed fixed JS and WUI tidy up

Consolidate js. Remove cruft.

  • Move stuff to vendor
    • Have all vendor libs local (?)
    • jquery, jquery-ui etc
  • Consolidate autocomplete to use jquery-ui
    • package autocomplete (groups)
    • tag autocomplete (package)
    • user autocomplete (authz group create)
      • turns out that authz group edit had autocomplete disabled so refactoring meant enabling ...
  • Move forms.css into main
  • Remove tag cloud from front page (can be example going forward)
  • Remove bookmarklet (since in wiki at: http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F)
  • Move name slug generation into standard js and clean up

Possible:

  1. JS unit tests for the given pages ...
  2. Option to use external site to serve js and css (e.g. CDN)

1314090389000000 1314404540000000
#1294 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] Package creation and editing UX improvements

Largely an integration of work from ckanjs

  • #1295 Simplify package create form
  • Improvements to edit form
    • Split edit form into sections
    • Remove preview from edit form (introduce inline preview on description/notes field)
  • #1296 Improved resource adding/editing on dataset/page page
  • #878 File upload integrated into resource creation
  • #1297 In-place editing of notes field for existing packages
1314405811000000 1315948257000000
#1353 defect nickstenning ckan-v1.5 closed fixed No UI to remove resources

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

1316729765000000 1317075904000000
#1354 enhancement amercader amercader ckan-v1.5 closed wontfix Pass the context object to plugins implementing the IPackageController and IGroupController

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

1316955680000000 1316970374000000
#1361 enhancement rgrp rgrp ckan-v1.5 closed fixed Simple search support

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

  • ckan.simple_search config option
  • If set:
    • query via simple query to database backend
    • do no specialized indexing

Extras

Remove

  • TagSearchQuery? from lib/search (just do a search directly)?
  • #1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

  • Re-introduce full-text search indexing where supported in e.g. postgres and use for querying
1317078276000000 1317082893000000
#1410 enhancement zephod zephod ckan-v1.5 closed duplicate Add Gravatars to user profiles

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

1319200774000000 1319213205000000
#1431 defect dread dread ckan-v1.5 closed fixed Captcha field - foreign chars cause exception

During registering a user, the user inputs foreign chars into the captcha field.

URL: http://thedatahub.org/user/register
...
Module ckan.lib.captcha:22 in check_recaptcha
<<                                     remoteip=client_ip_address,
                                          challenge=recaptcha_challenge_field,
                                          response=recaptcha_response_field))
           f = urllib2.urlopen(recaptcha_server_name, params)
           data = f.read()
>>  response=recaptcha_response_field))
Module urllib:1267 in urlencode
<<          for k, v in query:
                   k = quote_plus(str(k))
                   v = quote_plus(str(v))
                   l.append(k + '=' + v)
           else:
>>  v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128)
1320078849000000 1320084104000000
#1083 defect johnlawrenceaspden johnlawrenceaspden ckan-v1.5-sprint-1 closed fixed userobjectroles added twice can't be deleted

the add_user_to_role/remove_user_from_role functions are asymmetrical in that the add function is happy to add the same role twice but the remove asserts that it's only in the table once and crashes if that's not true.

an attempt has been made to guard against this, but fails, I think because the add functions rely on the caller committing the change to the db.

same problem affects corresponding authorization_group functions

I'll try to sort this out. Making a note here.

1302550660000000 1305537827000000
#1146 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed make logic layer control its own state

Logic layer should not use any vdm defined state and should manage it itself.

1305829117000000 1307957527000000
#1147 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Add expired_id to all revision tables.

Revision tables need expired_id to make querying history AND pending changes more efficient.

This involves making a session extension and a large table migration.

1305839833000000 1307957556000000
#1148 refactor kindly kindly ckan-v1.5-sprint-1 closed fixed test speed improvements and cleanup

The tests have been running slower recently and need fixing. They also could do with a bit more consistency to them.

1305969223000000 1305969925000000
#1149 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Change domain object modification plugin to use Session extension.

This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.

1305969863000000 1306090663000000
#1143 enhancement dread dread ckan-v1.5-sprint-2 closed fixed Improve stats page
  • Ensure we don't include deleted packages in the stats
  • Some visual improvements:
    • Number of packages:
      • fix x axis to start at first revision
      • fix y axis to start at zero
    • Fix problem with legend to 'Revisions to packages' graph.
    • Hide sidebar in the template, so it isn't right-cropped in the DGU theme
  • Add some testing of the stats lib - results of basic stats
1305734836000000 1307024681000000
#1169 enhancement dread dread ckan-v1.5-sprint-2 closed fixed Link to csv/json dumps on ckan.net 1306749768000000 1306754833000000
#1170 defect dread dread ckan-v1.5-sprint-2 closed fixed Production deployment docs out of date

Need to cover pip-requirements.txt on different branches in the doc/deployment.rst.

Also could do with tidying up upgrade.txt into this document too.

1306864171000000 1306864422000000
#1180 defect dread dread ckan-v1.5-sprint-3 closed fixed Links in markdown can be badly formed

User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error.

Need to improve filtering for anchors in markdown.

<a href="http://xxxsex.com>nasty/website

Also check this related exception:

Module ckan.controllers.user:59 in read
<<          c.is_myself = user.name == c.user
               c.api_key = user.apikey
               c.about_formatted = self._format_about(user.about)
               revisions_q = model.Session.query(model.Revision
                       ).filter_by(author=user.name)
>>  c.about_formatted = self._format_about(user.about)
Module ckan.controllers.user:167 in _format_about
<<      def _format_about(self, about):
               about_formatted = ckan.misc.MarkdownFormat().to_html(about)
               return genshi.HTML(about_formatted) 
       
           def _get_form_password(self):
>>  return genshi.HTML(about_formatted)
WebApp Error: <class 'genshi.input.ParseError'>: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3
1307469037000000 1307544223000000
#1187 enhancement dread dread ckan-v1.5-sprint-3 closed fixed Stop spam users with links in the 'about' field

When you edit a user, you should be stopped from putting links in the 'about' field, because this is a common tactic by spammers.

1308142999000000 1308143605000000
#1191 defect dread dread ckan-v1.5-sprint-3 closed fixed Unicode in user password gives 500 error on user registration

Create/register a user with a unicode character in the password. It creates the user, but the redirect to the user page doesn't work and results in 500 error.

1308312895000000 1308650930000000
#1196 defect dread dread ckan-v1.5-sprint-3 closed fixed Markdown logging ignores config settings

Markdown produces Debug logging, even when the pylons logging config is set to level of Info or Warning. This clogs up the logfile.

1308743476000000 1308743857000000
#1210 defect dread dread ckan-v1.5-sprint-4 closed fixed POST application/json error handling with newer WebOb

WebOb? from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types:

  • It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with WebOb? v1.0.7 but may go back to v1.0.4?)
  • A particular request caused an AssertionError? during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with WebOb? v1.0.7 but may go back to v1.0.2?)

Example:

Module ckan.controllers.api:206 in create
<<          log.debug('create: %s' % (context))
               try:
                   request_data = self._get_request_data()
               except ValueError, inst:
                   response.status_int = 400
>>  request_data = self._get_request_data()
Module ckan.lib.base:149 in _get_request_data
<<          cls.log.debug('Retrieving request params: %r' % request.params)
               cls.log.debug('Retrieving request POST: %r' % request.POST)
               cls.log.debug('Retrieving request POST body: %r' % request.body)
               if request.POST:
                   try:
>>  cls.log.debug('Retrieving request POST body: %r' % request.body)
Module paste.registry:137 in __getattr__
<<      
           def __getattr__(self, attr):
               return getattr(self._current_obj(), attr)
           
           def __setattr__(self, attr, value):
>>  return getattr(self._current_obj(), attr)
Module webob.request:470 in _body__get
<<          Return the content of the request body.
               """
               self.make_body_seekable() # we need this to have content_length
               r = self.body_file.read(self.content_length)
               self.body_file.seek(0)
>>  self.make_body_seekable() # we need this to have content_length
Module webob.request:697 in make_body_seekable
<<              self.body_file_raw.seek(0)
               else:
                   self.copy_body()
>>  self.copy_body()
Module webob.request:714 in copy_body
<<                  self.body = self.body_file_raw.read(length)
               elif self.is_body_readable:
                   self.body = self.body_file_raw.read()
                   self._copy_body_tempfile()
               else:
>>  self.body = self.body_file_raw.read()
Module webob.request:1190 in read
<<      def read(self, size=-1):
               body = self._get_body()
               if size < 0:
                   v = body[self.position:]
>>  body = self._get_body()
Module webob.request:1207 in _get_body
<<                  self._body = _encode_multipart(self.vars, self.content_type)
                   else:
                       assert 0, ('Bad content type: %r' % self.content_type)
               return self._body
>>  assert 0, ('Bad content type: %r' % self.content_type)
AssertionError: Bad content type: '; charset=utf-8'
1309949451000000 1309974781000000
#890 enhancement kindly pudo ckan-v1.6 closed invalid Introduce timed actions into ckanext-queue

The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows:

  • routing_key: Package
  • operation: daily
  • payload: pkg.as_dict()

Where operation is one of daily, weekly, monthly or any other interval.

1294050769000000 1318599247000000
#945 enhancement kindly kindly ckan-v1.6 closed fixed [super] Richer resources - Resource Groups, new fields, improved UI

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

  • New resource group table. #956
  • New kind field in resource. #957
  • UI for new kind field. #958
  • Resources in REST API ticket:358
  • Resources in WUI #1445
  • Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

1296475283000000 1325259350000000
#1032 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Resources in WUI

Add resources into Web User Interface.

  • Locate at: /dataset/{dataset}/resource/{id}
  • CRUD
  • Authorization

Done:

  • #945 - Richer resources - Resource Groups, new fields, improved UI
  • #1445 - Resource View page in WUI
  • #1450 - Dataset view pages to match

Moved to superticket #1506:

  • #978 - Edit Resource Extras in Web UI
1299782021000000 1330348463000000
#1065 enhancement zephod johnlawrenceaspden ckan-v1.6 closed fixed [super] Change Authorization System

Child tickets

  • #1198 Publisher hierarchy
  • #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
  • #1004 Group creation instructions missing
  • #1099 Strange interactions between two browsers while playing with authz groups
  • #1115 can have two authzgroups with the same name
  • #1133 command line rights manipulation doesn't work
  • #1138 minor navigations behave inconsistently

Old ticket description:

  1. Change name of AuthzGroup? to UserGroup? to reflect what it is for
  1. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
    • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
    • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)
  1. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

1301508331000000 1324550041000000
#1327 enhancement rgrp ckan-v1.6 closed duplicate [super] Dataset Archiving

Split out of #852. Automated archiving of datasets (related to QA).

Automated archiving using worker process

  • #890 - Timed actions in ckanext-queue
  • #891 - Resource download worker daemon
  • #892 - Make stored data available in WUI
1315821490000000 1320662446000000
#1402 enhancement kindly rgrp ckan-v1.6 closed fixed Migrate repository from mercurial to git

Plan to migrate from mercurial git

Process:

  • Do trial run
  • Announce conversion date / time
  • Require everyone to have pushed all outstanding changes at that time
  • Do conversion
  • Test
  • Announce on list and ckan.org/
1318811651000000 1324334011000000
#1435 enhancement rgrp ckan-v1.6 closed wontfix Switch to continuous.io for buildbot (?) 1320145831000000 1323283538000000
#1464 enhancement thejimmyg ckan-v1.6 closed wontfix Replace RabbitMQ with Celeryd to support running multiple instances

The current harvesting implementation can only have one instance per server.

We could either:

  • Accept this
  • Change the RabbitMQ code to support multiple instances
  • Switch the entire harvesting to use Celery
  • Wait for James's redis based live feedback based system
1321375709000000 1328529042000000
#1477 enhancement kindly kindly ckan-v1.6 closed fixed [Super] Group refactor

This is the implementation of the final draft in

http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

  • model migration. (done)
  • simplify revisioning (done)
  • Remove any sqlalchemy releations from groups. (done)
  • change logic functions (done)

Single type per group (#1531)

  • migration add type to group table. (done) 1d

Add capacity table (#1522)

  • model migration. (done)
  • add to logic functions.

Allow other objects to be added to groups. (#1531)

  • change logic functions (done)

Adapt thedatahub.org to use new model.

  • move auth groups over to groups/members
  • change user_object roles to use groups? keep user object roles for time being.
  • make create update package/group put the correct items in the group/members/capacity tables.
  • work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

  • adapt auth functions to use auth.
  • adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

  • #1279 Deprecate Authorization Groups
  • #1198 Publisher hierarchy
1321959320000000 1328016209000000
#1508 enhancement ross rgrp ckan-v1.6 closed fixed [super] Finalize and deploy some of our major (new) features as of Dec 2011

This ticket is about finalizing, then deploying (to http://thedatahub.org/), and finally announcing some of recent major new features such as:

  • Deploy wordpresser extension for thedatahub (so we can start blogging) - #1533 - WONTFIX
  • (Deploy new Resource and Dataset pages (invite comments))
  • Get thedatahub geo-ified i.e. ckanext-spatial - #1601 (?)
    • Any additional dev that needs to be done? (e.g. geo in search ...?)
  • (Deploy / re-enable ckanext-googleanalytics on thedatahub - #1451 (more stuff #1566))
    • Rename the extension to ckanext-analytics?
  • Custom forms (and use of 'dataset types') - Ticket (??)
  • (Get QA deployed and running - #1582)

It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc.

1323088817000000 1328000871000000
#1514 defect dread ckan-v1.6 closed duplicate Modifying user name loses connection with revisions

If you edit your user name, the number of revisions you made returns to 0. This is because in the Revision object, the user's name is stored, rather than the user's ID.

rgrp: We can reconnect that pretty easily (and have a longer term solution that involves not using the usernames but the userids in in the Revision objects so we don't hvae this problem in future!)

1323100659000000 1323279018000000
#1602 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Data Previewer / Viewer v2

We already have first pass of Data Preview that was done as part of #1357. This is about v2 of this. Two main components:

  • A new version of the separate Data Explorer library (now called Recline Data Explorer) - #1446 Est: 10-15d
    • Why:
      • Cleaner, backbone based solution.
      • Provides editing support if wanted.
      • Collaboration with Max Ogden
  • Integrate this into CKAN (into resource view page) - see #1630 Est: 1d
  • Improvements to Data Viewer / Previewer system in CKAN itself - Est: 2-3d
    • (Update Data Preview-er (View-er) to support images ...)
    • (WONTFIX here - (?) Update Data Preview-er to support geodata - #1151)

1325267544000000 1328175719000000
#1615 enhancement thejimmyg ckan-v1.6 closed worksforme CKAN Should work behind a proxy server

This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*.

We would like this work to facilitate testing within VMs as part of our new build infrastructure.

It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment.

1325687841000000 1328888870000000
#1637 enhancement seanh seanh ckan-v1.6 closed fixed API call for getting the list of activity detail items for a given activty stream item

(and add test cases for it)

1326304817000000 1326737169000000
#1677 enhancement amercader ckan-v1.6 closed duplicate Make synchronous search the default behaviour

Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour.

1326807604000000 1326807655000000
#1682 enhancement dread dread ckan-v1.6 closed duplicate [super] Translatability for EC

We need to be able to translate:

  • EC extension (templates, form)
    • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
  • Field values in taxonomy (e.g. country names - Eurovoc)
    • #1665 Research into Eurovoc
    • Display Taxonomy in different languages
  • Field values not in taxonomy (e.g. title & description)
    • Use extra fields e.g. _i18n_title_fr = Le data.
    • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)

Timescale to complete this - Mar/Apr?.

Related work to interface with:

  • Eurovoc - Sean
  • i18n in search index - Adria
1326911899000000 1330038786000000
#1743 defect seanh seanh ckan-v1.6 closed wontfix Activity streams still link to deleted things

For example, after deleting a dataset old activity stream events will still hyperlink to that dataset.

1328212016000000 1328527086000000
#1397 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Resource archiving

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

  • Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports

Work Items

  • Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
  • Generate archiving request on resource url change [0.25d][0.25d] - #1399
  • Make site user account.
  • Make entry point system for celery config
  • Archiver daemon #891
    1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
    2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
  • Write to resource and task status table.[1d][0.75d]
  • [Required?] Make archived data available in WUI - #892
  • Documentation - #1400
1318529593000000 1338202920000000
#1404 enhancement zephod zephod ckan-v1.7 closed wontfix Search Page UI improvements

[Refactored] :: Follows on from #1506 UX changes.

  • Declutter the sidebar. No yellow box.
  • Facets to go on the left, rather than the right. More logical flow.
  • Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
  • Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
    • Update that look-and-feel to include the new resource icons created in #1506
1318847818000000 1338203639000000
#1465 enhancement thejimmyg ckan-v1.7 closed fixed Upgrade the harvester to support publishers properly

At the moment most of the harvesting functionality is only available to sysadmins. This is fine for most cases but going forward we will need to grant each publisher access to their own sources, harvest objects and harvest object errors. This will probably be done based on the groups refactor.

One more pressing concern is giving DGU publishers the ability to make calls to /harvest/object/id. We want them to be able to see documents that faild validation and also to see *all* errors in each document.

1321375923000000 1338203858000000
#1496 enhancement seanh seanh ckan-v1.7 closed fixed Render a user's activity stream on her user page 1322495588000000 1334582338000000
#1510 enhancement seanh seanh ckan-v1.7 closed wontfix Migrate existing data into activity streams database tables

See '### Task 2' in this etherpad: http://ckan.okfnpad.org/notifications

I have no idea yet about how to do this or how long it might take me.

1323093901000000 1339677620000000
#1515 enhancement seanh seanh ckan-v1.7 closed fixed [super] Activity Streams

We want to add "activity streams" (chronological lists of activities) to CKAN.

Activity streams standard: http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: http://datahub.pudo.org/pudo

Tasks

Strikethrough means the task is done.

  • Migrate existing data into activity streams model [3.5d] #1510
  • Generate activities when objects are saved [4d] #1298
  • Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511
  • API call for getting a user's public activity stream as JSON [0.5d] #1495
  • API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494
  • Render a user's activity stream on her user page [3d] #1496 (needs review/page design work)
  • Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.
  • Logic functions for getting activity streams for packages, groups, and users.
  • API calls for getting activity streams of packages, groups, and users as JSON.
  • Rendering activity streams for packages and groups etc. into their pages. (done but disabled pending page design work)
  • API calls for inserting events into activity streams. (Still need to handle activity details in the API, ticketed).
  • Conform to activitystrea.ms standard
  • Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.
  • Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams
  • Email notifications
  • Extension point for extensions to register themselves as renderers for different types of activity
  • Deploy to UAT [0.5d]
  • Docs and announce [1d]
1323105048000000 1338204149000000
#1525 enhancement icmurray icmurray ckan-v1.7 closed fixed [super] DGU package form refactor

Form refactor. New create/edit package form wizard for DGU.

Etherpad: http://ckan.okfnpad.org/dgu-package-form

Tasks:

  • #1526 : Simple one-page form.
  • #1527 : Tab-ify the form using JS. No validation between steps.
  • #1541 : Setup server for the DGU form-refactor.
  • #1537 : Package create form wizard
  • #1538 : Store partially-filled in package form data as a draft.
  • #1539 : Package forms auto-save

How the rest of the work is carried out is up for discussion, as it depends on the approach taken. Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages. Google doc outlining design(s)

1323172839000000 1337159836000000
#1530 defect zephod lucychambers ckan-v1.7 closed fixed URL field help text in wrong place

Confusion between entering values in the title and name/url fields in datacatalogs.org.

The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser.

1323261984000000 1338202731000000
#1537 enhancement icmurray icmurray ckan-v1.7 closed fixed Package create form wizard

Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

  • separate pages for each section of the form
  • validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
  • no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

  • with the javascript approach it would be harder to test the workflow.
  • with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure.
  • the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
  • the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
  • the multi-page wizard wouldn't preclude a javascript-tabbing edit-form.
1323358451000000 1337159772000000
#1538 enhancement icmurray icmurray ckan-v1.7 closed wontfix Store partially-filled in package form data as a draft.

Allow a user to save a partially-complete and probably invalid package-create form in order to come back to it later.

We decided on blob storage over a state field on the package:

  • prevents a history of the drafts being stored in the revision history. This is a particular problem if we start auto-saving, say every 20s.
  • the blob can store arbitrary extra data, such as how far through the wizard we are.
1323359221000000 1337159799000000
#1539 enhancement icmurray icmurray ckan-v1.7 closed wontfix Package forms auto-save

Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms.

An AJAX call is made every N seconds, and the auto-saved blob is updated.

note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave.

1323359385000000 1337159806000000
#1566 enhancement David Raznik jilly mathews ckan-v1.7 closed fixed [super] Finalize Google analytics extension
  • (Deploy it)
  • Integrate with stats plugin (now that is in core) - #1101
  • Blog about it #1581
  • (Other tickets re upgrading for v1.5.1 - done but where are tickets?)
1324293168000000 1338203338000000
#1574 enhancement ross ross ckan-v1.7 closed fixed [super] Storage changes including automated save to Webstore

Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore.

  1. A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see #1608) but still using ofs [5d]
  1. The user is provided the link to the file as currently.
  1. The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see #1609) [5d] - took 6
  • It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file.
  1. The archiver will continue to move the file into permanent storage as per current practice.

Tickets

  • #1608 Merge storage into Core [5d]
  • #1609 Celery task for ckanext-archiver to write to webstore. [5d] (took 6)
  • #1687: Deploy to datahub [1d]
  • #1681: Core storage documentation [1d?]
    • Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc
    • External: here's how to configure it and here's what you get ... (goes in main sphinx docs)
1324315633000000 1338202955000000
#1586 enhancement toby dread ckan-v1.7 closed duplicate 'Sort by' field for package search

We default to sort packages search results by "search rank", but we can change it to be "name", "title", "metadata_modified" date, "metadata_created" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this.

Follows on from #191.

1324479588000000 1337079419000000
#1594 enhancement johnglover johnglover ckan-v1.7 closed wontfix [super] QA Improvements

Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement.

  • Integrate QA into frontend - 1d - #1583
  • QA report improvements - 2.5d - #1584
  • QA - Check license as part of 5 star rating - #1587
  • QA - Give SPARQL endpoints a 4 star rating - #1588
  • QA - Give 5 star rating to datasets with link metadata - #1589

Main pad: http://ckan.okfnpad.org/qa

Some QA discussion (relating to Celery in particular) at http://ckan.okfnpad.org/queue

1324489620000000 1338203215000000
#1595 defect kindly rgrp ckan-v1.7 closed wontfix Editing of a Group leads to recording a change against all associated datasets

Or at least it appears that way from revision log which displays all Group datasets against the changelog, see http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217 -- not apparent from that page but see my user page: http://thedatahub.org/user/rufuspollock

This may turn out to be a UX bug rather than a real bug ...

Assigning to kindly for review and comment.

1324504875000000 1338203959000000
#1600 enhancement rgrp rgrp ckan-v1.7 closed wontfix [super] User guide with integrated help in the WUI

Provide help to users in the WUI, with this linking to a detailed user guide.

  • User guide (suggest at help.thedatahub.org (or possibly help.ckan.org)
  • Snippets in the WUI with links to full user guide.
    • Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis
  • General info about publishing/working with/etc data (where we don't duplicate elsewhere)

Est: (?) 5-10d

Topics

See http://ckan.okfnpad.org/datahub-help

  • Step-by-step for adding a dataset including tips
  • API (perhaps link to main docs)
  • Command line interface and dpm
  • Using groups

Why do this

We already have wiki.ckan.org and docs.ckan.org. Why do this?

Answer for docs:

  • This is administrative guide and focused on general CKAN software.
  • Hard to provide completely generic help.
  • Integratable with the DataHub? site

Answer for wiki:

  • No spam
  • General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful.
  • Integratable with the DataHub?
1325257824000000 1333362051000000
#1601 enhancement amercader rgrp ckan-v1.7 closed fixed [super] Well integrated Geo-spatial support including search

Need thoughts from Adria and James. RP thoughts

  • Integration into WUI search: #1469
  • JS Editor for adding/editing spatial attribute: #1818
  • Theme/template hook for adding the map e.g. on dataset page (or elsewhere)
  • Clear Documentation
  • Deployment
  • Announce post
  • (?) Map page for stats section showing location of datasets

Extras (probably separate):

  • csw support ...
1325259821000000 1338204029000000
#1603 enhancement zephod rgrp ckan-v1.7 closed duplicate Search query builder

Super ticket: #1745

Ability to build up search query using a nice javascript-y interface.

  • Add facets by selecting attribute and adding -> search facet options in dropdown -> added to search (with 'x' to remove -- as we currently do).
    • (a bit like the data.hri.fi)
  • Some improvements to css
  • Improvements to faceting
    • Ability to configure faceting and number of items to show (?)
  • Pure JS search implementation to make it easy to reuse across site
1325268364000000 1338202654000000
#1605 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Multilingual support in CKAN

Multi-language:

  • dataset and resource metadata (and other objects such as groups?)
  • Field values in taxonomy (e.g. country names - Eurovoc)
    • #1665 Research into Eurovoc
    • Display Taxonomy in different languages
  • Field values not in taxonomy (e.g. title & description)
    • Use extra fields e.g. _i18n_title_fr = Le data.
    • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)
  • EC extension (templates, form)
    • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
1325473367000000 1338193641000000
#1607 enhancement icmurray dread ckan-v1.7 closed fixed [super] Data.gov.uk Maintenance Refactor and UKLP Development Support

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

  • 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1
  • 6. Publisher Registration Improvements MUST PHASE 1
  • 3. Themes/Taxonomy? as well as tags WONT PHASE 2
  • 32. Browse by Tags PHASE 1 MUST
  • 12. Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1
  • 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]
  • 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]
  • 22. Mixed Licenses MUST PHASE 2

Plus, brought forward from last sprint:

  • Publisher hierarchy support
  • Improved dataset creation tools (form) - #1525
  • Edit/delete for harvested datasets

Overview of tasks: see the design doc.

1325503348000000 1337159969000000
#1617 enhancement icmurray icmurray ckan-v1.7 closed fixed Update and test existing DGU package form : publisher / foi information

[Child of #1526]

Contact/FOI information is derived from Publisher/User?, but can be altered for each dataset.

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

1326103384000000 1332151332000000
#1618 enhancement icmurray icmurray ckan-v1.7 closed fixed Update and test existing DGU package form : provide stubs for the theme and sub-themes

[Child of #1526]

Provide the interface for the themes and subthemes hierarchy.

1326103557000000 1332151341000000
#1619 enhancement icmurray icmurray ckan-v1.7 closed fixed Update and test existing DGU package form : provide stubs for the publisher hierarchy

[Child of #1526]

Create the interface used to retrieve publisher and authz information from the publisher hierarchy.

1326103672000000 1332151351000000
#1632 enhancement seanh seanh ckan-v1.7 closed wontfix Render a group's activity stream on its' page

Add logic functions for getting the activity stream for a group in JSON and rendered HTML formats, add HTML activity stream into group page.

1326304199000000 1335877871000000
#1633 enhancement seanh seanh ckan-v1.7 closed wontfix Render a dataset's activity stream on its' page

Add logic functions for getting the activity stream for a dataset in JSON and rendered HTML formats, add HTML activity stream into dataset page.

1326304298000000 1335877927000000
#1634 enhancement seanh seanh ckan-v1.7 closed duplicate Allow users to follow/unfollow activity streams of other users, datasets and groups

This is a fairly big feature to add. Analysis: http://ckan.okfnpad.org/27

This ticket is related to #1635 (email notifications for activity streams) which is about managing a list of activity streams for which the user receives email notifications. There are decisions to be made about how the two will work together.

1326304542000000 1334587390000000
#1636 enhancement seanh seanh ckan-v1.7 closed wontfix Purge items from activity streams when object (package etc.) purged from CKAN 1326304754000000 1338204258000000
Note: See TracReports for help on using and creating reports.