{22} Trac tickets (2647 matches)

Results (1701 - 1800 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#315 defect dread rgrp v1.1 closed fixed Improvements and fixes to csv dump
  1. Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html
  1. Issues with package resource serialization into csv table.

<quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>

1274100887000000 1275846764000000
#317 enhancement pudo rgrp v1.1 closed fixed Make search pluggable

Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR).

Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable.

1274295418000000 1279286041000000
#320 enhancement dread rgrp v1.1 closed fixed site_title configuration variable which is used in template

As a sysadmin I want to configure basic site title information for use in the site templates.

Implementation:

  • ckan.site_title config variable
  • set this on g in app_globals.py e.g.
    • from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network')
  • use in head title and in main site title/logo section (use it as alt on logo image)
  • Also all other pages (e.g. index, about) which talk about CKAN
    • Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates?

Questions:

  • Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined?
  • Probably yes, but not part of this ticket.
1274378967000000 1279130535000000
#328 enhancement dread rgrp v1.1 closed fixed Mention code libraries in API documentation

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

  • Python: ckanclient
  • Perl: luke closs wrote something
  • PHP: drupal library?
  • ...?
1274988106000000 1275318745000000
#331 enhancement rgrp ckan-backlog new Timezone of CKAN timestamps should be configurable

Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either.

Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone.

1275302440000000 1339774701000000
#332 defect pudo rgrp closed fixed Autocomplete tag on click

During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box.

1275302887000000 1280743320000000
#337 enhancement johnbywater rgrp v1.1 closed fixed Download links for resources should open in new window

target = _blank

Cost: 30m

1275422151000000 1279300972000000
#355 defect rgrp rgrp closed fixed Dashes versus underscores in package names

Sort out how we deal with dashes versus underscores in package names.

1277221996000000 1311177552000000
#356 enhancement rgrp v1.1 closed fixed Search box in at top of page (UI)

A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

  • It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration).
1277235411000000 1278931830000000
#359 requirement pudo rgrp datapkg-0.7 closed fixed Create Storage with pluggable extensions

Storage represents something you can upload to (download is done in ticket:406).

  • storage sub package with defined interface (maybe just use OFS)
  • Only support uploading single files for the moment (though can do multiple files).
    • Relationship to build (ticket:306)
    • No support for syncing e.g. hg/git repos -- that's up to you as a user
  • Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication.

1277750654000000 1291135692000000
#360 defect rgrp rgrp datapkg-0.7 closed fixed New sqlite-based DB index

In 0.5 and 0.6 have been more oriented to getting file indexes working to support the convenience of using packages on disk directly. However we now need to go back to having a better central/default index and the best model for this is a simple db index using sqlite (which is now supported in the standard library).

NB: this is a retrospective ticket (been working on this for a couple of months)

1277803174000000 1288004891000000
#361 enhancement rgrp rgrp datapkg-0.7 closed fixed Allow specification of the default index in configuration 1277803305000000 1291135756000000
#405 task rgrp rgrp datapkg-0.8 closed fixed Retrieval options for package resources

Download Command (was install command) should support the following modes:

  • Download only the first listed resource (current behaviour, slightly arbitrary)
  • Download resources by interactive selection
  • Download resources by MIME type (cf #235)
  • Download all resources.
1281346806000000 1297214833000000
#406 requirement pudo rgrp datapkg-0.8 closed fixed Download sub-system

Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download.

And we should support interactive retrieving of resources ...

Implementation Ideas

for resources in pkg.resources:
    # datapkg/hg 
    load_storage_handler(resource.format)
1281346822000000 1296467427000000
#409 task rgrp rgrp datapkg-0.8 closed fixed Refactor cli to be more 'pluggable'
  • (Load cli commands from setuptools entry point name datapkg.cli)
  • Document how other can write new commands
  • Refactor base Command class to support this pluggability (if necessary)
1281348659000000 1297074197000000
#502 requirement dread rgrp ckan-v1.3 closed wontfix Data4nr uploading script shall remove deleted packages

Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed.

1282659304000000 1292586466000000
#503 task dread rgrp closed fixed Provide exemplar package relationships for some packages
  1. Find some relationships.
  2. Add them via API
  3. Notify DGU of completion and point to exemplars
1282659631000000 1314031851000000
#504 requirement dread rgrp ckan-v1.3 closed invalid The DGU fieldset shall conform to v3 of DGU metadata specification

No need for requirement - closing in favour of #505

1282660246000000 1292586686000000
#505 enhancement dread rgrp closed fixed DGU fieldset supports v3 of DGU metadata specification 1282660416000000 1298368280000000
#506 task dread rgrp closed fixed Plan transition to DGU metadata-fields v3 1282660490000000 1287080989000000
#527 story pudo rgrp closed fixed Create domain model and load sample data 1282899025000000 1293097531000000
#528 task pudo rgrp closed fixed Setup demonstration instance at iati.ckan.net 1282899110000000 1283536475000000
#529 task pudo rgrp closed fixed Obtain sample metadata 1282899161000000 1283536554000000
#531 task pudo rgrp closed fixed Load sample IATI metadata
  • Parse data
  • Load via API
  • Develop test for load via API (can be local or remote -- with paste webfixture may be easier)
1282899244000000 1283536676000000
#539 requirement wwaites rgrp ckan-v1.4 closed invalid The system shall improve performance 1283242447000000 1303118486000000
#540 story wwaites rgrp closed fixed Implement caching in a systematic manner 1283242705000000 1302694845000000
#541 enhancement wwaites rgrp closed fixed Full-page caching even for logged in users

The simplest way to improve performance on specific pages is to implement full page caching.

This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have "You are logged in as: {username}". Similarly we change the visibility of some material depending on the status of users:

  • Visibility of Edit or Authz link on packages and groups
1283243139000000 1294924872000000
#542 task wwaites rgrp closed fixed 'Show and Tell' page customization using cookie + js

To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/

In the first place we just need to apply this to:

  • You are logged in as
  • Package/group edit/authz tabs
1283244432000000 1283278942000000
#543 task wwaites rgrp closed duplicate Investigate partial page caching and edge-side includes

Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page.

Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish.

Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat.

1283244784000000 1311178918000000
#560 defect kindly rgrp ckan-v1.4 closed fixed Deleted packages showing up in tag lists

E.g. http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins).

Also suggested that these are showing up in the API (courtesy of Anja)

Cost: 2h

1283700001000000 1297084192000000
#730 task rgrp closed duplicate Back up package data from all CKAN packages to storage.ckan.net

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net.

  • Naming scheme?
    • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
      • What happens if names change
    • File: filename? hash?
  • Store hash back on ckan instance?
1287737109000000 1291139609000000
#782 defect pudo rgrp ckan-v1.3 closed fixed Add and remove users from sysadmin role from the command line

Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems).

Suggest creating a new paster command as first step.

# user identifier = openid (?)
paster sysadmin add {user identifier}
paster sysadmin remove {user identifier}

Suggest this also be added to main CKAN docs.

This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)

1289205287000000 1289218661000000
#797 enhancement rgrp rgrp ckan-v1.3 closed fixed Rework core html layout to mirror wordpress twentyten

WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes.

  • Convert to wp twentyten html
  • Switch css to be based off twentyten css (and perhaps rework somewhat)
1289402873000000 1289402982000000
#806 enhancement rgrp rgrp ckan-v1.3 closed fixed Metadata created and last modified timestamps for packages

Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)

1289854894000000 1324034356000000
#807 enhancement rgrp ckan-v1.3 closed wontfix Handle robot traffic better

We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest:

  • Add rel=nofollow to various links
  • Throttle some robots (not all seem to respect robots.txt)
1289899826000000 1297075372000000
#808 enhancement pudo rgrp ckan-v1.4-sprint-2 closed fixed Allow i18n language choice on site (and possibly set automatically)

Allow choice of language on ckan instance.

  • Configuration variable listing language choices (could try doing this automatically but recommend against this)
  • Switcher option in UI (perhaps on user profile page)
  • Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)

1289944394000000 1297783658000000
#828 enhancement rgrp rgrp ckan-v1.3 closed fixed Update cache support in home controller to use cache config in ini file

Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...).

This is especially frustrating in a development environment where is currently running into fact front page is being cached.

Fix this.

1290693154000000 1290696586000000
#830 enhancement rgrp closed fixed Themes for CKAN

Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/

1290760612000000 1311180263000000
#831 task rgrp rgrp ckan-v1.3 closed fixed Migrate ckan trac to ckan.org

Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.

1290760820000000 1290760864000000
#832 enhancement rgrp rgrp ckan-v1.3 closed fixed Convert stats to a plugin

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

1290765915000000 1296334980000000
#833 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed [super] Administrative dashboard extension

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

  • Purge revisions (or sets of revisions) and purge objects #1076
  • Set roles for users #1075
  • Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
    • WONTFIX
  • Overview of activity
    • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

https://bitbucket.org/okfn/ckanext-admin

Tickets

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system
1290765921000000 1303236364000000
#836 enhancement rgrp ckan-v1.3-sprint-1 closed fixed Use site_url config option in templates

A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h

1291132434000000 1302882808000000
#837 enhancement rgrp ckan-backlog new CKAN integration with freebase gridworks / google refine

Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

  1. User installs Refine and CKAN extension for refine
  2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
  3. They edit the dataset on Refine
  4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

  1. User visits a package on CKAN.net (or another CKAN instance)
  2. There is a button on the page "View and edit this dataset in Google Refine"
  3. Click button -- ask them if they have Google refine installed
    • Yes: instructions for loading dataset into refine
    • No: load dataset in hosted version of google refine (we could run this)
  4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.
1291140609000000 1339774605000000
#842 enhancement johnglover rgrp ckan-v1.4-sprint-7 closed fixed Todo list CKAN extension

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

  • Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
  • At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
    • If I am logged in
      • See a form for "Add to do" at the top of the todo section and can add one straight away
      • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

  • Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.

Implementation

The Todo form

  • One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
  • Add a description
  • Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

  • GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1

  • support limit (?)

/api/2/todo/category -> return list of todo categories

  • No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

  • Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...
1291467708000000 1305646487000000
#847 enhancement pudo rgrp ckan-v1.3-sprint-2 closed fixed CKAN search using SOLR backend

This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

  • Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
    • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
    • Asynchroously: via worker with queue
      • #324 Search indexing using notifications
      • #669: Refactor Solr (indexer) to become a generic worker
      • #874: Extract Solr search backend into an extension
  • In core support using SOLR backend as well as postgres for queries
    • #317: Make search pluggable (?)
  • Integrate into WUI (if changes needed)
    • #672 - Facets in CKAN search results

Extras:

  • Solr index testing tool #431
1291639273000000 1295259902000000
#869 enhancement rgrp rgrp ckan-v1.3-sprint-1 closed fixed Add plugin interface to allow editing and extending of config by extensions

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

1291989395000000 1292235262000000
#875 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Search backend supports solr interface and query API mimics solr

Consolidate search API interface (and backend) on solr (solrpy) type interface.

  • Support for standard query structure
  • Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

  1. Disable (happens automatically)
  2. Implement - suggest using group by etc

Extras

  • Front-page tag cloud: change this to use facets
    • Accept this means that if facets not functional in backend we have no tag cloud
1292844957000000 1297085261000000
#876 enhancement sebbacon rgrp ckan-v1.3-sprint-2 closed fixed Support sqlite as a database backend for CKAN

Among other things this will allow the tests to run much quicker.

1292858711000000 1294753889000000
#877 enhancement rgrp rgrp ckan-v1.4-sprint-2 closed fixed File upload in WUI (extension)

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

  • Implement as an Extension
  • Locate at /upload
  • Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
  • Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
  • This wants to be tied in to general package workflow. See parent ticket:852.
1292868146000000 1298624165000000
#878 enhancement rgrp rgrp ckan-sprint-2011-09-12 closed fixed Integrate file upload with workflow around package resources

Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

  • 'ResourceExtras?' for extra metadata on resources
  • Makes 'Resource' into first class object (and make available in WUI)?

Depends

  • #877 (Upload in WUI) - should be part of that extension
1292868269000000 1315820838000000
#879 enhancement wwaites rgrp ckan-v1.3 closed fixed Storage auth API

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

1292868509000000 1316030482000000
#888 enhancement johnglover rgrp ckan-sprint-2011-10-28 closed fixed Improvements to the dataproxy and the data API

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

  • Testing for dataproxy
    • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
  • Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
  • Google docs style row/column selections
  • Use the swiss library - https://bitbucket.org/okfn/swiss
    • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
  • Handle redirects for content-length?
  • Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

  • Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}
1293649783000000 1311773103000000
#889 enhancement rgrp rgrp ckan-v1.3-sprint-2 closed fixed Support extra footer material in config option (e.g. for google analytics)

Add a config option 'template_footer_end' which is inserted in layout_template just before </body>.

This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics.

Aside: going forward may want to turn this into a extension.

Cost: 1h

1293713189000000 1293715109000000
#907 enhancement rgrp ckan-v1.3-sprint-3 closed fixed Data previews: preview package resource data in the WUI

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

  1. Need a 'data proxy' - see ticket:698 and ticket:888
  2. Preview using javascript in WUI - do this in an extension http://bitbucket.org/okfn/ckanext-dataapi
1295261173000000 1297072303000000
#908 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Issues deploying extensions with modwsgi / uwsgi

Some extensions import pylons features (such as request) at root.

Have encountered following issues:

  1. You need to import ckanext before any ckan stuff in your wsgi script (extension stats and dataapi)
  1. Running under wsgi daemon mode (normal mode is fine!) the front page (front page only) will not work (does not occur with disqus or synchronous_search but does with stats and dataapi extensions). You get errors like this in the log:
[Mon Jan 17 09:01:19 2011] [error] Exception KeyError: KeyError(-1218594160,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Mon Jan 17 09:01:20 2011] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
1295264624000000 1296730578000000
#927 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed [super] Improvements to CKAN documentation

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

Miscellaneous

  • Contributors / credits page: ticket:928
  • (Data proxy documentation: move it to packages.python.org/dataproxy)
  • (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
  • Document license configuration and license system - ticket:973
  • Refactor trac frontpage moving general feature overview to ckan.org
  • Create proper roadmap information on http://trac.ckan.org/
  • Add link to wiki.ckan.net to ckan.org
1295869492000000 1300105638000000
#928 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Documentation - CKAN credits / contributors page
  1. Remove contributors list from about page (out-of-date and does not belong there)
    • A "Powered by CKAN" link in footer is enough
  2. Add contributors / team / credits page to ckan.org
    • May want to separate credits (e.g. software use), from contributors (also is team different from contributors)
    • (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket)
  3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info.

Cost: 1h

1295872661000000 1297115136000000
#936 enhancement johnglover rgrp ckan-v1.4-sprint-6 closed fixed Follow / watch package extension

As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). Decision: use follow

Implementation

Interface

Become a follower:

  • Follow button on packages (if already watching say 'unfollow')

Package-related changes:

  • Show number of followers on a package
  • List followers of a package at /package/{name}/followers
    • On a separate page

a package User-related changes:

  • List followed packages
    • Either on user's page on a separate 'following' page. (NB: called 'following')
  • Does watching involve notifications (by email)
    • Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
  • [Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page

Nitty-Gritty

  • Want to do this in ajax-y manner
  • API endpoint: /api/2/follower
  • Store data in a new follower table

API

/api/2/follower

follow => PUT / POST
{
   user_id
   object_type
   object_id
}

If this is submitted by a user with user.id != user_id => error (401)

unfollow => DELETE

/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

Table

Called 'follower'

user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]

Random Extras

  • What about following users as well
1296339079000000 1303838713000000
#937 enhancement sebbacon rgrp ckan-v1.4-sprint-5 closed fixed Record download stats for resources (extension)

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

Old Spec (do it in CKAN)

  • Record info of form: resource id (or url?), timestamp
  • Do this via javascript capturing of onclick event talking to an api
  • API: /api/resource/{id}/download
    • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
    • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

  • Do we record ip addresses (to handle de-botting etc)?
  • Do we count preview clicks as well?
1296340273000000 1302513831000000
#938 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Message flashing / notification in WUI

Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see http://flask.pocoo.org/docs/patterns/flashing/).

We already do some of this (slightly poorly), e.g. on adding an authorization the app says: "X authorization added".

Implementation

  • Standardize this process and add section to html for this purpose.
  • Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit
  • Testing: suggest keeping this very simple

Estimate

  • Cost: 0.5h (html etc) + 1h to do some integration
1296398920000000 1297077023000000
#939 enhancement rgrp closed fixed Notification bar at top of site shown to new users pointing them to simple instructions

On http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

  • Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
  • FAQ/instructions page (use the new wiki?)

Depends

Estimate

  • Cost: 1h
1296399008000000 1323171158000000
#940 defect pudo rgrp ckan-v1.3 closed wontfix OpenID login via google for same user generates 2 different openid urls

This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h

1296401521000000 1297071748000000
#941 enhancement thejimmyg rgrp ckan-v1.5 closed wontfix Submit apps or ideas for apps related to datasets (extension)

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

1296403099000000 1310134339000000
#947 enhancement rgrp closed duplicate Move the assignment of roles to a package on package creation to an extension

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

1296557478000000 1314031310000000
#953 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Add tagline/description to template and set in configuration

Have a tagline / description are under main title and set if from ckan.site_description config variable.

1296809808000000 1296809834000000
#954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3

Child tickets:

  • #1107 Move package autocomplete from package controller and move to API
  • #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

1296811899000000 1320142744000000
#960 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Support non-ascii character in internal error messages

Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way.

Should do this not just here but everywhere we can find in the code base.

Module ckan.controllers.package:302 in edit
<<          am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg)
               if not am_authz:
                   abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
       
               auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg)
>>  abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Estimate

Cost: 1h

1297069721000000 1297169056000000
#961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

  1. Generating (or just filling) a form template with 'form data' (and errors)
  2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
  3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

  • #926 - Pick a simpler form framework
  • #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
    • #1079 Refactor API to use new logic layer and dictization
    • #1078 Refactor WUI controllers and forms to use logic layer
    • cf existing dumper and importer code
    • This will fix #662
  • [not ticketed yet] - validation layer (should work on serialized objects?)
  • #662 - Can't put entity that is returned by posting to package register (Defect)
  • #972 - Merge 'extras' into main package dict rather than having separate key
  • #1035 - Form impressions are given IDs
  • #810 - Move "add packages" field up in group form (easier to do this once forms are done)
1297069849000000 1310126100000000
#962 enhancement rgrp rgrp ckan-v1.4-sprint-5 closed fixed Improvements to data preview extension

Basic implementation done (and deployed):

https://bitbucket.org/okfn/ckanext-datapreview

However plenty to improve, e.g.

  • Support more formats (use external systems for preview?)
    • json (!)
    • html (trivial!)
    • sparql
    • ...
  • Do not display preview if no preview

Also suggest reworking to use external services rather than doing preview 'in house' (doing in house places heavy reliance on data proxy service and on converting data to a standard format).

1297072524000000 1301364987000000
#963 enhancement thejimmyg rgrp ckan-v1.4-sprint-1 closed fixed Package CKAN as a debian package

Package CKAN as a debian package so it is easier to install.

1297076364000000 1298284252000000
#964 enhancement rgrp rgrp datapkg-0.8 closed fixed Create Resource Downloader and make it pluggable

In downloading packages we need to download resources and we need a ResourceDownloader? object for this. these should be pluggable so that we can add support for different types of resources.

1297211004000000 1297211237000000
#972 enhancement sebbacon rgrp ckan-v1.5 closed wontfix Merge 'extras' into main package dict rather than having separate key

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

  • Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

1297332282000000 1310134129000000
#973 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Document license configuration and license system.

Document how license system works and specifically how licenses can be configured.

Cost: 2h

1297332948000000 1297678851000000
#975 defect rgrp rgrp ckan-v1.4-sprint-1 closed fixed Placeholder attribute not displayed in some browsers leading to poor UX

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

1297343755000000 1297344448000000
#982 enhancement dread rgrp ckan-v1.4-sprint-2 closed fixed Remove various pip-requirements versions

We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head.

Remove them and update any client systems.

Cost: 20m

1297516060000000 1298887980000000
#992 defect rgrp rgrp ckan-v1.3 closed fixed Use X-forwarded-for whenever it is available to set remote IP address

At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache.

We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available.

Cost: 10m

1298058532000000 1298060474000000
#1000 enhancement kindly rgrp ckan-v1.4-sprint-3 closed fixed Remove call timings code

Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

  • lib/base.py: c.time_call_started
  • ....
1298489643000000 1298912726000000
#1001 enhancement rgrp rgrp ckan-v1.4-sprint-4 closed fixed API should use normal user credentials if available

When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController? to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author).

1298489705000000 1301310351000000
#1002 enhancement dread rgrp ckan-sprint-2011-12-05 closed fixed Remove changeset model code

This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed).

1298489773000000 1322515696000000
#1003 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed CKAN Javascript library and demonstration web interface

A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library.

Why?

  • Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core
    • E.g. for specific communities e.g. geodata, science
    • Specialized tasks - multi-package editing
  • Very easy deployment and integration (e.g. can drop in to getthedata or other sites)
1298490086000000 1300100411000000
#1006 enhancement kindly rgrp ckan-v1.4-sprint-4 closed fixed Deprecate stable branch

Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time).

Cost: 10m (giving high priority because of low cost)

(Assigning to dread as he has been managing the stable branch).

1298624996000000 1300372286000000
#1009 enhancement pudo rgrp ckan-backlog new Improvements to user accounts sytem
  • Forgot password (email a new password)
  • Confirm email
  • Do not show register page if you are logged in (redirect to home page)
  • ticket:1010 - listing of users.
    • Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)
1298635991000000 1310128574000000
#1010 enhancement rgrp rgrp ckan-v1.4-sprint-2 closed fixed List CKAN users in WUI

Should have user listing at /user/ rather than user account page.

  • list users, sorted by number of packages contributed/edited
  • Move user home page to /user/{user-id}
  • Paginated
1298649180000000 1298740889000000
#1031 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-4 closed fixed User lookup API

Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users).

  • API location: /api/util/user/lookup?q=querystr&limit=10
  • Return json objects containing {id: ..., name: ..., fullname: ...}
  • Put in a module called controllers/apiv2/user.py
1299780419000000 1300101520000000
#1032 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Resources in WUI

Add resources into Web User Interface.

  • Locate at: /dataset/{dataset}/resource/{id}
  • CRUD
  • Authorization

Done:

  • #945 - Richer resources - Resource Groups, new fields, improved UI
  • #1445 - Resource View page in WUI
  • #1450 - Dataset view pages to match

Moved to superticket #1506:

  • #978 - Edit Resource Extras in Web UI
1299782021000000 1330348463000000
#1069 enhancement tobes rgrp assigned Stub datasets (request for datasets)

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

  • Agree a new dedicated tag. e.g. todo.does-not-exist
1301666919000000 1340632215000000
#1070 enhancement rgrp rgrp ckan-v1.5 closed fixed Plan a new domain model and layer architecture for CKAN

See http://wiki.ckan.net/Domain_Model especially section on v2.

  • New domain model is planned but not yet finally agreed.
  • Layer architecture is complete and implemented
1301910940000000 1310117129000000
#1074 enhancement rgrp rgrp ckan-v1.5 closed fixed Refactor authz web user interface to have common code and templating

Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

  • Common template code (checkbox template)
  • Logic code (or just common code) for wiring into authz system
  • Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup.

1302271586000000 1314303581000000
#1075 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed Administrative dashboard - Edit Authorization related to System object

Roles on System object are important because admin role on system equates to being a 'sysadmin' (i.e. able to do anything).

  • Make users sysadmin (either as separate action or as part of editing roles on system object)
  • /authz subpage for editing roles on system object
    • Add and update user roles
    • Add and update authz group roles
    • List actions associated to roles at top of page (extra points for checkbox table with editability)
  • Document on http://wiki.ckan.net/Authorization what roles on System object 'mean' esp sysadmin role on System

Related Tickets

  • super ticket: #833
  • authz lib improvement and authztool refactor #1050
1302279799000000 1303227982000000
#1076 enhancement johnlawrenceaspden rgrp ckan-v1.4 closed fixed Improve revision and package purge system

Purging Revisions

  • Delete button displayed on:
    • /revision/list
    • (/package/history)
      • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
    • /revision/{id}
  • Delete button submits to delete action on revision and changes revision state to 'deleted'.
    • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
  • Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

  • Put into deleted state.
  • Listed on /ckan-admin/trash
  • Separate Empty trash button which deletes all associated revisions.
    • Should be separate from Empty trash for revisions

Current system

  • Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)
1302283442000000 1303236302000000
#1077 enhancement kindly rgrp ckan-backlog new Move to simpler vdm system

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

  • The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset

for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

  • does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

  • Easier support for pending state and similar behaviour
  • No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

  • Migration is required
  • More difficult to query revision history.
    • Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
  • Performance (?)
    • Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
      • Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

1302304464000000 1340034345000000
#1078 enhancement kindly rgrp ckan-v1.5 closed fixed Refactors WUI controllers and forms to use logic layer
  • Deserialize forms to new dict format.
  • Replace controllers/forms to use dictization.
1302509347000000 1305828973000000
#1079 enhancement kindly rgrp ckan-v1.4-sprint-5 closed fixed Refactor API to use new logic layer and dictization
  • Convert current api saves to the new standard dict format.
1302509530000000 1302777504000000
#1106 defect rgrp rgrp ckan-v1.4-sprint-6 closed fixed Bugs related to routes arising from API refactor + removal of default routes

Various bugs I've been encountering:

Latter issue was masked by existence of 'default' routes:

   map.connect('/{controller}', action='index')
   map.connect('/:controller/{action}')
   map.connect('/{controller}/{action}/{id}')

Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap.

In addition I think we should be cautious about 'default' routes in core such as:

    map.connect('/api/rest/:register', controller='api', action='list',
        conditions=dict(method=['GET'])
        )

As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map).

1303747360000000 1303834069000000
#1107 refactor tidy-up bitesize rgrp closed fixed Move package autocomplete from package controller and move to API

Currently autocomplete method on package controller. This method should be in API (like other autocomplete methods).

Will need to update client code (just forms atm I think).

1303808480000000 1340632612000000
#1119 enhancement rgrp rgrp ckan-v1.4-sprint-7 closed fixed Fully functional storage extension with file upload

Previous work in #877 and #879 + #853 (storage API). In this ticket:

  • Improve authorization
  • Establish convention for laying out files on disk
  • Add documentation
  • Fix bugs with #879 (does not currently work -- boto may have changed)
1304094382000000 1305037201000000
#1135 enhancement kindly rgrp assigned Changeset model for vdm

Move to Changeset model for vdm.

A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned.

This change would also incorporate significant simplication of vdm.

1305209986000000 1340632267000000
#1136 enhancement kindly rgrp assigned Move to SessionExtension in vdm

When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that SessionExtension? exists we should use it. We can also follow the existing SQLAlchemy recipe: <http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>

1305210855000000 1340632980000000
#1137 enhancement kindly rgrp assigned Remove need for statefulness in vdm

Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is required because, atm, revision objects have FKs to continuity objects.

This ticket proposes the following changes:

NB: this could be limited just to case of join tables (leaving state stuff on other tables)

  • Remove FKs from revision to continuity (or allow for them to be nullable).
    • We could just limit this to m2m stuff
  • Delete of an object leads to:
    • Deletion of continuity object
    • Adding an entry in revision table with state set to deleted (we retain state on revision table)

If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain "active" relationships.

If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state).

The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects).

1305211628000000 1340631974000000
#1142 enhancement annapowellsmith rgrp ckan-v1.5 closed fixed [super] Major Overhaul and Extension of CKAN Documentation

Child tickets:

  • #1041 Start Using the CKAN Wiki for Tutorial-style documentation
  • #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

  • CKAN 1-page overview (for enterprise and for data hackers)
  • Administrator's Guide (including install)
  • Extensions Guide
  • Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

Minor Items

1305727452000000 1313775665000000
Note: See TracReports for help on using and creating reports.