{22} Trac tickets (2647 matches)

Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR).

Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable.

Should prominently (at the top?) mention existing code libraries for working with ckan api. Have:

• Python: ckanclient
• Perl: luke closs wrote something
• PHP: drupal library?
• ...?

During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box.

Sort out how we deal with dashes versus underscores in package names.

Storage represents something you can upload to (download is done in ticket:406).

• storage sub package with defined interface (maybe just use OFS)
• Only support uploading single files for the moment (though can do multiple files).
  • Relationship to build (ticket:306)
  • No support for syncing e.g. hg/git repos -- that's up to you as a user
• Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication.

Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download.

• Also rename InstallCommand? -> DownloadCommand?

And we should support interactive retrieving of resources ...

Implementation Ideas

for resources in pkg.resources:
    # datapkg/hg 
    load_storage_handler(resource.format)

Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed.

No need for requirement - closing in favour of #505

• Parse data
• Load via API
• Develop test for load via API (can be local or remote -- with paste webfixture may be easier)

To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: ​http://www.ibm.com/developerworks/web/library/wa-rails2/

In the first place we just need to apply this to:

• You are logged in as
• Package/group edit/authz tabs

E.g. ​http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins).

Also suggested that these are showing up in the API (courtesy of Anja)

Cost: 2h

Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems).

Suggest creating a new paster command as first step.

# user identifier = openid (?)
paster sysadmin add {user identifier}
paster sysadmin remove {user identifier}

Suggest this also be added to main CKAN docs.

This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)

Packages should provide create and last modified attributes which are included in package dict exposed via api.

These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)

Allow choice of language on ckan instance.

• Configuration variable listing language choices (could try doing this automatically but recommend against this)
• Switcher option in UI (perhaps on user profile page)
• Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)

Already have the basic capacity using *_extras paths for templates and public directory (see ​http://wiki.okfn.org/ckan/doc/theme).

However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: ​http://packages.python.org/Flask-Themes/

Convert stats section to a ckan plugin.

Stats need not be in core and making it a plugin would make it easier to extend and expand.

A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

• Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
• At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
  • If I am logged in
    • See a form for "Add to do" at the top of the todo section and can add one straight away
    • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

• Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.

Implementation

The Todo form

• One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
• Add a description
• Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

• GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1

• support limit (?)

/api/2/todo/category -> return list of todo categories

• No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

• Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

Among other things this will allow the tests to run much quicker.

Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

• 'ResourceExtras?' for extra metadata on resources
• Makes 'Resource' into first class object (and make available in WUI)?

Depends

• #877 (Upload in WUI) - should be part of that extension

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

• Testing for dataproxy
  • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
• Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
• Google docs style row/column selections
• Use the swiss library - ​https://bitbucket.org/okfn/swiss
  • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
• Handle redirects for content-length?
• Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

• Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

1. Need a 'data proxy' - see ticket:698 and ticket:888
2. Preview using javascript in WUI - do this in an extension ​http://bitbucket.org/okfn/ckanext-dataapi

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

• (Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)
• Document caching system - ticket:841
• Howto guide for writing an extension - started on this at ​http://wiki.ckan.net/Extending_CKAN

Miscellaneous

• Contributors / credits page: ticket:928
• (Data proxy documentation: move it to packages.python.org/dataproxy)
• (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
• Document license configuration and license system - ticket:973
• Refactor trac frontpage moving general feature overview to ckan.org
• Create proper roadmap information on ​http://trac.ckan.org/
• Add link to wiki.ckan.net to ckan.org

As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). Decision: use follow

Implementation

Interface

Become a follower:

• Follow button on packages (if already watching say 'unfollow')

Package-related changes:

• Show number of followers on a package
• List followers of a package at /package/{name}/followers
  • On a separate page

a package User-related changes:

• List followed packages
  • Either on user's page on a separate 'following' page. (NB: called 'following')
• Does watching involve notifications (by email)
  • Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
• [Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page

Nitty-Gritty

• Want to do this in ajax-y manner
• API endpoint: /api/2/follower
• Store data in a new follower table

API

/api/2/follower

follow => PUT / POST
{
   user_id
   object_type
   object_id
}

If this is submitted by a user with user.id != user_id => error (401)

unfollow => DELETE

/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

Table

Called 'follower'

user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]

Random Extras

• What about following users as well

Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see ​http://flask.pocoo.org/docs/patterns/flashing/).

We already do some of this (slightly poorly), e.g. on adding an authorization the app says: "X authorization added".

Implementation

• Standardize this process and add section to html for this purpose.
• Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit
• Testing: suggest keeping this very simple

Estimate

• Cost: 0.5h (html etc) + 1h to do some integration

This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

• ​http://groups.google.com/group/google-federated-login-api/web/the-most-important-technical-issue-in-using-the-google-accounts-api?pli=1
• ​http://stackoverflow.com/questions/3549953/how-to-get-unique-identifier-from-openid-google

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

Child tickets:

• #1107 Move package autocomplete from package controller and move to API
• #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

• ​http://wiki.apache.org/solr/SolJSON
• ​http://www.flickr.com/services/api/response.json.html

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

1. Generating (or just filling) a form template with 'form data' (and errors)
2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

• #926 - Pick a simpler form framework
• #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
  • #1079 Refactor API to use new logic layer and dictization
  • #1078 Refactor WUI controllers and forms to use logic layer
  • cf existing dumper and importer code
  • This will fix #662
• [not ticketed yet] - validation layer (should work on serialized objects?)
• #662 - Can't put entity that is returned by posting to package register (Defect)
• #972 - Merge 'extras' into main package dict rather than having separate key
• #1035 - Form impressions are given IDs
• #810 - Move "add packages" field up in group form (easier to do this once forms are done)

Package CKAN as a debian package so it is easier to install.

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

• Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache.

We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available.

Cost: 10m

When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController? to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author).

A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library.

Why?

• Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core
  • E.g. for specific communities e.g. geodata, science
  • Specialized tasks - multi-package editing
• Very easy deployment and integration (e.g. can drop in to getthedata or other sites)

• Forgot password (email a new password)
• Confirm email
• Do not show register page if you are logged in (redirect to home page)
• ticket:1010 - listing of users.
  • Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)

Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users).

• API location: /api/util/user/lookup?q=querystr&limit=10
• Return json objects containing {id: ..., name: ..., fullname: ...}
• Put in a module called controllers/apiv2/user.py

Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki.

We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ...

(Just as we have datasets listed that exist but aren't available ...)

Alternative would be to have a request for datasets subsystem.

I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists.

Implementation

• Agree a new dedicated tag. e.g. todo.does-not-exist

Related

• This ticket has clear links to ​http://getthedata.org/

Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz.

Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core.

• Common template code (checkbox template)
• Logic code (or just common code) for wiring into authz system
• Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately.

Will also deliver a significant improvement in the form of ajax user lookup.

Purging Revisions

• Delete button displayed on:
  • /revision/list
  • (/package/history)
    • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
  • /revision/{id}
• Delete button submits to delete action on revision and changes revision state to 'deleted'.
  • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
• Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

• Put into deleted state.
• Listed on /ckan-admin/trash
• Separate Empty trash button which deletes all associated revisions.
  • Should be separate from Empty trash for revisions

Current system

• Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)

• Deserialize forms to new dict format.
• Replace controllers/forms to use dictization.

Various bugs I've been encountering:

• Autocomplete of tag names no longer works (no longer works on ​http://ckan.net/). Appears to be due to no longer have a routes for apiv2 (i'm seeing failing calls to: ​http://ckan.net/apiv2/package/autocomplete?callback=callback&incomplete=a)
• Incorrect url generated for API in page footer (e.g. ​http://ckan.net/rest/get_api) due to use of old 'rest' rather than new 'api'

Latter issue was masked by existence of 'default' routes:

   map.connect('/{controller}', action='index')
   map.connect('/:controller/{action}')
   map.connect('/{controller}/{action}/{id}')

Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap.

In addition I think we should be cautious about 'default' routes in core such as:

    map.connect('/api/rest/:register', controller='api', action='list',
        conditions=dict(method=['GET'])
        )

As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map).

Previous work in #877 and #879 + #853 (storage API). In this ticket:

• Improve authorization
• Establish convention for laying out files on disk
• Add documentation
• Fix bugs with #879 (does not currently work -- boto may have changed)

When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that SessionExtension? exists we should use it. We can also follow the existing SQLAlchemy recipe: <​http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>

Child tickets:

• #1041 Start Using the CKAN Wiki for Tutorial-style documentation
• #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): ​http://trac.ckan.org/ticket/927

• CKAN 1-page overview (for enterprise and for data hackers)
• Administrator's Guide (including install)
• Extensions Guide
• Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: ​http://wiki.ckan.net/Documentation_Plans

Minor Items