{22} Trac tickets (2647 matches)

Results (2201 - 2300 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1649 enhancement ross ross ckan-sprint-2012-03-19 closed fixed Verify RDF Extension (ckanext-rdf) works
  • Make sure that the extension works as it should and isn't broken on current releases [3d]
  • Make it work with the EC schema, including extras [2d]
1326709096000000 1338205274000000
#459 bug johnbywater ckan-v1.2 closed fixed Versions on branches are broken 1282299973000000 1282921783000000
#53 enhancement rgrp rgrp v0.8 closed fixed View Information about Package history (i.e. package revisions)

As A

Visitor/User?

I want to

View information about a package history, i.e. what revisions it has.

Details

  • History info should be available at package/history/pkg-name
  • Most basic info is just info from revision (when it occurred, author etc)
  • More sophisticated would be to have diff support to see differences between revisions
1223552139000000 1239133021000000
#2827 defect danieljohnlewis demo phase 4 closed fixed View Item - RSS Feed Issues

Problem: Server Error (HTTP Code 500) when trying to see information related to an RSS feed... example: http://s031.okserver.org:2375/en/dataset/testerisation/resource/974df29e-77bb-4b5e-982c-cb7a295c1923 Expected: A page that looks a bit more like this: http://s031.okserver.org:2375/dataset/epims-lod2/resource/97531bd8-e75c-4efa-bfa2-bb51d0ba90d0

1344505590000000 1344549139000000
#6 enhancement somebody johnbywater milestone1 closed fixed View a package

As a

Visitor

I want to

View the details of an individual package

So that

I know more about the package. Attributes available are detailed in updating a package (../UseCase12)

1152549960000000 1183648175000000
#103 enhancement kindly rgrp ckan-v1.5 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
  • package history page should provide links to these pages

Cost: 2h

1251289897000000 1311180850000000
#1236 enhancement dread dread ckan-sprint-2011-11-07 closed fixed View a package at a given revision

As a user I want to view a package at a given revision:

  • When I visit /package/osm@yyyy-mm-dd or /package/osm@rrrrrrr I should be shown package at date yyyy-mm-dd or revision rrrrrrr.
  • viewing this page, it should say it is the old revision, just like wikipedia
  • package history page should provide links to these pages
1311269333000000 1330019788000000
#104 enhancement rgrp rgrp closed fixed View a package at a given version

As a user i want to see a package at "version" X (NB: not revision X).

  • When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version)
  • Implementation:
    • Find revision for this version (search revision history for when version field was last 0.7)
    • Show pacakge at that revision (as in ticket:103)
  • On history page also shows versions in list of revisions associated with the package
1251292672000000 1311181303000000
#7 enhancement johnbywater johnbywater closed invalid View a release associated with a package

See Update a release (ticket 14) for more information on what should be shown.

1152550022000000 1204136209000000
#32 enhancement rgrp rgrp v0.3 closed fixed View a tag

As a

Visitor

I want to

View a tag's page which should list all packages associated with that tag (or link to a page containing that list).

Notes

  • The tags should have a nice primary url such as /tag/<tag-name> (could also use the plural i.e. /tag/<tag-name>)
1157375968000000 1185472746000000
#8 enhancement johnbywater johnbywater closed invalid View file

As a

Visitor

I want to

View the files associated with a package.

Notes

  • See also ticket:16 (updating a file).
  • Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release)

1152550057000000 1250619147000000
#24 enhancement somebody johnbywater v0.9 closed fixed View list of most recently registered or updated packages on front page 1152551291000000 1246441366000000
#1238 defect kindly dread ckan-sprint-2011-10-28 closed invalid Viewing a package @date misses non-PackageRevisions

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: http://ckan.net/package/history/osm with this view: http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

1311351303000000 1311759784000000
#1491 defect seanh dread closed fixed Visible strings need internationalisation

Sean spotted that some strings need internationalising, such as flash messages. Generally all strings that appear on the web front-end should be internationalised.

Particular places that need i18n:

  • ckan/controllers/admin
  • ckan/lib/base

Note: there are some exceptions, such as in i18n.py, very obscure error messages and stuff that only appears on the API.

NB: there is a cost in making a string internationalisable (all our volunteers have to translate it), so we should not be too zealous. We should also look at the i18n/ckan.pot to see if any of the existing strings can be reused.

1322481000000000 1330020630000000
#1090 defect dread dread ckan-v1.4-sprint-6 closed fixed Visitor can't create packages on new CKAN install

Default visitor roles in default config is reader, not anon_editor.

Problem caused by changes in #1066 (released in 1.3.3)

New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages.

The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions.

1302635219000000 1302635699000000
#462 story dread johnbywater ckan-v1.2 closed fixed Visitor downloads a daily database dump 1282305305000000 1285757238000000
#1 enhancement somebody rgrp milestone1 closed fixed Visitor visits service

Title

Visitor visits service

As a

Visitor (general web user)

I want to

Visit the website of the service (frontpage)

So that

  • See the front page and discover information about the service
  • Perform various activities related to the service such as registering and finding packages, registering as a user and logging in
1152549417000000 1183636342000000
#716 requirement pudo pudo iati-3 closed fixed Visual improvments to IATI CKAN 1287583741000000 1290000673000000
#799 requirement johnbywater johnbywater closed duplicate WAF harvesting shall be optimised in respect of reharvesting only records that have changed 1289483264000000 1294232675000000
#650 story pudo pudo ckan-v1.2 closed fixed WUI editing for Authorization Groups
  • Copy & adapt Group controller tests to suit AuthorizationGroups?
  • Copy & adapt Group controllers and templates for Authorization Groups
1284890073000000 1287391932000000
#2304 enhancement seanh seanh ckan-sprint-2012-05-15 closed fixed Watching/following of dataset/users

User story #037 here: http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: http://ckan.okfnpad.org/27

1334587233000000 1337158641000000
#2783 enhancement shevski closed wontfix Way to generate homepage backgrounds for demos/ckan hosted

Really we want this for CKAN Hosted & for creating demos.

Currently we're using http://leaflet.cloudmade.com/ - but by finding the area & taking screenshots. Would be better to interate with the app so that: a) this happens automatically and b) correct attribution will be visible on maps

1343812931000000 1343818702000000
#341 enhancement dread closed fixed Web UI accepts package IDs in URLs

(in the same way that we can refer to packages in the API by ID as well as name.)

As a

client of CKAN

I want to

link to a package page in the UI, referring to it by ID

1276162400000000 1277483030000000
#745 requirement johnbywater ckan-v1.3 closed wontfix Web UI shall accept requests for harvesting jobs 1288013148000000 1294412820000000
#646 requirement johnbywater closed wontfix Web UI shall support registering harvest sources 1284740303000000 1294413500000000
#340 enhancement dread dread v1.1 closed fixed Web UI theme easier to configure

An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in.

http://wiki.okfn.org/ckan/doc/theme

1276009729000000 1328807317000000
#1019 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Webhooks notification service

We propose the following push approach using individual packages:

  1. New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form:

{

payload: The entity data as available from the REST API entity-type: 'Package', operation-type: 'create'|'update'|'delete' # one of these options

}

It will be sent as the body of the request with content-type set to application/json.

  1. The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review.
  1. Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model)
  1. [optional] CKAN side will support configuring authorization for basic authentication if applied on WP side.
  1. [optional] List queue status (including failures) on ckan adminstrative dashboard.
1299166784000000 1299166930000000
#1208 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed Webstore: a datastore with web API
  • SQLite based
  • Use Cyclone for async http server on top with auth
  • Designed to not be specific to CKAN or anything else

Current code is here: https://github.com/okfn/webstore.

API Spec

Read

Two basic ways to query::

    GET: /{owner}/{db-name}/?sql=...
    GET: /{owner}/{db-name}/?table=...&attr=value&attr=value&limit=...

Returns::

  {
      u'keys': [u'id', u'name'],
      u'data': [
          [1, u'jones'],
          [u'aaa', u'jones']
          ]
  }

Write

POST to::

/{owner/{database}/{table}

Payload is json data structured as follows::

{

unique_keys: [list of key attributes] data: {dict of values}

}

Authentication and Authorization

Authentication: use basic auth header.

Authorization:

  • Default: all read, owner can write
  • Restricted: owner can read and write, everyone can do nothing

Possible future: config file can specify a python method (TODO: method signature)

Integration with Other Systems

TODO: Specify how to delegate authenatication to user database in some other system.

1309804460000000 1312191646000000
#2934 enhancement Dominik new Webtests for CKAN

The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes.

An example user story to be tested would be: "As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system." or: "As a user I want to use the search functionality and open a resource from the search page."

These user stories require a different testing system with interaction and processes.

1347966647000000 1348060201000000
#1822 enhancement kindly kindly ckan-sprint-2012-03-05 closed fixed Weight mulilingual searches correctly

Dismax query across languages. Titles and selected languages, weighted more highly.

1329772080000000 1330990127000000
#61 enhancement rgrp rgrp v0.10 closed fixed When dumping data to json do not dump private information like API keys

Cost: 1h

1239123529000000 1265890790000000
#1109 defect kindly kindly closed fixed When extras has a value other than a string an integrity error occurs in the api.

This is a regression that happened after refactoring the api.

It was shown by

http://pastebin.com/2v7QasZy

1303839943000000 1305124697000000
#39 enhancement johnbywater rgrp v0.7 closed fixed When listing packages show more information than just name.

When a package listing is shown (as on http://ckan.net/package/list) more information than just name should be shown. For example package titles should be shown as well as names (could also list tags and other stuff but simplest is just title).

Cost: 1

1194608920000000 1223908298000000
#156 enhancement johnbywater dread v1.0 closed fixed Whitelist for html in mark-down field

Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed.

1255623973000000 1271962871000000
#418 task dread closed invalid With COI, get SOLR working 1281431691000000 1291637919000000
#1569 enhancement David Raznik jilly mathews ckan-future new Wordpressser

How much effort will this be to be ready to use?

1324294056000000 1324294056000000
#580 story johnbywater closed duplicate Write (create or update) CKAN package for metadata document 1284210730000000 1284223068000000
#2397 task seanh seanh ckan-sprint-2012-05-29 closed fixed Write CKAN docstring guidelines 1337160193000000 1337963273000000
#572 story johnbywater johnbywater ckan-v1.3 closed Write CKAN package from metadata document

The attributes we need to read in are being defined here: http://okfnpad.org/uklii

The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes.

Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft.

1284040784000000 1288038237000000
#2234 enhancement seanh ckan-future assigned Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

  • Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds
  • Feed items should probably be cached somewhere

Suggested

  • The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.
  • For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference
  • Caching is pretty crucial and should probably obey the ttl of the feed.
1331902755000000 1346669567000000
#1326 enhancement thejimmyg ckan-backlog new Write a set of auth plugin functions to integrate with Druapl

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

1315821084000000 1315821084000000
#1777 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Write about multilingual features for ec.

Basic description of multilingual features intended to be supported.

1328542835000000 1329393295000000
#440 task dread dread closed duplicate Write and pass comprehensive performance tests

Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes.

1282226932000000 1294417436000000
#1825 enhancement seanh seanh ckan-sprint-2012-03-05 closed fixed Write docs for vocabularies API

Done on branch feature-1698-tag-taxonomies

1329845122000000 1330807330000000
#2757 enhancement Adria seanh ckan-v1.8 closed fixed Write documentation for upgrading a CKAN install from an older to a newer version of CKAN 1343235242000000 1343392006000000
#1767 enhancement johnglover johnglover ckan-sprint-2012-02-20 closed fixed Write end-to-end functional tests for tag vocabularies

Need to test:

  • API call to create new vocab
  • API call to add tags with a given vocab
  • API call to update tags
  • Dummy plugin to test that updating tags works via WUI
1328537586000000 1329326678000000
#413 task johnbywater johnbywater ckan-v1.3 closed invalid Write functional tests for DGU caching 1281431633000000 1294407458000000
#2300 enhancement seanh seanh ckan-sprint-2012-04-30 closed fixed Write initial docs for multilingual branch 1334579163000000 1335644486000000
#2436 enhancement seanh seanh ckan-sprint-2012-05-29 closed fixed Write instructions for booting CKAN demo instances on s031 1337968345000000 1337968356000000
#2279 enhancement rgrp rgrp ckan-v1.8 new Write tutorial for our DataStore API / Data Explorer

Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)

1333482263000000 1338205426000000
#1661 defect dread ckan-backlog assigned Wrong Routes version installed by CKAN package

Jaakko Louhio reported that the wrong Routes version got installed during CKAN's package install.

He is using Ubuntu 10.04 and he believes it install Routes 1.12.3 instead of what we use which is 1.11.

1326718061000000 1339773949000000
#2906 defect dominik demo phase 5 new Wrong flags in language dropdown

The flags in the dropdown on details pages are not right.

You can see the error on http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0.

1346861144000000 1346862889000000
#334 defect pudo dread closed fixed Wrong link for package feed icon

This is a problem with the package page e.g. http://ckan.net/package/open-election-data-project . The feed icon links to [1] and the text next to it "Subscribe" links to [2]. Surely these should be the same? The second link seems to right one to me.

1275407445000000 1280743667000000
#2714 defect johnmartin shevski demo phase 5 closed fixed XLS preview not working

see http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873

what XLS previews do we support?

1342706454000000 1352206138000000
#1405 defect dread jonathan.gray@… ckan-sprint-2011-11-07 closed fixed You can add a package twice to a group

To reproduce (from Jonathan Gray):

  • I go to add an item to the group
  • As I type, I get autocomplete options (cool!)
  • When I click on one, it adds this, and copies the same autocomplete option in the box
  • When I click submit, it seems to add the item twice
  • On the main group page I think it just shows up once, but when I click edit it shows two identical boxes, both ticked - and the count includes two copies of the same item

Basically the problem is allowing the same package associated multiple times with a group.

This is an issue on CKAN 1.5b and probably earlier.

1318856354000000 1320152291000000
#1442 defect dread dread ckan-sprint-2011-11-07 closed fixed You can't GET then POST a dataset as new

If you try to GET a dataset, purge it then POST it again, it fails checking authz for the group:

Module ckan.controllers.api:278 in create
<<                  (register, subregister))
               try:
                   response_data = action(context, data_dict)
                   location = None
                   if "id" in data_dict:
>>  response_data = action(context, data_dict)
Module ckan.logic.action.create:240 in package_create_rest
<<      api = context.get('api_version') or '1'
       
           check_access('package_create_rest', context, data_dict)
       
           dictized_package = package_api_to_dict(data_dict, context)
>>  check_access('package_create_rest', context, data_dict)
Module ckan.logic:129 in check_access
<<          #    log.debug('Valid API key needed to make changes')
               #    raise NotAuthorized
               logic_authorization = new_authz.is_authorized(action, context, data_dict)
               if not logic_authorization['success']:
                   msg = logic_authorization.get('msg','')
>>  logic_authorization = new_authz.is_authorized(action, context, data_dict)
Module ckan.new_authz:16 in is_authorized
<<      auth_function = _get_auth_function(action)
           if auth_function:
               return auth_function(context, data_dict)
           else:
               raise ValueError(_('Authorization function not found: %s' % action))
>>  return auth_function(context, data_dict)
Module ckan.logic.auth.create:115 in package_create_rest
<<          return {'success': False, 'msg': _('Valid API key needed to create a package')}
       
           return package_create(context, data_dict)
       
       def group_create_rest(context, data_dict):
>>  return package_create(context, data_dict)
Module ckan.logic.auth.create:15 in package_create
<<      else:
               
               check2 = check_group_auth(context,data_dict)
               if not check2:
                   return {'success': False, 'msg': _('User %s not authorized to edit these groups') % str(user)}
>>  check2 = check_group_auth(context,data_dict)
Module ckan.logic.auth.create:90 in check_group_auth
<<      groups = set()
           for group_dict in group_dicts:
               id = group_dict.get('id')
               if not id:
                   continue
>>  id = group_dict.get('id')
AttributeError: 'unicode' object has no attribute 'get'

(using RESTful Model API)

The reason is the POST is expecting the group to be expressed as a dict, not the list of names that GET returns.

1320263318000000 1320275623000000
#448 story thejimmyg dread closed invalid [Assist with] editing packages through Drupal 1282299478000000 1294417061000000
#1477 enhancement kindly kindly ckan-v1.6 closed fixed [Super] Group refactor

This is the implementation of the final draft in

http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

  • model migration. (done)
  • simplify revisioning (done)
  • Remove any sqlalchemy releations from groups. (done)
  • change logic functions (done)

Single type per group (#1531)

  • migration add type to group table. (done) 1d

Add capacity table (#1522)

  • model migration. (done)
  • add to logic functions.

Allow other objects to be added to groups. (#1531)

  • change logic functions (done)

Adapt thedatahub.org to use new model.

  • move auth groups over to groups/members
  • change user_object roles to use groups? keep user object roles for time being.
  • make create update package/group put the correct items in the group/members/capacity tables.
  • work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

  • adapt auth functions to use auth.
  • adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

  • #1279 Deprecate Authorization Groups
  • #1198 Publisher hierarchy
1321959320000000 1328016209000000
#1736 enhancement kindly kindly ckan-v1.7 closed fixed [Super] Multilingual Metadata

Translate all areas of metatdata to selected language. This includes finding datasets in all languages.

Multilingal Metadata: Tasks:

  • Add translation table. 2d
  • Api to add translation 3d
  • Add extention point to dataset view. 3d
  • Add extension point to search index to add translations. 2d.
  • Add all eurovoc data to translations. 1d.
  • Make multilingual solr schema 2d #1715
1328012697000000 1338205016000000
#1599 enhancement rgrp rgrp ckan-sprint-2012-01-09 closed fixed [contrib] Simple embeddable dataset count widget (esp for group count)

Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups.

  • Simple group count widget in JS for embedding in wordpress and elsewhere
    • Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box.
  • All you need to do is do a dataset query over the API e.g. http://thedatahub.org/api/search/dataset?groups=economics and then embed in some html!

Estimate: 30m (for someone who knows their jquery).

1325246358000000 1325555201000000
#1523 enhancement icmurray icmurray closed invalid [super]

Form refactor.

New create/edit package form wizard for DGU. See attached spec.

1323172661000000 1323172904000000
#1240 enhancement kindly rgrp ckan-backlog assigned [super] API v4

(Just creating this ticket as somewhere to keep notes)

  • Decide on REST api versus action API
    • Do we want to support both?
  • Tidying
    • Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
    • extras merged into normal field list in package
    • Get rid of /rest/ so just have api/v1/package
    • Get rid of separation of search api from 'rest' api
      • Propose that GET on REST index is search e.g. /package/?q=...
        • This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
  • Resource read/write in API (separate from package)
    • Does this need authorization work?
  • user/account API - read/write
  • Remove autocomplete -- can just use search
    • Do not worry about backwards compat as should only be used in our js (if others using it too bad!)
1311525660000000 1325473312000000
#954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3

Child tickets:

  • #1107 Move package autocomplete from package controller and move to API
  • #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

1296811899000000 1320142744000000
#1515 enhancement seanh seanh ckan-v1.7 closed fixed [super] Activity Streams

We want to add "activity streams" (chronological lists of activities) to CKAN.

Activity streams standard: http://activitystrea.ms/

Super branch where the feature was initially developed (this has now been merged into master): https://github.com/okfn/ckan/tree/feature-1515-activity-streams

Etherpad: http://ckan.okfnpad.org/notifications (most relevant parts inlined here)

Friedrich's mockup: http://datahub.pudo.org/pudo

Tasks

Strikethrough means the task is done.

  • Migrate existing data into activity streams model [3.5d] #1510
  • Generate activities when objects are saved [4d] #1298
  • Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] #1511
  • API call for getting a user's public activity stream as JSON [0.5d] #1495
  • API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] #1494
  • Render a user's activity stream on her user page [3d] #1496 (needs review/page design work)
  • Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.
  • Logic functions for getting activity streams for packages, groups, and users.
  • API calls for getting activity streams of packages, groups, and users as JSON.
  • Rendering activity streams for packages and groups etc. into their pages. (done but disabled pending page design work)
  • API calls for inserting events into activity streams. (Still need to handle activity details in the API, ticketed).
  • Conform to activitystrea.ms standard
  • Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page.
  • Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams
  • Email notifications
  • Extension point for extensions to register themselves as renderers for different types of activity
  • Deploy to UAT [0.5d]
  • Docs and announce [1d]
1323105048000000 1338204149000000
#833 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed [super] Administrative dashboard extension

Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview.

Possible features:

  • Purge revisions (or sets of revisions) and purge objects #1076
  • Set roles for users #1075
  • Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages)
    • WONTFIX
  • Overview of activity
    • WONTFIX - already have revision log

Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/.

https://bitbucket.org/okfn/ckanext-admin

Tickets

Notes

Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group):

# first remove permissions from roles
# this is hacky but have to do it because we hardcode assignment of 
# role permissions on package on package create (see model/authz.py)
paster roles deny editor edit
paster roles deny editor create-authorization-group
paster roles deny editor create-group
paster roles deny editor create-package
paster roles deny reader create-package
# make superuser group
# create authz group administrators / Administrators (if not exists)
paster rights make agroup:administrators admin system
1290765921000000 1303236364000000
#1096 defect rufuspollock pudo ckan-future new [super] CKAN Hosted

Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas:

  • introduce model.Site and c.site
    • site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI)
  • Subdomain detector to activate sites.
  • use site in Authorizer instead of System, have a NullSite? for global things
  • allow cross-site search
  • packages are in a list of sites, m:n rather than 1:n
    • list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally.
1303235062000000 1339774484000000
#1190 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] CREP 0004 Data API and Data Processing System

For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point):

http://wiki.ckan.org/CEP0004

We can distinguish 3 modules that are needed:

  1. "Webstore": A datastore with dataapi - #1208

Suggestion is this would be sqlite based with a simple sql based API. http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query}

  1. Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. #1398
  1. A data processing system which utilizes this datastore. One could

get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - #1432

More info

1308227611000000 1323270522000000
#1065 enhancement zephod johnlawrenceaspden ckan-v1.6 closed fixed [super] Change Authorization System

Child tickets

  • #1198 Publisher hierarchy
  • #1050 Authz lib improvement and refactor of ckan/lib/authztool.py
  • #1004 Group creation instructions missing
  • #1099 Strange interactions between two browsers while playing with authz groups
  • #1115 can have two authzgroups with the same name
  • #1133 command line rights manipulation doesn't work
  • #1138 minor navigations behave inconsistently

Old ticket description:

  1. Change name of AuthzGroup? to UserGroup? to reflect what it is for
  1. Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
    • Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
    • Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)
  1. Change UserGroups? so that they can have a hierarchical structure,

More info on Hierarchy change

e.g. UserGroup? NHS contains the User nhsysadmin, as well as the UserGroups? SURREY and BERKS, which themselves contain users.

One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not.

Each Group has permissions over various objects.

A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively.

Algorithm:

possible(user, action, package):

if user has permission for action on package

or any of have that permission

or any of his groups group-children (but not user-children), and so on recursively have the permission.

1301508331000000 1324550041000000
#1593 enhancement icmurray amercader ckan-sprint-2012-04-16 closed fixed [super] Create feeds from search results

Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:

http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*

On this case a message "Subscribe to these search results" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)

Other tickets related:

  • #1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.
1324489115000000 1334566668000000
#1525 enhancement icmurray icmurray ckan-v1.7 closed fixed [super] DGU package form refactor

Form refactor. New create/edit package form wizard for DGU.

Etherpad: http://ckan.okfnpad.org/dgu-package-form

Tasks:

  • #1526 : Simple one-page form.
  • #1527 : Tab-ify the form using JS. No validation between steps.
  • #1541 : Setup server for the DGU form-refactor.
  • #1537 : Package create form wizard
  • #1538 : Store partially-filled in package form data as a draft.
  • #1539 : Package forms auto-save

How the rest of the work is carried out is up for discussion, as it depends on the approach taken. Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages. Google doc outlining design(s)

1323172839000000 1337159836000000
#1602 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Data Previewer / Viewer v2

We already have first pass of Data Preview that was done as part of #1357. This is about v2 of this. Two main components:

  • A new version of the separate Data Explorer library (now called Recline Data Explorer) - #1446 Est: 10-15d
    • Why:
      • Cleaner, backbone based solution.
      • Provides editing support if wanted.
      • Collaboration with Max Ogden
  • Integrate this into CKAN (into resource view page) - see #1630 Est: 1d
  • Improvements to Data Viewer / Previewer system in CKAN itself - Est: 2-3d
    • (Update Data Preview-er (View-er) to support images ...)
    • (WONTFIX here - (?) Update Data Preview-er to support geodata - #1151)

1325267544000000 1328175719000000
#2283 enhancement rgrp ckan-v1.8 closed wontfix [super] Data Previewer / Viewer v3
  • Basic geo map
  • Facets
  • Filtering
  • Pivot table support (?)
  • Export data (?)
  • Save data query and share
  • More and better graphing
  • Embedding! - /resource/../embed

User stories and analysis: http://ckan.okfnpad.org/feature-2283-data-viewer-v3

1334138341000000 1340623843000000
#1432 enhancement rgrp ckan-backlog new [super] Data processing system for CKAN and Webstore

Super ticket: #1190

A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)

1320142747000000 1339774041000000
#1607 enhancement icmurray dread ckan-v1.7 closed fixed [super] Data.gov.uk Maintenance Refactor and UKLP Development Support

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

  • 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1
  • 6. Publisher Registration Improvements MUST PHASE 1
  • 3. Themes/Taxonomy? as well as tags WONT PHASE 2
  • 32. Browse by Tags PHASE 1 MUST
  • 12. Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1
  • 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]
  • 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]
  • 22. Mixed Licenses MUST PHASE 2

Plus, brought forward from last sprint:

  • Publisher hierarchy support
  • Improved dataset creation tools (form) - #1525
  • Edit/delete for harvested datasets

Overview of tasks: see the design doc.

1325503348000000 1337159969000000
#1327 enhancement rgrp ckan-v1.6 closed duplicate [super] Dataset Archiving

Split out of #852. Automated archiving of datasets (related to QA).

Automated archiving using worker process

  • #890 - Timed actions in ckanext-queue
  • #891 - Resource download worker daemon
  • #892 - Make stored data available in WUI
1315821490000000 1320662446000000
#852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving

3 (4) use cases: (With and without access to auth keys.)

  1. upload form on ckan (by default local system has auth key for storage)
  2. datapkg upload (client) (by default without key) (plumbing done)
  3. background job archiving downloads (by queue client)

Most important are:

  1. Upload form on ckan.net (with access to keys)
  2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

  • #826 - Resource 'extra' fields
  • #358 - Resources in REST API
  • #235 - Resource format normalization and detection

User upload via Web UI

  • #877 - File upload in WUI
  • #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

  • #853 - plumbing for credential-less rest upload
  • #879 - Storage auth API

Also related:

  • #405 - Retrieval options for package resources

More Info

1291722993000000 1315821628000000
#1211 enhancement kindly kindly ckan-v1.5 closed fixed [super] Drupal integration

Sub-tasks / tickets:

  • #1335 Action API - access to any logic layer function
  • Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
  • Add package_create_validate & package_update_validate logic functions
  • Allow extensions to provide logic layer functions (IActions)
1309958442000000 1315948703000000
#1566 enhancement David Raznik jilly mathews ckan-v1.7 closed fixed [super] Finalize Google analytics extension
  • (Deploy it)
  • Integrate with stats plugin (now that is in core) - #1101
  • Blog about it #1581
  • (Other tickets re upgrading for v1.5.1 - done but where are tickets?)
1324293168000000 1338203338000000
#1508 enhancement ross rgrp ckan-v1.6 closed fixed [super] Finalize and deploy some of our major (new) features as of Dec 2011

This ticket is about finalizing, then deploying (to http://thedatahub.org/), and finally announcing some of recent major new features such as:

  • Deploy wordpresser extension for thedatahub (so we can start blogging) - #1533 - WONTFIX
  • (Deploy new Resource and Dataset pages (invite comments))
  • Get thedatahub geo-ified i.e. ckanext-spatial - #1601 (?)
    • Any additional dev that needs to be done? (e.g. geo in search ...?)
  • (Deploy / re-enable ckanext-googleanalytics on thedatahub - #1451 (more stuff #1566))
    • Rename the extension to ckanext-analytics?
  • Custom forms (and use of 'dataset types') - Ticket (??)
  • (Get QA deployed and running - #1582)

It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc.

1323088817000000 1328000871000000
#1521 enhancement zephod shevski ckan-sprint-2012-01-23 closed fixed [super] Groups UI and UX improvements in v1.6 from Cygri

Various UX and UI improvements to groups.

Etherpad: http://ckan.okfnpad.org/feature-1521-groups-ux

Improved groups

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as http://ie.ckan.net/ could be turned into a simple group on the Data Hub without losing their separate identity.

  • Group homepages: Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.
  • Subdomains for groups: Enable subdomains such as ie.thedatahub.org as aliases for thedatahub.org/group/ireland
  • Search within the group: Have a big search box that searches just within the group on the group page, with a look like on http://thedatahub.org/dataset
  • Have an Add a dataset to this group link on the group page, perhaps in the sidebar.
  • Navigation within the group: Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from http://thedatahub.org/dataset
  • Group documentation: Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.
    • The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.
    • Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines About, Frequently Asked Questions and How to get involved, we'd then get three links in the sidebar that point to datahub.org/group/ireland/about, datahub.org/group/ireland/frequently-asked-questions, datahub.org/group/ireland/how-to-get-involved
      • RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

Additional

  • Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)
  • Better group index page (sort by number of dataset, allow searching ...) - #1403
  • #1597 Tag search, filtered by group
    • RP: not worth it in terms of benefit/cost IMO

Steps

  • "Add Dataset" button on action bar
    • Make it a dropdown? -> New Dataset / Existing Dataset
    • Redirect Add Existing -> Edit
    • Check permissions
  • Expandable "about this group" box
  • Implement simplest search-this-group box
    • turn the list of datasets into a dynamic list of search results
      • pagination
    • Matching set of facets in the sidebar (matches 'real' search page)
      • Deleteable, displayed facets

Nice-To-Have

  • Get rid of the old group_package_show action (confirm with team)
  • Tabbed "about this group" box (extract headers?)
  • Scrollable group members widget in the sidebar
1323169961000000 1330675930000000
#1787 enhancement dread ckan-future new [super] Improve RESTful API
  • Lists of entities should be full URLs, rather than just the names
  • Discoverability - /api/v3/rest should list the entity types that can be listed

This could be v3 of the RESTful interface.

1328702082000000 1328702082000000
#1233 enhancement thejimmyg ckan-backlog new [super] Improve wiki-style functionality for history

At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas:

  • #191 Searching by modification date
  • #193 Searching by time-related field
  • #301 Package discussion pages
  • #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages
  • #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page
  • Other improvements as per my word doc.
1311179392000000 1315948668000000
#927 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed [super] Improvements to CKAN documentation

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

Miscellaneous

  • Contributors / credits page: ticket:928
  • (Data proxy documentation: move it to packages.python.org/dataproxy)
  • (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
  • Document license configuration and license system - ticket:973
  • Refactor trac frontpage moving general feature overview to ckan.org
  • Create proper roadmap information on http://trac.ckan.org/
  • Add link to wiki.ckan.net to ckan.org
1295869492000000 1300105638000000
#1232 enhancement thejimmyg ckan-backlog new [super] Interface improvements

Child tickets:

  • #1194 "Welcome back" message for newly registered user
  • #1202 Links to datapkg utility don't lead to info about it
  • #925 Change the search box icon to remove the down arrow
  • #923 Search box doesn't work in leaderboard page in stats extension
  • #1034 Flash message cached
  • #737 Markdown syntax summary page
  • #811 Extra field editing form layout breaks when there are long field names
1311178296000000 1315948536000000
#1571 enhancement rgrp jilly mathews ckan-backlog assigned [super] Issues Extension

"Proper" issues extension.

Overview

  • This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues)
    • Current extension does work but lacks polish
    • Should we polish or do significant improvements?
  • Functionality - see etherpad

Steps

  • (Rename todo(s) -> issue(s) in existing extension)
1324294225000000 1330863183000000
#1142 enhancement annapowellsmith rgrp ckan-v1.5 closed fixed [super] Major Overhaul and Extension of CKAN Documentation

Child tickets:

  • #1041 Start Using the CKAN Wiki for Tutorial-style documentation
  • #1192 Convert CKAN Sphinx docs into admin/reference manual

Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927

  • CKAN 1-page overview (for enterprise and for data hackers)
  • Administrator's Guide (including install)
  • Extensions Guide
  • Separate CKAN.net info from Software Documentation (?)

Also now a wiki page with more detail: http://wiki.ckan.net/Documentation_Plans

Minor Items

1305727452000000 1313775665000000
#1231 requirement kindly thejimmyg ckan-backlog closed wontfix [super] Management Information Reporting

Child tickets:

  • #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

1311173919000000 1325474447000000
#1141 CREP johnglover ckan-backlog closed fixed [super] Moderated Edits User Interface

Proposer: John Glover
Seconder: James Gardner

Abstract

We are trying to achieve these goals:

  • To get people involved with making edits to CKAN metadata.
  • To have an ownership model as to who can moderate and validate these changes
  • To not put too huge a burden on these owners.

This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made "official".

There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface.

The Problem

We require the following functionality:

  • Allow a group of changes to be stored as a new revision.
  • Allow a linear stack of "community" revisions.
  • Provide a way for the editor and moderator to compare previous revisions to the current one.
  • When a moderator approves a change it creates a new revision flagged "moderated" (this is analogous to a merge commit)
  • Provide a way for the editor and moderator comment on revisions if necessary.

Extra features:

  • Need a way to summarise the changes (as part of the preview perhaps)
  • Sysadmin needs to purge a revision completely

Specification

UI/UX

UI Mockup:

Revisions:

  • Revisions are per package rather than per field.
  • Internally CKAN has separate revisions for resources, extras and package metadata. From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision.

On the Edit page:

  • We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?).
  • The values displayed in the form are by default populated from the latest revision (whether community or moderated)
  • Under each field is a "shadow", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too.
  • When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it
  • If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named "Replace fields with values from this revision" under the revision list. You click this, a warning pops up and then you say "Yes". You then select the moderated revision again.
  • We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this.
  • In addition to package comments, each revision will have a revision log (analogous to a commit message).

Technical Details

  • This CREP will result in a new CKAN extension.
  • It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised.
  • This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use.
  • We will then use suitable test data to mimic these API calls until CREP0002 is ready.

Why do it this way

This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX.

Implementation plan

Deliverables

A new CKAN extension, consisting of:

  • Code: Python, HTML, CSS, Javascript
  • Unit tests
  • Localization
  • Documentation

Participants

John Glover to do it.

Progress

John has implemented the bulk of this UI. Just some things to tidy up before it is complete:

  • Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates
  • history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API)

I've split these two off into a new ticket #1604.

Related Progress

The Todo extension is written and available at: https://bitbucket.org/johnglover/ckanext-todo.

In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done.

See also

#1129 Backend work

1305721003000000 1325352507000000
#1605 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Multilingual support in CKAN

Multi-language:

  • dataset and resource metadata (and other objects such as groups?)
  • Field values in taxonomy (e.g. country names - Eurovoc)
    • #1665 Research into Eurovoc
    • Display Taxonomy in different languages
  • Field values not in taxonomy (e.g. title & description)
    • Use extra fields e.g. _i18n_title_fr = Le data.
    • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)
  • EC extension (templates, form)
    • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
1325473367000000 1338193641000000
#1294 enhancement rgrp rgrp ckan-v1.5 closed fixed [super] Package creation and editing UX improvements

Largely an integration of work from ckanjs

  • #1295 Simplify package create form
  • Improvements to edit form
    • Split edit form into sections
    • Remove preview from edit form (introduce inline preview on description/notes field)
  • #1296 Improved resource adding/editing on dataset/page page
  • #878 File upload integrated into resource creation
  • #1297 In-place editing of notes field for existing packages
1314405811000000 1315948257000000
#1669 enhancement ross ross ckan-v1.7 closed fixed [super] Publisher profile

Implement the necessary backend changes to build profile for publishers using the group refactor.

See http://ckan.okfnpad.org/publisherprofiles

Subtickets are at:

  • Publisher profile analysis #1670 [2d]
  • Publisher profile sysadmin authz #1671 [1d]
  • Publisher profile user authz #1672 [1d]
  • Test auth logic hooks for publisher profile #1674 [3d]
  • Publisher admin authz #1675 [3d]
  • Publisher editor authz #1676 [2d]
  • Add approval status field to groups #1673 [1d]
1326802473000000 1338204071000000
#1594 enhancement johnglover johnglover ckan-v1.7 closed wontfix [super] QA Improvements

Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement.

  • Integrate QA into frontend - 1d - #1583
  • QA report improvements - 2.5d - #1584
  • QA - Check license as part of 5 star rating - #1587
  • QA - Give SPARQL endpoints a 4 star rating - #1588
  • QA - Give 5 star rating to datasets with link metadata - #1589

Main pad: http://ckan.okfnpad.org/qa

Some QA discussion (relating to Celery in particular) at http://ckan.okfnpad.org/queue

1324489620000000 1338203215000000
#1578 enhancement rgrp ckan-backlog new [super] Re-enable and refactor ratings 1324322443000000 1325473015000000
#2226 enhancement rgrp rgrp ckan-sprint-2012-03-19 closed fixed [super] Refactor and improve documentation (v1.7)

Documentation is key to the success of projects!

Move material into main docs from wiki

Create a User Guide

Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/)

  • Publishing data
  • Authorization and workflows
    • Publisher Profile (Workflow)

Break into sections

Suggest something like:

# General
* Intro
* Conceptual Overview
* What CKAN helps you do (http://ckan.org/)
* FAQ ? (or point to the wiki)

# For Administrators

* Installation and Setup
  * Configuration options
* Customization - Theming etc
* Authorization and Workflows
* Storage

# For Users (Publishers, Data Wranglers, etc etc)
* Walkthrough of publishing a dataset
* Storage
* API (see Developers section)

# CKAN Developers

* Domain Model
* API
* Extensions
* i18n

CKAN Developers
* Buildbot
1331481965000000 1332475494000000
#1094 enhancement thejimmyg thejimmyg ckan-v1.5 closed duplicate [super] Refactor the Auth System

Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including #787.

Two man reasons for the changes are:

  • We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer.
  • Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able.

The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package).

  1. The first thing the is_authorized method is a hook to a plugin

which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!)

Reason: authorization can be completely delegated to another system (or partially)

  1. is_authorized method currently takes (username, action, object)

but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model.

  1. Rename these two classes to better reflect what they are
  1. Rename the Editor role to PriveledgeUser? since Editors sometimes can't edit.

Although this sounds a bit radical we already have auth extensions.

Read-only CKAN Web UI

(Additional requirement from #764)

Whilse using CKAN web interface, you are not tempted to edit stuff:

  • You know at all times this CKAN is read-only
  • All editing facilities are still seen but greyed-out with an indication why it is.
1303117973000000 1311173649000000
#961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

  1. Generating (or just filling) a form template with 'form data' (and errors)
  2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
  3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

  • #926 - Pick a simpler form framework
  • #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
    • #1079 Refactor API to use new logic layer and dictization
    • #1078 Refactor WUI controllers and forms to use logic layer
    • cf existing dumper and importer code
    • This will fix #662
  • [not ticketed yet] - validation layer (should work on serialized objects?)
  • #662 - Can't put entity that is returned by posting to package register (Defect)
  • #972 - Merge 'extras' into main package dict rather than having separate key
  • #1035 - Form impressions are given IDs
  • #810 - Move "add packages" field up in group form (easier to do this once forms are done)
1297069849000000 1310126100000000
#2204 enhancement ross rgrp ckan-sprint-2012-04-30 closed fixed [super] Related (Stuff) Extension

This is a reworking of the existing apps extension.

Initial proposal at http://wiki.ckan.org/Proposals#Apps_in_CKAN and http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN

Naming

What do we call this extension?

  • related
    • RP best IMO but perhaps too close to separate relationships concept?
  • relatedstuff
  • links
  • references

Proposal

"Related Stuff": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension)

Implementation

New table named Related with following structure

|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp)
  • type = Idea | App | API | Visualization | Post | Paper | News Article
    • Suggest we make this a ckan.ini config option (comma separated ...?)
      • Do we want the possibility of different templates for different types of Related objects?
  • image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url.
  • owner_id = user_id or creating user (see authorization below)

Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off.

|| id || dataset_id || related_id || status

status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related.

Url

  • /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified}
    • If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id}
  • /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them.

/dataset/{id}/related <- list

  • use image_url for small icon etc, title description (shortened?)
  • Click through to full related item (optional)
  • dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} )
  • related owner sees an edit button / icon (pops up modal)

/dataset/{id}/related/{id} (optional)

/dataset/{id} will have a Related tab (with bubble with count).

  • Drop down with Add Related -> Pop-up modal and save via API

(Not used: /related/add with dataset prefilled ... )

/related/{id}/edit

Authorization

Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y).

Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete.

NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1.

Tasks and estimates {7.5d}

[x] Model + Migration for Related table. {0.75d}

[x] Controller for Related (or relevant sections in Package controller). {0.75d}

[x] Routing setup. {0.25d}

[x] Schema for related. {0.5d}

[x] Logic layer actions. {1.0d}

[x] Auth (default + publisher). {0.5d}

[x] Templates + Dataset changes (new tab etc). {1.0d}

[x] JS Application for adding Related objects in a modal. {1.0d}

[x] Testing. {0.75d}

[x] Dataset owner disabling of Related (via M2M table). {0.5d}

  • Updated to allow author of related to delete as well

[x] Documentation. {0.25d}

1330862992000000 1337594723000000
#1397 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Resource archiving

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

  • Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports

Work Items

  • Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
  • Generate archiving request on resource url change [0.25d][0.25d] - #1399
  • Make site user account.
  • Make entry point system for celery config
  • Archiver daemon #891
    1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
    2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
  • Write to resource and task status table.[1d][0.75d]
  • [Required?] Make archived data available in WUI - #892
  • Documentation - #1400
1318529593000000 1338202920000000
#1032 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Resources in WUI

Add resources into Web User Interface.

  • Locate at: /dataset/{dataset}/resource/{id}
  • CRUD
  • Authorization

Done:

  • #945 - Richer resources - Resource Groups, new fields, improved UI
  • #1445 - Resource View page in WUI
  • #1450 - Dataset view pages to match

Moved to superticket #1506:

  • #978 - Edit Resource Extras in Web UI
1299782021000000 1330348463000000
#945 enhancement kindly kindly ckan-v1.6 closed fixed [super] Richer resources - Resource Groups, new fields, improved UI

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

  • New resource group table. #956
  • New kind field in resource. #957
  • UI for new kind field. #958
  • Resources in REST API ticket:358
  • Resources in WUI #1445
  • Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

1296475283000000 1325259350000000
Note: See TracReports for help on using and creating reports.