{22} Trac tickets (2647 matches)

Results (601 - 700 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1039 enhancement dread dread ckan-v1.4 closed fixed Default user roles read from config

(instead of being hard coded)

1300212822000000 1300212856000000
#150 enhancement dread dread v0.11 closed fixed Loader for esw data uses REST API

(instead of sqlalchemy interface, to facilitate general usage of the loader)

  • use ckanclient

Follows on from ticket:148 and ticket:149

1255440828000000 1256751974000000
#1552 enhancement dread dread ckan-sprint-2011-12-19 closed fixed Be able to set password on the command line when creating users

(instead of using the prompt) - needed for deployment

1324056178000000 1324056240000000
#983 defect dread dread ckan-v1.3 closed fixed 'db upgrade' creates system priviledges

(should leave priviledges alone)

1297518265000000 1297773407000000
#358 enhancement rgrp dread ckan-v1.5 closed duplicate Resources in REST API

(spun out of ticket:336)

Resource added to model API at:

api/rest/resource

Example model request

GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028

returns:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Authorization

  1. Have it generic (ie. not per resource) and use an action/role on system
  2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages)
  3. Introduce Resource in authorization system (requires migration)

Mixed model

Create / Edit:

if resource associated to package:
    check_permissions(package, update)
else:
    check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System)
1277483282000000 1310128782000000
#2561 enhancement aron.carroll aron.carroll demo phase 1 closed fixed Demo theme edit dataset is missing functionality 1340035457000000 1341497328000000
#408 requirement pudo pudo ckan-v1.2 closed fixed Integrate CMS via proxying
  • As sysadmin/user of CKAN I want to integrate a CMS to support easy addition of additional content ...
  • I want to integrate with navigation
  • As a minimum I want to be able to add content pages (more stuff would be e.g. comments, sidebars, main menu ...)

Why: CMS are great at creating content and have very good editors and theme support ...

Previous work

http://groups.google.com/group/pylons-discuss/browse_thread/thread/f8969e28b0a007c7

  • deliverance: retheming on the fly

http://knowledgeforge.net/shakespeare/hg/file/677eef9402e6/shakespeare/controllers/template.py

  1. Rewrite urls for proxying
  2. You need a theme to substitute into
  3. pylons

Extreme version:

  • 2 sources -> pylons/ckan app, cms
  • 1 theme source: ckan or cms

1 frontend/proxy app which re-themes everything

Owned by CKAN:

  • package/
  • tag/
  • group/
  • api/
  • /

Everything else proxied.

1281348581000000 1281609572000000
#444 task dread closed duplicate Discuss package relationships ideas with JF
  • Create test data on visible ckan
  • Discuss with JF
1282299238000000 1294414008000000
#2355 enhancement toby toby closed fixed [super] make extensions cleaner
  • Do an analysis of why upgrades are problematic
  • What can we do better
  • Decide what to implement
1335885051000000 1338212946000000
#1649 enhancement ross ross ckan-sprint-2012-03-19 closed fixed Verify RDF Extension (ckanext-rdf) works
  • Make sure that the extension works as it should and isn't broken on current releases [3d]
  • Make it work with the EC schema, including extras [2d]
1326709096000000 1338205274000000
#411 requirement pudo pudo ckan-v1.2 closed fixed Deploy and test Varnish on eu3
  • Need to figure out cache invalidation for CKAN
  • Evaluate against an internal cache (i.e. memcached)
1281349275000000 1287393033000000
#443 task dread dread ckan-v1.2 closed fixed Get DGU daily dump working (local)
  • Producing daily dump file
  • Publishing at ckan.net/dumps/ or better
1282299046000000 1282735977000000
#865 defect pudo pudo ckan-v1.3-sprint-1 closed fixed Support external strings in JS
  • Provide and include a "language.js" file.
1291801938000000 1295259773000000
#955 defect dread dread ckan-v1.3 closed fixed Tidy up tests
  • Removal of init_db from every setup etc.
  • Get them working
1296819300000000 1297342534000000
#677 task dread dread ckan-v1.3 closed fixed License change script
  • Uses API to change licenses of all packages in a CKAN instance to a particular one.
  • Run it on DGU.
1286384756000000 1292587315000000
#2437 task icmurray icmurray ckan-v1.8 closed fixed Add coding standards to CKAN docs
  • [ ] Write up python coding standards as discussed at dev-meetup into CKAN docs. (Mostly written-up as part of dev-meetup notes.)
1338196895000000 1343124799000000
#2244 enhancement toby toby ckan-future new clean up auth
  • cache admins etc
  • aim to move check_access so independent of logic function
  • general spruce up
1332177771000000 1338205402000000
#2312 enhancement ross ross ckan-future closed duplicate Analysis of how datasets could belong to users instead of Groups

DUPLICATE OF #2548

Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how.

1334663770000000 1340188765000000
#2925 defect seanh ckan 2.0 new Remove trans mangle paster command?
  • Is trans mangle really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place.
1347530768000000 1347530768000000
#1105 defect nils.toedtmann closed invalid test ticket, please ignore

.

1303508261000000 1303508330000000
#860 defect wwaites wwaites ckan-v1.3 closed fixed full text indexing for semantic.ckan.net

...

1291726029000000 1291726067000000
#693 requirement wwaites ckan-v1.2 closed fixed Support ETag in cache decorator

... and make cache-control header optional

1286831051000000 1286831188000000
#1123 requirement dread nils.toedtmann closed fixed Please re-package CKAN packages as "noarch"

... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too.

Currently, http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise.

1304530050000000 1311863806000000
#1117 defect thejimmyg nils.toedtmann closed invalid Depend deb package "ckan" against ubuntu package "python-pastescript"

... otherwise the scripts fails.

1304089619000000 1304277240000000
#719 story pudo pudo iati-3 closed fixed Improve overall layout of IATI

... probably by adapting aidtransparency.net

1287584092000000 1289910497000000
#1666 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed Add activity stream events for tags

...and show them on the tag pages.

This means adding tag_activity_list() and tag_activity_list_html() logic functions, adding the templating to render tags as HTML, and adding the tag HTML into the tag pages.

What exactly should go into a tag's activity stream? As with datasets we may want a more complex SQL query, where events from objects related to the tag (datasets, users, etc.) appear in the tag's stream.

1326795985000000 1328527113000000
#2241 refactor seanh seanh ckan-sprint-2012-04-02 closed fixed Look into making test_activity.py faster

...but only if it can be done reasonably quickly.

1332172354000000 1332426423000000
#739 requirement thejimmyg johnbywater ckan-v1.3 closed fixed The system shall support retrieving a copy of the harvested metadata for a given package

...obviously, not all packages will have a harvested metadata document.

1287779710000000 1296592842000000
#1173 enhancement amercader amercader pdeu-1 closed fixed Offer a DCat representation for packages in the API

/api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf.

1306939639000000 1307615271000000
#2762 defect seanh ckan-v1.8 new test_related.py crashes

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

1343303753000000 1343303753000000
#178 enhancement rgrp dread v0.11 closed fixed Excel package importer

/tool New 'tools' section of the WUI with 'Excel package import' page.

/tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste.

/tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process.

File format:

First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes:

  • license - name of license
  • tags - space separated
  • groups - space separated
  • anything - for extra key-values

When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete.

User has to log in to use importer. Auth is checked for putting into groups.

1257244034000000 1265305558000000
#1486 defect seanh seanh closed fixed "allow_partial_update" option does not work with resource lists of packages
  1. Call ckan.logic.action.update:package_update()
  2. Pass in a context dict with "allow_partial_update":True
  3. Pass in a partial data dict containing only the package's ID and those fields that you want to update (e.g. title).

=> All of the package's resource will have their statuses changed to "deleted".

The "allow_partial_update" option should apply to the resource lists of packages, but it does not.

1322061318000000 1323172027000000
#2609 enhancement toby aron.carroll opendatasuite 1 closed fixed Demo theme errors when creating a dataset without a resource
  1. Complete step one of add dataset
  2. Click next without adding any resource data.

Page errors.

URL: http://localhost:5000/dataset/new_resource/asdasd
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond
  app_iter = self.application(environ, detect_start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
  return request.get_response(self.app)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__
  response = request.get_response(self.app)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__
  return self.wrap_app(environ, session_start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__
  response = self.app(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 495 in new_resource
  get_action('resource_create')(context, data)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/logic/action/create.py', line 240 in resource_create
  raise ValidationError(errors, _error_summary(errors))
ValidationError: {u'URL': u'Missing value'}

http://localhost:5000/dataset/new_resource/asdasd

1340810211000000 1340900271000000
#823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed.
  1. Create a package.
  2. Edit package and add a resource.
  3. View package in browser - no sign of the resource - ERROR.
  4. View package in API - resource is there.

Reason: hash doesn't change because package.revision only takes account of core fields.

1290505078000000 1290506116000000
#919 enhancement dread dread ckan-v1.4-sprint-6 closed fixed Package preview contains API & datapkg instructions
  1. Edit a package
  2. Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

1295346730000000 1303202627000000
#918 defect memespring dread closed invalid Package preview doesn't show tags or groups
  1. Edit any package
  2. Hit 'preview'

Notice that the preview at the bottom doesn't show what tags or groups the package is in.

This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview.

One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision.

1295346496000000 1315911507000000
#923 defect rgrp dread closed worksforme Search box doesn't work in leaderboard page
  1. Go to: http://ckan.net/stats/leaderboard#content
  2. In the far top-right of the browser, select the search box in 'water'.
  3. Press enter to search. Nothing happens.

Tried in: chrome, firefox

1295867328000000 1340632144000000
#2726 enhancement toby shevski demo phase 5 new confusing logic on data preview formats
  1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this
  1. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28
1342949927000000 1343030906000000
#315 defect dread rgrp v1.1 closed fixed Improvements and fixes to csv dump
  1. Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html
  1. Issues with package resource serialization into csv table.

<quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>

1274100887000000 1275846764000000
#66 enhancement rgrp rgrp v0.10 closed fixed Improve user account UI
  1. Link at top should say "Your Account" when you are logged in
    • when not logged in should say: "Log in via openid"
    • Open Id info on /account/ should move to /account/login/ or be deleted
    • Going to /account/ when not logged in should redirect to /account/login/
  1. When logged in the basic account index page should not give generic information but should reflect fact you are logged in by:
  • Saying something like: "You are logged in as: ..."
  • Showing your apikey or a link to apikey
  • Giving you a logout link

We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets.

1245263685000000 1250785405000000
#60 enhancement rgrp rgrp v0.9 closed fixed Front page usability improvements
  1. List in sidebar 3 latest created/updated packages.
  1. Put package search box on front page (might also want to move some of front page content into an about page).
  1. Similarly might want package creation box on front page.
1239094266000000 1246434921000000
#1717 enhancement shevski ckan-backlog new [super] Search UX improvements
  1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)
  1. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).
  1. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.
  1. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu
  1. Blue search button should be displayed in line with the search bar, not underneath
  1. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name
1327603981000000 1330088539000000
#2724 enhancement toby shevski demo phase 3 closed fixed Inconsistent labelling / UX on draft datasets
  1. One of the datasets here: http://s031.okserver.org:2375/en/user/shevski has [DRAFT] after it. It's not clear why this has been added -> user confusion
  2. The draft dataset is not clickable in the dataset list
  3. The ff dataset (same as the ff [DRAFT] dataset) in the Activity Steam section a) is clickable and b) does not have [DRAFT] after it
1342949402000000 1343319009000000
#1194 defect dread closed fixed "Welcome back" message for newly registered user
  1. Register a new user (/user/register)
  2. It redirects you to user page and displays "Welcome back" even though you are a new user.
1308563392000000 1317053688000000
#2722 defect aron.carroll shevski demo phase 3 closed fixed Ordering or search of datasets of a group brings up 'no results'
  1. go to a group, e.g. http://s031.okserver.org:2375/group/data-explorer
  2. Order by anything, e.g. choose 'popular' and click go
  3. "sorry no datasets found" message appears instead of ordering the group datasets in wanted order

http://s031.okserver.org:2375/group/data-explorer?q=&sort=views_recent+desc

1342948854000000 1344510010000000
#2856 defect toby shevski demo phase 3 closed fixed server error when clicking on deleted app link
  1. go to http://s031.okserver.org:2375/user/okfn
  2. Click on activity stream
  3. Click on 'Deletion test' related item that's linked to in entries 1 & 2
  4. Server error
1345049684000000 1345107503000000
#2366 enhancement ross amercader ckan-sprint-2012-06-25 closed fixed Remove Authorization Groups

1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8.

1335892563000000 1346316322000000
#852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving

3 (4) use cases: (With and without access to auth keys.)

  1. upload form on ckan (by default local system has auth key for storage)
  2. datapkg upload (client) (by default without key) (plumbing done)
  3. background job archiving downloads (by queue client)

Most important are:

  1. Upload form on ckan.net (with access to keys)
  2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

  • #826 - Resource 'extra' fields
  • #358 - Resources in REST API
  • #235 - Resource format normalization and detection

User upload via Web UI

  • #877 - File upload in WUI
  • #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

  • #853 - plumbing for credential-less rest upload
  • #879 - Storage auth API

Also related:

  • #405 - Retrieval options for package resources

More Info

1291722993000000 1315821628000000
#2814 enhancement shevski markw assigned Demo: upload file behaves oddly
  1. Uploading a file behaves counter-intuitively (I would suggest wrongly).

When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen:

  • the pathname of that file is filled in the box;
  • nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc).

Instead of this,

1344420360000000 1346670381000000
#241 defect rgrp dread v1.0 closed fixed License doesn't preview correctly

4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.

1265625546000000 1270569769000000
#43 enhancement rgrp rgrp v0.10 closed fixed Generic Attributes for Packages

As A

User

I Want To

Add arbitrary named attributes to packages (an attribute being a name, type, value triple).

Details

  • We will do this using a dedicated (versioned) table associated to Package
  • Do we allow multiple attributes of the same name?
    • For the present: No (since we will key by attribute name)
    • Could allow for single attribute but with multiple values using json list ...
  • What types do we allow or do we just rely on JSON to take care of this?

Questions (Original)

  • How complex is this to implement?
  • What would an arbitrary user be able to edit? Possibilities:
    1. 'create new attribute' and setting the value (so name and type would be chosen from predefined list).
    2. 'create', setting of name and value (but not type -- type already set in predefined list)
  • Could just use (machine) tags -- though this could be seen as a bit of a hack.
  • Would solve having to create special file/url attributes (though I think that perhaps file stuff is important enough to merit first class support in the domain model -- though, that said, since one won't want to have a file limit adding unlimited file support is very similar to unlimited attributes of arbitrary type).
1199788109000000 1253709802000000
#27 enhancement johnbywater johnbywater v0.6 closed fixed Register/Update a package via a web-api

As A

User

I want To

Create/register and update a package via a web api.

1152551379000000 1215543933000000
#44 enhancement johnbywater rgrp v0.7 closed fixed Provide RSS/Atom Feed of Repository History

As A

Visitor

I Want To

Get an RSS/Atom Feed of the Repository History to use in my feed reader (or elsewhere).

Details

  • Preference for Atom.
  • should just add parameter to /revision/list/ (or /revision/) to select atom format e.g. ?format=atom.
  • should have a 'days' attribute specifying number of days back to go e.g. &days=30

Cost

Low

1199788457000000 1223390660000000
#42 enhancement rgrp rgrp v0.6 closed fixed In WUI where List of Items to Display is Large Results should be Paged

As A

Visitor

I Want To

View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner.

Details

Main place this issue arises:

  • List of tags
  • List of packages
  • Repository history (/revision)

Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display.

1199787162000000 1204133257000000
#53 enhancement rgrp rgrp v0.8 closed fixed View Information about Package history (i.e. package revisions)

As A

Visitor/User?

I want to

View information about a package history, i.e. what revisions it has.

Details

  • History info should be available at package/history/pkg-name
  • Most basic info is just info from revision (when it occurred, author etc)
  • More sophisticated would be to have diff support to see differences between revisions
1223552139000000 1239133021000000
#49 enhancement rgrp rgrp closed invalid Filter Spam in Changes to CKAN Data

As A

sysadmin

I Want To

Have revisions to the CKAN data filtered in order to reduce the spam in the system.

Details

In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution.

Suggest we follow path of trac: http://trac.edgewall.org/wiki/SpamFilter

Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing').

Main initial plugins would be:

  • regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...)
    • could augment using the badcontent list approach (can find list on e.g. moinmoin)
  • spambayes and/or akismet
1204134691000000 1257244973000000
#54 enhancement johnbywater rgrp v0.7 closed fixed Support dump and load of CKAN data to JSON

As a

Sysadmin

I want to

Dump (serialize) CKAN data to a simple transport format (e.g. JSON) and be able to load it again.

Details

  • Already have simple db dump. However dumping to JSON has various advantages, particularly where changes to the data need to be made upon reloading (e.g. during a migration).
  • Dump should include *all* CKAN data (i.e. all the data in the CKAN db tables)
1223907883000000 1230211256000000
#77 enhancement dread rgrp v0.11 closed fixed Rate packages

As a User or Visitor

I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable.

Rating out of 5. One user or IP, one vote.

package_rating table:

package | user | rating annakarenina | joe.bloggs.openid.com | 4 annakarenina | 154.2.5.47 | 1

1247829193000000 1255176732000000
#253 enhancement dread ckan-backlog assigned Package relationships

Overview

Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

Examples

  1. There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.
  1. In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)
  1. OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.
  1. IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?
  1. Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.

See more examples discussed here: http://trac.ckan.org/ticket/253

Implementation

This is split into four tickets:

No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies).

1266854721000000 1339774726000000
#812 defect cygri ckan-backlog new Package edit form only allows three extra fields

Rationale

The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).

Implementation

A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

1289995010000000 1311176917000000
#1289 CREP dread ckan-backlog closed wontfix Remove 'relationships'

Abstract

Package Relationships have not taken off in the 18 months we've had them in the API. There are some issues with them and we need to spend more time improving them or consider getting rid of them.

The Problem

Original use cases are expressed here: #253 Here are comments about how we could handle these specific examples better:

  1. groups of packages - maybe better with a custom tag?
  2. fragment resources - soon to be covered by 'kind' resource field #957

3&5. derived resource - better to have some sort of resource relationship perhaps?

  1. linked resource - again better to have some sort of resource relationship perhaps?

Outstanding issues needing serious effort to fix:

  • #256 Editing them in Web UI (not done yet)
  • #1288 Package edit/creation can't include 'relationships' field

Specification

Remove relationships from model, API, tests, Web UI. Data migration to remove from db.

Why do it this way

Getting frustrated having problems with the code, when it's not used much. Often asked about what it's for, but rarely used. Seems an overly complicated design.

Backwards Compatibility

n/a

Implementation plan

Deliverables

See Specification

Risks and mitigations

Risk: a customer suddenly wants this, and the new ways to relate resources are not in place yet.

Mitigation: discuss this decision thoroughly to make sure we are confident the use cases are not important. Discuss with team, ckan-discuss and specifically the LOD people who have some related packages on thedatahub.org.

Participants

David Read

Progress

Not yet.

1314206502000000 1317315211000000
#20 enhancement johnbywater johnbywater v0.3 closed fixed Purge a package

As A

Administrator

I want to

Purge a deleted package

Notes

  • this need only be possible via the admin interface
1152550910000000 1185473187000000
#30 enhancement zool rgrp v0.3 closed fixed Tags are added to a package by package owner

As A

Site User

I Want To

Create/Edit/Update/Delete? tags associated with my package

Notes

  • Tags are service-wide and not package specific
  • Tags should not allow spaces (but allow everything else e.g. ':')
  • web interface details
    • should provide a text box where users can add space separated tags (a la delicious).
    • tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background)
    • would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious
1155905753000000 1185472559000000
#56 enhancement rgrp rgrp v0.8 closed fixed Show icons indicating package openness

As A

User or Visitor

I Want To

See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package.

So That

It is easy to see the openness status of a package.

Details

  • Suggested by Liz Turner
  • Openness = open license at present (no easy way to determine whether accessible)
  • Have separate tick for downloadability (i.e. existence of download url)
  • Could do more stuff in future (e.g. money where tag price- exists, etc etc)
1230211123000000 1239018857000000
#9 enhancement somebody johnbywater milestone1 closed fixed Register new package

As a

Authenticated User

I want to

Create/register a package

So that

The package will be listed in the system and visible to users

Notes

  • Full package registration process may be broken down into:
    1. Creating a package by provision of minimal attributes:
      1. name: unique and web-usable
      2. auto-generation of default values such as:
        1. date-registered (today's date)
        2. first owner (one owner should be created corresponding to the authenticated user performing this action)
      3. (?) auto-generate a default release
    2. updating the created package -> ../UseCase12
1152550171000000 1152555104000000
#15 enhancement johnbywater johnbywater closed invalid Create a file

As a

Authenticated user

I want to

Create a file associated with a release

So that

The release has a new file associated with it

Notes

  • Permissions are as for package updating (see ticket:10)
  • file attributes are described in file updating: ticket:16)
    • there are no attributes that are required to be supplied to create a file
1152550770000000 1250619139000000
#12 enhancement johnbywater johnbywater closed invalid Create a release associated with a package

As a

Authenticated user

I want to

Create a release associated with a package

So that

The package has a new release

Notes

  • Permissions are as for updating a package ticket:10
  • for details of release attributes see: updating a release ticket:14
1152550407000000 1204135071000000
#11 enhancement johnbywater johnbywater milestone16 closed fixed Delete package

As a

Authenticated user

I want to

Delete a package

So that

So that the package is no longer displayed for visitors of the site

Notes

  • The permissions for this action are the same as for updating the package details (see ticket:10)
  • When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator
1152550328000000 1157371568000000
#16 enhancement johnbywater rgrp closed invalid Update a file

As a

Authenticated user

I want to

Update a file associated with a release

So that

The file attributes are changed. File attributes are:

  • (internal) id
  • description
  • source: (url usually)

Notes

  • Permissions are as for package updating
1152550796000000 1250619162000000
#14 enhancement johnbywater johnbywater closed invalid Update a release

As a

Authenticated user

I want to

Update a release

So that

The release details are amended. Attributes of a release are:

  • name (may include '.' and '-'
  • nickname
  • date
  • description

Notes

  • permissions: as for updating a package (ticket:10)
1152550712000000 1204136163000000
#10 enhancement johnbywater johnbywater milestone15 closed worksforme Update package

As a

Authenticated user

I want to

Update the attributes of a package and add/remove releases

So that

The package attributes are changed. Package attributes are:

  • A title
  • A description (short or long)
  • Date added
  • Project url
  • package license(s)
  • download url
  • A list of releases with their dates and a summary line
  • a list of owners (referencing authenticated users)

I would also like to be able to add comments

Notes

  • Only an authenticated user is one of owners of a package may update it
  • Any authenticated user may add comments (but visitors may not)
  • can break down into:
    • Update a package by changing simple package attributes (value: 8)
    • Update a package by adding or removing an owner (value: 5)
    • Update a package by adding or removing comments (value: 7)
1152550243000000 1183648150000000
#336 defect dread donovanhide v1.1 closed fixed Resource Search API

As a

CKAN client such as ScraperWiki

I want to

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

Proposed implementation

Add resource search API at:

/api/search/resource

AND resource added to model API at:

api/rest/resource

(see ticket:358)

Functional differences from the ScraperWiki suggested patch:

  • URL is not normalised

  • URLs are not grouped
  • All fields of the resource object are returned, not just the URL
  • Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend 'http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

Example search

POST

{"url": "scraperwiki.com/", "all_fields": 1}

to: /api/2/search/resource

returns JSON:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe

Search Parameters

Key:  q
Description: Search all resource fields for the value

Key: url / description / format / 
Description: Search particular field for the value

Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

Original request

Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?

Donovan

1275411765000000 1279373842000000
#342 enhancement dread dread closed fixed JSONP parameter in API

As a

CKAN client using JQuery

I want to

call the CKAN API and instead of receiving back JSON I get JSONP. i.e. "%s(%s)" % (callback, json_content)

Suggested implementation

All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide:

http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch

Test

import re import unittest

def test_jsonp_callback():

response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com&callback=jsoncallback') match = re.match('jsoncallback\(.*\);',response) self.assertTrue(match)

response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com') match = re.match('jsoncallback\(.*\);',response) self.assertFalse(match)

I think the point needs to be made that JSONP only works for GET requests and not POST/PUT/DELETE, so there needs to be a check for that in the _finish_ok method.

(thanks to Donovan Hide for test)

1276166426000000 1276278485000000
#31 enhancement somebody rgrp closed wontfix Add a comment to a package

As a

Registered user

I want to

Add a comment to a package

Notes

  • The comment should be tagged with the username of the person making the comment
  • Should we allow comments per release as well as per package? No for the time being.
  • Comments will be 'flat' and not threaded (at least for the present)
  • Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)
1157374505000000 1185472236000000
#353 defect dread closed fixed SOLR search indexing

As a

SOLR instance

I want to

keep my search index of CKAN packages up-to-date

Implementation

  • Using asynchronous event notifications
  • Running in a separate process to CKAN
1277123480000000 1280756399000000
#130 enhancement dread rgrp v0.10 closed fixed Create a New Group via the WUI

As a

User

I want to

Create a new group via the WUI

Details

  • locate at /group/new/
  • You must be logged in to create a group
  • Group creator is automatically given role of group admin
  • Edit screen is same as edit screen ...
1254130212000000 1254735501000000
#2 enhancement somebody johnbywater milestone1 closed fixed Register as user on the service

As a

Visitor

I want to

Be able to register as a user on the service

So that

One can login and be an authenticated user and perform the actions that only authenticated users can perform

Notes

  • Registration involves the provisional of a minimal set of user details, namely:
    1. username (unique in lifetime of the service)
    2. fullname
    3. email
    4. password
1152549635000000 1152553157000000
#5 enhancement johnbywater johnbywater v0.3 closed fixed Browse list of packages and select one to view

As a

Visitor

I want to

Browse a list of packages resulting from a search or browse request (see other use cases)

So that

I can select one of the packages to view in more detail (-> viewing an individual package ticket:6)

Notes

  • When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list
  • The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable
1152549884000000 1185473622000000
#4 enhancement somebody johnbywater v0.7 closed fixed Find a package

As a

Visitor

I want to

Locate packages that I am, or might be, interested in

So that

I know what packages exist and can read the information associated with those packages

Notes

Find a package by searching by (or some combination thereof):

  1. tag
  2. string contained in title or description
  3. license
1152549820000000 1223908381000000
#3 enhancement somebody johnbywater milestone1 closed fixed Login to the service

As a

Visitor

I want to

Login to the service

So that

I am now an authenticated user and may perform the actions that only authenticated users may perform

Notes

  • Login by providing username and password
  • Login screen should also include a link or information about registration
  • If username do not match (or the username does not exist) a login failure is noted and the visitor is re-presented with the login screen as well as information on how to register
1152549717000000 1152553322000000
#33 enhancement rgrp rgrp v0.5 closed fixed List and search tags

As a

Visitor

I want to

View a list of the tags on the system and search for a tag by name

Notes

  • As for a search of packages should search for tags that contains a given search string (or could use a glob approach allowing wildcards).
1157375983000000 1199788094000000
#32 enhancement rgrp rgrp v0.3 closed fixed View a tag

As a

Visitor

I want to

View a tag's page which should list all packages associated with that tag (or link to a page containing that list).

Notes

  • The tags should have a nice primary url such as /tag/<tag-name> (could also use the plural i.e. /tag/<tag-name>)
1157375968000000 1185472746000000
#6 enhancement somebody johnbywater milestone1 closed fixed View a package

As a

Visitor

I want to

View the details of an individual package

So that

I know more about the package. Attributes available are detailed in updating a package (../UseCase12)

1152549960000000 1183648175000000
#8 enhancement johnbywater johnbywater closed invalid View file

As a

Visitor

I want to

View the files associated with a package.

Notes

  • See also ticket:16 (updating a file).
  • Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release)

1152550057000000 1250619147000000
#204 enhancement dread dread v0.11 closed fixed Custom package input form

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

  • all basic ckan fields
  • default license to OKD::Crown Copyright
  • co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

  • date_released
  • date_updated
  • temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

  • department

Selection or free text

Editable combo

  • update_frequency - selections: never, weekly, monthly, annually, discontinued
  • geographic_granularity - selections: national, regional, local authority, ward, point
  • temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

  • geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

  • national_statisic - UI is check box, store as 'Yes' or 'No'
  • resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.
  • categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

  • tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

1260188965000000 1285082169000000
#197 enhancement dread dread v0.11 closed fixed Simple dumper

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

  • Just do JSON & CSV for now.
  • Put this in CLI.
  • Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

1258972495000000 1258989700000000
#335 enhancement dread dread v1.1 closed fixed Post-package-edit redirect to configurable URL

As a

third-party interface to a CKAN instance

I want to

link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return.

Design

  1. The 'return URL' is passed as a parameter to CKAN.
  2. CKAN substitutes the package name into the return URL.

Example

Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded)

When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data

1275408834000000 1276179605000000
#407 requirement dread closed wontfix Link to Scraperwiki

As a

user browsing a CKAN package which has a resource in a bad format

I want to

be alerted to the possibility of scraping it on scraperwiki

Implementation

This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it
Image: (Scraperwiki dumper truck)
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can request this data is scraped or even write the scraper yourself.

1281348368000000 1330769956000000
#352 enhancement dread dread closed wontfix Package notification worker - sends XML-RPC

As an

external front-end

I want to

be notified (by XML-RPC) about package creations and updates.

Implementation

  1. A message queue worker waits for package update notifications
  2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI.

Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter.

Config - in the CKAN config will be:

  • URI to callback to
  • API version to use (version 2 gives packages referred by ID not name)
1276597996000000 1286375870000000
#362 defect dread dread closed invalid Ratings should not be created with a GET

Background

In the Web UI, when you rate a package it simply links to something like:

http://ckan.net/package/rate/mke-liquor-licenses?rating=3

This creates a GET request.

This is bad because:

  • Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
  • There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
  • Best practise for web requests is for GET to be a read-only request.
1278925451000000 1311176564000000
#1293 enhancement zephod rgrp ckan-sprint-2011-09-12 closed fixed Rename Package to Dataset throughout WUI, API

Changes

  • All templates
  • [DONE] Routing: /package/ -> /dataset/ (including API)
  • [DONE] Tests: self.app.get('/dataset/...') rather than package ...
    • One test to check redirect works?
  • Docs

Leave

  • Stay with Package in code e.g. stay with 'PackageController?'
    • also leave stuff like pkg = .... (gradual migration)
  • Leave extensions

Notes

  • Package -> Dataset
  • Data Package -> Dataset
  • respect capitalization
1314405700000000 1317649968000000
#1506 enhancement zephod rgrp ckan-sprint-2012-03-05 closed fixed [super] UX improvements to dataset edit, create and search + group + user page (Dec 2011)

Historical Parent Tickets

Step Zero

  • COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

  • Force format field to be lowercase
    • Modify backend to serve it up in that format
  • Subticket #978: Add extra fields to resources in Web UI
  • Fix display of large datasets (http://test.ckan.net/dataset/europeana-lod)
  • When submitting forms with errors, open the correct resource pane.
  • No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
  • Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

  • Subticket #1648: Clarify edit terminology
  • Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
  • Fix (i.e. allow / show) name editing on dataset edit.
    • Fix the slugify so it does not overwrite if i change title after fixing name.
    • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
  • Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

  • Fix animation glitch in expanding description
  • Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
  • Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

  • "Upload or link data files, APIs and other materials related to your dataset."
  • Include the description markdown editor again (keep it small)
  • Licence should maybe be restored
    • "(Don't worry if you don't know what your licence is)"
  • Show the log message field
    • Beautify
  • Pre-populate the log message field?
  • ? Show Home Page field when creating dataset [WONTFIX don't push it]
  • ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

  • Improvements to user page UX - 1d - #1396
  • Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

  • Take before and after screenshots
1323088306000000 1330702859000000
#25 enhancement johnbywater johnbywater v0.6 closed fixed Autocompletion (and suggestion) of package tags when adding tags to a package

Notes

  • Currently have a text box where users can add space separated tags
  • Tags should auto-complete in a manner similar to delicious
  • would also be good to have a list of suggested tags. This could either be:
    • [simpler] a list below the tag box chosen by existing popularity or
    • [more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far).
    • For first attempt to should probably just go for the simpler option.
1152551313000000 1204131522000000
#1077 enhancement kindly rgrp ckan-backlog new Move to simpler vdm system

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

  • The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset

for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

  • does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

  • Easier support for pending state and similar behaviour
  • No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

  • Migration is required
  • More difficult to query revision history.
    • Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
  • Performance (?)
    • Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
      • Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

1302304464000000 1340034345000000
#2500 defect ross icmurray ckan-v1.8 closed fixed get_action should raise an exception if the action requested doesn't exist

Original bug report:

Hi CKAN devs,

Can I request that ckan.logic.get_action raises an exception if the action requested doesn't exist?

Two of us had a frustrating time tracking down an exception this morning. It's difficult when the exception is raised in python in a template because you don't get a line number. The problem was caused by a new logic function being added to another extension, and we hadn't updated the code there. So if get_action raised an proper exception rather than silently returning None then I think we'd have found the problem quicker.

I'd write the code for this, but as I understand that the new system would require a ticket, branch, pull request, code review, merge. It seems a lot to do for such a small thing.

Dave

1339072580000000 1340293636000000
#1076 enhancement johnlawrenceaspden rgrp ckan-v1.4 closed fixed Improve revision and package purge system

Purging Revisions

  • Delete button displayed on:
    • /revision/list
    • (/package/history)
      • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
    • /revision/{id}
  • Delete button submits to delete action on revision and changes revision state to 'deleted'.
    • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
  • Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

  • Put into deleted state.
  • Listed on /ckan-admin/trash
  • Separate Empty trash button which deletes all associated revisions.
    • Should be separate from Empty trash for revisions

Current system

  • Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)
1302283442000000 1303236302000000
#2322 enhancement ross rgrp ckan-sprint-2012-05-29 closed fixed Convenience offset on end of resource urls: /raw, /api, /viewer

Raw

/raw or /download (latter may be better)

/dataset/{id}/resource/{resource-id}/raw

=> redirects to resource url (download)

API

/dataset/{id}/resource/{resource-id}/api[/....]

=> either redirects or directly serves /api/data/{resource-id}

Viewer (Explorer?)

Do data viewer but without anything else showing (full-screen?). this would not be just the data explorer but other cases too (including iframe etc ...)

/dataset/{id}/resource/{resource-id}/viewer

Resource (pre)-viewer without anything else (perhaps include resource title)

Assigning for kindly for review.

These would take about 20m to implement and would be *very* convenient from my experience of using the data API.


Implementation at enhancement-2322-convenience-urls

1335236581000000 1344086640000000
#329 defect rgrp dread v1.1 closed fixed Bad dates cause exception on Gov form

Reproduction

Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message.

Affects all versions of CKAN.

Why it's happening

The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught.

1275060617000000 1275079189000000
#2550 enhancement icmurray ross datahub-july assigned User types

Requirements

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

Required interface

Once changes have been made to the user schema, for a given user we want to be able to:

  • determine if they have a paid or a free account, and
  • get a string name of the type of paid account.

Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

User Stories

User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

  • As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely.
  • As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have.
  • As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else.
  • As a user, and only if I have one, I'd like to see my current payment level on my user profile page.

Tasks

[x] Tests

[x] Plugin based migration

[x] Code

[x] Model

[x] API

[x] Documentation

Estimates

1340017590000000 1346669497000000
#2552 enhancement ross ckan-future assigned Controlling access to features

Requirements

To provide a freemium service it is necessary to be able to provide differing levels of functionality based on the type of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks.

Initial implementation should focus on limiting access to datastore disk space.

Interface

These changes are currently only for the data hub and should be kept as much as possible within the data hub extension.

User Stories

  • As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited).
  • As a system administrator I don't expect to need to manage the levels of users or the features that this applies to.

Tasks

[ ] Clarification of requirements/analysis

[ ] Tests

[ ] Code

[ ] Model

[ ] API

[ ] UI

[ ] Documentation

Estimates

1340018770000000 1346669544000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Note: See TracReports for help on using and creating reports.