{22} Trac tickets (2647 matches)

Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id

create a new test file ckan/model/test_user.py

add tests for the following three functions in ckan/model/user.py

number_of_edits, number_administered_packages, search

merged in in changeset 0046f83aedcf

It is essential packages have a version attribute to assist in tracking releases etc (and to allow compatibility with other packaging systems e.g. distutils which is being used by datapkg).

cost: 2h

For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page.

We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar.

Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources.

On the organization read page there is only an 'Add Dataset' button, not a drop-down with add new dataset or add existing dataset as on the default group page.

On the organization edit page you do have the drop-down, but if you click add existing dataset nothing happens except it appends #datasets to the URL

To reproduce:

1. paster db init
2. paster create-test-data
3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
4. curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
5. paster search-index rebuild
6. curl http://localhost:5000/api/search/package?groups=tilo results in 1 result

Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception.

this want to

a) be a link on the left of the form like deletes

b) needs a confirm like delete

c) wants to work with the autoform

Is testing complete and ready for release?

Requirements:

• cater naturally for multiple packages with multiple resources
• international encoding
• large number of packages

Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages.

Motivation: each package's Atom feed is mentioned in the <link rel="alternate" type="application/atom+xml" ...> tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it.

The Repo History (/revision) page has a visual link the feed.

PS It would be nice to display the feed icon in both these places too.

Allow 'ignored' fields on versioned objects (i.e. attributes which are not 'versioned'). This is not hard to do as we already have most of the necessary mechanisms set up in the Revisioner object.

Cost: 2h

Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package.

I'm requesting the ability to diff against a "blank slate" initial state of a project, so I can see the content of the first project commit.

Not sure if this is a vdm feature, so I'm putting this ticket in against ckan.

On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually.

Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed)

Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource?

Ira, what are your thoughts?

GET api/search/package?q=

returns all packages. This is so you can filter them e.g. by openness, which is not currently possible.

Options:

1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works)

1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist).

Can do this using:

<xi:include href="base.html"><xi:fallback /></xi:include>

The AlphaPage? implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead.

We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible.

Spam:

• Package creation
• Package edit
• User creation
• Group creation

Systems to consider:

• Automatic
  • captcha
  • bayesian scoring
• Sysadmin
  • a tool to quickly analyse and remove packages, edits and users
• Helpful users
  • button to click indicating spam, leading to any/all of:
    • add TODO item (which emails package admin and sysadmins)
    • quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it

General thoughts:

• We should learn from the wikipedia tools, as they are the experts
• This is a serious problem that is only going to get worse and is a threat to the whole site.

Add api to translation to the term_translation table.

• add to logic 2d
• test 2d

For example:

• name and title should have a higher weight than the notes field

Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

$ paster archiver update -c ../ckan/development.ini 


2012-08-09 11:01:37,636 INFO  [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources)
2012-08-09 11:01:37,671 INFO  [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13
2012-08-09 11:01:37,900 INFO  [ckan.lib.base]  /api/action/package_show render time 0.043 seconds
Traceback (most recent call last):
  File "/home/toby/okfn/pyenv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command
    response = app.post(api_url + '/package_show', data)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post
    expect_errors=expect_errors)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request
    return self.do_request(req, status=status)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request
    self._check_status(status, res)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status
    res.body))
paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show)
{"help": "Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n\n    :rtype: dictionary\n\n    ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}}

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

• Naming scheme?
  • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
    • What happens if names change
  • File: filename? hash?
• Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

• Properly checking for source file modification (Last-modified, Etag)
• Using PIP VCS Backends for retrieval
• OFS/S3 Storage

I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update & their profile & fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates

Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard

DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation

John did this a while ago.

Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h

Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See [1].

[1] ​http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html

Auth Proposal

Use case: We'd like an authenticated and authorized Drupal user to be able to edit/delete packages using the API

To do this a user would need a CKAN API key so we need some way of Drupal telling CKAN who a user is so that they can get their API key.

Proposed Implementation

A user visits the CKAN API key page to get their key. Because CKAN is at catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal cookies.

If there is no DRXtrArgs or DRXtrArgs2 cookie, we know the user isn't signed in so we redirect them to Drupal to sign in.

WISHLIST: It would be really nice if we could pass the URL to redirect back to Drupal so tha Druapl can send them back to the CKAN API key page

Either way, they get back to CKAN and now the cookies exist. When the first HTTP request header is sent, CKAN will read the Drupal session ID and then call a Drupal API, server to server.

TODO: Implement an API on the drupal server which accepts a Drupal session ID as an argument and returns the username and credentials, but only if the request if from the CKAN server (perhaps we specify an API key in the request)

If the session is valid CKAN will set its own auth cookie and show them the page with the API key. CKAN only considers a user signed in if both the CKAN cookie *and* Drupal session ID are present. If at any time they sign out of Drupal the Drupal session disappears so they will be signed out of CKAN too.

Now the user has an API key they can use the standard CKAN command line API tools.

The API key is the only thing the user will need to use the CKAN API. This poses a problem. What if a user is removed from Drupal but still has a CKAN API key?

Two solutions:

1. API keys will be set to only be valid for the length of a Drupal session so that a user will only be able to use the write API for a little time after credentials are revoked

2. CKAN provides an API to allow Drupal to revoke keys

Option 1. seems easier to me. If we choose this we will write a get_api_key command line tool so that Drupal user can write things like this:

::

datapkg list_packages --api_key get_api_key

The get_api_key function will prompt for username and password and then perform the steps necessary to get an API key.

Edit a dataset and add an author, dataset read page still says 'author not given'. If you fill out both the author and author email fields then it works.

allow group-level authorization on all system entities

No authz on:

• Group creation/edit/listing
• Package relationship create/edit/delete

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

Apparently auto-complete in Organizations may not work. Check and fix.

See ​https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5

From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action()

The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource.

This would help improve data quality and consistency.

Extracted from ticket:216.

Dubious of its merit.

Notes

• Currently have a text box where users can add space separated tags
• Tags should auto-complete in a manner similar to delicious
• would also be good to have a list of suggested tags. This could either be:
  • [simpler] a list below the tag box chosen by existing popularity or
  • [more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far).
  • For first attempt to should probably just go for the simpler option.

• Possible sibling / requirement: #1397 (resource archiving)

• Setup test environment. 1d
• Make task to upload files to webstorer. 3d
  • This includes writing back webstore_url and updated date. 0.5d
  • Type guessing of file format. 1d
  • Make cron to upload all un webstored resources. 0.5d
• Testing for errors by running cron 2d
• Make nice way to deploy celery 1d.
• Documentation. 1d
• Deployment. 1d

We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided?

2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object
2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object
2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object
2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object
2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object
2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object

Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class.

cf. ​https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16

Reproduction

Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message.

Affects all versions of CKAN.

Why it's happening

The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught.

A bad action name: e.g. curl ​http://test.ckan.net/api/action/user_showaa -d '{}' causes 500 response.

Occurs only on 1.4.3 beta.

Using datapkg a user can:

• list packages on CKAN
• get info about package on CKAN
• register and update a package on CKAN

Nils flagged up that Ubuntu launches another LTS in April which we should probably move to. This ticket is to get the ball rolling with what we do. We can always include in our package the exact package versions of deps that we want, but it is good to be in line with the release if possible.

Here's how to see the packages included with it: ​http://packages.ubuntu.com/search?keywords=python-pylons&searchon=names&suite=all&section=all

Pylons 0.9.7 -> 1.0-2 Requires a small amount of work SQLAlchemy 0.7.3-2 OK Babel 0.9.6-1 OK Genshi 0.6-2 Perfect

Another alternative is to move to another web framework, especially minding that Pylons is not being further developed.