{22} Trac tickets (2647 matches)

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

• psql DB with UTF-8 tables

• psql search backend

• ckan version: 1.3.1b

Example:

​http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

1. Need a 'data proxy' - see ticket:698 and ticket:888
2. Preview using javascript in WUI - do this in an extension ​http://bitbucket.org/okfn/ckanext-dataapi

Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported:

• CKANrdf generated exports
• opengov.se RDF (not really DCat)
• Sunlight Nationdatacatalog as harvested by dcat-tools.

• Create a list of all valid predicates where licenses are stored.
• Create a license normalization registry in GSpread
• Fill spreadsheet from OKD/OSI lists.

We should present something there, why not start with a meta-search.

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in ​https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

• Most of the time nothing will change. A commit implies flush.
• You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
• A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
• You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

1. Edit a package
2. Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

Refactor harvesting code to simplify.

1. Go to: ​http://ckan.net/stats/leaderboard#content
2. In the far top-right of the browser, select the search box in 'water'.
3. Press enter to search. Nothing happens.

Tried in: chrome, firefox

Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed?

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

• (Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)
• Document caching system - ticket:841
• Howto guide for writing an extension - started on this at ​http://wiki.ckan.net/Extending_CKAN

Miscellaneous

• Contributors / credits page: ticket:928
• (Data proxy documentation: move it to packages.python.org/dataproxy)
• (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
• Document license configuration and license system - ticket:973
• Refactor trac frontpage moving general feature overview to ckan.org
• Create proper roadmap information on ​http://trac.ckan.org/
• Add link to wiki.ckan.net to ckan.org

I'm using v 1.3.1a and get a lot of "WebApp? Error: <type 'exceptions.Exception'>: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>" error messages.

Module ckan.forms.package:87 in build_package_form
<<      # Options/settings
           builder.set_field_option('name', 'validate', package_name_validator)
           builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
           builder.set_field_option('state', 'dropdown', {'options':model.State.all})
           builder.set_field_option('notes', 'textarea', {'size':'60x15'})
>>  builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
Module ckan.model.package:283 in get_license_options
<<      @classmethod
           def get_license_options(self):
               register = self.get_license_register()
               return [(l.title, l.id) for l in register.values()]
>>  register = self.get_license_register()
Module ckan.model.package:278 in get_license_register
<<      def get_license_register(self):
               if not hasattr(self, '_license_register'):
                   self._license_register = LicenseRegister()
               return self._license_register
>>  self._license_register = LicenseRegister()
Module ckan.model.license:46 in __init__
<<              from licenses.service import LicensesService2
                   self.service = LicensesService2(group_url)
                   entity_list = self.service.get_licenses()
               else:
                   from licenses import Licenses
>>  entity_list = self.service.get_licenses()
Module licenses.service:44 in get_licenses
<<          except Exception, inst:
                   msg = "Couldn't connect to licenses service: %s" % inst
                   raise Exception, msg
               try:
                   licenses = loads(response_body)
>>  raise Exception, msg
Exception: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>

ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically.

use pylint

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

• Stale pyc file
• Threading issue
• Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

• Use the existing support for this feature in google analytics or piwiki
• For google analytics see:
  • Integrate to record: ​http://www.google.com/support/analytics/bin/answer.py?answer=55529
  • Accessing analytics data via API (for showing download counts): ​http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
• Limitations only record downloads by browsers with js turned on.
  • This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

Old Spec (do it in CKAN)

• Record info of form: resource id (or url?), timestamp
• Do this via javascript capturing of onclick event talking to an api
• API: /api/resource/{id}/download
  • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

• Do we record ip addresses (to handle de-botting etc)?
• Do we count preview clicks as well?

On ​http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

• Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
• FAQ/instructions page (use the new wiki?)

Depends

• ticket:938

Estimate

• Cost: 1h

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

Set up a Mediawiki to contain info on CKAN.net (community) conventions etc.

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

• New resource group table. #956
• New kind field in resource. #957
• UI for new kind field. #958
• Resources in REST API ticket:358
• Resources in WUI #1445
• Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

• ​http://ckan.okfnpad.org/resources - discussion etherpad
  • Contains commented version of James Gardner's main email: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
• UseCasesResources

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

based on repoze.who-friendlyForms

I can't add a package (e.g. ​http://ckan.net/package/ub-konstanz) to a group (e.g. ​http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page.

Have a tagline / description are under main title and set if from ckan.site_description config variable.

• Removal of init_db from every setup etc.
• Get them working

The new field was outlined in

​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.

Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way.

Should do this not just here but everywhere we can find in the code base.

Module ckan.controllers.package:302 in edit
<<          am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg)
               if not am_authz:
                   abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
       
               auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg)
>>  abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Estimate

Cost: 1h

Basic implementation done (and deployed):

​https://bitbucket.org/okfn/ckanext-datapreview

However plenty to improve, e.g.

• Support more formats (use external systems for preview?)
  • json (!)
  • html (trivial!)
  • sparql
  • ...
• Do not display preview if no preview

Also suggest reworking to use external services rather than doing preview 'in house' (doing in house places heavy reliance on data proxy service and on converting data to a standard format).

In downloading packages we need to download resources and we need a ResourceDownloader? object for this. these should be pluggable so that we can add support for different types of resources.

I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system.

Document how license system works and specifically how licenses can be configured.

Cost: 2h

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

• db_upgrade (in ckan/model/init) makes call to validate_authorization_setup but forgets about rest of initialisation.

• it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)

Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API.

This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up.

(should leave priviledges alone)

• data.london.gov.uk (1d, ping DERI)
• catalogues from Paris and Vienna (1d)

Others?

• digitaliser.dk (1d)
• data-publica.com (2d)
• data.suomi.fi (2d)
• dati.piedmonte.it (1d)
• portalu.de (2d)

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

• CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
• Catalogue scraping for LOD2 experiments (yields RDF graphs)
• Atom/DCat for LOD2 production (yields RDF graphs)
• OAI-PMH for ​http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

• Source selection: find what to download/scrape/harvest/parse
• Index retrieval (i.e. package index)
• Item retrieval (i.e. package entity)
• (Optional: Serialization)
• Normalisation
• Loading/Merging? into CKAN

Exisiting harvesters are at:

• CSW: ​https://bitbucket.org/okfn/ckanext-csw/src/
• Scraper+CKAN: ​https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/

We need to support extending the model from plugins. This could involve:

• Adding a plugin hook to extend the mapper
• Adding an upgrade hook for plugin schema migrations
• Documenting how this is to be done
• Find a way to avoid conflicts

Form for new package has CheckboxExtraField? checked, when the value is False. (as used in ckanext-dgu package v3 form)

The link for About > 'Project Home Page' on the CKAN.net footer should be changed to point to:

http://ckan.org/

The current link is broken.

Thanks, Jason

Try and minimise the amount of different caching methodologies, especially for api calls.

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

Rather than polluting the ckan core code base on ckan.net install with ckan.net specific changes these adaptations should be moved out into a dedicated ckan.net theme.

When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController? to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author).

A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library.

Why?

• Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core
  • E.g. for specific communities e.g. geodata, science
  • Specialized tasks - multi-package editing
• Very easy deployment and integration (e.g. can drop in to getthedata or other sites)

Package edit form at hmg.ckan.net has no publishers any more.

Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class.

cf. ​https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16