{22} Trac tickets (2647 matches)

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

• New resource group table. #956
• New kind field in resource. #957
• UI for new kind field. #958
• Resources in REST API ticket:358
• Resources in WUI #1445
• Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

• ​http://ckan.okfnpad.org/resources - discussion etherpad
  • Contains commented version of James Gardner's main email: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
• UseCasesResources

Set up a Mediawiki to contain info on CKAN.net (community) conventions etc.

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

On ​http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

• Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
• FAQ/instructions page (use the new wiki?)

Depends

• ticket:938

Estimate

• Cost: 1h

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

• Use the existing support for this feature in google analytics or piwiki
• For google analytics see:
  • Integrate to record: ​http://www.google.com/support/analytics/bin/answer.py?answer=55529
  • Accessing analytics data via API (for showing download counts): ​http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
• Limitations only record downloads by browsers with js turned on.
  • This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

Old Spec (do it in CKAN)

• Record info of form: resource id (or url?), timestamp
• Do this via javascript capturing of onclick event talking to an api
• API: /api/resource/{id}/download
  • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

• Do we record ip addresses (to handle de-botting etc)?
• Do we count preview clicks as well?

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

• Stale pyc file
• Threading issue
• Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

use pylint

ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically.

I'm using v 1.3.1a and get a lot of "WebApp? Error: <type 'exceptions.Exception'>: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>" error messages.

Module ckan.forms.package:87 in build_package_form
<<      # Options/settings
           builder.set_field_option('name', 'validate', package_name_validator)
           builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
           builder.set_field_option('state', 'dropdown', {'options':model.State.all})
           builder.set_field_option('notes', 'textarea', {'size':'60x15'})
>>  builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
Module ckan.model.package:283 in get_license_options
<<      @classmethod
           def get_license_options(self):
               register = self.get_license_register()
               return [(l.title, l.id) for l in register.values()]
>>  register = self.get_license_register()
Module ckan.model.package:278 in get_license_register
<<      def get_license_register(self):
               if not hasattr(self, '_license_register'):
                   self._license_register = LicenseRegister()
               return self._license_register
>>  self._license_register = LicenseRegister()
Module ckan.model.license:46 in __init__
<<              from licenses.service import LicensesService2
                   self.service = LicensesService2(group_url)
                   entity_list = self.service.get_licenses()
               else:
                   from licenses import Licenses
>>  entity_list = self.service.get_licenses()
Module licenses.service:44 in get_licenses
<<          except Exception, inst:
                   msg = "Couldn't connect to licenses service: %s" % inst
                   raise Exception, msg
               try:
                   licenses = loads(response_body)
>>  raise Exception, msg
Exception: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

• (Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)
• Document caching system - ticket:841
• Howto guide for writing an extension - started on this at ​http://wiki.ckan.net/Extending_CKAN

Miscellaneous

• Contributors / credits page: ticket:928
• (Data proxy documentation: move it to packages.python.org/dataproxy)
• (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
• Document license configuration and license system - ticket:973
• Refactor trac frontpage moving general feature overview to ckan.org
• Create proper roadmap information on ​http://trac.ckan.org/
• Add link to wiki.ckan.net to ckan.org

Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed?

1. Go to: ​http://ckan.net/stats/leaderboard#content
2. In the far top-right of the browser, select the search box in 'water'.
3. Press enter to search. Nothing happens.

Tried in: chrome, firefox

Refactor harvesting code to simplify.

1. Edit a package
2. Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in ​https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

• Most of the time nothing will change. A commit implies flush.
• You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
• A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
• You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

We should present something there, why not start with a meta-search.

• Create a list of all valid predicates where licenses are stored.
• Create a license normalization registry in GSpread
• Fill spreadsheet from OKD/OSI lists.

Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported:

• CKANrdf generated exports
• opengov.se RDF (not really DCat)
• Sunlight Nationdatacatalog as harvested by dcat-tools.

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

1. Need a 'data proxy' - see ticket:698 and ticket:888
2. Preview using javascript in WUI - do this in an extension ​http://bitbucket.org/okfn/ckanext-dataapi

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

• psql DB with UTF-8 tables

• psql search backend

• ckan version: 1.3.1b

Example:

​http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. ​https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g.

mycssfile.css?v=12345678

At the moment, ExtrasField? cannot be used in conjunction with more specific extra fields, such as TextExtraField? or SuggestTextExtraField?.

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

• Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
• Should we have clever ways to dump APIs? ANS: no.
• Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
• Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
  • Complete support for ETags
  • Expires, Max-Age etc.
• Check

Functionality

• Download files via HTTP, HTTPS (will not do FTP)

Process:

1. [Archiver.Update checks queue (automated as part of celery)]
2. Open url and get any info from resource on cache / content-length etc
   1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
   2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
   • Add cache url to resource and updated date
   • Add other relevant info to resource such as md5, content-type etc
4. Update task_status

Optional functionality

• If result object is HTML, search for references to "proper data" (CSV download pages etc.)
• Download from POST forms (accepting licenses or weird proprietary systems)
• Support running on Google Apps Engine to save traffic costs.

Existing work

• ​https://bitbucket.org/okfn/ckanext-qa/overview
• out of date: ​https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental.
• out of date: ​https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc

Add a config option 'template_footer_end' which is inserted in layout_template just before </body>.

This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics.

Aside: going forward may want to turn this into a extension.

Cost: 1h

in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat

master ticket

• #884 implement harvest job delete command for an easier life
• #885 move csw code into dedicated ckanext plugin
• #886 move forms api into dgu extension module
• #887 move harvesting code to use generic harvesting module

I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

• Implement as an Extension
• Locate at /upload
• Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
• Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
• This wants to be tied in to general package workflow. See parent ticket:852.

Consolidate search API interface (and backend) on solr (solrpy) type interface.

• Support for standard query structure
• Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

1. Disable (happens automatically)
2. Implement - suggest using group by etc

Extras

• Front-page tag cloud: change this to use facets
  • Accept this means that if facets not functional in backend we have no tag cloud

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

The infamous ​exim bug only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers.

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like ​Rootkit hunter on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark? has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could.

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.

• Provide and include a "language.js" file.

e.g. license, owner etc

​http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

Test coverage for ckan.lib.cli is 34%. Need to improve that.

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

• master ticket #852

revisit link checker from ​http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py

revisit ollyc's parallel work

look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)

The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import.