{22} Trac tickets (2647 matches)

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

• psql DB with UTF-8 tables

• psql search backend

• ckan version: 1.3.1b

Example:

​http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

1. Need a 'data proxy' - see ticket:698 and ticket:888
2. Preview using javascript in WUI - do this in an extension ​http://bitbucket.org/okfn/ckanext-dataapi

Extra fields should be represented in some manner.

• Should license go in the bottom line?
• Formats should be styled as in CKAN

• Create a list of valid predicates for format information (accessURL et al)
• Execute #235 via Google Spreadsheets and include in DCat enrichment

We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG.

1. Edit any package
2. Hit 'preview'

Notice that the preview at the bottom doesn't show what tags or groups the package is in.

This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview.

One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision.

​http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed.

It should be purged !!!

Why?

• Will make it easier to give resources their own form, independent of packages.
• Makes uploading, harvesting and previewing data cleaner as its separate from the packages.
• Gives the option of sharing a resource between packages.
• Leads the way to supporting storing richer semantic information against them.

Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed?

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

• (Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)
• Document caching system - ticket:841
• Howto guide for writing an extension - started on this at ​http://wiki.ckan.net/Extending_CKAN

Miscellaneous

• Contributors / credits page: ticket:928
• (Data proxy documentation: move it to packages.python.org/dataproxy)
• (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
• Document license configuration and license system - ticket:973
• Refactor trac frontpage moving general feature overview to ckan.org
• Create proper roadmap information on ​http://trac.ckan.org/
• Add link to wiki.ckan.net to ckan.org

I'm using v 1.3.1a and get a lot of "WebApp? Error: <type 'exceptions.Exception'>: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>" error messages.

Module ckan.forms.package:87 in build_package_form
<<      # Options/settings
           builder.set_field_option('name', 'validate', package_name_validator)
           builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
           builder.set_field_option('state', 'dropdown', {'options':model.State.all})
           builder.set_field_option('notes', 'textarea', {'size':'60x15'})
>>  builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
Module ckan.model.package:283 in get_license_options
<<      @classmethod
           def get_license_options(self):
               register = self.get_license_register()
               return [(l.title, l.id) for l in register.values()]
>>  register = self.get_license_register()
Module ckan.model.package:278 in get_license_register
<<      def get_license_register(self):
               if not hasattr(self, '_license_register'):
                   self._license_register = LicenseRegister()
               return self._license_register
>>  self._license_register = LicenseRegister()
Module ckan.model.license:46 in __init__
<<              from licenses.service import LicensesService2
                   self.service = LicensesService2(group_url)
                   entity_list = self.service.get_licenses()
               else:
                   from licenses import Licenses
>>  entity_list = self.service.get_licenses()
Module licenses.service:44 in get_licenses
<<          except Exception, inst:
                   msg = "Couldn't connect to licenses service: %s" % inst
                   raise Exception, msg
               try:
                   licenses = loads(response_body)
>>  raise Exception, msg
Exception: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>

ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically.

use pylint

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

• Stale pyc file
• Threading issue
• Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

• Use the existing support for this feature in google analytics or piwiki
• For google analytics see:
  • Integrate to record: ​http://www.google.com/support/analytics/bin/answer.py?answer=55529
  • Accessing analytics data via API (for showing download counts): ​http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html
• Limitations only record downloads by browsers with js turned on.
  • This is OK I think (in any case machine download via e.g. datapkg does not get downloaded)

Old Spec (do it in CKAN)

• Record info of form: resource id (or url?), timestamp
• Do this via javascript capturing of onclick event talking to an api
• API: /api/resource/{id}/download
  • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
  • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

• Do we record ip addresses (to handle de-botting etc)?
• Do we count preview clicks as well?

On ​http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

• Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
• FAQ/instructions page (use the new wiki?)

Depends

• ticket:938

Estimate

• Cost: 1h

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

Set up a Mediawiki to contain info on CKAN.net (community) conventions etc.

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

• New resource group table. #956
• New kind field in resource. #957
• UI for new kind field. #958
• Resources in REST API ticket:358
• Resources in WUI #1445
• Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

• ​http://ckan.okfnpad.org/resources - discussion etherpad
  • Contains commented version of James Gardner's main email: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html
• UseCasesResources

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext.

second part of #908

Child tickets:

• #1107 Move package autocomplete from package controller and move to API
• #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

• ​http://wiki.apache.org/solr/SolJSON
• ​http://www.flickr.com/services/api/response.json.html

We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package <o2m> resource_group <o2m> resource

The use cases to this are outlined in this email. ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

Ui for related new field in resource (#957)

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From ​http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

1. Generating (or just filling) a form template with 'form data' (and errors)
2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

• #926 - Pick a simpler form framework
• #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
  • #1079 Refactor API to use new logic layer and dictization
  • #1078 Refactor WUI controllers and forms to use logic layer
  • cf existing dumper and importer code
  • This will fix #662
• [not ticketed yet] - validation layer (should work on serialized objects?)
• #662 - Can't put entity that is returned by posting to package register (Defect)
• #972 - Merge 'extras' into main package dict rather than having separate key
• #1035 - Form impressions are given IDs
• #810 - Move "add packages" field up in group form (easier to do this once forms are done)

Package CKAN as a debian package so it is easier to install.

In order to test migrations properly, we need to test the upgraded database against *all* the tests.

To do this a special flag to nosetests should be added. nosetests --migrations

This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db.

Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API.

Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras.

Issues

• Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being)

Possible subticket of forms refactoring: #961

Configuration option 'ckan.site_description' isn't documented

Part of super-ticket #1506.

Follows on from #826.

We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis.

This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up.

(should leave priviledges alone)

• data.london.gov.uk (1d, ping DERI)
• catalogues from Paris and Vienna (1d)

Others?

• digitaliser.dk (1d)
• data-publica.com (2d)
• data.suomi.fi (2d)
• dati.piedmonte.it (1d)
• portalu.de (2d)

We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include:

• CSW/WFS for INSPIRE/UKLII (yields CKAN packages)
• Catalogue scraping for LOD2 experiments (yields RDF graphs)
• Atom/DCat for LOD2 production (yields RDF graphs)
• OAI-PMH for ​http://datadryad.org/ and other dspace (yields CKAN packages)

We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios.

In general, such a system would have the following stages:

• Source selection: find what to download/scrape/harvest/parse
• Index retrieval (i.e. package index)
• Item retrieval (i.e. package entity)
• (Optional: Serialization)
• Normalisation
• Loading/Merging? into CKAN

Exisiting harvesters are at:

• CSW: ​https://bitbucket.org/okfn/ckanext-csw/src/
• Scraper+CKAN: ​https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/

It's not possible to use umlauts (like "CKAN Österreich") in the title / text.

At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache.

We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available.

Cost: 10m

It is too easy to make a mistake with the migrations at the moment. A more systematic way of doing them is needed and this needs to be documented.

This will involve running a sample of real requests synchronously against real data.

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

• lib/base.py: c.time_call_started
• ....

This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed).

Need instructions on the group page to tell people they need to login to create a group. Someone must have deleted this.

Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time).

Cost: 10m (giving high priority because of low cost)

(Assigning to dread as he has been managing the stable branch).

We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer.

We should also have page fragment caching in Genshi, which is not currently implmented.

cf. ​http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html

I have a requirement to allow an external source to define the groups of which a user is a member.

I propose to create an IAuthzUserGroups plugin interface that allows an extension to arbitrarily extend the list of AuthzGroups? that a user is in.