{22} Trac tickets (2647 matches)

Results (1 - 100 of 2647)

1 2 3 4 5 6 7 8 9 10 11
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#834 task Alexander ckan-v1.3 closed worksforme Searching in CKAN

Hello.

I've installed stable CKAN v1.1 from PyPI.

I can't find any docs about using CKAN API in order to query packages.

Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible?

Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient:

result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1})

As result nothing found.

Thanks.

1290769564000000 1291633657000000
#668 defect thejimmyg Colin Calnan closed invalid Caching issues on API v1

It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues.

Steps to test:

  1. Modify a dataset on datadotgc.ca, redirects to CKAN
  1. On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100
  1. You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present
  1. Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present.
  1. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either.
1285953542000000 1311176649000000
#2934 enhancement Dominik new Webtests for CKAN

The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes.

An example user story to be tested would be: "As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system." or: "As a user I want to use the search functionality and open a resource from the search page."

These user stories require a different testing system with interaction and processes.

1347966647000000 1348060201000000
#822 enhancement Javier de la Cueva <[email protected]…> closed fixed Docs: Configuration as site-available instead of site-enabled for apache

The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net

1290107103000000 1290506354000000
#904 task rgrp Stiivi ckan-v1.4-sprint-3 closed fixed Review CKAN documentation

What's bad at the moment?

  • lack of documentation e.g. config (very poorly documented)
  • too many sources of documentation
  • no common theming

Sources:

Resulting meta-ticket with things to do: ticket:927

1294832610000000 1299840539000000
#905 defect thejimmyg Stiivi ckan-sprint-2012-02-06 closed worksforme Unable to search with accented characters in package names

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

  • psql DB with UTF-8 tables
  • psql search backend
  • ckan version: 1.3.1b

Example:

http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

1294939120000000 1328638536000000
#906 enhancement thejimmyg Stiivi ckan-sprint-2012-03-05 closed fixed Ability to search without accents for accented words

In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCIITRANSLIT).

1294939588000000 1330990400000000
#699 defect pudo TimDavies closed fixed Paging search results re-sets Open Only and Downloadable Only flags

When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings.

1287143698000000 1296499061000000
#951 defect [email protected] closed invalid Can't add a package to group

I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page.

1296726886000000 1314031006000000
#1246 defect pudo [email protected] ckan-sprint-2011-10-28 closed fixed Search results on ckan.net are mistakenly all 'open'

All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: http://ckan.net/package

1311863353000000 1311892816000000
#1064 defect amercader closed duplicate Remove Workers from ckanext-queue

The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message). This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly.

1301417891000000 1323169787000000
#1134 CREP amercader ckan-backlog new CREP0003: Description and Configuration of Harvesters

Proposer: Adrià Mercader

Abstract

The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to:

  • Expose their type and other details so they can be used internally and on the UI.
  • Define configuration settings for particular harvester instances.

The Problem

Harvester description

The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current get_type method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. "CSW Server"), making it error prone.

Arbitrary configuration

In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the IHarvester interface and calls the appropiate methods for the current stage (gather_stage,fetch_stage,import_stage). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code).

Specification

Harvester description

Harvesters will need to provide the following information so the UI form can be built:

  • name: machine-readable name (e.g. "waf"). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester.
  • title: human-readable name (e.g. "Web Accessible Folder (WAF)"). This will appear in the form's select box.
  • description: a description of what the harvester does (e.g. "A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents"). This will appear on the form as a guidance to the user.

The way to provide it will be an info method that all harvesters must implement, which will return a dictionary with the previous elements:

    {
        'name': 'csw',
        'title': 'CSW Server',
        'description': 'A server that implements OGC's Catalog Service 
                        for the Web (CSW) standard'
    }

Arbitrary configuration

As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called config in the database) (Maybe using JsonType??).

This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an "Add a configuration flag" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an "Advance options" section.

Why do it this way

Harvester description

The info method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface.

Arbitrary configuration

There is an already existing config field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime.

Implementation plan

Deliverables

Risks and mitigations

The highest risk on the harvesters info method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI.

Participants

Adrià Mercader to do it.

Progress

None yet.

1305108868000000 1339774554000000
#1152 enhancement amercader amercader ckan-backlog new True support for generic CSW servers

The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are:

  • Handling the validators (right now are hardcoded in the harvester

code). This probably involves issues discussed in the CREP 3 (ticket #1134)

  • Changes in the model to adapt the specification to ISO 19139
  • Renaming objects and classes which are now Gemini-centric

List of CSW servers tested:

https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0

1306141334000000 1313411822000000
#1160 enhancement amercader amercader pdeu-1 closed wontfix Havest CSW servers on PDEU

It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include:

  • Compose Endpoint List (1d)
  • Harvest on staging system and evaluate which sources have useful material (2d)

First point is being performed in this GDoc: https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&authkey=CNu4hsEB&hl=en_US#gid=0

1306407271000000 1309804038000000
#1161 requirement pudo amercader pdeu-1 closed duplicate Build a simplified theme for PDEU

We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue.

This ticket relates to work on the PDEU theme only!

1306407835000000 1306408026000000
#1162 enhancement amercader amercader pdeu-1 closed fixed Define a common extra for storing the geographic extent

We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion: http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html

1306408704000000 1308647187000000
#1164 enhancement amercader amercader pdeu-1 closed fixed Cloropleth Map of European Data Availability for PDEU

A nice map in the homepage showing the availability of data across Europe

1306408824000000 1308647224000000
#1166 enhancement amercader amercader ckan-sprint-2011-10-10 closed fixed Less stringent filters for displaying the WMS preview

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

1306422646000000 1317381932000000
#1173 enhancement amercader amercader pdeu-1 closed fixed Offer a DCat representation for packages in the API

/api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf.

1306939639000000 1307615271000000
#1253 enhancement amercader amercader ckan-sprint-2011-10-28 closed fixed Authz refactor

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

1312199491000000 1313401367000000
#1272 enhancement amercader ckan-sprint-2011-10-10 closed fixed Store spatial extents provided in the package form

Consolidate the extra spatial as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features.

1313411787000000 1317381996000000
#1273 requirement amercader ckan-backlog new Create docs for API v3 1313412083000000 1313412083000000
#1278 enhancement amercader ckan-backlog new Refactor authorized_query calls

There are some functions that still use the Auhtorizer().authorized_query method:

./ckan/controllers/authorization_group.py:24:        query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup)
./ckan/lib/base.py:237:        groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, 
./ckan/lib/search/sql.py:55:        q = authz.Authorizer().authorized_query(username, model.Group)
./ckan/lib/search/sql.py:118:        q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package)
./ckan/logic/action/get.py:154:    query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT)

./ckan/tests/test_authz.py:158:        q = self.authorizer.authorized_query(self.notadmin.name, model.Package)
./ckan/tests/test_authz.py:353:        q = self.authorizer.authorized_query(self.notmember.name, model.Package)
./ckan/tests/test_authz.py:357:        q = self.authorizer.authorized_query(self.member.name, model.Package)
./ckan/tests/functional/test_authorization_group.py:44:        group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count()
1313415177000000 1313415177000000
#1279 enhancement ross amercader ckan-v1.8 closed duplicate Refactor or deprecate Authorization Groups

Authorization Groups are going to be deprecated in 1.8

It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets #1229 and #1253)

1313415367000000 1338212028000000
#1301 defect amercader amercader ckan-sprint-2011-09-12 closed fixed Give access to extensions from the old formalchemy controllers

This was removed during the auth refactoring, but it's needed in the old controllers

1314882888000000 1314893744000000
#1303 defect amercader amercader ckan-sprint-2011-09-12 closed fixed User can not login again if he has edited his profile

Only happens if the user does not update the password. Most probably introduced during #1229.

1314889472000000 1314978842000000
#1304 enhancement amercader amercader ckan-sprint-2011-09-12 closed fixed Show more descriptive error messages when sending emails

Sending emails is failing in some cases, and the error message just says 'Could not send reset link'.

1314893206000000 1314893749000000
#1305 defect nils.toedtmann amercader ckan-backlog closed fixed SMTP config for thedatahub.org and IATI

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'[email protected]…': (550, 'relay not permitted')},)

As I said, [email protected]… works fine. The SMTP server used mail.okfn.org

1314956657000000 1315317033000000
#1329 defect dread amercader ckan-sprint-2011-10-28 closed fixed i18n poor in some templates - missing i18n tags

Just a tip for developers regarding translations:

  • When writing templates, mark <p>, <div>, etc elements with the

i18:msg attribute, otherwise the messages don't get extracted properly.

Good: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40

<p i18n:msg="">More about CKAN, and the background to its creation can
   be found in <a
     href="http://m.okfn.org/files/talks/opentech_20080705/">these
     slides</a>.</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238

Bad: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211

<p class="hints">
<strong>Important:</strong> By submitting content, you agree to
release your contributions
 under the <a href="http://opendatacommons.org/licenses/odbl/1.0/">Open
Database License</a>. Please <strong>refrain</strong> from editing
this page if you are <strong>not</strong> happy to do this.
</p>

gives this: https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384

The message is split in different parts that not necessarily match the grammar or syntax of the translated language.

1315824288000000 1317404566000000
#1333 enhancement amercader amercader ckan-sprint-2011-10-28 closed fixed Move form converters to ckan core

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

1315909268000000 1316951285000000
#1334 defect amercader amercader ckan-sprint-2011-10-28 closed fixed "AttributeError: 'NoneType' object has no attribute 'name'" exception when logging in

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'
1315909508000000 1319798961000000
#1337 defect amercader amercader ckan-sprint-2011-10-28 closed fixed Group extras are JSON endoded in the REST API responses

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}
1315924727000000 1316951259000000
#1338 defect amercader amercader ckan-sprint-2011-10-28 closed fixed Exception when trying to use a custom form schema from an extension

Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators.

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit
  old_data, errors = validate(old_data, schema)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate
  converted_data, errors = _validate(flattened, schema, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate
  convert(converter, key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert
  converter(key, converted_data, errors, context)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator
  model = context['model']
KeyError: 'model'
1315925217000000 1317117738000000
#1342 defect johnglover amercader ckan-sprint-2011-10-10 closed fixed Home controller does not capture SearchError exceptions

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting http://localhost:5000/dataset).

Steps to reproduce:

1316001227000000 1318256532000000
#1352 enhancement amercader ckan-backlog new Use logic functions instead of as_dict when indexing entities

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)
1316615397000000 1339774086000000
#1354 enhancement amercader amercader ckan-v1.5 closed wontfix Pass the context object to plugins implementing the IPackageController and IGroupController

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

1316955680000000 1316970374000000
#1355 defect amercader ckan-backlog new Package extras property does not include the newly created ones

The extras in the package object sent to the extensions after editing (https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226) do not include the newly added.

1317034126000000 1339774056000000
#1356 enhancement kindly amercader ckan-sprint-2011-10-10 closed fixed Can not recreate a deleted extra

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

1317034180000000 1318279617000000
#1375 enhancement amercader amercader ckan-sprint-2011-10-10 closed fixed Define new package id before sending it to the IPackageController plugins

When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface

https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64

At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package.

One possible solution would be to set the package id in the data dict received by package_create (if it's not defined):

from ckan.model.types import make_uuid
if not 'id' in data or not data['id']:
   data['id'] = make_uuid()
1317910441000000 1318260008000000
#1430 defect amercader ckan-sprint-2011-11-07 closed fixed Documents get mixed between SOLR cores

On some occasions (apparently random), the documents indexed in a specific SOLR core get mixed with different site_ids.

E.g: We look for all documents in the testing.iatiregistry.org core, faceted by site_id. We would expect all documents to have site_id = iati_testing, but some of them have site_id = iatiregistry.org

http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=*:*&fq=+state:active&facet=true&facet.field=site_id

<lst name="facet_fields">
<lst name="site_id">
<int name="iati_testing">265</int>
<int name="iatiregistry.org">255</int>
</lst>
</lst>

If we compare one of the records which disappeared from the "iati_testing" site_id in both the production and testing SOLR cores of the server, the records are exactly the same, including the indexed_ts property:

http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&version=2.2&q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&facet=true&facet.field=site_id

Note that the response from the URLs shown may vary, as the testing site could have been reindexed.

1320068076000000 1324033923000000
#1456 enhancement amercader amercader ckan-sprint-2011-11-21 closed fixed Use resource description instead of name if both are present

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

1320862619000000 1324472178000000
#1470 defect dread amercader ckan-sprint-2011-11-21 closed fixed Check user name in the profile form 1321446143000000 1324473955000000
#1490 enhancement amercader amercader ckan-sprint-2011-12-05 closed fixed Standardize output from package listings coming from the logic layer

Right now, the two logic functions that return a list of packages (package_search [1] and group_packages_list [2])use custom functions to generate the output dict. That's suboptimal because:

  • The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts.
  • Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension.

In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the "isopen" property, needed by the template renderer.

[1] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685 [2] https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442

1322225239000000 1324033557000000
#1498 enhancement amercader amercader ckan-sprint-2011-12-05 closed fixed Support multiple SOLR schema versions

At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema. This process will involve:

  • Better documentation on how to deploy SOLR, as single instance or multicore
  • A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL.
  • (If possible) Checks in CKAN core that deal with different schema versions.
1322585264000000 1323165876000000
#1501 defect johnglover amercader ckan-sprint-2011-12-05 closed fixed Pagination links broken

Most of the different pagination links are broken and point to wrong locations. e.g:

  • Dataset listings in group page:

http://test.ckan.net/group/bibliographic

  • Tag letters in tag listing:

http://test.ckan.net/tag

  • User listing:

http://test.ckan.net/user

  • Revision listing:

http://test.ckan.net/revision

This is most probably related with:

https://github.com/okfn/ckan/commit/1772a5c

1322646397000000 1322684411000000
#1516 task amercader amercader ckan-sprint-2011-12-19 closed fixed Update SOLR schema after 1.5.1 release and set up multicore SOLR instances if needed

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

  • Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
    • Update CKAN
    • Update SOLR schema (symlink to suitable version in CKAN source)
    • Rebuild search index
  • Single SOLR instance used by multiple CKAN sites.
    • Update CKAN source
    • Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
    • Update solr_url on each of the CKAN sites with the suitable core
    • Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

  • Same machine as the CKAN site:
    • test.ckan.net
    • iati.test.ckan.net
    • data.gov.uk/ catalogue.data.gov.uk (confirm)
  • s046.okserver.org
    • iatiregistry.org
  • s052.okserver.org
    • datacatalogs.org
  • s004.okserver.org / eu4.okfn.org / solr.okfn.org
    • thedatahub.org (=www.ckan.net)
    • datagm.org.uk
    • publicdata.eu
    • hri.fi (dev.fvh.fi?)
    • it.ckan.net
    • ie.ckan.net
    • cz.ckan.net
    • register.data.overheid.nl (nl.ckan.net?)
    • no.ckan.net / datakilder.no
    • br.ckan.net
    • colorado.ckan.net
    • at.ckan.net

TODO: which SOLR server are using these instances?

  • data.norge.no
  • nederland.ckan.net
  • lt.ckan.net
  • pl.ckan.net
  • datadotmontreal.ca/
  • ca.ckan.net / datadotgc.com
1323108431000000 1323362689000000
#1533 defect rgrp amercader ckan-backlog closed wontfix Make sure ckanext-wordpresser works on CKAN 1.5.1

Superceded by #2234

Super ticket: #1508

Etherpad: http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

1323277660000000 1338202676000000
#1540 defect amercader amercader ckan-sprint-2012-01-09 closed fixed Search API returns an error if empty parameters are provided

Both in 1.5.1b:

http://thedatahub.org/api/search/dataset?groups=lodcloud&title=

and 1.5.2a (current master):

http://test.ckan.net/api/search/dataset?groups=lodcloud&title=

Although the error message in 1.5.2a is more verbose:

"Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.ParseException?: Cannot parse 'groups:lodcloud title:': Encountered \"<EOF>\" at line 1, column 22. Was expecting one of: \"(\" ... \"*\" ... <QUOTED> ... <TERM> ... <PREFIXTERM> ... <WILDTERM> ... \"[\" ... \"{\" ... <NUMBER> ..."

Some parameter validation before sending it to Solr should do the trick

1323359388000000 1326060385000000
#1543 defect johnglover amercader ckan-sprint-2012-01-09 closed fixed Pagination links in the dataset listings don't keep the current filters

E.g. Pagination links on this page don't include groups=lodcloud http://thedatahub.org/dataset?groups=lodcloud

Not sure if related to #1501 (probably not)

1323442623000000 1324483367000000
#1545 enhancement amercader ckan-sprint-2012-01-09 closed wontfix Remove external asset dependencies

CKAN is pulling a number of resources from external locations. This causes problems when connectivity is limited and you have to work locally. Maybe some of them cold be moved to CKAN source to avoid external requests.

Quick search:

./ckan/templates/layout_base.html:            <img src="http://assets.okfn.org/images/logo/okf_logo_white_and_green_tiny.png" id="footer-okf-logo" />
./ckan/templates/layout_base.html:            <a href="http://opendefinition.org/"><img alt="This Content and Data is Open" src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" style="border: none ; margin-bottom: -4px;"/></a>
./ckan/templates/package/resource_read.html:                <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/package/read.html:          <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /></a>
./ckan/templates/_util.html:                    <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/templates/_util.html:                  <img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" />
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      this.$dialog.html('<h2>Loading results...</h2><img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:          self.setMessage('Uploading file ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js:      self.setMessage('Checking upload permissions ... <img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" />');
Binary file ./ckan/lib/app_globals.pyc matches
./ckan/lib/app_globals.py:                                  'http://assets.okfn.org/p/ckan/img/ckan.ico')
./ckan/config/deployment.ini_tmpl:ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico
1323702635000000 1325260051000000
#1547 enhancement amercader amercader ckan-sprint-2012-01-23 closed fixed Add search methods (before and after) to the IPackageController interface

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

  • Modify the query being sent to Solr
def before_search(search_params):

    return search_params

  • Modify the results returned (after_search)
def after_search(search_results, search_params):

    return search_results

Please see this pad for implementation details:

http://ckan.okfnpad.org/search

1323778996000000 1330084379000000
#1590 enhancement amercader amercader closed fixed Create customized feeds for the IATI Registry

We need a way to track changes on the registry (datasets edited or updated), globally and on a per country/publisher/etc. basis. RSS and Atom feeds are really popular, and after closing #191 and #1498 crating them from the search results should be fairly easy.

The following URLs are pretty self-explanatory:

http://localhost:5000/feed/registry.rss

http://localhost:5000/feed/country/AF.rss
http://localhost:5000/feed/publisher/worldbank.rss
http://localhost:5000/feed/organisation_type/10.rss

As we need to implement custom wrappers for countries, publishers, etc, we might as well offer a fully customizable feed, e.g.:

http://localhost:5000/feed/custom.rss?q=activity_count:[* TO 100]
http://localhost:5000/feed/custom.rss?publishertype=primary_source

Apart from the actual feeds, there will be a small amount of work at the template level to add the links to the suitable pages (and maybe a generic page showing all available feeds)

1324486965000000 1326711608000000
#1592 enhancement amercader amercader ckan-sprint-2012-01-09 closed fixed Add metadata_modified and metadata_created to package_dictize output

The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level.

1324488909000000 1326304321000000
#1593 enhancement icmurray amercader ckan-sprint-2012-04-16 closed fixed [super] Create feeds from search results

Recent changes (see #191 and #1498) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy.

Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest.

Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags:

http://<ckan_server>/feed/group/{id}.{format:rss|atom}
http://<ckan_server>/feed/tag/{id}.{format:rss|atom}

These could be shown on the group/tag page, both on the <head> section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag".

Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on

http://<ckan_server>/feed/custom.{format:rss|atom}?res_format=api%2Fsparql
http://<ckan_server>/feed/custom.{format:rss|atom}?q=tags:animal*

On this case a message "Subscribe to these search results" would appear.

Note that a lot of work on this front needs to be done for the IATI registry project (see #1590)

Other tickets related:

  • #1592: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now.
1324489115000000 1334566668000000
#1616 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Catch exceptions when rebuilding the search index

Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop.

1325844669000000 1332327635000000
#1641 enhancement amercader amercader ckan-sprint-2012-01-23 closed fixed ckanext-archiver: Content-length header not reliable to check if resource has been modified

The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated [1].

The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.:

$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/xml
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly
X-AspNetMvc-Version: 3.0
content-disposition: attachment; filename=AFG_IATI_12012012.xml
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2012 12:36:43 GMT

Also worth noting that requests, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.:

$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 12:12:46 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: application/xml

curl -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 11:56:23 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Content-Length: 2686720
Vary: Accept-Encoding
Content-Type: application/xml

All this can lead to some resources never getting updated, and of course the size property of the resource not being set.

As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it).

[1] https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72

1326376420000000 1326376777000000
#1654 task amercader amercader ckan-v1.7 closed fixed [super] Update Publicdata.eu to the latest CKAN stable version

Tasks include:

  • #1813 Update ckanext-pdeu (4d)
  • #1814 Update harvesters (2d)
  • #1649 Verify ckanext-rdf works with latest CKAN (3d)
  • #1815 Reenable Sparql endpoint (?)
  • #1816 Update ckanext-apps (2-3d)
1326710647000000 1338203166000000
#1655 task amercader amercader ckan-sprint-2012-01-23 closed fixed Setup issues on s025 (Publicdata.eu)

Time estimate: 2d

  • Fix logs (apache, ckan, harvest): rotate, set suitable levels
  • Fix harvesting jobs: supervisord for gather consumer, cron job
  • Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

1326710844000000 1327312857000000
#1677 enhancement amercader ckan-v1.6 closed duplicate Make synchronous search the default behaviour

Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour.

1326807604000000 1326807655000000
#1678 enhancement amercader amercader ckan-v1.8 closed fixed Make synchronous search the default behaviour

Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour.

1326807604000000 1341405749000000
#1726 enhancement amercader amercader ckan-sprint-2012-02-06 closed fixed Update harvesting model

We need:

  • "current" column in harvest_object
  • "harvest_source_id" column in harvest_object, foreign key to harvest_source
  • "title" column in harvest_source

The extension should check on startup if the db needs updating and run the necessary scripts.

1328007127000000 1328523253000000
#1727 enhancement amercader amercader ckan-sprint-2012-02-06 closed fixed Add "current" property to harvest objects

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

  • Model update (See #1726)
  • Update queries for getting harvest sources' datasets
  • Update ckanext-csw queries for getting the objects
  • Update inspire and ckan harvesters
  • Document new field
1328007254000000 1328523641000000
#1728 enhancement amercader amercader ckan-sprint-2012-02-20 closed fixed Update UKLP harvesters

Three small changes:

  • Set Resource type to None instead of 'Undefined'
  • Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

  • Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:
    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

1328007610000000 1329498181000000
#1731 enhancement amercader amercader ckan-sprint-2012-03-19 closed fixed Publisher support for harvesting

Estimate: 4-5d

Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources.

We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model)

Tasks include:

  • Auth checks on the different controller actions (and the command line interface?)
  • Modify the index page to just show sources the logged user has permissions on.

(Talk with Ross about the specific functions to use)

  • Create/edit form to show a dropdown with the user available publishers.
1328007919000000 1332152540000000
#1733 enhancement amercader amercader ckan-sprint-2012-02-20 closed fixed Delete (inactivate) harvest sources

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

  • Expose this on the web interface (only for sysadmins, see #1731)
  • Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

  • Publisher should create a new record with new identifier and same/different content as applicable.
  • If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
  • You won't get the state where a package is updated in its deleted state
  • If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

1328008287000000 1329754689000000
#1734 enhancement amercader amercader ckan-v1.7 closed fixed [super] Updates on harvesting for UKLP

Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources.

  • #1726 Update harvesting model
  • #1727 Add "current" field to harvesting objects
  • #1728 Update UKLP harvesters
  • #1733 Delete (inactivate) harvest sources

At some point we also want to support a publisher model for the harvesting:

  • #1731 Publisher support for harvesting
1328008474000000 1332167315000000
#1813 task amercader amercader ckan-sprint-2012-03-05 closed fixed Upadate ckanext-pdeu to work with CKAN 1.6

Tasks include:

Estimate 4d

  • Update requirements (git)
  • Search facets error
  • Error on dataset read (extras in context)
  • Url generator for rdf
  • Make templates work with latest theme
  • Use dicts in templates (isopen())
1329757128000000 1330102557000000
#1814 enhancement amercader amercader ckan-sprint-2012-03-19 closed fixed Check publicadata.eu harvesters

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

1329757408000000 1332152596000000
#1815 task amercader amercader ckan-future new Reenable Sparql endpoint on publicdata.eu

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

1329757723000000 1338204641000000
#1816 enhancement amercader amercader ckan-sprint-2012-03-19 closed fixed Update ckanext-apps to work with CKAN 1.5.1 - 1.6

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

1329757864000000 1331302835000000
#1818 enhancement amercader amercader ckan-future new Spatial metadata editor

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

  • Integration with the form sections.
  • Allow some degree of configuration (default extent, limit geometry types...)

[1] http://openlayers.org/dev/examples/draw-feature.html [2] http://openlayers.org/dev/examples/modify-feature.html

1329763941000000 1338203195000000
#2231 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Base harvester broken on 1.6.1

On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1]

Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected

ValidationError: {'Id': u'The input field id was not expected.'}

We need a way to either:

  1. Be more flexible accepting schemas on the logic functions
  2. Modify the harvester plugin to implement the necessary interface

[1] https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115

[2] https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38

1331815499000000 1333374735000000
#2232 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Unicode Exception when rebuilding the search index

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
1331816582000000 1332331660000000
#2252 defect amercader amercader ckan-sprint-2012-04-16 closed fixed UKLP harvesters returning errors when importing due to schema changes

See trace below.

As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert
    converter(key, converted_data, errors, context)
  File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)
1332333907000000 1334567502000000
#2263 enhancement amercader amercader closed fixed IATI: Get "Record updated" field from the registry

Don't provide a field in the form or spreadsheet import for this, just use dataset.metadata_modified.

1332766145000000 1332846807000000
#2268 enhancement thejimmyg amercader ckan-backlog new UKLP Use same schema as form for harvested records

Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach.

1333108606000000 1333108606000000
#2271 defect amercader amercader ckan-sprint-2012-04-16 closed fixed IATI: Some datasets not being indexed

When importing a CSV file for a certain publisher, records don't come up on the search index. After some investigation this has tunrn out to be caused by the records being deleted. We need to investigate if this was caused by the import or just made by the users.

1333375469000000 1334567582000000
#2324 enhancement amercader ckan-v1.9 new Link API docs to relevant version not just latest

The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading

1335351532000000 1345199411000000
#2358 task amercader amercader ckan-sprint-2012-05-15 closed fixed Finish off Release 1.7

Various stuff related with release 1.7:

  • Review patches sent to the release branch
  • Regularly update TDH with latest changes
  • write changelog, announce email, etc
  • ...
1335887030000000 1336641577000000
#2359 task amercader amercader ckan-sprint-2012-05-15 closed fixed Investigate pycsw integration

Tom Kralidis kindly told us on IRC [1] that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration.

[1] http://irclogs.okfn.org/ckan/2012-04-25.txt 17:16

1335887299000000 1337016578000000
#2360 enhancement amercader amercader ckan-sprint-2012-05-15 closed fixed Finish INSPIRE support proposal

I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on #2359

1335887409000000 1337016284000000
#2361 enhancement amercader amercader ckan-sprint-2012-05-15 closed fixed Install ckanext-googleanalytics and ckanext-stats on PDEU

ditto

1335887476000000 1337016768000000
#2366 enhancement ross amercader ckan-sprint-2012-06-25 closed fixed Remove Authorization Groups

1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8.

1335892563000000 1346316322000000
#2382 task amercader amercader ckan-future new Investigate options for basic geocoding

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

1337017160000000 1338205325000000
#2388 enhancement amercader amercader ckan-v1.8 closed fixed Update "out of the box" settings for CKAN
  • Only logged in users able to create
  • Creation permissions have only creators able to update a dataset
  • Synchronous search is enabled (#1678)
1337095269000000 1342104475000000
#2389 defect amercader amercader ckan-v1.8 closed fixed More robust handling of DataExplorer errors

US-0a

Do not show failed preview with red-bar (esp when just empty datastore ...)

  • Data Viewer checks if datastore is empty nad falls back to ther options if emtpy
  • DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

Loading/showing error reports

1337095447000000 1341222201000000
#2400 defect amercader ckan-sprint-2012-05-29 closed worksforme Password reset fails

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"
1337188614000000 1338205947000000
#2467 defect amercader amercader ckan-v1.8 closed fixed Fix stats extension frontend

After moving to Bootstrap the tables on the frontend look broken.

1338213230000000 1340981885000000
#2468 enhancement amercader amercader ckan-sprint-2012-06-25 closed fixed Finish off SlickGrid based Recline view

Please see these GitHub? issues:

https://github.com/okfn/recline/issues/130

https://github.com/okfn/recline/issues/131

1338213375000000 1339757934000000
#2505 enhancement amercader amercader ckan-sprint-2012-06-25 closed fixed Docs improvements for 1.7.1

There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1.

  • Add an introduction to harvesting support
  • Add an introduction to CSW support
  • Better upgrade docs: Some users suggested improvements to the upgrade docs on the list [1]. Also we should mention how the upgrade would work for point point releases
  • Source install: see #2406

[1] http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html

1339166533000000 1340018770000000
#2575 enhancement toby amercader ckan-v1.9 closed fixed Licenses have i18n issues (was Inconsistent language on license dropdown)

License titles at least for some of them need translating. This wants to be done in a more efficient and simple manner than currently exists.

the original problem was:

The license dropdown on the dataset form shows titles in Swedish by default. Changing the language seems to default to English, but after refreshing the page a couple of times it shows a wrong language again.

Thanks to pudo for reporting.

1340125845000000 1343216544000000
#2604 enhancement amercader amercader ckan-v1.8 closed fixed Add Requests as requirement for tests

Requests (http://python-requests.org) is really useful to deal with http requests and saves a lot of time.

I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage.

1340790229000000 1340968458000000
#2639 enhancement amercader amercader ckan-v1.8 closed fixed Update recline version for 1.8 release

New recline version has significant improvements like the SlickGrid? based grid view, it'd be good to have it on 1.8.

1341845809000000 1350296205000000
#2640 enhancement amercader amercader demo phase 2 closed fixed Update recline on the demo site

Depends on #2639

1341845911000000 1342112730000000
#2641 enhancement johnmartin amercader demo phase 5 assigned Adapt spatial widgets to new theme

Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now

1341846147000000 1352658854000000
#2642 enhancement shevski amercader demo phase 3 closed fixed Adapt harvest frontend to new theme 1341846196000000 1345113645000000
#2745 defect amercader ckan-v1.9 new Password reset returns an exception if the key parameter is missing

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

1343145931000000 1343145931000000
#2788 enhancement amercader amercader ckan-v1.9 new Speed improvements on creating/updating and indexing

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

1343832992000000 1343832992000000
#2841 enhancement aron.carroll amercader demo phase 3 closed fixed Slug generation fails on demo dev

On the add dataset form, slug generation fails with JS error:

Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning'    basic-form.js:5
1344855653000000 1344857415000000
#2949 defect amercader amercader new Reenable Data API button on the new theme

The checks to show or not the button need to be updated for the latest datastore version

1349107464000000 1349107464000000
1 2 3 4 5 6 7 8 9 10 11
Note: See TracReports for help on using and creating reports.