Custom Query (2152 matches)

Reported by amercader, 2 years ago.

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.

Reported by amercader, 2 years ago.

On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1]

Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected

ValidationError: {'Id': u'The input field id was not expected.'}

We need a way to either:

1. Be more flexible accepting schemas on the logic functions
2. Modify the harvester plugin to implement the necessary interface

[1] ​https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115

[2] ​https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38

Reported by seanh, 2 years ago.

Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point):

The package_search() function is adding the search facets to the search results twice with two different data structures, with keys "facets" and "new_facets". It should be reduced to just the new facets (with the key changed to "facets").

Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets).

The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure.

In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets.

In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead.

Anywhere that "new_facets" appears it will have to be changed to "facets" (e.g. in the ckanext/multilingual/plugin.py.

This is the merge commit that introduced the duplication: ​https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d

This is the earlier commit where the search facets were refactored: ​https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c