{22} Trac tickets (2647 matches)

This is pretty key for standardization.

Also would like us to change the grey text to be: e.g. CSV, JSON, [more examples... i have no idea what they key ones are tbh]

Once you upload a file, all the buttons on the page stop working. You can't complete & go to next step or save & add another or go to previous

Clicking on these just seems to re-load the page

​http://s031.okserver.org:2375/dataset/new_resource/super

See attachment. Just need to amke sure that there's enough spacing around the edit button and that it lines up so it doesn't look too wierd

in dataset creation / edit form

lang box improvements: styling, images, submit upon selection

to reproduce run

paster color fuchsia -c development.ini

then go to ​http://localhost:5000/dataset/new

Add button is blue gradient at the top this should be overridden in variables.less

when added send this ticket to me and i'll update the color command to change it

1. go to a group, e.g. ​http://s031.okserver.org:2375/group/data-explorer
2. Order by anything, e.g. choose 'popular' and click go
3. "sorry no datasets found" message appears instead of ordering the group datasets in wanted order

​http://s031.okserver.org:2375/group/data-explorer?q=&sort=views_recent+desc

Can we please add some text next to the format field when user adding a new resource along the lines of:

"This will be generated automatically, but you can edit if you wish"

It always seems to work better when the user does not fill it themselves...

title broken for

​http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices

we have inherited the title from the extended page not from the base

should be 'add data - ckan' not 'add data - create dataset - ckan'

any idea how we fix this?

Specifically the 'add resource' button and the 'add item' button in related section

see spacing between 'download' button and 'data api' button ​http://s031.okserver.org:2375/dataset/afterfibre/resource/66f5b980-06db-4ccd-a005-c18de4d68a4b

This keeps the intended order.

apps dashboard available at ​http://localhost:5000/apps in main dev branch

I have got this to work in the demo however

a) it is a bit ugly

b) the template could do with some sorting eg the dropdowns should be being created in a nicer way

demo.ckan.org has a small field for adding text on homepage current text: "Welcome to the CKAN demo Try out standard CKAN functionality in a sandbox environment. Search for datasets directly from the homepage or by navigating to the Datasets search page where you can facet by tags, groups and format."

Want to make this editable for demos

@aron

We should be able to have longish tag lines that look ok currently they wrap and stop the logo text being clickable

Can we have white-space:nowrap; for the tagline so long ones don't wrap and maybe look at aligning long ones

​http://s031.okserver.org:2375/dataset/resource-markdown/resource_edit/ba1b094d-697c-47c4-a72a-6c2557da5409

should have back to resource as a minimum

be good to do this in phase 3

This is what we need the JSON to look like. Taken from the Jed documentation (​http://slexaxton.github.com/Jed/).

{

      // The empty string key is used as the configuration
      // block for each domain
      "" : {
        // Domain name
        "domain" : "the_domain",

        // Language code
        "lang" : "en",

        // Plural form function for language
        "plural_forms" : "nplurals=2; plural=(n != 1);"
      },

      // Other keys in a domain contain arrays as values
      // that map to the translations for that key.
      // Gettext suggests that you use english as your keys
      // in case the key isn't found, and it can just pass
      // the value directly through.
      // Note: by convention, the 0-index location of the translations
      // is never accessed. It's just a thing, I guess.
      "a key" : [ null, "the translation", "the plural translations", ... ],

      // The plural form string is converted into a function
      // and the value that's passed into the gettext call
      // is passed into the plural form function. It's result
      // (plus 1) is the index that the translation comes from.

      // We're using sprintf interpolation on our keys so we can
      // then sub in the _actual_ values into the result.
      "%d key" : [ null, "%d key", "%d keys" ],

      // Contexts are keys that are just prefixed with a context string
      // with a unicode \u0004 as the delimiter.
      // You can use it for anything. Usually it's just for being content aware
      // in some way (e.g. male vs. female, product vs. category)
      "context\u0004%d key": [ null, "context %d key", "context %d keys" ]
    }

This looks like an appropriate python library to do the formatting. ​http://pypi.python.org/pypi/pojson/ but I'll leave to your discretion.

Finally an API endpoint that returns the appropriate request would save us making requests to the i18n directory.

/api/i18n/{locale}

For example if the current url is ​http://localhost:5000/pl/dataset then the client would request:

http://localhost:5000/api/i18n/pl

This should return the Content-Type: application/json and a 200 with an appropriate JSON body (this can be the language requested or an appropriate fallback). I think that if the fallback is the default language (in most cases English) then we just return a 204 No Content to save data.

on resource pages (e.g. ​http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360), same as old ckan for continuity and to highlight!

If you make your browser window short - say half screen height on a page with delete eg ​http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada

the dialog box is partially off the screen - it should try to stay on screen

Is this done I know checkboxes are a pain but it looks a bit out of place compared to the other items.

On the add dataset form, slug generation fails with JS error:

Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning'    basic-form.js:5

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

• An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
• A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

• index.rst - copy some stuff from README.txt, keep short
• README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
• [NEW] install.rst - new file on installation
• [NEW] theming.rst - move over from wiki
• api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
• i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
• design.rst - REMOVE (can copy some over if you can be bothered to ​http://wiki.ckan.net/Vision (all philosophical stuff should go!))
• loaders.rst: move to wiki.ckan.net/Using_Loaders
• feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
• importer.rst: remove (don't even copy)
• forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) ​http://wiki.ckan.net/Customizing_Forms
• forms-integration.rst: remove
• model.rst: leave but move to reference (only for core devs)
• load-testing.rst -> move to ​http://wiki.ckan.net/Load_Testing
• distributed.rst -> remove
• admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
• deb.html -> go into reference (query james gardner on list about moving this to wiki?)

Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return <ows:ExceptionReport ... />.

Both cases would be resolved by checking capabilities (with CswClient's check_get_capabilities() method).

The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are:

• Handling the validators (right now are hardcoded in the harvester

code). This probably involves issues discussed in the CREP 3 (ticket #1134)

• Changes in the model to adapt the specification to ISO 19139

• Renaming objects and classes which are now Gemini-centric

List of CSW servers tested:

​https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0

We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city).

Relevant discussion: ​http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html

Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS".

This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough.

(The repository for this is ckanext-spatial, but it's not on the list)

We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system.

Only happens if the user does not update the password. Most probably introduced during #1229.

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

Instead of this:

extras: {
type: ""Primary source""
license_id: ""localauth-withrights""
publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0""
}

we should return this:

extras: {
type: "Primary source",
license_id: "localauth-withrights",
publisher_frequency: "2Preño € $ àdrià",
}

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

It is not currently possible to assign or remove groups by adding or removing their group name from the "groups" list on the REST API. This should be allowed, as it is easier than first adding a package and then editing each group it must belong to.

With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes.

Right now, the two logic functions that return a list of packages (package_search [1] and group_packages_list [2])use custom functions to generate the output dict. That's suboptimal because:

• The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts.
• Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension.

In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the "isopen" property, needed by the template renderer.

[1] ​https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685 [2] ​https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

• Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
  • Update CKAN
  • Update SOLR schema (symlink to suitable version in CKAN source)
  • Rebuild search index

• Single SOLR instance used by multiple CKAN sites.
  • Update CKAN source
  • Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
  • Update solr_url on each of the CKAN sites with the suitable core
  • Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

• Same machine as the CKAN site:
  • test.ckan.net
  • iati.test.ckan.net
  • data.gov.uk/ catalogue.data.gov.uk (confirm)

• s046.okserver.org
  • iatiregistry.org

• s052.okserver.org
  • datacatalogs.org

• s004.okserver.org / eu4.okfn.org / solr.okfn.org
  • thedatahub.org (=www.ckan.net)
  • datagm.org.uk
  • publicdata.eu
  • hri.fi (dev.fvh.fi?)
  • it.ckan.net
  • ie.ckan.net
  • cz.ckan.net
  • register.data.overheid.nl (nl.ckan.net?)
  • no.ckan.net / datakilder.no
  • br.ckan.net
  • colorado.ckan.net
  • at.ckan.net

TODO: which SOLR server are using these instances?

• data.norge.no
• nederland.ckan.net
• lt.ckan.net
• pl.ckan.net
• datadotmontreal.ca/
• ca.ckan.net / datadotgc.com

Time estimate: 1d

Add a two search method to the IPackageController interface so extensions can:

• Modify the query being sent to Solr

def before_search(search_params):

    return search_params

• Modify the results returned (after_search)

def after_search(search_results, search_params):

    return search_results

Please see this pad for implementation details:

​http://ckan.okfnpad.org/search

The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level.

Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop.

The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated [1].

The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.:

$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/xml
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly
X-AspNetMvc-Version: 3.0
content-disposition: attachment; filename=AFG_IATI_12012012.xml
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 12 Jan 2012 12:36:43 GMT

Also worth noting that ​requests, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.:

$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 12:12:46 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: application/xml

curl -I http://iatistandard.org/published-temp/adb-activities.xml
HTTP/1.1 200 OK
Date: Thu, 12 Jan 2012 11:56:23 GMT
Server: Apache
Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT
Accept-Ranges: bytes
Content-Length: 2686720
Vary: Accept-Encoding
Content-Type: application/xml

All this can lead to some resources never getting updated, and of course the size property of the resource not being set.

As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it).

[1] ​https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72

Time estimate: 2d

• Fix logs (apache, ckan, harvest): rotate, set suitable levels
• Fix harvesting jobs: supervisord for gather consumer, cron job
• Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

Make sure accented characters are normalized when indexed and when searched for.

We need:

• "current" column in harvest_object
• "harvest_source_id" column in harvest_object, foreign key to harvest_source
• "title" column in harvest_source

The extension should check on startup if the db needs updating and run the necessary scripts.

Three small changes:

• Set Resource type to None instead of 'Undefined'
• Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

• Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:

    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

• Expose this on the web interface (only for sysadmins, see #1731)
• Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

• Publisher should create a new record with new identifier and same/different content as applicable.
• If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
• You won't get the state where a package is updated in its deleted state
• If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

Tasks include:

Estimate 4d

• Update requirements (git)
• Search facets error
• Error on dataset read (extras in context)
• Url generator for rdf
• Make templates work with latest theme
• Use dicts in templates (isopen())

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

• Integration with the form sections.
• Allow some degree of configuration (default extent, limit geometry types...)

[1] ​http://openlayers.org/dev/examples/draw-feature.html [2] ​http://openlayers.org/dev/examples/modify-feature.html

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.