{22} Trac tickets (2647 matches)

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

• research ways to limit requests-per-process with the different web servers (see below for a start);
• explain these safeguards in the CKAN documentation and encourage users to apply them;
• consider the helper script ​ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of ​commented out. Or at least let it warn the user to use MaxRequestsPerChild;
• Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

• Apache:
  • Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on ​datahub.io and the ​ckan farm
  • (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
• nginx/supervisord: to be researched.

Quite a few of the current test dataset resources have their format incorrectly specified as text/html. Attempting to preview these results in the browser trying to automatically download the resource, so for now we will not preview this format.

When clicking 'add dataset' on a publisher/group page, the new dataset form should select the appropriate group.

The following pages are missing complete breadcrumbs.

Add related page:

​http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Needs to look like.

Dataset > {dataset_title} > Related > Create Related

Template here: ​https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/pages/form_page.html#L6

Same deal with Edit related.

​http://s031.okserver.org:2375/dataset/adur_district_spending/related/edit/5f9e665b-3e3c-428f-88b1-8554189cf14a

Needs to look like.

Dataset > {dataset_title} > Related > Edit Related

​https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/edit.html#L5

templates/error_document_template.html

I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content

just go to a non-existant url to see

Discusion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/

Implementation:

​http://s031.okserver.org:2375/404

​http://datahub.io/stats

if not logged in then go to add dataset url the language is lost

check if this is a ckan 1.7 issue

There's no docs for setting up the internal analytics, nor what it provides.

Setup

Add: ckan.tracking_enabled = true to .ini file

Run: paster tracking -c dev.ini

Related links

​http://trac.ckan.org/ticket/2251

New form should allow three steps.

1. Add basic fields to create a dataset
2. Add a data resource to the dataset (this can be repeated multiple times using "Save and add another"
3. Add final metadata to the dataset.

It should be possible to jump backward through the steps by clicking the arrows at the top. Errors in fields should be validated for each step when the next button is clicked.

Implementation:

​http://s031.okserver.org:2375/dataset/new

For example:

Creating a template in ckanext-demo/templates/base.html containing something like

{% ckan_extends "base.html" %}

Will extend ckan/templates/base.html

There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1.

• Add an introduction to harvesting support
• Add an introduction to CSW support
• Better upgrade docs: Some users suggested improvements to the upgrade docs on the list [1]. Also we should mention how the upgrade would work for point point releases
• Source install: see #2406

[1] ​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html

If you change the language to one where the word for 'Search' is longer than the English word (e.g. fr) then the search button wraps onto the next line, below the text field, which looks bad.

Looks like some of the style has been updated/changed on the group read page but the changes haven't been applied to the organizations read page.

Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8. Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set.

This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside.

The related tab on the dataset page doesn't change its appearance when selected like the other tabs do

• created repo for final list of instructions: ​http://github.com/okfn/ecportal-server-setup

Whilst the error messages returned from the dataproxy are informative, they aren't particularly human readable. It isn't clear to the user whether they should retry, or whether the format of their data is causing the problem etc...

Would be lovely if the error messages returned contained a human readable version of the error.

It still contains the old build_search_index_synchronously which no longer works

​http://localhost:5000/dataset/new_metadata/dataset

Controller : package Action : new_metadata

template package/snippets/package_metadata_form.html

JavaScript? needs to be able to insert html templates into the document.

There are three common solutions to templating at the moment.

1. Insert the html into <script> tags at the bottom of the page and pull them in using the DOM and innerHTML. This is a very simple solution but not very portable.
2. Request the HTML snippets as plain text/html using XHR. This adds an extra HTTP request per template but is very clean. It would also be possible to inline the HTML into the JS at the minification stage for production.
3. Include the HTML in JavaScript? strings (horrible to maintain and bloats the js)

It makes sense to keep these with the other templates so that we can take advantage of localisation where possible. This indicates 1 or 2.

Simple widgets in pure JS. For example:

• Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
• Embeddable list of top X (5) datasets for a given query
• Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. ​http://okfnlabs.org/ckanjs/

Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ...

Looks like recent CKAN cleanups broke some imports:

​https://gist.github.com/2935639

DataHub? is increasingly used and we should ensure it uses ssl as part of general security.

See also #1446 (Need to support https login for multiple instances as part of the CKAN package install)

​https://github.com/okfn/ckanbuild

Verify that what's there so far still works, write a README explaining how it works

Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh?

We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include.

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: ​http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

Requirements

We need to clone the demo ODS site for DataGM as soon as we have a mostly working demo so that the DataGM team can try it out and comment/feedback on functionality.

Interface

None

User Stories

None

Tasks

[ ] Deploy demo DataGM site.

Estimates

Requirements

The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS.

May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps.

Interface

None

User Stories

• As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed.

• As a bizdev person I would like to be able to demonstrate how ODS works with real datasets.

Tasks

[ ] Identify relevant sources for datasets

[ ] Pick datasets

[ ] Set them up for import

Estimates

Requirements

Merge the organizations plugin into the core groups functionality. Most of this is templating and a small number of changes to the group controller.

Investigate the possibility of removing the Membership model and using FKs.

Interface

This is likely to consist of:

• Updating the templates in templates/group/

• Configuring whether we want to constrain datasets to a single group (as per organizations)

• Implementing the separate user management (add users to group).

• Checking the publisher auth to not necessarily rely on group type = 'organization'

• Making sure that the changes still work with publisher auth.

User Stories

None

Tasks

[ ] Code

[ ] Configuration

[ ] Model/Migration?

[ ] Fix templates

[ ] Documentation

Estimates

Requirements

Would be useful if we could extend the Google Analytics extension to push track events to GA so that we can determine how the system is being used, and where we can optimise the workflow.

Assuming it isn't configured to be off, track events should be added to some key action buttons so that the information on what users are doing is logged for analysis.

Interface

This could possibly just be a list of JS calls to $('x').click(_gaq.push('track'...))

User Stories

• As a developer I would like to know which components of the system are used the most so I can streamline the processes.

• As a support developer I would like to know which parts add complexity to the system for minimal gain and could possibly be removed, de-prioritised.

For which metrics, as many of the following as possible that are achievable without changing the templates. I think for now it is enough to just record counts of these pages/actions.

• Any group link on homepage

• Clicking on username (to go to profile)

• In user profile, clicking on Edit Profile

• Clicking Save Changes on Edit Profile page

• Clicking on any dataset link on User Profile page

• Any of the group links on /group

• Clicking any of the right hand sidebar tags on /group/X

• Visiting /group/history/X

• Compare Button on /group/history/X

• Compare Button on /dataset/history/X

• Tags on right hand sidebar of /dataset/X

• Download button on any /dataset/X/resource/* page

• Data API button on any /dataset/X/resource/* page

Tasks

[ ] Analysis/Clarification?

[ ] Code/Config? changes

[ ] Documentation

Estimates

just passes template name

?template_name=...

Something to make building breadcrumbs a bit nicer

• Crosses have disappeared
• Formats should stretch across the page

• Show resources in the sidebar
• Allow people to go backward and forward
• Have a back button on each form
• Lose resource type
• Add extras as optional metadata
• Remove save
• Add back/Previous
• Add Another

• Add tags
• Add groups
• JavaScript? for custom fields

Allowing them to be displayed in many sizes, colours and resolutions.

​http://www.zurb.com/playground/foundation-icons

Share this (twitter/fb) links open in new tab (no JS) or nice lightbox of some description in demo

No real need/use for this for demo/ODS yet

Allow user to navigate directly from one resource page to another via resource box in sidebar

This function is incomplete

If you want I can do this but I need guidance over how to do this properly. ie would I go through lib.dictization.model_save.package_resource_list_save() which looks wrong to me as I'd need the package. resource_dict_save() seems better but to be honest the total lack of docstrings etc makes me feel ill.

Solr allows its facet parameters to be specified on a per-field basis, eg. facet.limit applies to all facet fields, but solr allows it to be overriden for a specific field, eg. facet.tags.limit.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See ckan.lib.search.query.VALID_SOLR_PARAMETERS.

License titles at least for some of them need translating. This wants to be done in a more efficient and simple manner than currently exists.

the original problem was:

The license dropdown on the dataset form shows titles in Swedish by default. Changing the language seems to default to English, but after refreshing the page a couple of times it shows a wrong language again.

Thanks to pudo for reporting.

When given a url parameter for a file it cannot fetch, the app raises a 500 instead of something useful.

Better error reporting in the JSON would be useful

make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc

pass some stuff through the context

in development many auth checks may have been lost we need to check they are still working etc