{22} Trac tickets (2647 matches)

To do this for the UKLP, the Ordnance Survey would need to add to their map service an operation to return image for a given bounding box, and COI would need to present the image on the search result page.

Otherwise, it would be possible to use Google.

We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer.

Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?).

We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP.

Add link to RDF representation of a package in our Atom feed.

Form refactor.

New create/edit package form wizard for DGU. See attached spec.

Re-create behaviour of old form without using form-alchemy. And unit test.

• [X] write high-level functional tests for the form. Creation / edition / validation.

Original estimate: 6 days (3 days x 2)

Time spent: 3 days

Time remaining: 0 days

• [X] write the html

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 0 days

• [X] modify the existing validation schema

Original estimate: 2 days (1 day x 2)

Time spent: 1.3 days

Time remaining: 1 days

• [X] split resources into types: individual, time-series, and additional.

Original estimate: 2 days

Time spent: 2.5 days

Time remaining: 0 days

• [X] contact information is derived from Publisher, but can be altered for each dataset. #1617

Original estimate: 4 days

Time spent: 0 days

Time remaining: 4 days

(Waiting on publisher integration)

• [X] foi contact information is derived from user, but can be altered for each dataset. #1617

Original estimate: 1 days

Time spent: 0 days

Time remaining: 1 days

(Waiting on publisher integration)

• [X] provide stubs for the theme and sub-themes #1618

Original estimate: 1 days

Time spent: 0 days

Time remaining: 0 days

• [X] provide stubs for the publisher hierarchy #1619

Original estimate: 2 days

Time spent: 0.5 days

Time remaining: 1.5 days

Note - the groups refactor is essentially ready to be used. And this would be a good test case for it.

• [X] don't allow package-type selection when editing a package

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (#1620)

Original estimate: 0.5 days

Time spent: 0 days

Time remaining: 0 days

Create the form wizard for the package-new form.

Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*)

• separate pages for each section of the form
• validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section.
• no draft saving to be performed in this ticket.

(*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that:

• with the javascript approach it would be harder to test the workflow.
• with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure.
• the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead.
• the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org)
• the multi-page wizard wouldn't preclude a javascript-tabbing edit-form.

Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms.

An AJAX call is made every N seconds, and the auto-saved blob is updated.

note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave.

CKAN 1.5 installations need to be upgrade to 1.5.1.

Working methodically through the CKANs on ​http://datacatalogs.org/group/ckan :

At 1.5:

• ckan-brazil ​http://br.ckan.net/ s057 - DONE
• ckan_ireland ​http://ie.ckan.net/ s057 - DONE
• cz-ckan-net ​http://cz.ckan.net/ s057 - DONE
• datakilder-no ​http://no.ckan.net/ s057 - DONE
• data_gov_uk ​http://catalogue.data.gov.uk/
• datacatalogs-org ​http://datacatalogs.org/ s054 - DONE

Other versions:

• berlin-open-data ​http://daten.berlin.de/ CKAN hidden
• ckan-italia ​http://it.ckan.net/ 1.3.3b
• ckan-poland ​http://pl.ckan.net/ Down
• data-norge-no ​http://data.norge.no/ CKAN hidden
• datadotgc-ca ​http://datadotgc.ca/ ​http://ca.ckan.net/ 1.4.2a
• datagm ​http://www.datagm.org.uk/ 1.3.3
• datagov-ru ​http://datagov.ru/ CKAN hidden
• datospublicos-org ​http://datospublicos.org/ 1.4.2
• iati-registry ​http://iatiregistry.org/ 1.5.1c
• katalog-offene-daten-oesterreich ​http://offener.datenkatalog.at/ 1.3.3b
• nosdonnees_fr ​http://www.nosdonnees.fr/ 1.3.1a
• opengov-fi ​http://opengov.fi/ CKAN hidden
• opengov_es ​http://opengov.es/ 1.2
• publicdata-eu ​http://publicdata.eu/ 1.4.2a
• the-data-hub ​http://thedatahub.org/ 1.5.1
• helsinki_finland ​http://hri.fi/ CKAN hidden

Now deleted from datacatalogs.org:

• ckan-belgium ​http://be.ckan.net/ Migrated
• si-ckan-net ​http://si.ckan.net/ migrated
• gr-ckan-net ​http://gr.ckan.net/ Migrated
• hu-ckan-net ​http://hu.ckan.net/ Migrated
• ckan-lithuania ​http://lt.ckan.net/ Migrated

A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU.

Design doc: ​https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB The design doc is the authoritative source of tasks. This ticket acts as a synopsis.

Current sprint (2012-02-20):

• 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1
• 6. Publisher Registration Improvements MUST PHASE 1
• 3. Themes/Taxonomy? as well as tags WONT PHASE 2
• 32. Browse by Tags PHASE 1 MUST
• 12. Edit/Delete? for Harvested Datasets MUST URGENT PHASE 1
• 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST]
• 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD]
• 22. Mixed Licenses MUST PHASE 2

Plus, brought forward from last sprint:

• Publisher hierarchy support
• Improved dataset creation tools (form) - #1525
• Edit/delete for harvested datasets

Overview of tasks: see the design doc.

[Child of #1526]

Provide the interface for the themes and subthemes hierarchy.

[Child of #1526]

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa.

• On the resources tab, there's a "Check Resources" button which, when clicked makes an ajax request with the list of URLs entered by the user.

• The server checka each link for errors and header information about the linked resource. (Using ckanext/archiver/tasks.py:link_checker()).

• The server returns a list of dicts (json), containing information about the linked resource, and the client uses that to:

• populate the format field of each resource

The (guessed) 'file_extension' populates the 'format' field. If it's 'htm' or 'html', then we assume it's a listing page, and so don't populate the format field with 'htm' or 'html'.

• provide feedback if a URL appears to be invalid

If the URL doesn't appear to be a URL at all, or returns a HTTP error, or times-out, then URL field is highlighted in red to indicate it's a bad URL. A tooltip shows the error message to the user.

• [Optional] provide feedback if a URL appears to point to "Additional Information" - ie the Resource should be entered under "Additional Resources", rather than "Timeseries" or "Individual" datasets.

Analysis and further description on etherpad: ​http://ckan.okfnpad.org/dgu-package-form? [Section I]

Search results page of DGU.

Breakdown:

[X] ensure faceting by group (publisher) works [X] allow faceting by UKLP dataset type [X] allow faceting by INSPIRE

Super ticket: #1745

• Convert /api/rest/dataset to be search query (i.e. take arguments in ?....)
• Directly expose solr though may want to override limit. See ​https://github.com/okfn/openspending/blob/master/openspending/ui/controllers/api.py#L48

Required for some improvements to UX (such as autocomplete and better search).

• [x] hide black stars on the data-view page
• [x] Edit on view form (if you have the right permission)
• [x] RES_FORMAT ~> Resource Format
• [x] Map Based Search -> UK Location Map Based Search
• [x] Note at the top right expalining that these are seach filters (if they are) -> this requires "add a dataset" and "other access" to be moved to a different place (just some whitespace is fine)
• [x] Display publisher and provider (UKLP records only).
• [x] analysis of filter by OGL / non-OGL
• [x] provide /dump on the build server

Atom feeds have been implemented in core in #1593. The IATI-specific implementation can now be removed when IATI move to CKAN >= 1.6.1

• Create Jenkins task for packaging CKAN
• ... and publishing to s3://apt.ckan.org

There are a number of features developed against 1.6.1 that need to be merged into master.

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

These two tests:

ckan.tests.lib.test_helpers.TestHelpers?.test_parse_rfc_2822_gmt_case

ckan.tests.lib.test_helpers.TestHelpers?.test_parse_rfc_2822_with_offset

have been failing in certain timezones. In the UK they pass, if you're in Germany they'll fail.

The function being tested uses datetime.datetime.fromtimestamp() which uses the local time, but the tests compare the results against hardcoded strings, so the tests cannot work in different timezones.

Some additions to the related stuff functionality to make it more useful post the 1.7 release.

[x] Edit item (after checking auth)

[x] My related things on my user page

[x] Show user created related things on the users' page with links direct to the related page of that dataset. {0.5d}

[x] Related stuff usage counts { 1d }

Record the number of times the link for a related item is clicked. This is likely to mean that the link fires to call CKAN and then redirects to the actual URL.

[x] Un-attached related items {0.25d}

Allow related items to be created without being attached to datasets. This may mean that the dataset field is optional in the API.

[ ] Dashboard for related

A dashboard for related items, site-wide (not package specific) and orderable by date, views, {rating}, popularity of the dataset. Would be useful (possibly) for site-admin to mark one or more as 'featured'

[x] Dashboard + Controller (Related) {1d}

[x] Ordering at logic layer (date, views, rating, popularity of package, featured) {1d}

[x] date [x] views [x] featured

[x] 'Featured' related item with mechanism to fetch it. {0.5d}

[ ] UI within dashboard for sysadmins to check/uncheck feature status {0.5d}

[x] Changing auth for related.

Currently auth is based on deletion being possible by dataset owner and the related owner but it may be that a lot of the auth for related items should be configured to have more restrictive auth.

[x] Implement auth changes for publisher auth profile. { 0.5d }

[x] General issues

[x] Serverside validation of URLs

[x] Activity updates

[x] Make optional

[x] Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

[ ] UI Changes

[ ] General tidy up, fix form etc (icon for menu bar)

[ ] Highlight featured items py:if="related.featured"

[ ] Show either view count or popularity ${ related.view_count }

[ ] A user's related items should be on their home page

[ ] Related item plus sign is very poorly aligned within box

[ ] Add related items should be in drop down like for resources (?) to make it easy to add items

[ ] Hover should show a title that gives some information about what related are (not at all clear)

"Apps, Ideas, Visualization and other material related to this dataset"

[x] Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. "Apps, Ideas, Visualization and other material related to this dataset") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!)

• Went with Apps & Ideas (for now)

[x] Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like ​http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says

• What it does
• How to enable (and perhaps includes a nice-screenshot!)
• How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...)
• Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...)

[ ] Clearly mark the old extension as deprecated from v1.7 forward: ​https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7)

Install and document (step-by-step) CKAN (and dependencies, and extensions) onto VM on s031.okserver.org which has CentOS 6.2 installed. Paying particular attention to the fact that we won't have root access on the staging nor production servers, and so a) anything that requires root access must be documented carefully; and b) we want to be able to perform maintenance without being root.

Maintenance

We want to have non-root access to:

• CKAN logs; (relevant) apache logs; qa process' logs; rdf cron logs
• restarting CKAN
• restarting qa processes
• changing/disabling rdf cronjob
• our database (through sql)
• restarting solr

CKAN and Dependencies

• CKAN installed from source
• Required plugins: synchronous_search ecportal ecportal_form organizations multilingual_dataset multilingual_group multilingual_tag qa

Extra configuration

• uploads
• analytics (toby)
• qa
• rdf cron
• caching
• datastore

Things to remember

• Use ckanext/multilingual/solr/schema.xml and stopword files
• Use nginx as proxy.

An extra field on datasets called "metadata_language". The language that the meta-data is in. Drop-down list needs to be translatable. Default value is "English".

• Decent powered by icon for the Embed for the DataHub?? (make this text customizable and allow html ...)
  • Current icon and text looks really bad
• Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
• Data viewer embed url includes all the resource info (not needed -- just url and backend info ...)
• Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org

This action is not currently usable via a GET request as it relies upon a nested dict of fields.

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

• research ways to limit requests-per-process with the different web servers (see below for a start);
• explain these safeguards in the CKAN documentation and encourage users to apply them;
• consider the helper script ​ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of ​commented out. Or at least let it warn the user to use MaxRequestsPerChild;
• Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

• Apache:
  • Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on ​datahub.io and the ​ckan farm
  • (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
• nginx/supervisord: to be researched.

The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result.

TSV support doesn't seem to work very well, may be the mimetype ( text/tab-separated-values )

See ​http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96

The current behaviour of TemplateController?.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want. In general, it leads to:

• 500s when attempting to access a normal template (eg - ​http://datahub.io/importer/preview)
• A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template).

Solution:

• Change the controller to 404
• Ensure there's a way for existing ckan instances to override that behaviour should they need it.

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

As part of merging Organisations into core, it is necessary that we clarify the capacity field with which the users/datasets are added as members to the group 'subclass'.

Rather than the capacity being an opaque string that implies auth but doesn't clearly specify it, we will use role names where roles are defined in the database - with a clearly defined set of standard roles. The Role table is expected to have simply a string name/representation and acts as a container for permissions.

Each permission is a string of the form object.action (such as package.add, group.delete) of which several are expected to be associated with a role. This means the permission table will contain a string and a reference to the role.

This work will require UI changes to the screens allowing users to be added to a group/organisation so that the list of available roles is available to add those users.

[x] Model for Role and Permission

[ ] Logic layer changes for managing roles/permissions etc.

[ ] Determine default roles, perhaps just admin/editor/viewer

[ ] Fix the auth layer to use the permissions/roles - may be better implemented as another ticket.