{22} Trac tickets (2647 matches)

Results (2601 - 2647 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#537 task wwaites wwaites closed duplicate Caching and Performance improvement

There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests.

Introducing caching has to be done carefully and should be done in a standards compliant manner.

General strategy

  • Where possible, cache output within the pylons app (beaker).
  • Facilitate external caching in an end-user's web browser or a caching proxy
  • Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes).
  • When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data.

Tasks

These tasks should be broken into sub-tickets:

  • caching of parts of templates that are expensive to render (package list, tag list, group list)
  • caching of entire output using beaker particularly for API read operations.
  • need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches.
  • cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL
    • Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well
    • Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner

Future Work

  • Investigate ckanclient library maintaining a local cache as a web browser would
  • Investigate using a CDN like Google Storage or Amazon for serving cached data.
1283184362000000 1311178929000000
#576 defect sebbacon wwaites ckan-v1.3 closed fixed CKAN Requires Old Version of SQLAlchemy

Requires 0.4. 0.5 is a maintenance branch, 0.6 is current.

VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why.

It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well

1284141573000000 1294753848000000
#693 requirement wwaites ckan-v1.2 closed fixed Support ETag in cache decorator

... and make cache-control header optional

1286831051000000 1286831188000000
#843 enhancement wwaites closed fixed user list ui for group curation
  • perhaps we should show the openid as well to distinguish between users with the same name.
  • perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name.
  • also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it.
1291484245000000 1319721601000000
#844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org

ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example:

B30E6D9F       2873 Sat Dec  4 14:05:40  MAILER-DAEMON
(delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out)
                                         [email protected]

both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null

1291556753000000 1296340486000000
#851 defect wwaites wwaites ckan-v1.3-sprint-1 closed fixed Link Checking

revisit link checker from http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py

revisit ollyc's parallel work

look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)

1291722860000000 1293025112000000
#852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving

3 (4) use cases: (With and without access to auth keys.)

  1. upload form on ckan (by default local system has auth key for storage)
  2. datapkg upload (client) (by default without key) (plumbing done)
  3. background job archiving downloads (by queue client)

Most important are:

  1. Upload form on ckan.net (with access to keys)
  2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

  • #826 - Resource 'extra' fields
  • #358 - Resources in REST API
  • #235 - Resource format normalization and detection

User upload via Web UI

  • #877 - File upload in WUI
  • #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

  • #853 - plumbing for credential-less rest upload
  • #879 - Storage auth API

Also related:

  • #405 - Retrieval options for package resources

More Info

1291722993000000 1315821628000000
#853 enhancement wwaites wwaites ckan-v1.3-sprint-1 closed fixed Client upload to storage without having primary storage keys

Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys.

This would lead to an extension to OFS.

This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN).

1291723063000000 1294594581000000
#860 defect wwaites wwaites ckan-v1.3 closed fixed full text indexing for semantic.ckan.net

...

1291726029000000 1291726067000000
#861 defect wwaites wwaites closed invalid unicode errors creating rdf output

http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d

1291726125000000 1311168845000000
#862 defect wwaites wwaites ckan-v1.3 closed fixed deploy rdf on catalogue.data.gov.uk

handle various dgu extras, configure apache for autonegotiation, etc.

1291726237000000 1291726261000000
#883 enhancement wwaites wwaites ckan-v1.4 closed fixed uklii harvesting refactor

master ticket

  • #884 implement harvest job delete command for an easier life
  • #885 move csw code into dedicated ckanext plugin
  • #886 move forms api into dgu extension module
  • #887 move harvesting code to use generic harvesting module

1293277713000000 1300196622000000
#884 enhancement thejimmyg wwaites ckan-v1.3 closed fixed harvest job delete operation

implement paster subcommand to delete harvest jobs

1293277800000000 1296592140000000
#885 defect wwaites wwaites ckan-v1.3 closed fixed reimplement csw module to use owslib 1293278099000000 1294253508000000
#886 defect wwaites wwaites ckan-v1.3 closed duplicate move uklii forms api to dgu module

doesn't belong in ckan core

1293278145000000 1294916538000000
#887 enhancement wwaites wwaites ckan-v1.4 closed fixed move harvesting to generic module

in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat

1293278215000000 1300196600000000
#894 defect wwaites wwaites ckan-v1.4 closed invalid harvest from http://locationmetadataeditor.data.gov.uk/geonetwork/srv/csw 1294253876000000 1300196388000000
#1084 task wwaites wwaites ckan-v1.4-sprint-6 closed fixed ckan.net RDF links changed

need to make some changes for the links to semantic.ckan.net. it should use http://semantic.ckan.net/record/<package_id> now

append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table) to taste (or just leave off the suffix and let content negotiation take care of it)

the base url is changed, but it now uses id not name.

see for example:

1302616717000000 1304934534000000
#1087 enhancement wwaites ckan-sprint-2011-11-21 closed fixed version and contact info api call

a simple api call that returns data like this:

{ "version": ckan_software_version,
  "contact": { "name": "Some Admin", "mbox": "[email protected]" },
  "description": "Site Description",
  "url": "http://canonical.name.ckan.net/"
1302628944000000 1320866159000000
#1088 defect wwaites ckan-v1.4 closed fixed content-type autonegotiation is wonky

in ckan/controllers/package.py around line 130 it does some strange things...

perhaps replace with https://github.com/wwaites/autoneg

and handle redirection of these content types:

application/rdf+xml
application/turtle
text/plain
text/x-graphviz
1302630261000000 1303035487000000
#1309 enhancement zephod zephod ckan-sprint-2011-09-12 closed duplicate Registering new user requires immediate login

Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of "Recent changes" and no ability to modify the page. Users must follow the login link in the top-right corner to continue.

Registering a new user should immediately log you in.

1315225510000000 1315586335000000
#1348 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed UX polishing (dataset create/view/edit experience)

Tweaking UIs and polishing the work done in #1294.

## Edit Page ##

  • [Done; cset:49e0b059b6f5] Hide upload if storage not enabled
  • [Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like.
  • [Done; cset:03904fc50cbf] Move edit form tabs to the left
  • [Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file
  • [Done] We used to autocomplete the format field - restore this functionality
    • Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here?
  • [Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated)
  • [Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover)
    • NB: reveal_instructions.js already disabled on edit page
    • [RP] have now done this on create -- think we can leave on edit
  • [Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ...
  • [Done; cset:7ad090cb7855] Add slideDown to Resource expanded view
  • [Done; cset:d3595143e506] Need ability to delete resources
    • No thought yet given to uploaded resources. Long term re-think probable.
  • [Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text)

## General ##

  • [Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history)
    • More generally, the presence of the #minornavigation bar is very inconsistent throughout the site.
  • [Done; cset:d5babc3629cf] Remove 'license' filtered search from backend
  • [Done] Move script stuff to footer of page
1316170410000000 1317650043000000
#1349 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed Template tidying

Various things in templates can be tidied up after work has finished on #1294:

  • remove styling in page in group/index.html (put in css main)
    • ditto for package/search.html
    • ditto for home/index.html
  • Forms.css - to merge into style.css; remove explicit imports elsewhere
    • @import at the top of style.css @import url(forms.css)
  • Make a note: Success/failure controllers clash CSS with success/failure flash banners
1316171025000000 1317650050000000
#1350 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed i18n of new javascript

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

  • application.js
  • templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

1316172576000000 1317650056000000
#1351 enhancement zephod zephod ckan-sprint-2011-09-26 closed fixed Tidy edit-user form; strip preview functionality

The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater.

  • Fix the duplicate form
  • Strip the preview functionality from the front-end, replacing it with js editor
  • Strip the preview functionality from the back-end and test
1316534753000000 1317650062000000
#1368 enhancement zephod zephod ckan-sprint-2011-10-24 closed fixed UX polishing and bugfixes

From Ira: PDF of bugs and suggestions is attached.

This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in #1404.

DONE ?

  • [dataset/edit]: Deletions are confirmed with a dialog box
  • [dataset/edit]: Long URLs break the edit resources tab. Show name (& 'no name') with a pencil icon.
  • [dataset/edit]: Rearrange the delete resources mechanism;
    • Use a little minus icon on the far left, followed by a pencil icon for editing.
    • The 'delete resource' button can become 'hide'
    • Deletions are confirmed with a dialog box
  • [dataset/edit]: Rearrange the 'you have unsaved changes' warning;
    • Delete the little green plus sign on the right;
    • Try a flash banner instead (more general purpose & beautiful)
  • [dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset.
  • [dataset/view]: Subscribe button (& subscribe to history) doesn't work as expected. Modal dialog carrying instructions?
  • [dataset/view]: Drop the 'add to a group>>' and 'add tag>>'. Needs inline tag-adder.
  • [markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out.
  • [dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels.
  • [groups/add]: Hide unimportant fields
  • [navigation] Add 3D bevel (@see github) to current tab highlight.
  • [dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see http://groupspaces.com/a/group/
  • [groups/view]: Show description and owner. To match dataset section.
  • [groups/authz]: Tidy to match admin/authz (pretty buttons & spacing). Add navigation bar.
  • [groups]: Entire section to use pretty buttons please.
  • [about]: Get the about page working on test.ckan.org. (Works locally and on live).

WONTFIX ?

  • [dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold. (wontfix)
  • [groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'. (ticket #1403)
  • ([groups]: (Investigate) Is it easy to implement a little search-groups feature?) (ticket #1403)
  • [*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...) (wontfix: we need to rethink this mechanism; authorization refactor is coming)
  • [general]: 'Authorization' refactor to 'Permissions' on nav-bars. (wontfix, further discussion. authorization might be less techy...)
  • [search]: (Investigate) Can we quickly & easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets? (wontfix: high cost and maintenance overhead; need compelling user story)
1317391960000000 1318847910000000
#1369 enhancement zephod zephod ckan-sprint-2011-10-10 closed fixed Remove showdown.js from the page download

ckan.js can be rearchitected to avoid instantiating showdown.js in the root namespace. That way, layout_base.html doesn't need to import the script (because we don't use the showdown-dependant parts of ckan.js).

1317392516000000 1318165086000000
#1377 defect zephod zephod ckan-sprint-2011-10-10 closed fixed Ckan admin repair

Integrating ckanext-admin into core has thrown up a number of problems:

  • Look & feel does not match the rest of the site
  • Tests are not passing
  • On the trash page, clicking 'undelete' triggers a purge
  • Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (& several corrupt datasets have been found on thedatahub.org)
  • Trash page can contain nested form tags in certain cases (breaking test harness & form redirection)
1318240018000000 1318245795000000
#1378 enhancement zephod zephod ckan-sprint-2011-10-24 closed fixed New to TheDataHub? - StackOverflow style banner

Implement a banner across the top of the screen giving newbies a link to the about page, and making them feel welcome. See stackoverflow for reference?

1318247329000000 1324047835000000
#1379 defect zephod zephod ckan-sprint-2011-11-07 closed fixed Markdown conversion of linked references broken

eg. http://thedatahub.org/dataset/us-gov-tarp

[This Paper][my-paper] is very interesting.

[my-paper]: http://somelink.com/paper

Write a test to check this markdown feature works - if not, use external markdown lib rather than webhelpers.markdown. Probably need to add markdown lib to list of requirements.

1318247719000000 1320161439000000
#1380 enhancement zephod zephod ckan-sprint-2011-11-07 closed fixed Making customizing CSS easier

While integrating the new theme and making a large number of UX tweaks, I've allowed the CSS file to bloat out of control with some very specific rules and subtle hacks, as well as a couple of clearfix <div> objects in the markup which isn't really good practice.

In general we want a logical CSS structure which is as easy to modify as possible. You should not find yourself being overwritten by highly specific rules all the time!

Actions

  1. Clean up css
  2. Add a config option ckan.template_head_end - arbitrary string inserted at end of <head> tag so you can add custom css etc (may also want to update theming docs to reflect possibility of using this)
  3. (While we're at it): Add ckan.template_footer_end config option to deployment.ini_template. (This already exists, we just haven't told anybody)
  4. The /users index page isn't linked anywhere. Clean it up and link it in the footer next to Groups etc.
  5. package/new, package/edit, group/new and group/edit all include their forms differently. Use h.literal in all cases.
  6. Fix the padding on minornavigation.

Background

David Read sent me the following as background:

  1. basic theming is described here: http://docs.ckan.org/en/latest/theming.html

Deep customisation of the templates is done by creating an extension, with this as a template: https://bitbucket.org/okfn/ckanext-exampletheme/overview

That's what Augusto and the Brazilians have done http://alpha.dados.gov.br/dados/ and their code is here: hg clone http://dev.dados.gov.br/codigo/dev/tema-ckan <snip> Another thing of interest was us becoming compatible with Wordpress themes last December. Richard Pope worked with Rufus on this. I believe the base theme is 'twentyten' and I'm guessing that one would swap that css file with another to change theme, but I don't know. I don't know if this feature has been dropped since then. http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html

Dave

1318263956000000 1324048324000000
#1403 enhancement zephod zephod ckan-backlog new Refactor groups index page

Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain.

Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column.

Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible).

The forthcoming groups refactor will probably have some bearing on this task.

1318847512000000 1318847566000000
#1404 enhancement zephod zephod ckan-v1.7 closed wontfix Search Page UI improvements

[Refactored] :: Follows on from #1506 UX changes.

  • Declutter the sidebar. No yellow box.
  • Facets to go on the left, rather than the right. More logical flow.
  • Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
  • Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
    • Update that look-and-feel to include the new resource icons created in #1506
1318847818000000 1338203639000000
#1406 enhancement zephod ckan-backlog new Re-enable RSS subscriptions

RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly.

Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?)

1318861327000000 1320930088000000
#1409 enhancement zephod zephod ckan-sprint-2011-11-07 closed fixed Add Gravatars to user profiles

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

[Reopened: Additional] Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history.

1319200773000000 1324047722000000
#1410 enhancement zephod zephod ckan-v1.5 closed duplicate Add Gravatars to user profiles

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

1319200774000000 1319213205000000
#1482 defect johnglover zephod ckan-sprint-2011-12-05 closed fixed Minor JS bugs

Three JS bugs have appeared:

  • The slug editor is not updating the text field again. [Fixed; commit:fadda17383]
  • The View Dataset page will stop executing javascript if there are no resources. [Fixed; commit:e54b089f6e]
  • The Add-Dataset and Add-Group URL editor shows the wrong URL for datasets and groups. Broken in this commit:

https://github.com/okfn/ckan/commit/5a01e67 by 'map.explicit=true'. JohnG: Can you look into this? [Fixed; commit:88e88380f6]

1321991091000000 1322062576000000
#1517 defect zephod zephod ckan-sprint-2011-12-05 closed fixed Accessing resource view sometimes auto-downloads the resource

Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file.

For example, being here: http://test.ckan.net/dataset/europeana-lod and clicking on any of the links to the first resource.

1323109647000000 1323175957000000
#1528 enhancement zephod zephod ckan-backlog closed fixed Gravatar next to My Account link on all pages

Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in.

This was suggested by Richard Cyganiak: http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html

1323197931000000 1330020444000000
#1788 defect zephod zephod ckan-sprint-2012-03-05 closed fixed IE7 compatibility problems

From Antonia Acuno, via Ira:

It appears that the current layout for the datahub does not work in IE7, which would mean a problem for us as most gov uses it and is one of those we have to support...any views?

Tested in several gov departments computers, so a common enough issue.

http://imgur.com/ip4xa

From Rufus: We haven't officially supported IE7 so far but can put engineering effort into this if absolutely needed. I note that IE 7 is now 3y out of date and is likely a security risk. Obviously this is something to ask Antonio about.

1328799250000000 1333133365000000
#2221 enhancement rgrp zephod closed fixed Tests are broken for some of us: NotAPairTreeStoreException

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: https://gist.github.com/2007985

1331318781000000 1331561095000000
#2222 enhancement rgrp zephod closed duplicate Tests are broken for some of us: NotAPairTreeStoreException

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: https://gist.github.com/2007985

1331318783000000 1331318825000000
#2247 enhancement zephod ckan-backlog new Resource preview glitch in some browsers

From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. http://i.imgur.com/KJaqz.png

1332246614000000 1332246614000000
#2286 enhancement zephod zephod ckan-sprint-2012-04-16 closed fixed Upgrade to boostrap v2.0.2

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

  • Switch to the latest minifed CSS + JS
  • QA-sweep the site, especially dataset/edit
1334251818000000 1338205222000000
#34 defect somebody zool closed worksforme Page Not Found - returns with 200 rather than 404 as it should 1163008961000000 1253781550000000
#2197 defect zydio ckan-backlog assigned Storage Metadata API: add/update not working with local file storage (Pairtree)

If OFS is configured with Pairtree to use a local file storage, all POST requests to add/update metadata ( /api/storage/metadata/{label} ) will fail.

This is due to the use of BotoOFS specific private methods in StorageAPIController.set_metadata(), eg: self.ofs._require_bucket(bucket), self.ofs._get_key(b, label), self.ofs._update_key_metadata(k, metadata) ... those methods can't be found in POTFS and this causes errors. The API should use only OFSInterface methods, or should conditionally make calls based off the actual type of self.ofs.

PS: I did set "ckan" as "Component" in the ticket because storage has been integrated back into the core in CKAN 1.6

1330421377000000 1346662128000000
#2198 enhancement zydio closed fixed API documentation is missing Storage Metadata API info

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on Storate Metadata API.

1330421743000000 1339771453000000
Note: See TracReports for help on using and creating reports.