id type owner reporter milestone status resolution summary description PosixTime ModifiedTime 1785 enhancement seanh seanh ckan-v1.9 accepted Replace 'Revisions' page with site-wide activity stream Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead? 1328632458000000 1340724312000000 2336 enhancement seanh seanh ckanbuild accepted Move Jenkins' install script into ckan core so it can be versioned 1335792914000000 1340639637000000 2341 enhancement seanh seanh ckan-v1.9 accepted Fix Jenkins issue when testing branches that contain model changes 1335876396000000 1340639472000000 2342 enhancement seanh seanh ckan-v1.9 accepted Fix Jenkins issue when testing branches with different solr schema versions 1335876445000000 1340639494000000 2357 enhancement aron.carroll aron.carroll demo phase 5 accepted Create build script for front-end resources Should minify and concatenate JavaScript and CSS as part of #2354 1335886181000000 1343220524000000 2362 enhancement toby kindly ckan-v1.9 accepted Improve plugin documentaion, including examples. Improve documentation for plugins and add examples. Could use ckanext-example as a base. 1335888791000000 1342085420000000 2416 enhancement toby aron.carroll demo phase 5 accepted Normalise resource/data types "Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json” See the following Basecamp thread for UI examples and discussion. https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056" 1337792946000000 1342617802000000 2430 enhancement seanh seanh ckan-v1.9 accepted Look into marking tickets as dependent on other tickets in trac 1337947086000000 1341234346000000 2431 enhancement seanh seanh ckan-v1.9 accepted Trac git integration 1337947128000000 1341234362000000 2466 task seanh seanh ckan-v1.9 accepted Fix spam handling on trac 1338213103000000 1341234539000000 2475 enhancement seanh seanh ckan-v1.9 accepted Upgrade trac.ckan.org to trac 0.12.3 "This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12. http://trac.edgewall.org/wiki/ChangeLog" 1338317176000000 1341234564000000 2477 enhancement seanh shevski ckan-v1.9 accepted Add extensions section to readthedocs Need a central up to date place for currently supported extensions (closest we have is http://wiki.ckan.org/List_of_Extensions) & their documentation. I suggest this gets added to a dedicated section of docs.ckan.org 1338393244000000 1343124123000000 2478 enhancement seanh seanh ckan-v1.9 accepted Get NeverNotifyUpdaterPlugin installed on trac.ckan.org 1338396734000000 1341234591000000 2493 enhancement aron.carroll toby demo phase 5 accepted Demo site 404 needs some love "templates/error_document_template.html I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content just go to a non-existant url to see Discusion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/ Implementation: http://s031.okserver.org:2375/404" 1338986081000000 1343135815000000 2516 enhancement seanh seanh ckan-v1.9 accepted Make 'Assign to:' field on trac.ckan.org into a dropdown list there's a setting for this 1339578442000000 1341234822000000 2537 enhancement seanh seanh ckanbuild accepted Test and document ckanbuild "https://github.com/okfn/ckanbuild Verify that what's there so far still works, write a README explaining how it works" 1339775328000000 1340639830000000 2538 enhancement seanh seanh ckanbuild accepted Add multiple-instance support to ckanbuild "Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command. May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command." 1339775499000000 1340639836000000 2539 enhancement seanh seanh ckanbuild accepted Investigate the existing ckan debian package for ckanbuild Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh? 1339775661000000 1340639845000000 2540 enhancement seanh seanh ckanbuild accepted Implement a way of upgrading ckan sites using ckanbuild When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once. 1339775740000000 1340639850000000 2541 enhancement seanh seanh ckanbuild accepted Add non-core extensions to ckanbuild We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include. 1339775826000000 1340639856000000 2542 enhancement seanh seanh ckanbuild accepted Create jenkins job to run ckanbuild, and run tests It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests. 1339775888000000 1340639863000000 3021 enhancement johnmartin johnmartin accepted Logout doesn't work without JS "Essentially, the functionality should be as follows: - Add logout link that has `.js-hide` attached to it within the header that isn't hidden within a dropdown See http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS" 1353410350000000 1353411968000000 140 enhancement rgrp ckan-backlog new News section on front page "Have a news section (suggest as a sidebar item). News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org. Details: * Suggest pulling via rss or similar. * Will want to cache this ... Cost: 4h?" 1254902541000000 1265625159000000 331 enhancement rgrp ckan-backlog new Timezone of CKAN timestamps should be configurable "Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either. Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone." 1275302440000000 1339774701000000 351 enhancement dread ckan-backlog new Homepage: list new, updated and 'hot' packages "Have a simpler list of exciting data, as opposed to the big revision list. For example: {{{ Hot data =========== New packages: package1, package2, package3 Updated resources: package1, package2, package3 Popular packages: }}}" 1276595816000000 1339774677000000 369 enhancement shudson@… ckan-backlog new """Package Listing Key"" should appear on Tag results" "Currently there's a nice legend titled ""Package Listing Key"" that appears in right side of ""Browse Packages"" results. The same key should show on other search results like when searching for a tag." 1279821634000000 1339774666000000 370 enhancement shudson@… ckan-backlog new Use better email encryption for author_email and maintainer_email "The JavaScript email encryption used is not very reassuring. Google's MailHide is a much better solution that is easily implemented. http://www.google.com/recaptcha/mailhide/ Check on the Mailhide API where there are even some Python libraries already built." 1279821819000000 1339774649000000 653 requirement dread ckan-backlog new Trackback links for packages "When people link to a package, a track-back link is auto-created. (Similar system as for blogs). As suggested by Tim Davies: Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already. I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset. " 1285062025000000 1339774636000000 737 enhancement dread ckan-backlog new Markdown syntax summary page "I suggest we produce a quick Markdown cheat-sheet page, showing the key runes: e.g. create a title and quote some text. This page can link to the full Markdown docs for advanced users. A user going to the Markdown docs that we link will have to read a couple of pages of the raison-d'etre of Markdown before he gets to the syntax. And it's not very easy to read, and being white on black it looks like proper geek stuff. " 1287766749000000 1323170239000000 811 defect cygri ckan-backlog new Extra field editing form layout breaks when there are long field names The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See http://ckan.net/package/edit/dbpedia for examples. 1289994812000000 1323170289000000 812 defect cygri ckan-backlog new Package edit form only allows three extra fields "= Rationale = The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview). = Implementation = A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has) Nice to have: a blank field is added when you tab from the last filled-in field in the table." 1289995010000000 1311176917000000 818 requirement cygri ckan-backlog new Rethinking the author and maintainer fields "The semantics of the Author and Maintainer fields are really unclear at the moment. This leads to very inconsistent usage. Also, perhaps Name and Email are not the only fields that are needed for a contact. Here is a table that shows the current usage of these fields in CKAN: http://richard.cyganiak.de/2010/ckan/ckan-ppl.html We note several problems: * Author and Maintainer are often the same * Author and Maintainer are often used interchangeably * People really want to specify URLs for the contacts and stick them into random places because there is no field for it * Multiple comma-separated names in a single field I'm not sure what to do about this, but a redesign is necessary in my opinion. Some ideas: * Remove the maintainer field? * Make really clear that Author doesn't refer to the metadata on CKAN, but to the original data * Add an “author URL” field?" 1290003524000000 1339774621000000 837 enhancement rgrp ckan-backlog new CKAN integration with freebase gridworks / google refine "Thread: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html === Scenario 1 === 1. User installs Refine and CKAN extension for refine 2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance) 3. They edit the dataset on Refine 4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package) NB: for the dataset sync back some form of ""CKAN"" storage would be required (we already have storage.ckan.net running but a closer integration would be required) === Scenario 2 === 1. User visits a package on CKAN.net (or another CKAN instance) 2. There is a button on the page ""View and edit this dataset in Google Refine"" 3. Click button -- ask them if they have Google refine installed * Yes: instructions for loading dataset into refine * No: load dataset in hosted version of google refine (we could run this) 4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset." 1291140609000000 1339774605000000 895 defect memespring ckan-backlog new Add version number (or simular) to css/js includes query string "Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g. mycssfile.css?v=12345678" 1294343382000000 1339774593000000 924 enhancement dread ckan-backlog new Search box has no search button "The search box at the top-right of CKAN's page doesn't have a 'go' button. I feel that a larger percentage of users expect a 'go' or 'search' button on the right-hand side of the box to press to start searching. Techies tend to know the keyboard shortcut of pressing 'carriage-return' but it might be better to follow standard practise on this. Examples with 'search' button: Internet Explorer, Firefox, Google, Amazon, trac Examples without: ?" 1295867533000000 1323170436000000 989 enhancement kindly pudo ckan-future new Extending the model from plugins "We need to support extending the model from plugins. This could involve: * Adding a plugin hook to extend the mapper * Adding an upgrade hook for plugin schema migrations * Documenting how this is to be done * Find a way to avoid conflicts " 1297689724000000 1340034311000000 1009 enhancement pudo rgrp ckan-backlog new Improvements to user accounts sytem " * Forgot password (email a new password) * Confirm email * Do not show register page if you are logged in (redirect to home page) * ticket:1010 - listing of users. * Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount) " 1298635991000000 1310128574000000 1077 enhancement kindly rgrp ckan-backlog new Move to simpler vdm system "== Option 1: 'Changeset' Model == See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this. Have developed a new ""changeset"" based model for revisioning in vdm. === Implementation === * The main challenge with this change is schema and data migration Every revisioned object has a revision_id and revision attribute. Approximate algorithm: {{{ Revision -> Changeset for revtype in [PackageRevision, ...]: for pkgrev in package_revision: changeset = lookupchangeset(package_revision) ChangeObject(cset, (table, id), dictize(pkgrev)) }}} Question: * does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter. == Option 2: Simplify Revision Object Model == Just use a simpler vdm, see ticket:1136 (move to !SessionExtension) and ticket:1137 (remove need for statefulness in vdm). == Discussion == Advantage of Option 1 versus 2: * Easier support for pending state and similar behaviour * No need to introduce new tables (and hence migrations) when making something revisioned (or not). === Disadvantages === * Migration is required * More difficult to query revision history. * Could be addressed by having !ChangeObject have separate cols for table name and id but would likely be more difficult. * Performance (?) * Have one big !ChangeObject table to query when looking at changed objects rather than many revision tables. * Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the !ChangeObject table == Conclusion == Implement Option 2 and leave Option 1 for present. Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2). Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date." 1302304464000000 1340034345000000 1096 defect rufuspollock pudo ckan-future new [super] CKAN Hosted "Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas: * introduce model.Site and c.site * site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI) * Subdomain detector to activate sites. * use site in Authorizer instead of System, have a NullSite for global things * allow cross-site search * packages are in a list of sites, m:n rather than 1:n * list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally. " 1303235062000000 1339774484000000 1101 enhancement sebbacon ckan-backlog new Integrate googlanalytics into site nav "There's a stats plugin (e.g. at http://trac.ckan.org/ticket/832). Output from the googleanalytics plugin should append to that page, if the stats plugin is present. Possibly the stats plugin and the googleanalytics plugin should be merged? Finally, if the stats plugin is active, then a link to the stats page should be added to the main site footer." 1303393926000000 1339774582000000 1120 enhancement tsm ckan-backlog new Atom feeds of each tag "Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See [1]. [1] http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html" 1304309285000000 1339774568000000 1134 CREP amercader ckan-backlog new CREP0003: Description and Configuration of Harvesters "'''Proposer''': Adrià Mercader == Abstract == The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to: * Expose their type and other details so they can be used internally and on the UI. * Define configuration settings for particular harvester instances. == The Problem == === Harvester description === The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current {{{get_type}}} method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. ""CSW Server""), making it error prone. === Arbitrary configuration === In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the {{{IHarvester}}} interface and calls the appropiate methods for the current stage ({{{gather_stage}}},{{{fetch_stage}}},{{{import_stage}}}). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code). == Specification == === Harvester description === Harvesters will need to provide the following information so the UI form can be built: * name: machine-readable name (e.g. ""waf""). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester. * title: human-readable name (e.g. ""Web Accessible Folder (WAF)""). This will appear in the form's select box. * description: a description of what the harvester does (e.g. ""A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents""). This will appear on the form as a guidance to the user. The way to provide it will be an {{{info}}} method that all harvesters must implement, which will return a dictionary with the previous elements: {{{ { 'name': 'csw', 'title': 'CSW Server', 'description': 'A server that implements OGC's Catalog Service for the Web (CSW) standard' } }}} === Arbitrary configuration === As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called {{{config}}} in the database) (Maybe using JsonType?). This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an ""Add a configuration flag"" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an ""Advance options"" section. == Why do it this way == === Harvester description === The {{{info}}} method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface. === Arbitrary configuration === There is an already existing {{{config}}} field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime. == Implementation plan == === Deliverables === === Risks and mitigations === The highest risk on the harvesters {{{info}}} method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI. === Participants === Adrià Mercader to do it. === Progress === None yet." 1305108868000000 1339774554000000 1144 enhancement timmcnamara ckan-backlog new Support DSPL "DSPL, the Dataset Publishing Language, is being promoted by Google for its ""Google Public Data Explorer"" system. It is an XML format with metadata. The format is described on the [http://code.google.com/apis/publicdata/docs/developer_guide.html developer docs ofthe Google Code site]. Google provides a [http://code.google.com/apis/publicdata/docs/dsplgen.html Python script which reads CSV data and generates DSPL] Sample from http://code.google.com/apis/publicdata/docs/dspl_sample.html: {{{ #!xml My statistics Some very interesting statistics about countries http://www.stats-bureau.com/mystats/info.html Bureau of Statistics http://www.stats-bureau.com Geography Social indicators Population indicators Poverty & income Health Population Size of the resident population. Country My list of countries Country name The official name of the country State US states
Gender Gender, Male or Female Genders Both genders
unemployment rate The percent of the labor force that is unemployed, not seasonally adjusted. http://www.bls.gov/cps/cps_htgm.htm true
countries.csv
country_slice.csv
US states.csv
state_slice.csv
genders.csv
gender_country_slice.csv
}}} " 1305763609000000 1339774517000000 1145 enhancement timmcnamara ckan-backlog new Support the Handle System "The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot. Its documentation is here: http://www.handle.net/. Servers running CKAN could host a ""Local Handle Service"", which redirects a hash of a resource to an actual URL. Some suggested use cases: - Researcher would like to cite where data came from - Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform" 1305764775000000 1339774502000000 1152 enhancement amercader amercader ckan-backlog new True support for generic CSW servers "The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested. The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are: * Handling the validators (right now are hardcoded in the harvester code). This probably involves issues discussed in the CREP 3 (ticket #1134) * Changes in the model to adapt the specification to ISO 19139 * Renaming objects and classes which are now Gemini-centric List of CSW servers tested: https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0" 1306141334000000 1313411822000000 1163 enhancement rgrp rgrp ckan-backlog new Improvements to Storage Extension "Storage is now working but there are * Integrate with Resources (e.g. create a resource for each file upload and give option to associate with a package) * Should we introduce rule that files *not* associated with a Resource are periodically deleted? * Allow setting of a file name/path before upload * Allow for file overwriting/deleting etc (how should this work -- do we want to allow this sort of thing) * Integrate local file upload stuff in api/auth/* == Different Backend Issues == Local file store is rather different from 'remote' storage in various ways: * For remote you don't want to use many buckets as there are bucket limits while for local you want to. Should we there have a single path that users provide which we then partition differently for different backends. " 1306408778000000 1310133808000000 1165 enhancement nils.toedtmann ckan-future new Add multi-site support to ckan "Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3). That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's """", or tracd). Depending on the ""Host:"" header in the HTTP1.1 request, it would choose which local ckan ini file to obey. I see two ways to constitute the map hostname-to-ini-file map: * ckan reads a set of ini files, and each ini file declares which servers names it is responsible for * In a global ini file, there are directives mapping servernames to ini files. In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc." 1306413667000000 1339774466000000 1179 enhancement timmcnamara ckan-backlog new Support tag aliases "A small number of tags are near-duplicates of each other. Perhaps we could support word stemming from NLTK and/or manual tag aliases: statistics statistik ... survey surveying surveys" 1307429221000000 1339774332000000 1182 defect timmcnamara ckan-backlog new "Comments from deleted packages appear in ""Recent Comments"" feed" "When a package has been deleted, say for spam moderation, comments still appear in the recent comments section. This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link. At CKAN.net currently, this affects the most recent comment." 1307658251000000 1339774319000000 1184 enhancement timmcnamara ckan-backlog new Support Wuala as CKAN storage option "Most of CKANs storage options are tied to the USA. This brings concerns of data security for some organisations who may wish to adopt the system. [http://wua.la Wuala] is a distributed file system that stores data in a peer-to-peer manner. The company behind it, LaCie sells storage for a fee. However, they also enable clients to have 'free' storage space when machines act as a storage node. In order to be a storage node, a machine needs to be online for more than 14% of the time - roughly 4h per day. Most CKAN servers are likely to have a far greater uptime than this. Supporting Wuala would go some way to enabling CKAN to be used in a secure manner. That is, CKAN could be promoted for organisational use where there is lots of data to be stored and large geographic distances to be managed. There is a [http://www.wuala.com/Wuala%20API/Sample%20Clients/Python/wualaAPI/ Python client available] and a fairly [http://www.youtube.com/watch?v=3xKZ4KGkQY8 long Google Tech Talk] that overviews the system." 1308034751000000 1339774306000000 1185 defect timmcnamara ckan-backlog new Administrators can't delete packages from web UI "Administrators have ""View"", ""Edit"" and ""History"" tabs. However, I can't see a way to delete a package from the web UI. Version: CKAN.net as of today" 1308111818000000 1339774289000000 1188 enhancement nickstenning ckan-backlog new Allow diffing against initial (blank) package version "Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package. I'm requesting the ability to diff against a ""blank slate"" initial state of a project, so I can see the content of the first project commit. Not sure if this is a vdm feature, so I'm putting this ticket in against ckan." 1308153160000000 1339774275000000 1198 enhancement dread ckan-backlog new Publisher hierarchy "'Publisher' entities in the model. They are hierarchical. 'User-Publisher' connections with one or more roles (e.g. drafter, moderator). Authorization settings can control who can set what values in a 'published by' type field. Publishers and User-Publishers available to read in the API. Future tickets will provide: * API to write Publishers and User-Publishers * UI to edit Publishers and User-Publishers (This feature deprecates authorization groups)" 1308820592000000 1339774200000000 1201 enhancement kindly ckan-backlog new seperate out logic in atom feeds to logic layer. Simplify the logic in the atom feed an make all feeds use logic layer to return lists. 1308928892000000 1310124297000000 1203 defect johnglover rolf ckan-backlog new "Moderated edits: html code shows as ""changed"" although it is not" "I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML. The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code — and in the rendered table that is an —" 1309274970000000 1313401579000000 1227 enhancement timmcnamara ckan-backlog new Display packages' tags in search results "In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information. ENV=datacatalos.org, with CKAN 1.4.2a" 1311034262000000 1339774147000000 1232 enhancement thejimmyg ckan-backlog new [super] Interface improvements "Child tickets: * #1194 ""Welcome back"" message for newly registered user * #1202 Links to datapkg utility don't lead to info about it * #925 Change the search box icon to remove the down arrow * #923 Search box doesn't work in leaderboard page in stats extension * #1034 Flash message cached * #737 Markdown syntax summary page * #811 Extra field editing form layout breaks when there are long field names " 1311178296000000 1315948536000000 1233 enhancement thejimmyg ckan-backlog new [super] Improve wiki-style functionality for history "At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas: * #191 Searching by modification date * #193 Searching by time-related field * #301 Package discussion pages * #1236 Package history page should provide links to pages at particular revisions, similar to the wikipedia pages * #1236 Viewing old revisions or unmoderated changes should have a message at the top of the package page * Other improvements as per my word doc." 1311179392000000 1315948668000000 1235 enhancement thejimmyg ckan-future new [super] Search Improvements "Child tickets: * #234 UI Review - Autocomplete package names & tags in search * #193 Searching by time-related field * #191 Searching by modification date * #905 Unable to search with accented characters in package names * #906 Ability to search without accents for accented words * #924 Search box has no search button Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two." 1311182641000000 1311182641000000 1257 enhancement dread ckan-backlog new Anti-Spam tools "We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible. Spam: * Package creation * Package edit * User creation * Group creation Systems to consider: * Automatic * captcha * bayesian scoring * Sysadmin * a tool to quickly analyse and remove packages, edits and users * Helpful users * button to click indicating spam, leading to any/all of: * add TODO item (which emails package admin and sysadmins) * quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it General thoughts: * We should learn from the wikipedia tools, as they are the experts * This is a serious problem that is only going to get worse and is a threat to the whole site." 1312283565000000 1339774129000000 1259 enhancement johnglover pudo ckan-backlog new """Add a row"" for Extras on Package form" "The default package form offers 4 empty extras fields. Like the resource section, it should have an ""add more"" button to add another row. " 1312302693000000 1312907056000000 1260 enhancement pudo ckan-backlog new Remove duplicate functions from _util.html There seems to be both a list view for dictized and non dictized data structures for all entities in _util.html at the moment. Probably in the back of someone's mind already, but cleanup here would be nice. 1312366652000000 1313401499000000 1261 defect pudo ckan-backlog new Investigate dots in extras search It seems that searching for extras_foo:value works with solr, but extras_foo.bar:value doesn't. No theory on why. 1312366768000000 1312366768000000 1262 enhancement pudo ckan-backlog new "Enforce ""create-user"" permission" This does not seem to have any implications at the moment, it should lock down registration and remove all related links. 1312375296000000 1323090112000000 1273 requirement amercader ckan-backlog new Create docs for API v3 1313412083000000 1313412083000000 1278 enhancement amercader ckan-backlog new Refactor authorized_query calls "There are some functions that still use the Auhtorizer().authorized_query method: {{{ ./ckan/controllers/authorization_group.py:24: query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup) ./ckan/lib/base.py:237: groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, ./ckan/lib/search/sql.py:55: q = authz.Authorizer().authorized_query(username, model.Group) ./ckan/lib/search/sql.py:118: q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package) ./ckan/logic/action/get.py:154: query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT) ./ckan/tests/test_authz.py:158: q = self.authorizer.authorized_query(self.notadmin.name, model.Package) ./ckan/tests/test_authz.py:353: q = self.authorizer.authorized_query(self.notmember.name, model.Package) ./ckan/tests/test_authz.py:357: q = self.authorizer.authorized_query(self.member.name, model.Package) ./ckan/tests/functional/test_authorization_group.py:44: group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count() }}}" 1313415177000000 1313415177000000 1286 enhancement rgrp ckan-backlog new Remove remaining formalchemy stuff "Stuff I've spotted: * forms/* * template/group/edit_form.html * template/package/edit_form.html This can go once new DGU form is in." 1314116996000000 1342436420000000 1288 defect dread ckan-backlog new Package edit/creation can't include 'relationships' field "When you create or edit a package (via the API), you aren't able to specify the relationships it has. (If you do you get 409 {{{ {""__junk"": [""The input field __junk was not expected.""]} }}}) The normal way to create relationships is via /api/rest/relationships/ and this works. But when you GET a package, the dictionary lists all relationship details. So this bug creates a problem for editing a package that has relationships - you want to GET it, make any edits and then PUT it back. The work-around is to delete the 'relationships' key from the dict before you PUT it back. == Options == Ideally, CKAN would read the 'relationships' key and make the necessary changes. This is a chunk of work. Another good option is to allow an unchanged 'relationships' value, but barf it is edited. This is also a chunk of work. A bad option would be to just ignore the 'relationships' value, since users will get frustrated changing this value and wonder why it never saves, not understanding it is different to all the rest, without error message. A final option is to get rid of relationships altogether." 1314203695000000 1339774098000000 1311 enhancement rgrp rgrp ckan-backlog new Modal user register and login form "Subticket of: #1294 Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...). == Why == * It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...) * It allows for quicker and easier logging in == Implementation == * See Friedrich's work on the datahub" 1315297227000000 1315297227000000 1326 enhancement thejimmyg ckan-backlog new Write a set of auth plugin functions to integrate with Druapl Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor. 1315821084000000 1315821084000000 1336 defect johnglover dread ckan-backlog new License fudge "cset:4b59ab34137d ckan/logic/action/get.py: {{{ - isopen = model.Package.get_license_register()[license_id].isopen() - result_dict['isopen'] = isopen + try: + isopen = model.Package.get_license_register()[license_id].isopen() + result_dict['isopen'] = isopen + except KeyError: + # TODO: create a log message this error? + result_dict['isopen'] = False }}} This change hides problems with the license server and returns potentially incorrect values for openness. This has been noted as 'temporary fix' but seems to be forgotten about, since it has been merged to default and gone into release 1.4.3. I suggest the licenses are cached (I thought this was already the case when CKAN first requests them after start-up?). I suggest failure would return 503." 1315912057000000 1323173073000000 1343 enhancement rgrp rgrp ckan-backlog new [super] User related improvements (login, user pages etc) " * Disallow account creation via openid - #1386 * Require email field - #1319 * Require email confirmation to be activated (?) * Improvements to user page (e.g. show activity and more info about user) - #1396 * Modal user login - #1311" 1316017098000000 1318528138000000 1352 enhancement amercader ckan-backlog new Use logic functions instead of as_dict when indexing entities "The current search implementation uses the output of the the as_dict method of the domain Package object to update the index https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48 It also uses package_to_api1 in the SynchronousSearch plugin: https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93 This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model). The search should use the logic function to get the package properties: {{{ get_action('package_show')(context,data_dict) }}} " 1316615397000000 1339774086000000 1355 defect amercader ckan-backlog new Package extras property does not include the newly created ones "The extras in the package object sent to the extensions after editing (https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226) do not include the newly added. " 1317034126000000 1339774056000000 1382 defect thejimmyg thejimmyg ckan-backlog new Deleted resources are present for harvested package Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate. 1318337889000000 1318337889000000 1384 task rgrp shevski ckan-backlog new CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages Most articles still refer and link to ckan.net, wiki.ckan.net and to packages 1318414077000000 1318414077000000 1403 enhancement zephod zephod ckan-backlog new Refactor groups index page "Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain. Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column. Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible). The forthcoming groups refactor will probably have some bearing on this task. " 1318847512000000 1318847566000000 1406 enhancement zephod ckan-backlog new Re-enable RSS subscriptions "RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly. Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?)" 1318861327000000 1320930088000000 1411 enhancement zephod rgrp ckan-backlog new Force resource format to be lower case (also mimetype) "Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering). * Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?)" 1319319604000000 1319319604000000 1414 enhancement shevski ckan-backlog new track user log-ins on thedatahub.org "Set up tracking for user logins so that we have stats about how many active users of thedatahub exist want to be able to see who logged in the the last x months" 1319454782000000 1319454782000000 1423 enhancement markbrough ckan-backlog new Edit resources suggestions " * Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: http://thedatahub.org/dataset/edit/iati-registry * Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in." 1319641906000000 1338203678000000 1424 enhancement dread ckan-backlog new Openness notice should be clearer "ckan-discuss discussion suggests changes to the 'openness' indicator ( http://lists.okfn.org/pipermail/ckan-discuss/2011-October/001786.html ) Dataset view page: * If there is an explicit but non-OKD compliant license, such as CC-BY-NC, then this should be stated explicitly, perhaps: “This dataset is Not Open. License: Creative Commons Attribution Noncommerical. This is not an open license as it does not meet the Open Knowledge Definition.” * If the license is marked as “Other::License Not Specified”, then this should be stated explicitly, perhaps: “This dataset is Not Open. It is published without an explicit license, the publisher reserves all rights to the dataset.” * 3. If the license field was left empty by the contributor of the Data Hub record, then again this should be stated explicitly, perhaps: “This dataset is Not Open. The license of this dataset is unknown or unspecified. Start an enquiry on IsItOpenData » * There is a bug so that non-open licenses doesn't have an openness notice. * If downloadable resources are not available, this should not affect 'openness' - check this has been removed." 1319648089000000 1319648089000000 1429 enhancement rgrp rgrp ckan-backlog new Provide DOIs for datasets in a CKAN instance "DOI = digital object identifier = http://www.doi.org/ As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way. == Details == * Probably implement as extension rather than core" 1319977305000000 1319977305000000 1432 enhancement rgrp ckan-backlog new [super] Data processing system for CKAN and Webstore "Super ticket: #1190 A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)" 1320142747000000 1339774041000000 1438 enhancement dread ckan-future new Action API - parameter discovery/checking "Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error. Currently they are listed in the docs, extracted from the code manually. So you could GET /action/api/package_list to receive not only the help text, but a list of arguments. And if you send an extra or missing argument then an intelligent error message can be returned. = implementation = How about some sort of decorator on the action function: {{{ @logic_params(id, offset, limit) def get_package_list(context, data_dict): ... }}} This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function? I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code. Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict." 1320161890000000 1338205050000000 1439 enhancement dread ckan-backlog new Action API discoverablility "A good service API needs to be discoverable, so you are not always having to refer to the documentation html. Maybe /api/action should return a list of actions available? (Currently this returns a 404.) * It would be nice to sort these into get/create/update/delete. * #1438 Parameters for each of the actions must be discoverable too /api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error)" 1320161970000000 1325474974000000 1457 defect jilly.mathews ckan-future new Bug with DataNL instance " ""when logging into http://register.data.overheid.nl/ with OpenID, the /user/me page gives a 404. CKAN version 1.3.4."" n the manual it says an API key kan be created via http://test.ckan.net/user/me /. However, when I try the corresponding http://register.data.overheid.nl/user/me, I get a 404 error (not found). " 1320925041000000 1320925041000000 1459 enhancement rgrp rgrp ckan-backlog new Featured Dataset feature "Provide way to mark a dataset as featured. Featured database show up on the front page. TODO: detail this more." 1321113012000000 1321113012000000 1466 enhancement thejimmyg ckan-backlog new Need to support https login for multiple instances as part of the CKAN package install 1321375978000000 1328529062000000 1534 enhancement rgrp ckan-backlog new Change revisions to record userid rather than username "The use of username is problematic because username's can change. * Change all revision creation code to use user id (simplest is to change c.author field in lib/base.py (?)) * (?) Add a field ipaddr for ip address of anonymous users? (or just keep putting this in author field on Revision and then acception that those won't match when we do a look up against user table) * Change user view page to look up against user id rather than name * Perform migration on existing Revision objects * Match should probably be against both openid and username when searching Revisions' author field (especially true on CKAN where some people have already changed their username from being their openid)" 1323278790000000 1338205050000000 1535 enhancement dread ckan-backlog new Plump for auth header of: X-CKAN-API-KEY "When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default ""X-CKAN-API-KEY"". Rufus suggests having *one* way for this. a) making this not configurable any more b) making X-CKAN-API-KEY the default (keep Authorization allowed, but not documented, for backwards compatibility)" 1323279082000000 1339774019000000 1542 enhancement dread ckan-backlog new Buttons to purge spam datasets and groups "A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it. The existing two-stage revision delete is currently unreliable and perhaps too laborious. Olav and Richard have needs along this line." 1323364930000000 1339774000000000 1544 task dread ckan-backlog new delete old git branches "We have 150 odd branches ({{{git branch -a}}}) - most of them old - we should prune them. At very least, branches that have been merged in should be deleted. Look at old branches that haven't been merged in and wonder why. May be of some use: {{{ git branch --merged git branch --no-merged }}}" 1323702610000000 1323702610000000 1557 enhancement David Rasnik jilly mathews ckan-future new Complete Webstore Preview Extension "Finish any work out standing on web store preview extension to be able to package and release. Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted." 1324291253000000 1324291253000000 1558 enhancement David Raznik jilly mathews ckan-future new Publisher Tools Summarise final set of requirements for this and finish development and test. Estimated 10 working days. 1324291573000000 1324291573000000 1560 enhancement David Raznik jilly mathews ckan-future new Follow extension "Estimate 2 days to finish dev and test. David can you add any info needed here?" 1324291879000000 1324291879000000 1561 enhancement David Raznik jilly mathews ckan-future new To do extension Can we finish this ready for release on data hub and CKAN Hosted. 1324291972000000 1324291972000000 1562 enhancement Adria jilly mathews ckan-future new Finish Geo Spatial Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients. 1324292193000000 1324292193000000 1564 enhancement David Raznik jilly mathews ckan-future new Structured Data (Data API) "Basic websotre exists but this may be not what is described yet. CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first." 1324292834000000 1324292834000000 1565 enhancement Rufus Pollock jilly mathews ckan-future new Admin dashboard finished? Is testing complete and ready for release? 1324293092000000 1324293092000000 1567 enhancement David Raznik jilly mathews ckan-future new Finish QA extension Requires change to celeryd. Estimated 4 weeks. 1324293599000000 1324293599000000 1569 enhancement David Raznik jilly mathews ckan-future new Wordpressser How much effort will this be to be ready to use? 1324294056000000 1324294056000000 1572 enhancement David Raznik jilly mathews ckan-future new Meta data Harvester Need to write custom harvesters for each client. Is it worth having one for data hub? 1324294509000000 1324294509000000 1573 enhancement David Raznik jilly mathews ckan-future new Apps and Ideas Estimate 2 weeks for someone to finish and test. 1324294593000000 1324294593000000 1577 defect rgrp dread ckan-backlog new Can't upload file with foreign chars in filename "Looks like uploading a file with foreign characters fails due to encoding reasons. {{{ URL: http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:123 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckanext.storage.controller:2 in auth_form Module ckan.lib.jsonp:26 in jsonpify << Very much modelled after pylons.decorators.jsonify . """""" data = func(*args, **kwargs) return to_jsonp(data) >> data = func(*args, **kwargs) Module ckanext.storage.controller:301 in auth_form << method = 'POST' authorize(method, bucket, label, c.userobj, self.ofs) data = self._get_form_data(label) return data >> authorize(method, bucket, label, c.userobj, self.ofs) Module ckanext.storage.controller:79 in authorize << if method != 'GET': # do not allow overwriting if ofs.exists(bucket, key): abort(409) # now check user stuff >> if ofs.exists(bucket, key): Module ofs.remote.botostore:53 in exists << if bucket is None: return False return (label is None) or (label in bucket) def claim_bucket(self, bucket): >> return (label is None) or (label in bucket) Module boto.s3.bucket:87 in __contains__ << def __contains__(self, key_name): return not (self.get_key(key_name) is None) def startElement(self, name, attrs, connection): >> return not (self.get_key(key_name) is None) Module boto.s3.bucket:144 in get_key << response = self.connection.make_request('HEAD', self.name, key_name, headers=headers, query_args=query_args) # Allow any success status (2xx) - for example this lets us # support Range gets, which return status 206: >> query_args=query_args) Module boto.s3.connection:388 in make_request << if isinstance(key, Key): key = key.name path = self.calling_format.build_path_base(bucket, key) boto.log.debug('path=%s' % path) auth_path = self.calling_format.build_auth_path(bucket, key) >> path = self.calling_format.build_path_base(bucket, key) Module boto.s3.connection:88 in build_path_base << def build_path_base(self, bucket, key=''): return '/%s' % urllib.quote(key) class SubdomainCallingFormat(_CallingFormat): >> return '/%s' % urllib.quote(key) Module urllib:1222 in quote << safe_map[c] = (c in safe) and c or ('%%%02X' % i) _safemaps[cachekey] = safe_map res = map(safe_map.__getitem__, s) return ''.join(res) >> res = map(safe_map.__getitem__, s) KeyError: u'\xed' CGI Variables AUTH_TYPE 'cookie' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=259200' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; ckan_user=elenaibp; ckan_display_name=""Elena Mondo""; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; __utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10.1324298316; __utmc=119670349; __utmz=119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' HTTP_HOST 'thedatahub.org' HTTP_REFERER 'http://thedatahub.org/dataset/edit/budget-library-czeck-republic' HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7' HTTP_VIA '1.1 localhost (squid/3.0.STABLE19)' HTTP_X_FORWARDED_FOR '87.114.74.190' HTTP_X_REQUESTED_WITH 'XMLHttpRequest' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' REMOTE_ADDR '193.34.146.142' REMOTE_PORT '55419' REMOTE_USER u'elenaibp' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SCRIPT_URL '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SERVER_ADDR '193.34.146.146' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
Apache/2.2.14 (Ubuntu) Server at thedatahub.org Port 80
\n' SERVER_SOFTWARE 'Apache/2.2.14 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1324298703.071357, '_creation_time': 1324293077.4139669} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan.net' mod_wsgi.reload_mechanism '1' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 8) paste.cookies (, 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; auth_tkt=""f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode""; ckan_user=elenaibp; ckan_display_name=""Elena Mondo""; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; _ _utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10...)|utmcmd=(none)') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'} repoze.who.identity repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multiprocess' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'}) }}}" 1324317659000000 1325473564000000 1578 enhancement rgrp ckan-backlog new [super] Re-enable and refactor ratings 1324322443000000 1325473015000000 1581 enhancement mark.wainwright@… johnglover ckan-future new Blog post about Google Analytics extension for CKAN "The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post. Can ping John Glover in January for any details required. Key link is: http://thedatahub.org/analytics/dataset/top though this should probably move to be under stats (e.g. http://thedatahub.org/stats/usage)" 1324402800000000 1325474274000000 1584 enhancement johnglover johnglover ckan-backlog new QA report improvements - 2.5d "Super: #1594 * qa/{username} * qa/{groupname} * paginate QA results * search / filter QA results * ~~list organisation report by default, but can disable via config option~~ (done) * UX tidy up of report pages - hide border if no sidebar, etc" 1324459433000000 1338981975000000 1588 enhancement johnglover johnglover ckan-backlog new QA - Give SPARQL endpoints a 4 star rating "Super: #1594 From Richard Cyganiak on the CKAN Discuss list: Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type ""api/sparql"". A few more are recorded with the format ""SPARQL"". I suggest that datasets with such resources should also be considered for the fourth star." 1324480405000000 1325475178000000 1589 enhancement johnglover johnglover ckan-backlog new QA - Give 5 star rating to datasets with link metadata "Super: #1594 From Richard Cyganiak on the CKAN Discuss list: Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field." 1324480600000000 1325475095000000 1596 enhancement dread ckan-future new Refactor authz roles "Suggestions from rgrp: * Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones. * Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system. * Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages) (This ticket is split off from #1065)" 1324549888000000 1338205019000000 1598 enhancement rgrp ckan-backlog new Reinstate Ratings "Ratings were disabled approximately a year ago because: * Unclear purpose and UX. What did ratings tell you? How useful were they? * Spamming (esp by bots: you could submit an anonymous rating via a GET request which caused problems) Both problems are solvable and it would be nice to have this feature reinstated. * Purpose: can make this more purposable by limiting to logged in users (or at least distinguishing logged in from non-logged in users) * Even better we could allow ratings to be made public (I'm interested in what someone else I respect finds important) * Spamming: limit to logged in users and / or use AJAX over an API to submit ..." 1325177524000000 1325474818000000 1604 enhancement dread ckan-backlog new Get ckanext-moderatededits working with CKAN 1.5+ templates "ckanext-moderatededits requires an old and possibly development version of CKAN. It would be good to update it for later CKAN versions. According to the README, you need CKAN from branch feature-1141-moderated-edits-ajax but the changelog suggests this branch went into version 1.4.2. So it possibly works with 1.4.2 and 1.4.3(.1). But CKAN 1.5 has revamped templates, so the genshi stream filters definitely don't work. BTW history_ajax/read_ajax calls have been deprecated in CKAN since 1.5.2a and will need fixing up to use the Action API too as part of this." 1325352429000000 1325352429000000 1606 enhancement dread ckan-backlog new metadata license config option "Add a config option to choose the metadata licence. Set default to Open Database License. Currently the dataset edit form says ""Important: By submitting content, you agree to release your contributions under the Open Database License."" This is hard-coded, but not suitable for when DGU uses the CKAN form - they use the OGL." 1325501130000000 1339773981000000 1635 enhancement seanh seanh ckan-backlog new Email notifications (e.g. for activity streams) "CKAN should be able to send email notifications to users. Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users. One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well. Need to implement (at least some of) #1634 before this can be implemented, in order to have something to send notifications about. Analysis here: http://ckan.okfnpad.org/27" 1326304587000000 1355141157000000 1642 defect pudo ckan-backlog new Extra link generators generate garbled HTML "I had a package descriptions with URLs that contain ""group:foo"". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link. Need to fix the parser. Text: Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als WebMapTileService (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5). Metadateneinträge zu den Daten im PortalU: * [apps4d_DOP40](http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&plugid=/kug-group:kug-iplug-udk-db_hh) * [apps4d_DISK](http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&plugid=/kug-group:kug-iplug-udk-db_hh) * [apps4d_DIRK](http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&plugid=/kug-group:kug-iplug-udk-db_hh) * [apps4d_DK5](http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&plugid=/kug-group:kug-iplug-udk-db_hh) One fix is quoting the URLs" 1326382171000000 1339773967000000 1643 enhancement shevski ckan-backlog new Add fixed tags to thedatahub for better browsing Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags 1326393293000000 1326393293000000 1644 enhancement shevski ckan-backlog new Order default dataset page by most downloaded resources on thedatahub "Instead of alphabetically as we do currently, alternatively by most viewed datasets for http://thedatahub.org/dataset" 1326393542000000 1326393542000000 1647 enhancement shevski ckan-backlog new add links to ckan discuss & dev to thedatahub In the footer as well as more clearly & directly on the About page 1326673852000000 1326707383000000 1667 enhancement seanh seanh ckan-future new Add an extension point for rendering activity streams "Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML. There needs to be an extension point where extensions can register their own rendering functions for particular activity types." 1326796151000000 1338204295000000 1668 defect dread ckan-backlog new repoze version discrepency "There's a discrepency in repoze.who versions between the source and package installs: * repoze.who - package 1.0.18 vs source 1.0.19 * repoze.who-friendlyform - package 1.0b3 vs source 1.0.8 We get a test failure [1] with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who. The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package. James suggests we 'do something horrible like dynamically patch repoze on CKAN import'. [1] http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode" 1326801746000000 1326801746000000 1679 enhancement dread ckan-backlog new Default roles problem "The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py). The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different. We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?" 1326823042000000 1339773923000000 1697 enhancement rgrp ckan-backlog new A Configurable list of states for a Dataset "Currently have 'active' and 'deleted' suggest also: * 'draft' * 'hidden' (Do we need both). Also write out workflows related to these." 1327400630000000 1338204189000000 1717 enhancement shevski ckan-backlog new [super] Search UX improvements "1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search) 2. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page). 3. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it. 4. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu 5. Blue search button should be displayed in line with the search bar, not underneath 6. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name" 1327603981000000 1330088539000000 1740 enhancement seanh ckan-future new Get rid of `from module import ...` "It's really bad to do {{{from module import *}}} and CKAN has a lot of them. I suggest a three-pronged approach: 1. Don't add any more of them. 2. When you're programming if you see an easy opportunity to remove one then do so. 3. At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things. We should also get rid of most or all of the {{{from module import foo}}} and {{{from module import foo, bar}}} statements. I think the right thing to do is just {{{import module}}} and then use {{{module.foo}}} in your code. But if you find yourself doing {{{module.foo.bar}}} then you may have a code smell. See: http://docs.python.org/dev/howto/doanddont.html" 1328094369000000 1328094884000000 1745 enhancement rgrp ckan-v1.9 new Dataset search UX improvements as of Jan 2012 "Changes to make search both more exploratory and more satisfying to use * Search query build - #1603 * Ability to add new facet fields ""live"" * That is add fields which then contain faceted options (a bit like data.hri.fi) * (??) Autocomplete / drop down on search (i.e. search while you type) * Dubious about value / cost ratio here Probably would involve to pure JS and HTML implementation. == Implementation == Probably require * API changes to expose solr style API directly #1737 " 1328224941000000 1340033358000000 1747 enhancement seanh ckan-backlog new Expire old activities Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits? 1328446589000000 1339773859000000 1748 enhancement seanh seanh ckan-future new Make activity streams conform to http://activitystrea.ms/ standard "At the very least we should make our JSON output conform to their spec: http://activitystrea.ms/" 1328450784000000 1338204337000000 1750 enhancement seanh ckan-backlog new Move ckan/lib/activity.py into the model Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers. 1328465888000000 1339773840000000 1782 enhancement zephod rgrp ckan-future new Improvements to datasets page UX (Feb 2012) " * Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?] * ~~Icons for *types* of resources in resource list~~ * List of 'Owners' (administrators) at top right of sidebar (a la group page) * Much prefer owner to administrator ..." 1328609441000000 1338205232000000 1787 enhancement dread ckan-future new [super] Improve RESTful API " * Lists of entities should be full URLs, rather than just the names * Discoverability - /api/v3/rest should list the entity types that can be listed This could be v3 of the RESTful interface." 1328702082000000 1328702082000000 1789 enhancement seanh ckan-backlog new Implement a tag_update() logic action function "So users can rename a tag and/or move it between vocabularies using the API. Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all. What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary)." 1328805413000000 1339773812000000 1790 enhancement dread ckan-future new Click to delete tags, rather than have all existing tags in the tag text box "From Pablo: Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box." 1328888674000000 1328888674000000 1800 refactor seanh ckan-future new Tidy up *_list() and *_search() functions in ckan/logic/action/get.py "For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed. Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality." 1329405129000000 1338204886000000 1815 task amercader amercader ckan-future new Reenable Sparql endpoint on publicdata.eu "Estimate: ? The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset." 1329757723000000 1338204641000000 1818 enhancement amercader amercader ckan-future new Spatial metadata editor "Estimate 2.5d Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset. We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers [1], [2]. Things to take into account: * Integration with the form sections. * Allow some degree of configuration (default extent, limit geometry types...) [1] http://openlayers.org/dev/examples/draw-feature.html [2] http://openlayers.org/dev/examples/modify-feature.html " 1329763941000000 1338203195000000 1823 enhancement rgrp ckan-backlog new Spring clean bin directory Huge number of accumulated (and likely unnecessary) scripts in /bin directory. 1329773331000000 1338203554000000 1824 enhancement seanh ckan-future new Add vocabulary pages "For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page. We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar." 1329845089000000 1338204958000000 1827 enhancement dread ckan-backlog new 'Register' link should be hidden if you not allowed to register "{{{ I have just deny visitors the create-user permission: sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message ""Unauthorized to create a user"" (when not logged in). But anyway that is an improvement. }}}" 1329924939000000 1339773730000000 2200 enhancement seanh seanh ckan-future new Add vocabulary_id option to tag_show() logic action function Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag. 1330533678000000 1338204900000000 2235 enhancement rgrp ckan-future new Group drop down on dataset edit should use chosen and sort groups by name 1331907357000000 1338204726000000 2244 enhancement toby toby ckan-future new clean up auth "* cache admins etc * aim to move check_access so independent of logic function * general spruce up" 1332177771000000 1338205402000000 2247 enhancement zephod ckan-backlog new Resource preview glitch in some browsers "From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. http://i.imgur.com/KJaqz.png" 1332246614000000 1332246614000000 2257 enhancement toby toby ckan-v1.9 new cleanup template vars "look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc" 1332513307000000 1340097071000000 2258 enhancement rgrp ckan-future new Customizable contributor agreement " * Customize text at bottom of forms * Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)" 1332751549000000 1338204747000000 2265 enhancement dread ckan-future new 'More Like This' for a dataset "When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar. SOLR has a feature for finding documents similar to a particular document, called 'More Like This'. We would like this for DGU." 1332865220000000 1339771350000000 2268 enhancement thejimmyg amercader ckan-backlog new UKLP Use same schema as form for harvested records Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach. 1333108606000000 1333108606000000 2277 refactor icmurray icmurray ckan-backlog new Use the new atom feeds in IATI Atom feeds have been implemented in core in #1593. The IATI-specific implementation can now be removed when IATI move to CKAN >= 1.6.1 1333386628000000 1333386628000000 2279 enhancement rgrp rgrp ckan-v1.8 new Write tutorial for our DataStore API / Data Explorer "Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this. This is documentation / example only (no code!)" 1333482263000000 1338205426000000 2282 enhancement toby toby ckan-future new log api calls for users "This was part of #2251 but is now it's own ticket. we need clarity on this. The use cases so far have been possible limiting api calls for users. so we want to monitor the user or ip address what about the api calls? do we want to record all calls or just some? do we want to record the type of call and to what level of detail? recording the url just seems to specific to me." 1334074725000000 1338210829000000 2287 enhancement seanh ckan-future new Consistent datetime and time period strings localization "Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support. For example, on the user index page right now there are time period strings like ""less than 1 month"" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly). ISO date format everywhere might be a good idea. Need to consider local timezone issues. Python standard library and Babel may have useful helpers for this. Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?" 1334254123000000 1339411445000000 2291 enhancement kindly rgrp ckan-v1.9 new DataStorer incorrectly attempts to store JSON in DataStore "See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670 This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do). Assigning to David for triage and review." 1334431147000000 1340636737000000 2298 enhancement icmurray icmurray ckan-v1.9 new Add sort-by controls to the search results page "Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page. * What should the sortBy controls/widget look like? * Which fields should be exposed? The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master. [1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c" 1334569162000000 1340635947000000 2301 enhancement seanh seanh ckan-future new Replace old-style string formatting with format() method (at least in strings marked for translation) "Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words. Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.: 'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible') http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated." 1334579264000000 1340635812000000 2310 enhancement icmurray icmurray ckan-backlog new Refactor the search-query construction in feeds.py "The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed). Refactor this to remove duplication." 1334592091000000 1337159386000000 2320 enhancement rgrp ckan-v1.9 new Introduce new state: archived Some datasets become obsolete: e.g. http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something. 1335211949000000 1340624119000000 2321 enhancement rgrp ckan-v1.9 new Use resource names/titles in URLs "Would make them much nicer but requires: * Ensure uniqueness (or deal with ambiguity) * (?) Slug/names versus titles * Aside: what is currently called name should probably be title " 1335236412000000 1340624174000000 2324 enhancement amercader ckan-v1.9 new Link API docs to relevant version not just latest "The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version. Also, remove the 'API' link (to /api/1) as pointless and misleading" 1335351532000000 1345199411000000 2326 enhancement toby toby ckan-future new dataset permalinks "create permalinks eg \dataset\~a5-DQ\nice-name ~XXXXX defines the permalink the nice-name is ignored * packages table needs a new field for this data * existing datasets need updating * new datasets need the identifier creating * routing needs updating * needs to work for all urls like edit Once this is implemented we can decide when we will use permalinks vs the normal urls." 1335371946000000 1338212799000000 2328 enhancement rgrp ckan-v1.9 new Link to Catalog API for each Dataset, Group and Tag At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page? 1335442501000000 1340624331000000 2333 enhancement rgrp ckan-v1.9 new Introduce new resource type datastore "This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly: * New Option: DataStore Table in add a resource form * URL field is disabled (or points at /api/data or something rather pointless) * Download button is disabled * Better would be to find a way to serialize an ES table or stream it ... " 1335651360000000 1340624473000000 2350 enhancement icmurray icmurray ckan-v1.9 new Hooks in package controller for validating form data "When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController subclasses to do form validation/processing. Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override. NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs. - [ ] Add hooks - [ ] Use in ckanext-example" 1335881761000000 1340635966000000 2363 enhancement toby kindly ckan-v1.8 new Documentation of best caching practice. Need better documentation on best practices in making page loads faster for non logged in users. 1335889017000000 1340099794000000 2382 task amercader amercader ckan-future new Investigate options for basic geocoding A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk) 1337017160000000 1338205325000000 2395 defect dread ckan-backlog new paster db clean/init don't work when spatial extension enabled "If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init. Can't you just modify the clean and init functions to run without extensions enabled? The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket." 1337159793000000 1339771313000000 2406 defect kindly markw ckan-v1.9 new Errors in source install doc "There are a few errors or unclarities in the source install doc - see http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html " 1337595093000000 1340636804000000 2407 enhancement amercader markw ckan-v1.9 new 'Access denied' message is unhelpful "This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied': http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html The problem was he didn't have access to the group. A message like 'You don't have access to the group ""LODcloud""' would have been clearer." 1337612500000000 1340633440000000 2411 enhancement toby markw ckan-future new .ini file is a mess The .ini file needs cleaning up in assorted ways which Toby will expand upon. 1337686900000000 1338210872000000 2415 defect kindly markw ckan-v1.9 new Confusing message on attempting to register new account with upper case "On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message: ""The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"" What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...' But even better would be to make the login case-insensitive and allow capitals to be entered. " 1337779189000000 1341268304000000 2426 enhancement kindly seanh ckan-v1.9 new Get rid of CKAN's formalchemy dependency 1337946462000000 1340616095000000 2427 enhancement kindly seanh ckan-v1.9 new Reduce cruft installed into CKAN's virtualenv due to Pylons 1337946507000000 1341268323000000 2433 defect toby toby ckan-v1.8 new API uses name not id for some version 3 calls see https://github.com/okfn/ckan/pull/5 1337957648000000 1340099820000000 2479 enhancement markw ckan-v1.9 new Meanings of Author and Maintainer fields are unclear "CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank. The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest: (1) Rename the fields 'Owner' and 'Maintainer'. (2) Change the explanatory text for the relevant fields: Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.' (3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile. " 1338452898000000 1339771115000000 2480 enhancement markw ckan-v1.9 new Better message when dataset has no resources "If a dataset has no resources the resources list currently says '(none)'. Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'" 1338453093000000 1339771086000000 2483 enhancement markw ckan-v1.9 new Non-local resources should not have Download links "At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally. This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table. Suggested fix: * Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource) * Disable the Download button unless the resource is stored locally. " 1338468734000000 1339771043000000 2485 defect icmurray nils.toedtmann ckan-v1.9 new Encourage leak containment by limiting the number of requests a CKAN process serves "CKAN has [ticket:1345 memory leaks]. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced. One should ... * research ways to limit requests-per-process with the different web servers (see below for a start); * explain these safeguards in the CKAN documentation and encourage users to apply them; * consider the helper script '''[https://github.com/okfn/ckan/blob/master/ckan_deb/usr/bin/ckan-create-instance ckan-create-instance]''' to generate Apache configs with '''WSGIDaemonProcess ... maximum-requests=XY''' being active instead of [https://github.com/okfn/ckan/blob/master/ckan_deb/usr/lib/ckan/common.sh#L262 commented out]. Or at least let it warn the user to use '''!MaxRequestsPerChild'''; * Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them. ---- How to limit requests-per-process * Apache: * Use '''WSGIDaemonProcess''' with '''maximum-requests=50''' or whatever limit is appropriate. We did this sucessfully on [http://trac.okfn.org/ticket/904 datahub.io] and the [http://trac.okfn.org/ticket/1245 ckan farm] * (This need to be verified) Without WSGIDaemonProcess, '''!MaxRequestsPerChild 50''' should achieve the same. * nginx/supervisord: to be researched. " 1338551229000000 1340636028000000 2486 defect seanh ckan-backlog new Should be able to use . in dataset names 1338655583000000 1339770978000000 2499 enhancement toby icmurray ckan-v1.8 new Documentation for the internal analytics "There's no docs for setting up the internal analytics, nor what it provides. == Setup == Add: `ckan.tracking_enabled = true` to .ini file Run: `paster tracking -c dev.ini` == Related links == http://trac.ckan.org/ticket/2251" 1339070297000000 1339070297000000 2508 enhancement seanh ckan-backlog new Make it possible to run CKAN tests for each language "Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language. This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc. In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files." 1339411335000000 1339770771000000 2524 enhancement kindly kindly ckan-ecportal new If there are no translation files for selected language fall back to default lang. If a user selects a language there are no mo files for then an error is raised. Revert to default language instead. 1339609048000000 1340117608000000 2529 enhancement rgrp ckan-v1.9 new DataHub (or CKAN) widgets "Simple widgets in pure JS. For example: * Count of datasets in a group (could generalise to a query but not sure how useful that is ...) * Embeddable list of top X (5) datasets for a given query * Embeddable list of *my* datasets Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query). Or: we have this attached to areas of site where relevant. Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page). Cf. http://okfnlabs.org/ckanjs/" 1339750049000000 1340624917000000 2530 enhancement kindly rgrp ckan-v1.9 new DataHub purge fails on some revisions "See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like: Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError) update or delete on table ""revision"" violates foreign key constraint ""group_extra_revision_revision_id_fkey"" on table ""group_extra_revision"" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table ""group_extra_revision"". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}" 1339750498000000 1341268280000000 2531 enhancement rgrp ckan-backlog new New state option: archived / deprecated "Deleted means things will get purged at some point. Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated. @richard cyganiak ..." 1339750787000000 1339770649000000 2543 enhancement icmurray ckan-v1.9 new facet.sort is not available in the package_search action "Not all solr facet parameters are available through the pcakage_search action. In particular, `facet.sort` has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too. See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort" 1340013335000000 1340633091000000 2555 enhancement toby aron.carroll demo phase 5 new Demo site needs a breadcrumb helper Something to make building breadcrumbs a bit nicer 1340026983000000 1342618384000000 2572 enhancement toby toby ckan-v1.9 new clean up stats plugin attempt to disengage the stats plugin from core as much as possible 1340105054000000 1340899524000000 2573 enhancement icmurray ckan-future new package_search does not allow solr's per-field facet parameters "Solr allows its facet parameters to be specified on a per-field basis, eg. `facet.limit` applies to ''all'' facet fields, but solr allows it to be overriden for a specific field, eg. `facet.tags.limit`. We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See `ckan.lib.search.query.VALID_SOLR_PARAMETERS`." 1340112439000000 1340633101000000 2579 enhancement toby toby ckan-v1.9 new move sort_by functions into lib.helpers "make these more available but keep existing functionality so not to break any users remove_field() drill_down_url() etc" 1340281798000000 1340899337000000 2582 enhancement rgrp ckan-v1.9 new Do not hide notes / readme on dataset pages "Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?) Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket." 1340312340000000 1340625009000000 2583 enhancement toby toby demo phase 5 new make sure that we implement authentication where needed in development many auth checks may have been lost we need to check they are still working etc 1340359478000000 1342086274000000 2585 enhancement seanh ckan-v1.9 new Escape solr control characters in search queries, add advanced search screen "Suggestion from David Read: We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for ""Spend over £25,000 - NHS Leeds"" then it would not come up with the dataset with that exact name. It was excluding datasets with the word ""NHS"" due to the dash/minus sign. It works fine if you escape the minus sign: ""Spend over £25,000 \- NHS Leeds"". So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine: http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/ Perhaps you would consider providing this in CKAN core in future? I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?" 1340360773000000 1340625078000000 2603 refactor icmurray icmurray ckan-v1.9 new Remove deprecated 'fields' parameter from resource_search The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result. 1340730601000000 1340730601000000 2621 refactor icmurray icmurray ckan-v1.9 new Remove the deprecated 'fields' parameter from tag_search and tag_autocomplete "This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict. See #2439 In a future release of CKAN (probably 1.9) it can be removed. Internal uses of it were removed in #2439, but there are tests that still use it." 1340900569000000 1340900569000000 2622 defect seanh new Login fails in Opera 12 "Try to login to CKAN using Opera 12, get ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)""" 1340902602000000 1340902602000000 2625 enhancement seanh ckan-v1.9 new Add i18n strings from non-core but supported extensions to ckan.pot file Have to decide which non-core extensions are going to be supported first. 1341236903000000 1341236903000000 2635 enhancement dread new Non-destructive SOLR reindex "You can't run the search-index reindex on a live server because it will give us bad results for 2 to 3 hours while it runs. Can there be an option that doesn't delete the entire index at the start? Instead it could just delete any items that don't exist any more, then delete them and regenerate them one by one. So the total number of datasets doesn't change much." 1341829394000000 1341829394000000 2656 defect seanh dread new Feed with few results has bad paging link, causing exception "This page http://thedatahub.org/feeds/custom.atom?q=wombat has 0 results and contains a link to http://thedatahub.org/feeds/custom.atom?q=wombat&page=0 which the page=0 causes this exception: {{{ ckan.lib.search.common.SearchError'>: SOLR returned an error running query Error: ""'start' parameter cannot be negative"" }}}" 1342001112000000 1342001112000000 2663 enhancement toby toby ckan-v1.9 new h.resource_display_name needs love "This function is shit and needs cleaning up and a doc string description is markdown and should be treated properly either we should truncate all or leave it to the templates but work universally url if no name / desc this is in demo-theme branch" 1342017746000000 1342017746000000 2673 enhancement rgrp new simplify set of options for resources Far too many resource options. Lets restrict back to data file and API. Visualizations etc can either get linked in description or in the Related items. 1342300559000000 1342300559000000 2679 enhancement icmurray icmurray ckan-v1.9 new Change default behaviour of TemplateController.view to 404. "The current behaviour of TemplateController.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file. Although this may be a feature that some instances want. In general, it leads to: - 500s when attempting to access a normal template (eg - http://datahub.io/importer/preview) - A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template). Solution: - Change the controller to 404 - Ensure there's a way for existing ckan instances to override that behaviour should they need it." 1342436133000000 1342436133000000 2683 enhancement seanh new Add no-cache header to _tracking API call's response to make sure it doesn't get cached 1342446577000000 1342446577000000 2688 enhancement ross new Allow ordering of groups in WUI Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed) 1342520875000000 1342520875000000 2708 enhancement kindly toby ckan-v1.9 new limit extra data for package/group show " context['package_limits'] = { 'tags': 5, <- get first 5 'extras': 0, <- get all } only get what you ask for have to be explicit context['group_limits'] = {} only main item start with datasets/groups expand if we like it" 1342622420000000 1342622420000000 2709 enhancement icmurray markw new Atom feeds are undocumented There doesn't seem to be any documentation yet for Atom feeds. 1342624310000000 1342626212000000 2718 enhancement toby shevski demo phase 4 new can't add dataset to more than one group "trying to add a dataset to another group means it's no longer part of the first group http://demo.ckan.org/dataset/edit/afterfibre" 1342780550000000 1344544203000000 2719 defect dread new Feeds controller does not catch NotAuthorized exception Results in bad user experience and WebApp errors emailed out. Seen in 1.7.1 1342872863000000 1342872863000000 2721 defect toby shevski demo phase 4 new deleted groups should not show on 'Add to Groups' dropdown "Groups previously deleted still show up in the add dataset process in step 3 'Additional info' http://s031.okserver.org:2375/dataset/new_metadata/ff" 1342948632000000 1344544214000000 2725 enhancement toby shevski demo phase 5 new Case sensitivity on tags "My feeling is that 'country-US' and 'country-us' should be the same tag. However currently tags with caps are treated differently see http://s031.okserver.org:2375/en/dataset/test-dataset with TEST and test - there also get indexed twice in the search page" 1342949667000000 1343030773000000 2726 enhancement toby shevski demo phase 5 new confusing logic on data preview formats "1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this 2. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28" 1342949927000000 1343030906000000 2728 defect toby shevski demo phase 4 new deleted group shows on search index - for admins 'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see http://s031.okserver.org:2375/dataset?groups=test-group 1342950784000000 1345023944000000 2729 enhancement kindly shevski ckan-backlog new searching for tags:[tag] works but tag:[tag] doesn't "which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work! http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc" 1342951109000000 1342951176000000 2731 enhancement markw new Some sites permanently 'down for maintenance' "A large number of XXX.ckan.net sites give the following message: ""This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins."" The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed): http://trac.okfn.org/ticket/933 However, the redirection only seems to have worked in one case, http://si.ckan.net. The problem still affects the following sites - the first 4 of which have supposedly been merged: * http://lt.ckan.net * http://hu.ckan.net * http://be.ckan.net * http://gr.ckan.net * http://nz.ckan.net * http://bg.ckan.net * http://fi.ckan.net * http://katalogdanych.centrumcyfrowe.pl * http://pl.ckan.net * http://southampton.ckan.net * http://sl.ckan.net Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate." 1343045168000000 1343051608000000 2733 enhancement johnglover johnglover ckan-v1.9 new Datastore logic functions "== Where does the data go? == In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url. The user should have rights to create tables. == Whats the api like? == We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern. == What are the initial logic functions? == * datastore_create * datastore_delete * datastore_show == What is the JSON input format for datastore_create == To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id. {{{ { resource_id: resource_id # the data is going to be stored against. fields: a list of dictionaries of fields/columns and their extra metadata. records: a list of dictionaries of the data eg [{""dob"": ""2005"", ""some_stuff"": ['a', b']}, ..] } }}} * The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions. * rows: can be empty so that you can just set the fields * fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering. eg: [{""id"": ""dob"", ""type"": ""timestamp"" }, {""id"": ""some_stuff"", ""type"": ""text""}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though. * Any error results in total failure!! For now pass back the actual error. * Should be transactional == What json does datastore_delete take? == {{{ { resource_id: resource_id # the data is going to be deleted. filters: dictionary of matching conditions to delete e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to ""delete from table where key1 = 'a' and key2 = 'b' "". No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table. } }}} == What json does datastore_search take? == {{{ { resource_id: resource_id # the data is going to be selected. filters : dictionary of matching conditions to select e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to ""select * from table where key1 = 'a' and key2 = 'b' "" q: full text query limit: limit the amount of rows to size default 100 offset: offset the amount of rows fields: list of fields return in that order, defaults (empty or not present) to all fields in fields order. sort: comma separated field names with ordering e.g ""fieldname1, fieldname2 desc"" } }}} Some free code: https://gist.github.com/3163864 == What json does datastore_search return? == {{{ { fields: same type as datastore_create accepts (i.e. with metadata) offset: The same offset that was supplied in datastore_show limit: The original limit filters: The filters that were applied in data_show total: # total matching records without size or offset records: [same as data_create] # list of matching results } }}} On error will return: {{{ { __error__: … sql error … } }}} == What types are allowed? == Aim to support as many postgres/postgis types that have string representations. http://www.postgresql.org/docs/9.1/static/datatype.html http://www.postgresql.org/docs/9.1/static/sql-createdomain.html == IDs == Each row in a table will be given an _id column which has an id generated by us which you can use in queries. == Other Features == Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc." 1343058886000000 1343656105000000 2745 defect amercader ckan-v1.9 new Password reset returns an exception if the key parameter is missing "Instead of showing a notice, the password reset page throws an exception if the key parameter is missing: {{{ Module ckan.controllers.user:329 in perform_reset c.reset_key = request.params.get('key') if not mailer.verify_reset_link(user_obj, c.reset_key): h.flash_error(_('Invalid reset key. Please try again.')) abort(403) if not mailer.verify_reset_link(user_obj, c.reset_key): Module ckan.lib.mailer:100 in verify_reset_link if not user.reset_key or len(user.reset_key) < 5: return False return key.strip() == user.reset_key return key.strip() == user.reset_key AttributeError: 'NoneType' object has no attribute 'strip' }}} Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param: http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8 instead of: http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8 " 1343145931000000 1343145931000000 2748 enhancement shevski demo phase 5 new add 'add new resource' button to sidebar "When editing a resource you see the current and any other existing resources in sidebar (e.g. see http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True) Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page" 1343212878000000 1344503744000000 2751 enhancement toby toby demo phase 5 new check translations for full demo site need to check everything gets translated - sean did this before so will have info 1343216443000000 1344243046000000 2758 enhancement toby toby ckan-v1.9 new file storage gives error if config not available but no useful user information "We get an error which should be improved the actual problem is this but not passed to user ⇝ KeyError: 'ofs.impl' View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view >> data = func(*args, **kwargs) Module ckan.controllers.storage:407 in auth_form view >> authorize(method, bucket, label, c.userobj, self.ofs) Module ckan.controllers.storage:200 in ofs view >> StorageAPIController._ofs_impl = get_ofs() Module ckan.controllers.storage:71 in get_ofs view >> storage_backend = config['ofs.impl'] Module paste.registry:146 in __getitem__ view >> return self._current_obj()[key] KeyError: 'ofs.impl'" 1343287709000000 1343287709000000 2761 enhancement seanh ckan-v1.8 new Document all the errors you can get when setting up filestore, and how to fix them "Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html For the error messages and their solutions, see various threads on ckan-dev" 1343302566000000 1343302566000000 2762 defect seanh ckan-v1.8 new test_related.py crashes "/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py ImportError (cannot import name assert_regexp_matches)" 1343303753000000 1343303753000000 2763 defect seanh ckan-v1.8.1 new Multilingual tests failing test_multilingual_plugin.TestDatasetTermTranslation.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation.test_group_read_translation both failing for me on master 1343303819000000 1350303864000000 2768 enhancement toby shevski demo phase 5 new normalise excel to xls so that data proxy works 1343319382000000 1344351663000000 2771 enhancement seanh seanh ckan-v1.8 new Documentation and examples for IDatasetForm and IGroupForm "Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests. The IDatasetForm example should use tag vocabularies (two birds with one stone) The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples. Tab Vocabularies docs should reference IDatasetForm example. When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe) There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date." 1343392238000000 1350303564000000 2773 enhancement markw markw ckan-v1.9 new About page needs improving The about page for the DataHub (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh. 1343646795000000 1343646795000000 2775 enhancement toby aron.carroll demo phase 4 new Add bin/less to paster serve command "Ideally the ./bin/less command would be run when the server is started. * Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }} * The command could also detect missing node binaries and redirect to the documentation." 1343685686000000 1344543962000000 2777 enhancement icmurray new bug: user attributes 1343726363000000 1343726363000000 2780 enhancement toby shevski demo phase 4 new way for admins to undelete datasets "Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?" 1343737248000000 1345023811000000 2784 defect icmurray icmurray new model dictize sensitive data The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not. 1343814685000000 1343814685000000 2786 enhancement shevski demo phase 5 new target blank HTML downloads e.g. if I click on download here: http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b can it take me to a new tab instead of navigating away from ckan? 1343819814000000 1343819814000000 2788 enhancement amercader amercader ckan-v1.9 new Speed improvements on creating/updating and indexing "Specially needed when importing large numbers of datasets. Profiling the import command from the harvesting extension has shown some areas where improvements could be made. " 1343832992000000 1343832992000000 2790 enhancement kindly toby demo phase 4 new logic.action.user_show is slow "This is a very slow call it would benefit from the sort of speed-ups that package-search received for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization can we just grab json blobs from solr? also is it possible to specify a sort order/paging? I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now" 1343852483000000 1345023734000000 2795 enhancement toby demo phase 5 new Check validation of HTML, CSS, JS Ensure that we are being standards compliant 1343903128000000 1343903128000000 2796 enhancement mark.wainwright ross new Need a datahub one-pager "A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data. " 1343924916000000 1345129495000000 2810 enhancement kindly ckan-future new heroku ckan support Get ckan working on heroku 1344364858000000 1344364858000000 2813 enhancement toby markw demo phase 5 new Confusing sidebar on demo dataset page "On a dataset page on demo.ckan.org, the left sidebar is confusing. * It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear. * The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll)." 1344420206000000 1344445419000000 2815 defect seanh seanh ckan-v1.8.1 new db_to_form_package_schema() strips tracking summary, isopen If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates. 1344444427000000 1350303821000000 2820 defect danieljohnlewis demo phase 5 new English Language: Visualization -> Visualisation "Problem: In the English version (which has a UK flag, indicating British English), the word ""Visualization"" is used. For an example see the ""Filter by type"" drop down on the /apps page. Expected: This should be ""Visualisation"" in British English. Any instances of ""Visualize"" should be changed to ""Visualise"" too. " 1344504455000000 1344504455000000 2821 enhancement danieljohnlewis demo phase 5 new Featured Items on Filter "Problem: On /apps page in the Filter Results box there is a ""Only show featured items"" checkbox, on selection it comes up with 0 solutions. Expected: Presumably an admin can create ""featured items"" so that they can be randomly selected on front page (is this correct)? If there are no ""featured items"" in the whole database can this check box be hidden? Bug is: no UI or obvious way to create featured items. Also the checkbox looks un-styled" 1344504504000000 1344505492000000 2822 enhancement toby toby demo phase 4 new Resource additional info titles format/i18n "the title for additional info should be translated capitalised etc" 1344504620000000 1344543985000000 2823 enhancement toby toby demo phase 5 new resource additional info title order "Order the items so that none user fields are first from ticket #2707 " 1344504773000000 1344504773000000 2828 enhancement toby shevski demo phase 4 new Draft datasets are confusing - tickets need creating "reported as editing datasets incorrect e.g. clicking on edit here http://s031.okserver.org:2375/dataset/ff takes you to create dataset page http://s031.okserver.org:2375/dataset/edit/ff but this is the correct behaviour of a draft dataset We need to show draft datasets correctly proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc" 1344506178000000 1344547324000000 2829 enhancement johnglover toby ckan-v1.9 new Archiver fails on 403 http response "Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end {{{ $ paster archiver update -c ../ckan/development.ini 2012-08-09 11:01:37,636 INFO [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources) 2012-08-09 11:01:37,671 INFO [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13 2012-08-09 11:01:37,900 INFO [ckan.lib.base] /api/action/package_show render time 0.043 seconds Traceback (most recent call last): File ""/home/toby/okfn/pyenv/bin/paster"", line 8, in load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')() File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 104, in run invoke(command, command_name, options, args[1:]) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 143, in invoke exit_code = runner.run(args) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py"", line 238, in run result = self.command() File ""/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py"", line 98, in command response = app.post(api_url + '/package_show', data) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 262, in post expect_errors=expect_errors) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 243, in _gen_request return self.do_request(req, status=status) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 406, in do_request self._check_status(status, res) File ""/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py"", line 439, in _check_status res.body)) paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show) {""help"": ""Return the metadata of a dataset (package) and its resources.\n\n :param id: the id or name of the dataset\n :type id: string\n\n :rtype: dictionary\n\n "", ""success"": false, ""error"": {""message"": ""Access denied"", ""__type"": ""Authorization Error""}} }}}" 1344508484000000 1344508484000000 2830 enhancement toby toby demo phase 4 new Need method to undelete groups need controller action and front-end method 1344509408000000 1344547341000000 2831 enhancement aron.carroll ckan 2.0 new Create a limited subset of markdown that's supported "Allowing people to use the full range of markdown results in extremely messy output across the site. I'd suggest limiting support to only a subset of common use cases. * Allow all inline elements, this allows bold, italic, code and links. * Allow lists. Disallow * Horizontal Rules * Headings * Block quote and code (this may turn out to be useful and so could be included) This way you get markdowns paragraph handling and a few inline styles without breaking the entire layout of the page. Here's the full syntax if anyone is interested http://daringfireball.net/projects/markdown/syntax" 1344512467000000 1344512467000000 2833 enhancement aron.carroll demo phase 5 new Load module templates before calling .initialize() "I think this would be a nice feature for remote loading templates if the options.template value ends in "".html"". {{{ ckan.module('my-module', { options: { template: 'my-template.html' }, initialize: function () { this.template // This is the loaded template. } }); }}}" 1344531939000000 1344531939000000 2834 enhancement aron.carroll demo phase 4 new Defer all publish calls until all modules have been initialised "Currently if a module uses sandbox.publish() in its initilaize() function then any modules initialised afterwards will miss the event. We should queue all calls to .publish() in the pubsub module in an array until all modules have loaded then iterate over and execute them." 1344532093000000 1344856692000000 2835 enhancement aron.carroll demo phase 5 new Client module needs a template loading method "{{{ Client#getTemplate(name, params, success, error); }}} Where params, success and error are optional arguments. test/index.html already has an implementation called loadFixture()." 1344532233000000 1344532233000000 2838 enhancement seanh ckan-v1.9 new Context variables accepted by action functions need to be documented "I was doing this: {{{ context = {'model': base.model, 'session': base.model.Session, 'user': toolkit.c.user or toolkit.c.author, 'extras_as_string': True} group_dict = logic.get_action('group_show')(context, {'id': group_id}) }}} in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass {{{'extras_as_string': True}}} in the context. I don't think this or other context variables are documented anywhere." 1344630911000000 1344630911000000 2844 enhancement rgrp new SQL-only (no solr) version of CKAN " * ~~Search needs to run of local DB (again)~~ * paster db clean attemps to connect to SOLR (still works as does db first but then excepts which is not nice UX)" 1344859168000000 1345454527000000 2846 defect seanh ckan-v1.8.1 new Organizations allows you to set an organization as the parent organization of itself that doesn't make any sense 1344860536000000 1350303844000000 2850 enhancement aron.carroll demo phase 4 new Add JSON support for IE7 "The JavaScript just isn't going to work without it. Look into either the newer JSON3: http://bestiejs.github.com/json3/ or the old faithful: https://github.com/douglascrockford/JSON-js/" 1344940692000000 1344940692000000 2851 enhancement ross new Preview of PDF tries to connect to datastore "e.g. http://datahub.io/dataset/ccc-gistemp/resource/80ebdbd6-d91b-4fef-9db4-d3dfbd7e868e CKAN attempts to render the PDF as a datastore tabular date even though the mimetype of the PDF is correctly specified." 1345032766000000 1345032766000000 2854 enhancement shevski demo phase 5 new no way to see which dataset an app is related to on /apps page e.g. from http://s031.okserver.org:2375/apps?page=1 you cannot edit a related item as an admin or view what dataset the item is related to 1345048728000000 1345048728000000 2857 defect toby shevski demo phase 4 new no UI to follow another user although option on dashboard "e.g. second option in left hand sidebar is follow another user' http://s031.okserver.org:2375/user/dashboard but if you click on that and then any user, there's no follow button to me the dashboard user stuff is highly unfinished and probably should not be included unless someone is very keen for it" 1345049983000000 1345107097000000 2859 defect icmurray icmurray ckan-v1.8 new Fix the build "The requires files have been removed. These were required by buildkit. We've lost the information regarding which packages were already in ubuntu repos, which conflicted and which were missing. But perhaps we can just package everything up." 1345109217000000 1345109217000000 2861 enhancement toby shevski demo phase 4 new Add qa breadcrumb can we add in the breadcrumb navigation to qa pages so that you can go back to http://s031.okserver.org:2375/qa from sub pages such as http://s031.okserver.org:2375/qa/dataset/five_stars 1345114046000000 1345483270000000 2862 enhancement toby markw demo phase 4 new Revised revised groups description "Revised text for 'What are groups?' box at demo.ckan.org/group (after discussion with IB re #2812): What are groups? Groups allow you to group users and data together so that they are easier to manage. Group owners can assign roles and authorisations, giving each project or department control of its own data publishing. Users can browse or facet by groups, which could be an organisation (for example, the Department of Health) or topic (e.g. Transport, Health), making it easier to find the data they are looking for." 1345114322000000 1345115072000000 2863 defect toby shevski demo phase 4 new non-admin users can edit other's draft datasets edit button shows up for daniel lewis for http://s031.okserver.org:2375/dataset/ff 1345114479000000 1345201502000000 2867 enhancement ross new 1.8 url_for error "From: Date: Thu, Aug 16, 2012 at 10:30 AM Subject: WebApp Error: : url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 To: ckan-sysadmin@okfn.org, kindly@gmail.com URL: http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.storage:190 in file << return fapp(request.environ, self.start_response) else: h.redirect_to(file_url) >> h.redirect_to(file_url) Module ckan.lib.helpers:57 in redirect_to << if are_there_flash_messages(): kw['__no_cache__'] = True return _redirect_to(url_for(*args, **kw)) def url(*args, **kw): >> return _redirect_to(url_for(*args, **kw)) Module ckan.lib.helpers:79 in url_for << # fix ver to include the slash kw['ver'] = '/%s' % ver my_url = _routes_default_url_for(*args, **kw) kw['__ckan_no_root'] = no_root return _add_i18n_to_url(my_url, locale=locale, **kw) >> my_url = _routes_default_url_for(*args, **kw) Module routes.util:265 in url_for << if not isinstance(url, str) and url is not None: raise GenerationException(""url_for can only return a string, got "" ""unicode instead: %s"" % url) if url is None: raise GenerationException( >> ""unicode instead: %s"" % url) GenerationException: url_for can only return a string, got unicode instead: https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&Expires=1345109432&AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6 CGI Variables CKAN_CURRENT_URL '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)' PATH_INFO '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '37236' REQUEST_METHOD 'GET' REQUEST_URI '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80
\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345109433.273328, '_creation_time': 1345109433.273328} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'}) " 1345114669000000 1345128279000000 2869 defect shevski demo phase 4 new user profile not linked on disqus until page refresh "Reported by Daniel: Problem: DISQUS comments don't appear with profile details (inc. username). How to reproduce: 1) Go to a dataset, e.g. I went to http://s031.okserver.org:2375/dataset/ff 2) I logged into DISQUS using the Google Authentication, e.g. in my case that is okfn.daniel.lewis@gmail.com 3) I made a comment and submitted 4) It comes up with the comment, but no linked profile until the page gets refreshed" 1345114714000000 1345114714000000 2870 enhancement seanh ross new 1.8 tag_list not defined "From: Date: Thu, Aug 16, 2012 at 3:20 AM Subject: WebApp Error: : ""tag_list"" not defined To: ckan-sysadmin@okfn.org, kindly@gmail.com URL: http://ckan.net/dataset/uk-postboxes Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:155 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:131 in __call__ << r'\1', oldpath) response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:239 in __call__ << try: res = WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> res = WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.package:322 in read << template = template[:template.index('.') + 1] + format return render(template, loader_class=loader) def comments(self, id): >> return render(template, loader_class=loader) Module ckan.lib.base:153 in render << try: return cached_template(template_name, render_template, loader_class=loader_class) except ckan.exceptions.CkanUrlException, e: raise >> loader_class=loader_class) Module pylons.templating:249 in cached_template << return content else: return render_func() >> return render_func() Module ckan.lib.base:102 in render_template << if loader_class == NewTextTemplate: return literal(stream.render(method=""text"", encoding=None)) return literal(stream.render(method=method, encoding=None, >> return literal(stream.render(method=""text"", encoding =None)) Module genshi.core:183 in render << method = self.serializer or 'xml' generator = self.serialize(method=method, **kwargs) return encode(generator, method=method, encoding=encoding, out=out) def select(self, path, namespaces=None, variables=None): >> return encode(generator, method=method, encoding=encoding, out =out) Module genshi.output:57 in encode << _encode = lambda string: string if out is None: return _encode(''.join(list(iterator))) for chunk in iterator: out.write(_encode(chunk)) >> return _encode(''.join(list(iterator))) Module genshi.output:569 in __call__ << def __call__(self, stream): strip_markup = self.strip_markup for event in stream: if event[0] is TEXT: data = event[1] >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module ckanext.googleanalytics.plugin:93 in download_adder << [downloaded %s times]''' count = None for mark, (kind, data, pos) in stream: if mark and kind == START: href = data[1].get('href') >> for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1175 in __call__ << """""" callable_value = hasattr(self.value, '__call__') for mark, (kind, data, pos) in stream: if mark is ENTER: if callable_value: >> for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.filters.transform:686 in _unmark << def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): >> for mark, event in stream: Module genshi.filters.transform:1145 in __call__ << :param stream: The marked event stream to filter """""" for mark, event in stream: yield mark, event if mark is ENTER: >> for mark, event in stream: Module genshi.filters.transform:714 in __call__ << stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event >> for mark, event in stream: Module genshi.filters.transform:682 in _mark << def _mark(self, stream): for event in stream: yield OUTSIDE, event >> for event in stream: Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound for event in stream: if event[0] is INCLUDE: href, cls, fallback = event[1] >> for event in stream: Module genshi.template.base:565 in _flatten << elif kind is EXPR: result = _eval_expr(data, ctxt, vars) if result is not None: # First check for a string, otherwise the iterable test >> result = _eval_expr(data, ctxt, vars) Module genshi.template.base:277 in _eval_expr << if vars: ctxt.push(vars) retval = expr.evaluate(ctxt) if vars: ctxt.pop() >> retval = expr.evaluate(ctxt) Module genshi.template.eval:178 in evaluate << __traceback_hide__ = 'before_and_this' _globals = self._globals(data) return eval(self.code, _globals, {'__data__': data}) >> return eval(self.code, _globals, {'__data__': data}) Module ?:51 in <<
  • Tags

    ${tag_list(c.pkg_dict.get('tags', ''))}
  • >> ${tag_list(c.pkg_dict.get('tags', ''))} Module genshi.template.eval:309 in lookup_name << val = BUILTINS.get(name, val) if val is UNDEFINED: val = cls.undefined(name) return val >> val = cls.undefined(name) Module genshi.template.eval:410 in undefined << """"""Raise an ``UndefinedError`` immediately."""""" __traceback_hide__ = True raise UndefinedError(key, owner=owner) >> raise UndefinedError(key, owner=owner) UndefinedError: ""tag_list"" not defined CGI Variables CKAN_CURRENT_URL '/dataset/uk-postboxes' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,text/plain,text/xml' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'vi,en-us;q=0.7,en;q=0.3' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'coccoc/1.0 ()' PATH_INFO '/dataset/uk-postboxes' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/dataset/uk-postboxes' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '47065' REQUEST_METHOD 'GET' REQUEST_URI '/dataset/uk-postboxes' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '
    Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1345083636.079552, '_creation_time': 1345083636.079552} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (, '') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'} repoze.who.logger repoze.who.plugins {'openid': , 'friendlyform': , 'ckan.lib.authenticator:UsernamePasswordAuthenticator': , 'auth_tkt': , 'ckan.lib.authenticator:OpenIDAuthenticator': } routes.route routes.url webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgi.version (1, 1) wsgiorg.routing_args (, {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'}) " 1345114732000000 1345114732000000 2881 defect seanh ckan-v1.8.1 new Remove or improve 'Domain Model' docs page "http://docs.ckan.org/en/ckan-1.8/domain-model.html Currently not very useful. Also these pages: http://docs.ckan.org/en/latest/domain-model-dataset.html http://docs.ckan.org/en/latest/domain-model-resource.html http://docs.ckan.org/en/latest/domain-model-task-status.html This just seems very incomplete to me (surely there is more to the domain model the dataset, resource and task status?) and very likely to not get updated as changes are mode to the code. Would anyone seriously refer to this rather than just looking at the source code? If we want to have something along these lines in the docs, I suggest adding docstrings to packages, modules, classes and methods and then using autodoc to pull them into sphinx." 1345539917000000 1345626056000000 2882 defect seanh ckan-v1.8.1 new Remove User Stories Overview from docs? "Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation? http://docs.ckan.org/en/latest/user-stories.html" 1345540723000000 1345540723000000 2883 enhancement seanh ckan-v1.8.1 new Add high level dev overview to 'For CKAN Developers' section of docs Here's a draft: https://gist.github.com/3414107 1345546355000000 1345629890000000 2884 defect seanh ckan-v1.8.1 new Lock icon shown next to unknown licenses Set the license of a dataset to blank (the first option) or to 'License Not Specified' or 'Other (Non-Commercial)' and a lock icon appears next to the license on the dataset read page. Shouldn't this be a question mark or something? 1345564873000000 1345564873000000 2885 enhancement seanh ckan-v1.9 new Labels (editor, admin) not translated in authorization pages for datasets or groups 1345625351000000 1345625524000000 2886 enhancement ross ckan 2.0 new Configurable related items "Related items (Apps & Ideas) have a collection of types, which are currently fixed. Some of these types are ambiguous, such as ideas in that they may not have a link - and if they do it is likely to be to a blog post - another existing type. Whilst the URL is required, and we think this should stay required, we should also allow users to change the types found in Apps & Ideas to a shortened list that suits their requirements." 1345625718000000 1345625718000000 2887 defect seanh ckan-v1.8.1 new """Welcome to CKAN!"" on front page is untranslatable" "ckan/templates/home/index.html contains:

    Welcome to ${g.site_title}!

    which means that just ""Welcome to"" ends up in the ckan.po files for translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence." 1345627079000000 1345627233000000 2888 enhancement kindly new Datapreview in Iframe Try and put datapreview in Iframe. 1345745461000000 1345745461000000 2889 enhancement seanh ckan-v1.9 new Support JSON values in extras, when returning JSON from API "For example, edit the warandpeace dataset and add an extra with key ""comments"" and value ""[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]"", now visit API URLs such as: /api/search/dataset?sort=metadata_modified+desc&all_fields=1&limit=10 /api/rest/dataset/warandpeace /api/action/package_show?id=warandpeace The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects." 1345811737000000 1345811737000000 2890 enhancement seanh ckan-v1.8.1 new Collect data previews and data store docs in one chapter "Currently there is this page: http://docs.ckan.org/en/latest/data-viewer.html which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it. Separately there is this page: http://docs.ckan.org/en/ckan-1.7.1/datastore.html which covers datastore, datastorer, and the data api. I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be: CKAN has builtin previews of data resources on resource pages, enabled by default. Images, Google Documents, and web page resources will be loaded into embedded iframes for preview. Text-like files will be displayed raw. CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer. Additionally, you can enable CKAN's DataStore, requires you to install ElasticSearch and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data. Does having DataStore enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline, You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd." 1346149236000000 1346175867000000 2891 defect seanh ckan-v1.8.1 new Update or remove DataStore google presentation "Currently this page: http://docs.ckan.org/en/ckan-1.8/datastore.html has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient. Suggest moving the google presentation into the documentation proper, so it works with grep etc." 1346149423000000 1346154177000000 2892 defect seanh ckan-v1.8.1 new IDatasetForm's form_to_db_schema() breaks API? "It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict. It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method." 1346154125000000 1346154163000000 2893 defect seanh ckan-v1.8.1 new Tags missing from form_to_db_package_schema() "With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this: [ckan.controllers.api] Validation error: ""{'__type': 'Validation Error', '__junk': [u'The input field __junk was not expected.']}"" They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active). In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema." 1346155864000000 1346155864000000 2894 defect seanh ckan-v1.8.1 new Empty lists get stripped from package dicst in package_show() "If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active). If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value. Suspect this also effects packages with no tags, no groups, etc." 1346157634000000 1346157634000000 2895 enhancement rgrp new QA pages (e.g. broken resource links) are not paginated And as a result timeout on e.g. the datahub ... 1346175485000000 1346175485000000 2896 defect seanh ckan-v1.8.1 new resource_create crashes on ValidationErrors with KeyError: 'resources' "ckan/logic/action/create.py:resource_create(): try: pkg_dict = _get_action('package_update')(context, pkg_dict) except ValidationError, e: errors = e.error_dict['resources'][-1] raise ValidationError(errors) This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold." 1346257754000000 1346257754000000 2897 defect seanh ckan-v1.8.1 new db_to_form_package_schema() crashes historical dataset page "If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191) crashes. Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like: schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp': [validators.ignore_missing, unicode]}) " 1346330495000000 1346330495000000 2898 defect johnmartin demo phase 5 new Looses data entered on step 1 of create dataset after login "To reproduce: 1. Enter details for new dataset without being logged in 2. Hit login page for just before step 2 3. Once logged in you end up back at the empty add dataset form (instead of expected 2nd step of dataset edit)" 1346332094000000 1346332120000000 2899 enhancement johnmartin demo phase 5 new Step 2 > Save & add another bug "Goto: 1. Add dataset 2. Get to step 2 3. Click ""Save & add another"" without adding any data 4. Ends on step 3 Should: fail to validate and remain on step 2" 1346332519000000 1346332519000000 2900 defect seanh ckan-v1.8.1 new snippets/package_list.html template crashes on dataset that has no 'notes' "Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages. Fix in the template? Or in the schema? The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well" 1346335207000000 1346335788000000 2901 enhancement johnmartin demo phase 5 new Language Dropdown bug in footer with IE7 "The language dropdown in Internet Explorer 7 behaves oddly. Essentially I think the problem is with the way that the dropdown decides to go into ""drop-up mode"" (e.g there aint enough screen space below the initial state so it has to drop up)." 1346754674000000 1346754674000000 2902 enhancement ross ckan 2.0 new genshi is used in Group controller "The Group controller uses the genshi module where it may be better for it to use Markup() in the template. See https://github.com/okfn/ckan/pull/117/files#r1524863" 1346782230000000 1346782230000000 2904 enhancement johnmartin demo phase 5 new Show more formats/groups/tags shouldn't display if there aren't more On the search results page when there are no more formats, groups or tags to display it should not display the show more links. 1346854444000000 1346854444000000 2905 enhancement johnmartin demo phase 5 new Add dataset URL key behaviour "On step 1 for adding a dataset can we add a ajax ping that checks if the URL that is generated by the JS is taken? Behaviour should be as follows: 1. User types title in box (as current) 2. URL key is regexed by JS (as current) 3. The JS does a ajax call to check if the generated URL key is valid. The ajax callback returns a JSON dump which contains what the URL key should be (e.g. if 'test' was taken it returns 'test-1') 4. If the URL key returned in the ajax is different than the one generated by the JS then: 4a. If the user hasn't clicked the edit button then: it automatically updates the URL key field with the URL key supplied in the ajax 4b. If the user has clicked the edit button then: a form error next to the input appears saying that the URL key is taken and within that error is suggests the URL key returned by the ajax as alternative (the user can click said URL key within the error to automatically choose that one) The URL key isn't reserved by the ajax and upon form submission the usual validation should still apply." 1346855283000000 1346855283000000 2906 defect dominik demo phase 5 new Wrong flags in language dropdown "The flags in the dropdown on details pages are not right. You can see the error on http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0." 1346861144000000 1346862889000000 2908 enhancement johnmartin demo phase 5 new Dataset related with image should retain aspect ratio At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio. 1346931162000000 1346931162000000 2909 enhancement johnmartin demo phase 5 new User profile headers are a bit weird They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain. 1346931348000000 1346931348000000 2910 enhancement johnmartin demo phase 5 new User listings are not very compelling They are currently are just a small image / name. They should be: Larger image, Full name (username), description, edits/dataset 1346931587000000 1346931587000000 2911 enhancement kindly new Internal documentation of Organization Groups A summary/user story doc of how organizations and groups are expected to work. 1346941384000000 1346941384000000 2912 defect seanh ckan-v1.8.1 new ckanext/organizations: Neither public or private initially selected when adding dataset When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected. 1347271076000000 1350303897000000 2913 defect seanh ckan 2.0 new IGroupForm has package_form() method Should be group_form() I think 1347374739000000 1350303779000000 2914 defect seanh ckan-v1.8.1 new Add form_to_db_schema_options() to IDatasetForm and IGroupForm This optional method is supported when creating or updating groups and packages and is used in DefaultGroupForm and DefaultPackageForm, but seems to be missing from the interfaces. Should be added with docstring. 1347445795000000 1350303928000000 2915 refactor seanh ckan 2.0 new Refactor form_to_db_schema_options() "Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)? Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay. The same thing goes for db_to_form_schema_options(). Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None?" 1347446269000000 1347447588000000 2917 enhancement johnmartin ross ckan 2.0 new Organization admins can delete themselves Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end. 1347451843000000 1347455636000000 2919 defect seanh ckan-v1.8 new Remove IController from docs Still in docs but seems gone from code 1347461253000000 1347461253000000 2920 defect seanh ckan 2.0 new Genshi text template message extractor no longer needed? "In setup.py, do we still need: ('templates/**.txt', 'genshi', { 'template_class': 'genshi.template:TextTemplate' }), in message_extractors? Doesn't like there are any txt files in there anymore." 1347530249000000 1347530249000000 2921 defect seanh ckan 2.0 new Add docstring to top of lib/extract.py file "I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious. Also a couple of other small fixes: jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export. jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future." 1347530407000000 1347530407000000 2922 defect seanh ckan 2.0 new Better docstring for CKANInternationalizationExtension "I'm unsure about what's going on here. As I understand it, when we run `python setup.py extract_messages` it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files. But then we have the parse() method of CkanInternationalizationExtension also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice? My guess is that CkanInternationalizationExtension is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output? Maybe CkanInternationalizationExtension needs a better docstring saying what it does?" 1347530507000000 1347530507000000 2923 defect seanh ckan 2.0 new Change regularise -> regularize The function is called regularise_html(), can't remember what file it's in. 1347530582000000 1347530582000000 2924 defect seanh ckan 2.0 new Better docs for trans js command, and add to release process "Add a better docstring to for trans js command explaining what it does and why, and how to use it. Also add this command to the CKAN Release Process as it's needed there" 1347530671000000 1347530671000000 2925 defect seanh ckan 2.0 new Remove trans mangle paster command? "- Is `trans mangle` really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place." 1347530768000000 1347530768000000 2926 enhancement seanh ckan 2.0 new I*Form tests for per-type templates "Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods). The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions." 1347552081000000 1347552081000000 2927 enhancement seanh ckan 2.0 new Test new package/group/organization type URLs support IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet. 1347552153000000 1347552153000000 2928 enhancement seanh ckan 2.0 new Run CKAN tests with example_i*form extensions enabled Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process. 1347552334000000 1347552334000000 2929 enhancement seanh ckan 2.0 new Remove is_fallback() from I*Form extension interfaces? "If a plugin wants to take over the default package type for example, it can return ['dataset'] from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed? Same for IGroupForm and IOrganizationForm" 1347552597000000 1347552597000000 2931 enhancement seanh ckan 2.0 new Better docstring for app_globals.py "''' The application's Globals object ''' is not very informative. " 1347891194000000 1347891378000000 2932 enhancement seanh ckan 2.0 new Add docstring to system_info.py What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table. 1347893429000000 1347893429000000 2933 enhancement seanh ckan 2.0 new Update theming docs "There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs. I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents." 1347895272000000 1347895272000000 2934 enhancement Dominik new Webtests for CKAN "The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes. An example user story to be tested would be: ""As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system."" or: ""As a user I want to use the search functionality and open a resource from the search page."" These user stories require a different testing system with interaction and processes. " 1347966647000000 1348060201000000 2936 defect seanh ckan-v1.8 new Updating a group via the API clears its packages "If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work. Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent" 1348048066000000 1348048066000000 2938 enhancement Toby dominik new Explain bundling in doc/resources.rst "Since it makes sense to bundle resources into one js file that are commonly used together, there should be some explanations on how separate js scripts are merged into one file for production. Explanations could include how groups are handled or different resource files." 1348488835000000 1348488835000000 2939 enhancement toby toby new Orgs are groups holder for ticket number 1348564034000000 1348564034000000 2943 enhancement dominik new Chrome does not resize preview Chrome does not resize iframe after a full refresh/ on first load 1349089686000000 1349090759000000 2944 enhancement dominik new Recline preview does not work in Opera Opera cannot show a recline preview. I shows 0 records. 1349090802000000 1349090802000000 2945 enhancement dominik new Pdf preview does not load in IE The pdf preview does not load in IE 9. 1349090869000000 1349090869000000 2947 enhancement dominik new Redirect to the resource page from /resource/{res-id} "Use story: As a user of the datasore I want to be able to put the resource id without the dataset name in a url and see the resource page. Proposed redirect: /resource/{res-id} -> dataset/foo-bar/resource/{res-id}" 1349097325000000 1349097325000000 2948 enhancement dominik new Negative range breaks datastore backend Recline allows negative ranges. The backend returns an error 500 for that which breaks the recline preview. 1349101854000000 1349101865000000 2949 defect amercader amercader new Reenable Data API button on the new theme The checks to show or not the button need to be updated for the latest datastore version 1349107464000000 1349107464000000 2951 enhancement icmurray ckan 2.0 new Paster command for building css from less "With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is: - the less files should be compiled to css (main.css, not just main.debug.css) This ticket is to provide a paster command which will compile the .less into the main.css file. The idea of the paster command is that it will be run by developers, and they will check in the resulting .css files. It will also provide a convenient mechanism for production scenarios with node.js installed on the production server, as they will be able to compile the .less there as well. ## Background The .css files that need to be served are built using `less`. The css files that are generated *are* checked-in to the repo; but they are a build artifact. So the general workflow is: - commit changes to the less files - build the main.css file and commit This paster command slots into the above workflow for convenience. ## Notes - doc/frontend-development.rst - bin/less to run this paster command, `Node` will be required. So that dependency should be checked. ## The paster command This is the proposed behaviour of the paster command: - ensure `custom.less` does not exist - for each colour in{fuchsia,green,maroon,red}: - generate a `custom.less` file for $colour. There's a paster command that does this already: `paster color . - generate the `css` from the `less` files. ie - the equivelant of running `bin/less --production`. - this will generate a `main.css` file, which should be renamed to `$color.css`. - ensure `custom.less` does not exist - generate the `css` from the `less` files. This will create a `main.css` with the default colour scheme. - call the paster command that minifies css and js files. (This command does not exist yet, see #2950)" 1349118740000000 1349175030000000 2952 enhancement seanh icmurray ckan 2.0 new incorporate javascriopt translations into translations workflow There are now javascript translations. There's a paster command, `trans js`. This is a ticket to ensure that using `trans js` is documented in the normal ckan translation workflow (which is followed after a feature freeze, prior to release). And to see if there's any need for further incorporation into the current translation process. 1349118945000000 1349118945000000 2957 task dominik new New datastore on postgres prior to 9.0 The ``GRANT SELECT ON ALL`` and ``ALTER DEFAULT PRIVILEGES`` are not available in Postgres prior to version 9.0. 1349292214000000 1349292214000000 2958 defect seanh ckan-v1.8 new Uploading files with unicode characters in filename fails in CKAN 1.7 and 1.8 "e.g. 2012_08(주요국가).xls I tested in CKAN 2.0 and it seemed to work fine there." 1349342976000000 1349342976000000 2959 defect icmurray icmurray ckan 2.0 new Changing a Group's name through the action api disassociates it from its datasets in the index "Repro: - Create a new Group, named ""test-group"". - Add a dataset to it. - Verify the dataset belongs to the group by visiting the Group's read-page - Update the Group through the action api (group_update), using the uid in the ""id"" field, and a new name in the ""name"" field. - Visit the group's read-page. The list of datasets will be empty. This was an issue when editing a Group through the web interface, which was fixed in [1]. However it only fixes the issue in the group controller. [1] https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335" 1349363935000000 1349363935000000 2960 task dominik ckan-backlog new Private datasets are not private At the moment everyone can read all resources. There is no support for hiding resources that are in private datasets. 1349694255000000 1349694255000000 2962 enhancement dominik ckan-backlog new Search across multiple ckan instances "Could be done by: - using the solr distributed search - difficult set up - merging result sets from apis - make sure that results can be merged properly (score, facets, ...)" 1349736622000000 1349736622000000 2963 defect amercader ckan-v1.8 new Timeout on tag pages with lots of datasets "e.g. http://thedatahub.org/tag/lod Tags with less datasets work fine (e.g. http://thedatahub.org/tag/railways)" 1350295999000000 1350295999000000 2964 defect seanh ckan 2.0 new Last organization admin can remove herself "If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it. Last admin should not be able to remove or demote herself. Also applies to groups." 1350296058000000 1350296058000000 2965 defect amercader ckan-v1.8 new Stats extension broken on 1.8 " * Graphs not showing (looks like a flot related file is missing) * Wrong groups counts (e.g. Data Explorer Examples show 1800 datasets when it onlu has 8)" 1350296141000000 1350296157000000 2966 enhancement seanh ckan 2.0 new 'Add' button text is wrong when editing organization members e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe 1350296149000000 1350296149000000 2967 defect seanh ckan 2.0 new Organization members edit page reloads after demoting self Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page. 1350296227000000 1350296227000000 2968 defect seanh ckan 2.0 new Anyone can access organization members page The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all 1350296355000000 1350297070000000 2969 defect seanh ckan 2.0 new Group members page 500s The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s 1350296454000000 1350296454000000 2970 defect seanh ckan 2.0 new Organization and group member links use id not name e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar 1350296531000000 1350296531000000 2971 enhancement seanh ckan 2.0 new """Are ytou sure you want to delete this member?"" should say which member" when deleting members from groups and orgs 1350296840000000 1350296840000000 2972 enhancement seanh ckan 2.0 new Remove any imports of authz.py and delete file it is no longer used 1350297442000000 1350297442000000 2973 enhancement seanh ckan 2.0 new Move new_authz.py into logic/auth/__init__.py "Makes sense to keep all auth stuff under logic/auth. Also decide which functions from new_authz.pu should be marked private with _. Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py" 1350297545000000 1350297545000000 2974 enhancement seanh ckan 2.0 new General of all auth functions "From Toby: A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc." 1350297612000000 1350297612000000 2975 enhancement seanh ckan 2.0 new Tests for auth functions and new actions "From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions " 1350297660000000 1350297660000000 2976 enhancement seanh ckan 2.0 new Polish group and organization member pages "From Toby: A little polish to the member pages and probably a little hardening of the controller and logic actions" 1350297709000000 1350297709000000 2977 enhancement seanh ckan 2.0 new Fix user autocomplete on group and organization member pages "From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it " 1350297749000000 1350297749000000 2978 enhancement seanh ckan 2.0 new Tests for permissions for organizations and groups "From Toby: checking the logic around the who can do what from the user stories - via some tests may be a good approach " 1350297806000000 1350297806000000 2979 enhancement seanh ckan 2.0 new Requesting membership to groups and organizations "From Toby: The requesting membership user stories and implementation remain needed " 1350297844000000 1350297844000000 2981 refactor seanh ckan 2.0 new Remove config from jinja templates "From Toby: remove the config from jinja2 templates get all such stuff via g. (before 2.0) " 1350297960000000 1350297960000000 2982 refactor seanh ckan 2.0 new Move functionality from controllers into template helpers "From Toby: strip more functionality from controllers and add via template helpers eg activity streams " 1350298002000000 1350298002000000 2983 refactor seanh ckan 2.0 new Refactor lib/base.py to remove circular import issues "From Toby: c) refactor lib/base.py to remove the circular import issues (render functions to lib/render.py - would fix much of that) " 1350298044000000 1350298044000000 2984 refactor seanh ckan 2.0 new Auto-populate context with user, model, session, etc. "From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action()" 1350298101000000 1350298101000000 2985 refactor seanh ckan 2.0 new Make lib/helpers.py more template-specific "From Toby: make lib/helpers more template specific and less generally shared with none template uses (some functions are shared but many shouldn't be) - again help avoid circular imports " 1350298188000000 1350298188000000 2986 refactor seanh ckan 2.0 new Make lib/base.py more like it was originally intended "From Toby: make lib/base.py be more like it was originally intended eg defines stuff that is shared eg _ but maybe try to include stuff like render/get_action if the circular import stuff can be solved - which should be doable if hard and likely needs an extra file or two for the externals _ etc " 1350298245000000 1350298245000000 2987 refactor seanh ckan 2.0 new Remove all direct calls to logic action and auth functions From Toby 1350298287000000 1350298287000000 2988 enhancement seanh ckan 2.0 new UI functional tests for CKAN 2.0 Afaik the new CKAN 2.0 frontend has no functional tests 1350298935000000 1350298935000000 2989 enhancement seanh ckan 2.0 new """Add dataset to organization"" should auto-select the organization" "'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected. I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the ""Add dataset to group"" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently." 1350299052000000 1350299052000000 2990 enhancement seanh ckan 2.0 new Fix descriptions of groups and organizations on /groups and /organizations pages 1350299091000000 1350299091000000 2991 enhancement seanh ckan 2.0 new Add blocks to header.html template Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are. 1350385820000000 1350385820000000 2992 enhancement johnmartin new Delete resource should send me back to edit dataset page 1350407956000000 1350407956000000 2993 defect seanh ckan 2.0 new """logged_in"" and ""visitor"" show in user list at /users" 1350466922000000 1350484826000000 2994 enhancement seanh ckan 2.0 new Add dataset and user popovers throughout the site "If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button. These popovers should appear wherever datasets or users are listed, throughout the site." 1350483964000000 1350483964000000 2995 enhancement seanh ckan 2.0 new Popovers for resources, groups, organizations, tags... "If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button. It would be nice to add similar popovers for other types of object in CKAN, e.g. resources, groups, organizations, tags..." 1350484041000000 1350484041000000 2996 enhancement seanh ckan 2.0 new Fix behaviour of View Profile button in user popover when already on user's profile "The user popovers that appear when you hover the mouse over a user in an activity stream include a View Profile button that takes you to that user's profile page. If you are already on that user's profile page then it simply reloads the same page, but it sends you to the Datasets tab if you were on another tab. Possible solutions: Don't show button if already on user's page? Button just makes popover disappear, if already on user's page? Button does reload the page, but reloads the same tab (datasets, followers, activity stream) that you were on. Marking this low priority" 1350484238000000 1350484238000000 2997 enhancement seanh ckan 2.0 new Add activity streams to dataset pages Add an activity stream tab to dataset pages, like we have on user profile pages. Dataset activity streams are already implemented in the backend. 1350484306000000 1350484306000000 2998 enhancement seanh ckan 2.0 new Add activity streams to group pages Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups. 1350484394000000 1350484394000000 2999 enhancement seanh ckan 2.0 new Add activity streams to organization pages Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations. 1350484457000000 1350484457000000 3000 enhancement seanh ckan 2.0 new Add number of followers to dataset pages User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend). 1350484560000000 1350484560000000 3001 defect seanh ckan 2.0 new Multilingual plugin crashes CKAN on add dataset when some languages are default "Enable the multilingual plugins: ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag and set your default language to one not supported by the multilingual plugin, e.g. ckan.locale_default = cs_CZ now run CKAN and try to add a dataset: File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index text_field_items['text_' + default_lang].extend(all_terms) KeyError: 'text_cs_CZ' It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not. A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes. I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0" 1350579048000000 1350579048000000 3002 defect amercader ckan-v1.8 new API v1/2 'legacy' search parameters must be escaped before they are put into a Solr query string "Just to track @tauberer patch on Github. Would be nice to write a test for it. Probably going to 1.8.1" 1350639142000000 1350639142000000 3003 enhancement amercader ckan-v1.8 new New pg databases should be created with UTF8 encoding rather than system default "Just to track @tauberer patch on Github. Probably going to 1.8.1" 1350639199000000 1350639199000000 3005 enhancement seanh seanh ckan 2.0 new Add following/unfollowing of groups: model, API, frontend, tests 1351013744000000 1351013744000000 3006 enhancement seanh ckan-v1.8.1 new Update translations from Transifex There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too. 1351506144000000 1351506144000000 3007 enhancement seanh ckan 2.0 new Adding a dataset creates multiple activities Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity. 1351516507000000 1351517019000000 3010 enhancement seanh ckan 2.0 new Pin images don't appear in data explorer "When minified files are in use (set debug = false in ini file) the pin images for points don't show up in the data explorer map view when viewing a resource with geolocation data. set debug = true will fix the problem by using unminified files instead but also turns on a bunch of debug stuff you wouldn't want on a production site. middleware.py, around line 73, is where the decision to turn minified files on or off based on the debug setting is made. This can be hacked on a production site to run with unminified files but otherwise run on production mode thereby getting around the issue with the pins. But that doesn't fix the underlying bug." 1351705506000000 1351705506000000 3011 enhancement johnglover johnglover ckan 2.0 new Recline fixes and updates for CKAN 2.0 Fixes and updates to Recline / Datapreview for CKAN 2.0. 1352288455000000 1352288455000000 3012 enhancement toby toby new data.gov auth 1352393409000000 1352393409000000 3013 defect dominik dominik new common-error-messages is unreadable "Since the update of the doc theme, the page became unreadable. http://docs.ckan.org/en/latest/common-error-messages.html" 1352553505000000 1352553505000000 3014 defect seanh ckan 2.0 new Crash when deleting a non-empty vocabulary "From Knud Möller: when I try to delete a non-empty tag vocabulary via the API (through HTTP), I get an internal server error. Checking the logs, this turns out to be a consistency error raised by sqlalchemy: Error - : (IntegrityError) update or delete on table ""vocabulary"" violates foreign key constraint ""tag_vocabulary_id_fkey"" on table ""tag"" DETAIL: Key (id)=(21421955-7560-467c-af30-9f790b73e6ae) is still referenced from table ""tag"". 'DELETE FROM vocabulary WHERE vocabulary.id = %(id)s' {'id': u'21421955-7560-467c-af30-9f790b73e6ae'} URL: http://33.33.33.10:5000/api/action/vocabulary_delete The error makes sense, but I'm wondering if it would be useful to extend the API to also allow the deletion of non-empty vocabularies, possibly via a parameter (not sure what best practice in API design is). At the very least, it would be cool if the error message coming back in the response had more information in it." 1352803808000000 1352803808000000 3016 enhancement johnmartin johnmartin ckan 2.0 new CKAN 2.0 template tweaks Just a ticket to keep track of a few suggested template changes. 1352813417000000 1352813417000000 3017 enhancement johnglover johnglover ckan 2.0 new New stable branch for the demo server 1352819088000000 1352819088000000 3018 enhancement johnmartin johnmartin ckan 2.0 new Load more in activity streams Activity streams should be able to load more than 15 items within them. Suggest the default amount of loading is around 30 and then click to load more. 1352900051000000 1355140950000000 3019 defect seanh ckan 2.0 new Cannot delete dataset extras Deleting extras in the web interface is broken 1352918678000000 1352918678000000 3020 enhancement seanh new Update CKAN coding standards 1353096711000000 1353096711000000 3023 enhancement amercader amercader new New methods on IPackageController to provide access to the data_dict Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it 1353603657000000 1353603657000000 3025 enhancement amercader amercader new Add requests to core requirements Because yes please 1354052124000000 1354052124000000 3026 enhancement amercader amercader new Support icons on nav_named_link nav_named_link won't include the nice icon even when passing the icon keyword param. 1354301516000000 1354301516000000 3027 enhancement kindly kindly new solr for 2.0 "change mm support solr 3 and 4 add *_date field" 1354587729000000 1354587729000000 3028 enhancement seanh seanh ckan 2.0 new Feature: dashboard activity stream filtering https://github.com/okfn/ckanext-pdeu/issues/13 1355141244000000 1355141244000000 3030 enhancement toby new clean up helper functions 1355834038000000 1355834038000000 84 enhancement kindly rgrp ckan-future assigned Revert support on versioned objects "Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support: 1. Cases where multiple objects changed in a revision but only want to revert 1 (low priority) 2. Want to revert but have reversion as a new revision of that object. Seems low priority at present. Cost: ?" 1248339543000000 1340626385000000 143 enhancement thejimmyg dread ckan-backlog assigned Most active users listed on homepage Display league of users' recent activity on homepage. 1255010373000000 1312372381000000 235 enhancement tobes dread ckan-v1.9 assigned Resource format normalization and detection "Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service. Sub-tasks: * Create a Google Spreadsheet with two Worksheets: ""MIME-Mappings"", i.e. ""CSV"" -> ""text/csv"" and ""Name mappings"", i.e. ""text/csv"" -> ""Comma-Separated Spreadsheet"". * Collect and map surface forms from all CKANs * Access this via Swiss and apply, store as a PackageResource extra field pending #826 (Resource extras). * Add heuristics for format auto-detections: * Map well-known file extensions * Recognize obvious magic (Zip, Tar) * Peek into Zipfile/Tarfiles * Define a convention for generic data types (many CKAN packages have only ""Spreadsheet"" defined, either detect specific type or set MIME to */tabular-data or similar) * See also: #816 (Autocomplete for the resource format field)" 1263827604000000 1340627624000000 250 enhancement icmurray dread ckan-v1.9 assigned RDF link in Atom feed Add link to RDF representation of a package in our Atom feed. 1266507695000000 1340631430000000 253 enhancement dread ckan-backlog assigned Package relationships "= Overview = Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management. = Examples = 1. There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial. 2. In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.) 3. OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it. 4. IPCC collection of data linked / mirrored. Not sure if there are useful relationships here? 5. Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM. See more examples discussed here: http://trac.ckan.org/ticket/253 = Implementation = This is split into four tickets: * Model: ticket:254 * Read in WUI: ticket:255 * Edit in WUI: ticket:256 * API: ticket:257 No need for write access to be provided API for the moment. This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies)." 1266854721000000 1339774726000000 256 requirement dread ckan-backlog assigned Package relationships - 3. Edit in WUI "WUI: * Editable as part of package or separately? (e.g. like authz) * Do we normalize to only one type name of the pair? * Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?) " 1266928561000000 1339774714000000 277 enhancement zephod dread ckan-backlog assigned Set some config options / settings in WUI (extension) "== Use case == As a ckan administrator I want to easily change options about the CKAN install. == Implementation == === Settings to be in DB === Suggested: {{{ ## Title of site (using in several places including templates and tag ckan.site_title = CKAN ## Logo image to use (replaces site_title string on front page if defined) ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png ## Site tagline / description (used on front page) ckan.site_description = ## Used in creating some absolute urls (such as rss feeds, css files) and ## dump filenames ckan.site_url = ## Favicon (default is the CKAN software favicon) ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico ## An 'id' for the site (using, for example, when creating entries in a common search index) ## If not specified derived from the site_url # ckan.site_id = ckan.net ## API url to use (e.g. in AJAX callbacks) ## Enable if the API is at a different domain # ckan.api_url = http://www.ckan.net ## html content to be inserted just before </body> tag (e.g. google analytics code) ## NB: can use html e.g. <strong>blah</strong> ## NB: can have multiline strings just indent following lines # ckan.template_footer_end = }}} NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ... === Settings / Options / KeyValues Table === Columns: * [namespace]: ? only if KeyValues (for settings this would then always be settings) * key * label * value (json) * type (e.g. date and to specify in advance what type should be) * description * tags: ?? (for grouping ...) === Loading settings from DB === Do this in ckan/config/environment.py === WUI === * /ckan-admin/settings * Show label, plus description plus text field == Depends == * Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard) " 1269274861000000 1318247121000000 285 enhancement rgrp assigned Paginate list of packages on tag read page "Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ... " 1270664606000000 1340631923000000 301 enhancement rgrp assigned Package discussion pages "A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package. It provides a way for people to make suggestions about a package without needing access to main package." 1272301033000000 1340632055000000 728 requirement amercader johnbywater ckan-backlog assigned CSW Harvesting shall be optimised in respect of reharvesting only records that have changed Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please? 1287675340000000 1310124784000000 763 enhancement dread ckan-future assigned Read-only mode - Setup "Admin configures entering read-only mode in one of two places: * CKAN config file (e.g. ckan.ini) * environment variable from Apache config Once enabled, no writes can occur to the database (including user ratings and other usage stats)." 1288091506000000 1338206204000000 765 enhancement dread ckan-backlog assigned Read-only mode - API usage "All writes to the API are captured and you are returned an error explaining the reason. Possible errors: * 503 temporary maintenance * 403 forbidden (if server if permanently read-only)" 1288091897000000 1338206123000000 794 requirement amercader johnbywater ckan-backlog assigned Investigate reconciling UKLP Publisher and Provider with DGU "This needs more analysis, but the GEMINI2 attribute ""metadata point of contact"" must be reconciled with the registered publisher (or agent). This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider." 1289227811000000 1311179581000000 909 enhancement pudo ckan-backlog assigned DCat importer for CKAN "Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported: * CKANrdf generated exports * opengov.se RDF (not really DCat) * Sunlight Nationdatacatalog as harvested by dcat-tools. " 1295265958000000 1346669602000000 948 enhancement dread ckan-future assigned Highlight (to a sysadmin) which packages are deleted "When a customer logs in as a sysadmin then he/she see all packages, including deleted and pending ones. These are hidden to the average user, but the sysadmin has no idea of this until he clicks on the package and sees at the bottom 'state: deleted'. It should be more obvious than that on the search view - an icon, message or crossed-out name to packages are deleted." 1296646369000000 1338206280000000 979 enhancement kindly dread ckan-backlog assigned Edit Resource extras in the API Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API. 1297429777000000 1315222244000000 1041 enhancement thejimmyg thejimmyg ckan-backlog assigned Start Using the CKAN Wiki for Tutorial-style documentation "For example, I will document the following: * A specific guide on using mercurial when working with our branching and merging policy * DONE - http://wiki.ckan.net/Becoming_a_CKAN_Developer * Policy document on how we use tickets, plan sprints and make releases * DONE - http://wiki.ckan.net/Becoming_a_CKAN_Developer * How to package software as .deb file for project deployment I'd love if someone else would write: * An authorisation tutorial covering the core model, the command line tools and examples of every possible way of using the system * A HOWTO guide with screenshots for adding a package " 1300284715000000 1312372367000000 1062 defect johnglover sebbacon ckan-backlog assigned Data preview encoding error "The preview of ""Species Misc Turtle Download"" at http://ckan.net/package/taxonconcept results in the following error: Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte " 1301396143000000 1311773731000000 1069 enhancement tobes rgrp assigned Stub datasets (request for datasets) "Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki. We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ... (Just as we have datasets listed that exist but aren't available ...) Alternative would be to have a request for datasets subsystem. I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists. == Implementation == * Agree a new dedicated tag. e.g. todo.does-not-exist == Related == * This ticket has clear links to http://getthedata.org/" 1301666919000000 1340632215000000 1130 enhancement lucychambers assigned First time users Send users to FAQ first time on CKAN 1304938761000000 1340633514000000 1135 enhancement kindly rgrp assigned Changeset model for vdm "Move to Changeset model for vdm. A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned. This change would also incorporate significant simplication of vdm." 1305209986000000 1340632267000000 1136 enhancement kindly rgrp assigned Move to SessionExtension in vdm When vdm was created there was no !SessionExtension so we use !MapperExtension for doing revisioning. Now that SessionExtension exists we should use it. We can also follow the existing SQLAlchemy recipe: <http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects> 1305210855000000 1340632980000000 1137 enhancement kindly rgrp assigned Remove need for statefulness in vdm "Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is ''required'' because, atm, revision objects have FKs to continuity objects. This ticket proposes the following changes: NB: this could be limited just to case of join tables (leaving state stuff on other tables) * Remove FKs from revision to continuity (or allow for them to be nullable). * We could just limit this to m2m stuff * Delete of an object leads to: * Deletion of continuity object * Adding an entry in revision table with state set to deleted (we retain state on revision table) If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain ""active"" relationships. If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state). The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects)." 1305211628000000 1340631974000000 1168 enhancement thejimmyg dread ckan-backlog assigned Test system for deb packaging "Get buildbot to: * build the deb packages * install them into a fresh virtual machine * run smoke tests on the installed ckan" 1306441994000000 1330990423000000 1171 enhancement mark.wainwright dread ckan 2.0 assigned Citation instructions on dataset and resource view pages "Some sort of citation helper. Something small on the dataset and resource page that would show how to cite. wwaites: Some related thoughts on this from opb: http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf timclicks: I'm looking at Dataverse for the first time[0]. It seems very popular in the social sciences. I noticed that there is a recommended citation for each dataset. For example, [1] is has this one: ""Targeted Input Programme (TIP) 2000-01"", http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1 V1 [Version]"" == Implementation == Add a small box at bottom of dataset / resource page (or in sidebar on dataset page) with title ""Cite this"" with contents like: %title. %author. Retrieved %date. %site_title. For resource: %title = %dataset_title. %resource_name. Could also add export to ref managers (e.g. to bibtex) but that is for later." 1306920799000000 1347358705000000 1240 enhancement kindly rgrp ckan-backlog assigned [super] API v4 "(Just creating this ticket as somewhere to keep notes) * Decide on REST api versus action API * Do we want to support both? * Tidying * Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...) * extras merged into normal field list in package * Get rid of /rest/ so just have api/v1/package * Get rid of separation of search api from 'rest' api * Propose that GET on REST index is search e.g. /package/?q=... * This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned) * Resource read/write in API (separate from package) * Does this need authorization work? * user/account API - read/write * Remove autocomplete -- can just use search * Do not worry about backwards compat as should only be used in our js (if others using it too bad!) " 1311525660000000 1325473312000000 1244 enhancement dread assigned Notes field carriage-returns converted to CRLF "When you edit a package in the web form, if the notes field had {{{\n}}} as the End Of Line symbol, it gets lost when you preview or save the package, and the notes field is displayed all on one line. This can be seen when editing annakarenina (as created by 'paster create-test-data'). The diff shows for example: {{{ - Some test notes + Some test notes ? + }}} but it would more clearly be shown as: {{{ - Some test notes\n + Some test notes ? ++ }}} This is a significant problem with DGU, since a lot comes in via the API. It's not clear what we should do about it. We could standardise on \n or \r\n when the form submission comes in. Do different browsers on different platforms do different things with EOLs? == Analysis == Displaying the package: the Markdown processor respects both EOLs when displaying the field, putting each line in a <p> tag. Creating the package edit form: placed into <textfield>. Browser displaying package edit form: <textfield> displays \n and \r\n as EOL. But \n\n gets compressed to one EOL. But on submission, both are returned as \r\n. Receiving the edited package: Somewhere along the line the EOL gets converted to \n\n. " 1311689456000000 1340191253000000 1255 enhancement kindly kindly ckan-backlog assigned Drupal consistancy checks. Make a robust way to make sure the drupal database is consistent with the ckan data. 1312219968000000 1313400054000000 1285 enhancement dread ckan-future assigned Errors cause emails "Currently a sysadmin gets an email when an exception is not caught. But there are occasions when we DO want to catch an exception so we can fail nicely for the user, but the sysadmin STILL gets an email to know to fix something. e.g. if there is an exception when search indexes a package. You want to catch the exception and still run any other notify calls. " 1314116944000000 1338206151000000 1287 enhancement thejimmyg dread ckan-backlog assigned NAVL validation errors - Junk fields should be listed explicitly "When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message: {{{ {""__junk"": [""The input field __junk was not expected.""]} }}} It should mention the actual key which is not expected. e.g. {{{ {""relationships"": [""The input field 'relationships' was not expected.""]} }}} Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that." 1314203102000000 1330990459000000 1314 enhancement dread ckan-backlog assigned ckanclient search - generator improvements "Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised? Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit & offset)." 1315395410000000 1340191233000000 1317 defect dread ckan-backlog assigned password reset - improve user search "In password reset, it gets confused if you have two similar users. This is because with the string the user provides, it searches several fields, not just name but also fullname and email address, allowing you to search for these. But only name is unique. Specific problem: Ira searches for ""Irina"" then it finds both: {{{<User name=irina fullname=Irina email=irina.bolychevsky@okfn.org ]}}} and {{{<User name=shevski fullname=Ira email=>}}} (I think) Maybe need to choose which field it searches?" 1315415539000000 1340191221000000 1322 enhancement dread assigned Action API improvements "Focusing on improving Action API as the v3 API: * have an optional parameter of the data_dict called ""options"". Options would contain items that would get passed into the context. e.g. {{{{""options"" : {""ref_package_by"": ""id""}}. }}} * instead of using API version to change the way packages are referenced, use the ref_package_by. * All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'. * Action API is v3 of api, replacement for v1 & v2. Default for most urls is still v1, but if url is /api/action then default to v3. Next steps: * Add search API (package, resource, * Add Util API * Clarify JSONP still applies * Add doc strings, clarifying parameters" 1315474749000000 1340191088000000 1328 defect minspamboks@… assigned Unicode & paster commands "A possible bug in CKAN when I tried deleting users using ""paster --plugin=ckan user delete"" command. To reproduce the bug do the following: 1. Create a user with an ID (which in my case was a user's full name) that contains non-unicode caracters like Norwegian ""æ"", ""ø"", or ""å"". 2. Make sure that you can see something like the example below: (pyenv) rm@mycomputer:$ paster --plugin=ckan user Users: name=Rustæm 3. Then try deleting the user with following command: (pyenv) rm@mycomputer:$ paster --plugin=ckan user delete ""Rustæm"" You should now get a python encoding error. I know that this is quite rare case, but in our case it caused some trouble. Could you guys have a look at this bug? CKAN ver. 1.3.3." 1315823110000000 1340191065000000 1358 enhancement zephod rgrp ckan-backlog assigned Generate configuration documentation automatically from the deployment_ini_tmpl file "At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs. Suggest we write a script that automatedly generates reference documentation for the config from the source. May be obsoleted by #277 (some config in db)" 1317076350000000 1318257823000000 1366 defect dread ckan-future assigned Search inside extra fields "SOLR search doesn't support searching for part of an extra field, but it does for other fields. i.e. title=""One Two Three"" matches q=one AND q=title:one and geographic_coverage=""England Scotland"" matches q=England BUT NOT q=geographic_coverage:England This problem emerged when we went to SOLR in #1275 (CKAN 1.5a). Tests were skipped. This is could be a problem for DGU and maybe elsewhere. " 1317290992000000 1338206707000000 1393 enhancement johnglover dread ckan-backlog assigned Don't skip search tests Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break. 1318505453000000 1320153590000000 1447 defect kindly dread ckan-backlog assigned disk space leakage "Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it. One problem in the past has been file handles running out when creating lots of tiny files in the data directory. Another problem has been several enourmous backups being created every day - pdeu on eu25." 1320666843000000 1340727330000000 1458 enhancement rgrp rgrp ckan-future assigned Support previewing kml files in data viewer "Super ticket: #1151 (viewing geo data) E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map" 1320936488000000 1340632932000000 1460 enhancement rgrp assigned Improve extensions documentation "Current extensions documentation needs some work: http://docs.ckan.org/en/latest/plugins.html * Queue extension section may now be out of date (?) * Think about how it integrates with https://github.com/okfn/ckanext-example (especially tutorial and example extension) * Document all plugin points (auto-extract from CKAN source??)" 1321114523000000 1340191001000000 1489 enhancement dread ckan-backlog assigned Updating example theme/extension "ckanext-example needs updating for CKAN 1.5: * theme changes * new forms About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look & operation of CKAN. This moved to github and renamed 'ckanext-example'. " 1322137920000000 1324292384000000 1507 enhancement zephod rgrp ckan-backlog assigned Minor fixes to dataset add on Group edit form - 0.5d "Group edit dataset add form needs some work * Dataset name is not cleared when you add * No way to remove item from list of datasets to be added if I make a mistake * (2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...) " 1323088429000000 1338205220000000 1509 defect dread assigned Mis-dated old revisions of datasets "e.g. http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358 gives: {{{ This is the current revision of this dataset, as edited at 2011-07-12 12:16. }}} but it should say the date 2011-07-12 (which is the data being displayed). The problem is looking at PackageRevision, when it might be a tag or extra." 1323090398000000 1340190814000000 1550 enhancement ross ckan-backlog assigned Allow simple auth via the API It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly. 1324049369000000 1346670055000000 1571 enhancement rgrp jilly mathews ckan-backlog assigned [super] Issues Extension """Proper"" issues extension. * Etherpad: http://ckan.okfnpad.org/issues * Code: https://github.com/okfn/ckanext-issues Overview * This would be an extension and improvement of existing todo extension - #842 (https://github.com/okfn/ckanext-issues) * Current extension does work but lacks polish * Should we polish or do significant improvements? * Functionality - see etherpad == Steps == * ~~(Rename todo(s) -> issue(s) in existing extension)~~" 1324294225000000 1330863183000000 1652 enhancement kindly ross ckan-backlog assigned How we intergrate with Drupal Multiligual? " * Analysis of how/where we can integrate with Drupal Multilingual * Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/... * Check with Ian for other issues. " 1326709894000000 1340187535000000 1661 defect dread ckan-backlog assigned Wrong Routes version installed by CKAN package "Jaakko Louhio reported that the wrong Routes version got installed during CKAN's package install. He is using Ubuntu 10.04 and he believes it install Routes 1.12.3 instead of what we use which is 1.11." 1326718061000000 1339773949000000 1684 enhancement ross ckan-backlog assigned Remove all config from ckanext-archiver "ckanext-archiver currently has a settings file (and a default) and it should be passed in all relevant information from the context. Remove all settings (ARCHIVE_DIR and MAX_CONTENT_LENGTH and others) and pass them in from CKAN." 1326983821000000 1346670037000000 1689 enhancement kindly dread assigned List deleted datasets in API "The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets. 'state' could be a parameter on the package_list call * Easy to call and implement * But getting permissions right for the different state values may not be easy - maybe a separate logic function is better? " 1327062214000000 1340190040000000 1749 enhancement seanh ckan-future assigned Allow creating activity details through API Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts. 1328465406000000 1338213933000000 1778 enhancement seanh seanh ckan-future assigned Replace classmethod's with module-level functions "In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the {{{get()}}} classmethods in the model/ classes. Doing it with module functions lets us type module.function() instead of module.Class.method(). In many cases we're importing classes directly with {{{from module import Class}}} and then doing {{{Class.method()}}}, but {{{from module import foo}}} is bad style: http://docs.python.org/dev/howto/doanddont.html" 1328543614000000 1340623743000000 1831 enhancement ross ckan-backlog assigned Login with email address "Users should be able to log in using either their username, or their email address, both of which are unique. Will require a change to UsernamePasswordAuthenticator in ckan.lib.authenticator.py and possible a useful User.by_email in the user model if it doesn't already exist." 1330073906000000 1346670504000000 1832 enhancement dread ckan-backlog assigned dataset purge API "Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance. Without this, they need to resort to more difficult methods such as: * cleaning and reloading the database * setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names. Requested for NHSIC. == Implementation == * This could slot into the Action API. * Of course we would need to ensure the user's had been given the specific right to purge. * I suggest we log the full details of the dataset being purged." 1330077745000000 1339773706000000 2197 defect zydio ckan-backlog assigned Storage Metadata API: add/update not working with local file storage (Pairtree) "If OFS is configured with Pairtree to use a local file storage, all POST requests to add/update metadata ( /api/storage/metadata/{label} ) will fail. This is due to the use of [https://github.com/okfn/ofs/blob/master/ofs/remote/botostore.py BotoOFS] specific ''private'' methods in [https://github.com/okfn/ckan/blob/master/ckan/controllers/storage.py StorageAPIController.set_metadata()], eg: self.ofs.'''_require_bucket'''(bucket), self.ofs.'''_get_key'''(b, label), self.ofs.'''_update_key_metadata'''(k, metadata) ... those methods can't be found in [https://github.com/okfn/ofs/blob/master/ofs/local/pairtreestore.py POTFS] and this causes errors. The API should use only [https://github.com/okfn/ofs/blob/master/ofs/base.py OFSInterface] methods, or should conditionally make calls based off the actual type of ''self.ofs''. ''PS: I did set ""ckan"" as ""Component"" in the ticket because storage has been integrated back into the core in CKAN 1.6''" 1330421377000000 1346662128000000 2203 enhancement kindly rgrp ckan-v1.9 assigned Data Viewer support for PDF " * Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf * propose we use pdf.js - https://github.com/mozilla/pdf.js * We could also try iframing for browsers with native support but this seems more complex" 1330860293000000 1348568961000000 2234 enhancement seanh ckan-future assigned Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN templates "You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page. This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful. Details - Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds - Feed items should probably be cached somewhere Suggested - The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry. - For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference - Caching is pretty crucial and should probably obey the ttl of the feed. " 1331902755000000 1346669567000000 2284 enhancement ross ckan-backlog assigned Local queries/views/viz in Related stuff. "From #2204 not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ... ### Comments In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ... " 1334218700000000 1346669513000000 2294 defect kindly rgrp ckan-v1.9 assigned Unauthorized action on Data API gets 302 rather than 401 or similar "Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients). Fix this by ensuring that error messsages on API get passed through correctly. Minor because affect is annoying but not terminal" 1334432104000000 1340624003000000 2329 enhancement rgrp ckan 2.0 assigned Add back in RSS/Atom links on relevant pages E.g. on dataset page, on revision page, on user page and on search results. 1335442540000000 1346664271000000 2334 enhancement kindly rgrp ckan-v1.9 assigned Improved data import "Placeholder for thinking about this ... * Allow users to configure info for CSV import (e.g. separators, quotes etc)" 1335651662000000 1340624538000000 2335 defect seanh seanh ckan-v1.9 assigned Unicode characters don't work in CKAN ini files You get a UnicodeDecodeError, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example. 1335789726000000 1340624635000000 2337 enhancement seanh seanh ckan-v1.9 assigned Sort out ini file confusion in docs "For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same. Either fix the docs or fix CKAN to use same ini file names in package and source installs/" 1335797876000000 1340624600000000 2348 enhancement ross ckan-future assigned Java client library for CKAN "Start a Java library for interacting with the CKAN Action API. Currently requires support for Groups, Tags, Resources, and Search. An example app would be very useful." 1335879980000000 1346670024000000 2370 enhancement seanh seanh ckan-v1.9 assigned Add examples of all config settings in default config file The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there. 1336042237000000 1340624662000000 2378 enhancement ross ckan-future assigned Extract metadata directly from resources that contain it. "Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data. [ ] Obtain list of embedded metadata 'standards' [ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded. [ ] HTML? [ ] Image formats (exif/gif etc) [ ] PDF [ ] Eventually consider ... [ ] MS Office documents " 1336651088000000 1346669985000000 2404 enhancement rgrp ckan-future assigned Auto-screenshot for related items "As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere Implementation: (?) Use phantomjs or something like http://snapito.com/ Investigate for 1.8, will be needed for TDH Oct." 1337583347000000 1346669526000000 2405 defect icmurray rgrp ckan-v1.9 assigned Improvements to data viewer embed " * Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...) * Current icon and text looks really bad * Embed link uses uuid rather than dataset name (this goes to perma-url question ...) * Data viewer embed url includes all the resource info (not needed -- just url and backend info ...) * Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org " 1337583565000000 1340635981000000 2412 defect zephod icmurray ckan-sprint-2012-06-25 assigned More than one resource invalidatiing breaks dataset edit form "When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state. Repro: 1. Go to /dataset/new 2. Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size... 3. Add another resource, doing the same thing: make one of the fields invalid. 4. Try to save the dataset. 5. The entered resource information will be lost, and a js error ""Uncaught Error: Can't add the same model to a set twice,: backbone.js:586"" will be thrown." 1337717148000000 1338203234000000 2446 enhancement johnmartin aron.carroll demo phase 4 assigned Create demo dataset history/comparison page "This is a low priority for the demo site. Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments Implementation: http://s031.okserver.org:2375/dataset/history/adur_district_spending" 1338210738000000 1352658819000000 2451 enhancement johnmartin aron.carroll demo phase 5 assigned Create demo user list page "Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments Implementation: http://s031.okserver.org:2375/user" 1338211213000000 1352658902000000 2457 enhancement johnmartin aron.carroll demo phase 4 assigned Create demo tags list page "This includes the tag page as well for now. Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments Implementation: http://s031.okserver.org:2375/en/tag" 1338211735000000 1352658878000000 2461 enhancement toby aron.carroll demo phase 4 assigned Create demo revisions page "I believe this is to be merged in with site activity. Discussion: https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts Implementation: http://s031.okserver.org:2375/en/revision" 1338212055000000 1344543838000000 2473 enhancement kindly kindly ckan-v1.9 assigned Make datstorer store field ordering in _meta field Recline views should have a default table order, being the same as the csv that was imported. 1338293492000000 1340636871000000 2513 enhancement ross ckan-backlog assigned Dataproxy should not default to utf8 "Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8. Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set. This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside." 1339575820000000 1346669646000000 2520 defect seanh seanh ckan-v1.9 assigned Document undocumented config options "There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either: ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled There are also some options that are in the default deployment.ini even though they're deprecated: ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously See email to ckan-dev from David Read: http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see #1358" 1339588368000000 1340624908000000 2535 enhancement rgrp assigned SSL certificate for DataHub + https by default "DataHub is increasingly used and we should ensure it uses ssl as part of general security. See also #1446 (Need to support https login for multiple instances as part of the CKAN package install)" 1339758027000000 1346662082000000 2546 requirement ross ckan-backlog assigned ODS Managing homepage content " == Requirements == Require the ability for users to control some level of content that is visible on the home page of their ODS installation. This may be through RSS/Atom feeds (see #2234) or another mechanism but should result in admins being able to change blocks of text on their homepage. This should not be configuration, but accessible through WUI. == Interface == None == User Stories == * As a system administrator I want to have control over content displayed on the front page beyond featured/popular items. * As a system administrator I don't want to manage content through having to write an extension. == Tasks == [ ] Analysis == Estimates == " 1340016842000000 1346663437000000 2547 enhancement shevski ross opendatasuite 2 assigned ODS Initial data sets " == Requirements == The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS. May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps. == Interface == None == User Stories == * As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed. * As a bizdev person I would like to be able to demonstrate how ODS works with real datasets. == Tasks == [ ] Identify relevant sources for datasets [ ] Pick datasets [ ] Set them up for import == Estimates == " 1340016906000000 1340705614000000 2548 enhancement kindly ross datahub-july assigned Object ownership for groups/package " == Requirements == We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'. At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner. More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature. == Interface == None == User Stories == None == Tasks == [ ] Analysis/Clarification [ ] Tests [ ] Migration [ ] Code/Schema changes [ ] Documentation == Estimates == " 1340017331000000 1340706539000000 2550 enhancement icmurray ross datahub-july assigned User types "== Requirements == In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub). == Required interface == Once changes have been made to the user schema, for a given user we want to be able to: * determine if they have a paid or a free account, and * get a string name of the type of paid account. Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin. == User Stories == User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands. Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all. * As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely. * As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have. * As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else. * As a user, and only if I have one, I'd like to see my current payment level on my user profile page. == Tasks == [x] Tests [x] Plugin based migration [x] Code [x] Model [x] API [x] Documentation == Estimates == " 1340017590000000 1346669497000000 2552 enhancement ross ckan-future assigned Controlling access to features "== Requirements == To provide a freemium service it is necessary to be able to provide differing __levels__ of functionality based on the '''type''' of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks. Initial implementation should focus on limiting access to datastore disk space. == Interface == These changes are currently only for the data hub and should be kept as much as possible within the data hub extension. == User Stories == * As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited). * As a system administrator I don't expect to need to manage the levels of users '''or''' the features that this applies to. == Tasks == [ ] Clarification of requirements/analysis [ ] Tests [ ] Code [ ] Model [ ] API [ ] UI [ ] Documentation == Estimates == " 1340018770000000 1346669544000000 2554 enhancement ross ckan-backlog assigned Research Virtuoso cartridges "Look into writing a cartridge for importing CKAN data into a Virtuoso quadstore http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger#How Does It Work?" 1340026645000000 1346670433000000 2590 enhancement shevski ross ckan-backlog assigned Publisher dashboard "Need proper user stories but ... Publisher admins/editors may need a more useful group read page showing things like: * The current search * Recent activity * People within the group * Followers * Others? " 1340618617000000 1346663416000000 2607 defect seanh ckan-backlog assigned 'Upload a file' appears on resource form when storage not enabled "if the user tries to upload a file they will get ""Failed to get credentials for storage upload. Upload cannot proceed"" Maybe add a test for it this time, this bug has appeared and reappeared before" 1340803808000000 1346663383000000 2619 enhancement seanh seanh ckan-v1.9 assigned Omit private datasets from public activity streams "Activities about private datasets should not appear in public activity streams. I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams. I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out' I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?) The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets. An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private. Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered." 1340884140000000 1351531137000000 2641 enhancement johnmartin amercader demo phase 5 assigned Adapt spatial widgets to new theme Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now 1341846147000000 1352658854000000 2644 enhancement shevski toby demo phase 5 assigned user dashboard for demo theme "we now have a user dashboard that needs theming not sure if we need sam to look at it http://localhost:5000/user/dashboard" 1341910821000000 1344255836000000 2654 enhancement ross ckan 2.0 assigned UI support for ordering groups on group_read page "The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name Add support for this for datahub now, and discuss for new 1.9 UI" 1341943891000000 1346662156000000 2674 defect kindly shevski demo phase 5 assigned Data preview not loading on s031 "Not loading for all resources as far as I can tell; e.g. http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360 Fine on demo.ckan.org" 1342435102000000 1344349324000000 2686 defect shevski ckan-backlog assigned enabling datastore & data API breaks recline "First I noticed that the gold prices dataset preview was not displaying & has data API enabled Secondly I tried enabling datastore for http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f and after I saved the preview no longer worked. Unticking the datastore & data api checkbox brought it back Is this a known issue?" 1342516011000000 1346663300000000 2697 enhancement johnmartin shevski demo phase 5 assigned create dataset validation Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check) 1342620035000000 1346235925000000 2698 enhancement toby shevski demo phase 4 assigned markdown preview for description / other fields with markdown support 1342620085000000 1344543252000000 2699 enhancement shevski shevski demo phase 5 assigned workflow for associating datasets with groups "needs review & speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group" 1342620176000000 1344507133000000 2702 enhancement shevski shevski demo phase 5 assigned Future Javascript wishlist for demo "tooltip on popular datasets with number of views facets to update automatically creating a dataset without reloading page between steps hover on licences information autocomplete on search terms group filtering social share buttons in lightboxes dataset counts on homepage" 1342620475000000 1344255984000000 2732 enhancement ross ckan-backlog assigned New file upload functionality "We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is http://ckan.okfnpad.org/uploads '''Simplifying uploads''' Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces. We want to fix: * File uploads themselves * Storage of uploaded files * Notification of the upload to other components '''File uploads''' Things file upload should do: * Allow sysadmin to disable * Allow auth'ed users to upload * Store whatever they send on disk, and store DB entry linking the file to the person * When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files. {{{ Can we do the upload asynchronously and then associate the uploaded key with the resource before the save ? What happens if the user tries to submit before asymc upload finishes ? Should we delay them? }}} The upload workflow should look like... 1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data. 2. a. When called via ajax then the ID of the newly created file should be returned, b. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param. 3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource. This should enable: * Separate file upload into a user's temporary store, either individually or as a batch. * Creating resources and simply choosing from previously uploaded, unassigned files * Adding files/data to a resource after the fact. '''File storage''' File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request. File storage should: * allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths). * provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up. * allow for collection of information (i.e. estimate of storate space used) * check whether there is enough space and handling the conequences cleanly * ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage. * Have a configurable maximum accepted blob size during upload. * Should store what meta-data was provided with the upload, such as mimetype. Somewhere in the DB we should store ... ||'''Column'''||'''Notes'''|| ||id||An identifier|| ||owner||The owning user, who uploaded the file|| ||path||The path (from the 'storage root') to the file|| ||size||The size in bytes of the file on disk|| ||mimetype||The mimetype of the file, as provided by the uploader|| ||upload_date||When the data was uploaded|| ||resource||The ID of the resource it belongs to. A unidirectional relationship.|| ||archived_url||The URL where this file has been archived|| Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files. '''Notifications''' We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store. We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file). " 1343058789000000 1346663270000000 2735 enhancement toby shevski demo phase 5 assigned Dataset order on user page "I think the datasets on user pages http://s031.okserver.org:2375/user/me should be ordered by latest updated (with most recent at the top) instead of in alphabetical order. What do you think?" 1343062877000000 1344349245000000 2766 enhancement seanh shevski demo phase 4 assigned prevent draft datasets making it to activity stream "The new ckan creates datasets as part of a 3 phase process. To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'. These datasets should not be seen as active by the activity stream. If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created. When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created We do not want these adding. Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it. finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream. The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there. Let me know if you need any help with this ticket or a better explanation of the problem." 1343318795000000 1344543193000000 2785 enhancement johnmartin aron.carroll demo phase 5 assigned Allow resources to be re-ordered "Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource? Ira, what are your thoughts?" 1343816523000000 1346235916000000 2814 enhancement shevski markw assigned Demo: upload file behaves oddly "3. Uploading a file behaves counter-intuitively (I would suggest wrongly). When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen: * the pathname of that file is filled in the box; * nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc). Instead of this, * the file is immediately and, seemingly, irrevocably uploaded; * the box is filled with a mysterious URL (https://commondatastorage.googleapis.com/ckan-demo ...) * the mysterious URL is so long I can't even see the final element, which is my file name. " 1344420360000000 1346670381000000 2818 defect seanh danieljohnlewis demo phase 4 assigned Improve related item schema "Problem: When creating a related item (e.g. a Visualisation), if you don't put in a URL it succeeds, but on the related items and apps pages it renders it as a link to the same page. Expected: Always require a URL and it should only submit if one is added" 1344504176000000 1346231718000000 2873 defect toby danieljohnlewis demo phase 4 assigned ckanext-qa: dataset summary incorrect " For example see: http://s031.okserver.org:2375/qa/dataset/five_stars Which has: Example dataset as a score of: 4 After examination of the dataset ( http://s031.okserver.org:2375/dataset/example-dataset ) it is made up of a CSV and an HTML file. " 1345126429000000 1345208283000000 2874 enhancement rgrp rgrp assigned Clean up bin directory Full of obsolete material 1345190508000000 1345190515000000 2877 enhancement kindly rgrp assigned Bugs with datastore v2 "In progress 1. ~~[major] q does not seem to work reliably. e.g. using the setup from this gist https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with ""q=de"" either)~~ * ~~q=second does work ...~~ 1. [major] q does not work with 2 values (see below) 2. Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT * Also weird fact that limit is returned but as as as string - should it not be an integer 3. Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...) === Multiple query values === Try a query such as: ""second UK"" and you will get 500 error: http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK {{{ ProgrammingError: (ProgrammingError) syntax error in tsquery: ""second UK"" 'select ""_id"", ""id"", ""date"", ""x"", ""y"", ""z"", ""country"", ""title"", ""lat"", ""lon"", count(*) over() as ""_full_count""\n from ""4f1299ab-a100-4e5f-ba81-e6d234a2f3bd"" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',) }}} === Suggestions === Filter support: should think in more detail about this (may want to follow recline style) Simple filters in query parameters would be nice too ..." 1345250002000000 1346320395000000 2956 enhancement shevski johnmartin ckan 2.0 assigned Allow for resource editing in CKAN 2.0 On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually. 1349277100000000 1349346442000000 2961 enhancement dominik dominik ckan 2.0 assigned Preview plugin endpoint "Users could write ckan extensions that offer previews for a certain datatype. Advantages: - We can say that you can add your own previews - Every preview would have to define it's own dependencies - External development (community creates new extensions?) Questions to answer: - Specification of the interface - Precedence of previews for the same data types - Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible. As far as I can see, this should be fairly easy to implement by following these instructions: http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface" 1349696367000000 1350577990000000 3029 defect seanh dread assigned JSONP parameter scuppers Search in API "{{{http://datahub.io/api/2/search/package?jsonp=jsonpcallback&q=canada}}} returns {{{ {""count"": 0, ""results"": []} }}} I believe this worked in CKAN 1.4 or 1.5, but it is broken on 1.7.1, 1.8 and whatever demo.ckan.org is running. I suspect the jsonpcallback parameter is getting sent to SOLR. This bug prevents using javascript on another site to search CKAN (although hopefully the action API would work)." 1355238035000000 1355243824000000 350 enhancement dread ckan-backlog reopened Search engine optimisation "Need to research what can easily be done to improve CKAN packages in the search rankings. Comments from Glen Barnes: We've been pretty successful at SEO without even really trying (see http://www.google.co.nz/search?client=safari&rls=en&q=auckland+google+transit+feed&ie=UTF-8&oe=UTF-8&redir_esc=&ei=dsYSTOzJLs2eceuZiI8I as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (http://www.ckan.net/package/ascoe -> http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment), setting the H1 tag correctly (""Atmospheric Chemistry Studies in the Oceanic Environment"" on the example above). Some basic SEO 101 on page optimisations. " 1276594541000000 1339774690000000 1341 enhancement kindly kindly ckan-backlog reopened Delete spam users from ckan Spam users where added to thedatahub and we need to clean them. 1315995034000000 1320141540000000 2202 enhancement rgrp ckan-future reopened Display page view count on dataset and resource pages Just like we display download counts we should display view counts. 1330765455000000 1338204929000000 2243 enhancement seanh seanh ckan-v1.9 reopened Fix ckanext-example 1332172710000000 1340635768000000 2331 defect kindly rgrp ckan-sprint-2012-05-29 reopened Search should AND terms not OR terms "Appears current default search in CKAN ORs terms rather than ANDing them (i.e. adding more terms increasing number of items found rather than reducing it). Not sure when this crept in or if it has been there for a long time." 1335637485000000 1356474344000000 1 enhancement somebody rgrp milestone1 closed fixed Visitor visits service "== Title == Visitor visits service == As a == Visitor (general web user) == I want to == Visit the website of the service (frontpage) == So that == * See the front page and discover information about the service * Perform various activities related to the service such as registering and finding packages, registering as a user and logging in" 1152549417000000 1183636342000000 2 enhancement somebody johnbywater milestone1 closed fixed Register as user on the service "== As a == Visitor == I want to == Be able to register as a user on the service == So that == One can login and be an authenticated user and perform the actions that only authenticated users can perform == Notes == * Registration involves the provisional of a minimal set of user details, namely: 1. username (unique in lifetime of the service) 2. fullname 3. email 4. password " 1152549635000000 1152553157000000 3 enhancement somebody johnbywater milestone1 closed fixed Login to the service "== As a == Visitor == I want to == Login to the service == So that == I am now an authenticated user and may perform the actions that only authenticated users may perform == Notes == * Login by providing username and password * Login screen should also include a link or information about registration * If username do not match (or the username does not exist) a login failure is noted and the visitor is re-presented with the login screen as well as information on how to register " 1152549717000000 1152553322000000 4 enhancement somebody johnbywater v0.7 closed fixed Find a package "== As a == Visitor == I want to == Locate packages that I am, or might be, interested in == So that == I know what packages exist and can read the information associated with those packages == Notes == Find a package by searching by (or some combination thereof): 1. tag 2. string contained in title or description 3. license " 1152549820000000 1223908381000000 5 enhancement johnbywater johnbywater v0.3 closed fixed Browse list of packages and select one to view "== As a == Visitor == I want to == Browse a list of packages resulting from a search or browse request (see other use cases) == So that == I can select one of the packages to view in more detail (-> viewing an individual package ticket:6) == Notes == * When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list * The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable " 1152549884000000 1185473622000000 6 enhancement somebody johnbywater milestone1 closed fixed View a package "== As a == Visitor == I want to == View the details of an individual package == So that == I know more about the package. Attributes available are detailed in updating a package (../UseCase12) " 1152549960000000 1183648175000000 7 enhancement johnbywater johnbywater closed invalid View a release associated with a package See Update a release (ticket 14) for more information on what should be shown. 1152550022000000 1204136209000000 8 enhancement johnbywater johnbywater closed invalid View file "== As a == Visitor == I want to == View the files associated with a package. == Notes == * See also ticket:16 (updating a file). * Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release) " 1152550057000000 1250619147000000 9 enhancement somebody johnbywater milestone1 closed fixed Register new package "== As a == Authenticated User == I want to == Create/register a package == So that == The package will be listed in the system and visible to users == Notes == * Full package registration process may be broken down into: 1. Creating a package by provision of minimal attributes: 1. name: unique and web-usable 2. auto-generation of default values such as: 1. date-registered (today's date) 2. first owner (one owner should be created corresponding to the authenticated user performing this action) 3. (?) auto-generate a default release 2. updating the created package -> ../UseCase12 " 1152550171000000 1152555104000000 10 enhancement johnbywater johnbywater milestone15 closed worksforme Update package "== As a == Authenticated user == I want to == Update the attributes of a package and add/remove releases == So that == The package attributes are changed. Package attributes are: * A title * A description (short or long) * Date added * Project url * package license(s) * download url * A list of releases with their dates and a summary line * a list of owners (referencing authenticated users) I would also like to be able to add comments == Notes == * Only an authenticated user is one of owners of a package may update it * Any authenticated user may add comments (but visitors may not) * can break down into: * Update a package by changing simple package attributes (value: 8) * Update a package by adding or removing an owner (value: 5) * Update a package by adding or removing comments (value: 7) " 1152550243000000 1183648150000000 11 enhancement johnbywater johnbywater milestone16 closed fixed Delete package "== As a == Authenticated user == I want to == Delete a package == So that == So that the package is no longer displayed for visitors of the site == Notes == * The permissions for this action are the same as for updating the package details (see ticket:10) * When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator " 1152550328000000 1157371568000000 12 enhancement johnbywater johnbywater closed invalid Create a release associated with a package "== As a == Authenticated user == I want to == Create a release associated with a package == So that == The package has a new release == Notes == * Permissions are as for updating a package ticket:10 * for details of release attributes see: updating a release ticket:14 " 1152550407000000 1204135071000000 13 enhancement somebody rgrp closed invalid Delete a release 1152550611000000 1218545330000000 14 enhancement johnbywater johnbywater closed invalid Update a release "== As a == Authenticated user == I want to == Update a release == So that == The release details are amended. Attributes of a release are: * name (may include '.' and '-' * nickname * date * description == Notes == * permissions: as for updating a package (ticket:10)" 1152550712000000 1204136163000000 15 enhancement johnbywater johnbywater closed invalid Create a file "== As a == Authenticated user == I want to == Create a file associated with a release == So that == The release has a new file associated with it == Notes == * Permissions are as for package updating (see ticket:10) * file attributes are described in file updating: ticket:16) * there are no attributes that are required to be supplied to create a file " 1152550770000000 1250619139000000 16 enhancement johnbywater rgrp closed invalid Update a file "== As a == Authenticated user == I want to == Update a file associated with a release == So that == The file attributes are changed. File attributes are: * (internal) id * description * source: (url usually) == Notes == * Permissions are as for package updating " 1152550796000000 1250619162000000 17 enhancement johnbywater johnbywater closed invalid Delete a file See ticket:16 (update a file) 1152550822000000 1250619180000000 18 enhancement somebody johnbywater closed invalid Purge a user (registered person) 1152550871000000 1199787607000000 19 enhancement somebody rgrp closed invalid Undelete a user (registered person) 1152550881000000 1199787631000000 20 enhancement johnbywater johnbywater v0.3 closed fixed Purge a package "== As A == Administrator == I want to == Purge a deleted package == Notes == * this need only be possible via the admin interface" 1152550910000000 1185473187000000 21 enhancement somebody johnbywater v0.11 closed fixed Undelete a package 1152550936000000 1265891796000000 22 enhancement somebody johnbywater closed invalid Update person attributes 1152551014000000 1199787243000000 23 enhancement somebody johnbywater closed invalid Request new password (or a reminder) if old has been lost 1152551256000000 1199786799000000 24 enhancement somebody johnbywater v0.9 closed fixed View list of most recently registered or updated packages on front page 1152551291000000 1246441366000000 25 enhancement johnbywater johnbywater v0.6 closed fixed Autocompletion (and suggestion) of package tags when adding tags to a package "== Notes == * Currently have a text box where users can add space separated tags * Tags should auto-complete in a manner similar to delicious * would also be good to have a list of suggested tags. This could either be: * [simpler] a list below the tag box chosen by existing popularity or * [more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far). * For first attempt to should probably just go for the simpler option." 1152551313000000 1204131522000000 26 enhancement somebody johnbywater closed duplicate A registered person creates their own tags for a package 1152551351000000 1152555283000000 27 enhancement johnbywater johnbywater v0.6 closed fixed Register/Update a package via a web-api "= As A = User = I want To = Create/register and update a package via a web api." 1152551379000000 1215543933000000 28 enhancement somebody johnbywater closed wontfix Provide a dump of all packages listed on the system in XML 1152551416000000 1200902911000000 29 enhancement rgrp rgrp milestone16 closed fixed Improve installation process " 1. install documentation (currently source:trunk/INSTALL is empty) 2. improve ckan-admin as necessary 3. create default data (such as licenses)" 1152553638000000 1183648166000000 30 enhancement zool rgrp v0.3 closed fixed Tags are added to a package by package owner "== As A == Site User == I Want To == Create/Edit/Update/Delete tags associated with my package == Notes == * Tags are service-wide and '''not''' package specific * Tags should not allow spaces (but allow everything else e.g. ':') * web interface details * should provide a text box where users can add space separated tags (a la delicious). * tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background) * would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious " 1155905753000000 1185472559000000 31 enhancement somebody rgrp closed wontfix Add a comment to a package "== As a == Registered user == I want to == Add a comment to a package == Notes == * The comment should be tagged with the username of the person making the comment * Should we allow comments per release as well as per package? '''No''' for the time being. * Comments will be 'flat' and '''not''' threaded (at least for the present) * Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)" 1157374505000000 1185472236000000 32 enhancement rgrp rgrp v0.3 closed fixed View a tag "== As a == Visitor == I want to == View a tag's page which should list all packages associated with that tag (or link to a page containing that list). == Notes == * The tags should have a nice primary url such as /tag/<tag-name> (could also use the plural i.e. /tag/<tag-name>)" 1157375968000000 1185472746000000 33 enhancement rgrp rgrp v0.5 closed fixed List and search tags "== As a == Visitor == I want to == View a list of the tags on the system and search for a tag by name == Notes == * As for a search of packages should search for tags that contains a given search string (or could use a glob approach allowing wildcards)." 1157375983000000 1199788094000000 34 defect somebody zool closed worksforme Page Not Found - returns with 200 rather than 404 as it should 1163008961000000 1253781550000000 35 defect rgrp rgrp v0.4 closed fixed IP Address for non-logged in users is always 127.0.0.1 when deployed behind a proxy (e.g. on ckan.net) "When IP address is used to identify non-logged in users (for example on create and editing of packages) it is always 127.0.0.1 when using a reverse proxy (standard deployment configuration on a production machine and used e.g. on http://www.ckan.net). This could be corrected by using HTTP_X_FORWARDED_FOR when available." 1177006633000000 1185471537000000 36 enhancement rgrp rgrp v0.4 closed fixed Edit package page should support previews When editing a package one would like to be able to preview one's changes before saving. 1177007112000000 1185470035000000 37 enhancement rgrp rgrp v0.5 closed fixed Purge a Revision (i.e. purge all changes associated with that revision) "Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged). In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort)." 1192650660000000 1199786536000000 38 enhancement rgrp rgrp v0.5 closed fixed Make data available in machine-usable form "Currently all data is only available via the web interface which makes it difficult to obtain in bulk. Data should be provided in other more convenient, machine usable forms: * DB Dump * JSON for each package page (maybe Atom as well)" 1194521614000000 1200903004000000 39 enhancement johnbywater rgrp v0.7 closed fixed When listing packages show more information than just name. "When a package listing is shown (as on http://ckan.net/package/list) more information than just name should be shown. For example package titles should be shown as well as names (could also list tags and other stuff but simplest is just title). Cost: 1" 1194608920000000 1223908298000000 40 defect rgrp rgrp v0.5 closed fixed Reserved html characters (such as &) in urls mean package does not render for read view "A url such as: http://someurl.com/xyz?x=1&VERSION=1.1&Service=WFS when set as url or download_url breaks the rendering of the package with an error like: There was an error rendering the package: not well-formed (invalid token): line 1, column 181 Have checked that removing the & stuff makes the error go away so this looks like an issue with escaping urls when displaying them ..." 1195565228000000 1200993319000000 41 task rgrp rgrp v0.5 closed fixed Upgrade to Latest Version of Pylons (0.9.6.*) and Deploy on Production Machine Should do this sooner rather than later to avoid 'clear water' opening up and need to deploy then asap onto production machine to avoid 'clear water' between trunk and production (which then prevents pushing from trunk to production). 1199786854000000 1199787967000000 42 enhancement rgrp rgrp v0.6 closed fixed In WUI where List of Items to Display is Large Results should be Paged "= As A = Visitor = I Want To = View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner. = Details = Main place this issue arises: * List of tags * List of packages * Repository history (/revision) Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display." 1199787162000000 1204133257000000 43 enhancement rgrp rgrp v0.10 closed fixed Generic Attributes for Packages "= As A = User = I Want To = Add arbitrary named attributes to packages (an attribute being a name, type, value triple). = Details = * We will do this using a dedicated (versioned) table associated to Package * Do we allow multiple attributes of the same name? * For the present: No (since we will key by attribute name) * Could allow for single attribute but with multiple values using json list ... * What types do we allow or do we just rely on JSON to take care of this? = Questions (Original) = * How complex is this to implement? * What would an arbitrary user be able to edit? Possibilities: 1. 'create new attribute' and setting the value (so name and type would be chosen from predefined list). 2. 'create', setting of name and value (but not type -- type already set in predefined list) * Could just use (machine) tags -- though this could be seen as a bit of a hack. * Would solve having to create special file/url attributes (though I think that perhaps file stuff is important enough to merit first class support in the domain model -- though, that said, since one won't want to have a file limit adding unlimited file support is very similar to unlimited attributes of arbitrary type)." 1199788109000000 1253709802000000 44 enhancement johnbywater rgrp v0.7 closed fixed Provide RSS/Atom Feed of Repository History "= As A = Visitor = I Want To = Get an RSS/Atom Feed of the Repository History to use in my feed reader (or elsewhere). = Details = * Preference for Atom. * should just add parameter to /revision/list/ (or /revision/) to select atom format e.g. ?format=atom. * should have a 'days' attribute specifying number of days back to go e.g. &days=30 = Cost = Low" 1199788457000000 1223390660000000 45 enhancement rgrp johnbywater v0.6 closed fixed Immunity to SQL injection attacks 1201110894000000 1204133342000000 46 enhancement rgrp johnbywater v0.9 closed fixed Return error documents that look and feel like normal CKAN pages 1201111018000000 1265891789000000 47 enhancement rgrp johnbywater v0.6 closed fixed Return OpenID signin pages that look and feel like normal pages 1201183920000000 1215543616000000 48 enhancement rgrp johnbywater ckan-v1.4-sprint-1 closed fixed Create favicon Because CKAN's worth it! 1201202789000000 1297077492000000 49 enhancement rgrp rgrp closed invalid Filter Spam in Changes to CKAN Data "= As A = sysadmin = I Want To = Have revisions to the CKAN data filtered in order to reduce the spam in the system. = Details = In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution. Suggest we follow path of trac: http://trac.edgewall.org/wiki/SpamFilter Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing'). Main initial plugins would be: * regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...) * could augment using the badcontent list approach (can find list on e.g. moinmoin) * spambayes and/or akismet" 1204134691000000 1257244973000000 50 task rgrp rgrp closed fixed Investigate apt-get in more detail "Research apt-get especially wajig and python wrappers for apt-get. Post results on the wiki. " 1215456052000000 1267648356000000 51 enhancement rgrp rgrp v0.7 closed fixed Upgrade CKAN to use sqlalchemy and vdm v0.2 "Move CKAN codebase to use sqlalchemy (rather than sqlobject) along with vdm v0.2. NB: the upgrade itself should not be too hard, the work will be in providing a satisfactory migration script to convert existing data on ckan.net service." 1220900438000000 1223908230000000 52 enhancement rgrp rgrp v0.7 closed invalid Paginated results sets should be customizable across domain objects " 1. Tag results could/should show number of associated packages 2. Package results should show title (and be in list) 3. Pagination should be able to be applied to other result sets than simply register listing (e.g. search results ...) 4. Revisions should be listed in a table with full details" 1223549648000000 1223908425000000 53 enhancement rgrp rgrp v0.8 closed fixed View Information about Package history (i.e. package revisions) "= As A = Visitor/User = I want to = View information about a package history, i.e. what revisions it has. = Details = * History info should be available at package/history/pkg-name * Most basic info is just info from revision (when it occurred, author etc) * More sophisticated would be to have diff support to see differences between revisions" 1223552139000000 1239133021000000 54 enhancement johnbywater rgrp v0.7 closed fixed Support dump and load of CKAN data to JSON "= As a = Sysadmin = I want to = Dump (serialize) CKAN data to a simple transport format (e.g. JSON) and be able to load it again. = Details = * Already have simple db dump. However dumping to JSON has various advantages, particularly where changes to the data need to be made upon reloading (e.g. during a migration). * Dump should include *all* CKAN data (i.e. all the data in the CKAN db tables)" 1223907883000000 1230211256000000 55 enhancement rgrp rgrp v0.7 closed fixed Code to migrate data from v0.6 to v0.7 using dump and load "Associated to ticket:51 (upgrade CKAN to new vdm) and ticket:54 (dump/load) need to convert v0.6 data for v0.7. Obvious way to do this is via alteration to data load method." 1223908240000000 1223909891000000 56 enhancement rgrp rgrp v0.8 closed fixed Show icons indicating package openness "== As A == User or Visitor == I Want To == See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package. == So That == It is easy to see the openness status of a package. == Details == * Suggested by Liz Turner * Openness = open license at present (no easy way to determine whether accessible) * Have separate tick for downloadability (i.e. existence of download url) * Could do more stuff in future (e.g. money where tag price- exists, etc etc)" 1230211123000000 1239018857000000 57 enhancement rgrp rgrp v0.8 closed fixed Basic datapkg integration with CKAN "Using datapkg a user can: * list packages on CKAN * get info about package on CKAN * register and update a package on CKAN " 1238750826000000 1239018717000000 58 enhancement rgrp rgrp v0.9 closed fixed Make bookmarklet to enable easy package registration on CKAN "Make bookmarklet to enable easy package registration on CKAN == Details == 1. This requires redoing web interface to allow 1-step package creation/editing (as opposed to name then everything else approach at the moment)" 1239018517000000 1239974365000000 59 enhancement johnbywater rgrp v1.0 closed fixed Provide a basic guide to CKAN on the site " 1. Use contents from ckan package notes section (rgrp) 2. Update that section to be more guide-like (jwyg)" 1239018583000000 1273080019000000 60 enhancement rgrp rgrp v0.9 closed fixed Front page usability improvements "1. List in sidebar 3 latest created/updated packages. 2. Put package search box on front page (might also want to move some of front page content into an about page). 3. Similarly might want package creation box on front page." 1239094266000000 1246434921000000 61 enhancement rgrp rgrp v0.10 closed fixed When dumping data to json do not dump private information like API keys Cost: 1h 1239123529000000 1265890790000000 62 enhancement dread rgrp v0.10 closed fixed Change tags to contain any character (other than space) Requires us to url encode the tag names when displaying them ... 1240585095000000 1250181376000000 63 defect rgrp rgrp v0.9 closed fixed Purge is broken after upgrade of vdm "Since we upgraded to new vdm (v0.4) which has built in purge support existing purge support looks to be broken. Should be reasonably easy to fix this. Cost: 1h" 1245148623000000 1245656000000000 64 enhancement rgrp rgrp v0.10 closed fixed Switch to repoze.who for authentication "Switch to repoze.who + openid plugin for authentication from authkit. Already did this in microfacts so should be fairly easy." 1245168853000000 1246127063000000 65 enhancement rgrp rgrp v0.9 closed fixed Add version attribute to package "It is essential packages have a version attribute to assist in tracking releases etc (and to allow compatibility with other packaging systems e.g. distutils which is being used by datapkg). cost: 2h" 1245262864000000 1246434569000000 66 enhancement rgrp rgrp v0.10 closed fixed Improve user account UI "1. Link at top should say ""Your Account"" when you are logged in * when not logged in should say: ""Log in via openid"" * Open Id info on /account/ should move to /account/login/ or be deleted * Going to /account/ when not logged in should redirect to /account/login/ 2. When logged in the basic account index page should not give generic information but should reflect fact you are logged in by: * Saying something like: ""You are logged in as: ..."" * Showing your apikey or a link to apikey * Giving you a logout link We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets." 1245263685000000 1250785405000000 67 enhancement rgrp rgrp v0.10 closed fixed List all of a user's recent edits on their home page "Additional feature related to ticket:66. cost: 2h" 1245263731000000 1250785122000000 68 enhancement rgrp rgrp v0.9 closed fixed Show affected packages in revision list view "When listing revisions (at /revision/ or in atom feed) show affected packages. Minor but v. useful UI improvement. Cost: 1h" 1245656153000000 1245697554000000 69 enhancement rgrp rgrp v0.9 closed fixed Change to text-only license field and use external license repo "Switch from license domain object to a simple license field and use license list from new centralised license repo: <http://knowledgeforge.net/okfn/licenses/> * This will '''require''' a migration Cost: 4h (plus migration ...)" 1245687449000000 1246437494000000 70 enhancement rgrp rgrp v0.9 closed fixed Convert from py.test to nosetests Cost: 1h 1246434753000000 1247827053000000 71 enhancement rgrp rgrp v0.10 closed fixed Upgrade to Pylons 0.9.7 Cost: 2h 1246435041000000 1250181211000000 72 enhancement rgrp rgrp v0.10 closed fixed Integrate new logo "Integrate new logo from http://wiki.okfn.org/ckan/logo into site: * In title Also can do favicon (separate ticket:48) Cost: 1h" 1246441021000000 1251451954000000 73 enhancement rgrp rgrp closed invalid Put enquiry template in db so it can be edited by admins "Currently stored in the python controller file. This is not the right place and putting it in db will allow editing by admins. May also want to do this for the footer (also in the controller file). Details: * Will need to create a new domain object/db table. Suggest called miscellaneous and consist of key value pairs utilizing JSON type for values. Cost: 2h." 1247705786000000 1266510385000000 74 enhancement dread rgrp v0.10 closed fixed Add Is It Open links to package pages "On each package page in the openness field if the package is not open provide a link to is it open create enquiry page. Details: * may want to add to the url a query parameter indicating which package this relates to e.g. ?ckan-package=... * http://isitopen.ckan.net/enquiry/create/?ckan-package=... * do we provide link even if package is open (e.g. so that people can make enquiries about e.g. adding open data/content buttons ...) Cost: 0.5h" 1247828218000000 1250182938000000 75 enhancement dread rgrp closed duplicate "Record and display package ""usage"" information" " * Number of package page visits on ckan (can we get this straight from google analytics) * Number of times url or download url is used - now ticket:937 (Record download stats for resoures) How do we do this? * Google analytics will miss a lot of this usage (and how do we get that data out anyway) * Could use javascript but again misses usage. * One option is to redirect link but that is kind of nasty (but may be only option ...) " 1247828785000000 1296341223000000 76 enhancement dread rgrp v0.10 closed fixed Convert to use formalchemy for all forms "Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation. As an extra we could utilize the formalchemy pylons admin interface (pretty much for free). Cost: 12h == Details == 1. Replace htmlfill and formencode extract in controllers/templates with formalchemy 1. Crude and simple 2. Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before. 3. Sort out tag field with a special renderer 2. Validation - testing definitely required. 3. Fix up description and pretty css etc" 1247829041000000 1251301765000000 77 enhancement dread rgrp v0.11 closed fixed Rate packages "= As a User or Visitor = I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable. Rating out of 5. One user or IP, one vote. package_rating table: package | user | rating annakarenina | joe.bloggs.openid.com | 4 annakarenina | 154.2.5.47 | 1" 1247829193000000 1255176732000000 78 enhancement rgrp rgrp ckan-sprint-2011-10-28 closed fixed Extend login cookie lifetime "When you login you get given a cookie but it is very short (life of your browser session I think). Work out how to extend to something reasonable (30 days or perhaps forever with logout unsetting cookie). 1. A quick investigation to see whether this is possible 2. If possible do it Cost: ?" 1247829310000000 1314877169000000 79 enhancement dread rgrp v0.10 closed fixed Improvements to package search "Several things: * Default package search (used e.g. on front page) should deal well with multiple items: ""abc xyz"" means search for both abc and xyz (as separate searches) * Should automatically display a list of matching tags in a line at top with number of associated packages * have an option to only search for fully open material (or openly licensed material ...) Since we should be moving to a more sophisticated search solution anyway (to allow prioritisation etc) maybe we should cautious as to what we do now." 1247829864000000 1265890806000000 80 enhancement dread rgrp v0.10 closed fixed Refactor or remove modes code "Get rid of modes in the RESTful API. Do json stuff directly in controllers. * package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff." 1247844263000000 1265890912000000 81 enhancement rgrp rgrp vdm-0.7 closed fixed Support for sqlalchemy 0.5 1248285433000000 1268487327000000 82 enhancement rgrp rgrp closed fixed Support diffing of versioned objects "Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs. Details: * Do we deal with m2m relationships (and m2one on the many side)? * What is diff? * For text fields output of python diff command * For non-text field guess just simple +/- for what was there before and now Cost: 4h (don't think this is a huge request)" 1248289499000000 1256565441000000 83 enhancement rgrp rgrp vdm-0.6 closed fixed "Allow ""ignored"" fields on versioned objects" "Allow 'ignored' fields on versioned objects (i.e. attributes which are not 'versioned'). This is not hard to do as we already have most of the necessary mechanisms set up in the Revisioner object. Cost: 2h " 1248339384000000 1267648301000000 85 enhancement rgrp rgrp closed fixed Convert state from an object to an enumeration "No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible. Main hassle here is that it requires migration in all projects that use vdm. If we're going to do this we should do it sooner rather than later. Cost: 2h (+ 1h for a conversion method)" 1248339662000000 1263206391000000 86 enhancement rgrp rgrp closed fixed Support for stateful dict-like collections "Already support stateful list-like collections and should extend this to dict-like collections. (This is prerequisite for implementing versioned ""extra"" (key/value) attributes on packages in CKAN and elsewhere)." 1248430798000000 1249050202000000 87 enhancement rgrp rgrp closed duplicate Multiple download links Multiple download links, including links to mirrors and multiple formats/versions 1248693302000000 1258470719000000 88 enhancement rgrp rgrp v0.11 closed fixed Download (link) metadata "Support for download metadata (file size, file formats, version information). See also: ticket:87 (multiple download links)" 1248693385000000 1265891491000000 89 enhancement nickstenning rgrp closed fixed Tag cloud for package tags Should not be hard to do (lots of existing libraries) but not sure that this is very important. 1248693939000000 1265892698000000 90 enhancement nickstenning rgrp v0.11 closed fixed Link to RDF versions of CKAN data "We should link to representations on semantic.ckan.net * Each package page could link to its RDF representation * ""Alternative formats"" near star rating. Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f). * Some instances of ckan won't need this, so have option in ini file for this. rdf_store_url=xyz * Include machine-readable link to the RDF: {{{ <meta http-equiv=... /> }}} * 303 redirect to RDF when content negotiation requires RDF. Test: {{{ curl -I -H ""Accept: application/rdf+xml"" http://localhost:5000/package/warandpeace }}} Cost: 3h == Future == * Search page to link to SPARQL interface: http://semantic.ckan.net/sparql * License/dump page could link to full RDF dump. " 1248943742000000 1265890334000000 91 enhancement dread rgrp v0.10 closed fixed Add author and maintainer attributes to package "Add the following attributes to package: * author, author_email * maintainer, maintainer_email Gives us full compatibility to: http://docs.python.org/distutils/setupscript.html#additional-meta-data Column ordering - should come after name, title, url, download_url." 1249049780000000 1250864156000000 92 enhancement rgrp rgrp closed wontfix Add RDFa to package pages 1249049927000000 1297344859000000 93 enhancement rgrp rgrp v0.10 closed fixed Access control for packages "Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl. Cost: 8d " 1249055049000000 1253613274000000 94 enhancement dread rgrp v0.10 closed fixed Use sqlalchemy-migrate to handle db/model upgrades "Details of how to go about this here <http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/> Cost: 2h" 1249134185000000 1250604808000000 95 enhancement rgrp rgrp closed fixed Add manifest support "Support for listing files (manifest) contained within a package. Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs." 1249981842000000 1251454716000000 96 defect rgrp rgrp closed invalid info command must give up to date information for python-type distributions "(2008-09-10) at present when setup.py is edited info command may not give up to date information if pkg-info (in egg.info) is not rebuilt. Suggest: info command needs to rebuilds pkg-info in python-type distributions. " 1249982410000000 1311176063000000 97 enhancement rgrp rgrp closed duplicate Do not create a distribution on a path is something already exists there "(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option). Cost: 1h" 1249983557000000 1318181317000000 98 enhancement rgrp rgrp datapkg-0.7 closed fixed Replace use of pastescript templates in PythonDistribution with of something simpler "Replace use of pastescript templates in PythonDistribution (write) with of something simpler (allowing us to remove dependency on pastescript). " 1249983810000000 1297210774000000 99 enhancement rgrp rgrp closed wontfix Test DbIndex with sqla 0.5 "Test DbIndex with sqla 0.5 (and get it working if it does not work). Cost: ?? Priority: low because DbIndex is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel). " 1249985537000000 1297081088000000 100 enhancement rgrp rgrp closed fixed Convert existing data on disk to a datapkg distribution "(2008-10-29) convert existing directory on disk to a datapkg distribution (PythonDistribution by default). This would be presented in the CLI as a convert command. Details: * Add a metadata file (setup.py), perhaps prompting for input * List all existing data in that directory Cost: 4h " 1249986628000000 1318181227000000 101 enhancement rgrp rgrp closed fixed (Improved) Download support " 1. Given a url download from that url to disk 2. Given a package use download url to download to disk Once on disk should uncompress (if necessary). == Details == * Should download either to tmp directory (defaulted in config) or a specified path on disk * In download would like to support as many types of target urls as possible: 1. Single files (usually compressed filesets but could be individual files) 2. Index pages listing files to download 3. Revision control repositories 4. ... This will obviously be a significant amount of work. Suggest: 1. Start off just supporting the first option. 2. Have something like a plugin system to handle new target types 3. Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc) Cost: 1d " 1249988866000000 1267648607000000 102 enhancement rgrp rgrp closed fixed Switch to using UUIDs for revision ids "This is better for long term sustainability, especially if we want to ""push and pull"" from unrelated repositories in the future. Details: * May wish to retain a number field updated in the usual incremental way (what's the cost/benefit here?)" 1250068175000000 1260285104000000 103 enhancement kindly rgrp ckan-v1.5 closed fixed View a package at a given revision "As a user I want to view a package at a given revision: * When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy * package history page should provide links to these pages Cost: 2h" 1251289897000000 1311180850000000 104 enhancement rgrp rgrp closed fixed View a package at a given version "As a user i want to see a package at ""version"" X (NB: not revision X). * When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version) * Implementation: * Find revision for this version (search revision history for when version field was last 0.7) * Show pacakge at that revision (as in ticket:103) * On history page also shows versions in list of revisions associated with the package " 1251292672000000 1311181303000000 105 enhancement dread rgrp v0.10 closed fixed Package groups (view) "As a user I want to be able to create groups of packages. This functionality is different from tags: * The ability to add a package to a given group is restricted whereas anyone can add a given tag to a package * Specifically groups have owners and only the owners can add a package to that group Group properties * id (primarykey, uuid style) * name (restricted content - same rules as package) * title (no restrictions) * owners -- many:many with user object * description (markdown) * packages -- many:many with package object Don't version groups for now. Groups address in the WUI will be: * /group - browse list of groups (reuse list action) * /group/list - same thing for now * /group/<groupname> - display: group properties with links to packages. No links to user pages (yet). Group editing and searching will be another ticket. Cost: 3d " 1251392282000000 1252488496000000 106 enhancement dread rgrp closed duplicate Regularly convert CKAN data to RDF and put on Talis CC "Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons). Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC. In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required. Attached is the ruby script used by Talis for conversion Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)" 1251454474000000 1256140649000000 107 enhancement rgrp rgrp closed invalid Provide information about uploading material "Often people who are registering resources want to make the associated material available. We should: * Provide a dedicated upload page giving instructions as to upload process 1. Explain we don't store the data itself on CKAN. Suggest uploading to a store somewhere and then linking using download_url (that is its purpose) 2. Upload to * Talis CC for RDF * archive.org * grid.okfn.org * link to this page from next to download_url item on new package page In long run (now ticket:186) we may automate this by providing a ckan upload facility which caches the data and then reuploads it to relevant service (disadvantage is ""we"" are the owner of the data on that service ...)" 1251466274000000 1296341644000000 108 enhancement dread rgrp v0.10 closed fixed Package search in the REST API "Add package search facility in the rest api at /api/search Queries can be provided as for the normal package search either by posting to that url or by performing a get with a query string. E.g. .../api/search/package?q=xyz Query parameters: * q is keyword string (searches name, title, tags by default) * split by words and ANDed * additional parameters for specific fields * qjson is alternative to q for Searching by specific fields (in addition to keyword string). Payload to json is a JSON-encoded dict which is a dictionary with a q field for free text (keywords) and additional key/value pairs for specific fields * if q and qjson specified ignore q and just process qjson value Additional parameters in addition to query (""q"" or ""qjson"") are: * limit * offset * fullinfo=0/1 - return full record for each result (default=0) * order_by=field_name * search_notes=0/1 (default 0) do we search notes field in a keyword search Return value is json encoded dictionary with keys: * 'results': list of results * 'count': total number of results === Extras (for the future) === * Prioritisation is not part of this ticket (requires fulltext support in DB or in external app such as Xapian) " 1251915845000000 1252340511000000 109 enhancement rgrp dread closed fixed Tag search in the REST API "Similar to ticket:108 but for tags: /api/search/tag?q For tags search is extremely simple since you can only search by name. == Possible Extras == * Allow ""exotic"" ordering of results e.g. ordering by number of packages with that tag. * Return this number with tag list." 1251976297000000 1291829457000000 110 enhancement dread dread v0.10 closed fixed Integrate groups into packages " 1. Package needs a groups property (backref Group.packages). 2. Put groups in the search api. 3. Display them on the package, but not editable. (Only editable from the group page.) Follow up to ticket:105 (Groups). Cost: 1d" 1252315956000000 1252488660000000 111 enhancement rgrp dread v0.10 closed fixed Create user object "Object properties: * id (uuid) * apikey (uuid) -- migrated from the apikey table * name (username = openid) /account is being renamed to /user in the wui." 1252315994000000 1252331626000000 112 enhancement rgrp dread v0.11 closed wontfix Test db schema created via migration scripts "'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass. Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work)." 1252316495000000 1265294069000000 113 enhancement dread dread v0.11 closed fixed Simple script to test data migration on a production db "Must not alter the db. Related to ticket:112." 1252316523000000 1265294054000000 114 enhancement dread dread v0.10 closed fixed Access Control - model "Create in the model basic operation of Access Control. roles table {{{ name | context | action ----------------------- admin| package | edit admin| package | edit-permissions admin| package | read editor| package | update editor| package | read reader| package | read This data is set-up on db init and will have no interface. }}} user-roles table: {{{ username | context_type | objectid | role rgrp | system | n/a | admin visitor | package | * | reader bob | package | geonames | admin visitor | package | geonames | editor visitor | package | geonames | reader john | group | ukgov | admin dread | group | ukgov | editor visitor | group | ukgov | reader This data will be added when someone is given permissions for the system, a package or a group. }}} Pseudo code: class Package def is_allowed(name, action): is_allowed(name, action, context=self) class Group def is_allowed(name, action): is_allowed(name, action, context=self) def is_allowed(name, action, context=None): ''' name: string - a username or IP for 'visitor' action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system) ''' # look up user from name. # look up in user-roles table what roles this user has for this context. # for each roles, look up in roles table what actions are allowed. # return True if action is allowed, else False. " 1252494527000000 1253034529000000 115 enhancement dread dread v0.10 closed fixed Access Control - wui constrained by model "Based on a section of AccessControl design: Reading and writing to packages in the WUI and REST API are now dependent on the authz tables." 1253034394000000 1253091426000000 116 enhancement dread dread v0.10 closed fixed Access Control - edited in wui "Based on a section of AccessControl design: WUI gives controls to user and administrator to change permissions on a package." 1253034802000000 1253709460000000 117 enhancement dread dread v0.10 closed fixed Access Control - group core functionality "Based on a section of AccessControl design: Group reads and edits are controlled by access control. WUI and REST interfaces covered." 1253271333000000 1254735855000000 118 enhancement dread dread v0.10 closed fixed Use paginate in webhelpers "Take out import of paginate in setup.py. Use paginate in webhelpers instead. Make changes to take account of any i/f changes." 1253273657000000 1253784902000000 119 enhancement dread dread v0.10 closed fixed Ensure non-active packages don't show up "Ensure pending packages don't show up in search or browse cost: 4h" 1253529414000000 1253791147000000 120 enhancement dread dread v0.10 closed fixed Security audit "Look for all places where model is accessed and check authorization is checked. Document holes (and, as necessary, suggestions for fixes) as new tickets. Likely areas that need looking at: * search i/f * package WUI commit Write holes are obviously much more significant to us than read holes." 1253529427000000 1254406544000000 121 enhancement dread dread v0.10 closed fixed Add 'Group' to main menu And associated page to browse group. 1253694827000000 1253716782000000 122 enhancement dread dread v0.10 closed fixed Add Group authz page 1253694842000000 1253716757000000 123 enhancement dread dread v0.10 closed fixed Ability to edit Group in WUI "Add Group editing page. If no permissions to change group can't edit group. Also cannot view edit page. Editable attributes: name, title, description No preview needed " 1253708041000000 1254321447000000 124 enhancement rgrp rgrp v1.0 closed fixed Display Generic Package Attributes in WUI Split out from ticket:43 1253709702000000 1254735558000000 125 enhancement dread rgrp v1.0 closed duplicate Edit Generic Package Attributes in WUI Split out from ticket:43 1253709712000000 1258377621000000 126 enhancement dread dread v0.10 closed fixed Change package state in the WUI (delete and undelete) "As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it. (NB: this is quite separate from ""purging"" objects which is the term we shall use for irrevocable removal of an object from the domain model). * Only Package Admins (and sysadmins) should be able to change state == Implementation Suggestions == * 'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...) * Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active'). * This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset " 1253789571000000 1254740244000000 127 enhancement dread dread v0.10 closed fixed Minor form and UI improvements " * group form: description field for needs to be a text area * group form: make title and name field wider * package view: need to move tags and license above notes " 1253798659000000 1253868048000000 128 enhancement dread rgrp v0.11 closed fixed Add ckan_url attribute to REST JSON representation of a Package "Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN. Cost: 30m" 1253866713000000 1275694573000000 129 enhancement rgrp dread ckan-backlog closed invalid Secure db access by channelling query generation through authz module "Controllers and templates should not access db objects directly - they should do all access via authz module giving username. They are handed by a query that has already been filtered by the packages they are authorized to read. (Additional idea to be discussed: When they request a package object, they are handed an copy of the db object - disconnected from the database - so it the db object can't be changed.) A couple of tests can be reenabled when this is done: ckan.tests.functional.test_authz.TestUsage.test_admin_list_deleted ckan.tests.functional.test_authz.TestUsage.test_search_deleted" 1253886136000000 1267719162000000 130 enhancement dread rgrp v0.10 closed fixed Create a New Group via the WUI "== As a == User == I want to == Create a new group via the WUI == Details == * locate at /group/new/ * You must be logged in to create a group * Group creator is automatically given role of group admin * Edit screen is same as edit screen ..." 1254130212000000 1254735501000000 131 enhancement dread dread v0.10 closed fixed Groups REST interface Controlling Groups through a REST interface. 1254307959000000 1254308115000000 132 defect rgrp dread closed fixed Security hole - read package/group list (REST) "Using REST interface you can list packages and groups without authorization being checked. Can be fixed using more advanced query to check authz." 1254389493000000 1273254514000000 133 defect rgrp dread closed fixed Security hole - search package/group (WUI & REST) "Using WUI or REST interface you can search packages and groups without authorization being checked. On the REST interface you can also read all the attributes of the packages using the 'all-fields' option. Can be fixed using more advanced query to check authz. " 1254390168000000 1273253977000000 134 defect rgrp dread v0.11 closed fixed admin interface is only available to sysadmins 1254735314000000 1255430998000000 135 enhancement rgrp dread v0.11 closed fixed sysadmins defined in the db 1254735368000000 1265293940000000 136 enhancement rgrp dread v0.11 closed fixed User has publicly viewable page "A user's 'home page' is at: user/<user.id> e.g. user/28394723982-03849472 Step 1: * Current user home page with recent edits Step 2: Readonly * Number of edits * Number of Packages they are admin of How do we do a nicer URL for the home page - can we use their openid login? Follow on tickets: ticket:138 and ticket:142" 1254741650000000 1255169466000000 137 enhancement rgrp dread closed duplicate User has editable home page " * Generic text box for markdown about the user 'About' Model's user table reflects these: * 'about' attribute " 1254741703000000 1254741830000000 138 enhancement rgrp dread v1.0 closed fixed User has editable home page " * Generic text box for markdown about the user 'About' Model's user table reflects these: * 'about' attribute * 'created' attribute is a default=datetime.now() - so when user was created Need edit page for this. Only user should be able to edit it. Follows on from ticket:136 and ticket:142 Cost: 2h" 1254741789000000 1258466074000000 139 defect rgrp rgrp v0.11 closed worksforme Ensure you cannot add same tag twice to a package "Seems some package have ended up with same tag twice: http://www.ckan.net/package/fred2dot0 Check whether this is still possible and if it is fix it ... Cost: 2h" 1254768552000000 1255188974000000 141 task dread rgrp v0.11 closed fixed Search api docs "Write up search api documentation and put it in a template that shows up at api/search/. Cost: 2h" 1254903008000000 1255007583000000 142 defect pudo dread closed fixed Update User object with additional fields such as 'nickname' "Suggested fields: * id, name (username/profile name), openid, email, password, extras (JsonType) * + current items: apikey, about, created) * current contents of name is usually openid and should migrate to openid field (could match based on starting with http://) * May also want a fullname field (or perhaps that can just go in extras - may be better to have its own field as useful to search on this ) Related changes (probably separate tickets): * Improve user home page to include this additional information (and have it be editable) * Use user 'name' (username) for home page links (rather than current user/<id>) * Where we show lists of revisions, each revision author is now a link to the author's user page. Related to ticket:136 and ticket:138." 1255010314000000 1289219098000000 144 enhancement rgrp dread v0.11 closed duplicate Most popular packages listed on homepage "Based on number of views. Related to ticket:143." 1255010391000000 1265284457000000 145 enhancement rgrp dread v0.11 closed fixed Group shows its administrators When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators) 1255010403000000 1265294041000000 146 defect pudo rgrp closed worksforme Problem when an admin user logs in between commencing editing and saving " 1. Visitor starts editing a package (or creates a new package) 2. Visitor logs in before previewing or committing 3. Visitor is an admin (or sysadmin) 4. Preview or commit fails Reason (conjecture): the form we use for package editing is now dependent on whether you have admin privileges (we allow editing of state by admins). When you login the old form is now incorrect and this results in the formalchemy fieldset breaking." 1255013773000000 1291829862000000 147 enhancement dread dread v0.11 closed duplicate Parser and loader for esw.org data 1255440695000000 1255515162000000 148 enhancement dread dread v0.11 closed fixed Parser and loader for esw.org data " * parse like data4nr code Follow up tickets: ticket:149 and ticket:150" 1255440696000000 1255515222000000 149 enhancement dread dread v0.11 closed fixed Loader for esw data recognises existing packages "Follows on from ticket:148. Sister to ticket:150." 1255440751000000 1257414916000000 150 enhancement dread dread v0.11 closed fixed Loader for esw data uses REST API "(instead of sqlalchemy interface, to facilitate general usage of the loader) * use ckanclient Follows on from ticket:148 and ticket:149" 1255440828000000 1256751974000000 151 enhancement dread rgrp v0.11 closed duplicate User object should have a created attribute "User object should have a ""created"" attribute initialized to current datetime. Require a db migration but o/w very simple. Cost: 1.5h" 1255589694000000 1257414545000000 152 enhancement dread dread v0.11 closed fixed Package has editable 'Extra' fields "Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows: {{{ Key Value delete [ country ] [ uk, spain ] [x] }}} When presenting the form, existing pairs and 3 blank pairs are provided. When saving the data, keys that are not changed use the existing !PackageExtra object (perhaps with new value). Deleted keys put the !PackageExtra state to deleted. For new keys, only create a new !PackageExtra if there isn't a state=deleted one to resurrect. Validators ensure no repeated keys." 1255615316000000 1256056193000000 153 enhancement dread dread v0.11 closed duplicate Group's packages listed alphabetically "This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there. Suggested by Jonathan Gray" 1255621515000000 1258971895000000 154 enhancement rgrp dread v0.11 closed wontfix Package form guidance "Give guidance on what to put in 'author' and 'maintainer' fields (e.g. name? URL?) Suggestion from Jonathan Gray" 1255621572000000 1257535066000000 155 enhancement dread dread v1.0 closed duplicate Adding multiple packages to a group "Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously) Use a bit of javascript to add more dropdowns. Suggested by Jonathan Gray " 1255621779000000 1271760041000000 156 enhancement johnbywater dread v1.0 closed fixed Whitelist for html in mark-down field Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed. 1255623973000000 1271962871000000 157 enhancement dread rgrp v0.11 closed fixed Log message not persisted through preview "Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this. cost: 0.5h" 1255694396000000 1256062680000000 158 enhancement dread dread v0.11 closed fixed Add 'extras' to REST API " * Ensure you can add/change/delete extra fields from packages using the REST API. * Update API docs to cover this. On Package get, ensure separation of extra attributes from other attributes. What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone * Delete field if field value is None Option 2 - user must supply all fields * Delete field if field absent Prefer option 1. Document this. Follows on from ticket:152" 1255703259000000 1255949818000000 159 defect dread rgrp v0.11 closed fixed Searching for tags:... resulting in lots of tags being found "Search of form: tags: ... behaves differently depending on whether there is a leading space: * tags: postcode - tags found correctly but no packages found * tags:postcode - tags incorrectly found but correct packages found Let's fix this. Cost: 0.5h" 1256030097000000 1256060264000000 160 enhancement dread rgrp v0.11 closed fixed Implement internal links in CKAN notes markup "Suggest form is: key:value (with optional space after ':'. Keys would be for an object, i.e. 'package', 'tag', 'group'. Becomes a link. This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace. Cost: 4h" 1256030751000000 1265312986000000 161 defect dread rgrp v0.11 closed fixed Tag listing is incorrect "E.g. http://ckan.net/tag/read/postcodes this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ... I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ..." 1256030925000000 1258573607000000 162 enhancement dread rgrp v0.11 closed fixed Tags should be automatially lowere-cased if not already "Rather than raising a validation error automatically lower-case tags on submission. Cost: 0.5h" 1256048269000000 1256142462000000 163 enhancement rgrp dread v0.11 closed fixed Regularly put CKAN RDF on Talis CC "Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN Regularly re-upload the CKAN RDF data to Talis CC. In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required. Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is) " 1256052587000000 1265891374000000 164 enhancement wwaites dread v1.0 closed fixed Creating RDF data from CKAN "Related to: ticket:90 Link to RDF version of CKAN data and ticket:163 (regularly convert CKAN data to RDF). Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net." 1256052643000000 1271251422000000 165 enhancement rgrp dread closed invalid Use 'tag:' instead of 'tags:' for wui package search "Because people might try and do ""tags:gov sweden"" when ""tag:gov sweden"" makes more sense (sweden is searched for in all fields in these cases). But in REST i/f search allow ""tag""=""gov"" OR ""tags""=""gov sweden"" in parameters." 1256200526000000 1311181391000000 166 enhancement dread dread v0.11 closed fixed Add extras editing to ckanclient Ensure logging is working on server. 1256203581000000 1265891199000000 167 defect dread dread v0.11 closed fixed REST interface you can't update multiple tags Only first tag is updated. 1256209993000000 1265891178000000 168 enhancement rgrp dread closed duplicate Show admins for a group in group view 1256291481000000 1257414795000000 169 enhancement dread dread closed duplicate Package derivations "A 'Derived' relationship can be applied from one package to another. e.g. sussex-demography is derived from census-2001 'Derived' relationship is: * directional * many:many * stateful 'derived' table columns: * id (primary key) * source_package (foreign key) * result_package (foreign key) * description (markdown text) Further tickets: * WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y * WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate * REST if - expose reading and writing this property" 1256304927000000 1266928708000000 170 enhancement rgrp dread v0.11 closed fixed Paginate search results "e.g. http://www.ckan.net/package/search?q=science&search=Search+Packages+%C2%BB Results are capped at 20, but can't see past first page." 1256313584000000 1265891157000000 171 enhancement rgrp rgrp datapkg-0.7 closed fixed Create consolidated config for CLI Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!) 1256488011000000 1297210925000000 172 enhancement rgrp rgrp v0.11 closed fixed Build ckan documentation using sphinx and upload "Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible. NB: improving the documentation is another matter (as is integrating e.g. existing api docs). Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net): * http://knowledgeforge.net/ckan/ckan/doc/ * (Other options: http://www.ckan.net/doc/ http://doc.ckan.net) " 1256489019000000 1257532331000000 173 enhancement rgrp dread v0.11 closed fixed Diffing WUI "Use vdm diff functionality (see ticket:82) to implement diffing of packages between versions in WUI. In package history you should be able to select to revisions and show the diff. Estimated cost: 12h" 1256565412000000 1265293968000000 174 defect rgrp dread v0.11 closed fixed Packages search broken for 'only openly licensed' "the ""only openly licensed packages"" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The ""only downloadable packages"" tick box by itself seems to work fine. E.g. try searching for 'science'. Raised by Jonathan Gray" 1256650867000000 1265891093000000 175 enhancement rgrp dread v0.11 closed fixed Creation date "'created' attribute on User, Group and Rating datetime object: default=datetime.datetime.now() See example on revision Requires migrate script" 1256754070000000 1265291037000000 176 enhancement dread dread closed duplicate Package dependencies "(Related to ticket:169 - Package derivations) A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.) e.g. london-traffic-visualisation depends on road-map 'Dependency' relationship is: * directional * many:many * stateful 'dependency' table columns: * id (primary key) * dependent (foreign key) * dependency (foreign key) Further tickets: * WUI - package view - have list of dependencies (do not need to list packages which depend on this one) * WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package') * REST api - expose reading and writing 'depends on' property. === Issues === * How do we deal with dependency at a particular version?" 1257162812000000 1266928721000000 177 enhancement rgrp dread closed invalid Service documentation "This is docs for users of a CKAN service (e.g. ckan.net), as opposed to someone installing / administering the software. (The latter has docs on knowledgeforge. Editability - wiki?" 1257243285000000 1273050236000000 178 enhancement rgrp dread v0.11 closed fixed Excel package importer "/tool New 'tools' section of the WUI with 'Excel package import' page. /tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&paste. /tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process. File format: First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes: * license - name of license * tags - space separated * groups - space separated * anything - for extra key-values When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete. User has to log in to use importer. Auth is checked for putting into groups." 1257244034000000 1265305558000000 179 defect dread rgrp v1.0 closed fixed Restore 404 and 500 messages in WUI "Cost: 0.5h (?) Conjecture this went missing in cset:a35db862a841" 1257412668000000 1265305549000000 180 enhancement rgrp jwyg v0.11 closed duplicate Tag cloud as way to view CKAN tags Create big tag cloud with all CKAN tags - perhaps weighting with size and colour... 1257534254000000 1265284374000000 181 enhancement rgrp jwyg closed invalid New 'stats' box on right hand sidebar "With up to date statistics such as: * total packages * total tags * total users * most active users * last update * etc. Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184" 1257534385000000 1296339510000000 182 enhancement rgrp v1.0 closed fixed Improve openness icons "Tick and cross aren't clear enough. Propose: * Icon for downloadability (nothing if absent ...(?)) * Icon for open license? (Need something if closed ...) Cost: 2h (?)" 1257534590000000 1270567116000000 183 enhancement rgrp rgrp closed worksforme Browse packages by rating At moment order packages by title. 1257534606000000 1290604779000000 184 enhancement dread rgrp closed fixed Stats page "Create /stats/ page displaying main statistics, e.g: * Most highly rated packages * Most edited packages * Largest groups * Top tags (by packages) * Package addition rate * Users with most packages Related to ticket:181 - Stats side-bar == Details == * Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display) * For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see * http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py * http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html" 1257534756000000 1266837414000000 185 enhancement rgrp dread closed invalid Package form field hiding "Make parts of the package form hidden by default, to make it look simpler. Users are invited to click a button to reveal the less important fields. Fields hidden by default: version, author, author_email, maintainer, maintainer_email" 1257763437000000 1291830039000000 186 enhancement rgrp rgrp closed duplicate Automated upload to archive.org s3 "(Follows on from ticket:107). We want to provide facility for users to automatically upload material. * Create an upload page * Store the material on archive.org using s3 - http://www.archive.org/help/abouts3.txt * Main issue is how we cache large datasets ..." 1257803430000000 1296341182000000 187 enhancement rgrp rgrp v0.11 closed fixed Full-text search "Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results. If we do this we need: 1. To weight across fields in a sensible way 2. We can also use proper text search on title or ... Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: http://www.postgresql.org/docs/8.3/static/textsearch.html Using this with sqlalchemy: http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/ Issues with fulltext search: * tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking. * if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better. * can we split the name on dash or underscore before being indexed? * natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'. * do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres? * we want to weight name and title higher than other fields - achievable with custom trigger." 1257869950000000 1265892403000000 188 enhancement rgrp rgrp v0.11 closed duplicate Improve package listing views "Propose change to tabular-like format showing these attributes (perhaps should be configurable?) * Openness status * Title (not sure name is needed) * Tags Cost: 4h" 1257870031000000 1265294090000000 189 enhancement dread dread v0.11 closed fixed Multiple download urls "See also: PackageResources which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data. == As a == User (package editor and package user) == I want to == (package editor) Associate multiple distributions with a package. (package user) Easily have access to multiple distributions of a package should they exist. == Implementation == There is an object called 'Resource' with the following attributes: * url (text) (required in UI, but not in db table) * format (text) * description (text) * package (association) - a resource is always associated with 1 package A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution'). === How to replace download url === download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility. REST API has new attribute in package dictionary 'resouces' - an ordered list. In WUI, replace download url in package read with a table of the distributions. In the WUI package edit, be able to add/remove/edit/reorder distributions. == Example == Example part of package: === Distributions: === || !http://some.com/pollution.pdf || PDF || Original report || || !http://some.com/pollution.csv || csv || Original data || || !http://some.com/pollution || sparql || Linked version of data || Existing data to be migrated to new structure. Cost: 4 days " 1258386120000000 1265284961000000 190 defect pudo dread closed fixed Package comments "Cost 7 days When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API. The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user. Example at bottom of package page: '''Leave a comment:''' ||Subject|| _________ || ||Comment|| _________ || [[br]] ||Submit button|| '''Comments:''' ||'''Explanation doc'''|| ||''Posted on 25h May 2009 by !http://bertdavies.myopenid.com''|| ||It says on the pollution data web page that not all the stations have a CO2 sensor, so you have to extrapolate from the ones that do. See my visualisation of CO2 across London for an idea of what you can do: bertdavies.com/pollution-2008.jpg|| ||'''More info'''|| ||''Posted on 24th May 2009 by !http://ronsmith.myopenid.com''|| ||Excellent data, but why is there no value in the CO2 column for some of the testing stations?|| = Implementation details: = Comments table is with columns: ||id || package_id || date (date) || comment (multi-line text)|| " 1258387105000000 1280820852000000 191 enhancement johnglover dread ckan-sprint-2011-12-19 closed fixed Searching by modification date "Cost - 2 days Search interface has new options to filter and sort the results by the date the package has been last modified in ckan. Search options are included in both Web UI and Search API. The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. Example search parameters: || modification-range=5/4/09- || Exclude packages last modified earlier than 5/4/09 || || modification-range=5/4/09-5/12/09 || Exclude packages last modified outside of 5/4/09-5/12/09 || || order_by=mod ||Sort by metadata modification. Defaults to newest first. || || order_by=mod-newest || Sort by metadata modification, newest first. || || order_by=mod-oldest || Sort by metadata modification, oldest first. || " 1258387778000000 1330020983000000 192 enhancement rgrp dread closed fixed Time-related package field "Cost - 2 days The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09). Example part of package: '''Date:''' 5/10/09 [[br]] or [[br]] '''Date:''' 1/1/09 – 1/4/09" 1258388163000000 1291733895000000 193 enhancement rgrp dread closed wontfix Searching by time-related field "Cost - 2 days Search interface has new options to filter and sort the results by the time-related field of the package. Search options are included in both Web UI and Search API. The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. Need to decide for a time-related field value that is date range, what date is used for the search. Example search parameters: || reldate-range=5/4/09- || Exclude packages related to earlier than 5/4/09 || || reldate-range=5/4/09-5/12/09 || Exclude packages related to date outside of 5/4/09-5/12/09 || || order_by=reldate || Sort by date package is related to. Defaults to newest first. || || order_by=reldate-newest || Sort by date package is related to, newest first. || || order_by=reldate-oldest || Sort by date package is related to, oldest first. || Related to ticket:192" 1258388169000000 1340626463000000 194 defect rgrp dread v0.11 closed fixed Check star ratings aren't influenced by search engine crawlers rel=nofollow or robots.txt ? 1258471512000000 1265284389000000 195 enhancement rgrp rgrp v0.11 closed fixed Association listings should be alphabetically sorted (e.g. a group's packages) "Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name. Cost: 2h" 1258475448000000 1258585500000000 196 defect dread dread v1.0 closed fixed RDF URI to resolve on CKAN "Provide RDF version of a package at the RDF URI: http://ckan.net/package/rdf/32000-naples-florida-businesses-kml See email on OKFN help: For the information to be Linked Data, the following URI (from the api.talis.com SPARQL call) should be resolvable to RDF, or an RDFa enhanced HTML page." 1258727988000000 1265389771000000 197 enhancement dread dread v0.11 closed fixed Simple dumper "== As a == admin / user == I want to == Admin: export all HEAD data to JSON / CSV / RDF format dump. User: wants to get hold of this dump. == Details == * Just do JSON & CSV for now. * Put this in CLI. * Just do list of packages - as_dict info. CSV needs a bit of flattening. Cost: 2h" 1258972495000000 1258989700000000 198 enhancement rgrp dread closed fixed Change package and tag ids to uuids "See how we did it already for other things. Note: on ckan.net older PackageRevision.id might not be identical to Package.id but this may need sorting at this point." 1258980613000000 1266837606000000 199 defect rgrp casbon closed fixed Fix failing tests and provide documentation on how to run tests in datapkg "As of 80:099bdc5b07a6 datapkg has 4 test fails and 1 test error. There is no documentation on how to run them, so I am using 'python setup.py test'. {{{ datapkg.tests.test_cli.TestCLI.test_ckan ... FAIL datapkg.tests.test_cli.TestCLI.test_walkthrough ... FAIL datapkg.tests.test_index.TestDbIndex.test_get ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/index.py:97: SADeprecationWarning: Use session.add() self.session.update(pkg) FAIL datapkg.tests.test_index.TestDbIndex.test_get_when_loaded_as_new_and_init_not_called ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/tests/test_index.py:57: SADeprecationWarning: Use session.expunge_all() self.index.session.clear() FAIL }}} " 1259087361000000 1267649255000000 200 defect rgrp rgrp closed invalid PythonDistribution write method should write all available metadata Current write method just creates a bare bones package and does not write any of the available metadata other than name. 1259229583000000 1311176118000000 201 enhancement rgrp dread v1.0 closed fixed Stress test ckan "Check: * high numbers of simultaneous users * incomplete transactions * etc." 1259844836000000 1265284419000000 202 enhancement tonfa dread v0.11 closed fixed Internationalization (i18n) try-out "Try i18n for one language (German) for ckan (de.ckan.net). Basic approach to i18n is well-covered in Pylons book: http://pylonsbook.com/en/1.1/internationalization-and-localization.html However, we need to integrate with our Genshi templates. Details on how to do this in comment: http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization == Steps == 1. i18n templates * Extract content using Babel * Get genshi translation working * Do the translation! 2. i18n all strings in controllers (if any) 3. i18n forms" 1260179192000000 1265892368000000 203 enhancement dread dread v0.11 closed fixed show package counts for groups in WUI "In two places in the WUI * list of groups * group page in title e.g. ukgov (5) Cost: 1h" 1260187690000000 1265891080000000 204 enhancement dread dread v0.11 closed fixed Custom package input form "== As a == admin == I want to == input government data in a constrained way == so that == you get consistency in the db which is easily searched, discovered and automatically linked. == implementation == Form has fields suitable for government data. (related to custom excel importer ticket) In pylons config specify the form schema to use - whether the existing one or the government one. '''On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)''' === Simple fields === * all basic ckan fields * default license to OKD::Crown Copyright * co_id, precision, taxonomy url, agency === Date fields === Date format: machine readable. Can be specific or vague: year or month/year or day/month/year. * date_released * date_updated * temporal_coverage_from, temporal_coverage_to - two date fields combined === Strict selection === Drop down combo * department === Selection or free text === Editable combo * update_frequency - selections: never, weekly, monthly, annually, discontinued * geographic_granularity - selections: national, regional, local authority, ward, point * temporal_granularity - selections: yes, months, weeks, days, hours, point === Specialised fields === * geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global (Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?) * national_statisic - UI is check box, store as 'Yes' or 'No' * resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets. * categories - multi values from strict selection of ONS categories. Stored as comma-separated. === Field data processed === * tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories. === Future - optional === Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views. When you edit a package it detects the schema based on the extra fields that are present. " 1260188965000000 1285082169000000 205 enhancement rgrp dread closed wontfix Custom package importer "Importer function is exposed to logged in users. Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'. * Basic is as it is currently * Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords. All fields should be in step with the government custom input form. For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text. Estimate: " 1260197819000000 1297068450000000 206 enhancement nickstenning nickstenning v1.0 closed fixed UI review "Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular: * Form usability * best example: http://ckan.net/package/new * ""Extras"" fields * Adding to groups (from group page or from package page?) * Overview pages: reduction in visual noise and unnecessary elements. * best example: package overview page * consider openness/downloadability icons * display common metadata fields in a more structured fashion? Est cost: 2 hrs." 1260199941000000 1260881123000000 207 enhancement dread nickstenning v1.0 closed fixed Reimplement tag selection in jQuery "Remove the current dependency on Yahoo YUI toolkit for doing tag autocompletion on package edit forms. The (KForge-derived) behaviour files for CKAN can easily be switched to the new jQuery-based ones: see ticket:212. Once this is done, it would make sense to implement the tag-selection code with jQuery. Relevant pages include: * http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/ * e.g. http://ckan.net/package/edit/acawiki" 1260200340000000 1271250740000000 208 enhancement dread rgrp v0.11 closed fixed Obfuscate author email and maintainer email field "Obfuscate author email and maintainer email field. Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@jones.com', encode='javascript') Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate? " 1260203938000000 1265891070000000 209 enhancement rgrp rgrp closed fixed Validate email address when creating an enquiry p:wui 2009-09-09 validate email address when creating an enquiry 1260288671000000 1260288728000000 210 enhancement johnbywater johnbywater closed invalid User registration "Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid. " 1260292578000000 1266509788000000 211 enhancement johnbywater johnbywater closed invalid User login "Native login scenario: Complete login form, require email address. Any session is resumed. Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff. See CKAN or Microfacts for usage of repose with openid." 1260292624000000 1266509848000000 212 enhancement rgrp rgrp v0.11 closed fixed "Switch to using m.okfn.org ""kforge"" media (css and behaviour)" "We are already using standard kforge css though copied locally and with a few modifications. We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made. Cost: 2h" 1260703728000000 1260703969000000 213 enhancement dread dread v0.11 closed fixed Automate CKAN deployment fabric script to automate ckan deployment 1260811707000000 1265891061000000 214 enhancement nickstenning dread v0.11 closed fixed UI Review - General template " * Set hover names on navigation links. * Do we really need to display ""About CKAN"" everywhere? Just on home page. * Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions. " 1260879581000000 1263406056000000 215 enhancement rgrp dread v1.0 closed fixed UI Review - Home page " * 'Login via OpenID' -> 'Login with OpenID'? * Elements: * Search * about CKAN * 3 column Packages view (see ticket:219) * Small tag cloud (RHS) Also, what about ticket:143 ?" 1260879732000000 1266513523000000 216 enhancement nickstenning dread v1.0 closed fixed UI Review - Search " * allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them <label>s. * cut 'only's -- use word 'filter' instead * doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package. * autocomplete package names & tags " 1260879802000000 1273596102000000 217 enhancement nickstenning dread v0.11 closed fixed UI Review - Search results "Package search results page [/package/search?q=something] * need clearer delineation between search entry form and results. * 0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result. * ""N tags found"" -- why do I care? this should be presented as ""filter results by tag"" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search. * tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background) * filter out particular tags * tag cloud? * copywriting: ""Packages - Search"" -> ""Search packages"" " 1260879870000000 1263406092000000 218 enhancement nickstenning dread v0.11 closed fixed UI Review - Package listing " * openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS. * maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence. * results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise. * maybe links should go on the full titles? Get rid of package names. * Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS? * same comments as search listing (ticket:217) " 1260879903000000 1263406033000000 219 enhancement nickstenning dread closed wontfix UI Review - Package " * Propose three-column package listing: two lists of highest-scorers on some metric and last column is an invitation to register a new package, with a little copy explaining what's involved and why you'd want to. * Add in search widget " 1260879947000000 1294914901000000 220 enhancement nickstenning dread v0.11 closed fixed UI Review - Package read "Package page [/package/mypackagename] * Design * In general: a huge amount of visual and typographic noise: ""Package:"" is redundant. * ""{edit}"", ""{history}"" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is ""make an enquiry with Is It Open?"" bigger than these? * Openness/downloadability icons: need help, as discussed earlier * ""Rating:"" redundant, as is the second set of stars -- why can't both the display and rate widget * Rating - alternative to GET on set_rating. * ""Metadata:"" technospeak, probably redundant. * <li> bullets add nothing. * Title, Version, URL can probably go in one line. * Resources can be made more prominent. * Author, author email and Maintainer, maintainer email can be merged. * Groups, tags in little boxes? * Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface? * Don't show extras unless there are any. " 1260880136000000 1265286499000000 221 enhancement dread dread v1.0 closed fixed UI Review - Package edit "Package edit page [/package/edit/mypackagename] * Ability to add the package to groups on this page. * Fields could be grouped. * Inconsistent capitalisation (Url -> URL). * Tag editor YUI script appears to break occasionally. * Perhaps move to two column forms to save vertical space? * Notes field to monospace? * Extras fields need some jQuery love -- shouldn't have a fixed number available. * Typography is a bit of a mess. " 1260880175000000 1271756757000000 222 defect nickstenning dread closed fixed UI Review - Package history "Package history page [/package/history/mypackagename] * ""Revisions"" is redundant * ""Compare"" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it! * Ditto to overall ""Recent Changes"" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links. " 1260880198000000 1282909280000000 223 enhancement nickstenning dread v1.0 closed fixed UI Review - Tag "Tags landing page [/tag] * Both search for tags *and* browse the list of tags right here * Big cloud of tags (sorted by popularity/usage?) filterable with a search box. * Tags search page [/tag/search] and list [/tag/list] are both redundant and should be merged straight into [/tag]. " 1260880276000000 1273596111000000 224 enhancement nickstenning dread v0.11 closed fixed UI Review - Groups "Groups landing page [/group] * Yeesh! What's up with the typography for ""Just login...""? * Truncated description along with group name. * Explanation text about what groups are for (RHS) * Another case of <li> bullets adding nothing. " 1260880360000000 1267100560000000 225 enhancement rgrp dread closed invalid UI Review - swap URL order "Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure. " 1260880487000000 1311178276000000 226 enhancement nickstenning dread v1.1 closed fixed UI Review - History "Repository History [/revision] * What is this page called? ""Recent Changes"", ""Repository History"", ""/revision"": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar? * Will change page title on /revision/ to Revision History. Will not change route for the time being. * ~~needs copyediting.~~ * ~~Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.~~ * ~~Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say ""Older""/""More recent"" rather than ""Previous""/""Next"".~~ (wontfix: now have text saying we are showing most recent changes) * Table layout is pretty ugly (yes, I'm aware this is my fault). * wontfix - nothing better at the moment * Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail? * Can we link to an author page? Yes! * ~~Atom feed should have a feed icon!~~ * Why are we adding another <link rel=""alternate""> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page. " 1260880515000000 1279130390000000 227 enhancement nickstenning dread v1.1 closed fixed UI Review - General checks " * There is huge inconsistency in the titles of pages across the site: to give one example ""Edit Package:"" vs ""Package: mypkg [not linked]"" vs ""History - mypkg [linked to package page]"" * Normalize and use terminology: ""Data Package"" - also change in navbar (but nowhere else for the time being). So hvae Data Package - mypkg, Data Package - mypkg - Edit etc * pagination is currently lost in amongst the results list. needs to be *much* more obvious, and should appear top and bottom, or at the very least at the bottom! " 1260880552000000 1279192621000000 228 enhancement rgrp rgrp closed duplicate Deal with duplicate packages This needs to be thought out ... 1262085763000000 1290596875000000 229 enhancement dread dread v0.11 closed fixed Resource hashes "New field for resources - hash of the resource file. * CKAN itself will not calculate the hash value - user just pastes it in. * Display text field in resource table. * Requires migration script. Questions for the field's value: 1. Which hash to use? Restrict to python hashlib and other major hash libraries. 2. Should we use merkle trees? Thanks to Julien D'Assanges for the suggestion. " 1262686287000000 1265891612000000 230 task dread dread v1.0 closed fixed Refactor 'package preview' to use 'package read' " * Requires Package Read to not use the side-bar * Need to sync params to a package object that is taken out of the session so that it doesn't go into the database." 1262788084000000 1265293953000000 231 defect pudo rgrp ckan-v1.3 closed fixed Make group versioned "Make group object revisioned/versioned. * Involves upgrade script both for group object and group_revision object * No changes required to WUI for present" 1262853923000000 1288516929000000 232 enhancement rgrp rgrp closed invalid Add strapline to CKAN logo "Strapline (?): Sharing, discovery and reuse of data and content Need to consult on this. Cost: 1h" 1262975291000000 1297246297000000 233 enhancement rgrp rgrp v1.1 closed fixed Allow simple site-specific customization/overriding of templates "Options: 1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works) 1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist). Can do this using: <xi:include href=""base.html""><xi:fallback /></xi:include>" 1263000396000000 1273080352000000 234 enhancement memespring rgrp closed wontfix UI Review - Autocomplete package names & tags in search Broken out from ticket:216 1263056678000000 1338206486000000 236 enhancement johnbywater johnbywater closed fixed The system shall present to all users a form for starting data openness enquiries 1264157100000000 1264157324000000 237 enhancement johnbywater johnbywater closed fixed The system shall accept submissions from authenticated owners of activated accounts using the data openness enquiry form by presenting the enquiry summary and prompting for confirmation 1264157377000000 1264157420000000 238 enhancement johnbywater johnbywater closed invalid The system shall accept submissions from unauthenticated users of the data openness enquiry form by creating an anonymous pending enquiry action, and by redirecting the user to login form, passing a reference to the pending action 1264157814000000 1266509806000000 239 task dread dread v1.0 closed fixed Test migrate scripts "Integrate new test upgrade into testmigrate stuff. New ""migrate.ini"" config file programmed with the db to be used for migrate tests and a param saying where the db dump is." 1264439926000000 1265286521000000 240 defect johnbywater rgrp v1.0 closed fixed Do not allow creation of PackageResource object without a url Currently it is possible (v0.11) to have a PackageResource object without a url. This should be impossible to do (either at creation time or via modification). 1265399747000000 1272383770000000 241 defect rgrp dread v1.0 closed fixed License doesn't preview correctly 4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package. 1265625546000000 1270569769000000 242 defect pudo rgrp v1.1 closed fixed Miscellaneous tidying up (v0.11) "Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp * controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)? done: * lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage, SearchTag etc)" 1265715103000000 1280823876000000 243 defect rgrp dread closed worksforme Repetition of number of packages in a group "http://ckan.net/group/publicdomain has: ""There is 16 package in this group. There are 16 packages in this group.""" 1265750250000000 1266837796000000 244 defect rgrp dread closed fixed Several links preceded by link URL "e.g. Text on ckan.net home page: ""All material available under an /licenseopen license"" Also seen on the package edit page: ""/user/loginClick here to sign in"" I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime." 1265750419000000 1271248968000000 245 enhancement rgrp rgrp closed duplicate Support for composite primary keys "Problem here is that foreign key then becomes ""complicated"" (composite). * Could also deprecate continuity_id field in favour of the basic foreign key on ie" 1265882630000000 1297066620000000 246 enhancement rgrp rgrp closed duplicate Support for primary key not named id At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys) 1265882862000000 1297066757000000 247 enhancement dread rgrp v1.0 closed fixed Alternative formats links on package page """Alternative formats"". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f). * How do we know if RDF available? Ans: see details in ticket:90. Cost: 1h" 1265890326000000 1271248813000000 248 enhancement dread rgrp v1.0 closed fixed i18n of forms Need to do i18n of forms (not in templates but in code). Should consult with tonfa about this. 1265892313000000 1270745304000000 249 enhancement rgrp dread closed invalid Regex search "Search of package name and title (and other fields) using regular expressions. Current example use-case: Wanting to specify packages with title beginning with 'B'. Issues: * Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields? Implementation: * Postgres reg ex searching detailed here: http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE" 1265994509000000 1311182450000000 251 enhancement johnbywater johnbywater v1.0 closed fixed Present in the API a list of revision IDs, and present individual revisions when referenced by ID "Want to have: * /api/search/revision/ Support querying by ""since time"" and ""since revision"". Each individual revision should be available in the REST API (read-only): * /api/rest/revision/{revision ID} * should include revision attributes plus revision.packages e.g. like { ""id"": ""..."", ... ""packages"": [ list of packages names from e.g. revision.packages ] } " 1266519743000000 1271423917000000 252 enhancement dread johnbywater closed invalid Change revision object so that it has parent(s) attribute 1266519767000000 1296477560000000 254 enhancement dread dread v1.0 closed fixed Package relationships - 1. model "New domain object: PackageRelationship (revisioned) Attributes: * subject (Package reference) * object (Package reference) * type (string) * comment Relationship type values: depends_on (dependency_of) derives_from (has_derivation) child_of (parent_of) Relationship type is stored as a string, as given in the first column. The reverse relationship (bracketed) is given just for display purposes only. Subject / Object - as in a sentence: ""Dracos is derived from OSM"" is <subject> <predicate> <object>. We'll use 'type' instead of 'predicate' as that is a more familiar word. " 1266928449000000 1273596180000000 255 enhancement dread dread v1.0 closed fixed Package relationships - 2. Read in WUI "WUI: * View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...) " 1266928542000000 1273596174000000 257 enhancement dread dread v1.0 closed fixed Package relationships - 4. Read in API "API: * Appear in package listing Example: 'relationships': [{'is_dependency_of':'osm', comments:'Since version 0.2'}, {'is_parent_of':'bobs_maps'}] " 1266928630000000 1273596170000000 258 enhancement rgrp dread v1.0 closed fixed Add uuids into package in REST Add uuids into package in REST interface. 1266954722000000 1273596163000000 259 defect rgrp dread v1.0 closed fixed isitopen enquiry broken "Pressing the 'make an enquiry' link on the package read page gives 404. (Also links should also be changed from isitopen.ckan.net to isitopendata.org) Example link is: http://isitopen.ckan.net/enquiry/create/?ckan_package=zeno which gets forwarded to: http://www.isitopendata.org/enquiry/create/?ckan_package=zeno which gives 404" 1267012666000000 1273596153000000 260 enhancement dread rgrp v1.0 closed fixed Display group package list in standard (richer) way "Currently use old ""bullet-point"" style. Should change to now standard style used on e.g. package search results page. Cost: 0.5h" 1267100660000000 1267122137000000 261 defect dread rgrp v1.0 closed fixed License does not work through REST API "Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted). Using normal license field does not cause breakage but has no effect. (Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install). {{{ ourjson = '''{ ""maintainer"" : ""Applications Division"", ""version"" : ""1999-01-01"", ""license_id"" : 9, ""name"" : ""geogratiswaterfraction1km_v100"", ""author_email"" : ""..."", ""author"" : ""Dissemination Branch"", ""tags"" : [ ""GeoGratis"", ""Canada"", ""GCMD"", ""Hydrosphere"", ""Ground Water"", ""Surface Water"", ""Hydropattern"", ""Earth Science"", ""Land Surface"", ""Land Cover"", ""Landscape"" ], ""extras"" : { ""License"" : ""http://geogratis.ca/geogratis/en/licence.jsp"", ""Ministry"" : ""Natural Resources Canada"", ""Level of Government"" : ""Federal"" }, ""maintainer_email"" : ""richard.fernandes@nrcan.gc.ca"", ""notes"" : ""This product is a raster coverage"", ""url"" : ""http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67"", ""title"" : ""Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada"" } ''' def test_15_random(self): offset = '/api/rest/package' import urllib tmpjson = simplejson.dumps(simplejson.loads(self.ourjson)) postparams = urllib.urlencode({tmpjson: 1}) res = self.app.post(offset, params=postparams, status=[200], extra_environ=self.extra_environ) model.Session.remove() }}}" 1267134778000000 1267207776000000 262 enhancement rgrp rgrp closed fixed Mark package as duplicate of another package "Need a way to mark packages as duplicate. Ideas/Questions: * Could work off a tag (meta-duplicate or somesuch) * What happens if we want to delete the package (to we leave a place-holder, do we merge content?) " 1267139682000000 1296468392000000 263 enhancement johnbywater rgrp v1.1 closed fixed Improve and test openid login "At the moment the user has to figure out to login with providers -- this is not always obvious (e.g. google). We could improve this with a small bit of javascript. As an example see: http://standalone.demo.civicrm.org/standalone/ Not sure exactly what library that is using (or if bespoke) so alternatives include (NB: we already use jquery): * http://code.google.com/p/openid-realselector/ * http://code.google.com/p/openid-selector/ In addition to implementing this we also need to test logging in with main providers: google, wordpress -- as we have had reports of it not working with e.g. wordpress (not sure if this testing can be automated really -- best hope would be selenium I think -- so OK if done by hand). Cost: 3h (2h UI), (1h testing)" 1267605039000000 1279292730000000 264 defect dread rgrp v1.0 closed fixed Investigate why there are revisions with no associated packages "Revision.packages should not include all changes so why are we getting revisions with no associated changes? See e.g. http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb? Investigate and fix. NB: if this is bugfix this should go against stable or metastable branch if at all possible and then be merged into default. Cost: 1.5h (?)" 1267605428000000 1272390013000000 265 enhancement johnbywater dread v1.0 closed fixed More detail shown in Atom feed "== Use case == As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package. == Implementation == Add into the Atom feed: 1. a package is a new one 2. the resources have been changed 3. the last_updated field has been changed " 1267708364000000 1271636891000000 266 enhancement johnbywater dread v1.0 closed fixed Package feed "As a user I want to track changes to a particular package. Provide an Atom feed, linked from the package read page." 1267716417000000 1271636901000000 267 defect johnbywater rgrp vdm-0.7 closed fixed Diffs for object that is created in that revision Currently this raises an exception because there is no package/package_revision in existence at from_revision. 1268041383000000 1272960518000000 268 defect rgrp dread closed duplicate Select groups in Package edit form 1268068896000000 1285070682000000 269 enhancement rgrp dread closed fixed Improve gov package form "If the notes field could use a WYSIWYG editor with word cleanup this would really help users who may well be pasting in text from Word (and I guess may have been approved or written by someone else) having auto-complete on tags would both make the system easier to use - and reduce the risk of synonymous tags being created inadvertently. Does the department drop down options list interact with user permissions - so that users who only have rights to one department only see that option ? Similarly for licenses - can this reduce to the set of allowed options for this user/ What level of validation is there on the fields?" 1268220853000000 1291897538000000 270 enhancement johnbywater johnbywater closed fixed Refactor ckan/lib/importer and ckanclient/loader. "There is code to create packages in-process (ckan/lib/importer) and via HTTP (ckanclient/loaders/base). There is also code to read different kinds of package source (Google Spreadsheets spreadsheet, other spreadsheet). Propose to refactor all this to have template method in base command class (subclasses may implement command line interface, graphical user interface, or Web interface), where the template method calls on aggregated strategy objects for: reading raw spreadsheet data from spreadsheet application; for inferring CKAN package entities from the raw spreadsheet data; and for writing package entities to CKAN service. Spreadsheet readers to be factored to allow for variations of spreadsheet application. Raw data inferrers to allow for variations in spreadsheet cell structure. CKAN writer strategies to allow for either CKAN client or direct linking. Actual cases then supported by specifying existing strategies (or defining new ones) in a script that invokes the command class." 1268671804000000 1290596640000000 271 enhancement johnbywater johnbywater v1.0 closed fixed The system shall support distributing changes. See DistributingChanges for analysis and functional requirements. 1268672137000000 1273596144000000 272 enhancement rgrp rgrp closed fixed Atom/RSS feeds for individual tags and groups "I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group. * This would allow people to embed feed widgets in other places" 1268988652000000 1340631775000000 273 task pudo rgrp v1.1 closed fixed Investigate search index options and create tickets "Work already here: http://knowledgeforge.net/ckan/trac/wiki/SearchEngine Add knowledge there or to: http://wiki.okfn.org/SoftwareTools/Search " 1268996933000000 1278578527000000 274 defect rgrp rgrp closed fixed Allow searching by any extra field "At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...) Cost: 1h" 1269034562000000 1287402800000000 275 defect dread rgrp closed fixed Add forms renderer for JsonType fixing bugs in admin interface "This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type) Cost: 0.5h {{{ from ckan.model.types import JsonType forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer }}}" 1269122740000000 1281002082000000 276 defect dread rgrp v1.0 closed fixed Resources in Package form seen multiple times upon preview Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'. 1269255399000000 1272996237000000 278 defect dread dread v1.0 closed fixed Spaces in extra field keys If the key to an extra field has a space in it then various form code doesn't work. Either need to disallow spaces or cope with them. 1269520845000000 1271173752000000 279 enhancement johnbywater dread v1.0 closed fixed Ensure package's license is an option when it's edited "With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost. So for this case we should add the package's license to the list displayed." 1270205931000000 1272451384000000 280 defect dread dread v1.0 closed fixed Disappearing extra fields "When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear. Metastable branch." 1270209684000000 1271173769000000 281 enhancement dread dread v1.0 closed fixed Refactor forms to be plugin-able 1270542248000000 1270723675000000 282 enhancement thejimmyg dread closed wontfix Provide diverts when package name changes When a package's name is changed, references to the old name (in the WUI and REST) are redirected to the new name (assuming the old name is not being reused by another package). 1270659003000000 1338206417000000 283 enhancement rgrp dread closed wontfix Manage deletions of unwanted packages "== Use case == As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion. == Suggested solution == In the package view side-bar, there is a note: ""To have this package completely removed, contact the [ca.ckan.net administrators admin@ca.ckan.net]."" == Other solutions == A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill. c.f. http://en.wikipedia.org/wiki/Deleting_an_article" 1270660210000000 1311325526000000 284 defect johnbywater rgrp v1.0 closed fixed Move tag/read/xxx to simple tag/xxx like package and group "Also put in redirect from old locations (see examples with plurals in routing.py) Estimated cost: 20m Actual cost: 40m" 1270664552000000 1271852451000000 286 enhancement dread dread v1.0 closed fixed Refactor forms templates "Use Nick's examples as a basis to revamp the formalchemy usage, principally in the package edit form. * http://static.whiteink.com/css/examples/forms.html * http://whiteink.com/temp/ckan-forms/" 1270723513000000 1270723629000000 287 enhancement dread dread v1.0 closed fixed Customise Canadian package edit form "Spec - see attached, which is a copy of the Pirate Pad from Lauren: http://piratepad.net/2C2iwiLDhd" 1270801086000000 1270801210000000 288 defect dread dread v1.0 closed fixed Foreign and escaped characters in extras fields These don't work. 1271171899000000 1271173777000000 289 enhancement rgrp dread v1.0 closed fixed Document config options 1271248690000000 1271249368000000 290 defect johnbywater dread v1.0 closed fixed Revision API - docs doc/api.rst needs to cover the new Revision REST interface. 1271268759000000 1271636910000000 291 defect rgrp dread closed worksforme Investigate search options encoding "On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in. {{{ diff -r 813ad8b5de0b ckan/lib/search.py --- a/ckan/lib/search.py Mon Mar 01 22:23:36 2010 +0100 +++ b/ckan/lib/search.py Thu Apr 15 19:11:42 2010 +0200 @@ -63,6 +63,9 @@ def query(self, options): '''For the given search options, returns a query object.''' self._options = options + # temporary hack! + if self._options.q: + self._options.q = self._options.q.encode('utf8') general_terms, field_specific_terms = self._parse_query_string() if not general_terms and \ }}} " 1271351670000000 1273254895000000 292 defect dread dread v1.0 closed fixed PackageResources are created new on every package edit When you edit a package, all of its resources are deleted and a fresh set created. This causes all history of them to be lost. 1271755964000000 1272286005000000 293 defect johnbywater dread v1.0 closed fixed REST PackageResource update causes exception "This is caused by editing a package with resources over the REST interface. Exception: {{{ URL: http://de.ckan.net/api/rest/package/destatis Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module repoze.who.middleware:107 in __call__ << wrapper = StartResponseWrapper(start_response) app_iter = app(environ, wrapper.wrap_start_response) # The challenge decider almost(?) always needs information from the >> app_iter = app(environ, wrapper.wrap_start_response) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:51 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.rest:145 in update << fs = ckan.forms.group_fs_combined fs = fs.bind(entity, data=request_fa_dict) validation = fs.validate_on_edit(entity.name, entity.id) if not validation: response.status_int = 409 >> validation = fs.validate_on_edit(entity.name, entity.id) Module ckan.forms.package:385 in validate_on_edit << temp_name = orig_pkg_name self.data['Package-%s-name' % record_id] = u'something_unique' validation = self.validate() if temp_name: # restore it >> validation = self.validate() Module formalchemy.forms:175 in validate << if self.readonly: raise Exception('Cannot validate a read-only FieldSet') return AbstractFieldSet.validate(self) def sync(self): >> return AbstractFieldSet.validate(self) Module formalchemy.forms:104 in validate << success = True for field in self.render_fields.itervalues(): success = field._validate() and success # run this _after_ the field validators, since each field validator # resets its error list. we want to allow the global validator to add >> success = field._validate() and success Module formalchemy.fields:763 in _validate << try: value = self._deserialize() except validators.ValidationError, e: self.errors.append(e) >> value = self._deserialize() Module formalchemy.fields:1354 in _deserialize << if self.is_composite_foreign_key: return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize())) return self.renderer.deserialize() >> return self.renderer.deserialize() Module formalchemy.fields:183 in deserialize << if self.field.is_collection: return [self._deserialize(subdata) for subdata in self._serialized_value()] return self._deserialize(self._serialized_value()) def _deserialize(self, data): >> return self._deserialize(self._serialized_value()) Module formalchemy.fields:167 in _serialized_value << if self.field.is_collection: return self._params.getall(self.name) return self._params.getone(self.name) def deserialize(self): >> return self._params.getone(self.name) Module formalchemy.base:63 in getone << if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage): return v return v[0] def getall(self, key): v = dict.get(self, key) >> return v[0] TypeError: 'int' object is unsubscriptable }}} Thanks to Friedrich for spotting this." 1271756102000000 1271940083000000 294 enhancement thejimmyg dread closed duplicate Add/remove extra fields in Package edit form Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources. 1271756591000000 1291830960000000 295 enhancement dread dread v1.0 closed fixed Add multiple packages when editing a group "In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add. Ticket is split off from ticket:221 See also ticket:155 (duplicate)." 1271756713000000 1272384758000000 296 enhancement johnbywater johnbywater closed duplicate Commit CKAN revisions to changeset system 1272279521000000 1294407032000000 297 enhancement johnbywater johnbywater closed duplicate Update CKAN repository from changeset system 1272279556000000 1294407051000000 298 enhancement johnbywater johnbywater closed duplicate Pull changesets from remote CKAN instance 1272279591000000 1294407080000000 299 enhancement johnbywater johnbywater closed duplicate Merge diverging lines of changesets 1272279698000000 1294407099000000 300 defect rgrp dread v1.0 closed fixed Resource ordering issue "Failing test: ckan.tests.models.test_resource.TestResourceLifecycle.test_03_reorder_resources Not clear how visible this is to the user. Related to ticket:292" 1272285994000000 1272384474000000 302 enhancement johnbywater dread v1.0 closed fixed Advertise a package's feed visually "Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages. Motivation: each package's Atom feed is mentioned in the <link rel=""alternate"" type=""application/atom+xml"" ...> tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it. The Repo History (/revision) page has a visual link the feed. PS It would be nice to display the feed icon in both these places too." 1272370397000000 1272453821000000 303 defect dread dread v1.0 closed fixed Package history & diff & feed to show changes to tags, extras and resources "Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a PackageResource. This doesn't include changes to tags, extras and resources. These need to be added. There is the complication that the 'diff' function doesn't currently work beyond PackageResource. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now. Rufus suggests alongside the existing methods: {{{ pkg.all_revisions() pkg.diff() }}} we have ones which also include the related objects (tags, extras, resources): {{{ pkg.all_revisions_full() pkg.diff_full() }}} You could have a function which returned these related objects. Ideas: {{{ revobjlist = [ self, resources, extras ] list of tuples: (revobj, changed_revobjs) list of tuples: (revobj, {objtype: [changed_revobjs]}) tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...} }}}" 1272370659000000 1272989728000000 304 defect johnbywater johnbywater v1.0 closed fixed Previewing package with resource with missing url is broken "Intended outcome: See resource url validation error when previewing package with a resources that has no url. Recurring Error: {{{ URL: http://127.0.0.1:5000/package/new File 'pyenv-ckan/lib/python2.5/site-packages/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File 'pyenv-ckan/lib/python2.5/site-packages/repoze/who/middleware.py', line 107 in __call__ app_iter = app(environ, wrapper.wrap_start_response) File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Routes-1.10.3-py2.5.egg/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File 'pyenv-ckan/src/ckan/ckan/lib/base.py', line 50 in __call__ return WSGIController.__call__(self, environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 216 in new clear_session=True) File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 357 in _render_edit_form edit_form_html = fs.render() File 'pyenv-ckan/src/ckan/ckan/forms/builder.py', line 13 in render return render(self.form_template) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render format=format, namespace=kargs, **cache_args) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render **options) File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 533 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate File 'pyenv-ckan/src/ckan/ckan/templates/package/form_fields.html', line 52 in <Expression u'h.literal(field.render())'> ${h.literal(field.render())} File 'pyenv-ckan/lib/python2.5/site-packages/FormAlchemy-1.3.4-py2.5.egg/formalchemy/fields.py', line 1217 in render return self.renderer.render(**opts) File 'pyenv-ckan/src/ckan/ckan/forms/common.py', line 311 in render return render('package/form_resources') File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render format=format, namespace=kargs, **cache_args) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render **options) File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 510 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate File 'pyenv-ckan/src/ckan/ckan/templates/package/form_resources.html', line 14 in <Expression u""res['id'] if res else ''""> <td><input name=""${c.id}-${i}-id"" type=""hidden"" value=""${res['id'] if res else ''}"" /></td> File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 335 in lookup_item File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 405 in undefined UndefinedError: {'url': u'a', 'hash': u'a', 'description': u'a', 'format': u'a'} has no member named ""id"" }}} How to reproduce: 1. Go to package edit form, enter a package resource **but without a URL**. 2. Click preview. " 1272379550000000 1272447296000000 305 defect johnbywater johnbywater v1.0 closed fixed Search result pagination is broken "Expect to page through results. Only page 1 is shown, all other pages fail to display. Reproduce by searching for something common and browsing to the second page." 1272468229000000 1272994804000000 306 enhancement rgrp rgrp closed duplicate datapkg build command "Need to be able to build a distribution. Need: * new 'build' command * specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution. " 1272474212000000 1318181194000000 307 defect rgrp rgrp datapkg-0.7 closed fixed datapkg upload command "Support uploading a distribution somewhere. * New cli command: upload * Pluggable uploader framework. Suggest initial support Depends on ticket:306" 1272474224000000 1288027815000000 308 enhancement rgrp rgrp closed duplicate Autocomplete package names & tags in package search "Extracted from ticket:216. Dubious of its merit." 1273050549000000 1275302577000000 309 defect pudo dread closed fixed Tag list shows unused tags "Deleted tags are visible on the /tag page and are searchable. Delete tags are ones that were in use once and then discarded, so this includes ones which are not used on any packages any more, including misspelling, mistakes, and deprecated tags. It would makes sense to remove them. Example tag that is viewed at /tag and searchable: http://ckan.net/tag/2rand[0" 1273247648000000 1280743432000000 310 defect dread rgrp v1.1 closed fixed Commit message box looks wrong in edit page since edit style overhaul "Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site). * Also change label to: Edit summary (Briefly describe the changes you have made) * Remove: you can markdown formatting here. * Move author: if you have not signed in smaller and closer (like markdown instructions are nwo). * Change commit -> save * Remove ""please save"" just have the bullet points" 1273348714000000 1279300525000000 311 defect dread rgrp v1.1 closed fixed Reordering of package resources can lead to integri "Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have: {{{ [Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - <class 'sqlalchemy.exceptions.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint ""package_resource_revision_pkey"", referer: http://ckan.net/package/edit/cofog [Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] 'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog }}} " 1273348968000000 1274282065000000 312 defect rgrp dread closed invalid Race condition creating PackageRating "On ckan.net there are a few packages which cause a 403 exception when you try to rate them: http://www.ckan.net/package/rate/coins-data?rating=3 The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError: Multiple rows returned for one() It looks like this package got rated twice in quick succession, creating two similar PackageRating objects. This race condition needs to be solved and these particular rating objects repaired." 1273482785000000 1311176173000000 313 enhancement rgrp dread v1.1 closed fixed Allow packages to be specified by IDs in REST interface "If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID. It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface. " 1273606248000000 1275404524000000 314 defect johnbywater johnbywater closed fixed Bugs getting revisions from the REST API "Bug report regarding getting revisions: Getting revisons by ID (on the latest ID) GET ""http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"" Returns a whole load of stuff (it's also quite slow - about 10 seconds) I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API? GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45"" Returns the empty string - I'd expect an empty array ie [] GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45"" Returns an internal server error 500 - I think it should probably be ""bad Request"" 400 (the date is invalid) " 1273743755000000 1276523983000000 315 defect dread rgrp v1.1 closed fixed Improvements and fixes to csv dump "1. Issues with quote in fields: http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html 2. Issues with package resource serialization into csv table. <quote> In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. </quote>" 1274100887000000 1275846764000000 316 defect rgrp dread closed fixed Search URL escaping "If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error. e.g. http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk returns this exception: {{{ URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module repoze.who.middleware:107 in __call__ << wrapper = StartResponseWrapper(start_response) app_iter = app(environ, wrapper.wrap_start_response) # The challenge decider almost(?) always needs information from the >> app_iter = app(environ, wrapper.wrap_start_response) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:50 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.package:52 in search << collection=query, page=request.params.get('page', 1), items_per_page=50 ) # filter out ranks from the query result >> items_per_page=50 Module webhelpers.paginate:333 in __init__ << self.item_count = item_count else: self.item_count = len(self.collection) # Compute the number of the first and last available page >> self.item_count = len(self.collection) Module webhelpers.paginate:204 in __len__ << def __len__(self): return self.obj.count() # Since the items on a page are mainly a list we subclass the ""list"" type >> return self.obj.count() Module sqlalchemy.orm.query:1094 in count << q = q.params(params) q = q._legacy_select_kwargs(**kwargs) return q._count() def _count(self): >> return q._count() Module sqlalchemy.orm.query:1103 in _count << """""" return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key)) def _col_aggregate(self, col, func, nested_cols=None): >> return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key)) Module sqlalchemy.orm.query:1125 in _col_aggregate << if self._autoflush and not self._populate_existing: self.session._autoflush() return self.session.scalar(s, params=self._params, mapper=self.mapper) def compile(self): >> return self.session.scalar(s, params=self._params, mapper=self.mapper) Module sqlalchemy.orm.session:635 in scalar << engine = self.get_bind(mapper, clause=clause, instance=instance) return self.__connection(engine, close_with_result=True).scalar(clause, params or {}) def close(self): >> return self.__connection(engine, close_with_result=True).scalar(clause, params or {}) Module sqlalchemy.engine.base:834 in scalar << """""" return self.execute(object, *multiparams, **params).scalar() def statement_compiler(self, statement, **kwargs): >> return self.execute(object, *multiparams, **params).scalar() Module sqlalchemy.engine.base:844 in execute << for c in type(object).__mro__: if c in Connection.executors: return Connection.executors[c](self, object, multiparams, params) else: raise exceptions.InvalidRequestError(""Unexecutable object type: "" + str(type(object))) >> return Connection.executors[c](self, object, multiparams, params) Module sqlalchemy.engine.base:895 in execute_clauseelement << else: keys = None return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params) def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None): >> return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params) Module sqlalchemy.engine.base:907 in _execute_compiled << context.pre_execution() self.__execute_raw(context) context.post_execution() self._autocommit(context) >> self.__execute_raw(context) Module sqlalchemy.engine.base:916 in __execute_raw << self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context) else: self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) def _execute_ddl(self, ddl, params, multiparams): >> self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) Module sqlalchemy.engine.base:958 in _cursor_execute << self.engine.logger.info(repr(parameters)) try: self.dialect.do_execute(cursor, statement, parameters, context=context) except Exception, e: self._handle_dbapi_exception(e, statement, parameters, cursor) >> self.dialect.do_execute(cursor, statement, parameters, context=context) Module sqlalchemy.engine.default:133 in do_execute << def do_execute(self, cursor, statement, parameters, context=None): cursor.execute(statement, parameters) def is_disconnect(self, e): >> cursor.execute(statement, parameters) UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128) }}}" 1274265928000000 1291831177000000 317 enhancement pudo rgrp v1.1 closed fixed Make search pluggable "Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR). Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable." 1274295418000000 1279286041000000 318 defect johnglover wwaites ckan-sprint-2011-10-28 closed fixed Insufficient validation of resource URIs "The CKAN instance on data.gov.uk serves invalid URIs out of its API. For example the following can be found, http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls] In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396 Also URIs are not stripped of whitespace at the end. This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs. ""Be liberal in what you accept and conservative in what you send"" would seem apt. == Actions == * Validation of urls as part of form entry or data loading * Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal * This code should support analysis of existing data so we can go through existing database and find invalid urls * Also useful to have this so we can do out of band validation " 1274302723000000 1349778662000000 319 defect dread dread closed fixed Local Authority license misnamed in migration script 18 "Migration script 18 converts license ""OKD Compliant::Local Authority Copyright with data.gov.uk rights"" into ""uklocalauthority-withrights"" when it should be ""localauth-withrights"", according to the license service SoS 2. This causes clients difficulties looking it up. " 1274349714000000 1274366882000000 320 enhancement dread rgrp v1.1 closed fixed site_title configuration variable which is used in template "As a sysadmin I want to configure basic site title information for use in the site templates. Implementation: * ckan.site_title config variable * set this on g in app_globals.py e.g. * from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network') * use in head title and in main site title/logo section (use it as alt on logo image) * Also all other pages (e.g. index, about) which talk about CKAN * Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates? Questions: * Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined? * Probably yes, but '''not''' part of this ticket." 1274378967000000 1279130535000000 321 enhancement thejimmyg johnbywater closed duplicate Delegate authentication to Drupal "When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model. The Drupal front-end shall have: 1. Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER. 2. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions. 3. Access denied page - fixed location, static resource, gently indicates what has happened, and how to ask for permission. The CKAN slave edit page shall: 1. Try to detect a Drupal session key (passed as cookie or as request param). 2. Redirect to Drupal login page if no session key. 3. Check authorisation if session key is found. 4. Redirect to access denied page if session key not authorised. 5. Present the Package edit page. 6. Reject unauthenticated or unauthorised edit submissions. 7. Snag invalid edit submissions from authenticated and authorised users. 8. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end. " 1274705234000000 1291831399000000 322 enhancement dread dread v1.1 closed fixed Client interface for Notification Service "=== Use cases === * Register for package changes * Register for all revisions * Notified of a package change * Notified of a revision * Deregistration * Configuration of port in pylons config === Design === * Default port: 5672 (standard for AMQP) * Exchange name: 'ckan' * Exchange type: topic exchange (most flexible) * Routing keys: (see below) === Routing detail === Routing key format: ""OBJ_TYPE"" (NB tags should be identified by their name, not ID) Example routing keys * 'package' - Package edited/created * 'resource' - Resource edited/created * 'revision' - Any change * 'db.clean' * 'db.rebuild' Example queue bindings that clients may use: * * - no filtering - client receives all notifications * package - only changes to packages * revision - all revisions * db - all database operations === Versioning === Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps? === Documentation === * How to use * simple example of an external client?" 1274720042000000 1277722821000000 323 enhancement dread dread v1.1 closed fixed Notification message "=== Which events to notify on === Listed by domain object, these are the notification message 'change types' that will be sent: * Package * !PackageResource Also it is clear that it could be useful to know when db-wide maintenance is carried out: * db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration) === Ignored domain objects === These parts of the domain model will not carry notifications as no use case has been identified for them: * Revision * Group * Tag * Rating * User - list of users is sensitive info * Relationships - complicated * Authz - complicated and sensitive info * License - change of a license's metadata is a question for the 'license service' === Message format === A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity. For the 'db' notifications there shall be no payload." 1274723333000000 1278578841000000 324 enhancement dread dread v1.1 closed fixed Search indexing using notifications "Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead. The indexing shall run in a separate shell/process, managed by supervisord." 1274723483000000 1278599927000000 325 enhancement dread dread v1.1 closed fixed Event push notification "=== As a === CKAN client program === I want to === be notified when changes to the CKAN metadata occur. === Examples of use === 1. An external search engine needing to (re)index a package. (interest: Package) 2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333. 3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: !PackageResource) 4. Do some processing on resource (e.g. extract sample data for display) (interest: !PackageResource) === Context === The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling. === Design === Split-off into two tickets: * Notification message - ticket:323 * Which events to notify on * Message format * Interface for Notifier Service - ticket:322 === Testing === To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine. === Related === * Run CKAN search indexing using this system - ticket:324 * Run SOLR indexing using this system - ticket:353 * Web hooks for notifications - ticket:327 " 1274723512000000 1278599979000000 326 task dread dread v1.1 closed fixed Centralise importation of json library Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers. 1274784223000000 1274789296000000 327 defect pudo pudo closed wontfix Create a web hook worker for CKAN " This will be useful as an example consumer of the new queue notifications (#325). * A user can register any URL to be notified upon an event. * Possible extensions: GET with ID only vs. POST with serialized object * E-Mail notifications are also hooks, essentially. We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)" 1274807361000000 1296467361000000 328 enhancement dread rgrp v1.1 closed fixed Mention code libraries in API documentation "Should prominently (at the top?) mention existing code libraries for working with ckan api. Have: * Python: ckanclient * Perl: luke closs wrote something * PHP: drupal library? * ...?" 1274988106000000 1275318745000000 329 defect rgrp dread v1.1 closed fixed Bad dates cause exception on Gov form "== Reproduction == Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message. Affects all versions of CKAN. == Why it's happening == The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught." 1275060617000000 1275079189000000 330 defect dread dread closed fixed getdata/ons timezone not recognised on non-British servers "The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error: DateConvertError: Could not read date as ISO format ""%a, %d %b %Y %H:%M:%S %Z"". Date provided: ""Mon, 04 Jan 2010 09:30:00 BST"" It turns out that it only recognises local names of timezones. The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems." 1275300271000000 1275303122000000 332 defect pudo rgrp closed fixed Autocomplete tag on click "During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box. " 1275302887000000 1280743320000000 333 enhancement dread v1.1 closed wontfix CKAN front end requirements for package notifications "== Use case: new package == 1. An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package. 2. User: clicks 'new package'. 3. CKAN presents the package/new form to the user. 4. (After a couple of previews) User: clicks 'commit'. 5. Notification message goes from CKAN to the front-end detailing the new package. 6. The user is redirected back to the front-end web page displaying the list of packages, which contains the new one. The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6). In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages? == Use case: CKAN imports packages == 1. CKAN administrator runs a script that adds 100 new packages into CKAN. 2. CKAN sends notification message to front-end to report the new packages/revisions. 3. Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages. 4. The front-end queries CKAN for each new package one-by-one. 5. A new user request to the front-end will include the info about the new packages. The package addition could be achieved in 1 revision, 100 revisions or some compromise: * If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'. * If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN. This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages." 1275324042000000 1275407987000000 334 defect pudo dread closed fixed Wrong link for package feed icon "This is a problem with the package page e.g. http://ckan.net/package/open-election-data-project . The feed icon links to [1] and the text next to it ""Subscribe"" links to [2]. Surely these should be the same? The second link seems to right one to me. * [1] http://ckan.net/package/open-election-data-project?format=atom * [2] http://ckan.net/package/history/open-election-data-project?format=atom&days=7" 1275407445000000 1280743667000000 335 enhancement dread dread v1.1 closed fixed Post-package-edit redirect to configurable URL "== As a == third-party interface to a CKAN instance == I want to == link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return. == Design == 1. The 'return URL' is passed as a parameter to CKAN. 2. CKAN substitutes the package name into the return URL. == Example == Front-end links to: !http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/<NAME> (but with the parameter URL-encoded) When finished editing and the user commits, CKAN redirects the user to: !http://datadotgc.ca/dataset/pollution_data " 1275408834000000 1276179605000000 336 defect dread donovanhide v1.1 closed fixed Resource Search API "== As a == CKAN client such as !ScraperWiki == I want to == search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL. == Proposed implementation == Add resource search API at: /api/search/resource AND resource added to model API at: api/rest/resource (see ticket:358) Functional differences from the !ScraperWiki suggested patch: * URL is not normalised * URLs are not grouped * All fields of the resource object are returned, not just the URL * Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend 'http://ckan.net/package/' to the package ID) This is to make our API more general, simple and consistent. It means that the !ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok? === Example search === POST {{{ {""url"": ""scraperwiki.com/"", ""all_fields"": 1} }}} to: /api/2/search/resource returns JSON: {{{ [{""id"": ""a3dd8f64-9078-4f04-845c-e3f047125028"", ""package_id"": ""b8a325c8-af2a-43f3-8245-9db7d73dfbfe"", ""URL"": ""http://scraperwiki.com/lincolnshire-councillors"", ""format"": ""CSV"", ""Description"": ""Scrape of www.lincs.gov/councillors.pdf by ScraperWiki."", ""hash"": """", ""position"": 2 }] }}} Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe === Search Parameters === {{{ Key: q Description: Search all resource fields for the value Key: url / description / format / Description: Search particular field for the value Key: all_fields Value: 0 or 1 (0 is default) Description: If 1 (true), the full record of the package resource (and it's package reference) are returned, rather than just the PackageResource ID. }}} May also choose to introduce 'offset' and 'limit' to page through a large number of results. JSONP achieved through API-wide parameter - see ticket:342 Search is case insensitive. == Original request == Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388. Could also add a search version. Not sure what the best url would be for that though. Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere? Donovan" 1275411765000000 1279373842000000 337 enhancement johnbywater rgrp v1.1 closed fixed Download links for resources should open in new window "target = _blank Cost: 30m" 1275422151000000 1279300972000000 338 story johnbywater johnbywater v1.1 closed Reference groups by ID in addition to name, since group names can change 1275901137000000 1280446480000000 339 defect johnbywater closed fixed Remove download_url from package data format in API Version 2, but maintain the old data format in the old version of the interface (API Version 1) 1275901499000000 1278074705000000 340 enhancement dread dread v1.1 closed fixed Web UI theme easier to configure "An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in. http://wiki.okfn.org/ckan/doc/theme" 1276009729000000 1328807317000000 341 enhancement dread closed fixed Web UI accepts package IDs in URLs "(in the same way that we can refer to packages in the API by ID as well as name.) == As a == client of CKAN == I want to == link to a package page in the UI, referring to it by ID" 1276162400000000 1277483030000000 342 enhancement dread dread closed fixed JSONP parameter in API "== As a == CKAN client using JQuery == I want to == call the CKAN API and instead of receiving back JSON I get JSONP. i.e. ""%s(%s)"" % (callback, json_content) == Suggested implementation == All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide: http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch == Test == import re import unittest def test_jsonp_callback(): response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com&callback=jsoncallback') match = re.match('jsoncallback\(.*\);',response) self.assertTrue(match) response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com') match = re.match('jsoncallback\(.*\);',response) self.assertFalse(match) I think the point needs to be made that JSONP only works for GET requests and not POST/PUT/DELETE, so there needs to be a check for that in the _finish_ok method. (thanks to Donovan Hide for test)" 1276166426000000 1276278485000000 343 defect johnbywater dread closed fixed Packages referred by ID in API When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID. 1276180179000000 1278066420000000 344 defect dread closed fixed REST Create package with incorrect format gives 500 error "PUT to /rest/api/package of {{{ {""name"": ""name"", ""resources"": [""someurl.com/data""] } }}} (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message. Exception: {{{ Module ckan.controllers.rest:154 in create << if register == 'package' and not subregister: fs = ckan.forms.get_standard_fieldset() request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data) fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session) elif register == 'package' and subregister in model.PackageRelationship.get_all_types(): >> request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data) Module ckan.forms.package_dict:88 in edit_package_dict << for res_dict in value: res_dict_str = {} for key, value in res_dict.items(): res_dict_str[str(key)] = value resources.append(res_dict_str) >> for key, value in res_dict.items(): AttributeError: 'unicode' object has no attribute 'items' }}}" 1276341172000000 1277477712000000 345 requirement dread johnbywater closed invalid metastable, stable & ultrastable branches shall be automatically built and tested 1276523083000000 1291831615000000 346 defect dread johnbywater ckan-v1.3 closed wontfix Revision search API (response data format and documentation issue) "Whilst going through the API docs for the revision search API, it was noticed that the ""Gdu"" SoS doc doesn't match up. It returns revision IDs (perhaps this is useful to note in the spec?) so the format is probably not 'limitedstring'. Also, they appear to be ordered youngest first, not oldest as stated. And in the revision model, it refers to 'simplestring' which it doesn't define - I guess the names should be 'limitedstring'? Could this be checked out?" 1276523517000000 1296477510000000 347 enhancement johnbywater johnbywater closed fixed The system shall present the package edit form in an API 1276523692000000 1277820440000000 348 enhancement johnbywater johnbywater closed fixed The system shall accept package edit form submissions in an API 1276523793000000 1277820496000000 349 defect johnbywater johnbywater ckan-v1.2 closed invalid The system shall support authorised users to edit packages on data.gov.uk 1276523851000000 1282908974000000 352 enhancement dread dread closed wontfix Package notification worker - sends XML-RPC "== As an == external front-end == I want to == be notified (by XML-RPC) about package creations and updates. == Implementation == 1. A message queue worker waits for package update notifications 2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI. Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter. Config - in the CKAN config will be: * URI to callback to * API version to use (version 2 gives packages referred by ID not name)" 1276597996000000 1286375870000000 353 defect dread closed fixed SOLR search indexing "== As a == SOLR instance == I want to == keep my search index of CKAN packages up-to-date == Implementation == * Using asynchronous event notifications * Running in a separate process to CKAN" 1277123480000000 1280756399000000 354 defect johnbywater johnbywater closed invalid Collect together requirements and top-level design for user/package 'groups' "Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff. http://knowledgeforge.net/ckan/trac/wiki/AccessControl Do we add these into user-role table somehow or new table? To present this to team" 1277131335000000 1282908983000000 355 defect rgrp rgrp closed fixed Dashes versus underscores in package names Sort out how we deal with dashes versus underscores in package names. 1277221996000000 1311177552000000 356 enhancement rgrp v1.1 closed fixed Search box in at top of page (UI) "A small but useful ui improvement would be to have a search box at top right on every page. As an example see the one here on trac or on github.com or bitbucket.org. * It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration)." 1277235411000000 1278931830000000 357 defect johnbywater johnbywater closed fixed Fix API documentation, to correct misleading HTTP_AUTHORIZATION statement. "Sean Burlington wrote: > I recently came across a documentation issue > > http://knowledgeforge.net/ckan/doc/ckan/api.html > ----- > The key should be passed in the API request header: > Header Example value > HTTP_AUTHORIZATION fde34a3c-b716-4c39-8dc4-881ba115c6d4 > > ----- > > But the header name is actually just 'AUTHORIZATION' > > For example to set it in PHP > > curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION: > xxxxx-xxxx-xxxx-xxxx-xxxxxxx')); > " 1277459886000000 1277461466000000 358 enhancement rgrp dread ckan-v1.5 closed duplicate Resources in REST API "(spun out of ticket:336) Resource added to model API at: api/rest/resource === Example model request === GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028 returns: {{{ [{""id"": ""a3dd8f64-9078-4f04-845c-e3f047125028"", ""package_id"": ""b8a325c8-af2a-43f3-8245-9db7d73dfbfe"", ""URL"": ""http://scraperwiki.com/lincolnshire-councillors"", ""format"": ""CSV"", ""Description"": ""Scrape of www.lincs.gov/councillors.pdf by ScraperWiki."", ""hash"": """", ""position"": 2 }] }}} == Authorization == 1. Have it generic (ie. not per resource) and use an action/role on system 2. Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages) 3. Introduce Resource in authorization system (requires migration) Mixed model {{{ Create / Edit: if resource associated to package: check_permissions(package, update) else: check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System) }}}" 1277483282000000 1310128782000000 359 requirement pudo rgrp datapkg-0.7 closed fixed Create Storage with pluggable extensions "Storage represents something you can upload to (download is done in ticket:406). * storage sub package with defined interface (maybe just use OFS) * Only support uploading single files for the moment (though can do multiple files). * Relationship to build (ticket:306) * No support for syncing e.g. hg/git repos -- that's up to you as a user * Upload command At end of this we can deprecate Repository. Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication." 1277750654000000 1291135692000000 360 defect rgrp rgrp datapkg-0.7 closed fixed New sqlite-based DB index "In 0.5 and 0.6 have been more oriented to getting file indexes working to support the convenience of using packages on disk directly. However we now need to go back to having a better central/default index and the best model for this is a simple db index using sqlite (which is now supported in the standard library). NB: this is a retrospective ticket (been working on this for a couple of months)" 1277803174000000 1288004891000000 361 enhancement rgrp rgrp datapkg-0.7 closed fixed Allow specification of the default index in configuration 1277803305000000 1291135756000000 362 defect dread dread closed invalid Ratings should not be created with a GET "== Background == In the Web UI, when you rate a package it simply links to something like: http://ckan.net/package/rate/mke-liquor-licenses?rating=3 This creates a GET request. This is bad because: * Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this) * There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept GET> Apache instruction in. But the database may still get written for these ratings. * Best practise for web requests is for GET to be a read-only request." 1278925451000000 1311176564000000 363 defect kindly dread ckan-backlog closed wontfix Blank revisions "Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they? They appear on the 'Recently changed' list on the home page with an empty 'Packages' column." 1278947772000000 1310125872000000 364 defect dread ckan-v1.3 closed fixed Search for 'statistic' returns nothing "On ckan.net there are plenty of packages (and indeed their tags) with the word 'statistic' in them, but no packages turn up when you search for it: http://ckan.net/package/search?q=statistic&search=Search+Packages+%C2%BB (Using Postgres full text search)" 1278949620000000 1291637291000000 365 enhancement dread dread closed fixed ResourceNotifications If you change a resource then you not only get a PackageNotification, but also a ResourceNofication. 1279037411000000 1279300621000000 366 enhancement pudo dread ckan-v1.4 closed fixed A flash message says why you are redirected to the 'user login' page "You get redirected for a number of different reasons - often confuses me let alone the average user! Cost: 1h" 1279128058000000 1300212171000000 367 enhancement dread dread closed fixed Notfication monitor Runs on the commandline and prints out notifications. 1279303310000000 1279303693000000 368 defect anonymous closed wontfix 500 Server error when creating package " I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response: ""publish: New Zealand Coastline (new-zealand-coastline)"" opening connection to nz.ckan.net... opened <- ""POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n"" <- ""{\""name\"":\""new-zealand-coastline\"",\""title\"":\""New Zealand Coastline\""}"" -> ""HTTP/1.1 500 Internal Server Error\r\n"" -> ""Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n"" -> ""Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n"" -> ""Vary: Accept-Encoding\r\n"" -> ""Content-Type: text/html; charset=iso-8859-1\r\n"" -> ""Cache-Control: proxy-revalidate\r\n"" -> ""Content-Length: 640\r\n"" -> ""Connection: close\r\n"" -> ""\r\n"" reading 640 bytes... -> ""<!DOCTYPE HTML PUBLIC \""-//IETF//DTD HTML 2.0//EN\"">\n<html><head>\n<title>500 Internal Server Error\n\n

    Internal Server Error

    \n

    The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.

    \n

    Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.

    \n

    More information about this error may be available\nin the server error log.

    \n
    \n
    Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80
    \n\n"" read 640 bytes Conn close The server should not respond with a 500 error if the package was created. " 1279622462000000 1291831811000000 371 requirement nils.toedtmann johnbywater ckan-v1.4 closed fixed The system shall monitor QoS against SLA Requested by DGU. 1279885622000000 1300217820000000 372 bug johnbywater johnbywater ckan-v1.2 closed Fix system limits on CKAN for DGU Set limits in /etc/security/limits.conf so that we can always ssh in at least. Requested by DGU. 1279885752000000 1281522535000000 373 requirement dread johnbywater closed fixed Sort out gov daily script "Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally. Requested by DGU." 1279885887000000 1286376176000000 374 task johnbywater johnbywater v1.1 closed fixed Design servicization of CKAN API 1280223118000000 1280398501000000 375 task johnbywater johnbywater v1.1 closed fixed Integrate Forms API into Drupal Requested by DGU. 1280223168000000 1280405812000000 376 task johnbywater johnbywater v1.1 closed fixed Implement servicization of CKAN API Requested by DGU. 1280223226000000 1281000755000000 377 enhancement glen@… ckan-future closed wontfix API Should return JSON in all cases "When the API returns any response it should always retunr JSON. For example a 404 response should return something like {{{ {'status': 404, 'message': 'package somename not found'} }}} When errors are encountered when creating a package: {{{ {'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be blank', 'description':'license code must be an integer']} }}} Not authorised: {{{ {'status': 503, 'message': 'You are not authorised to create this package.'} }}} The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created. " 1280268158000000 1338206349000000 378 story johnbywater johnbywater v1.1 closed fixed My user story #1 1280325186000000 1280326706000000 379 task johnbywater johnbywater v1.1 closed fixed My dev task #1 1280325277000000 1280326496000000 380 task johnbywater johnbywater v1.1 closed fixed My dev task #2 1280325304000000 1280326595000000 381 defect johnbywater closed invalid Test defect please ignore 1280345937000000 1294410466000000 382 story johnbywater johnbywater v1.1 closed Measure quality of service parameters As a service administrator, I want to measure responsiveness, throughput, and availability of a CKAN service. 1280346974000000 1280854608000000 383 task johnbywater johnbywater v1.1 closed fixed Create middleware feature to generate QoS throughput log 1280347595000000 1280854578000000 384 story johnbywater johnbywater v1.1 closed Send alert when QoS measurements break expectation 1280347841000000 1280496812000000 385 task johnbywater johnbywater v1.1 closed fixed Send email if QoS measurement breaks QoS criteria 1280355527000000 1280496127000000 386 task johnbywater johnbywater v1.1 closed fixed Set limits in /etc/security/limits.conf 1280356594000000 1281088994000000 387 task johnbywater johnbywater v1.1 closed fixed Fix failing tests in search sub-system trigged by recent changes 1280357804000000 1280446047000000 388 task johnbywater johnbywater v1.1 closed fixed "Reply to ""two projects"" question from RS" 1280395503000000 1280405288000000 389 task johnbywater johnbywater v1.1 closed fixed Investigate timezone issue raised by SB 1280395554000000 1280422414000000 390 task dread johnbywater ckan-v1.2 closed fixed Remove a package for SB 1280418625000000 1282214629000000 391 task johnbywater johnbywater v1.1 closed fixed Create monitoring feature to generate QoS responsiveness log 1280446714000000 1280854592000000 392 task johnbywater johnbywater v1.1 closed fixed Create monitoring feature to generate QoS availability log 1280446769000000 1280496140000000 393 task johnbywater johnbywater v1.1 closed fixed Create aggregation feature to aggregate QoS log 1280446811000000 1280496962000000 394 task johnbywater johnbywater closed duplicate Fix munin on DGU (?) 1280485351000000 1294407189000000 395 task pudo ckan-v1.3 closed duplicate Set up profiling to analyze performance issues "At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls. This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance. NB: We should also agree on a maximum request latency. TODO: Read up on all those QoS tickets to avoid overlapping efforts. " 1280824739000000 1294417538000000 396 task johnbywater johnbywater closed fixed Integrate Forms API into Drupal 1280854226000000 1282324934000000 397 task johnbywater johnbywater v1.1 closed fixed Implement QoS monitoring for DGU 1280854935000000 1281000745000000 398 task johnbywater johnbywater ckan-v1.3 closed fixed Create Public API documentation for DGU (draft) 1281000828000000 1294407372000000 399 task dread johnbywater ckan-v1.2 closed fixed Import BIS data for SD 1281000890000000 1282294177000000 400 task johnbywater johnbywater ckan-v1.2 closed fixed Configure DGU Public API with TSO Have TSO configure data.ov.uk/api/catalogue. Have TSO register catalogue.data.gov.uk. 1281000956000000 1288003690000000 401 task pudo pudo closed fixed Provide base worker as queue consumer We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app) 1281018724000000 1286189243000000 402 task pudo pudo ckan-v1.3 closed duplicate Archiving worker to back up package resources from a CKAN instance "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance? The caching worker will consumer update notifications and fetch packages. Extra points for: * Properly checking for source file modification (Last-modified, Etag) * Using PIP VCS Backends for retrieval * OFS/S3 Storage " 1281018912000000 1296467635000000 403 requirement pudo closed fixed Create web service for OFS storage "Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources. * relates to #307" 1281346110000000 1287392451000000 404 requirement rgrp pudo datapkg-0.8 closed fixed Make registries pluggable Allow using different indexes, including ckan, file, dcat, ... 1281346690000000 1297072955000000 405 task rgrp rgrp datapkg-0.8 closed fixed Retrieval options for package resources "Download Command (was install command) should support the following modes: * Download only the first listed resource (current behaviour, slightly arbitrary) * Download resources by interactive selection * Download resources by MIME type (cf #235) * Download all resources. " 1281346806000000 1297214833000000 406 requirement pudo rgrp datapkg-0.8 closed fixed Download sub-system "Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download. * Also rename InstallCommand -> DownloadCommand And we should support interactive retrieving of resources ... == Implementation Ideas == {{{ for resources in pkg.resources: # datapkg/hg load_storage_handler(resource.format) }}} " 1281346822000000 1296467427000000 407 requirement dread closed wontfix Link to Scraperwiki "== As a == user browsing a CKAN package which has a resource in a bad format == I want to == be alerted to the possibility of scraping it on scraperwiki == Implementation == This should be done in a CKAN extension. It should appear in a side-bar on the package read page. If the package does not have a resource with scraperwiki.com in the url, then it displays this message: Title: Scrape it[[BR]] Image: (Scraperwiki dumper truck)[[BR]] Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can [http://scraperwiki.com/market/request/ request] this data is scraped or even [http://scraperwiki.com/editor/ write the scraper] yourself. " 1281348368000000 1330769956000000 408 requirement pudo pudo ckan-v1.2 closed fixed Integrate CMS via proxying "* As sysadmin/user of CKAN I want to integrate a CMS to support easy addition of additional content ... * I want to integrate with navigation * As a minimum I want to be able to add content pages (more stuff would be e.g. comments, sidebars, main menu ...) Why: CMS are great at creating content and have very good editors and theme support ... === Previous work http://groups.google.com/group/pylons-discuss/browse_thread/thread/f8969e28b0a007c7 * deliverance: retheming on the fly http://knowledgeforge.net/shakespeare/hg/file/677eef9402e6/shakespeare/controllers/template.py 1. Rewrite urls for proxying 2. You need a theme to substitute into 3. pylons Extreme version: * 2 sources -> pylons/ckan app, cms * 1 theme source: ckan or cms 1 frontend/proxy app which re-themes everything Owned by CKAN: * package/ * tag/ * group/ * api/ * / Everything else proxied. " 1281348581000000 1281609572000000 409 task rgrp rgrp datapkg-0.8 closed fixed Refactor cli to be more 'pluggable' " * ~~(Load cli commands from setuptools entry point name datapkg.cli)~~ * Document how other can write new commands * Refactor base Command class to support this pluggability (if necessary)" 1281348659000000 1297074197000000 410 requirement pudo pudo ckan-v1.2 closed fixed Latest comments widget on home page This would be an extension to the disqus plugin that would patch itself into home/index. 1281348733000000 1281619720000000 411 requirement pudo pudo ckan-v1.2 closed fixed Deploy and test Varnish on eu3 "* Need to figure out cache invalidation for CKAN * Evaluate against an internal cache (i.e. memcached) " 1281349275000000 1287393033000000 412 requirement johnbywater closed fixed Data format for package importing shall be documented "Requirements: * cater naturally for multiple packages with multiple resources * international encoding * large number of packages" 1281430809000000 1289822092000000 413 task johnbywater johnbywater ckan-v1.3 closed invalid Write functional tests for DGU caching 1281431633000000 1294407458000000 414 task johnbywater dread ckan-v1.2 closed fixed Change the Apache and Varnish ports Ask Paul for a new machine for testing. Then one for varnish-live and one for varnish-test. 1281431639000000 1288003770000000 415 task dread ckan-v1.2 closed fixed Chase Talis about loading RDF from CKAN. 1281431656000000 1288003954000000 416 task johnbywater dread ckan-v1.2 closed fixed Meet Evan at COI at 1 o'clock Tuesday 10th August 2010 (to progress Forms API) 1281431677000000 1281529736000000 418 task dread closed invalid With COI, get SOLR working 1281431691000000 1291637919000000 419 task johnbywater johnbywater ckan-v1.2 closed fixed Meet Paul at COI at 2 o'clock Tuesday 10th August 2010 1281431794000000 1281529739000000 420 task dread dread ckan-v1.2 closed fixed Release ckan 1.1 1281432518000000 1281522091000000 421 task dread dread ckan-v1.2 closed fixed Upgrade test-hmg.ckan.net to latest ckan 1.1 1281432541000000 1282909772000000 422 story dread ckan-v1.3 closed fixed Administrator imports meta-data into CKAN "Requirements: * cater naturally for multiple packages with multiple resources * international encoding * large number of packages" 1281433490000000 1292586586000000 423 task dread johnbywater closed fixed Add examples to documentation about uploading data 1281435260000000 1282812768000000 424 task dread dread ckan-v1.2 closed fixed Tell JF the format spec & examples 1281436534000000 1282919429000000 425 task johnbywater johnbywater ckan-v1.2 closed fixed Document how licenses can be configured in CKAN 1281436551000000 1281605524000000 426 task dread dread ckan-v1.2 closed fixed Ask list for suggestions improving API docs 1281436574000000 1282925305000000 427 task thejimmyg dread ckan-v1.4-sprint-3 closed fixed Match licenses from GEMINI document " * examples of posting a new package * example license_id - explain" 1281436611000000 1299164063000000 428 requirement dread ckan-v1.2 closed fixed Daily dump of DGU CKAN available End users want to get hold of a JSON & CSV dump of the DGU data. 1281703251000000 1285757316000000 429 requirement dread ckan-v1.2 closed fixed ONS data is imported daily to DGU 1281703294000000 1285757274000000 430 task thejimmyg pudo closed wontfix Upgrade CKAN to Pylons 1.0 "Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism. http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed. " 1281962485000000 1298283075000000 431 task pudo pudo closed wontfix Solr index testing tool There seem to be a few conditions under which either queue processing or the indexer fail in their current state. To get a more systematic picture of these failures, we should have a small testing tool to compare the index to the database of a live CKAN instance. 1281963532000000 1295260262000000 432 defect dread dread ckan-v1.2 closed fixed Creating package over REST gives 500 error "This occurs when CKAN is run with mod_wsgi (not under paster). This is because of a unicode header being creating in rest.py." 1282065101000000 1282552819000000 433 defect wwaites datapkg-0.8 closed fixed Data package metadata in the Egg "Still not sure if we shouldn't use the existing setuptools machinery to manage this -- there is already a way to get at the metadata. In any event, I've just made an addition to datapkg that makes it possible to put datapkg_sources as a dictionary in your package's setup.py. Afterwards it is possible to pull the metadata out of the egg. Of course this could easily be changed to save the information in whatever form, indeed if you pass it a string instead of a dictionary it will just write whatever you gave it into the datapkg_sources.spec. The point is, I think that the egg is a good place to stuff this information. For non-python users, it is always possible to simply put up the datapkg_sources.spec somewhere on the web so they can directly retrieve the data files. From the docstring:: {{{ This is the implementation for an [egg_info.writers] entrypoint. Datapkg adds an argument to setuptools's setup() function called datapkg_sources. The argument should be a dictionary of the form: .. code-block:: python setup( ..., datapkg_sources = { ""cra2009"" : ""http://www.hm-treasury.gov.uk/d/cra_2009_db.csv"" } ) The result of this is that there will be a file in the egg called datapkg_sources.spec that looks like this:: [sources] cra2009=http://www.hm-treasury.gov.uk/d/cra_2009_db.csv How do you get at this data? Simple:: .. code-block:: python import pkg_resources dist = pkg_resources.get_distribution(""ukgov_treasury_cra"") spec = dist.get_metadata(""datapkg_sources.spec"") and 'spec' will be the contents of the file as a string. }}} " 1282078660000000 1283183647000000 434 defect dread ckan-v1.2 closed fixed Logged in user to si.ckan.net causes genshi exception "Here's the trace: {{{ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Error - : list index out of range [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] mod_wsgi (pid=6195): Exception occurred processing WSGI script '/home/okfn/var/srvc/si.ckan.net/py env/bin/si.ckan.net.py'. [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Traceback (most recent call last): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/cascade.py"", line 130, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.apps[-1](environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/registry.py"", line 375, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] app_iter = self.application(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/repoze/who/middleware.py "", line 107, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] app_iter = app(environ, wrapper.wrap_start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/middleware.py"", l ine 214, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] self.app, new_environ, catch_exc_info=True) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/util.py"", line 94 , in call_wsgi_application [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] app_iter = application(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/weberror/errormiddleware .py"", line 156, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.application(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py"", l ine 73, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.app(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py"", l ine 152, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.wrap_app(environ, session_start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/routes/middleware.py"", l ine 130, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self.app(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py"", line 125, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self.dispatch(controller, environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py"", line 324, in dispatch [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return controller(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py"", line 73, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return WSGIController.__call__(self, environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py"", line 221, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self._dispatch_call() [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py"", line 172, in _dispatch_call [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self._inspect_call(func) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py"", line 107, in _inspect_call [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] result = self._perform_call(func, args) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py"", line 60, in _perform_call [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return func(**args) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/controllers/error.py"", line 32, in doc ument [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] ckan_template = render('error_document_template.html') [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py"", line 38, in render [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return render_template() [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py"", line 36, in render_templ ate [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return literal(stream.render(method=method, encoding=None)) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py"", line 18 3, in render [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return encode(generator, method=method, encoding=encoding, out=out) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 57, in encode [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return _encode(''.join(list(iterator))) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 339, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 670, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 771, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in chain(stream, [(None, None, None)]): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py"", line 586, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for ev in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py"", line 288, in _ensure [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 618, in _include [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in tmpl.generate(ctxt, **vars): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 618, in _include [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in tmpl.generate(ctxt, **vars): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 605, in _include [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/markup.py"", line 327, in _match [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py"", line 545, in _flatten [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py"", line 177, in _generate [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in msgbuf.translate(gettext(msgbuf.format())): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py"", line 1029, in translate [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] parts = parse_msg(string) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File ""/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py"", line 1143, in parse_msg [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] parts.append((stack[-1], string)) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] IndexError: list index out of range }}} ckan: 1e403927f2bb (metastable) genshi: 0.6 lang: slovenian" 1282130515000000 1283189807000000 435 defect thejimmyg dread ckan-v1.4 closed wontfix Switching between ckan instances causes orphaned postgres processes "Analysis using pg_top reveals orphaed processes: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/460#comment:6 This followed switching ckan instances on the dgu machine. Do these hang around forever clogging up a db server? Can switching between instances be smoothed to avoid this?" 1282147736000000 1298284084000000 436 bug dread ckan-v1.2 closed wontfix Investigate exception: resource search JSON "Here's the dump from 22:10 last night: URL: http://ckan.net/api/search/resource?all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class= Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.rest:400 in search << response.status_int = 400 return gettext('Blank qjson parameter') params = json.loads(request.params['qjson']) elif request.params.values() and request.params.values() != [u''] and request.params.values() != [u'1']: params = request.params >> params = json.loads(request.params['qjson']) Module simplejson:384 in loads << parse_constant is None and object_pairs_hook is None and not use_decimal and not kw): return _default_decoder.decode(s) if cls is None: cls = JSONDecoder >> return _default_decoder.decode(s) Module simplejson.decoder:402 in decode << """""" obj, end = self.raw_decode(s, idx=_w(s, 0).end()) end = _w(s, end).end() if end != len(s): >> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) Module simplejson.decoder:420 in raw_decode << obj, end = self.scan_once(s, idx) except StopIteration: raise JSONDecodeError(""No JSON object could be decoded"", s, idx) return obj, end >> raise JSONDecodeError(""No JSON object could be decoded"", s, idx) JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/json' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_ACCEPT_LANGUAGE 'en-us,en;q=0.5' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE '__utma=27730403.1245320310.1281386803.1281386803.1282164955.2; __utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; __utmb=27730403.3.10.1282164955; __utmc=27730403' HTTP_HOST 'ckan.net' HTTP_KEEP_ALIVE '300' HTTP_REFERER 'http://jira/browse/PLATFORM-892' HTTP_USER_AGENT 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/search/resource' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/search/resource' QUERY_STRING 'all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=' REMOTE_ADDR '64.235.97.218' REMOTE_PORT '20720' REQUEST_METHOD 'GET' REQUEST_URI '/api/search/resource?all_fields=1&offset=0&limit=20&qjson=%3Cspan%20class=' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/api/search/resource' SCRIPT_URL '/api/search/resource' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
    Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282165818.0880959, '_creation_time': 1282165818.0880959} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '__utma=27730403.1245320310.1281386803.1281386803.1282164955.2; __utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; __utmb=27730403.3.10.1282164955; __utmc=27730403') paste.parsed_querystring ([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', ' paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'search', 'controller': u'rest', 'register': u'resource'} repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', ' wsgiorg.routing_args (, {'action': u'search', 'controller': u'rest', 'register': u'resource'}) " 1282206959000000 1288003983000000 437 bug dread ckan-v1.2 closed fixed Buildbot test failures - ascii codec "On today's buildbot: http://buildbot.okfn.org/builders/buildbot-test/builds/201 2 failures about ascii (ignore other 2)" 1282223640000000 1288004009000000 438 requirement dread dread ckan-v1.3 closed invalid Data importers use minimal processing e.g. ONS importer 1282224383000000 1294661079000000 439 task dread closed fixed Plan server requirements for DGU "Using use cases for security etc. Draw diagram to satisfy use cases. Communicate to Paul and therefore TSO." 1282224452000000 1294413903000000 440 task dread dread closed duplicate Write and pass comprehensive performance tests "Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes." 1282226932000000 1294417436000000 441 requirement dread dread ckan-v1.3 closed duplicate CKAN read-only state "When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages. Examples of use: * Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. * A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.: * another (but writable) CKAN instance is connected to the same db * restoring database dumps from another CKAN db * Should a security be breached, all editing could be stopped " 1282227314000000 1292586309000000 442 requirement dread dread ckan-v1.3 closed worksforme Importing data "Review what chunks of code we have and how to move forward with requirements. Requirements: * Bulk parsing is done away from the CKAN server Possible future requirements: * /importer for importing spreadsheets in WUI * validation-light - 'scoring' a package Current importers: * /importer controller uses: PackageImporter, form, PackageSaver * loader uses: package_importer -> API * API uses: form, PackageSaver Modules * package_importer: * xls/csv --SpreadsheetData--> table * table --DataRecords--> records * records --PackageImporter--> pkg_dicts * pkg_dict --form--> fs & validation * fs --PackageSaver--> package" 1282228977000000 1294661028000000 443 task dread dread ckan-v1.2 closed fixed Get DGU daily dump working (local) "* Producing daily dump file * Publishing at ckan.net/dumps/ or better" 1282299046000000 1282735977000000 444 task dread closed duplicate Discuss package relationships ideas with JF "* Create test data on visible ckan * Discuss with JF" 1282299238000000 1294414008000000 445 task johnbywater johnbywater ckan-v1.2 closed fixed Add dumps to DGU SLA "Thinking about putting something like this: Dumps Interface (this is NOT going in the API, it is another interface of the service): - resources: Daily Dump Folder Daily Dump File - locations: /dump/hmg.ckan.net-YYYY-MM-DD.[json|csv].zip - data format: Dump Folder is a HTML document containing a table of links to Dump Files Dump File has either JSON or CSV format. - status codes (200, 404, 500) " 1282299380000000 1283250620000000 446 task dread closed invalid Server configuration query (?) " * Ensure use case is in design - ""an admin writes to the CKAN API"" (and see JB notes on paper) * Communicate how this is planned to be achieved in design to JF" 1282299385000000 1294414077000000 447 task memespring johnbywater closed fixed Presentation on Package edit form (labels IE7, horiz alignment in chrome) Follow up with NS and RP. 1282299423000000 1311182895000000 448 story thejimmyg dread closed invalid [Assist with] editing packages through Drupal 1282299478000000 1294417061000000 449 task wwaites johnbywater closed invalid Start email thread about RDF in CKAN with JF and WW 1282299482000000 1311182945000000 450 task dread closed fixed Assist with proxying of static media for forms John did this a while ago. 1282299573000000 1287997487000000 451 task dread dread ckan-v1.3 closed invalid Show/Allow only departments associated with user Use drupal api 1282299641000000 1292587233000000 452 task dread closed fixed Assist with using package update error form 1282299647000000 1287997540000000 453 task thejimmyg johnbywater closed invalid Analysis and planning for UKLII 1282299725000000 1294411374000000 454 story dread dread ckan-v1.2 closed fixed Import ONS data via API 1282299851000000 1286376044000000 455 task johnbywater closed invalid SOLR - suggest 1 pager about how system would work Either CKAN writes to SOLR and Drupal reads from SOLR, or CKAN writes to SOLR and Drupal reads SOLR via CKAN API (so search resource locations are unaffected). 1282299913000000 1291637172000000 456 story johnbywater dread ckan-v1.2 closed duplicate Daily dump 1282299917000000 1282665858000000 457 story dread dread ckan-v1.2 closed fixed Dump database on a daily basis 1282299921000000 1282909051000000 458 task rgrp johnbywater closed wontfix Request to DGU for access to database machine 1282299950000000 1294415537000000 459 bug johnbywater ckan-v1.2 closed fixed Versions on branches are broken 1282299973000000 1282921783000000 460 defect pudo johnbywater closed fixed State field changed by non-sysadmin "This package: http://ckan.net/package/dbtune-audioscrobbler was: 1. created by Richard (logged-in) 2. edited by Richard (logged-in) (According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just ''? 3. pudo changed the state back to active Similarly an incident with bibbase package where field set to '' (see http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409) Need to investigate how this got changed, fix if necessary and report back to Richard. Note that package 'admins' as well as sysadmins can change the state of a package (though note that bibbase did not appear to have an owner). Suggested solution (for setting to ''): * Ensure in ckan/forms.py that there is a validator for state field that ensures only set to valid values. * Check that we do not allow state to be changed in the api except by package owner or sysadmin" 1282300002000000 1311176868000000 461 task dread johnbywater ckan-v1.2 closed fixed Change ONS data importing to work via API " * Move script out to ckandgu repo * Change script to convert xml into package dicts * Test (against test.ckan.net, hmg.test.ckan.net) * Deploy" 1282303411000000 1283250478000000 462 story dread johnbywater ckan-v1.2 closed fixed Visitor downloads a daily database dump 1282305305000000 1285757238000000 463 defect dread ckan-v1.3 closed worksforme package diff exception "Investigate exception: {{{ URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.revision:119 in diff << c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() >> diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff << display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] >> display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=0' HTTP_COOKIE '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173' HTTP_HOST 'ckan.net' HTTP_REFERER 'http://ckan.net/package/history/dbtune-audioscrobbler' HTTP_USER_AGENT 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '92.226.62.168' REMOTE_PORT '55102' REMOTE_USER 'http://pudo.org/' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
    Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; auth_tkt=""e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!""; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.identity repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) }}}" 1282306037000000 1294916148000000 464 task rgrp dread closed duplicate Request dgu db server access 1282306104000000 1282325194000000 465 requirement johnbywater ckan-v1.3 closed wontfix Require API key for all API requests. 1282308904000000 1294411534000000 466 requirement dread johnbywater ckan-v1.3 closed wontfix Name of HTTP header for API key shall be configurable 1282308995000000 1294835610000000 467 story johnbywater ckan-v1.3 closed duplicate Admin configures CKAN to expect API key in named HTTP header 1282310562000000 1294411681000000 468 task johnbywater johnbywater ckan-v1.2 closed fixed Default config file has commented out configuration variable 1282310659000000 1283250799000000 469 task johnbywater ckan-v1.2 closed fixed Unit test to check API key in correctly named HTTP header is accepted by server 1282310760000000 1282932682000000 470 task dread johnbywater ckan-v1.3 closed fixed "Adjust API key's HTTP header name on DGU ""live"" and ""test""" 1282310854000000 1292587187000000 471 story johnbywater ckan-v1.2 closed API user sends API key in correctly named header 1282312088000000 1283248729000000 472 story johnbywater ckan-v1.2 closed API user sends API key in incorrectly named header 1282312108000000 1282932740000000 473 story johnbywater ckan-v1.2 closed API user discovers correct header for sending API key 1282312203000000 1283248736000000 474 task dread dread closed wontfix Questions related to fi.ckan.net "Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list. 1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding. To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters. 2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en. 3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ? 4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN. 5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ? Best Regards, Henri" 1282312255000000 1294916760000000 475 task johnbywater ckan-v1.2 closed fixed Unit test to check API key in incorrectly named HTTP header is not accepted by server 1282312802000000 1282932705000000 476 requirement johnbywater ckan-v1.3 closed wontfix Daily database dumps shall be discoverable and downloadable 1282313433000000 1294411741000000 477 story johnbywater ckan-v1.3 closed duplicate Discover location of the daily database dumps 1282313788000000 1294411761000000 478 task johnbywater ckan-v1.2 closed fixed "Unit test for ""interface resource"" in API" We don't want /api to be an HTML page, but rather provide information about the API in a JSON structure. 1282315067000000 1283248713000000 479 task dread johnbywater ckan-v1.2 closed fixed Suggest DGU presents database dumps through front end 1282324365000000 1288004211000000 480 requirement thejimmyg johnbywater ckan-v1.4 closed fixed Catalogue service shall conform to specification "Common requirements for running CKAN behind a (e.g Wordpress or Drupal) front-end: 1. Unrestricted total read-only access to catalogue API for general public (e.g. voluntary organisation). - monitored by API key - not monitored by API key 2. Restricted total read-write access to catalogue API for authorized clients (e.g. front-end system, bulk upload clients). - restricted by CKAN access controller - restricted by HTTP Auth - restricted by IP address 3. Restricted total read-write access to catalogue Web UI for authorized users (e.g. site admins). - restricted by CKAN access controller - restricted by HTTP Auth 4. Restricted partial read-write access to catalogue Web UI for authorized users (e.g. group admins). - restricted by CKAN access controller - restricted by HTTP Auth [[Image(fig128130.png)]]" 1282422612000000 1300281551000000 481 story johnbywater ckan-v1.3 closed fixed Unrestricted total read-only access to catalogue API 1282422697000000 1294248359000000 482 story thejimmyg johnbywater ckan-backlog closed wontfix API Rate Limiting 1282422756000000 1298284158000000 483 story johnbywater closed invalid Restricted total read-write access to catalogue Web UI 1282422801000000 1294417216000000 484 story johnbywater closed invalid Restricted partial read-write access to catalogue Web UI 1282422858000000 1294417248000000 485 story johnbywater closed duplicate Performance beats QoS criteria 1282425219000000 1294411946000000 486 requirement johnbywater ckan-v1.3 closed duplicate Catalogue service shall notify and query SOLR service 1282425790000000 1291639321000000 487 story johnbywater closed duplicate Notify SOLR service of model events 1282425910000000 1291639404000000 488 requirement johnbywater closed wontfix Catalogue service shall notify RDF service 1282426021000000 1320930240000000 489 story wwaites johnbywater closed fixed Notify RDF service of model events 1282426051000000 1294416189000000 490 requirement johnbywater closed fixed The system shall support registering a harvest source via the CKAN API 1282426875000000 1286200804000000 491 story johnbywater ckan-v1.2 closed Get form for creating harvest source entity 1282427008000000 1284493173000000 492 story johnbywater ckan-v1.2 closed Submit harvest source create form response to the API 1282427042000000 1284493145000000 493 story johnbywater ckan-v1.2 closed Get harvest source entity 1282427083000000 1284493130000000 494 story johnbywater ckan-v1.2 closed Get form for updating remote metadata entity 1282427106000000 1285198894000000 495 story johnbywater ckan-v1.2 closed Put form for updating remote metadata entity 1282427150000000 1285198898000000 496 requirement wwaites johnbywater ckan-v1.4-sprint-2 closed fixed Provide a mechanism for exporting GEMINII metadata via a CSW interface 1282427307000000 1299164106000000 497 story johnbywater johnbywater closed duplicate "Respond to CSW ""GetRecords"" request" 1282427334000000 1294407718000000 498 story johnbywater closed invalid "Search packages within location ""bounding box""" 1282427412000000 1294412520000000 499 story thejimmyg johnbywater closed wontfix Registration and management of UKLII related code lists, controlled vocabularies and other forms of reference/static data 1282432571000000 1296593038000000 500 defect dread ckan-v1.2 closed duplicate Exception from diff "Investigate exception occured occasionally in last couple of days on ckan.net: {{{ WebApp Error: : 'NoneType' object has no attribute 'key' URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.revision:119 in diff << c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() >> diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff << display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] >> display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'zh-cn,zh-tw' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Baiduspider+(+http://www.baidu.com/search/spider.htm)' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '123.125.66.32' REMOTE_PORT '63767' REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
    Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
    \n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) }}}" 1282553033000000 1287747652000000 501 requirement pudo ckan-v1.2 closed duplicate Read-only maintenance mode CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache. 1282554617000000 1282724566000000 502 requirement dread rgrp ckan-v1.3 closed wontfix Data4nr uploading script shall remove deleted packages "Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed. " 1282659304000000 1292586466000000 503 task dread rgrp closed fixed Provide exemplar package relationships for some packages " 1. Find some relationships. 2. Add them via API 3. Notify DGU of completion and point to exemplars" 1282659631000000 1314031851000000 504 requirement dread rgrp ckan-v1.3 closed invalid The DGU fieldset shall conform to v3 of DGU metadata specification No need for requirement - closing in favour of #505 1282660246000000 1292586686000000 505 enhancement dread rgrp closed fixed DGU fieldset supports v3 of DGU metadata specification 1282660416000000 1298368280000000 506 task dread rgrp closed fixed Plan transition to DGU metadata-fields v3 1282660490000000 1287080989000000 507 story johnbywater closed fixed Upgrade a CKAN service 1282661646000000 1282909852000000 508 task dread dread ckan-v1.2 closed fixed Create branch ultrastable " * Create a branch named ultrastable that represents latest version of 1.0 ''as found on dgu'' * Going forward ultrastable will act as stable - 1 (and either ultrastable or stable will be on the rock-solid production systems such as dgu) * Do '''not''' deprecate the stable or metastable branch for the present * Ask for recommendations on mailing list about best practice for branching (e.g. versioned based versus a simple HEAD versus stable) and how this affects standard operations such as applying bugfixes. * For useful discussion of how to do backporting and applying bugfix patches see: http://www.python.org/dev/peps/pep-0374/#backport * NB: in general do '''not''' always have to apply in stable/ultrastable and merge upwards for bugfixes -- can use transplant extension as well. " 1282663663000000 1282908795000000 509 story dread dread closed invalid ONS data imported daily 1282665870000000 1291734435000000 510 enhancement dread dread closed fixed Setup cron for ons import 1282665939000000 1294138332000000 511 requirement dread ckan-v1.3 closed worksforme Better warnings and errors when using API 1282754677000000 1297075354000000 512 story dread closed duplicate User creates package via API with incorrect core fields specified 1282754750000000 1294917121000000 513 task dread closed wontfix Creating a package with API should warn if there are incorrect fields "When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK. " 1282754802000000 1302774329000000 514 defect dread ckan-v1.2 closed duplicate Inconsistent use of 'location' header in API "When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all. I've removed the info about the header in the docs in the meantime." 1282757357000000 1282757391000000 515 defect kindly dread ckan-v1.4-sprint-6 closed fixed Inconsistent use of 'location' header in API "When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all. I've removed the info about the header in the docs in the meantime." 1282757360000000 1302774268000000 516 defect dread ckan-v1.2 closed fixed Double-escaped text visible - revision page "At: http://ckan.net/revision you can see: ""Next »"" on pager This doesn't seem to be a problem with earlier (1.0.1) version." 1282829539000000 1288002933000000 517 requirement pudo pudo iati-1 closed fixed Standard domain model for entries "cf. Tariqs mail Updated version: https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0 == Original Version == {{{ Publishing Entity: Publishing Entity Type: (Donor, Recipient, Community Data..) Donor Country Activity period: Verification status: enumeration of statuses (checked, not checked etc) Resource links: to the actual IATI record Number of activities: ... Date record updated: Date data updated: License: Need this field even if it may be a standard license So naively mashing these together, we get something like: Identifier Title Abstract Donor Country Publisher Publisher Type Verification Status Department Contact Contact e-mail Licence Resource format Resource URL Resource ID Activity period Number of activities Date record updated Date data updated }}}" 1282893081000000 1284041561000000 518 requirement pudo pudo iati-1 closed fixed IATI WUI Editing * Create custom form 1282893167000000 1283896718000000 519 requirement pudo pudo iati-1 closed fixed IATI look and feel * Use what is on iatiregistry.org at the moment. 1282893224000000 1283536828000000 520 requirement pudo pudo iati-1 closed fixed Browseable web interface onto the data * e.g. find/browse by country and by publishing entity and by donor 1282893270000000 1283538080000000 521 requirement pudo pudo iati-1 closed fixed IATI Customized search 1282893401000000 1283897124000000 522 requirement pudo pudo iati-2 closed fixed Create publishing entity account " * Publishing entity emails system editors * System editor creates publishing entity account and emails publishing entity with details * Priority: 4/5" 1282893668000000 1287392999000000 523 requirement pudo pudo iati-1 closed fixed Read/write APIs for users and publishing entities "Read/write APIs for users and publishing entities to access and register data * Users have read access * Publishing entities have both (though can only write to 'their' records) * Priority: 4 for write (read access is 3)" 1282893743000000 1283897688000000 524 requirement pudo pudo iati-2 closed fixed Limit registering data to authenticated users either via API or manually. "Limit registering data to authenticated users either via API or manually. * Edits may only be made by authenticated entities * Priority: 5" 1282893904000000 1285594971000000 525 requirement pudo pudo iati-2 closed fixed Confirm and approve publishing entity account " * So some kind of confirmation or approval by someone more senior? * Alternative: just have an account request. * Priority: 3" 1282894040000000 1285595152000000 526 requirement pudo pudo closed wontfix Integrate validation tool into the registry "Assumes 8 is done: 8. Tool for validation an IATI XML record (referenced by a URL) for compliance with the IATI specification * Outside of scope of registry * Multiple validators with different levels of rigour" 1282894300000000 1340626152000000 527 story pudo rgrp closed fixed Create domain model and load sample data 1282899025000000 1293097531000000 528 task pudo rgrp closed fixed Setup demonstration instance at iati.ckan.net 1282899110000000 1283536475000000 529 task pudo rgrp closed fixed Obtain sample metadata 1282899161000000 1283536554000000 530 task pudo pudo closed fixed Map IATI attributes to CKAN metadata schema "From lopad: {{{ Publishing Entity (often/always? equal to the logged in user) * Domain logic: only can add entries for ""yourself"" Donor Country Activity period: period to which this record relates Verification status: enumeration of statuses (checked, not checked etc) Resource links: to the actual IATI record Number of activities: ... Date record updated: Date data updated: License: Need this field even if it may be a standard license }}}" 1282899187000000 1287392210000000 531 task pudo rgrp closed fixed Load sample IATI metadata " * Parse data * Load via API * Develop test for load via API (can be local or remote -- with paste webfixture may be easier)" 1282899244000000 1283536676000000 532 defect johnbywater johnbywater ckan-v1.2 closed invalid Just a test - please ignore 1282906842000000 1282907197000000 533 requirement johnbywater ckan-v1.3 closed invalid The performance of CKAN shall scale "CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses. How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware? Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases? Are there any other options for scaling the persistence mechanism?" 1282910499000000 1292957374000000 534 defect johnbywater dread ckan-v1.2 closed fixed buildbot config - database machine moved "Configure buildbot to use postgres db which has been moved to eu5.okfn.org pudo said: Could you perhaps also comment on http://knowledgeforge.net/okfn/tasks/ticket/466 with a hint on how to update this properly? I only edited /home/buildslave/okfn/full/build/buildandsmoke/buildandsmoke.ini which was the only occurence of the DB credentials I could find there." 1283165568000000 1288002762000000 535 defect dread ckan-v1.2 closed duplicate genshi error when logged into sl.ckan.net Genshi exception when rendering the page whilst logged in to sl.ckan.net. 1283165774000000 1283167040000000 536 task dread dread ckan-v1.2 closed fixed Extend base importer merging in resources "Multiple ONS records can go into one package e.g. Stats for Jan, Stats for Feb, Stats for Mar etc. We already identify an existing package by 'external_id'. Now need to find package by title and an extra field (department in ONS case) and allow it to merge in resource and tags. Alert if rest of meta data is different." 1283173313000000 1286376029000000 537 task wwaites wwaites closed duplicate Caching and Performance improvement "There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests. Introducing caching has to be done carefully and should be done in a ''standards compliant manner''. == General strategy == * Where possible, cache output within the pylons app (beaker). * Facilitate external caching in an end-user's web browser or a caching proxy * Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes). * When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data. == Tasks == These tasks should be broken into sub-tickets: * caching of parts of templates that are expensive to render (package list, tag list, group list) * caching of entire output using beaker particularly for API read operations. * need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches. * cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL * Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well * Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner == Future Work == * Investigate ckanclient library maintaining a local cache as a web browser would * Investigate using a CDN like Google Storage or Amazon for serving cached data." 1283184362000000 1311178929000000 538 task rgrp johnbywater closed wontfix Update CKAN Trac ticket report queries "Update report queries with more recent versions (below are report queries shipped with trac 0.11.7): NB: Report 1 has already been done. id | query ----+--------------------------------------------------------------------------------------------------- 1 | : SELECT p.value AS __color__, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY CAST(p.value AS integer), milestone, t.type, time : 2 | : SELECT p.value AS __color__, : version AS __group__, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time : 3 | : SELECT p.value AS __color__, : 'Milestone '||milestone AS __group__, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time : 4 | : : SELECT p.value AS __color__, : owner AS __group__, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time : 5 | : SELECT p.value AS __color__, : owner AS __group__, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : description AS _description_, : changetime AS _changetime, reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time : 6 | : SELECT p.value AS __color__, : t.milestone AS __group__, : (CASE status : WHEN 'closed' THEN 'color: #777; background: #ddd; border-color: #ccc;' : ELSE : (CASE owner WHEN $USER THEN 'font-weight: bold' END) : END) AS __style__, : id AS ticket, summary, component, status, : resolution,version, t.type AS type, priority, owner, : changetime AS modified, : time AS _time,reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), : (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC : 7 | : SELECT p.value AS __color__, : (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS __group__, : id AS ticket, summary, component, version, milestone, : t.type AS type, priority, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE t.status <> 'closed' AND owner = $USER : ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time : 8 | : SELECT p.value AS __color__, : (CASE owner : WHEN $USER THEN 'My Tickets' : ELSE 'Active Tickets' : END) AS __group__, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (COALESCE(owner, '') = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time " 1283190081000000 1294414537000000 539 requirement wwaites rgrp ckan-v1.4 closed invalid The system shall improve performance 1283242447000000 1303118486000000 540 story wwaites rgrp closed fixed Implement caching in a systematic manner 1283242705000000 1302694845000000 541 enhancement wwaites rgrp closed fixed Full-page caching even for logged in users "The simplest way to improve performance on specific pages is to implement full page caching. This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have ""You are logged in as: {username}"". Similarly we change the visibility of some material depending on the status of users: * Visibility of Edit or Authz link on packages and groups" 1283243139000000 1294924872000000 542 task wwaites rgrp closed fixed 'Show and Tell' page customization using cookie + js "To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/ In the first place we just need to apply this to: * You are logged in as * Package/group edit/authz tabs" 1283244432000000 1283278942000000 543 task wwaites rgrp closed duplicate Investigate partial page caching and edge-side includes "Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page. Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish. Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat." 1283244784000000 1311178918000000 544 requirement pudo ckan-v1.3 closed duplicate Backport facet browsing to CKAN 1.2 "This is in IATI, would be nice to have in generic CKAN. * http://bitbucket.org/okfn/iati/changeset/378431974c76" 1283267292000000 1291638966000000 545 requirement johnbywater johnbywater ckan-v1.2 closed The system shall support creating packages via the Form API 1283339310000000 1283458648000000 546 story johnbywater johnbywater ckan-v1.2 closed Get the package create form from the API 1283339414000000 1286896814000000 547 story johnbywater johnbywater ckan-v1.2 closed Submit package create form response to the API 1283339487000000 1283458645000000 548 task johnbywater johnbywater ckan-v1.2 closed fixed Functional test for getting package create form from the API 1283339598000000 1283351141000000 549 task johnbywater johnbywater ckan-v1.2 closed fixed Method on API controller to return package create form 1283339748000000 1283353945000000 550 task johnbywater johnbywater ckan-v1.2 closed fixed Routing configuration for package create form API resource 1283339785000000 1283351840000000 551 task johnbywater johnbywater ckan-v1.2 closed fixed Documentation for getting package create form from API 1283339845000000 1283354302000000 552 task johnbywater johnbywater ckan-v1.2 closed fixed Method on API controller to accept valid package create form submission 1283340009000000 1283436028000000 553 task johnbywater johnbywater ckan-v1.2 closed fixed "Functional test for submitting valid ""package create form"" response to the API" 1283340203000000 1283433851000000 554 task johnbywater johnbywater ckan-v1.2 closed fixed "Functional test for submitting invalid ""package create form"" response to the API" 1283340226000000 1283436393000000 555 task johnbywater johnbywater ckan-v1.2 closed fixed Documentation for submitting package create form submission to API 1283340290000000 1283354313000000 556 task johnbywater johnbywater ckan-v1.2 closed fixed Method on API controller to accept invalid package create form submission 1283340448000000 1283436605000000 557 defect pudo ckan-v1.3 closed invalid Remove siblings from link_to/from relationship display Displaying siblings is not needed for link_to/from. 1283542595000000 1297075159000000 558 defect pudo ckan-v1.2 closed fixed Sort extras, relationships by key/object alphabetically 1283542680000000 1283781883000000 559 defect pudo pudo ckan-v1.5 closed invalid Fix package lookup on group edit form "Lookup is needed to add a package but won't show up at some times. Originally reported by Anja Jentzsch, re questions. " 1283542776000000 1310127694000000 560 defect kindly rgrp ckan-v1.4 closed fixed Deleted packages showing up in tag lists "E.g. http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins). Also suggested that these are showing up in the API (courtesy of Anja) Cost: 2h" 1283700001000000 1297084192000000 561 defect pudo ckan-v1.2 closed duplicate Deleted packages are returned in the API Anja is reporting this, severe bug, I think. 1283775578000000 1283775711000000 562 defect pudo ckan-v1.2 closed fixed OpenID Login fails using Yahoo and Google "reported by LOD folks. subsidary bug: error messages from OpenID providers are not shown on the form return page. " 1283941119000000 1284063574000000 563 requirement thejimmyg johnbywater closed duplicate Support a minimal CSW server interface or export to GeoNetwork 1284033576000000 1296592472000000 564 requirement johnbywater ckan-v1.3 closed wontfix The system shall support location based search 1284033829000000 1294412714000000 565 requirement thejimmyg johnbywater ckan-v1.3 closed invalid The system shall support UKLP code lists 1284033864000000 1294417752000000 566 requirement johnbywater johnbywater ckan-v1.3 closed invalid The system shall support harvesting remote metadata 1284039619000000 1294232284000000 567 story johnbywater ckan-v1.2 closed Post new harvest job for given harvest source 1284039943000000 1284689374000000 568 story johnbywater johnbywater ckan-v1.3 closed Pull metadata documents from given harvest source entity 1284040131000000 1288038207000000 569 requirement thejimmyg johnbywater ckan-v1.4 closed fixed Harvested GEMINI document metadata validation 1284040221000000 1297347214000000 570 story johnbywater johnbywater ckan-v1.3 closed duplicate Validate metadata document against UKLP schematron 1284040256000000 1294407974000000 571 story johnbywater johnbywater ckan-v1.3 closed fixed Save metadata document and associate with harvest source entity 1284040495000000 1288038218000000 572 story johnbywater johnbywater ckan-v1.3 closed Write CKAN package from metadata document "The attributes we need to read in are being defined here: http://okfnpad.org/uklii The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes. Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft." 1284040784000000 1288038237000000 573 story johnbywater closed duplicate Add metadata entity to harvesting queue 1284045353000000 1284220987000000 574 story johnbywater closed duplicate Create UKLII package with attributes from remote metadata record 1284045805000000 1284222410000000 575 requirement johnbywater ckan-v1.2 closed fixed The system shall notify remote metadata harvesting errors 1284047959000000 1288012749000000 576 defect sebbacon wwaites ckan-v1.3 closed fixed CKAN Requires Old Version of SQLAlchemy "Requires 0.4. 0.5 is a maintenance branch, 0.6 is current. VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why. It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well" 1284141573000000 1294753848000000 577 story johnbywater ckan-v1.2 closed Get remote metadata harvest job errors 1284209381000000 1285348049000000 578 story johnbywater ckan-v1.2 closed Get remote metadata harvest job 1284209407000000 1284493202000000 579 story johnbywater ckan-v1.2 closed Delete remote metadata harvest job 1284209440000000 1285260356000000 580 story johnbywater closed duplicate Write (create or update) CKAN package for metadata document 1284210730000000 1284223068000000 581 task rgrp johnbywater ckan-v1.2 closed fixed Determine package name from metadata document Need to establish procedure for automatically generating package names from metadata documents. 1284210874000000 1284689274000000 582 task johnbywater ckan-v1.2 closed fixed Domain object class for harvest source 1284211104000000 1284261402000000 583 task johnbywater ckan-v1.2 closed fixed Define fieldset object for harvest source forms 1284211169000000 1284261424000000 584 task johnbywater ckan-v1.2 closed fixed Migration script to add harvest source table 1284211204000000 1284483760000000 585 task johnbywater ckan-v1.2 closed fixed Method on API controller to return harvest source create form 1284211302000000 1284261448000000 586 task johnbywater ckan-v1.2 closed fixed Routing configuration for harvest source create form API resource 1284211336000000 1284261447000000 587 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest source create form from API 1284211364000000 1284492381000000 588 task johnbywater ckan-v1.2 closed fixed Method on API controller to accept valid harvest source create form submission 1284211575000000 1284318555000000 589 task johnbywater ckan-v1.2 closed fixed Method on API controller to reject invalid harvest source create form submission 1284211620000000 1284318561000000 590 task johnbywater ckan-v1.2 closed fixed Documentation for submitting harvest source create form submission to API 1284211659000000 1284492397000000 591 task johnbywater ckan-v1.2 closed fixed Routing configuration for harvest source entity API resource 1284212082000000 1284342925000000 592 task johnbywater ckan-v1.2 closed fixed Methods to return harvest source entity as JSON 1284212338000000 1284342904000000 593 task johnbywater ckan-v1.2 closed fixed Method on API controller to return remote metadata edit form 1284212498000000 1285198853000000 594 task johnbywater ckan-v1.2 closed fixed Routing configuration for remote metadata edit form API resource 1284212532000000 1285198859000000 595 task johnbywater ckan-v1.2 closed fixed Documentation for getting remote metadata edit form from API 1284212572000000 1285198863000000 596 task johnbywater ckan-v1.2 closed fixed Method on API controller to accept valid remote metadata edit form submission 1284212662000000 1285198874000000 597 task johnbywater ckan-v1.2 closed fixed Method on API controller to reject invalid remote metadata edit form submission 1284212689000000 1285198885000000 598 story johnbywater ckan-v1.2 closed List remote metadata entities for given publisher 1284213730000000 1286200795000000 599 task johnbywater ckan-v1.2 closed fixed Routing configuration for listing remote metadata entities for a given publisher 1284213814000000 1284347290000000 600 task johnbywater ckan-v1.2 closed fixed Method on API controller for listing remote metadata entities for a given publisher 1284213854000000 1284347296000000 601 task johnbywater ckan-v1.2 closed fixed Documentation for listing remote metadata entities for a given publisher via API 1284213928000000 1284493045000000 602 task johnbywater ckan-v1.2 closed fixed Method on API controller to accept valid harvest job 1284217587000000 1284336801000000 603 task johnbywater ckan-v1.2 closed fixed Method on API controller to reject invalid harvest job 1284217624000000 1284338918000000 604 task johnbywater ckan-v1.2 closed fixed Domain object class for harvest job 1284217689000000 1284326615000000 605 task johnbywater ckan-v1.2 closed fixed Routing configuration for harvest job register 1284217966000000 1284336788000000 606 task johnbywater ckan-v1.2 closed fixed Methods to return harvest job as JSON 1284218102000000 1284344127000000 607 task johnbywater ckan-v1.2 closed fixed Routing configuration for harvest job entity 1284218136000000 1284344113000000 608 task johnbywater ckan-v1.2 closed fixed Routing configuration for register of harvest jobs with error status 1284218378000000 1285347281000000 609 task johnbywater ckan-v1.2 closed fixed Methods to return harvest jobs with error status as JSON 1284218437000000 1285347285000000 610 task johnbywater johnbywater ckan-v1.2 closed fixed Load local copy of metadata document 1284218592000000 1288041706000000 611 task johnbywater johnbywater ckan-v1.3 closed invalid Load local copy of XML schema for metadata document 1284218639000000 1294408239000000 612 task johnbywater johnbywater ckan-v1.3 closed duplicate Check given XML schema validates given metadata document 1284218750000000 1294408188000000 613 task johnbywater johnbywater closed Store result of schema validation check on metadata document object 1284218828000000 1286798466000000 614 task johnbywater johnbywater closed invalid Acquire local copy of XML schema for remote metadata documents 1284219077000000 1288038898000000 615 task johnbywater johnbywater closed invalid Acquire local implementation of UKLP schematron "We ""just"" need a schematron.validate(metadata) method. :-)" 1284219174000000 1288038926000000 616 task johnbywater johnbywater closed invalid Store result of schematron validation check on metadata document object 1284219246000000 1288038907000000 617 task johnbywater johnbywater ckan-v1.3 closed duplicate Check UKLP schematron validates given metadata document 1284219298000000 1294408164000000 618 task johnbywater ckan-v1.2 closed fixed Domain object class for metadata document 1284219669000000 1284598039000000 619 task johnbywater ckan-v1.2 closed fixed Get HTTP response for given resource locator 1284220327000000 1284675079000000 620 task johnbywater ckan-v1.2 closed fixed Identify remote metadata entity as metadata document, WAF, or CSW 1284220527000000 1284689400000000 621 task johnbywater johnbywater ckan-v1.3 closed fixed Metadata document links extracted from WAF response 1284220707000000 1288038326000000 622 task johnbywater johnbywater ckan-v1.2 closed fixed Metadata document identifiers extracted from CSW GetRecords responses 1284220740000000 1287510408000000 623 task johnbywater johnbywater ckan-v1.2 closed fixed CSW GetRecords request for all identifiers (with CSW authentication) 1284220777000000 1287507837000000 624 task johnbywater ckan-v1.2 closed fixed Read attribute 'title' from local copy of metadata document 1284221083000000 1284675746000000 625 task johnbywater ckan-v1.2 closed fixed Read attribute 'abstract' from local copy of metadata document 1284221101000000 1284675753000000 626 task johnbywater ckan-v1.2 closed fixed Read attribute 'point of contact' from local copy of metadata document 1284221117000000 1285443038000000 627 task johnbywater ckan-v1.2 closed fixed Read attribute 'guid' from local copy of metadata document 1284221134000000 1284598028000000 628 task johnbywater ckan-v1.2 closed fixed Read attribute 'bounding-box' from local copy of metadata document 1284221156000000 1285443047000000 629 task johnbywater ckan-v1.2 closed fixed Read attribute 'keywords' from local copy of metadata document 1284221191000000 1285443055000000 630 task johnbywater ckan-v1.2 closed fixed Read attribute 'use constraints' from local copy of metadata document 1284221210000000 1285443069000000 631 task johnbywater ckan-v1.2 closed fixed Read attribute 'resource-locator' from local copy of metadata document 1284221223000000 1284675741000000 632 task johnbywater ckan-v1.2 closed fixed Read attribute 'topic category' from local copy of metadata document 1284221246000000 1285443078000000 633 task johnbywater ckan-v1.2 closed fixed Read attribute '' from local copy of metadata document 1284221272000000 1285443112000000 634 task johnbywater ckan-v1.2 closed fixed Read attribute '' from local copy of metadata document 1284221284000000 1285443115000000 635 task johnbywater ckan-v1.2 closed fixed Read attribute '' from local copy of metadata document 1284221305000000 1285443119000000 636 task johnbywater ckan-v1.2 closed fixed Method on API controller to delete given harvest job 1284221790000000 1285260342000000 637 task johnbywater closed invalid Read UKLII package data from given metadata document 1284222536000000 1284223104000000 638 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest source entity from API 1284225805000000 1284491185000000 639 task johnbywater ckan-v1.2 closed fixed Documentation for posting harvest job to API 1284226008000000 1284491239000000 640 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest job entity from API 1284226036000000 1284491233000000 641 task johnbywater ckan-v1.2 closed fixed Documentation for getting harvest job errors from API 1284226049000000 1285348033000000 642 task johnbywater ckan-v1.2 closed fixed Migration script to add harvest job table 1284226154000000 1284483747000000 643 task johnbywater ckan-v1.2 closed fixed Migration script to add metadata document table 1284226255000000 1284675721000000 644 task johnbywater ckan-v1.2 closed fixed Documentation for deleting harvest job entity 1284226402000000 1285260346000000 645 task johnbywater johnbywater ckan-v1.2 closed fixed Domain object and migration script to associate packages and harvester objects 1284227152000000 1287774090000000 646 requirement johnbywater closed wontfix Web UI shall support registering harvest sources 1284740303000000 1294413500000000 647 requirement pudo pudo ckan-v1.3 closed fixed Authorization Groups for group-based access control allow group-level authorization on all system entities 1284889418000000 1287391981000000 648 requirement pudo ckan-v1.3 closed fixed Enable lock-down of package creation " - copy exisiting tests, modify authz in setUp, adapt and extend tests * Problem: default_role_actions is read by init_db * Solution: nuke db after monkey-patching * role = model.Role('Reader'), del role.actions[...] self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions) mauthz.default_role_actions.remove((Role.READER, Action.CREATE)) #raise Exception(mauthz.default_role_actions) model.Session.remove() model.repo.rebuild_db() 3. Start from the functional * Both the api controllers and wui controllers to do * tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. http://bitbucket.org/pudo/ckan-authz2 is_authorized(user, Action.Create, model.Package) -> Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards PackageRole in all queries. Since for class-level role assignments there never is a PackageRole join table entry, this will never return any results. * cf branch ""classes"": http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66 * No possibility of including ""instance"" column in inheritance decision. * Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling. * have a ckan install that would not allow visitors to either list packages or list groups * two paths: create the listing, but for each group/pkg decide that you cannot show this * lock down the whole page (/package/list) - this is class-based, not object-based is_authorized(user, Action.Package_Create, model.System()) is_authorized(user, Action.Group_Create, model.System()) [Separating Package and Group roles may be useful going forward: PackageEditor, GroupEditor etc] 4. Find a standard way to lock down classes * possibly add default rows in user_object_role * introduce lock-down into controllers: group new, package new, REST equivalents, * confirm tests" 1284889519000000 1284889656000000 649 story pudo closed fixed Introduce Authorization Groups in model "Authorization Groups: * Rename UserObjectRole to SubjectObjectRole * Create database migration to do the rename in DB * Make sure all the tests are passing again (Doing this first seems more stable since we first want to ""re-gain"" exisiting functionality) * Create authz.py and model tests for authorization_group-based auth * tests: functional: create fixtures, do actual requests lib-tests: test for membership addition, removal, group-right additions affecting all members, group-right removal, group rights affecting non-members. # * Adapt model to get these passing * Integration into existing controllers should be negligible but may want ot have tests - do not want that. would mean design is broken * Need to create a new controller for editing authz groups (will have same permissions as any other object -- Reader, Editor, Administrator) " 1284889884000000 1284889983000000 650 story pudo pudo ckan-v1.2 closed fixed WUI editing for Authorization Groups " * Copy & adapt Group controller tests to suit AuthorizationGroups * Copy & adapt Group controllers and templates for Authorization Groups" 1284890073000000 1287391932000000 651 requirement pudo closed fixed Improve javascript helper for package/user addition on groups/authzgroups This needs to be abstracted and we still need to be able to remove items before saving. 1284972211000000 1311171104000000 652 requirement pudo ckan-v1.2 closed fixed Allow for addition and removal of groups on package/edit Much more natural place to do this, even at the risk of overloading the package edit form. 1284998134000000 1284998201000000 654 bug johnbywater ckan-v1.2 closed Harvest sources and jobs should return 404 when missing (not 500) 1285170717000000 1285254097000000 655 story johnbywater ckan-v1.2 closed Return status code 404 when harvest source is not found 1285252399000000 1285254088000000 656 story johnbywater ckan-v1.2 closed Return status code 404 when harvesting job is not found 1285252429000000 1285254084000000 657 task johnbywater ckan-v1.2 closed fixed Fix harvesting job entity get method to default to returning 'None' rather than raising an exception 1285252536000000 1285254068000000 658 task johnbywater ckan-v1.2 closed fixed Fix harvest source entity get method to default to returning 'None' rather than raising an exception 1285252579000000 1285254072000000 659 enhancement nils.toedtmann dread closed fixed Check CKAN instance works automatically Auto way to check web and API interface of a CKAN instance basically works. Several gotchas can be quickly determined, such as logging in, search not working. Needs to be configurable per site basis. 1285348333000000 1311183031000000 660 requirement dread closed invalid Check CKAN instance works As an admin I want to check a CKAN instance works having just upgraded it or configured it. 1285348463000000 1311183115000000 661 task dread ckan-v1.3 closed fixed Create CKAN site tester "From Rufus: {{{ 'blackbox' testing command (read-only only would be fine) -- very useful for upgrade on sunday On blackbox thinking of something like: paster test-site http://hmg.ckan.net {package-name} Or (maybe better): paster test-site --site-spec {myconfig.json} my-config is a json file with info for testing (could overload --config rather than site-spec): site-url: http://hmg.ckan.net/ package-name: package-title: ...etc ... Then: Test front-page Test /revision/ Test /package/{page} -- check title Test /package/history Test /api/rest/package/ ... Test /api/search/package ... etc }}}" 1285348558000000 1288271266000000 662 defect sebbacon johnbywater ckan-v1.4 closed fixed Can't put entity that is returned by posting to package register "It's because Package carries several out-of-band values, which are snagged on the way back out. Entity get response also can't be posted. However, post response can be re-posted (because it isn't the same as the register-post/entity-get responses. An issue for CKAN too. Sub-ticket of #961 (form, validation, model sync meta-ticket) and depends on that work." 1285410546000000 1301076463000000 663 defect kindly dread ckan-v1.4 closed fixed Duplicate sibling relationships displayed "Looking at: http://ckan.net/package/rkb-explorer-acm 'rkb-explorer-resex' gets display multiple times and shouldn't" 1285576369000000 1298913603000000 664 defect kindly pudo ckan-v1.4 closed fixed Sibling relationships shown for deleted packages 1285576600000000 1300371645000000 665 requirement johnbywater johnbywater ckan-v1.3 closed duplicate The system shall support withdrawing a harvested dataset or service from publication "Discussion between John and Peter: > Given we can identify a document, does the disappearance of a document > from a registered source imply the disappearance of the metadata (such > that we delete packages once the documents disappear from the registered > source)? > I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as ""withdraw a dataset or service from publication"" (part of UCD03 Maintain resources). " 1285588250000000 1297268097000000 666 requirement johnbywater ckan-v1.2 closed fixed The system shall support updating packages from new versions of previously harvested dataset or service metadata 1285590374000000 1288013526000000 667 defect thejimmyg dread ckan-v1.3 closed wontfix API is slow "See message and script: http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html 1500 requests shouldn't take hours." 1285885228000000 1297069121000000 668 defect thejimmyg Colin Calnan closed invalid Caching issues on API v1 "It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues. Steps to test: 1. Modify a dataset on datadotgc.ca, redirects to CKAN 2. On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100 3. You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present 4. Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present. 5. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either." 1285953542000000 1311176649000000 669 task pudo pudo closed fixed Refactor Solr to become a generic worker 1286189594000000 1293057161000000 670 task pudo pudo closed fixed HowTo: Creating a sysadmin 1286190347000000 1293056844000000 671 task pudo pudo closed fixed Implement and show user names in WUI 1286190454000000 1293056852000000 672 enhancement pudo pudo ckan-v1.3-sprint-1 closed fixed Facets in CKAN search results "This has been done in IATI ckan but should be ported over to CKAN main and put into the UI in a nice fashion. IATI: http://bitbucket.org/okfn/iati/changeset/378431974c76 (related to #520) Requires SOLR search backend (for postgres should fall back gracefully -- just do not show them)." 1286190736000000 1291715338000000 673 task johnbywater closed Construct and send CSW GetRecordById request 1286214712000000 1286786758000000 674 task johnbywater closed Extract metadata documents from CSW GetRecordById 1286214780000000 1286787032000000 675 task johnbywater johnbywater ckan-v1.3 closed fixed Draft specification of CSW harvesting behaviour with guide for publishing agents 1286214877000000 1288038316000000 676 requirement thejimmyg johnbywater ckan-v1.3 closed wontfix The system shall support browsing to correct version of the CKAN documentation for a given service 1286276136000000 1295869107000000 677 task dread dread ckan-v1.3 closed fixed License change script "* Uses API to change licenses of all packages in a CKAN instance to a particular one. * Run it on DGU." 1286384756000000 1292587315000000 678 task dread dread ckan-v1.2 closed fixed Form for DGU metadata v3 " * Create form * Deploy on test * Deploy on live " 1286385712000000 1286979133000000 679 task thejimmyg dread ckan-v1.3 closed fixed Improve standard package edit form " * Add help text, drawing from: * ca.ckan.net * http://wiki.okfn.org/ckan/doc/package * http://wiki.okfn.org/ckan/doc/faq * Make longer text revealed on click" 1286386050000000 1294166120000000 680 task dread dread closed fixed Prepare for DGU tech meeting 1286386074000000 1287141809000000 681 task dread dread ckan-v1.3 closed fixed Department field bug https://trac.dataco.coi.gov.uk/projects/datagov/ticket/613 1286386685000000 1290507180000000 682 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Add ckanclient tests to buildbot Some work done on this to make tests run independently. 1286386769000000 1297358266000000 683 story dread dread closed fixed metastable, stable & ultrastable smoketests run by buildbot 1286388134000000 1287142289000000 684 task dread dread closed fixed metastable tested on buildbot 1286388181000000 1286452710000000 685 task dread dread closed fixed stable tested on buildbot 1286388206000000 1287997008000000 686 task dread dread closed wontfix ultrastable tested on buildbot 1286388225000000 1287997047000000 687 task dread dread ckan-v1.2 closed fixed Release ckanclient 0.4 "Done. Added release process here: wiki:ReleaseProcess" 1286452098000000 1286452650000000 688 task johnbywater johnbywater ckan-v1.2 closed Example GeoNetworks service for CSW development 1286786893000000 1286980827000000 689 task johnbywater johnbywater ckan-v1.3 closed fixed Updated package for updated metadata document 1286789134000000 1288038308000000 690 requirement johnbywater johnbywater ckan-v1.2 closed invalid The system shall support Gemini2 service records Wish to fold service records into package resources. Trouble is that service records don't necessarily refer to a data record, so should this be a new package? 1286822613000000 1288013718000000 691 requirement thejimmyg johnbywater ckan-backlog closed duplicate Package Relationships 1286822735000000 1295610145000000 692 requirement johnbywater johnbywater ckan-v1.3 closed invalid The system shall support presenting map thumbnail for bounding box. "To do this for the UKLP, the Ordnance Survey would need to add to their map service an operation to return image for a given bounding box, and COI would need to present the image on the search result page. Otherwise, it would be possible to use Google." 1286823049000000 1294247841000000 693 requirement wwaites ckan-v1.2 closed fixed Support ETag in cache decorator ... and make cache-control header optional 1286831051000000 1286831188000000 694 bug wwaites dread closed fixed No postgres tools for current version Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them. 1286977052000000 1287087916000000 695 bug pudo dread closed fixed Search indexing broken on ckan.net "e.g. searching for 'buddhist' or 'sanskrit', you don't get this newly created package: http://ckan.net/package/digitalsanskritbuddhistcanon" 1286991201000000 1287766973000000 696 task johnbywater johnbywater ckan-v1.2 closed fixed Decoupled generation of package group choices from validation of package group selection 1287066070000000 1287392206000000 697 requirement pudo johnbywater iati-3 closed fixed The system shall indicate which package groups can be removed when editing a package This seems to require doing something else with such a group's 'delete' button, such as showing it greyed, or perhaps not showing it at all. 1287066103000000 1291644457000000 698 task Stiivi thejimmyg ckan-v1.3-sprint-1 closed fixed CKAN Data API v1 "This proposal is to discuss adding a new API for proxying certain spreadsheet data via JSON-P to make it possible to build simple browser apps directly off the API. See the attached proposal for information." 1287073433000000 1293649815000000 699 defect pudo TimDavies closed fixed Paging search results re-sets Open Only and Downloadable Only flags When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings. 1287143698000000 1296499061000000 700 bug pudo dread iati-3 closed fixed Groups in package form "Editing groups in forms doesn't work for me, with latest code from this morning: 1. Clean db 2. paster create-test-data 3. paster sysadmin create http://davidread.myopenid.com/ 4. paster serve development.ini 5. In browser, log in to CKAN 6. Create new package: name=abc group=Roger's books 7. [Preview] - yes group appears 8. [Save] - shows package and group hasn't appeared ERROR 9. Check http://127.0.0.1:5000/api/rest/package/abc and reedit package it also doesn't appear here either. " 1287394476000000 1290000656000000 701 requirement pudo johnbywater iati-3 closed fixed The system shall support changing package groups when editing a package 1287403776000000 1291644605000000 702 requirement johnbywater johnbywater ckan-v1.2 closed duplicate The system shall support changing package groups when editing a package 1287403778000000 1287403850000000 703 story pudo johnbywater iati-3 closed fixed Add package to package group using the package form fieldset 1287403920000000 1289909096000000 704 story pudo johnbywater iati-3 closed fixed Remove package from package group using the packge form fieldset 1287403961000000 1289910491000000 705 story pudo johnbywater iati-3 closed fixed Show package groups on the package read page 1287404004000000 1291644479000000 706 task pudo johnbywater iati-3 closed fixed Test for removing package from package group using the package form fieldset 1287404073000000 1289909028000000 707 task pudo johnbywater iati-3 closed fixed Test for adding package to package group using the package form fieldset 1287404126000000 1289910513000000 708 task pudo johnbywater iati-3 closed fixed Test for showing package groups on package read page 1287404169000000 1289910506000000 709 task johnbywater johnbywater ckan-v1.2 closed fixed Metadata document extracted from CSW GetRecordById response 1287432440000000 1287684969000000 710 task johnbywater johnbywater ckan-v1.2 closed fixed CSW GetRecordById request for given identifier 1287432675000000 1287507854000000 711 requirement thejimmyg johnbywater ckan-v1.3 closed fixed Packages shall support both UK government and location information with a common set of attributes "Previously, a package type attribute was intended to distinguish between package data that came from government depts. and publishers location information (GEMINI). Now there is an intention use a common set of attributes to support data from either source. Hence, there currently a v.3 of the government attributes, a document about extracting the attributes from Gemini document using XPath, and a comparison documents mapping between governemtn attributes and Gemini attributes. That means there are potentially four different list of attributes. I'm not sure how to go about reviewing the lists and converging any differences. But we do want to end up with a list of names, a description of their purpose, and then perhaps the XPath which would reveal the value from a Gemini doc?" 1287580866000000 1296592589000000 712 story johnbywater johnbywater ckan-v1.3 closed invalid "Model government data package with ""v0.4"" of the attributes" "Where said ""v0.4"" would be the result of converging four lists mentioned in requirement #711." 1287581039000000 1294408364000000 713 task johnbywater johnbywater ckan-v1.3 closed fixed Review four definitions of DGU package attributes 1287581158000000 1288260915000000 714 bug johnbywater closed fixed DGU package form shall have a read only field for ??? attribute So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission. 1287581408000000 1291733788000000 715 defect rgrp johnbywater ckan-v1.4-sprint-2 closed fixed The group list page shall show the group title *or* group name "Symptom #1: Groups without titles don't get a clickable link, because there is no text. Defaulting to the group name would solve this trouble. Symptom #2: Package groups drop-down list show's 'None' when a group doesn't have a title. Therefore, we need to centralize presentation of group title, so it defers to group name. And we need tests. Also please sort groups by title on group list page." 1287583089000000 1297796784000000 716 requirement pudo pudo iati-3 closed fixed Visual improvments to IATI CKAN 1287583741000000 1290000673000000 717 bug pudo pudo closed fixed "Fix and validate ""setup_default_user_roles"" in IATI" 1287583892000000 1290005099000000 718 story pudo pudo iati-3 closed fixed Show recent activities on home page 1287583989000000 1288459823000000 719 story pudo pudo iati-3 closed fixed Improve overall layout of IATI ... probably by adapting aidtransparency.net 1287584092000000 1289910497000000 720 story pudo pudo iati-3 closed wontfix Include CMS pages in IATI registry "Perhaps using deliverance? * Move About to the right of the nav tabs and make it an about IATI page * Make a Help / FAQ section linked of main menu * Front page more dynamic - latest data registered and news (microblog thing) * make the front page blurb noticable / shorter etc. * link to iatistandard.org from the front page blurb" 1287584189000000 1288459344000000 721 story pudo pudo iati-3 closed fixed Make the table preview a link to a new page. 1287584244000000 1288459534000000 722 story pudo pudo iati-3 closed fixed Further adapt l10n for IATI " * Change ""Publishing Entity"" to ""Publisher""" 1287584314000000 1288459703000000 723 story pudo pudo iati-3 closed fixed Hide action links pointing to unavailable actions (package create) 1287584448000000 1288459610000000 724 requirement pudo pudo iati-3 closed fixed Improve publisher workflow 1287584548000000 1289835202000000 725 story pudo pudo iati-3 closed fixed Change to allow anyone (logged in) to create a publisher "With a pending state set (""unapproved"") " 1287584630000000 1289296038000000 726 story pudo pudo iati-3 closed fixed Send message to sysadmins to approve new publishers 1287584670000000 1289835192000000 727 story pudo pudo iati-3 closed fixed Sysadmin can approve a publisher 1287584754000000 1289835196000000 729 bug pudo closed fixed Assets to be loaded from assets.okfn.org, not m.okfn.org Move from hetzner box to s3. 1287736519000000 1288012854000000 730 task rgrp closed duplicate Back up package data from all CKAN packages to storage.ckan.net "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance?" 1287737109000000 1291139609000000 731 bug dread dread ckan-v1.3 closed worksforme Geo coverage field losses in Form API Sometimes editing a package via api on dgu results in some countries being lost in geo coverage. 1287738539000000 1288038226000000 732 task dread dread ckan-v1.3 closed fixed Reproduce and investigate (geo coverage issue) 1287738761000000 1288038335000000 733 task dread dread ckan-v1.3 closed invalid Fix problem (geo coverage issue) 1287738764000000 1288038341000000 734 task dread dread ckan-v1.3 closed fixed Investigate department issue (ONS import) "Creation of duplicate packages named with trailing underscore ww 22/9/10: >> Need to ... >> merge some packages (the ""find packages by attributes"" >> is not 100% reliable because of inconsistent data >> already in there). The ""find by tags"" algorithm uses (title, department). Unfortunately some of the data already there has no department. This causes it to make a new package (with trailing underscore). dread 24/9/10: I've done a test for the missing department issue and I can't reproduce the problem. This may be due to my general tidy up and getting tests passing using the latest ckanclient. Anyway, can you check over my test to see if this is what you mean?: dgu/ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadMissingDept ww 20/10/10: you should uncover it if you run against (a copy of) the dgu database." 1287747273000000 1292587603000000 735 task dread dread closed fixed Investigate 'deduping' (ONS import) "ww 22/9/10: (discussion of ticket:509) This leads us to deduping which should be the next order of business oin this thread... ww 20/10/10: if a dataset is missing a department and the ons script makes a new version of it with a department we now have two reccords for the same dataset. these should be manually merged - need a merge tool where you can tell it that two datasets are the same." 1287748786000000 1292957248000000 736 requirement amercader johnbywater ckan-v1.4-sprint-7 closed fixed URL validation for CSW and WMS links "Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return . Both cases would be resolved by checking capabilities (with !CswClient's check_get_capabilities() method)." 1287749416000000 1304963598000000 738 requirement kindly johnbywater ckan-v1.4-sprint-4 closed fixed Store revisions of harvested documents 1287775892000000 1299761436000000 739 requirement thejimmyg johnbywater ckan-v1.3 closed fixed The system shall support retrieving a copy of the harvested metadata for a given package ...obviously, not all packages will have a harvested metadata document. 1287779710000000 1296592842000000 740 requirement thejimmyg johnbywater closed duplicate Get copy of harvested metadata for a given package 1287779799000000 1296592889000000 741 enhancement ollyc ckan-v1.2 closed fixed Extension / Plugin system for CKAN "Create plugin system using Specification in CEP-003 (browser:doc/cep/cep-003.txt) and " 1287996890000000 1291989194000000 742 task dread dread ckan-v1.3 closed fixed Release ckan 1.2 to metastable Need to make sure #700 is fixed. 1288002397000000 1289824163000000 743 story johnbywater closed wontfix Create harvest source registration via Web UI 1288012860000000 1294412807000000 744 story johnbywater closed wontfix Update harvest source registration via Web UI 1288012960000000 1294412813000000 745 requirement johnbywater ckan-v1.3 closed wontfix Web UI shall accept requests for harvesting jobs 1288013148000000 1294412820000000 746 story johnbywater closed wontfix Request new harvesting job via Web UI 1288013198000000 1294412827000000 747 story johnbywater closed wontfix Track harvesting jobs via Web UI 1288013217000000 1294412834000000 748 story johnbywater closed duplicate Link new sample package to previous sample package of continuous series 1288013849000000 1294412976000000 749 story johnbywater closed duplicate Fold up continuous series in search results behind newest sample package 1288014002000000 1294412986000000 750 enhancement thejimmyg johnbywater closed duplicate Get CSW records modified since given time 1288014402000000 1296592940000000 751 story johnbywater closed duplicate Get harvested document for a given package 1288014518000000 1288014616000000 752 task johnbywater johnbywater ckan-v1.3 closed duplicate "Change package attribute names used by Gemini harvesting to DGU ""v.4""" 1288039205000000 1294408472000000 753 requirement johnbywater johnbywater ckan-v1.3 closed wontfix The system shall support filtered harvest of CSW sources 1288040290000000 1294247922000000 754 enhancement wwaites johnbywater ckan-v1.4-sprint-5 closed wontfix Filtering for harvesting sysem This has once again been ruled out for UKLP for the timebeing. 1288040469000000 1303118020000000 755 task johnbywater johnbywater ckan-v1.3 closed duplicate Add filter attribute to harvest source entity 1288040506000000 1294408632000000 756 task johnbywater johnbywater ckan-v1.3 closed duplicate Add filter field to harvest source form 1288040545000000 1294408642000000 757 task thejimmyg johnbywater ckan-v1.3 closed duplicate Create migration script to add harvest source filter attribute to existing tables 1288040584000000 1296593448000000 758 task johnbywater johnbywater ckan-v1.3 closed duplicate Change API documentation to indicate harvest source entity has filter attribute 1288040643000000 1294409053000000 759 story johnbywater johnbywater ckan-v1.3 closed duplicate Construct and send filtered CSW GetRecords request 1288040753000000 1294408652000000 760 task johnbywater johnbywater ckan-v1.3 closed duplicate "Change ""CSW Get Records"" request class to accept and used given CSW filter" 1288040993000000 1294409111000000 761 story dread closed invalid Read-only CKAN for maintenance Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. 1288090334000000 1288091977000000 762 story dread closed invalid Permanently Read-Only CKAN instance "A CKAN is used just for distributing metadata. Updates may still arrive through direct db manipulation, e.g.: * another (but writable) CKAN instance is connected to the same db * restoring database dumps from another CKAN db " 1288090578000000 1288091982000000 764 enhancement thejimmyg dread ckan-v1.5 closed duplicate Read-only CKAN Web UI "Whilse using CKAN web interface, you are not tempted to edit stuff: * You know at all times this CKAN is read-only * All editing facilities are still seen but greyed-out with an indication why it is." 1288091619000000 1310133334000000 766 requirement wwaites johnbywater ckan-v1.3 closed duplicate The system shall support harvesting metadata from ArcGIS "Perhaps 80% of the users of the UKLII may be using ArcGIS." 1288177933000000 1296593519000000 767 task thejimmyg johnbywater ckan-v1.4 closed wontfix Harvest metadata records from ArcGIS CSW API 1288178056000000 1304936413000000 768 task johnbywater johnbywater ckan-v1.3 closed duplicate Obtain example ArcGIS CSW server 1288178134000000 1294409308000000 769 task johnbywater johnbywater ckan-v1.3 closed duplicate Tests for harvesting metadata from ArcGIS CSW API 1288178179000000 1294409318000000 770 defect thejimmyg johnbywater ckan-v1.3 closed wontfix Improve the paster harvester help command "Issue 1. Help command without arguments fails: $paster help --plugin ckan --config development.ini gives: EITHER Command help not recognized OR paster help: error: no such option: --plugin Issue 2: Harvester command with arguments fails. $ paster harvester Traceback (most recent call last): ... cmd = self.args[0] IndexError: list index out of range " 1288190220000000 1296593925000000 771 story johnbywater ckan-v1.3 closed worksforme Run CLI help command without arguments Should return something sensible (currently reports an error). 1288190318000000 1294413256000000 772 story johnbywater ckan-v1.3 closed duplicate Run CLI harvester command without arguments Should return help for the harvester (currently crashes). 1288190373000000 1294413313000000 773 task johnbywater ckan-v1.3 closed duplicate Fix CLI help command to return CLI help message 1288190475000000 1294413400000000 774 task johnbywater ckan-v1.3 closed duplicate Fix CLI harvester command to return harvester help message 1288190522000000 1294413412000000 775 task pudo dread closed fixed Search warning "We're getting this warning a great deal on live servers. Is these really a sign of the system not operating correctly or can we reduce the level to an INFO? e.g. on hmg.ckan.net: {{{ 2010-10-29 17:12:08,262 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,333 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,375 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,406 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,480 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,613 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position }}}" 1288372692000000 1295260144000000 776 enhancement rgrp dread closed duplicate Avoid generating vdm warnings "We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided? {{{ 2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object 2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object 2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object 2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object 2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object 2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object }}}" 1288372797000000 1297066840000000 777 task dread dread ckan-v1.3 closed fixed Put 503-returning middleware into a module Also configure hmgckan, hmgapi, test-hmg and dev-hmg machines. 1288608324000000 1292586843000000 778 task thejimmyg dread ckan-v1.3 closed fixed Package edit form - name auto-generated from title 1288630777000000 1294416567000000 779 task thejimmyg dread ckan-v1.3 closed fixed Package edit form - stylesheet in DGU updated 1288630800000000 1294416581000000 780 task thejimmyg dread ckan-v1.3 closed fixed Package edit form - 'preview' button shows preview, with form lower in page This ticket refers to the form shown in Drupal. This doesn't have a preview function, but we don't have a requirement for it right now, so leaving ticket closed. 1288630829000000 1294660720000000 781 task thejimmyg dread ckan-v1.3 closed duplicate Package edit form - 'extra' fields have 'add' and 'delete' buttons "(Just like resource table, but extra fields don't need up/down buttons) Nice to have: a blank field is added when you tab from the last filled-in field in the table." 1288630935000000 1294415081000000 782 defect pudo rgrp ckan-v1.3 closed fixed Add and remove users from sysadmin role from the command line "Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems). Suggest creating a new paster command as first step. {{{ # user identifier = openid (?) paster sysadmin add {user identifier} paster sysadmin remove {user identifier} }}} Suggest this also be added to main CKAN docs. This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other)" 1289205287000000 1289218661000000 783 story johnbywater johnbywater ckan-v1.3 closed duplicate Delete harvested documents when remote metadata disappears from harvest source We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer. 1289210029000000 1294409399000000 784 task thejimmyg johnbywater ckan-v1.4 closed wontfix End to end testing for UKLP 1289210251000000 1304936251000000 785 task johnbywater johnbywater ckan-v1.3 closed fixed Review document from PP regarding CSW and WAF guidance Scope of CSW guidance document has been broadened to include WAF. 1289210338000000 1289482499000000 786 requirement wwaites johnbywater ckan-v1.3 closed duplicate The system shall support deleting a registered harvest source 1289210982000000 1294410004000000 787 task thejimmyg thejimmyg ckan-sprint-2011-09-12 closed fixed Auth API "Auth Proposal Use case: We'd like an authenticated and authorized Drupal user to be able to edit/delete packages using the API To do this a user would need a CKAN API key so we need some way of Drupal telling CKAN who a user is so that they can get their API key. Proposed Implementation ----------------------- A user visits the CKAN API key page to get their key. Because CKAN is at catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal cookies. If there is no ``DRXtrArgs`` or ``DRXtrArgs2`` cookie, we know the user isn't signed in so we redirect them to Drupal to sign in. WISHLIST: It would be really nice if we could pass the URL to redirect back to Drupal so tha Druapl can send them back to the CKAN API key page Either way, they get back to CKAN and now the cookies exist. When the first HTTP request header is sent, CKAN will read the Drupal session ID and then call a Drupal API, server to server. TODO: Implement an API on the drupal server which accepts a Drupal session ID as an argument and returns the username and credentials, but only if the request if from the CKAN server (perhaps we specify an API key in the request) If the session is valid CKAN will set its own auth cookie and show them the page with the API key. CKAN only considers a user signed in if both the CKAN cookie *and* Drupal session ID are present. If at any time they sign out of Drupal the Drupal session disappears so they will be signed out of CKAN too. Now the user has an API key they can use the standard CKAN command line API tools. The API key is the only thing the user will need to use the CKAN API. This poses a problem. What if a user is removed from Drupal but still has a CKAN API key? Two solutions: 1. API keys will be set to only be valid for the length of a Drupal session so that a user will only be able to use the write API for a little time after credentials are revoked 2. CKAN provides an API to allow Drupal to revoke keys Option 1. seems easier to me. If we choose this we will write a ``get_api_key`` command line tool so that Drupal user can write things like this: :: datapkg list_packages --api_key `get_api_key` The ``get_api_key`` function will prompt for username and password and then perform the steps necessary to get an API key." 1289211035000000 1315821118000000 788 story johnbywater johnbywater ckan-v1.3 closed duplicate API user deletes harvest source Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?). 1289211196000000 1294410007000000 789 requirement wwaites johnbywater ckan-v1.3 closed duplicate The harvest source form shall validate location field by positive identification of source type 1289212106000000 1296593257000000 790 story johnbywater johnbywater ckan-v1.3 closed duplicate Submit harvest source form with valid CSW location 1289212479000000 1294409709000000 791 story johnbywater johnbywater ckan-v1.3 closed duplicate Submit harvest source form with invalid CSW location 1289212751000000 1294409723000000 792 task johnbywater johnbywater ckan-v1.3 closed fixed Meeting to review pre-alpha testing report from TM 1289212851000000 1289482513000000 793 task johnbywater johnbywater ckan-v1.3 closed invalid Fix up changes in changeset aa9aa32e00a9 1289212907000000 1294409836000000 795 story thejimmyg johnbywater ckan-v1.3 closed duplicate """Match up"" GEMINI2 ""metadata point of contact"" with registered entities" "TM wants to know how this will be ""matched up"". I don't know whether this always the data provider, or sometimes the publisher." 1289228025000000 1296593361000000 796 defect pudo ckan-v1.3 closed invalid AlphaPage for non-latin languages "The AlphaPage implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead. " 1289296662000000 1291640040000000 797 enhancement rgrp rgrp ckan-v1.3 closed fixed Rework core html layout to mirror wordpress twentyten "WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes. * Convert to wp twentyten html * Switch css to be based off twentyten css (and perhaps rework somewhat)" 1289402873000000 1289402982000000 798 task dread dread ckan-v1.3 closed fixed cospread importer using API Move existing getdata/cospread.py into dgu repo and use the importer/loader mechanism. 1289475362000000 1294916973000000 799 requirement johnbywater johnbywater closed duplicate WAF harvesting shall be optimised in respect of reharvesting only records that have changed 1289483264000000 1294232675000000 800 story johnbywater johnbywater ckan-v1.3 closed duplicate Get WAF records modified since given time 1289483343000000 1294245610000000 801 enhancement wwaites johnbywater ckan-v1.4-sprint-3 closed duplicate Get/Set last harvested time for a given harvest source Implies that last harvested time must be stored on source by harvesting job (see #802). 1289483546000000 1300196714000000 802 story johnbywater johnbywater closed duplicate Save last harvested time on source 1289484226000000 1294233294000000 803 story thejimmyg johnbywater closed invalid Auto-generate next migration version unit and test stub files As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually. 1289486678000000 1314031451000000 804 task johnbywater johnbywater ckan-v1.3 closed fixed Rework analysis for publisher/provider in UKLP We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP. 1289816054000000 1294233156000000 805 task dread dread ckan-v1.4-sprint-2 closed fixed Migration scripts on buildbot 1289816755000000 1298379084000000 806 enhancement rgrp rgrp ckan-v1.3 closed fixed Metadata created and last modified timestamps for packages "Packages should provide create and last modified attributes which are included in package dict exposed via api. These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc)" 1289854894000000 1324034356000000 807 enhancement rgrp ckan-v1.3 closed wontfix Handle robot traffic better "We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest: * Add rel=nofollow to various links * Throttle some robots (not all seem to respect robots.txt) " 1289899826000000 1297075372000000 808 enhancement pudo rgrp ckan-v1.4-sprint-2 closed fixed Allow i18n language choice on site (and possibly set automatically) "Allow choice of language on ckan instance. * Configuration variable listing language choices (could try doing this automatically but recommend against this) * Switcher option in UI (perhaps on user profile page) * Default it automatically based on browser settings (?) Think this is high value (and cost is low) Cost: low (4h) " 1289944394000000 1297783658000000 809 requirement pudo ckan-v1.3 closed wontfix Link to the Wiki pages in package form "This would allow for a community-driven definition of field values, conventions, etc. Plus: create wiki.ckan.net " 1289992231000000 1297075561000000 810 enhancement pudo pudo ckan-v1.5 closed wontfix "Move ""add packages"" field up in group form" Subticket of forms super ticket #961 1289992287000000 1310128477000000 813 requirement cygri ckan-v1.3 closed fixed Add link to “Register new package” to the homepage "A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to http://ckan.net in order to create a new package. I don't want to hunt around for the link to the “new package” form. The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage. " 1289995253000000 1294411047000000 814 requirement cygri ckan-v1.3 closed fixed Have an “About CKAN” link on every page "A link “About CKAN” pointing to http://ckan.net/about should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is. I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar. But anywhere else would be fine as well." 1289995821000000 1294411109000000 815 requirement cygri ckan-v1.3 closed duplicate Autocomplete for the search field The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page. 1289995951000000 1294410951000000 816 enhancement johnglover cygri ckan-sprint-2011-10-28 closed fixed Autocomplete for the resource format field "The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource. This would help improve data quality and consistency." 1289996574000000 1319812324000000 817 requirement cygri ckan-v1.3 closed worksforme Proposed redesign of Resources table on package pages "There are some problems with the Resources table on package pages: 1. The column labelled “URL” doesn't contain URLs, but just the word “Download”. 2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links). Attached is a screenshot for a proposed redesign. The changes are: 1. Move the Format column to the right of the Description column (description is more important) 2. Make the description a clickable link 3. Drop the “URL/Download” column because it's now redundant 4. Rename “Description” to “Resource” (not that important)" 1289997384000000 1297073724000000 819 defect cygri ckan-backlog closed fixed Tag autocompletion widget broken "The widget for tag autocompletion is broken in various ways. For example, if I edit a package that is tagged ""music"", and just tab through the form fields to get to the Author field, then the widget changes to ""industrial-music"" as I tab through it. Or if the tag is ""foo bar"" and I hit alt-right to jump to the end of the text field (with the intent of adding a third tag), then the contents change to ""foo barbecue""." 1290004121000000 1323171435000000 820 defect dread closed fixed Exception browsing /user/apikey when not logged in "Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in. {{{ WebApp Error: : 'NoneType' object has no attribute 'apikey' URL: http://test-hmg.ckan.net/user/apikey }}}" 1290006079000000 1311325226000000 821 defect dread ckan-v1.4-sprint-2 closed worksforme Some resources have disappeared in CKAN "Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end. Here are the details: * in CKAN's package view (in web interface, API and dump) no resources display * but they WERE created in CKAN, as shown by the revision diffs. * they are in the Drupal front-end So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal. Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices" 1290015776000000 1298486642000000 822 enhancement Javier de la Cueva closed fixed Docs: Configuration as site-available instead of site-enabled for apache "The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net " 1290107103000000 1290506354000000 823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed. "1. Create a package. 2. Edit package and add a resource. 3. View package in browser - no sign of the resource - ERROR. 4. View package in API - resource is there. Reason: hash doesn't change because package.revision only takes account of core fields." 1290505078000000 1290506116000000 824 enhancement dread dread closed fixed 'Getdata' moved out of CKAN "Data importers should use API instead of object model directly: * saves load on server * much better error checking and authentication * eating our own dogfood (tests the API well)" 1290595507000000 1290595559000000 825 enhancement dread dread closed fixed Document fabric tool 1290609160000000 1290624449000000 826 enhancement kindly dread ckan-v1.4-sprint-1 closed fixed Resource 'extra' fields "There are a number of extra properties we want to store in a resource: * resource size (in bytes, no. triples etc.) - https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673 * resource hash is 'static' (default) or 'changeable' - https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674 * alternative link (e.g. TNA Web Archive) to use the link is broken - https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173 i.e. this is a much needed enhancement Need to be able to read/write these values in web interface and API. (At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.) = Implementation = I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra code as much as possible. (Other much more radical alternative is to move model to RDF.)" 1290611905000000 1306766057000000 827 defect rgrp dread closed fixed metadata_created_and_modified error "Intermittent failure on ckan default branch. Looks like brittle test. {{{ ====================================================================== FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified ---------------------------------------------------------------------- Traceback (most recent call last): File ""/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py"", line 186, in runTest self.test(*self.arg) File ""/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py"", line 250, in test_02_metadata_created_and_modified assert out == exp, (out, exp) AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938)) }}} http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio" 1290678624000000 1290766239000000 828 enhancement rgrp rgrp ckan-v1.3 closed fixed Update cache support in home controller to use cache config in ini file "Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...). This is especially frustrating in a development environment where is currently running into fact front page is being cached. Fix this." 1290693154000000 1290696586000000 829 defect rgrp dread ckan-v1.4-sprint-6 closed wontfix Admin CRUD broken Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception. 1290695043000000 1303838115000000 830 enhancement rgrp closed fixed Themes for CKAN "Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme). However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/" 1290760612000000 1311180263000000 831 task rgrp rgrp ckan-v1.3 closed fixed Migrate ckan trac to ckan.org Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc. 1290760820000000 1290760864000000 832 enhancement rgrp rgrp ckan-v1.3 closed fixed Convert stats to a plugin "Convert stats section to a ckan plugin. Stats need not be in core and making it a plugin would make it easier to extend and expand." 1290765915000000 1296334980000000 833 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-6 closed fixed [super] Administrative dashboard extension "Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview. Possible features: * Purge revisions (or sets of revisions) and purge objects #1076 * Set roles for users #1075 * Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages) * WONTFIX * Overview of activity * WONTFIX - already have revision log Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/. https://bitbucket.org/okfn/ckanext-admin === Tickets === * ticket:1031 - user autocomplete api in ckan === Notes === Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group): {{{ # first remove permissions from roles # this is hacky but have to do it because we hardcode assignment of # role permissions on package on package create (see model/authz.py) paster roles deny editor edit paster roles deny editor create-authorization-group paster roles deny editor create-group paster roles deny editor create-package paster roles deny reader create-package # make superuser group # create authz group administrators / Administrators (if not exists) paster rights make agroup:administrators admin system }}}" 1290765921000000 1303236364000000 834 task Alexander ckan-v1.3 closed worksforme Searching in CKAN "Hello. I've installed stable CKAN v1.1 from PyPI. I can't find any docs about using CKAN API in order to query packages. Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible? Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&open_only=1&downloadable_only=1 and Ckanclient: result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1}) As result nothing found. Thanks." 1290769564000000 1291633657000000 835 enhancement pudo pudo ckan-v1.3 closed fixed Authorization CLI and Documentation "As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages. Suggest put this as an internal command in authz system and add to cli as paster command." 1291029108000000 1291644820000000 836 enhancement rgrp ckan-v1.3-sprint-1 closed fixed Use site_url config option in templates "A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html Instead we should use the site_url config option where necessary. Cost: 1h" 1291132434000000 1302882808000000 838 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed UI improvements December 2010 "This is a meta ticket for a bunch of UI changes: * #839 Package page redesign * #846 Merge css files * #850 Show welcome message * #863 Prompt users to enter missing info * #864 Search results changes * #866 Add download formats to search results" 1291296811000000 1295259773000000 839 enhancement memespring anonymous ckan-v1.3-sprint-1 closed fixed Package page redesign as per http://ckan.org/wiki/UIRedesignPackage 1291299579000000 1292239918000000 840 enhancement dread dread ckan-v1.4-sprint-6 closed fixed On/off switch for ETags cache " * Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'. * Default setting for (all) caching should be off. * Needs documenting in configuration.rst" 1291308727000000 1302694133000000 841 enhancement kindly dread ckan-v1.4-sprint-4 closed duplicate Caching docs (as a whole) "Documentation article on caching / improving performance. (To complement configuration docs.) * Different sorts of cache - beaker style, etags, package_dict in search results(?) * How each one affects performance * How to turn them on/off and configure them * Is it possible to bypass each of them in the browser or with wget/curl?" 1291308879000000 1300364333000000 842 enhancement johnglover rgrp ckan-v1.4-sprint-7 closed fixed Todo list CKAN extension "An extension that provides a todo list feature on CKAN so that people can register and find things to do. Extension name: ckanext-todo == User Story == === Package page === As a user I come to a package: * Have a todo count at that top that takes you down to the todo list (which may say nothing todo) * At the bottom is a section of the package display titled ""ToDo"" where I see a list of all toDos for the package most recent at the top * If I am logged in * See a form for ""Add to do"" at the top of the todo section and can add one straight away * I see a ""now resolved"" button next to each which goes green when you hover. When clicked the todo fades away. * Not logged in: I see a button that says ""login to add todo"" === Todo list page === When a user comes to todo overview at /todo At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category. Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo. == Implementation == The Todo form * One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed) * Add a description * Submit, the todo gets added via AJAX to the list at the top as the most recent todo Model: {{{ todo table id (autoincrement integer) package_id todo_category_id (required) description (required) created=NOW() resolved=null (unresolved) or a datetime (datetime of resolution) creator=user resolver=user todo_category table id name Prepopulate with: broken-resource-link, no-author, bad-format, add-description }}} API at /api/2/todo * GET / POST / PUT ... /api/2/todo?package=package_id_or_name&category=...&resolved=0/1 * support limit (?) /api/2/todo/category -> return list of todo categories * No GET / PUT / POST (these are auto-created by creation of todo) == Optional Extras (Will not be done atm) == * Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ..." 1291467708000000 1305646487000000 843 enhancement wwaites closed fixed user list ui for group curation " * perhaps we should show the openid as well to distinguish between users with the same name. * perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name. * also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it." 1291484245000000 1319721601000000 844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org "ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example: {{{ B30E6D9F 2873 Sat Dec 4 14:05:40 MAILER-DAEMON (delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out) no-reply@ckan.net }}} both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null" 1291556753000000 1296340486000000 845 enhancement dread ckan-v1.3-sprint-1 closed fixed Required fields "Allow a form to specify fields that are required. Currently name and resource url are required. * Show required fields with asterisks by the input box. * Resource fields to be included in this." 1291634995000000 1291723492000000 846 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed merge style.css and ckan.css no need for separate files 1291635478000000 1291719074000000 847 enhancement pudo rgrp ckan-v1.3-sprint-2 closed fixed CKAN search using SOLR backend "This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010). 3 key aspects of this: * Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN) * Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable * Asynchroously: via worker with queue * #324 Search indexing using notifications * #669: Refactor Solr (indexer) to become a generic worker * #874: Extract Solr search backend into an extension * In core support using SOLR backend as well as postgres for queries * #317: Make search pluggable (?) * Integrate into WUI (if changes needed) * #672 - Facets in CKAN search results Extras: * Solr index testing tool #431" 1291639273000000 1295259902000000 848 enhancement pudo ckan-v1.3-sprint-1 closed fixed Ability to hide extras fields on package read for helsinki regional infoshare. 1291714316000000 1291751275000000 849 defect pudo closed duplicate Import extra CSS from main template The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import. 1291714923000000 1291715179000000 850 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Show welcome message on first visit to package page Control using cookies. 1291721923000000 1291721964000000 851 defect wwaites wwaites ckan-v1.3-sprint-1 closed fixed Link Checking "revisit link checker from http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py revisit ollyc's parallel work look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user)" 1291722860000000 1293025112000000 852 enhancement rgrp wwaites ckan-v1.5 closed fixed [super] Dataset upload and archiving "3 (4) use cases: (With and without access to auth keys.) 1. upload form on ckan (by default local system has auth key for storage) 2. datapkg upload (client) (by default without key) (plumbing done) 3. background job archiving downloads (by queue client) Most important are: 1. Upload form on ckan.net (with access to keys) 2. API for getting credentials for upload from ckan.net api (without access to keys). Aside: Once 2 was done we could deprecate storage.ckan.net === Sub-tickets === ''' Required infrastructure ''' * #826 - Resource 'extra' fields * #358 - Resources in REST API * #235 - Resource format normalization and detection ''' User upload via Web UI ''' * #877 - File upload in WUI * #878 - Integrate file upload with workflow around package resources ''' User upload via CLI/datapkg ''' * #853 - plumbing for credential-less rest upload * #879 - Storage auth API Also related: * #405 - Retrieval options for package resources === More Info === * Upload/storage discussion: http://ckan.okfnpad.org/storage * For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/ " 1291722993000000 1315821628000000 853 enhancement wwaites wwaites ckan-v1.3-sprint-1 closed fixed Client upload to storage without having primary storage keys "Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys. This would lead to an extension to OFS. This analysis should inform (and go hand-in-hand) with the implementation of ticket:879 (Storage Auth API in CKAN). * master ticket #852" 1291723063000000 1294594581000000 854 defect johnlawrenceaspden dread ckan-v1.4-sprint-6 closed fixed Tests for authorization_group "The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests. " 1291723815000000 1304351843000000 855 defect pudo dread closed invalid Authenticator tests The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests. 1291723967000000 1311176988000000 856 defect wwaites dread closed invalid Tests for caching Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved? 1291724064000000 1311177085000000 857 defect dread closed invalid Tests for CLI Test coverage for ckan.lib.cli is 34%. Need to improve that. 1291724131000000 1311182581000000 858 defect dread dread closed fixed Tests for diff Test coverage for ckan.lib.diff is 24%. Need to improve 1291724171000000 1291734297000000 859 defect pudo dread closed invalid Tests for User model Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement? 1291724348000000 1311177461000000 860 defect wwaites wwaites ckan-v1.3 closed fixed full text indexing for semantic.ckan.net ... 1291726029000000 1291726067000000 861 defect wwaites wwaites closed invalid unicode errors creating rdf output http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d 1291726125000000 1311168845000000 862 defect wwaites wwaites ckan-v1.3 closed fixed deploy rdf on catalogue.data.gov.uk handle various dgu extras, configure apache for autonegotiation, etc. 1291726237000000 1291726261000000 863 enhancement memespring memespring closed wontfix prompt users for missing package information e.g. license, owner etc 1291729787000000 1338206455000000 864 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Search results UI changes as per http://ckan.org/wiki/UIRedesignSearch 1291736441000000 1291741028000000 865 defect pudo pudo ckan-v1.3-sprint-1 closed fixed Support external strings in JS "* Provide and include a ""language.js"" file. " 1291801938000000 1295259773000000 866 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Include download formats in search results as above 1291812213000000 1292240191000000 867 enhancement dread dread ckan-v1.3-sprint-1 closed fixed ckanclient raises exceptions "To be more pythonic, raise exceptions when ckanclient gets status which isn't 200. " 1291915162000000 1299866685000000 868 enhancement sebbacon thejimmyg closed fixed Test improvements "The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach. We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!) Some suggestions for achieving this include: * Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database * Not setting up and tearing down the database so frequently. " 1291987811000000 1294753596000000 869 enhancement rgrp rgrp ckan-v1.3-sprint-1 closed fixed Add plugin interface to allow editing and extending of config by extensions "Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc). Allowing plugins to edit the config dictionary on load would allow this (and more!). Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)" 1291989395000000 1292235262000000 870 defect memespring ckan-v1.3-sprint-1 closed fixed Menu items not selecting correctly "Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly. Specifically if Home / About or Add Package / Search are selected both menu items are highlighted. This is because the code assumes only one top-level menu item per controller." 1292243583000000 1294914243000000 871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too "The infamous [http://www.exim.org/lurker/message/20101207.215955.bb32d4f2.en.html exim bug] only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers. We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not. BTW did we already run a rootkit checker like [http://rkhunter.sourceforge.net/ Rootkit hunter] on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could. " 1292264117000000 1296340558000000 872 enhancement johnbywater dread closed fixed ONS loader assigns packages to same package even when dept name changes Solutions: ResourceSeriesLoader allows synonyms for values. 1292424112000000 1292957110000000 873 defect dread dread closed fixed ONS loader assigns packages to same package even when dept name added manually "e.g. national_child_measurement_programme had no department but ""Information Centre for Health and Social Care"" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_." 1292437253000000 1292939274000000 874 enhancement pudo ckan-v1.3-sprint-2 closed fixed Extract solr search backend into an extension Move the solr SearchBackend to a seperate code base, remove any mentions of the CKAN model. 1292844422000000 1293057151000000 875 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Search backend supports solr interface and query API mimics solr "Consolidate search API interface (and backend) on solr (solrpy) type interface. * Support for standard query structure * Support for facet options Do '''not''' need to change response formats. (Or do we?) 2 options here for advanced features like facets in non-solr: 1. Disable (happens automatically) 2. Implement - suggest using group by etc == Extras == * Front-page tag cloud: change this to use facets * Accept this means that if facets not functional in backend we have no tag cloud " 1292844957000000 1297085261000000 876 enhancement sebbacon rgrp ckan-v1.3-sprint-2 closed fixed Support sqlite as a database backend for CKAN Among other things this will allow the tests to run much quicker. 1292858711000000 1294753889000000 877 enhancement rgrp rgrp ckan-v1.4-sprint-2 closed fixed File upload in WUI (extension) "Support uploading files to a storage system from WUI with a simple upload form. === Implementation === * Implement as an [wiki:Extensions Extension] * Locate at /upload * Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users * Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists) * This wants to be tied in to general package workflow. See parent ticket:852." 1292868146000000 1298624165000000 878 enhancement rgrp rgrp ckan-sprint-2011-09-12 closed fixed Integrate file upload with workflow around package resources "Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata? If yes suggests (need to ticket): * 'ResourceExtras' for extra metadata on resources * Makes 'Resource' into first class object (and make available in WUI)? == Depends == * #877 (Upload in WUI) - should be part of that extension" 1292868269000000 1315820838000000 879 enhancement wwaites rgrp ckan-v1.3 closed fixed Storage auth API "API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage. Implement as a CKAN extension." 1292868509000000 1316030482000000 880 defect dread ckan-v1.3-sprint-2 closed fixed ONS loader tripping up on deleted packages Deleted packages with the same title and extra fields as an active package trip up the loader. It finds them in the search, but due to ckanclient not using the apikey when you 'get' the package to examine it further, it causes an exception. 1292939379000000 1292945137000000 881 defect thejimmyg fccoelho closed invalid http authentication requirement when installing I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code. 1293021889000000 1296335072000000 882 defect pudo pudo closed wontfix Update i18n user docs and backport genshi i18n:domain fixes from WDMMG 1293192044000000 1340626094000000 883 enhancement wwaites wwaites ckan-v1.4 closed fixed uklii harvesting refactor "== master ticket == * #884 implement harvest job delete command for an easier life * #885 move csw code into dedicated ckanext plugin * #886 move forms api into dgu extension module * #887 move harvesting code to use generic harvesting module " 1293277713000000 1300196622000000 884 enhancement thejimmyg wwaites ckan-v1.3 closed fixed harvest job delete operation implement paster subcommand to delete harvest jobs 1293277800000000 1296592140000000 885 defect wwaites wwaites ckan-v1.3 closed fixed reimplement csw module to use owslib 1293278099000000 1294253508000000 886 defect wwaites wwaites ckan-v1.3 closed duplicate move uklii forms api to dgu module doesn't belong in ckan core 1293278145000000 1294916538000000 887 enhancement wwaites wwaites ckan-v1.4 closed fixed move harvesting to generic module in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat 1293278215000000 1300196600000000 888 enhancement johnglover rgrp ckan-sprint-2011-10-28 closed fixed Improvements to the dataproxy and the data API "First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...): For dataproxy: * Testing for dataproxy * Can start by using known good remote urls (moving forward could switch to providing/mocking these locally) * Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum) * Google docs style row/column selections * Use the swiss library - https://bitbucket.org/okfn/swiss * Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet) * Handle redirects for content-length? * Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?) For dataapi: * Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}" 1293649783000000 1311773103000000 889 enhancement rgrp rgrp ckan-v1.3-sprint-2 closed fixed Support extra footer material in config option (e.g. for google analytics) "Add a config option 'template_footer_end' which is inserted in layout_template just before . This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics. Aside: going forward may want to turn this into a extension. Cost: 1h" 1293713189000000 1293715109000000 890 enhancement kindly pudo ckan-v1.6 closed invalid Introduce timed actions into ckanext-queue "The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows: * routing_key: Package * operation: daily * payload: pkg.as_dict() Where operation is one of daily, weekly, monthly or any other interval. " 1294050769000000 1318599247000000 891 task johnglover pudo ckan-sprint-2011-11-07 closed fixed Resource download worker daemon "Superticket: #1397 Write a worker daemon to download all resources from a CKAN instance to a local repository. == Questions == * Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues) * Should we have clever ways to dump APIs? ANS: no. * Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving) * Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache. * Complete support for ETags * Expires, Max-Age etc. * Check == Functionality == * Download files via HTTP, HTTPS (will not do FTP) Process: 1. [Archiver.Update checks queue (automated as part of celery)] 2. Open url and get any info from resource on cache / content-length etc 1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery 2. Check headers for content-length and content-type ... * IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?) * ELSE: check content-type. * IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource) * ELSE: archive it (compute md5 hash etc) * IF: get content-length and content-length unchanged GOTO step 4 3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext * Add cache url to resource and updated date * Add other relevant info to resource such as md5, content-type etc 4. Update task_status == Optional functionality == * If result object is HTML, search for references to ""proper data"" (CSV download pages etc.) * Download from POST forms (accepting licenses or weird proprietary systems) * Support running on Google Apps Engine to save traffic costs. == Existing work == * https://bitbucket.org/okfn/ckanext-qa/overview * out of date: https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental. * out of date: https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc" 1294052979000000 1320149841000000 892 enhancement johnglover pudo ckan-sprint-2012-01-09 closed fixed Make stored data available in WUI - 0.5d "Once we have storage, make the data available in the following ways: * Now have a cached_url field can show in the frontend ... * Add a [cached] link to right of real url on resource listing on dataset page. * On resource page: will not add it yet. * At the moment no clear place to pu this given nice big download button (could put in list of items on left but that does not seem right and note that it will turn up in big list of info at bottom) * Add test (?) * Deploy " 1294053293000000 1324402480000000 893 defect pudo ckan-v1.4 closed wontfix ExtrasField should not overwrite more specific extras At the moment, ExtrasField cannot be used in conjunction with more specific extra fields, such as TextExtraField or SuggestTextExtraField. 1294135783000000 1298293527000000 894 defect wwaites wwaites ckan-v1.4 closed invalid harvest from http://locationmetadataeditor.data.gov.uk/geonetwork/srv/csw 1294253876000000 1300196388000000 896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances "Based on the ideas in: * DistributingChanges * SyncingInstances Merging the tickets #296, #297, #298, #299 into this one. Related ideas include: * Using dcat to exchange data * Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance This should be considered a wishlist item for the timebeing. " 1294406999000000 1300217375000000 897 defect dread dread closed duplicate """New package"" highlighted in Navigator when searching" "On ckan.net 1.3.1b, click on ""Search"" in the Navigator. Now both ""Search"" and ""New Package"" are highlighted." 1294648815000000 1294914333000000 898 defect rgrp dread closed fixed Changes stored indefinitely "Every change to every object is being stored in memory, which could add up to quite a lot of memory. This fixes it by making sure the objects are in a weakref. https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f" 1294659490000000 1294662408000000 899 enhancement rgrp dread closed fixed Optimise check_real_change "There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e From David Raznick" 1294659583000000 1294916615000000 900 defect dread closed fixed Intermittent error with new revision not having id "There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e. From David Raznick" 1294659696000000 1294916587000000 901 enhancement rgrp dread closed fixed vdm support for sqla 0.6 and sqlite 1294660144000000 1294662466000000 902 defect rgrp dread ckan-v1.4-sprint-1 closed fixed Package buttons not clickable / Improve submenu tabs "On a package page there are three buttons to the right - View, Edit, History. Each of these buttons contain an icon and the text, but you can only activate the button by clicking on the icon. When you click on the part of the button where the text is (about 50% of the button) then nothing happens. Using: Chrome, Firefox" 1294756898000000 1297078885000000 903 defect pudo pudo ckan-v1.3 closed fixed Proper handling of deleted packages in Solr serps Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr. 1294831054000000 1296588070000000 904 task rgrp Stiivi ckan-v1.4-sprint-3 closed fixed Review CKAN documentation "What's bad at the moment? * lack of documentation e.g. config (very poorly documented) * too many sources of documentation * no common theming Sources: * http://ckan.org/ * http://packages.python.org/ckan * http://wiki.okfn.org/ckan/ - user * Deprecate: http://knowledgeforge.net/ckan/doc/ckan/ Resulting meta-ticket with things to do: ticket:927" 1294832610000000 1299840539000000 905 defect thejimmyg Stiivi ckan-sprint-2012-02-06 closed worksforme Unable to search with accented characters in package names "I have a package with name: ""Štatistické prehľady agendy cestovných pasov"" and when I try to search for ""Štatistické"" there are no results. CKAN setup: - psql DB with UTF-8 tables - psql search backend - ckan version: 1.3.1b Example: http://verejnedata.sk/package/agenda-cestovnch-pasov try to search for the first word in the package name. " 1294939120000000 1328638536000000 906 enhancement thejimmyg Stiivi ckan-sprint-2012-03-05 closed fixed Ability to search without accents for accented words "In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results. This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents. For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCII//TRANSLIT). " 1294939588000000 1330990400000000 907 enhancement rgrp ckan-v1.3-sprint-3 closed fixed Data previews: preview package resource data in the WUI "As a user I want to preview a dataset ('resource') before downloading it. == Implementation == This will be implemented as an [wiki:Extensions Extension] NB: This ticket requires a way to get data out of a remote url (the dataset). We do '''not want to put this data load/streaming into ckan core''' due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file) 1. Need a 'data proxy' - see ticket:698 and ticket:888 2. Preview using javascript in WUI - do this in an extension http://bitbucket.org/okfn/ckanext-dataapi " 1295261173000000 1297072303000000 908 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Issues deploying extensions with modwsgi / uwsgi "Some extensions import pylons features (such as request) at root. Have encountered following issues: 1. You need to import ckanext before any ckan stuff in your wsgi script (extension stats and dataapi) 2. Running under wsgi daemon mode (normal mode is fine!) the front page (front page only) will not work (does not occur with disqus or synchronous_search but does with stats and dataapi extensions). You get errors like this in the log: {{{ [Mon Jan 17 09:01:19 2011] [error] Exception KeyError: KeyError(-1218594160,) in ignored [Mon Jan 17 09:01:20 2011] [error] Exception AttributeError: ""'NoneType' object has no attribute 'clearing'"" in ' }}} " 1295264624000000 1296730578000000 910 enhancement pudo pudo ckan-v1.5 closed invalid Refactor CKANrdf to support extras Extra fields should be represented in some manner. 1295266026000000 1310128544000000 911 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Add pager to dcat-tools/rdfsolr 1295266211000000 1296467375000000 912 defect pudo pudo ckan-v1.5 closed wontfix Rethink result row representation in dcat-tools/rdfsolr " * Should license go in the bottom line? * Formats should be styled as in CKAN " 1295266299000000 1306774876000000 913 enhancement pudo pudo ckan-v1.5 closed wontfix Handle licenses in dcat-tools " * Create a list of all valid predicates where licenses are stored. * Create a license normalization registry in GSpread * Fill spreadsheet from OKD/OSI lists. " 1295266407000000 1306774962000000 914 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Handle formats in dcat-tools " * Create a list of valid predicates for format information (accessURL et al) * Execute #235 via Google Spreadsheets and include in DCat enrichment " 1295266569000000 1297069958000000 915 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Deploy dcat-tools/rdfsolr to publicdata.eu as a placeholder We should present something there, why not start with a meta-search. 1295267040000000 1296467518000000 916 enhancement rgrp pudo vdm-0.8 closed duplicate Port new vdm to Mongodb "We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG. " 1295277011000000 1297066902000000 917 enhancement kindly kindly closed fixed Turning off Autoflush in SQLAlchemy "David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior. I have a patch in https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default. This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet). Things to know about using flush: * Most of the time nothing will change. A commit implies flush. * You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query. * A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated. * You can use them as a limited form of nested transaction. It increased the performance on the tests by about 5%. There are less flushes altogether so is faster. I do not see any downsides." 1295279694000000 1295280232000000 918 defect memespring dread closed invalid Package preview doesn't show tags or groups "1. Edit any package 2. Hit 'preview' Notice that the preview at the bottom doesn't show what tags or groups the package is in. This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview. One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision." 1295346496000000 1315911507000000 919 enhancement dread dread ckan-v1.4-sprint-6 closed fixed Package preview contains API & datapkg instructions "1. Edit a package 2. Hit 'preview'. The preview contains the section ""CKAN API / datapkg"" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the ""CKAN API / datapkg"" section too?" 1295346730000000 1303202627000000 920 defect thejimmyg hellmann@… closed fixed empty tags are kind of confusing "http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed. It should be purged !!!" 1295355280000000 1300319140000000 921 enhancement sebbacon thejimmyg closed fixed Refactor DGU Harvesting "Refactor harvesting code to simplify. " 1295389237000000 1300197629000000 922 enhancement kindly kindly closed duplicate Make Resources first class entity. "Why? * Will make it easier to give resources their own form, independent of packages. * Makes uploading, harvesting and previewing data cleaner as its separate from the packages. * Gives the option of sharing a resource between packages. * Leads the way to supporting storing richer semantic information against them." 1295713858000000 1320664187000000 923 defect rgrp dread closed worksforme Search box doesn't work in leaderboard page "1. Go to: http://ckan.net/stats/leaderboard#content 2. In the far top-right of the browser, select the search box in 'water'. 3. Press enter to search. Nothing happens. Tried in: chrome, firefox" 1295867328000000 1340632144000000 925 defect dread ckan-backlog closed fixed Change the search box icon to remove the down arrow Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed? 1295867593000000 1323168588000000 926 enhancement sebbacon sebbacon ckan-v1.4-sprint-1 closed fixed Pick a simpler form framework "The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise. Review existing (and potentially non-existing) frameworks with a view to porting forms over to something more explicit and lightweight. Implement the current Package forms as an example of how this would work. Document and circulate. Sub-ticket of #961 (form, validation, model sync meta-ticket)" 1295869056000000 1298541597000000 927 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed [super] Improvements to CKAN documentation "List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along. == Major == * ~~(Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)~~ * Document caching system - ticket:841 * Howto guide for writing an extension - ''started on this at http://wiki.ckan.net/Extending_CKAN'' == Miscellaneous == * Contributors / credits page: ticket:928 * ~~(Data proxy documentation: move it to packages.python.org/dataproxy)~~ * ~~(List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))~~ * Document license configuration and license system - ticket:973 * Refactor trac frontpage moving general feature overview to ckan.org * Create proper roadmap information on http://trac.ckan.org/ * Add link to wiki.ckan.net to ckan.org " 1295869492000000 1300105638000000 928 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Documentation - CKAN credits / contributors page " 1. Remove contributors list from about page (out-of-date and does not belong there) * A ""Powered by CKAN"" link in footer is enough 2. Add contributors / team / credits page to ckan.org * May want to separate credits (e.g. software use), from contributors (also is team different from contributors) * (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket) 3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info. Cost: 1h" 1295872661000000 1297115136000000 929 defect rgrp rene.kapusta ckan-v1.4 closed fixed Handle the case when the licenses service is down better "I'm using v 1.3.1a and get a lot of ""WebApp Error: : Couldn't connect to licenses service: "" error messages. {{{ Module ckan.forms.package:87 in build_package_form << # Options/settings builder.set_field_option('name', 'validate', package_name_validator) builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()}) builder.set_field_option('state', 'dropdown', {'options':model.State.all}) builder.set_field_option('notes', 'textarea', {'size':'60x15'}) >> builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()}) Module ckan.model.package:283 in get_license_options << @classmethod def get_license_options(self): register = self.get_license_register() return [(l.title, l.id) for l in register.values()] >> register = self.get_license_register() Module ckan.model.package:278 in get_license_register << def get_license_register(self): if not hasattr(self, '_license_register'): self._license_register = LicenseRegister() return self._license_register >> self._license_register = LicenseRegister() Module ckan.model.license:46 in __init__ << from licenses.service import LicensesService2 self.service = LicensesService2(group_url) entity_list = self.service.get_licenses() else: from licenses import Licenses >> entity_list = self.service.get_licenses() Module licenses.service:44 in get_licenses << except Exception, inst: msg = ""Couldn't connect to licenses service: %s"" % inst raise Exception, msg try: licenses = loads(response_body) >> raise Exception, msg Exception: Couldn't connect to licenses service: }}}" 1295995409000000 1299840884000000 930 defect wwaites dread closed fixed call_timing files created for every request "On a production server we produce literally millions of little files saying how long every single request took to process. They are here: {{{ hmg.ckan.net.2/pylonsdata/call_timing }}} They were added by ckan/lib/base.py in cset:da438a9085d3. I don't believe anyone uses these stats." 1296061721000000 1297066292000000 931 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Search results generator hides paging functionality ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically. 1296147360000000 1298379187000000 932 enhancement dread dread ckan-v1.3 closed fixed Move to SQLAlchemy-Migrate 0.6 "Basic steps from kindly: * changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions. * running the migrate script on the repository (or changing the repository scripts manually). * checking in both these changes at once." 1296156660000000 1297076710000000 933 enhancement kindly ckan-v1.4 closed fixed get rid of self when it should be cls in classmethods use pylint 1296174526000000 1297348975000000 934 enhancement kindly closed invalid Key value store for extensions to store data "We need to allow plugins to store information. Each plug-in should have its own name space and allow any data to be set. There are two options. * make a key value table in ckan that will hold random information. * use redis (or other key value store) If in sql suggest schema of form: * (namespace, obj_id, key, value [, value_type]) * value is json" 1296228758000000 1323171047000000 935 defect pudo pudo closed fixed """groups"" field bug in IATI" "An internal server error was reported when trying to create a new package (not when editing): WebApp Error: : (IntegrityError) null value in column ""name"" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'} The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the ""groups"" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server. Possible causes include: * Stale pyc file * Threading issue * Dependency issue As the problem is gone for now, it cannot be further investigated atm. " 1296302915000000 1306778682000000 936 enhancement johnglover rgrp ckan-v1.4-sprint-6 closed fixed Follow / watch package extension "As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis). NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'. Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). '''Decision: use follow''' == Implementation == === Interface === Become a follower: * Follow button on packages (if already watching say 'unfollow') Package-related changes: * Show number of followers on a package * List followers of a package at /package/{name}/followers * On a separate page a package User-related changes: * List followed packages * Either on user's page on a separate 'following' page. (NB: called 'following') * Does watching involve notifications (by email) * Probably not: you can already subscribe to RSS feed after all and email not that necessary (?) * [Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page === Nitty-Gritty === * Want to do this in ajax-y manner * API endpoint: /api/2/follower * Store data in a new follower table ==== API ==== /api/2/follower {{{ follow => PUT / POST { user_id object_type object_id } }}} If this is submitted by a user with user.id != user_id => error (401) {{{ unfollow => DELETE /api/2/follower/package/{id} => list of followers [ { safe dictized user } ] }}} NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method. ==== Table ==== Called 'follower' {{{ user_id, table, object_id, created xxx, package, yyy, ... xxx, user, yyy, ... [future] }}} === Random Extras === * What about following users as well " 1296339079000000 1303838713000000 937 enhancement sebbacon rgrp ckan-v1.4-sprint-5 closed fixed Record download stats for resources (extension) "As a User (especially as a Package Owner/Maintainer) I want to know how many times a resource has been downloaded (and when). So let's record download stats (as in clicks on the link for a resource). == Implementation == * Use the existing support for this feature in google analytics or piwiki * For google analytics see: * Integrate to record: http://www.google.com/support/analytics/bin/answer.py?answer=55529 * Accessing analytics data via API (for showing download counts): http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html * Limitations only record downloads by browsers with js turned on. * This is OK I think (in any case machine download via e.g. datapkg does not get downloaded) === Old Spec (do it in CKAN) === * Record info of form: resource id (or url?), timestamp * Do this via javascript capturing of onclick event talking to an api * API: /api/resource/{id}/download * POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js) * GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] } == Questions == * Do we record ip addresses (to handle de-botting etc)? * Do we count preview clicks as well?" 1296340273000000 1302513831000000 938 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Message flashing / notification in WUI "Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see http://flask.pocoo.org/docs/patterns/flashing/). We already do some of this (slightly poorly), e.g. on adding an authorization the app says: ""X authorization added"". == Implementation == * Standardize this process and add section to html for this purpose. * Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit * Testing: suggest keeping this very simple == Estimate == * Cost: 0.5h (html etc) + 1h to do some integration " 1296398920000000 1297077023000000 939 enhancement rgrp closed fixed Notification bar at top of site shown to new users pointing them to simple instructions "On http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions. Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better. == Implementation == * Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing) * FAQ/instructions page (use the new wiki?) == Depends == * ticket:938 == Estimate == * Cost: 1h" 1296399008000000 1323171158000000 940 defect pudo rgrp ckan-v1.3 closed wontfix OpenID login via google for same user generates 2 different openid urls "This has been noted twice in the last couple of weeks. Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See: * http://groups.google.com/group/google-federated-login-api/web/the-most-important-technical-issue-in-using-the-google-accounts-api?pli=1 * http://stackoverflow.com/questions/3549953/how-to-get-unique-identifier-from-openid-google To fix this we should set the openid.realm (not sure how atm). If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache). Cost: 1h" 1296401521000000 1297071748000000 941 enhancement thejimmyg rgrp ckan-v1.5 closed wontfix Submit apps or ideas for apps related to datasets (extension) The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted. 1296403099000000 1310134339000000 942 defect pudo pudo ckan-v1.4-sprint-1 closed fixed Fix SERP pager argument passing "It only passes ""q"", but the whole query must be serialized. " 1296469490000000 1297074807000000 943 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Create CKAN Wiki Set up a Mediawiki to contain info on CKAN.net (community) conventions etc. 1296469547000000 1297076498000000 944 defect pudo pudo iati-4 closed fixed Upgrade IATI to CKAN 'default' " * Move code base, fix any breakage. * Deploy to eu8 * Test * Deploy to eu14 == Why == Later CKAN codebase has a bunch of improvements / fixes that it would be good to have. == Estimate == * 8h " 1296469688000000 1306774766000000 945 enhancement kindly kindly ckan-v1.6 closed fixed [super] Richer resources - Resource Groups, new fields, improved UI "Super ticket: #1032 This is a meta ticket for changes that are going to happen in resources. * New resource group table. #956 * New kind field in resource. #957 * UI for new kind field. #958 * Resources in REST API ticket:358 * Resources in WUI #1445 * Make Resources first class entity. #922 (duplicate?) Background on this change can be found at: * http://ckan.okfnpad.org/resources - discussion etherpad * Contains commented version of James Gardner's main email: http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html * UseCasesResources " 1296475283000000 1325259350000000 946 defect pudo dread ckan-v1.3 closed fixed No default system rights "CKAN currently comes out of the box being uneditable. We'd like to encourage people to have open CKANs and therefore by default init the db with this right: {{{ visitor -> is editor on -> System system }}}" 1296499922000000 1296833383000000 947 enhancement rgrp closed duplicate Move the assignment of roles to a package on package creation to an extension "At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded. This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833). An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own)." 1296557478000000 1314031310000000 949 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Generic username/password login "based on repoze.who-friendlyForms " 1296658126000000 1297074827000000 950 enhancement dread dread ckan-v1.3 closed fixed Move importer controller to ckanext-importer "ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN. In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext." 1296666038000000 1297079743000000 951 defect adrian.pohl@… closed invalid Can't add a package to group "I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is ""history"") nor on the group page." 1296726886000000 1314031006000000 952 task wwaites pudo ckan-backlog closed invalid CKAN should run under nginx/uswgi " second part of #908" 1296730498000000 1310125204000000 953 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Add tagline/description to template and set in configuration Have a tagline / description are under main title and set if from ckan.site_description config variable. 1296809808000000 1296809834000000 954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3 "Child tickets: * #1107 Move package autocomplete from package controller and move to API * #1087 version and contact info api call Move to a format that has a separate responseHeader and response. === A standard package response === {{{ { responseHeader: { status: 0, } response: {package-dict} } }}} On error: {{{ { responseHeader: { status: {err-code}, error: 'message' } response: none } }}} === A search query === Based directly on solr. {{{ { responseHeader: { status: 0, } response: { numFound: 5, start: 0 docs: [ ] } } }}} == Issues == This is a breaking change for clients == References == * http://wiki.apache.org/solr/SolJSON * http://www.flickr.com/services/api/response.json.html " 1296811899000000 1320142744000000 955 defect dread dread ckan-v1.3 closed fixed Tidy up tests "* Removal of init_db from every setup etc. * Get them working" 1296819300000000 1297342534000000 956 enhancement kindly kindly ckan-v1.4-sprint-2 closed fixed Add a resource group table. "We need a way to group resources by various different factors. The most important of which is timeseries. The model will look like Package resource_group resource The use cases to this are outlined in this email. http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html " 1296821112000000 1299489084000000 957 enhancement kindly kindly closed wontfix "new ""kind "" field added to resources" "The new field was outlined in http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html it says '''Can be ""data"", ""metadata"", ""service"", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.'''" 1296821398000000 1314218701000000 958 enhancement kindly ckan-v1.5 closed duplicate ui for new kind field Ui for related new field in resource (#957) 1296822896000000 1320664462000000 960 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Support non-ascii character in internal error messages "Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way. Should do this not just here but everywhere we can find in the code base. {{{ Module ckan.controllers.package:302 in edit << am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg) if not am_authz: abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id))) auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg) >> abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) }}} == Estimate == Cost: 1h" 1297069721000000 1297169056000000 961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization "This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN. ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html == The Issue == From #926: The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise. From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html: ... !FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing. At least 3 functions involved [in this area]: 1. Generating (or just filling) a form template with 'form data' (and errors) 2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization' 3. Converting form data to model data (and validating) (inverse of previous step) == Related Tickets == * #926 - Pick a simpler form framework * #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict * #1079 Refactor API to use new logic layer and dictization * #1078 Refactor WUI controllers and forms to use logic layer * cf existing dumper and importer code * This will fix #662 * [not ticketed yet] - validation layer (should work on serialized objects?) * #662 - Can't put entity that is returned by posting to package register (Defect) * #972 - Merge 'extras' into main package dict rather than having separate key * #1035 - Form impressions are given IDs * #810 - Move ""add packages"" field up in group form (easier to do this once forms are done)" 1297069849000000 1310126100000000 962 enhancement rgrp rgrp ckan-v1.4-sprint-5 closed fixed Improvements to data preview extension "Basic implementation done (and deployed): https://bitbucket.org/okfn/ckanext-datapreview However plenty to improve, e.g. * Support more formats (use external systems for preview?) * json (!) * html (trivial!) * sparql * ... * Do not display preview if no preview Also suggest reworking to use external services rather than doing preview 'in house' (doing in house places heavy reliance on data proxy service and on converting data to a standard format)." 1297072524000000 1301364987000000 963 enhancement thejimmyg rgrp ckan-v1.4-sprint-1 closed fixed Package CKAN as a debian package Package CKAN as a debian package so it is easier to install. 1297076364000000 1298284252000000 964 enhancement rgrp rgrp datapkg-0.8 closed fixed Create Resource Downloader and make it pluggable In downloading packages we need to download resources and we need a ResourceDownloader object for this. these should be pluggable so that we can add support for different types of resources. 1297211004000000 1297211237000000 965 enhancement kindly kindly ckan-v1.4 closed fixed make migration testing plugin to nose "In order to test migrations properly, we need to test the upgraded database against *all* the tests. To do this a special flag to nosetests should be added. nosetests --migrations This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db." 1297213492000000 1298280637000000 971 enhancement sebbacon sebbacon closed duplicate Support middleware as a plugin I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system. 1297277473000000 1299245064000000 972 enhancement sebbacon rgrp ckan-v1.5 closed wontfix Merge 'extras' into main package dict rather than having separate key "Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API. Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras. == Issues == * Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being) Possible subticket of forms refactoring: #961" 1297332282000000 1310134129000000 973 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Document license configuration and license system. "Document how license system works and specifically how licenses can be configured. Cost: 2h" 1297332948000000 1297678851000000 974 defect dread ckan-v1.3 closed fixed Document site_description Configuration option 'ckan.site_description' isn't documented 1297342254000000 1297342599000000 975 defect rgrp rgrp ckan-v1.4-sprint-1 closed fixed Placeholder attribute not displayed in some browsers leading to poor UX "Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE). This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search. We should fix this by finding a way to display placeholder attribute in all browsers." 1297343755000000 1297344448000000 976 defect dread ckan-v1.3 closed fixed Documentation references knowledgeforge.net 1297346649000000 1297346954000000 977 defect dread ckan-v1.3 closed fixed db_upgrade errors " * db_upgrade (in ckan/model/__init__) makes call to validate_authorization_setup but forgets about rest of initialisation. * it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)" 1297420558000000 1297420742000000 978 enhancement zephod dread ckan-sprint-2012-03-05 closed fixed Edit Resource including Extras in Web UI "Part of super-ticket #1506. Follows on from #826. We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis. " 1297429619000000 1330547181000000 980 defect pudo pudo ckan-v1.4-sprint-1 closed fixed Fixes to HR Infoshare "We've fixed a few things in CKAN that were discovered and tracked by HRI in their own issue tacker. This includes: * add authz checks to package edit links (cset:0752316cd2fe) * replace gettext with _ in controllers, to support unicode error msgs (cset:822340e6077e) * handle broken html in notes field without crashing on package read (cset:4b6be037dda0) * update i18n (cset:37d57dc3c492, cset:ea03173f5e77, cset:f16f4ee40fe7)" 1297430918000000 1297503177000000 981 defect kindly kindly ckan-v1.4-sprint-2 closed fixed fix authorization_group_user.id so it's added in migrate This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up. 1297434480000000 1298278654000000 982 enhancement dread rgrp ckan-v1.4-sprint-2 closed fixed Remove various pip-requirements versions "We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head. Remove them and update any client systems. Cost: 20m" 1297516060000000 1298887980000000 983 defect dread dread ckan-v1.3 closed fixed 'db upgrade' creates system priviledges (should leave priviledges alone) 1297518265000000 1297773407000000 984 defect kindly kindly ckan-v1.4-sprint-2 closed fixed Fix migration, where upgrade diverges from actual model. "The migration scripts have not been inline with the upgrade scripts. There are lots of discrepancies. " 1297627447000000 1298280689000000 985 enhancement pudo pudo pdeu-1 closed duplicate Harvest other data catalogues " * data.london.gov.uk (1d, ping DERI) * catalogues from Paris and Vienna (1d) Others? * digitaliser.dk (1d) * data-publica.com (2d) * data.suomi.fi (2d) * dati.piedmonte.it (1d) * portalu.de (2d)" 1297679799000000 1306408134000000 986 defect wwitzel3 thejimmyg ckan-v1.4-sprint-2 closed fixed Broken link report from the ckanext-qa code "Should have the following features: * A list of all packages with broken links on the resource * Under each packages a description of any resources with broken links to include: * Link * Description * Format * First found to be broken date [If you can't do the last one yet, don't worry]" 1297683994000000 1297812401000000 987 defect pudo pudo closed duplicate Common harvesting framework "We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: * CSW/WFS for INSPIRE/UKLII (yields CKAN packages) * Catalogue scraping for LOD2 experiments (yields RDF graphs) * Atom/DCat for LOD2 production (yields RDF graphs) * OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages) We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. In general, such a system would have the following stages: * Source selection: find what to download/scrape/harvest/parse * Index retrieval (i.e. package index) * Item retrieval (i.e. package entity) * (Optional: Serialization) * Normalisation * Loading/Merging into CKAN Exisiting harvesters are at: * CSW: https://bitbucket.org/okfn/ckanext-csw/src/ * Scraper+CKAN: https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/" 1297684756000000 1311177705000000 988 task wwaites thejimmyg ckan-v1.4-sprint-2 closed fixed Get a better HTML view of a GEMINI document using XSL 1297685975000000 1299493598000000 990 defect rene.kapusta closed fixed umlauts in title / text "It's not possible to use umlauts (like ""CKAN Österreich"") in the title / text." 1297801036000000 1311180850000000 991 defect dread ckan-v1.3 closed fixed Checkbox defaults to True "Form for new package has CheckboxExtraField checked, when the value is False. (as used in ckanext-dgu package v3 form)" 1298035175000000 1298037717000000 992 defect rgrp rgrp ckan-v1.3 closed fixed Use X-forwarded-for whenever it is available to set remote IP address "At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache. We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available. Cost: 10m" 1298058532000000 1298060474000000 993 defect thejimmyg jason.kitcat@… ckan-v1.4-sprint-2 closed fixed Link on ckan.net footer needs changing "The link for About > 'Project Home Page' on the CKAN.net footer should be changed to point to: http://ckan.org/ The current link is broken. Thanks, Jason" 1298154006000000 1298373114000000 994 defect kindly kindly ckan-v1.4-sprint-3 closed fixed document a new migration methodology It is too easy to make a mistake with the migrations at the moment. A more systematic way of doing them is needed and this needs to be documented. 1298281192000000 1298912842000000 995 defect kindly kindly closed wontfix consolidate caching Try and minimise the amount of different caching methodologies, especially for api calls. 1298283499000000 1311179009000000 996 task kindly kindly ckan-v1.4 closed fixed Run some basic load testing. This will involve running a sample of real requests synchronously against real data. 1298283994000000 1300364398000000 997 defect kindly kindly closed duplicate change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1298288665000000 998 defect kindly kindly closed fixed change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1300364423000000 999 enhancement rgrp dread ckan-v1.4-sprint-3 closed fixed Factor out ckan.net theme changes to a separate theme repo and apply Rather than polluting the ckan core code base on ckan.net install with ckan.net specific changes these adaptations should be moved out into a dedicated ckan.net theme. 1298386729000000 1300707328000000 1000 enhancement kindly rgrp ckan-v1.4-sprint-3 closed fixed Remove call timings code "Not used and extra overhead. Very useful to strip this out and low cost. Places where it seems to be: * lib/base.py: c.time_call_started * ...." 1298489643000000 1298912726000000 1001 enhancement rgrp rgrp ckan-v1.4-sprint-4 closed fixed API should use normal user credentials if available "When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key. The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...). This is critical to incorporating any ajax editing into the frontend. As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author)." 1298489705000000 1301310351000000 1002 enhancement dread rgrp ckan-sprint-2011-12-05 closed fixed Remove changeset model code This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed). 1298489773000000 1322515696000000 1003 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed CKAN Javascript library and demonstration web interface "A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library. Why? * Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core * E.g. for specific communities e.g. geodata, science * Specialized tasks - multi-package editing * Very easy deployment and integration (e.g. can drop in to getthedata or other sites) " 1298490086000000 1300100411000000 1004 defect zephod dread ckan-backlog closed fixed Group creation instructions missing Need instructions on the group page to tell people they need to login to create a group. Someone must have deleted this. 1298559144000000 1323195485000000 1005 defect dread dread closed duplicate Editing in ckan missing publishers Package edit form at hmg.ckan.net has no publishers any more. 1298561348000000 1300100085000000 1006 enhancement kindly rgrp ckan-v1.4-sprint-4 closed fixed Deprecate stable branch "Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time). Cost: 10m (giving high priority because of low cost) (Assigning to dread as he has been managing the stable branch)." 1298624996000000 1300372286000000 1007 defect pudo pudo closed wontfix Backport WDMMG translations for extensions i18n "Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class. cf. https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16 " 1298630621000000 1340626203000000 1008 defect rgrp pudo ckan-v1.4-sprint-2 closed fixed eval() of user display name in template head "We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer. We should also have page fragment caching in Genshi, which is not currently implmented. cf. http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html" 1298632686000000 1298821826000000 1010 enhancement rgrp rgrp ckan-v1.4-sprint-2 closed fixed List CKAN users in WUI "Should have user listing at /user/ rather than user account page. * list users, sorted by number of packages contributed/edited * Move user home page to /user/{user-id} * Paginated" 1298649180000000 1298740889000000 1011 enhancement sebbacon sebbacon ckan-v1.4-sprint-3 closed fixed Make pluggable AuthzGroups implementation "I have a requirement to allow an external source to define the groups of which a user is a member. I propose to create an IAuthzUserGroups plugin interface that allows an extension to arbitrarily extend the list of AuthzGroups that a user is in." 1298819657000000 1299245206000000 1012 defect kindly kindly ckan-v1.4 closed fixed Add package revision history to api Revision history information is not accessible, dgu want this. 1298887351000000 1301943113000000 1013 defect sebbacon sebbacon ckan-v1.4-sprint-5 closed fixed Refactor & write tests for Wordpresser extension The Wordpresser extension seems to be working well for DataGM, but needs caching and tests. 1298887392000000 1301909717000000 1014 defect sebbacon sebbacon ckan-v1.4-sprint-3 closed fixed Decouple oAuth repoze extension from DataNO, write tests The current oAuth extension tightly couples repoze.who, ckan, and data.no. These components should be decoupled; they also need more tests 1298887493000000 1299245293000000 1015 defect kindly dread ckan-v1.4-sprint-3 closed fixed Editing group gives exception "Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?) Example page: http://ckan.net/group/edit/civil-society Exception: {{{ WebApp Error: : (IntegrityError) duplicate key value violates unique constraint ""group_revision_pkey"" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u""A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society"", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09 }}}" 1298898588000000 1299788821000000 1017 defect pudo sebbacon closed fixed Problem assigning users to authz groups through web interface "Against ckan-1.3.1, when I create an authz group called ""administrators"" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it. Each time I add additional users, the existing user on the list is replaced with the new one." 1299071127000000 1299668555000000 1018 task dread dread ckan-v1.4-sprint-3 closed fixed Remove gov form (It has been moved into ckanext-dgu) 1299072516000000 1299073340000000 1019 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Webhooks notification service "We propose the following push approach using individual packages: 1. New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form: { payload: The entity data as available from the REST API entity-type: 'Package', operation-type: 'create'|'update'|'delete' # one of these options } It will be sent as the body of the request with content-type set to application/json. 2. The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review. 3. Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model) 4. [optional] CKAN side will support configuring authorization for basic authentication if applied on WP side. 5. [optional] List queue status (including failures) on ckan adminstrative dashboard. " 1299166784000000 1299166930000000 1020 defect kindly closed fixed harvesting doc revision table add revisioning to harvested document table 1299205012000000 1300196215000000 1021 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Config option to disable OpenID HRI don't like federation, want to login normal way only. Make this a config option and perhaps even mess with runtime repoze config 1299492920000000 1299518828000000 1022 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Error reporting in CKAN worker API Report on errors by any worker daemons, send them out via E-Mail 1299493047000000 1299512991000000 1023 defect pudo pudo closed wontfix Re-queueing of worker requests with errors Schedule worker calls to be ran several times, with certain intervals. 1299493179000000 1340626231000000 1024 enhancement pudo pudo ckan-v1.4-sprint-3 closed duplicate Lock down all controller actions in CKAN When a user visits the site, he/she may only see a login box. 1299493428000000 1299668648000000 1025 enhancement dread dread ckan-v1.4-sprint-3 closed fixed Default authz can be set in config "Currently the default authz for a package is hard-coded to: {{{ , , }}} This should be configurable in the config, so that you can have a more locked down instance etc. " 1299596110000000 1299751045000000 1026 enhancement dread dread ckan-v1.4-sprint-3 closed fixed cli for creating users It's handy to be able to create users using the cli (e.g. dgu migration) 1299604652000000 1299605128000000 1027 enhancement pudo pudo ckan-v1.4-sprint-3 closed fixed Authorization checks on all controller actions We want to have authz checks on all controller actions so that we can lock down CKAN to a login-only mode. 1299666256000000 1299682082000000 1028 defect dread dread ckan-v1.4-sprint-3 closed fixed Open redirect in locale setting 1299759883000000 1299760360000000 1029 defect kindly ckan-v1.4-sprint-3 closed fixed synchronous search erroring when harvesting is run. Errors are caused when harvesting documents. This also makes the count not show up correctly on the ckan search page. 1299768337000000 1301311643000000 1030 defect amercader thejimmyg closed fixed Move harvesting out of the rest API 1299776418000000 1303117978000000 1031 enhancement johnlawrenceaspden rgrp ckan-v1.4-sprint-4 closed fixed User lookup API "Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users). * API location: /api/util/user/lookup?q=querystr&limit=10 * Return json objects containing {id: ..., name: ..., fullname: ...} * Put in a module called controllers/apiv2/user.py" 1299780419000000 1300101520000000 1032 enhancement rgrp rgrp ckan-v1.6 closed fixed [super] Resources in WUI "Add resources into Web User Interface. * Locate at: /dataset/{dataset}/resource/{id} * CRUD * Authorization Done: * ~~#945 - Richer resources - Resource Groups, new fields, improved UI~~ * ~~#1445 - Resource View page in WUI~~ * ~~#1450 - Dataset view pages to match~~ Moved to superticket #1506: * #978 - Edit Resource Extras in Web UI " 1299782021000000 1330348463000000 1033 defect dread dread closed fixed Register user with blank password causes 500 Go to http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error. 1299796274000000 1308310446000000 1034 defect dread closed duplicate Flash message cached " 1. Login (shows ckan home page with flash message ""Welcome back xyz"") 2. Click away, to ""Add package"" say 3. Click back to ckan ""Home"". It shows ""Welcome back xyz"" again - it shouldn't, I assume? " 1299845308000000 1320174353000000 1035 enhancement thejimmyg dread closed wontfix Form impressions given an ID "To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form. (Something to consider for the new form mechanism)" 1299857978000000 1338206251000000 1036 defect johnlawrenceaspden johnlawrenceaspden ckan-v1.4-sprint-4 closed fixed Add tests for three functions in ckan/model/user.py "create a new test file ckan/model/test_user.py add tests for the following three functions in ckan/model/user.py number_of_edits, number_administered_packages, search merged in in changeset 0046f83aedcf " 1300127840000000 1301304575000000 1037 defect amercader thejimmyg ckan-v1.4-sprint-6 closed fixed More Robust Harvesting for DGU "CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in #987. Some of the key issues: * Error reports do not currently contain the ID or title of the document with the error. * We only have ""added"" and ""error"" logging on jobs when we really need a report of ""added"", ""updated"", ""not changed"" and ""errors"" with the items in each referencing a real metadata document for which harvesting was attempted * We need deletion and editing of sources, without deleting the harvested documents or packages * We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once. * We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched. " 1300197602000000 1304937601000000 1038 enhancement dread dread ckan-v1.4 closed fixed Authz tool - operate on all packages at once Add 'package:all' to authz tool to allow mass changes of authz. 1300212788000000 1300212841000000 1039 enhancement dread dread ckan-v1.4 closed fixed Default user roles read from config (instead of being hard coded) 1300212822000000 1300212856000000 1040 defect thejimmyg dread ckan-v1.4 closed fixed File system mounted Public file path code adds a trailing , and thus adds / as a static file app in the Pylons middleware cascade. 1300213855000000 1328806824000000 1042 task dread dread closed fixed 'Ckanext' split-up "It's not good to have ckanext doing lots of different things with different dependencies. Split it off into: * ckanext-importlib etc. And then deprecate the ckanext repo itself." 1300293907000000 1300969865000000 1043 defect kindly closed fixed stop sqlalchemy message saying at least one scoped seession already present see summary. 1300310326000000 1300321033000000 1044 defect pudo dread closed fixed Sysadmins locked-out of API without Right: (visitor, SITE_READ, System) "The problem is that in ckan/controllers/rest.py the BaseApiController has this method: {{{ def __before__(self, action, **env): BaseController.__before__(self, action, **env) if not self.authorizer.am_authorized(c, model.Action.SITE_READ, model.System): abort(401, _('Not authorized to see this page')) }}} which works on the basis of your c.user, rather than your apikey. All API users are treated as visitors (since API users don't get a login cookie) and even a sysadmin's apikey is blocked unless there is a right for a Visitor to SITE_READ. Also needs tests. (Also, why is this restriction only on the API, package search, group index and tags and agroup index? I'm guessing SITE_READ is only for places where other authz don't apply, but maybe it should not be called 'SITE_READ' but 'OTHER_READ' or something?)" 1300358919000000 1302096155000000 1045 defect dread dread ckan-v1.4-sprint-4 closed fixed Group identified by ID in API "{{{ http://ckan.net/api/2/rest/group }}} returns group IDs but I can only reach a group by name: {{{ http://ckan.net/api/2/rest/group/economics }}} when I also want to get a group by ID: {{{ http://ckan.net/api/2/rest/group/04fb43d2-8ddf-4485-9bf5-66d47d3672f3 }}}" 1300360642000000 1300793261000000 1046 enhancement kindly thejimmyg ckan-v1.4-sprint-5 closed fixed Dictization and the new logic layer "The stages involved with doing this. * Convert model objects to standard dict format (DONE) * Convert standard dicts to current api formats (DONE) * Make standard dicts savable (DONE) * Validate standard dict format. (DONE) * Authorize actions" 1300364694000000 1302777668000000 1047 defect sebbacon sebbacon ckan-v1.4-sprint-4 closed fixed Package edit form claims you're not logged in at the end when you are "At the bottom, it says something like Author: Bob Bumgardner Since you have not signed in this will just be your IP address. Click here to sign in before saving (opens in new window)." 1300384556000000 1300387309000000 1048 enhancement dread dread ckan-v1.4-sprint-4 closed fixed Complete making groups versioned " * Deleting a group changes state to 'deleted' rather than purging it * Adding authz tests for deleted groups" 1300387655000000 1300702752000000 1049 defect dread closed invalid Fix database errors on offener.datenkatalog.at etc. "We're getting these sorts of exceptions from http://offener.datenkatalog.at/ like we had from ckan.net, which was due to small errors in migration scripts. Is it worth fixing this database, and in fact all the ckans on eu3 at once? {{{ WebApp Error: : (IntegrityError) duplicate key value violates unique constraint ""group_revision_pkey"" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, ... }}} Thoughts James and Rufus?" 1300444912000000 1323169424000000 1050 enhancement thejimmyg johnlawrenceaspden closed invalid Authz lib improvement and refactor of ckan/lib/authztool.py "Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface. The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code. Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests. all to go on a branch feature-1050-refactor-authtoolz == Optional extras == * Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py " 1300451937000000 1315394117000000 1051 defect sebbacon sebbacon ckan-v1.4-sprint-4 closed fixed Ability to set custom favicon Allow deployers to set a URL pointing to their own favicon 1300703160000000 1301305079000000 1052 defect dread dread ckan-v1.4-sprint-4 closed fixed Authz holes "No authz on: * Group creation/edit/listing * Package relationship create/edit/delete" 1300709144000000 1300895410000000 1053 defect dread dread ckan-v1.5 closed fixed Deletion in Model API "Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted. There is no way to delete objects at the moment - changes to 'state' are ignored in the API. Do we need an alternative way to purge objects in the API?" 1300790039000000 1310126546000000 1054 defect kindly dread ckan-v1.4-sprint-4 closed fixed Ordering of resources "Changing an old resource and creating a new resource on the end results in the old resource moving to the end in the ordering. This breaks tests: * (ckanext-dgu) ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadBasic.test_fields * (ckanext-importlib) ckanext/importlib/tests/test_loader.py:TestLoaderInsertingResources.test_0_reload You can make a ckan test break with this patch: {{{ diff -r e6643cf1324c ckan/tests/models/test_resource.py --- a/ckan/tests/models/test_resource.py Wed Mar 23 13:25:52 2011 +0000 +++ b/ckan/tests/models/test_resource.py Wed Mar 23 19:22:35 2011 +0000 @@ -297,6 +297,8 @@ 'url':self.urls[1], 'format':u'OTHER FORMAT', 'description':self.description, 'hash':self.hash, 'id':original_res_ids[2]}, + { #new + 'url':'new'}, ] pkg.update_resources(res_dicts) model.repo.commit_and_remove() }}} There seems to be a problem with vdm creating a replacement Resource for the old resource - because it has a duplicate position it is put to the end by the SQLAlchemy ordering_list function." 1300969236000000 1301305615000000 1055 defect dread dread ckan-v1.4-sprint-4 closed fixed @search_related tests not running "Tests marked decorated ""@search_related"" should only be run against postgresql, but in fact they don't get run at all. " 1300985228000000 1300992395000000 1056 defect dread pudo ckan-v1.4-sprint-6 closed fixed User links for OpenID users are broken "Use case: * Login using OpenID * Click on 'My account' - results in 404 Solutions: * User user.id instead of their name * Escape the URL properly. " 1301060249000000 1302882616000000 1057 defect dread closed fixed JSONP parameter isn't escaped "{{{ $ curl ""http://127.0.0.1:5000/api/rest/package/annakarenina?callback=