id type owner reporter milestone status resolution summary description PosixTime ModifiedTime 2783 enhancement shevski closed wontfix Way to generate homepage backgrounds for demos/ckan hosted "Really we want this for CKAN Hosted & for creating demos. Currently we're using http://leaflet.cloudmade.com/ - but by finding the area & taking screenshots. Would be better to interate with the app so that: a) this happens automatically and b) correct attribution will be visible on maps" 1343812931000000 1343818702000000 7 enhancement johnbywater johnbywater closed invalid View a release associated with a package See Update a release (ticket 14) for more information on what should be shown. 1152550022000000 1204136209000000 8 enhancement johnbywater johnbywater closed invalid View file "== As a == Visitor == I want to == View the files associated with a package. == Notes == * See also ticket:16 (updating a file). * Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release) " 1152550057000000 1250619147000000 12 enhancement johnbywater johnbywater closed invalid Create a release associated with a package "== As a == Authenticated user == I want to == Create a release associated with a package == So that == The package has a new release == Notes == * Permissions are as for updating a package ticket:10 * for details of release attributes see: updating a release ticket:14 " 1152550407000000 1204135071000000 13 enhancement somebody rgrp closed invalid Delete a release 1152550611000000 1218545330000000 14 enhancement johnbywater johnbywater closed invalid Update a release "== As a == Authenticated user == I want to == Update a release == So that == The release details are amended. Attributes of a release are: * name (may include '.' and '-' * nickname * date * description == Notes == * permissions: as for updating a package (ticket:10)" 1152550712000000 1204136163000000 15 enhancement johnbywater johnbywater closed invalid Create a file "== As a == Authenticated user == I want to == Create a file associated with a release == So that == The release has a new file associated with it == Notes == * Permissions are as for package updating (see ticket:10) * file attributes are described in file updating: ticket:16) * there are no attributes that are required to be supplied to create a file " 1152550770000000 1250619139000000 16 enhancement johnbywater rgrp closed invalid Update a file "== As a == Authenticated user == I want to == Update a file associated with a release == So that == The file attributes are changed. File attributes are: * (internal) id * description * source: (url usually) == Notes == * Permissions are as for package updating " 1152550796000000 1250619162000000 17 enhancement johnbywater johnbywater closed invalid Delete a file See ticket:16 (update a file) 1152550822000000 1250619180000000 18 enhancement somebody johnbywater closed invalid Purge a user (registered person) 1152550871000000 1199787607000000 19 enhancement somebody rgrp closed invalid Undelete a user (registered person) 1152550881000000 1199787631000000 22 enhancement somebody johnbywater closed invalid Update person attributes 1152551014000000 1199787243000000 23 enhancement somebody johnbywater closed invalid Request new password (or a reminder) if old has been lost 1152551256000000 1199786799000000 26 enhancement somebody johnbywater closed duplicate A registered person creates their own tags for a package 1152551351000000 1152555283000000 28 enhancement somebody johnbywater closed wontfix Provide a dump of all packages listed on the system in XML 1152551416000000 1200902911000000 31 enhancement somebody rgrp closed wontfix Add a comment to a package "== As a == Registered user == I want to == Add a comment to a package == Notes == * The comment should be tagged with the username of the person making the comment * Should we allow comments per release as well as per package? '''No''' for the time being. * Comments will be 'flat' and '''not''' threaded (at least for the present) * Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)" 1157374505000000 1185472236000000 34 defect somebody zool closed worksforme Page Not Found - returns with 200 rather than 404 as it should 1163008961000000 1253781550000000 49 enhancement rgrp rgrp closed invalid Filter Spam in Changes to CKAN Data "= As A = sysadmin = I Want To = Have revisions to the CKAN data filtered in order to reduce the spam in the system. = Details = In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution. Suggest we follow path of trac: http://trac.edgewall.org/wiki/SpamFilter Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing'). Main initial plugins would be: * regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...) * could augment using the badcontent list approach (can find list on e.g. moinmoin) * spambayes and/or akismet" 1204134691000000 1257244973000000 50 task rgrp rgrp closed fixed Investigate apt-get in more detail "Research apt-get especially wajig and python wrappers for apt-get. Post results on the wiki. " 1215456052000000 1267648356000000 73 enhancement rgrp rgrp closed invalid Put enquiry template in db so it can be edited by admins "Currently stored in the python controller file. This is not the right place and putting it in db will allow editing by admins. May also want to do this for the footer (also in the controller file). Details: * Will need to create a new domain object/db table. Suggest called miscellaneous and consist of key value pairs utilizing JSON type for values. Cost: 2h." 1247705786000000 1266510385000000 75 enhancement dread rgrp closed duplicate "Record and display package ""usage"" information" " * Number of package page visits on ckan (can we get this straight from google analytics) * Number of times url or download url is used - now ticket:937 (Record download stats for resoures) How do we do this? * Google analytics will miss a lot of this usage (and how do we get that data out anyway) * Could use javascript but again misses usage. * One option is to redirect link but that is kind of nasty (but may be only option ...) " 1247828785000000 1296341223000000 82 enhancement rgrp rgrp closed fixed Support diffing of versioned objects "Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs. Details: * Do we deal with m2m relationships (and m2one on the many side)? * What is diff? * For text fields output of python diff command * For non-text field guess just simple +/- for what was there before and now Cost: 4h (don't think this is a huge request)" 1248289499000000 1256565441000000 85 enhancement rgrp rgrp closed fixed Convert state from an object to an enumeration "No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible. Main hassle here is that it requires migration in all projects that use vdm. If we're going to do this we should do it sooner rather than later. Cost: 2h (+ 1h for a conversion method)" 1248339662000000 1263206391000000 86 enhancement rgrp rgrp closed fixed Support for stateful dict-like collections "Already support stateful list-like collections and should extend this to dict-like collections. (This is prerequisite for implementing versioned ""extra"" (key/value) attributes on packages in CKAN and elsewhere)." 1248430798000000 1249050202000000 87 enhancement rgrp rgrp closed duplicate Multiple download links Multiple download links, including links to mirrors and multiple formats/versions 1248693302000000 1258470719000000 89 enhancement nickstenning rgrp closed fixed Tag cloud for package tags Should not be hard to do (lots of existing libraries) but not sure that this is very important. 1248693939000000 1265892698000000 92 enhancement rgrp rgrp closed wontfix Add RDFa to package pages 1249049927000000 1297344859000000 95 enhancement rgrp rgrp closed fixed Add manifest support "Support for listing files (manifest) contained within a package. Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs." 1249981842000000 1251454716000000 96 defect rgrp rgrp closed invalid info command must give up to date information for python-type distributions "(2008-09-10) at present when setup.py is edited info command may not give up to date information if pkg-info (in egg.info) is not rebuilt. Suggest: info command needs to rebuilds pkg-info in python-type distributions. " 1249982410000000 1311176063000000 97 enhancement rgrp rgrp closed duplicate Do not create a distribution on a path is something already exists there "(2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option). Cost: 1h" 1249983557000000 1318181317000000 99 enhancement rgrp rgrp closed wontfix Test DbIndex with sqla 0.5 "Test DbIndex with sqla 0.5 (and get it working if it does not work). Cost: ?? Priority: low because DbIndex is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel). " 1249985537000000 1297081088000000 100 enhancement rgrp rgrp closed fixed Convert existing data on disk to a datapkg distribution "(2008-10-29) convert existing directory on disk to a datapkg distribution (PythonDistribution by default). This would be presented in the CLI as a convert command. Details: * Add a metadata file (setup.py), perhaps prompting for input * List all existing data in that directory Cost: 4h " 1249986628000000 1318181227000000 101 enhancement rgrp rgrp closed fixed (Improved) Download support " 1. Given a url download from that url to disk 2. Given a package use download url to download to disk Once on disk should uncompress (if necessary). == Details == * Should download either to tmp directory (defaulted in config) or a specified path on disk * In download would like to support as many types of target urls as possible: 1. Single files (usually compressed filesets but could be individual files) 2. Index pages listing files to download 3. Revision control repositories 4. ... This will obviously be a significant amount of work. Suggest: 1. Start off just supporting the first option. 2. Have something like a plugin system to handle new target types 3. Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc) Cost: 1d " 1249988866000000 1267648607000000 102 enhancement rgrp rgrp closed fixed Switch to using UUIDs for revision ids "This is better for long term sustainability, especially if we want to ""push and pull"" from unrelated repositories in the future. Details: * May wish to retain a number field updated in the usual incremental way (what's the cost/benefit here?)" 1250068175000000 1260285104000000 104 enhancement rgrp rgrp closed fixed View a package at a given version "As a user i want to see a package at ""version"" X (NB: not revision X). * When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version) * Implementation: * Find revision for this version (search revision history for when version field was last 0.7) * Show pacakge at that revision (as in ticket:103) * On history page also shows versions in list of revisions associated with the package " 1251292672000000 1311181303000000 106 enhancement dread rgrp closed duplicate Regularly convert CKAN data to RDF and put on Talis CC "Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons). Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC. In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required. Attached is the ruby script used by Talis for conversion Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)" 1251454474000000 1256140649000000 107 enhancement rgrp rgrp closed invalid Provide information about uploading material "Often people who are registering resources want to make the associated material available. We should: * Provide a dedicated upload page giving instructions as to upload process 1. Explain we don't store the data itself on CKAN. Suggest uploading to a store somewhere and then linking using download_url (that is its purpose) 2. Upload to * Talis CC for RDF * archive.org * grid.okfn.org * link to this page from next to download_url item on new package page In long run (now ticket:186) we may automate this by providing a ckan upload facility which caches the data and then reuploads it to relevant service (disadvantage is ""we"" are the owner of the data on that service ...)" 1251466274000000 1296341644000000 109 enhancement rgrp dread closed fixed Tag search in the REST API "Similar to ticket:108 but for tags: /api/search/tag?q For tags search is extremely simple since you can only search by name. == Possible Extras == * Allow ""exotic"" ordering of results e.g. ordering by number of packages with that tag. * Return this number with tag list." 1251976297000000 1291829457000000 132 defect rgrp dread closed fixed Security hole - read package/group list (REST) "Using REST interface you can list packages and groups without authorization being checked. Can be fixed using more advanced query to check authz." 1254389493000000 1273254514000000 133 defect rgrp dread closed fixed Security hole - search package/group (WUI & REST) "Using WUI or REST interface you can search packages and groups without authorization being checked. On the REST interface you can also read all the attributes of the packages using the 'all-fields' option. Can be fixed using more advanced query to check authz. " 1254390168000000 1273253977000000 137 enhancement rgrp dread closed duplicate User has editable home page " * Generic text box for markdown about the user 'About' Model's user table reflects these: * 'about' attribute " 1254741703000000 1254741830000000 142 defect pudo dread closed fixed Update User object with additional fields such as 'nickname' "Suggested fields: * id, name (username/profile name), openid, email, password, extras (JsonType) * + current items: apikey, about, created) * current contents of name is usually openid and should migrate to openid field (could match based on starting with http://) * May also want a fullname field (or perhaps that can just go in extras - may be better to have its own field as useful to search on this ) Related changes (probably separate tickets): * Improve user home page to include this additional information (and have it be editable) * Use user 'name' (username) for home page links (rather than current user/) * Where we show lists of revisions, each revision author is now a link to the author's user page. Related to ticket:136 and ticket:138." 1255010314000000 1289219098000000 146 defect pudo rgrp closed worksforme Problem when an admin user logs in between commencing editing and saving " 1. Visitor starts editing a package (or creates a new package) 2. Visitor logs in before previewing or committing 3. Visitor is an admin (or sysadmin) 4. Preview or commit fails Reason (conjecture): the form we use for package editing is now dependent on whether you have admin privileges (we allow editing of state by admins). When you login the old form is now incorrect and this results in the formalchemy fieldset breaking." 1255013773000000 1291829862000000 165 enhancement rgrp dread closed invalid Use 'tag:' instead of 'tags:' for wui package search "Because people might try and do ""tags:gov sweden"" when ""tag:gov sweden"" makes more sense (sweden is searched for in all fields in these cases). But in REST i/f search allow ""tag""=""gov"" OR ""tags""=""gov sweden"" in parameters." 1256200526000000 1311181391000000 168 enhancement rgrp dread closed duplicate Show admins for a group in group view 1256291481000000 1257414795000000 169 enhancement dread dread closed duplicate Package derivations "A 'Derived' relationship can be applied from one package to another. e.g. sussex-demography is derived from census-2001 'Derived' relationship is: * directional * many:many * stateful 'derived' table columns: * id (primary key) * source_package (foreign key) * result_package (foreign key) * description (markdown text) Further tickets: * WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y * WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate * REST if - expose reading and writing this property" 1256304927000000 1266928708000000 176 enhancement dread dread closed duplicate Package dependencies "(Related to ticket:169 - Package derivations) A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.) e.g. london-traffic-visualisation depends on road-map 'Dependency' relationship is: * directional * many:many * stateful 'dependency' table columns: * id (primary key) * dependent (foreign key) * dependency (foreign key) Further tickets: * WUI - package view - have list of dependencies (do not need to list packages which depend on this one) * WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package') * REST api - expose reading and writing 'depends on' property. === Issues === * How do we deal with dependency at a particular version?" 1257162812000000 1266928721000000 177 enhancement rgrp dread closed invalid Service documentation "This is docs for users of a CKAN service (e.g. ckan.net), as opposed to someone installing / administering the software. (The latter has docs on knowledgeforge. Editability - wiki?" 1257243285000000 1273050236000000 181 enhancement rgrp jwyg closed invalid New 'stats' box on right hand sidebar "With up to date statistics such as: * total packages * total tags * total users * most active users * last update * etc. Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184" 1257534385000000 1296339510000000 183 enhancement rgrp rgrp closed worksforme Browse packages by rating At moment order packages by title. 1257534606000000 1290604779000000 184 enhancement dread rgrp closed fixed Stats page "Create /stats/ page displaying main statistics, e.g: * Most highly rated packages * Most edited packages * Largest groups * Top tags (by packages) * Package addition rate * Users with most packages Related to ticket:181 - Stats side-bar == Details == * Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display) * For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see * http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py * http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html" 1257534756000000 1266837414000000 185 enhancement rgrp dread closed invalid Package form field hiding "Make parts of the package form hidden by default, to make it look simpler. Users are invited to click a button to reveal the less important fields. Fields hidden by default: version, author, author_email, maintainer, maintainer_email" 1257763437000000 1291830039000000 186 enhancement rgrp rgrp closed duplicate Automated upload to archive.org s3 "(Follows on from ticket:107). We want to provide facility for users to automatically upload material. * Create an upload page * Store the material on archive.org using s3 - http://www.archive.org/help/abouts3.txt * Main issue is how we cache large datasets ..." 1257803430000000 1296341182000000 190 defect pudo dread closed fixed Package comments "Cost 7 days When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API. The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user. Example at bottom of package page: '''Leave a comment:''' ||Subject|| _________ || ||Comment|| _________ || [[br]] ||Submit button|| '''Comments:''' ||'''Explanation doc'''|| ||''Posted on 25h May 2009 by !http://bertdavies.myopenid.com''|| ||It says on the pollution data web page that not all the stations have a CO2 sensor, so you have to extrapolate from the ones that do. See my visualisation of CO2 across London for an idea of what you can do: bertdavies.com/pollution-2008.jpg|| ||'''More info'''|| ||''Posted on 24th May 2009 by !http://ronsmith.myopenid.com''|| ||Excellent data, but why is there no value in the CO2 column for some of the testing stations?|| = Implementation details: = Comments table is with columns: ||id || package_id || date (date) || comment (multi-line text)|| " 1258387105000000 1280820852000000 192 enhancement rgrp dread closed fixed Time-related package field "Cost - 2 days The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09). Example part of package: '''Date:''' 5/10/09 [[br]] or [[br]] '''Date:''' 1/1/09 – 1/4/09" 1258388163000000 1291733895000000 193 enhancement rgrp dread closed wontfix Searching by time-related field "Cost - 2 days Search interface has new options to filter and sort the results by the time-related field of the package. Search options are included in both Web UI and Search API. The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. Need to decide for a time-related field value that is date range, what date is used for the search. Example search parameters: || reldate-range=5/4/09- || Exclude packages related to earlier than 5/4/09 || || reldate-range=5/4/09-5/12/09 || Exclude packages related to date outside of 5/4/09-5/12/09 || || order_by=reldate || Sort by date package is related to. Defaults to newest first. || || order_by=reldate-newest || Sort by date package is related to, newest first. || || order_by=reldate-oldest || Sort by date package is related to, oldest first. || Related to ticket:192" 1258388169000000 1340626463000000 198 enhancement rgrp dread closed fixed Change package and tag ids to uuids "See how we did it already for other things. Note: on ckan.net older PackageRevision.id might not be identical to Package.id but this may need sorting at this point." 1258980613000000 1266837606000000 199 defect rgrp casbon closed fixed Fix failing tests and provide documentation on how to run tests in datapkg "As of 80:099bdc5b07a6 datapkg has 4 test fails and 1 test error. There is no documentation on how to run them, so I am using 'python setup.py test'. {{{ datapkg.tests.test_cli.TestCLI.test_ckan ... FAIL datapkg.tests.test_cli.TestCLI.test_walkthrough ... FAIL datapkg.tests.test_index.TestDbIndex.test_get ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/index.py:97: SADeprecationWarning: Use session.add() self.session.update(pkg) FAIL datapkg.tests.test_index.TestDbIndex.test_get_when_loaded_as_new_and_init_not_called ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/tests/test_index.py:57: SADeprecationWarning: Use session.expunge_all() self.index.session.clear() FAIL }}} " 1259087361000000 1267649255000000 200 defect rgrp rgrp closed invalid PythonDistribution write method should write all available metadata Current write method just creates a bare bones package and does not write any of the available metadata other than name. 1259229583000000 1311176118000000 205 enhancement rgrp dread closed wontfix Custom package importer "Importer function is exposed to logged in users. Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'. * Basic is as it is currently * Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords. All fields should be in step with the government custom input form. For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text. Estimate: " 1260197819000000 1297068450000000 209 enhancement rgrp rgrp closed fixed Validate email address when creating an enquiry p:wui 2009-09-09 validate email address when creating an enquiry 1260288671000000 1260288728000000 210 enhancement johnbywater johnbywater closed invalid User registration "Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid. " 1260292578000000 1266509788000000 211 enhancement johnbywater johnbywater closed invalid User login "Native login scenario: Complete login form, require email address. Any session is resumed. Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff. See CKAN or Microfacts for usage of repose with openid." 1260292624000000 1266509848000000 219 enhancement nickstenning dread closed wontfix UI Review - Package " * Propose three-column package listing: two lists of highest-scorers on some metric and last column is an invitation to register a new package, with a little copy explaining what's involved and why you'd want to. * Add in search widget " 1260879947000000 1294914901000000 222 defect nickstenning dread closed fixed UI Review - Package history "Package history page [/package/history/mypackagename] * ""Revisions"" is redundant * ""Compare"" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it! * Ditto to overall ""Recent Changes"" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links. " 1260880198000000 1282909280000000 225 enhancement rgrp dread closed invalid UI Review - swap URL order "Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure. " 1260880487000000 1311178276000000 228 enhancement rgrp rgrp closed duplicate Deal with duplicate packages This needs to be thought out ... 1262085763000000 1290596875000000 232 enhancement rgrp rgrp closed invalid Add strapline to CKAN logo "Strapline (?): Sharing, discovery and reuse of data and content Need to consult on this. Cost: 1h" 1262975291000000 1297246297000000 234 enhancement memespring rgrp closed wontfix UI Review - Autocomplete package names & tags in search Broken out from ticket:216 1263056678000000 1338206486000000 236 enhancement johnbywater johnbywater closed fixed The system shall present to all users a form for starting data openness enquiries 1264157100000000 1264157324000000 237 enhancement johnbywater johnbywater closed fixed The system shall accept submissions from authenticated owners of activated accounts using the data openness enquiry form by presenting the enquiry summary and prompting for confirmation 1264157377000000 1264157420000000 238 enhancement johnbywater johnbywater closed invalid The system shall accept submissions from unauthenticated users of the data openness enquiry form by creating an anonymous pending enquiry action, and by redirecting the user to login form, passing a reference to the pending action 1264157814000000 1266509806000000 243 defect rgrp dread closed worksforme Repetition of number of packages in a group "http://ckan.net/group/publicdomain has: ""There is 16 package in this group. There are 16 packages in this group.""" 1265750250000000 1266837796000000 244 defect rgrp dread closed fixed Several links preceded by link URL "e.g. Text on ckan.net home page: ""All material available under an /licenseopen license"" Also seen on the package edit page: ""/user/loginClick here to sign in"" I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime." 1265750419000000 1271248968000000 245 enhancement rgrp rgrp closed duplicate Support for composite primary keys "Problem here is that foreign key then becomes ""complicated"" (composite). * Could also deprecate continuity_id field in favour of the basic foreign key on ie" 1265882630000000 1297066620000000 246 enhancement rgrp rgrp closed duplicate Support for primary key not named id At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys) 1265882862000000 1297066757000000 249 enhancement rgrp dread closed invalid Regex search "Search of package name and title (and other fields) using regular expressions. Current example use-case: Wanting to specify packages with title beginning with 'B'. Issues: * Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields? Implementation: * Postgres reg ex searching detailed here: http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE" 1265994509000000 1311182450000000 252 enhancement dread johnbywater closed invalid Change revision object so that it has parent(s) attribute 1266519767000000 1296477560000000 262 enhancement rgrp rgrp closed fixed Mark package as duplicate of another package "Need a way to mark packages as duplicate. Ideas/Questions: * Could work off a tag (meta-duplicate or somesuch) * What happens if we want to delete the package (to we leave a place-holder, do we merge content?) " 1267139682000000 1296468392000000 268 defect rgrp dread closed duplicate Select groups in Package edit form 1268068896000000 1285070682000000 269 enhancement rgrp dread closed fixed Improve gov package form "If the notes field could use a WYSIWYG editor with word cleanup this would really help users who may well be pasting in text from Word (and I guess may have been approved or written by someone else) having auto-complete on tags would both make the system easier to use - and reduce the risk of synonymous tags being created inadvertently. Does the department drop down options list interact with user permissions - so that users who only have rights to one department only see that option ? Similarly for licenses - can this reduce to the set of allowed options for this user/ What level of validation is there on the fields?" 1268220853000000 1291897538000000 270 enhancement johnbywater johnbywater closed fixed Refactor ckan/lib/importer and ckanclient/loader. "There is code to create packages in-process (ckan/lib/importer) and via HTTP (ckanclient/loaders/base). There is also code to read different kinds of package source (Google Spreadsheets spreadsheet, other spreadsheet). Propose to refactor all this to have template method in base command class (subclasses may implement command line interface, graphical user interface, or Web interface), where the template method calls on aggregated strategy objects for: reading raw spreadsheet data from spreadsheet application; for inferring CKAN package entities from the raw spreadsheet data; and for writing package entities to CKAN service. Spreadsheet readers to be factored to allow for variations of spreadsheet application. Raw data inferrers to allow for variations in spreadsheet cell structure. CKAN writer strategies to allow for either CKAN client or direct linking. Actual cases then supported by specifying existing strategies (or defining new ones) in a script that invokes the command class." 1268671804000000 1290596640000000 272 enhancement rgrp rgrp closed fixed Atom/RSS feeds for individual tags and groups "I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group. * This would allow people to embed feed widgets in other places" 1268988652000000 1340631775000000 274 defect rgrp rgrp closed fixed Allow searching by any extra field "At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...) Cost: 1h" 1269034562000000 1287402800000000 275 defect dread rgrp closed fixed Add forms renderer for JsonType fixing bugs in admin interface "This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type) Cost: 0.5h {{{ from ckan.model.types import JsonType forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer }}}" 1269122740000000 1281002082000000 282 enhancement thejimmyg dread closed wontfix Provide diverts when package name changes When a package's name is changed, references to the old name (in the WUI and REST) are redirected to the new name (assuming the old name is not being reused by another package). 1270659003000000 1338206417000000 283 enhancement rgrp dread closed wontfix Manage deletions of unwanted packages "== Use case == As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion. == Suggested solution == In the package view side-bar, there is a note: ""To have this package completely removed, contact the [ca.ckan.net administrators admin@ca.ckan.net]."" == Other solutions == A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill. c.f. http://en.wikipedia.org/wiki/Deleting_an_article" 1270660210000000 1311325526000000 285 enhancement rgrp assigned Paginate list of packages on tag read page "Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ... " 1270664606000000 1340631923000000 291 defect rgrp dread closed worksforme Investigate search options encoding "On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in. {{{ diff -r 813ad8b5de0b ckan/lib/search.py --- a/ckan/lib/search.py Mon Mar 01 22:23:36 2010 +0100 +++ b/ckan/lib/search.py Thu Apr 15 19:11:42 2010 +0200 @@ -63,6 +63,9 @@ def query(self, options): '''For the given search options, returns a query object.''' self._options = options + # temporary hack! + if self._options.q: + self._options.q = self._options.q.encode('utf8') general_terms, field_specific_terms = self._parse_query_string() if not general_terms and \ }}} " 1271351670000000 1273254895000000 294 enhancement thejimmyg dread closed duplicate Add/remove extra fields in Package edit form Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources. 1271756591000000 1291830960000000 296 enhancement johnbywater johnbywater closed duplicate Commit CKAN revisions to changeset system 1272279521000000 1294407032000000 297 enhancement johnbywater johnbywater closed duplicate Update CKAN repository from changeset system 1272279556000000 1294407051000000 298 enhancement johnbywater johnbywater closed duplicate Pull changesets from remote CKAN instance 1272279591000000 1294407080000000 299 enhancement johnbywater johnbywater closed duplicate Merge diverging lines of changesets 1272279698000000 1294407099000000 301 enhancement rgrp assigned Package discussion pages "A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package. It provides a way for people to make suggestions about a package without needing access to main package." 1272301033000000 1340632055000000 306 enhancement rgrp rgrp closed duplicate datapkg build command "Need to be able to build a distribution. Need: * new 'build' command * specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution. " 1272474212000000 1318181194000000 308 enhancement rgrp rgrp closed duplicate Autocomplete package names & tags in package search "Extracted from ticket:216. Dubious of its merit." 1273050549000000 1275302577000000 309 defect pudo dread closed fixed Tag list shows unused tags "Deleted tags are visible on the /tag page and are searchable. Delete tags are ones that were in use once and then discarded, so this includes ones which are not used on any packages any more, including misspelling, mistakes, and deprecated tags. It would makes sense to remove them. Example tag that is viewed at /tag and searchable: http://ckan.net/tag/2rand[0" 1273247648000000 1280743432000000 312 defect rgrp dread closed invalid Race condition creating PackageRating "On ckan.net there are a few packages which cause a 403 exception when you try to rate them: http://www.ckan.net/package/rate/coins-data?rating=3 The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError: Multiple rows returned for one() It looks like this package got rated twice in quick succession, creating two similar PackageRating objects. This race condition needs to be solved and these particular rating objects repaired." 1273482785000000 1311176173000000 314 defect johnbywater johnbywater closed fixed Bugs getting revisions from the REST API "Bug report regarding getting revisions: Getting revisons by ID (on the latest ID) GET ""http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"" Returns a whole load of stuff (it's also quite slow - about 10 seconds) I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API? GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45"" Returns the empty string - I'd expect an empty array ie [] GET ""http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45"" Returns an internal server error 500 - I think it should probably be ""bad Request"" 400 (the date is invalid) " 1273743755000000 1276523983000000 316 defect rgrp dread closed fixed Search URL escaping "If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error. e.g. http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk fjdkf2B´gfhgfkgf{gpk returns this exception: {{{ URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&search=Search+Packages+%C2%BB Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module repoze.who.middleware:107 in __call__ << wrapper = StartResponseWrapper(start_response) app_iter = app(environ, wrapper.wrap_start_response) # The challenge decider almost(?) always needs information from the >> app_iter = app(environ, wrapper.wrap_start_response) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:50 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.package:52 in search << collection=query, page=request.params.get('page', 1), items_per_page=50 ) # filter out ranks from the query result >> items_per_page=50 Module webhelpers.paginate:333 in __init__ << self.item_count = item_count else: self.item_count = len(self.collection) # Compute the number of the first and last available page >> self.item_count = len(self.collection) Module webhelpers.paginate:204 in __len__ << def __len__(self): return self.obj.count() # Since the items on a page are mainly a list we subclass the ""list"" type >> return self.obj.count() Module sqlalchemy.orm.query:1094 in count << q = q.params(params) q = q._legacy_select_kwargs(**kwargs) return q._count() def _count(self): >> return q._count() Module sqlalchemy.orm.query:1103 in _count << """""" return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key)) def _col_aggregate(self, col, func, nested_cols=None): >> return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key)) Module sqlalchemy.orm.query:1125 in _col_aggregate << if self._autoflush and not self._populate_existing: self.session._autoflush() return self.session.scalar(s, params=self._params, mapper=self.mapper) def compile(self): >> return self.session.scalar(s, params=self._params, mapper=self.mapper) Module sqlalchemy.orm.session:635 in scalar << engine = self.get_bind(mapper, clause=clause, instance=instance) return self.__connection(engine, close_with_result=True).scalar(clause, params or {}) def close(self): >> return self.__connection(engine, close_with_result=True).scalar(clause, params or {}) Module sqlalchemy.engine.base:834 in scalar << """""" return self.execute(object, *multiparams, **params).scalar() def statement_compiler(self, statement, **kwargs): >> return self.execute(object, *multiparams, **params).scalar() Module sqlalchemy.engine.base:844 in execute << for c in type(object).__mro__: if c in Connection.executors: return Connection.executors[c](self, object, multiparams, params) else: raise exceptions.InvalidRequestError(""Unexecutable object type: "" + str(type(object))) >> return Connection.executors[c](self, object, multiparams, params) Module sqlalchemy.engine.base:895 in execute_clauseelement << else: keys = None return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params) def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None): >> return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params) Module sqlalchemy.engine.base:907 in _execute_compiled << context.pre_execution() self.__execute_raw(context) context.post_execution() self._autocommit(context) >> self.__execute_raw(context) Module sqlalchemy.engine.base:916 in __execute_raw << self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context) else: self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) def _execute_ddl(self, ddl, params, multiparams): >> self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) Module sqlalchemy.engine.base:958 in _cursor_execute << self.engine.logger.info(repr(parameters)) try: self.dialect.do_execute(cursor, statement, parameters, context=context) except Exception, e: self._handle_dbapi_exception(e, statement, parameters, cursor) >> self.dialect.do_execute(cursor, statement, parameters, context=context) Module sqlalchemy.engine.default:133 in do_execute << def do_execute(self, cursor, statement, parameters, context=None): cursor.execute(statement, parameters) def is_disconnect(self, e): >> cursor.execute(statement, parameters) UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128) }}}" 1274265928000000 1291831177000000 319 defect dread dread closed fixed Local Authority license misnamed in migration script 18 "Migration script 18 converts license ""OKD Compliant::Local Authority Copyright with data.gov.uk rights"" into ""uklocalauthority-withrights"" when it should be ""localauth-withrights"", according to the license service SoS 2. This causes clients difficulties looking it up. " 1274349714000000 1274366882000000 321 enhancement thejimmyg johnbywater closed duplicate Delegate authentication to Drupal "When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model. The Drupal front-end shall have: 1. Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER. 2. Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions. 3. Access denied page - fixed location, static resource, gently indicates what has happened, and how to ask for permission. The CKAN slave edit page shall: 1. Try to detect a Drupal session key (passed as cookie or as request param). 2. Redirect to Drupal login page if no session key. 3. Check authorisation if session key is found. 4. Redirect to access denied page if session key not authorised. 5. Present the Package edit page. 6. Reject unauthenticated or unauthorised edit submissions. 7. Snag invalid edit submissions from authenticated and authorised users. 8. Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end. " 1274705234000000 1291831399000000 327 defect pudo pudo closed wontfix Create a web hook worker for CKAN " This will be useful as an example consumer of the new queue notifications (#325). * A user can register any URL to be notified upon an event. * Possible extensions: GET with ID only vs. POST with serialized object * E-Mail notifications are also hooks, essentially. We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes)" 1274807361000000 1296467361000000 330 defect dread dread closed fixed getdata/ons timezone not recognised on non-British servers "The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error: DateConvertError: Could not read date as ISO format ""%a, %d %b %Y %H:%M:%S %Z"". Date provided: ""Mon, 04 Jan 2010 09:30:00 BST"" It turns out that it only recognises local names of timezones. The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems." 1275300271000000 1275303122000000 332 defect pudo rgrp closed fixed Autocomplete tag on click "During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box. " 1275302887000000 1280743320000000 334 defect pudo dread closed fixed Wrong link for package feed icon "This is a problem with the package page e.g. http://ckan.net/package/open-election-data-project . The feed icon links to [1] and the text next to it ""Subscribe"" links to [2]. Surely these should be the same? The second link seems to right one to me. * [1] http://ckan.net/package/open-election-data-project?format=atom * [2] http://ckan.net/package/history/open-election-data-project?format=atom&days=7" 1275407445000000 1280743667000000 339 defect johnbywater closed fixed Remove download_url from package data format in API Version 2, but maintain the old data format in the old version of the interface (API Version 1) 1275901499000000 1278074705000000 341 enhancement dread closed fixed Web UI accepts package IDs in URLs "(in the same way that we can refer to packages in the API by ID as well as name.) == As a == client of CKAN == I want to == link to a package page in the UI, referring to it by ID" 1276162400000000 1277483030000000 342 enhancement dread dread closed fixed JSONP parameter in API "== As a == CKAN client using JQuery == I want to == call the CKAN API and instead of receiving back JSON I get JSONP. i.e. ""%s(%s)"" % (callback, json_content) == Suggested implementation == All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide: http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch == Test == import re import unittest def test_jsonp_callback(): response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com&callback=jsoncallback') match = re.match('jsoncallback\(.*\);',response) self.assertTrue(match) response = self.app.get('/api/search/resource/?url=http://www.scraperwiki.com') match = re.match('jsoncallback\(.*\);',response) self.assertFalse(match) I think the point needs to be made that JSONP only works for GET requests and not POST/PUT/DELETE, so there needs to be a check for that in the _finish_ok method. (thanks to Donovan Hide for test)" 1276166426000000 1276278485000000 343 defect johnbywater dread closed fixed Packages referred by ID in API When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID. 1276180179000000 1278066420000000 344 defect dread closed fixed REST Create package with incorrect format gives 500 error "PUT to /rest/api/package of {{{ {""name"": ""name"", ""resources"": [""someurl.com/data""] } }}} (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message. Exception: {{{ Module ckan.controllers.rest:154 in create << if register == 'package' and not subregister: fs = ckan.forms.get_standard_fieldset() request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data) fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session) elif register == 'package' and subregister in model.PackageRelationship.get_all_types(): >> request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data) Module ckan.forms.package_dict:88 in edit_package_dict << for res_dict in value: res_dict_str = {} for key, value in res_dict.items(): res_dict_str[str(key)] = value resources.append(res_dict_str) >> for key, value in res_dict.items(): AttributeError: 'unicode' object has no attribute 'items' }}}" 1276341172000000 1277477712000000 345 requirement dread johnbywater closed invalid metastable, stable & ultrastable branches shall be automatically built and tested 1276523083000000 1291831615000000 347 enhancement johnbywater johnbywater closed fixed The system shall present the package edit form in an API 1276523692000000 1277820440000000 348 enhancement johnbywater johnbywater closed fixed The system shall accept package edit form submissions in an API 1276523793000000 1277820496000000 352 enhancement dread dread closed wontfix Package notification worker - sends XML-RPC "== As an == external front-end == I want to == be notified (by XML-RPC) about package creations and updates. == Implementation == 1. A message queue worker waits for package update notifications 2. On reception, it constructs XML detailing the changes and PUSHes it to a configured URI. Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter. Config - in the CKAN config will be: * URI to callback to * API version to use (version 2 gives packages referred by ID not name)" 1276597996000000 1286375870000000 353 defect dread closed fixed SOLR search indexing "== As a == SOLR instance == I want to == keep my search index of CKAN packages up-to-date == Implementation == * Using asynchronous event notifications * Running in a separate process to CKAN" 1277123480000000 1280756399000000 354 defect johnbywater johnbywater closed invalid Collect together requirements and top-level design for user/package 'groups' "Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff. http://knowledgeforge.net/ckan/trac/wiki/AccessControl Do we add these into user-role table somehow or new table? To present this to team" 1277131335000000 1282908983000000 355 defect rgrp rgrp closed fixed Dashes versus underscores in package names Sort out how we deal with dashes versus underscores in package names. 1277221996000000 1311177552000000 357 defect johnbywater johnbywater closed fixed Fix API documentation, to correct misleading HTTP_AUTHORIZATION statement. "Sean Burlington wrote: > I recently came across a documentation issue > > http://knowledgeforge.net/ckan/doc/ckan/api.html > ----- > The key should be passed in the API request header: > Header Example value > HTTP_AUTHORIZATION fde34a3c-b716-4c39-8dc4-881ba115c6d4 > > ----- > > But the header name is actually just 'AUTHORIZATION' > > For example to set it in PHP > > curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION: > xxxxx-xxxx-xxxx-xxxx-xxxxxxx')); > " 1277459886000000 1277461466000000 362 defect dread dread closed invalid Ratings should not be created with a GET "== Background == In the Web UI, when you rate a package it simply links to something like: http://ckan.net/package/rate/mke-liquor-licenses?rating=3 This creates a GET request. This is bad because: * Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this) * There are occasions when we want to make a CKAN instance read-only, so we put a Apache instruction in. But the database may still get written for these ratings. * Best practise for web requests is for GET to be a read-only request." 1278925451000000 1311176564000000 365 enhancement dread dread closed fixed ResourceNotifications If you change a resource then you not only get a PackageNotification, but also a ResourceNofication. 1279037411000000 1279300621000000 367 enhancement dread dread closed fixed Notfication monitor Runs on the commandline and prints out notifications. 1279303310000000 1279303693000000 368 defect anonymous closed wontfix 500 Server error when creating package " I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response: ""publish: New Zealand Coastline (new-zealand-coastline)"" opening connection to nz.ckan.net... opened <- ""POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n"" <- ""{\""name\"":\""new-zealand-coastline\"",\""title\"":\""New Zealand Coastline\""}"" -> ""HTTP/1.1 500 Internal Server Error\r\n"" -> ""Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n"" -> ""Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n"" -> ""Vary: Accept-Encoding\r\n"" -> ""Content-Type: text/html; charset=iso-8859-1\r\n"" -> ""Cache-Control: proxy-revalidate\r\n"" -> ""Content-Length: 640\r\n"" -> ""Connection: close\r\n"" -> ""\r\n"" reading 640 bytes... -> ""\n\n500 Internal Server Error\n\n

Internal Server Error

\n

The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.

\n

Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.

\n

More information about this error may be available\nin the server error log.

\n
\n
Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80
\n\n"" read 640 bytes Conn close The server should not respond with a 500 error if the package was created. " 1279622462000000 1291831811000000 373 requirement dread johnbywater closed fixed Sort out gov daily script "Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally. Requested by DGU." 1279885887000000 1286376176000000 381 defect johnbywater closed invalid Test defect please ignore 1280345937000000 1294410466000000 394 task johnbywater johnbywater closed duplicate Fix munin on DGU (?) 1280485351000000 1294407189000000 396 task johnbywater johnbywater closed fixed Integrate Forms API into Drupal 1280854226000000 1282324934000000 401 task pudo pudo closed fixed Provide base worker as queue consumer We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app) 1281018724000000 1286189243000000 403 requirement pudo closed fixed Create web service for OFS storage "Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources. * relates to #307" 1281346110000000 1287392451000000 407 requirement dread closed wontfix Link to Scraperwiki "== As a == user browsing a CKAN package which has a resource in a bad format == I want to == be alerted to the possibility of scraping it on scraperwiki == Implementation == This should be done in a CKAN extension. It should appear in a side-bar on the package read page. If the package does not have a resource with scraperwiki.com in the url, then it displays this message: Title: Scrape it[[BR]] Image: (Scraperwiki dumper truck)[[BR]] Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can [http://scraperwiki.com/market/request/ request] this data is scraped or even [http://scraperwiki.com/editor/ write the scraper] yourself. " 1281348368000000 1330769956000000 412 requirement johnbywater closed fixed Data format for package importing shall be documented "Requirements: * cater naturally for multiple packages with multiple resources * international encoding * large number of packages" 1281430809000000 1289822092000000 418 task dread closed invalid With COI, get SOLR working 1281431691000000 1291637919000000 423 task dread johnbywater closed fixed Add examples to documentation about uploading data 1281435260000000 1282812768000000 430 task thejimmyg pudo closed wontfix Upgrade CKAN to Pylons 1.0 "Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism. http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed. " 1281962485000000 1298283075000000 431 task pudo pudo closed wontfix Solr index testing tool There seem to be a few conditions under which either queue processing or the indexer fail in their current state. To get a more systematic picture of these failures, we should have a small testing tool to compare the index to the database of a live CKAN instance. 1281963532000000 1295260262000000 439 task dread closed fixed Plan server requirements for DGU "Using use cases for security etc. Draw diagram to satisfy use cases. Communicate to Paul and therefore TSO." 1282224452000000 1294413903000000 440 task dread dread closed duplicate Write and pass comprehensive performance tests "Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes." 1282226932000000 1294417436000000 444 task dread closed duplicate Discuss package relationships ideas with JF "* Create test data on visible ckan * Discuss with JF" 1282299238000000 1294414008000000 446 task dread closed invalid Server configuration query (?) " * Ensure use case is in design - ""an admin writes to the CKAN API"" (and see JB notes on paper) * Communicate how this is planned to be achieved in design to JF" 1282299385000000 1294414077000000 447 task memespring johnbywater closed fixed Presentation on Package edit form (labels IE7, horiz alignment in chrome) Follow up with NS and RP. 1282299423000000 1311182895000000 448 story thejimmyg dread closed invalid [Assist with] editing packages through Drupal 1282299478000000 1294417061000000 449 task wwaites johnbywater closed invalid Start email thread about RDF in CKAN with JF and WW 1282299482000000 1311182945000000 450 task dread closed fixed Assist with proxying of static media for forms John did this a while ago. 1282299573000000 1287997487000000 452 task dread closed fixed Assist with using package update error form 1282299647000000 1287997540000000 453 task thejimmyg johnbywater closed invalid Analysis and planning for UKLII 1282299725000000 1294411374000000 455 task johnbywater closed invalid SOLR - suggest 1 pager about how system would work Either CKAN writes to SOLR and Drupal reads from SOLR, or CKAN writes to SOLR and Drupal reads SOLR via CKAN API (so search resource locations are unaffected). 1282299913000000 1291637172000000 458 task rgrp johnbywater closed wontfix Request to DGU for access to database machine 1282299950000000 1294415537000000 460 defect pudo johnbywater closed fixed State field changed by non-sysadmin "This package: http://ckan.net/package/dbtune-audioscrobbler was: 1. created by Richard (logged-in) 2. edited by Richard (logged-in) (According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just ''? 3. pudo changed the state back to active Similarly an incident with bibbase package where field set to '' (see http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409) Need to investigate how this got changed, fix if necessary and report back to Richard. Note that package 'admins' as well as sysadmins can change the state of a package (though note that bibbase did not appear to have an owner). Suggested solution (for setting to ''): * Ensure in ckan/forms.py that there is a validator for state field that ensures only set to valid values. * Check that we do not allow state to be changed in the api except by package owner or sysadmin" 1282300002000000 1311176868000000 464 task rgrp dread closed duplicate Request dgu db server access 1282306104000000 1282325194000000 474 task dread dread closed wontfix Questions related to fi.ckan.net "Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&A to the ckan-discuss list. 1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding. To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters. 2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar UnicodeDecoding errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en. 3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ? 4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN. 5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ? Best Regards, Henri" 1282312255000000 1294916760000000 483 story johnbywater closed invalid Restricted total read-write access to catalogue Web UI 1282422801000000 1294417216000000 484 story johnbywater closed invalid Restricted partial read-write access to catalogue Web UI 1282422858000000 1294417248000000 485 story johnbywater closed duplicate Performance beats QoS criteria 1282425219000000 1294411946000000 487 story johnbywater closed duplicate Notify SOLR service of model events 1282425910000000 1291639404000000 488 requirement johnbywater closed wontfix Catalogue service shall notify RDF service 1282426021000000 1320930240000000 489 story wwaites johnbywater closed fixed Notify RDF service of model events 1282426051000000 1294416189000000 490 requirement johnbywater closed fixed The system shall support registering a harvest source via the CKAN API 1282426875000000 1286200804000000 497 story johnbywater johnbywater closed duplicate "Respond to CSW ""GetRecords"" request" 1282427334000000 1294407718000000 498 story johnbywater closed invalid "Search packages within location ""bounding box""" 1282427412000000 1294412520000000 499 story thejimmyg johnbywater closed wontfix Registration and management of UKLII related code lists, controlled vocabularies and other forms of reference/static data 1282432571000000 1296593038000000 503 task dread rgrp closed fixed Provide exemplar package relationships for some packages " 1. Find some relationships. 2. Add them via API 3. Notify DGU of completion and point to exemplars" 1282659631000000 1314031851000000 505 enhancement dread rgrp closed fixed DGU fieldset supports v3 of DGU metadata specification 1282660416000000 1298368280000000 506 task dread rgrp closed fixed Plan transition to DGU metadata-fields v3 1282660490000000 1287080989000000 507 story johnbywater closed fixed Upgrade a CKAN service 1282661646000000 1282909852000000 509 story dread dread closed invalid ONS data imported daily 1282665870000000 1291734435000000 510 enhancement dread dread closed fixed Setup cron for ons import 1282665939000000 1294138332000000 512 story dread closed duplicate User creates package via API with incorrect core fields specified 1282754750000000 1294917121000000 513 task dread closed wontfix Creating a package with API should warn if there are incorrect fields "When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK. " 1282754802000000 1302774329000000 526 requirement pudo pudo closed wontfix Integrate validation tool into the registry "Assumes 8 is done: 8. Tool for validation an IATI XML record (referenced by a URL) for compliance with the IATI specification * Outside of scope of registry * Multiple validators with different levels of rigour" 1282894300000000 1340626152000000 527 story pudo rgrp closed fixed Create domain model and load sample data 1282899025000000 1293097531000000 528 task pudo rgrp closed fixed Setup demonstration instance at iati.ckan.net 1282899110000000 1283536475000000 529 task pudo rgrp closed fixed Obtain sample metadata 1282899161000000 1283536554000000 530 task pudo pudo closed fixed Map IATI attributes to CKAN metadata schema "From lopad: {{{ Publishing Entity (often/always? equal to the logged in user) * Domain logic: only can add entries for ""yourself"" Donor Country Activity period: period to which this record relates Verification status: enumeration of statuses (checked, not checked etc) Resource links: to the actual IATI record Number of activities: ... Date record updated: Date data updated: License: Need this field even if it may be a standard license }}}" 1282899187000000 1287392210000000 531 task pudo rgrp closed fixed Load sample IATI metadata " * Parse data * Load via API * Develop test for load via API (can be local or remote -- with paste webfixture may be easier)" 1282899244000000 1283536676000000 537 task wwaites wwaites closed duplicate Caching and Performance improvement "There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests. Introducing caching has to be done carefully and should be done in a ''standards compliant manner''. == General strategy == * Where possible, cache output within the pylons app (beaker). * Facilitate external caching in an end-user's web browser or a caching proxy * Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes). * When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data. == Tasks == These tasks should be broken into sub-tickets: * caching of parts of templates that are expensive to render (package list, tag list, group list) * caching of entire output using beaker particularly for API read operations. * need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches. * cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: http://www.mnot.net/cache_docs/#CACHE-CONTROL * Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well * Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner == Future Work == * Investigate ckanclient library maintaining a local cache as a web browser would * Investigate using a CDN like Google Storage or Amazon for serving cached data." 1283184362000000 1311178929000000 538 task rgrp johnbywater closed wontfix Update CKAN Trac ticket report queries "Update report queries with more recent versions (below are report queries shipped with trac 0.11.7): NB: Report 1 has already been done. id | query ----+--------------------------------------------------------------------------------------------------- 1 | : SELECT p.value AS __color__, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY CAST(p.value AS integer), milestone, t.type, time : 2 | : SELECT p.value AS __color__, : version AS __group__, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time : 3 | : SELECT p.value AS __color__, : 'Milestone '||milestone AS __group__, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time : 4 | : : SELECT p.value AS __color__, : owner AS __group__, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time : 5 | : SELECT p.value AS __color__, : owner AS __group__, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : description AS _description_, : changetime AS _changetime, reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time : 6 | : SELECT p.value AS __color__, : t.milestone AS __group__, : (CASE status : WHEN 'closed' THEN 'color: #777; background: #ddd; border-color: #ccc;' : ELSE : (CASE owner WHEN $USER THEN 'font-weight: bold' END) : END) AS __style__, : id AS ticket, summary, component, status, : resolution,version, t.type AS type, priority, owner, : changetime AS modified, : time AS _time,reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), : (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC : 7 | : SELECT p.value AS __color__, : (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS __group__, : id AS ticket, summary, component, version, milestone, : t.type AS type, priority, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE t.status <> 'closed' AND owner = $USER : ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time : 8 | : SELECT p.value AS __color__, : (CASE owner : WHEN $USER THEN 'My Tickets' : ELSE 'Active Tickets' : END) AS __group__, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (COALESCE(owner, '') = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time " 1283190081000000 1294414537000000 540 story wwaites rgrp closed fixed Implement caching in a systematic manner 1283242705000000 1302694845000000 541 enhancement wwaites rgrp closed fixed Full-page caching even for logged in users "The simplest way to improve performance on specific pages is to implement full page caching. This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have ""You are logged in as: {username}"". Similarly we change the visibility of some material depending on the status of users: * Visibility of Edit or Authz link on packages and groups" 1283243139000000 1294924872000000 542 task wwaites rgrp closed fixed 'Show and Tell' page customization using cookie + js "To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: http://www.ibm.com/developerworks/web/library/wa-rails2/ In the first place we just need to apply this to: * You are logged in as * Package/group edit/authz tabs" 1283244432000000 1283278942000000 543 task wwaites rgrp closed duplicate Investigate partial page caching and edge-side includes "Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page. Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish. Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat." 1283244784000000 1311178918000000 563 requirement thejimmyg johnbywater closed duplicate Support a minimal CSW server interface or export to GeoNetwork 1284033576000000 1296592472000000 573 story johnbywater closed duplicate Add metadata entity to harvesting queue 1284045353000000 1284220987000000 574 story johnbywater closed duplicate Create UKLII package with attributes from remote metadata record 1284045805000000 1284222410000000 580 story johnbywater closed duplicate Write (create or update) CKAN package for metadata document 1284210730000000 1284223068000000 613 task johnbywater johnbywater closed Store result of schema validation check on metadata document object 1284218828000000 1286798466000000 614 task johnbywater johnbywater closed invalid Acquire local copy of XML schema for remote metadata documents 1284219077000000 1288038898000000 615 task johnbywater johnbywater closed invalid Acquire local implementation of UKLP schematron "We ""just"" need a schematron.validate(metadata) method. :-)" 1284219174000000 1288038926000000 616 task johnbywater johnbywater closed invalid Store result of schematron validation check on metadata document object 1284219246000000 1288038907000000 637 task johnbywater closed invalid Read UKLII package data from given metadata document 1284222536000000 1284223104000000 646 requirement johnbywater closed wontfix Web UI shall support registering harvest sources 1284740303000000 1294413500000000 649 story pudo closed fixed Introduce Authorization Groups in model "Authorization Groups: * Rename UserObjectRole to SubjectObjectRole * Create database migration to do the rename in DB * Make sure all the tests are passing again (Doing this first seems more stable since we first want to ""re-gain"" exisiting functionality) * Create authz.py and model tests for authorization_group-based auth * tests: functional: create fixtures, do actual requests lib-tests: test for membership addition, removal, group-right additions affecting all members, group-right removal, group rights affecting non-members. # * Adapt model to get these passing * Integration into existing controllers should be negligible but may want ot have tests - do not want that. would mean design is broken * Need to create a new controller for editing authz groups (will have same permissions as any other object -- Reader, Editor, Administrator) " 1284889884000000 1284889983000000 651 requirement pudo closed fixed Improve javascript helper for package/user addition on groups/authzgroups This needs to be abstracted and we still need to be able to remove items before saving. 1284972211000000 1311171104000000 659 enhancement nils.toedtmann dread closed fixed Check CKAN instance works automatically Auto way to check web and API interface of a CKAN instance basically works. Several gotchas can be quickly determined, such as logging in, search not working. Needs to be configurable per site basis. 1285348333000000 1311183031000000 660 requirement dread closed invalid Check CKAN instance works As an admin I want to check a CKAN instance works having just upgraded it or configured it. 1285348463000000 1311183115000000 668 defect thejimmyg Colin Calnan closed invalid Caching issues on API v1 "It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues. Steps to test: 1. Modify a dataset on datadotgc.ca, redirects to CKAN 2. On save, redirects to http://www.datadotgc.ca/update/geogratisnat_hydrography_v100 which in turn redirects to http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100 3. You can see that the Dataset has not updated correctly. Run a check on the API v1 - http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100 the updates are not present 4. Check the v2 of the API - http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100, the updates are present. 5. Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either." 1285953542000000 1311176649000000 669 task pudo pudo closed fixed Refactor Solr to become a generic worker 1286189594000000 1293057161000000 670 task pudo pudo closed fixed HowTo: Creating a sysadmin 1286190347000000 1293056844000000 671 task pudo pudo closed fixed Implement and show user names in WUI 1286190454000000 1293056852000000 673 task johnbywater closed Construct and send CSW GetRecordById request 1286214712000000 1286786758000000 674 task johnbywater closed Extract metadata documents from CSW GetRecordById 1286214780000000 1286787032000000 680 task dread dread closed fixed Prepare for DGU tech meeting 1286386074000000 1287141809000000 683 story dread dread closed fixed metastable, stable & ultrastable smoketests run by buildbot 1286388134000000 1287142289000000 684 task dread dread closed fixed metastable tested on buildbot 1286388181000000 1286452710000000 685 task dread dread closed fixed stable tested on buildbot 1286388206000000 1287997008000000 686 task dread dread closed wontfix ultrastable tested on buildbot 1286388225000000 1287997047000000 694 bug wwaites dread closed fixed No postgres tools for current version Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them. 1286977052000000 1287087916000000 695 bug pudo dread closed fixed Search indexing broken on ckan.net "e.g. searching for 'buddhist' or 'sanskrit', you don't get this newly created package: http://ckan.net/package/digitalsanskritbuddhistcanon" 1286991201000000 1287766973000000 699 defect pudo TimDavies closed fixed Paging search results re-sets Open Only and Downloadable Only flags When getting a search results page (e.g. http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1) and then using the on-page links to page through results, the next link / numbered page links go to http://ckan.net/package/search?q=development&page=2 rather than http://ckan.net/package/search?q=development&open_only=1&downloadable_only=1&page=2 meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings. 1287143698000000 1296499061000000 714 bug johnbywater closed fixed DGU package form shall have a read only field for ??? attribute So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission. 1287581408000000 1291733788000000 717 bug pudo pudo closed fixed "Fix and validate ""setup_default_user_roles"" in IATI" 1287583892000000 1290005099000000 729 bug pudo closed fixed Assets to be loaded from assets.okfn.org, not m.okfn.org Move from hetzner box to s3. 1287736519000000 1288012854000000 730 task rgrp closed duplicate Back up package data from all CKAN packages to storage.ckan.net "Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net. * Naming scheme? * Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? * What happens if names change * File: filename? hash? * Store hash back on ckan instance?" 1287737109000000 1291139609000000 735 task dread dread closed fixed Investigate 'deduping' (ONS import) "ww 22/9/10: (discussion of ticket:509) This leads us to deduping which should be the next order of business oin this thread... ww 20/10/10: if a dataset is missing a department and the ons script makes a new version of it with a department we now have two reccords for the same dataset. these should be manually merged - need a merge tool where you can tell it that two datasets are the same." 1287748786000000 1292957248000000 740 requirement thejimmyg johnbywater closed duplicate Get copy of harvested metadata for a given package 1287779799000000 1296592889000000 743 story johnbywater closed wontfix Create harvest source registration via Web UI 1288012860000000 1294412807000000 744 story johnbywater closed wontfix Update harvest source registration via Web UI 1288012960000000 1294412813000000 746 story johnbywater closed wontfix Request new harvesting job via Web UI 1288013198000000 1294412827000000 747 story johnbywater closed wontfix Track harvesting jobs via Web UI 1288013217000000 1294412834000000 748 story johnbywater closed duplicate Link new sample package to previous sample package of continuous series 1288013849000000 1294412976000000 749 story johnbywater closed duplicate Fold up continuous series in search results behind newest sample package 1288014002000000 1294412986000000 750 enhancement thejimmyg johnbywater closed duplicate Get CSW records modified since given time 1288014402000000 1296592940000000 751 story johnbywater closed duplicate Get harvested document for a given package 1288014518000000 1288014616000000 761 story dread closed invalid Read-only CKAN for maintenance Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. 1288090334000000 1288091977000000 762 story dread closed invalid Permanently Read-Only CKAN instance "A CKAN is used just for distributing metadata. Updates may still arrive through direct db manipulation, e.g.: * another (but writable) CKAN instance is connected to the same db * restoring database dumps from another CKAN db " 1288090578000000 1288091982000000 775 task pudo dread closed fixed Search warning "We're getting this warning a great deal on live servers. Is these really a sign of the system not operating correctly or can we reduce the level to an INFO? e.g. on hmg.ckan.net: {{{ 2010-10-29 17:12:08,262 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,333 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,375 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,406 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,480 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,613 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position }}}" 1288372692000000 1295260144000000 776 enhancement rgrp dread closed duplicate Avoid generating vdm warnings "We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided? {{{ 2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object 2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object 2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object 2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object 2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object 2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object }}}" 1288372797000000 1297066840000000 799 requirement johnbywater johnbywater closed duplicate WAF harvesting shall be optimised in respect of reharvesting only records that have changed 1289483264000000 1294232675000000 802 story johnbywater johnbywater closed duplicate Save last harvested time on source 1289484226000000 1294233294000000 803 story thejimmyg johnbywater closed invalid Auto-generate next migration version unit and test stub files As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually. 1289486678000000 1314031451000000 820 defect dread closed fixed Exception browsing /user/apikey when not logged in "Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in. {{{ WebApp Error: : 'NoneType' object has no attribute 'apikey' URL: http://test-hmg.ckan.net/user/apikey }}}" 1290006079000000 1311325226000000 822 enhancement Javier de la Cueva closed fixed Docs: Configuration as site-available instead of site-enabled for apache "The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net " 1290107103000000 1290506354000000 823 defect dread dread closed fixed Etags cache doesn't update when only non-core package fields changed. "1. Create a package. 2. Edit package and add a resource. 3. View package in browser - no sign of the resource - ERROR. 4. View package in API - resource is there. Reason: hash doesn't change because package.revision only takes account of core fields." 1290505078000000 1290506116000000 824 enhancement dread dread closed fixed 'Getdata' moved out of CKAN "Data importers should use API instead of object model directly: * saves load on server * much better error checking and authentication * eating our own dogfood (tests the API well)" 1290595507000000 1290595559000000 825 enhancement dread dread closed fixed Document fabric tool 1290609160000000 1290624449000000 827 defect rgrp dread closed fixed metadata_created_and_modified error "Intermittent failure on ckan default branch. Looks like brittle test. {{{ ====================================================================== FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified ---------------------------------------------------------------------- Traceback (most recent call last): File ""/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py"", line 186, in runTest self.test(*self.arg) File ""/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py"", line 250, in test_02_metadata_created_and_modified assert out == exp, (out, exp) AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938)) }}} http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio" 1290678624000000 1290766239000000 830 enhancement rgrp closed fixed Themes for CKAN "Already have the basic capacity using *_extras paths for templates and public directory (see http://wiki.okfn.org/ckan/doc/theme). However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: http://packages.python.org/Flask-Themes/" 1290760612000000 1311180263000000 843 enhancement wwaites closed fixed user list ui for group curation " * perhaps we should show the openid as well to distinguish between users with the same name. * perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name. * also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it." 1291484245000000 1319721601000000 844 defect wwaites wwaites closed fixed bad mail config ckan.net/ckan.org "ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example: {{{ B30E6D9F 2873 Sat Dec 4 14:05:40 MAILER-DAEMON (delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out) no-reply@ckan.net }}} both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null" 1291556753000000 1296340486000000 849 defect pudo closed duplicate Import extra CSS from main template The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import. 1291714923000000 1291715179000000 855 defect pudo dread closed invalid Authenticator tests The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests. 1291723967000000 1311176988000000 856 defect wwaites dread closed invalid Tests for caching Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved? 1291724064000000 1311177085000000 857 defect dread closed invalid Tests for CLI Test coverage for ckan.lib.cli is 34%. Need to improve that. 1291724131000000 1311182581000000 858 defect dread dread closed fixed Tests for diff Test coverage for ckan.lib.diff is 24%. Need to improve 1291724171000000 1291734297000000 859 defect pudo dread closed invalid Tests for User model Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement? 1291724348000000 1311177461000000 861 defect wwaites wwaites closed invalid unicode errors creating rdf output http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d 1291726125000000 1311168845000000 863 enhancement memespring memespring closed wontfix prompt users for missing package information e.g. license, owner etc 1291729787000000 1338206455000000 868 enhancement sebbacon thejimmyg closed fixed Test improvements "The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach. We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!) Some suggestions for achieving this include: * Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database * Not setting up and tearing down the database so frequently. " 1291987811000000 1294753596000000 871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too "The infamous [http://www.exim.org/lurker/message/20101207.215955.bb32d4f2.en.html exim bug] only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers. We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not. BTW did we already run a rootkit checker like [http://rkhunter.sourceforge.net/ Rootkit hunter] on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could. " 1292264117000000 1296340558000000 872 enhancement johnbywater dread closed fixed ONS loader assigns packages to same package even when dept name changes Solutions: ResourceSeriesLoader allows synonyms for values. 1292424112000000 1292957110000000 873 defect dread dread closed fixed ONS loader assigns packages to same package even when dept name added manually "e.g. national_child_measurement_programme had no department but ""Information Centre for Health and Social Care"" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_." 1292437253000000 1292939274000000 881 defect thejimmyg fccoelho closed invalid http authentication requirement when installing I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code. 1293021889000000 1296335072000000 882 defect pudo pudo closed wontfix Update i18n user docs and backport genshi i18n:domain fixes from WDMMG 1293192044000000 1340626094000000 896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances "Based on the ideas in: * DistributingChanges * SyncingInstances Merging the tickets #296, #297, #298, #299 into this one. Related ideas include: * Using dcat to exchange data * Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance This should be considered a wishlist item for the timebeing. " 1294406999000000 1300217375000000 897 defect dread dread closed duplicate """New package"" highlighted in Navigator when searching" "On ckan.net 1.3.1b, click on ""Search"" in the Navigator. Now both ""Search"" and ""New Package"" are highlighted." 1294648815000000 1294914333000000 898 defect rgrp dread closed fixed Changes stored indefinitely "Every change to every object is being stored in memory, which could add up to quite a lot of memory. This fixes it by making sure the objects are in a weakref. https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f" 1294659490000000 1294662408000000 899 enhancement rgrp dread closed fixed Optimise check_real_change "There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e From David Raznick" 1294659583000000 1294916615000000 900 defect dread closed fixed Intermittent error with new revision not having id "There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e. From David Raznick" 1294659696000000 1294916587000000 901 enhancement rgrp dread closed fixed vdm support for sqla 0.6 and sqlite 1294660144000000 1294662466000000 917 enhancement kindly kindly closed fixed Turning off Autoflush in SQLAlchemy "David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior. I have a patch in https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default. This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet). Things to know about using flush: * Most of the time nothing will change. A commit implies flush. * You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query. * A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated. * You can use them as a limited form of nested transaction. It increased the performance on the tests by about 5%. There are less flushes altogether so is faster. I do not see any downsides." 1295279694000000 1295280232000000 918 defect memespring dread closed invalid Package preview doesn't show tags or groups "1. Edit any package 2. Hit 'preview' Notice that the preview at the bottom doesn't show what tags or groups the package is in. This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview. One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision." 1295346496000000 1315911507000000 920 defect thejimmyg hellmann@… closed fixed empty tags are kind of confusing "http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed. It should be purged !!!" 1295355280000000 1300319140000000 921 enhancement sebbacon thejimmyg closed fixed Refactor DGU Harvesting "Refactor harvesting code to simplify. " 1295389237000000 1300197629000000 922 enhancement kindly kindly closed duplicate Make Resources first class entity. "Why? * Will make it easier to give resources their own form, independent of packages. * Makes uploading, harvesting and previewing data cleaner as its separate from the packages. * Gives the option of sharing a resource between packages. * Leads the way to supporting storing richer semantic information against them." 1295713858000000 1320664187000000 923 defect rgrp dread closed worksforme Search box doesn't work in leaderboard page "1. Go to: http://ckan.net/stats/leaderboard#content 2. In the far top-right of the browser, select the search box in 'water'. 3. Press enter to search. Nothing happens. Tried in: chrome, firefox" 1295867328000000 1340632144000000 930 defect wwaites dread closed fixed call_timing files created for every request "On a production server we produce literally millions of little files saying how long every single request took to process. They are here: {{{ hmg.ckan.net.2/pylonsdata/call_timing }}} They were added by ckan/lib/base.py in cset:da438a9085d3. I don't believe anyone uses these stats." 1296061721000000 1297066292000000 934 enhancement kindly closed invalid Key value store for extensions to store data "We need to allow plugins to store information. Each plug-in should have its own name space and allow any data to be set. There are two options. * make a key value table in ckan that will hold random information. * use redis (or other key value store) If in sql suggest schema of form: * (namespace, obj_id, key, value [, value_type]) * value is json" 1296228758000000 1323171047000000 935 defect pudo pudo closed fixed """groups"" field bug in IATI" "An internal server error was reported when trying to create a new package (not when editing): WebApp Error: : (IntegrityError) null value in column ""name"" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'} The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the ""groups"" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server. Possible causes include: * Stale pyc file * Threading issue * Dependency issue As the problem is gone for now, it cannot be further investigated atm. " 1296302915000000 1306778682000000 939 enhancement rgrp closed fixed Notification bar at top of site shown to new users pointing them to simple instructions "On http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions. Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better. == Implementation == * Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing) * FAQ/instructions page (use the new wiki?) == Depends == * ticket:938 == Estimate == * Cost: 1h" 1296399008000000 1323171158000000 947 enhancement rgrp closed duplicate Move the assignment of roles to a package on package creation to an extension "At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded. This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833). An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own)." 1296557478000000 1314031310000000 951 defect adrian.pohl@… closed invalid Can't add a package to group "I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is ""history"") nor on the group page." 1296726886000000 1314031006000000 957 enhancement kindly kindly closed wontfix "new ""kind "" field added to resources" "The new field was outlined in http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html it says '''Can be ""data"", ""metadata"", ""service"", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.'''" 1296821398000000 1314218701000000 971 enhancement sebbacon sebbacon closed duplicate Support middleware as a plugin I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system. 1297277473000000 1299245064000000 987 defect pudo pudo closed duplicate Common harvesting framework "We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: * CSW/WFS for INSPIRE/UKLII (yields CKAN packages) * Catalogue scraping for LOD2 experiments (yields RDF graphs) * Atom/DCat for LOD2 production (yields RDF graphs) * OAI-PMH for http://datadryad.org/ and other dspace (yields CKAN packages) We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. In general, such a system would have the following stages: * Source selection: find what to download/scrape/harvest/parse * Index retrieval (i.e. package index) * Item retrieval (i.e. package entity) * (Optional: Serialization) * Normalisation * Loading/Merging into CKAN Exisiting harvesters are at: * CSW: https://bitbucket.org/okfn/ckanext-csw/src/ * Scraper+CKAN: https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/" 1297684756000000 1311177705000000 990 defect rene.kapusta closed fixed umlauts in title / text "It's not possible to use umlauts (like ""CKAN Österreich"") in the title / text." 1297801036000000 1311180850000000 995 defect kindly kindly closed wontfix consolidate caching Try and minimise the amount of different caching methodologies, especially for api calls. 1298283499000000 1311179009000000 997 defect kindly kindly closed duplicate change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1298288665000000 998 defect kindly kindly closed fixed change create on cli to upgrade When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. 1298287981000000 1300364423000000 1005 defect dread dread closed duplicate Editing in ckan missing publishers Package edit form at hmg.ckan.net has no publishers any more. 1298561348000000 1300100085000000 1007 defect pudo pudo closed wontfix Backport WDMMG translations for extensions i18n "Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class. cf. https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16 " 1298630621000000 1340626203000000 1017 defect pudo sebbacon closed fixed Problem assigning users to authz groups through web interface "Against ckan-1.3.1, when I create an authz group called ""administrators"" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it. Each time I add additional users, the existing user on the list is replaced with the new one." 1299071127000000 1299668555000000 1020 defect kindly closed fixed harvesting doc revision table add revisioning to harvested document table 1299205012000000 1300196215000000 1023 defect pudo pudo closed wontfix Re-queueing of worker requests with errors Schedule worker calls to be ran several times, with certain intervals. 1299493179000000 1340626231000000 1030 defect amercader thejimmyg closed fixed Move harvesting out of the rest API 1299776418000000 1303117978000000 1033 defect dread dread closed fixed Register user with blank password causes 500 Go to http://ckan.net/user/register and fill in all the fields apart from password. On submit you get 500 error. 1299796274000000 1308310446000000 1034 defect dread closed duplicate Flash message cached " 1. Login (shows ckan home page with flash message ""Welcome back xyz"") 2. Click away, to ""Add package"" say 3. Click back to ckan ""Home"". It shows ""Welcome back xyz"" again - it shouldn't, I assume? " 1299845308000000 1320174353000000 1035 enhancement thejimmyg dread closed wontfix Form impressions given an ID "To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form. (Something to consider for the new form mechanism)" 1299857978000000 1338206251000000 1042 task dread dread closed fixed 'Ckanext' split-up "It's not good to have ckanext doing lots of different things with different dependencies. Split it off into: * ckanext-importlib etc. And then deprecate the ckanext repo itself." 1300293907000000 1300969865000000 1043 defect kindly closed fixed stop sqlalchemy message saying at least one scoped seession already present see summary. 1300310326000000 1300321033000000 1044 defect pudo dread closed fixed Sysadmins locked-out of API without Right: (visitor, SITE_READ, System) "The problem is that in ckan/controllers/rest.py the BaseApiController has this method: {{{ def __before__(self, action, **env): BaseController.__before__(self, action, **env) if not self.authorizer.am_authorized(c, model.Action.SITE_READ, model.System): abort(401, _('Not authorized to see this page')) }}} which works on the basis of your c.user, rather than your apikey. All API users are treated as visitors (since API users don't get a login cookie) and even a sysadmin's apikey is blocked unless there is a right for a Visitor to SITE_READ. Also needs tests. (Also, why is this restriction only on the API, package search, group index and tags and agroup index? I'm guessing SITE_READ is only for places where other authz don't apply, but maybe it should not be called 'SITE_READ' but 'OTHER_READ' or something?)" 1300358919000000 1302096155000000 1049 defect dread closed invalid Fix database errors on offener.datenkatalog.at etc. "We're getting these sorts of exceptions from http://offener.datenkatalog.at/ like we had from ckan.net, which was due to small errors in migration scripts. Is it worth fixing this database, and in fact all the ckans on eu3 at once? {{{ WebApp Error: : (IntegrityError) duplicate key value violates unique constraint ""group_revision_pkey"" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, ... }}} Thoughts James and Rufus?" 1300444912000000 1323169424000000 1050 enhancement thejimmyg johnlawrenceaspden closed invalid Authz lib improvement and refactor of ckan/lib/authztool.py "Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface. The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code. Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests. all to go on a branch feature-1050-refactor-authtoolz == Optional extras == * Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py " 1300451937000000 1315394117000000 1057 defect dread closed fixed JSONP parameter isn't escaped "{{{ $ curl ""http://127.0.0.1:5000/api/rest/package/annakarenina?callback=