http://localhost/report/22 Trac Report - https://github.com/roskakori/tratihubis/blob/master/query_tickets.sql en-us http://assets.okfn.org/p/ckan/img/ckan_logo_shortname.png http://localhost/report/22 Trac v0.12.3 rgrp <h2 id="Title">Title</h2> <p> Visitor visits service </p> <h2 id="Asa">As a</h2> <p> Visitor (general web user) </p> <h2 id="Iwantto">I want to</h2> <p> Visit the website of the service (frontpage) </p> <h2 id="Sothat">So that</h2> <ul><li>See the front page and discover information about the service </li><li>Perform various activities related to the service such as registering and finding packages, registering as a user and logging in </li></ul> http://localhost/ticket/1 http://localhost/ticket/1 Report johnbywater <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> Be able to register as a user on the service </p> <h2 id="Sothat">So that</h2> <p> One can login and be an authenticated user and perform the actions that only authenticated users can perform </p> <h2 id="Notes">Notes</h2> <ul><li>Registration involves the provisional of a minimal set of user details, namely: <ol><li>username (unique in lifetime of the service) </li><li>fullname </li><li>email </li><li>password </li></ol></li></ul> http://localhost/ticket/2 http://localhost/ticket/2 Report johnbywater <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> Login to the service </p> <h2 id="Sothat">So that</h2> <p> I am now an authenticated user and may perform the actions that only authenticated users may perform </p> <h2 id="Notes">Notes</h2> <ul><li>Login by providing username and password </li><li>Login screen should also include a link or information about registration </li><li>If username do not match (or the username does not exist) a login failure is noted and the visitor is re-presented with the login screen as well as information on how to register </li></ul> http://localhost/ticket/3 http://localhost/ticket/3 Report johnbywater <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> Locate packages that I am, or might be, interested in </p> <h2 id="Sothat">So that</h2> <p> I know what packages exist and can read the information associated with those packages </p> <h2 id="Notes">Notes</h2> <p> Find a package by searching by (or some combination thereof): </p> <ol><li>tag </li><li>string contained in title or description </li><li>license </li></ol> http://localhost/ticket/4 http://localhost/ticket/4 Report johnbywater <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> Browse a list of packages resulting from a search or browse request (see other use cases) </p> <h2 id="Sothat">So that</h2> <p> I can select one of the packages to view in more detail (-&gt; viewing an individual package <a class="closed ticket" href="http://localhost/ticket/6" title="enhancement: View a package (closed: fixed)">ticket:6</a>) </p> <h2 id="Notes">Notes</h2> <ul><li>When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list </li><li>The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (&lt;= 50). Response time should be kept reasonable </li></ul> http://localhost/ticket/5 http://localhost/ticket/5 Report johnbywater <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> View the details of an individual package </p> <h2 id="Sothat">So that</h2> <p> I know more about the package. Attributes available are detailed in updating a package (../UseCase12) </p> http://localhost/ticket/6 http://localhost/ticket/6 Report johnbywater <p> See Update a release (ticket 14) for more information on what should be shown. </p> http://localhost/ticket/7 http://localhost/ticket/7 Report johnbywater <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> View the files associated with a package. </p> <h2 id="Notes">Notes</h2> <ul><li>See also <a class="closed ticket" href="http://localhost/ticket/16" title="enhancement: Update a file (closed: invalid)">ticket:16</a> (updating a file). </li><li>Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also <a class="closed ticket" href="http://localhost/ticket/14" title="enhancement: Update a release (closed: invalid)">ticket:14</a> (update a release) </li></ul><p> </p> http://localhost/ticket/8 http://localhost/ticket/8 Report johnbywater <h2 id="Asa">As a</h2> <p> Authenticated User </p> <h2 id="Iwantto">I want to</h2> <p> Create/register a package </p> <h2 id="Sothat">So that</h2> <p> The package will be listed in the system and visible to users </p> <h2 id="Notes">Notes</h2> <ul><li>Full package registration process may be broken down into: <ol><li>Creating a package by provision of minimal attributes: <ol><li>name: unique and web-usable </li><li>auto-generation of default values such as: <ol><li>date-registered (today's date) </li><li>first owner (one owner should be created corresponding to the authenticated user performing this action) </li></ol></li><li>(?) auto-generate a default release </li></ol></li><li>updating the created package -&gt; ../UseCase12 </li></ol></li></ul> http://localhost/ticket/9 http://localhost/ticket/9 Report johnbywater <h2 id="Asa">As a</h2> <p> Authenticated user </p> <h2 id="Iwantto">I want to</h2> <p> Update the attributes of a package and add/remove releases </p> <h2 id="Sothat">So that</h2> <p> The package attributes are changed. Package attributes are: </p> <ul><li>A title </li><li>A description (short or long) </li><li>Date added </li><li>Project url </li><li>package license(s) </li><li>download url </li><li>A list of releases with their dates and a summary line </li><li>a list of owners (referencing authenticated users) </li></ul><p> I would also like to be able to add comments </p> <h2 id="Notes">Notes</h2> <ul><li>Only an authenticated user is one of owners of a package may update it </li><li>Any authenticated user may add comments (but visitors may not) </li><li>can break down into: <ul><li>Update a package by changing simple package attributes (value: 8) </li><li>Update a package by adding or removing an owner (value: 5) </li><li>Update a package by adding or removing comments (value: 7) </li></ul></li></ul> http://localhost/ticket/10 http://localhost/ticket/10 Report johnbywater <h2 id="Asa">As a</h2> <p> Authenticated user </p> <h2 id="Iwantto">I want to</h2> <p> Delete a package </p> <h2 id="Sothat">So that</h2> <p> So that the package is no longer displayed for visitors of the site </p> <h2 id="Notes">Notes</h2> <ul><li>The permissions for this action are the same as for updating the package details (see <a class="closed ticket" href="http://localhost/ticket/10" title="enhancement: Update package (closed: worksforme)">ticket:10</a>) </li><li>When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator </li></ul> http://localhost/ticket/11 http://localhost/ticket/11 Report johnbywater <h2 id="Asa">As a</h2> <p> Authenticated user </p> <h2 id="Iwantto">I want to</h2> <p> Create a release associated with a package </p> <h2 id="Sothat">So that</h2> <p> The package has a new release </p> <h2 id="Notes">Notes</h2> <ul><li>Permissions are as for updating a package <a class="closed ticket" href="http://localhost/ticket/10" title="enhancement: Update package (closed: worksforme)">ticket:10</a> </li><li>for details of release attributes see: updating a release <a class="closed ticket" href="http://localhost/ticket/14" title="enhancement: Update a release (closed: invalid)">ticket:14</a> </li></ul> http://localhost/ticket/12 http://localhost/ticket/12 Report rgrp http://localhost/ticket/13 http://localhost/ticket/13 Report johnbywater <h2 id="Asa">As a</h2> <p> Authenticated user </p> <h2 id="Iwantto">I want to</h2> <p> Update a release </p> <h2 id="Sothat">So that</h2> <p> The release details are amended. Attributes of a release are: </p> <ul><li>name (may include '.' and '-' </li><li>nickname </li><li>date </li><li>description </li></ul><h2 id="Notes">Notes</h2> <ul><li>permissions: as for updating a package (<a class="closed ticket" href="http://localhost/ticket/10" title="enhancement: Update package (closed: worksforme)">ticket:10</a>) </li></ul> http://localhost/ticket/14 http://localhost/ticket/14 Report johnbywater <h2 id="Asa">As a</h2> <p> Authenticated user </p> <h2 id="Iwantto">I want to</h2> <p> Create a file associated with a release </p> <h2 id="Sothat">So that</h2> <p> The release has a new file associated with it </p> <h2 id="Notes">Notes</h2> <ul><li>Permissions are as for package updating (see <a class="closed ticket" href="http://localhost/ticket/10" title="enhancement: Update package (closed: worksforme)">ticket:10</a>) </li><li>file attributes are described in file updating: <a class="closed ticket" href="http://localhost/ticket/16" title="enhancement: Update a file (closed: invalid)">ticket:16</a>) <ul><li>there are no attributes that are required to be supplied to create a file </li></ul></li></ul> http://localhost/ticket/15 http://localhost/ticket/15 Report rgrp <h2 id="Asa">As a</h2> <p> Authenticated user </p> <h2 id="Iwantto">I want to</h2> <p> Update a file associated with a release </p> <h2 id="Sothat">So that</h2> <p> The file attributes are changed. File attributes are: </p> <ul><li>(internal) id </li><li>description </li><li>source: (url usually) </li></ul><h2 id="Notes">Notes</h2> <ul><li>Permissions are as for package updating </li></ul> http://localhost/ticket/16 http://localhost/ticket/16 Report johnbywater <p> See <a class="closed ticket" href="http://localhost/ticket/16" title="enhancement: Update a file (closed: invalid)">ticket:16</a> (update a file) </p> http://localhost/ticket/17 http://localhost/ticket/17 Report johnbywater http://localhost/ticket/18 http://localhost/ticket/18 Report rgrp http://localhost/ticket/19 http://localhost/ticket/19 Report johnbywater <h2 id="AsA">As A</h2> <p> Administrator </p> <h2 id="Iwantto">I want to</h2> <p> Purge a deleted package </p> <h2 id="Notes">Notes</h2> <ul><li>this need only be possible via the admin interface </li></ul> http://localhost/ticket/20 http://localhost/ticket/20 Report johnbywater http://localhost/ticket/21 http://localhost/ticket/21 Report johnbywater http://localhost/ticket/22 http://localhost/ticket/22 Report johnbywater http://localhost/ticket/23 http://localhost/ticket/23 Report johnbywater http://localhost/ticket/24 http://localhost/ticket/24 Report johnbywater <h2 id="Notes">Notes</h2> <ul><li>Currently have a text box where users can add space separated tags </li><li>Tags should auto-complete in a manner similar to delicious </li><li>would also be good to have a list of suggested tags. This could either be: <ul><li>[simpler] a list below the tag box chosen by existing popularity or </li><li>[more complex] a list of suggestions that updates as you add tags (based again on popularity but 'filtered/reranked' to reflect choices so far). </li><li>For first attempt to should probably just go for the simpler option. </li></ul></li></ul> http://localhost/ticket/25 http://localhost/ticket/25 Report johnbywater http://localhost/ticket/26 http://localhost/ticket/26 Report johnbywater <h1 id="AsA">As A</h1> <p> User </p> <h1 id="IwantTo">I want To</h1> <p> Create/register and update a package via a web api. </p> http://localhost/ticket/27 http://localhost/ticket/27 Report johnbywater http://localhost/ticket/28 http://localhost/ticket/28 Report rgrp <ol><li>install documentation (currently <a class="source" href="http://localhost/browser/trunk/INSTALL">source:trunk/INSTALL</a> is empty) </li><li>improve ckan-admin as necessary </li><li>create default data (such as licenses) </li></ol> http://localhost/ticket/29 http://localhost/ticket/29 Report rgrp <h2 id="AsA">As A</h2> <p> Site User </p> <h2 id="IWantTo">I Want To</h2> <p> <a class="missing wiki">Create/Edit/Update/Delete?</a> tags associated with my package </p> <h2 id="Notes">Notes</h2> <ul><li>Tags are service-wide and <strong>not</strong> package specific </li><li>Tags should not allow spaces (but allow everything else e.g. ':') </li><li>web interface details <ul><li>should provide a text box where users can add space separated tags (a la delicious). </li><li>tag creation should be seemless (so if a tag does not already tag exists it should automatically be created in the background) </li><li>would like to provide a list of relevant existing tags. For time being could just list all existing tags below entry box but in long run would like auto-updating/auto-suggesting listing a la delicious </li></ul></li></ul> http://localhost/ticket/30 http://localhost/ticket/30 Report rgrp <h2 id="Asa">As a</h2> <p> Registered user </p> <h2 id="Iwantto">I want to</h2> <p> Add a comment to a package </p> <h2 id="Notes">Notes</h2> <ul><li>The comment should be tagged with the username of the person making the comment </li><li>Should we allow comments per release as well as per package? <strong>No</strong> for the time being. </li><li>Comments will be 'flat' and <strong>not</strong> threaded (at least for the present) </li><li>Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown) </li></ul> http://localhost/ticket/31 http://localhost/ticket/31 Report rgrp <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> View a tag's page which should list all packages associated with that tag (or link to a page containing that list). </p> <h2 id="Notes">Notes</h2> <ul><li>The tags should have a nice primary url such as /tag/&lt;tag-name&gt; (could also use the plural i.e. /tag/&lt;tag-name&gt;) </li></ul> http://localhost/ticket/32 http://localhost/ticket/32 Report rgrp <h2 id="Asa">As a</h2> <p> Visitor </p> <h2 id="Iwantto">I want to</h2> <p> View a list of the tags on the system and search for a tag by name </p> <h2 id="Notes">Notes</h2> <ul><li>As for a search of packages should search for tags that contains a given search string (or could use a glob approach allowing wildcards). </li></ul> http://localhost/ticket/33 http://localhost/ticket/33 Report zool http://localhost/ticket/34 http://localhost/ticket/34 Report rgrp <p> When IP address is used to identify non-logged in users (for example on create and editing of packages) it is always 127.0.0.1 when using a reverse proxy (standard deployment configuration on a production machine and used e.g. on <a class="ext-link" href="http://www.ckan.net"><span class="icon">​</span>http://www.ckan.net</a>). </p> <p> This could be corrected by using HTTP_X_FORWARDED_FOR when available. </p> http://localhost/ticket/35 http://localhost/ticket/35 Report rgrp <p> When editing a package one would like to be able to preview one's changes before saving. </p> http://localhost/ticket/36 http://localhost/ticket/36 Report rgrp <p> Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged). </p> <p> In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort). </p> http://localhost/ticket/37 http://localhost/ticket/37 Report rgrp <p> Currently all data is only available via the web interface which makes it difficult to obtain in bulk. Data should be provided in other more convenient, machine usable forms: </p> <ul><li>DB Dump </li><li>JSON for each package page (maybe Atom as well) </li></ul> http://localhost/ticket/38 http://localhost/ticket/38 Report rgrp <p> When a package listing is shown (as on <a class="ext-link" href="http://ckan.net/package/list"><span class="icon">​</span>http://ckan.net/package/list</a>) more information than just name should be shown. For example package titles should be shown as well as names (could also list tags and other stuff but simplest is just title). </p> <p> Cost: 1 </p> http://localhost/ticket/39 http://localhost/ticket/39 Report rgrp <p> A url such as: <a class="ext-link" href="http://someurl.com/xyz?x=1&amp;VERSION=1.1&amp;Service=WFS"><span class="icon">​</span>http://someurl.com/xyz?x=1&amp;VERSION=1.1&amp;Service=WFS</a> when set as url or download_url breaks the rendering of the package with an error like: </p> <p> There was an error rendering the package: not well-formed (invalid token): line 1, column 181 </p> <p> Have checked that removing the &amp; stuff makes the error go away so this looks like an issue with escaping urls when displaying them ... </p> http://localhost/ticket/40 http://localhost/ticket/40 Report rgrp <p> Should do this sooner rather than later to avoid 'clear water' opening up and need to deploy then asap onto production machine to avoid 'clear water' between trunk and production (which then prevents pushing from trunk to production). </p> http://localhost/ticket/41 http://localhost/ticket/41 Report rgrp <h1 id="AsA">As A</h1> <p> Visitor </p> <h1 id="IWantTo">I Want To</h1> <p> View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner. </p> <h1 id="Details">Details</h1> <p> Main place this issue arises: </p> <ul><li>List of tags </li><li>List of packages </li><li>Repository history (/revision) </li></ul><p> Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display. </p> http://localhost/ticket/42 http://localhost/ticket/42 Report rgrp <h1 id="AsA">As A</h1> <p> User </p> <h1 id="IWantTo">I Want To</h1> <p> Add arbitrary named attributes to packages (an attribute being a name, type, value triple). </p> <h1 id="Details">Details</h1> <ul><li>We will do this using a dedicated (versioned) table associated to Package </li><li>Do we allow multiple attributes of the same name? <ul><li>For the present: No (since we will key by attribute name) </li><li>Could allow for single attribute but with multiple values using json list ... </li></ul></li><li>What types do we allow or do we just rely on JSON to take care of this? </li></ul><h1 id="QuestionsOriginal">Questions (Original)</h1> <ul><li>How complex is this to implement? </li><li>What would an arbitrary user be able to edit? Possibilities: <ol><li>'create new attribute' and setting the value (so name and type would be chosen from predefined list). </li><li>'create', setting of name and value (but not type -- type already set in predefined list) </li></ol></li><li>Could just use (machine) tags -- though this could be seen as a bit of a hack. </li><li>Would solve having to create special file/url attributes (though I think that perhaps file stuff is important enough to merit first class support in the domain model -- though, that said, since one won't want to have a file limit adding unlimited file support is very similar to unlimited attributes of arbitrary type). </li></ul> http://localhost/ticket/43 http://localhost/ticket/43 Report rgrp <h1 id="AsA">As A</h1> <p> Visitor </p> <h1 id="IWantTo">I Want To</h1> <p> Get an RSS/Atom Feed of the Repository History to use in my feed reader (or elsewhere). </p> <h1 id="Details">Details</h1> <ul><li>Preference for Atom. </li><li>should just add parameter to /revision/list/ (or /revision/) to select atom format e.g. ?format=atom. </li><li>should have a 'days' attribute specifying number of days back to go e.g. &amp;days=30 </li></ul><h1 id="Cost">Cost</h1> <p> Low </p> http://localhost/ticket/44 http://localhost/ticket/44 Report johnbywater http://localhost/ticket/45 http://localhost/ticket/45 Report johnbywater http://localhost/ticket/46 http://localhost/ticket/46 Report johnbywater http://localhost/ticket/47 http://localhost/ticket/47 Report johnbywater <p> Because CKAN's worth it! </p> http://localhost/ticket/48 http://localhost/ticket/48 Report rgrp <h1 id="AsA">As A</h1> <p> sysadmin </p> <h1 id="IWantTo">I Want To</h1> <p> Have revisions to the CKAN data filtered in order to reduce the spam in the system. </p> <h1 id="Details">Details</h1> <p> In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution. </p> <p> Suggest we follow path of trac: <a class="ext-link" href="http://trac.edgewall.org/wiki/SpamFilter"><span class="icon">​</span>http://trac.edgewall.org/wiki/SpamFilter</a> </p> <p> Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing'). </p> <p> Main initial plugins would be: </p> <ul><li>regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...) <ul><li>could augment using the badcontent list approach (can find list on e.g. moinmoin) </li></ul></li><li>spambayes and/or akismet </li></ul> http://localhost/ticket/49 http://localhost/ticket/49 Report rgrp <p> Research apt-get especially wajig and python wrappers for apt-get. Post results on the wiki. </p> http://localhost/ticket/50 http://localhost/ticket/50 Report rgrp <p> Move CKAN codebase to use sqlalchemy (rather than sqlobject) along with vdm v0.2. </p> <p> NB: the upgrade itself should not be too hard, the work will be in providing a satisfactory migration script to convert existing data on ckan.net service. </p> http://localhost/ticket/51 http://localhost/ticket/51 Report rgrp <ol><li>Tag results could/should show number of associated packages </li><li>Package results should show title (and be in list) </li><li>Pagination should be able to be applied to other result sets than simply register listing (e.g. search results ...) </li><li>Revisions should be listed in a table with full details </li></ol> http://localhost/ticket/52 http://localhost/ticket/52 Report rgrp <h1 id="AsA">As A</h1> <p> <a class="missing wiki">Visitor/User?</a> </p> <h1 id="Iwantto">I want to</h1> <p> View information about a package history, i.e. what revisions it has. </p> <h1 id="Details">Details</h1> <ul><li>History info should be available at package/history/pkg-name </li><li>Most basic info is just info from revision (when it occurred, author etc) </li><li>More sophisticated would be to have diff support to see differences between revisions </li></ul> http://localhost/ticket/53 http://localhost/ticket/53 Report rgrp <h1 id="Asa">As a</h1> <p> Sysadmin </p> <h1 id="Iwantto">I want to</h1> <p> Dump (serialize) CKAN data to a simple transport format (e.g. JSON) and be able to load it again. </p> <h1 id="Details">Details</h1> <ul><li>Already have simple db dump. However dumping to JSON has various advantages, particularly where changes to the data need to be made upon reloading (e.g. during a migration). </li><li>Dump should include *all* CKAN data (i.e. all the data in the CKAN db tables) </li></ul> http://localhost/ticket/54 http://localhost/ticket/54 Report rgrp <p> Associated to <a class="closed ticket" href="http://localhost/ticket/51" title="enhancement: Upgrade CKAN to use sqlalchemy and vdm v0.2 (closed: fixed)">ticket:51</a> (upgrade CKAN to new vdm) and <a class="closed ticket" href="http://localhost/ticket/54" title="enhancement: Support dump and load of CKAN data to JSON (closed: fixed)">ticket:54</a> (dump/load) need to convert v0.6 data for v0.7. </p> <p> Obvious way to do this is via alteration to data load method. </p> http://localhost/ticket/55 http://localhost/ticket/55 Report rgrp <h2 id="AsA">As A</h2> <p> User or Visitor </p> <h2 id="IWantTo">I Want To</h2> <p> See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package. </p> <h2 id="SoThat">So That</h2> <p> It is easy to see the openness status of a package. </p> <h2 id="Details">Details</h2> <p> </p> <ul><li>Suggested by Liz Turner </li><li>Openness = open license at present (no easy way to determine whether accessible) </li><li>Have separate tick for downloadability (i.e. existence of download url) </li><li>Could do more stuff in future (e.g. money where tag price- exists, etc etc) </li></ul> http://localhost/ticket/56 http://localhost/ticket/56 Report rgrp <p> Using datapkg a user can: </p> <ul><li>list packages on CKAN </li><li>get info about package on CKAN </li><li>register and update a package on CKAN </li></ul> http://localhost/ticket/57 http://localhost/ticket/57 Report rgrp <p> Make bookmarklet to enable easy package registration on CKAN </p> <h2 id="Details">Details</h2> <ol><li>This requires redoing web interface to allow 1-step package creation/editing (as opposed to name then everything else approach at the moment) </li></ol> http://localhost/ticket/58 http://localhost/ticket/58 Report rgrp <ol><li>Use contents from ckan package notes section (rgrp) </li><li>Update that section to be more guide-like (jwyg) </li></ol> http://localhost/ticket/59 http://localhost/ticket/59 Report rgrp <ol><li>List in sidebar 3 latest created/updated packages. </li></ol><ol start="2"><li>Put package search box on front page (might also want to move some of front page content into an about page). </li></ol><ol start="3"><li>Similarly might want package creation box on front page. </li></ol> http://localhost/ticket/60 http://localhost/ticket/60 Report rgrp <p> Cost: 1h </p> http://localhost/ticket/61 http://localhost/ticket/61 Report rgrp <p> Requires us to url encode the tag names when displaying them ... </p> http://localhost/ticket/62 http://localhost/ticket/62 Report rgrp <p> Since we upgraded to new vdm (v0.4) which has built in purge support existing purge support looks to be broken. Should be reasonably easy to fix this. </p> <p> Cost: 1h </p> http://localhost/ticket/63 http://localhost/ticket/63 Report rgrp <p> Switch to repoze.who + openid plugin for authentication from authkit. </p> <p> Already did this in microfacts so should be fairly easy. </p> http://localhost/ticket/64 http://localhost/ticket/64 Report rgrp <p> It is essential packages have a version attribute to assist in tracking releases etc (and to allow compatibility with other packaging systems e.g. distutils which is being used by datapkg). </p> <p> cost: 2h </p> http://localhost/ticket/65 http://localhost/ticket/65 Report rgrp <ol><li>Link at top should say "Your Account" when you are logged in <ul><li>when not logged in should say: "Log in via openid" </li><li>Open Id info on /account/ should move to /account/login/ or be deleted </li><li>Going to /account/ when not logged in should redirect to /account/login/ </li></ul></li></ol><ol start="2"><li>When logged in the basic account index page should not give generic information but should reflect fact you are logged in by: </li></ol><ul><li>Saying something like: "You are logged in as: ..." </li><li>Showing your apikey or a link to apikey </li><li>Giving you a logout link </li></ul><p> We can think of a bunch of other stuff that could go on their (recent edits, packages you own etc) but these will be separate tickets. </p> http://localhost/ticket/66 http://localhost/ticket/66 Report rgrp <p> Additional feature related to <a class="closed ticket" href="http://localhost/ticket/66" title="enhancement: Improve user account UI (closed: fixed)">ticket:66</a>. </p> <p> cost: 2h </p> http://localhost/ticket/67 http://localhost/ticket/67 Report rgrp <p> When listing revisions (at /revision/ or in atom feed) show affected packages. Minor but v. useful UI improvement. </p> <p> Cost: 1h </p> http://localhost/ticket/68 http://localhost/ticket/68 Report rgrp <p> Switch from license domain object to a simple license field and use license list from new centralised license repo: </p> <p> &lt;<a class="ext-link" href="http://knowledgeforge.net/okfn/licenses/"><span class="icon">​</span>http://knowledgeforge.net/okfn/licenses/</a>&gt; </p> <ul><li>This will <strong>require</strong> a migration </li></ul><p> Cost: 4h (plus migration ...) </p> http://localhost/ticket/69 http://localhost/ticket/69 Report rgrp <p> Cost: 1h </p> http://localhost/ticket/70 http://localhost/ticket/70 Report rgrp <p> Cost: 2h </p> http://localhost/ticket/71 http://localhost/ticket/71 Report rgrp <p> Integrate new logo from <a class="ext-link" href="http://wiki.okfn.org/ckan/logo"><span class="icon">​</span>http://wiki.okfn.org/ckan/logo</a> into site: </p> <ul><li>In title </li></ul><p> Also can do favicon (separate <a class="closed ticket" href="http://localhost/ticket/48" title="enhancement: Create favicon (closed: fixed)">ticket:48</a>) </p> <p> Cost: 1h </p> http://localhost/ticket/72 http://localhost/ticket/72 Report rgrp <p> Currently stored in the python controller file. This is not the right place and putting it in db will allow editing by admins. </p> <p> May also want to do this for the footer (also in the controller file). </p> <p> Details: </p> <ul><li>Will need to create a new domain object/db table. Suggest called miscellaneous and consist of key value pairs utilizing JSON type for values. </li></ul><p> Cost: 2h. </p> http://localhost/ticket/73 http://localhost/ticket/73 Report rgrp <p> On each package page in the openness field if the package is not open provide a link to is it open create enquiry page. </p> <p> Details: </p> <ul><li>may want to add to the url a query parameter indicating which package this relates to e.g. ?ckan-package=... </li><li><a class="ext-link" href="http://isitopen.ckan.net/enquiry/create/?ckan-package="><span class="icon">​</span>http://isitopen.ckan.net/enquiry/create/?ckan-package=</a>... </li><li>do we provide link even if package is open (e.g. so that people can make enquiries about e.g. adding open data/content buttons ...) </li></ul><p> Cost: 0.5h </p> http://localhost/ticket/74 http://localhost/ticket/74 Report rgrp <ul><li>Number of package page visits on ckan (can we get this straight from google analytics) </li><li>Number of times url or download url is used - now <a class="closed ticket" href="http://localhost/ticket/937" title="enhancement: Record download stats for resources (extension) (closed: fixed)">ticket:937</a> (Record download stats for resoures) </li></ul><p> How do we do this? </p> <ul><li>Google analytics will miss a lot of this usage (and how do we get that data out anyway) </li><li>Could use javascript but again misses usage. </li><li>One option is to redirect link but that is kind of nasty (but may be only option ...) </li></ul> http://localhost/ticket/75 http://localhost/ticket/75 Report rgrp <p> Current form stuff is not very good (uses formencode). Switch to formalchemy would improve this, especially on validation. </p> <p> As an extra we could utilize the formalchemy pylons admin interface (pretty much for free). </p> <p> Cost: 12h </p> <h2 id="Details">Details</h2> <ol><li>Replace htmlfill and formencode extract in controllers/templates with formalchemy <ol><li>Crude and simple </li><li>Suppress unwanted fields (revision, state, all revisions) and sort out ordering to be similar to before. </li><li>Sort out tag field with a special renderer </li></ol></li><li>Validation - testing definitely required. </li><li>Fix up description and pretty css etc </li></ol> http://localhost/ticket/76 http://localhost/ticket/76 Report rgrp <h1 id="AsaUserorVisitor">As a User or Visitor</h1> <p> I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable. </p> <p> Rating out of 5. One user or IP, one vote. </p> <p> package_rating table: </p> <p> package | user | rating annakarenina | joe.bloggs.openid.com | 4 annakarenina | 154.2.5.47 | 1 </p> http://localhost/ticket/77 http://localhost/ticket/77 Report rgrp <p> When you login you get given a cookie but it is very short (life of your browser session I think). </p> <p> Work out how to extend to something reasonable (30 days or perhaps forever with logout unsetting cookie). </p> <ol><li>A quick investigation to see whether this is possible </li><li>If possible do it </li></ol><p> Cost: ? </p> http://localhost/ticket/78 http://localhost/ticket/78 Report rgrp <p> Several things: </p> <ul><li>Default package search (used e.g. on front page) should deal well with multiple items: "abc xyz" means search for both abc and xyz (as separate searches) </li><li>Should automatically display a list of matching tags in a line at top with number of associated packages </li><li>have an option to only search for fully open material (or openly licensed material ...) </li></ul><p> Since we should be moving to a more sophisticated search solution anyway (to allow prioritisation etc) maybe we should cautious as to what we do now. </p> http://localhost/ticket/79 http://localhost/ticket/79 Report rgrp <p> Get rid of modes in the RESTful API. Do json stuff directly in controllers. </p> <ul><li>package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff. </li></ul> http://localhost/ticket/80 http://localhost/ticket/80 Report rgrp http://localhost/ticket/81 http://localhost/ticket/81 Report rgrp <p> Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs. </p> <p> Details: </p> <ul><li>Do we deal with m2m relationships (and m2one on the many side)? </li><li>What is diff? <ul><li>For text fields output of python diff command </li><li>For non-text field guess just simple +/- for what was there before and now </li></ul></li></ul><p> Cost: 4h (don't think this is a huge request) </p> http://localhost/ticket/82 http://localhost/ticket/82 Report rgrp <p> Allow 'ignored' fields on versioned objects (i.e. attributes which are not 'versioned'). This is not hard to do as we already have most of the necessary mechanisms set up in the Revisioner object. </p> <p> Cost: 2h </p> http://localhost/ticket/83 http://localhost/ticket/83 Report rgrp <p> Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support: </p> <ol><li>Cases where multiple objects changed in a revision but only want to revert 1 (low priority) </li><li>Want to revert but have reversion as a new revision of that object. </li></ol><p> Seems low priority at present. </p> <p> Cost: ? </p> http://localhost/ticket/84 http://localhost/ticket/84 Report rgrp <p> No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible. </p> <p> Main hassle here is that it requires migration in all projects that use vdm. </p> <p> If we're going to do this we should do it sooner rather than later. </p> <p> Cost: 2h (+ 1h for a conversion method) </p> http://localhost/ticket/85 http://localhost/ticket/85 Report rgrp <p> Already support stateful list-like collections and should extend this to dict-like collections. </p> <p> (This is prerequisite for implementing versioned "extra" (key/value) attributes on packages in CKAN and elsewhere). </p> http://localhost/ticket/86 http://localhost/ticket/86 Report rgrp <p> Multiple download links, including links to mirrors and multiple formats/versions </p> http://localhost/ticket/87 http://localhost/ticket/87 Report rgrp <p> Support for download metadata (file size, file formats, version information). </p> <p> See also: <a class="closed ticket" href="http://localhost/ticket/87" title="enhancement: Multiple download links (closed: duplicate)">ticket:87</a> (multiple download links) </p> http://localhost/ticket/88 http://localhost/ticket/88 Report rgrp <p> Should not be hard to do (lots of existing libraries) but not sure that this is very important. </p> http://localhost/ticket/89 http://localhost/ticket/89 Report rgrp <p> We should link to representations on semantic.ckan.net </p> <ul><li>Each package page could link to its RDF representation <ul><li>"Alternative formats" near star rating. Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f). </li><li>Some instances of ckan won't need this, so have option in ini file for this. rdf_store_url=xyz </li><li>Include machine-readable link to the RDF: <pre class="wiki">&lt;meta http-equiv=... /&gt; </pre></li><li>303 redirect to RDF when content negotiation requires RDF. Test: <pre class="wiki">curl -I -H "Accept: application/rdf+xml" http://localhost:5000/package/warandpeace </pre></li></ul></li></ul><p> Cost: 3h </p> <h2 id="Future">Future</h2> <ul><li>Search page to link to SPARQL interface: <a class="ext-link" href="http://semantic.ckan.net/sparql"><span class="icon">​</span>http://semantic.ckan.net/sparql</a> </li><li>License/dump page could link to full RDF dump. </li></ul> http://localhost/ticket/90 http://localhost/ticket/90 Report rgrp <p> Add the following attributes to package: </p> <ul><li>author, author_email </li><li>maintainer, maintainer_email </li></ul><p> Gives us full compatibility to: <a class="ext-link" href="http://docs.python.org/distutils/setupscript.html#additional-meta-data"><span class="icon">​</span>http://docs.python.org/distutils/setupscript.html#additional-meta-data</a> </p> <p> Column ordering - should come after name, title, url, download_url. </p> http://localhost/ticket/91 http://localhost/ticket/91 Report rgrp http://localhost/ticket/92 http://localhost/ticket/92 Report rgrp <p> Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: <a class="wiki" href="http://localhost/wiki/AccessControl">AccessControl</a>. </p> <p> Cost: 8d </p> http://localhost/ticket/93 http://localhost/ticket/93 Report rgrp <p> Details of how to go about this here &lt;<a class="ext-link" href="http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/"><span class="icon">​</span>http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/</a>&gt; </p> <p> Cost: 2h </p> http://localhost/ticket/94 http://localhost/ticket/94 Report rgrp <p> Support for listing files (manifest) contained within a package. </p> <p> Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs. </p> http://localhost/ticket/95 http://localhost/ticket/95 Report rgrp <p> (2008-09-10) at present when setup.py is edited info command may not give up to date information if pkg-info (in egg.info) is not rebuilt. </p> <p> Suggest: info command needs to rebuilds pkg-info in python-type distributions. </p> http://localhost/ticket/96 http://localhost/ticket/96 Report rgrp <p> (2009-03-09) Do not create a distribution at path X if path X already exists and contains material (unless forced via a force option). </p> <p> Cost: 1h </p> http://localhost/ticket/97 http://localhost/ticket/97 Report rgrp <p> Replace use of pastescript templates in <a class="missing wiki">PythonDistribution?</a> (write) with of something simpler (allowing us to remove dependency on pastescript). </p> http://localhost/ticket/98 http://localhost/ticket/98 Report rgrp <p> Test <a class="missing wiki">DbIndex?</a> with sqla 0.5 (and get it working if it does not work). </p> <p> Cost: ?? </p> <p> Priority: low because <a class="missing wiki">DbIndex?</a> is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel). </p> http://localhost/ticket/99 http://localhost/ticket/99 Report rgrp <p> (2008-10-29) convert existing directory on disk to a datapkg distribution (<a class="missing wiki">PythonDistribution?</a> by default). This would be presented in the CLI as a convert command. </p> <p> Details: </p> <ul><li>Add a metadata file (setup.py), perhaps prompting for input </li><li>List all existing data in that directory </li></ul><p> Cost: 4h </p> http://localhost/ticket/100 http://localhost/ticket/100 Report rgrp <ol><li>Given a url download from that url to disk </li><li>Given a package use download url to download to disk </li></ol><p> Once on disk should uncompress (if necessary). </p> <h2 id="Details">Details</h2> <ul><li>Should download either to tmp directory (defaulted in config) or a specified path on disk </li><li>In download would like to support as many types of target urls as possible: <ol><li>Single files (usually compressed filesets but could be individual files) </li><li>Index pages listing files to download </li><li>Revision control repositories </li><li>... </li></ol></li></ul><blockquote> <p> This will obviously be a significant amount of work. Suggest: </p> </blockquote> <ol><li>Start off just supporting the first option. </li><li>Have something like a plugin system to handle new target types </li><li>Reuse existing work (e.g. easy_install and pip handle standard vcs such as svn, hg, git etc) </li></ol><p> Cost: 1d </p> http://localhost/ticket/101 http://localhost/ticket/101 Report rgrp <p> This is better for long term sustainability, especially if we want to "push and pull" from unrelated repositories in the future. </p> <p> Details: </p> <ul><li>May wish to retain a number field updated in the usual incremental way (what's the cost/benefit here?) </li></ul> http://localhost/ticket/102 http://localhost/ticket/102 Report rgrp <p> As a user I want to view a package at a given revision: </p> <ul><li>When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy </li><li>package history page should provide links to these pages </li></ul><p> Cost: 2h </p> http://localhost/ticket/103 http://localhost/ticket/103 Report rgrp <p> As a user i want to see a package at "version" X (NB: not revision X). </p> <ul><li>When I visit e.g. /package/read/xyz?version=0.7 I should be shown package at version 0.7 (or a message saying no such version) </li><li>Implementation: <ul><li>Find revision for this version (search revision history for when version field was last 0.7) </li><li>Show pacakge at that revision (as in <a class="closed ticket" href="http://localhost/ticket/103" title="enhancement: View a package at a given revision (closed: fixed)">ticket:103</a>) </li></ul></li><li>On history page also shows versions in list of revisions associated with the package </li></ul> http://localhost/ticket/104 http://localhost/ticket/104 Report rgrp <p> As a user I want to be able to create groups of packages. </p> <p> This functionality is different from tags: </p> <ul><li>The ability to add a package to a given group is restricted whereas anyone can add a given tag to a package </li><li>Specifically groups have owners and only the owners can add a package to that group </li></ul><p> Group properties </p> <ul><li>id (primarykey, uuid style) </li><li>name (restricted content - same rules as package) </li><li>title (no restrictions) </li><li>owners -- many:many with user object </li><li>description (markdown) </li><li>packages -- many:many with package object </li></ul><p> Don't version groups for now. </p> <p> Groups address in the WUI will be: </p> <ul><li>/group - browse list of groups (reuse list action) </li><li>/group/list - same thing for now </li><li>/group/&lt;groupname&gt; - display: group properties with links to packages. No links to user pages (yet). </li></ul><p> Group editing and searching will be another ticket. </p> <p> Cost: 3d </p> http://localhost/ticket/105 http://localhost/ticket/105 Report rgrp <p> Sister to <a class="closed ticket" href="http://localhost/ticket/90" title="enhancement: Link to RDF versions of CKAN data (closed: fixed)">ticket:90</a> (Link to RDF version of CKAN data on Talis Connected Commons). </p> <p> Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC. </p> <p> In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required. </p> <p> Attached is the ruby script used by Talis for conversion </p> <p> Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is) </p> http://localhost/ticket/106 http://localhost/ticket/106 Report rgrp <p> Often people who are registering resources want to make the associated material available. We should: </p> <ul><li>Provide a dedicated upload page giving instructions as to upload process <ol><li>Explain we don't store the data itself on CKAN. Suggest uploading to a store somewhere and then linking using download_url (that is its purpose) </li><li>Upload to <ul><li>Talis CC for RDF </li><li>archive.org </li><li>grid.okfn.org </li></ul></li></ol></li><li>link to this page from next to download_url item on new package page </li></ul><p> In long run (now <a class="closed ticket" href="http://localhost/ticket/186" title="enhancement: Automated upload to archive.org s3 (closed: duplicate)">ticket:186</a>) we may automate this by providing a ckan upload facility which caches the data and then reuploads it to relevant service (disadvantage is "we" are the owner of the data on that service ...) </p> http://localhost/ticket/107 http://localhost/ticket/107 Report rgrp <p> Add package search facility in the rest api at /api/search </p> <p> Queries can be provided as for the normal package search either by posting to that url or by performing a get with a query string. </p> <p> E.g. .../api/search/package?q=xyz </p> <p> Query parameters: </p> <ul><li>q is keyword string (searches name, title, tags by default) <ul><li>split by words and ANDed </li></ul></li><li>additional parameters for specific fields </li><li>qjson is alternative to q for Searching by specific fields (in addition to keyword string). Payload to json is a JSON-encoded dict which is a dictionary with a q field for free text (keywords) and additional key/value pairs for specific fields <ul><li>if q and qjson specified ignore q and just process qjson value </li></ul></li></ul><p> Additional parameters in addition to query ("q" or "qjson") are: </p> <ul><li>limit </li><li>offset </li><li>fullinfo=0/1 - return full record for each result (default=0) </li><li>order_by=field_name </li><li>search_notes=0/1 (default 0) do we search notes field in a keyword search </li></ul><p> Return value is json encoded dictionary with keys: </p> <ul><li>'results': list of results </li><li>'count': total number of results </li></ul><h3 id="Extrasforthefuture">Extras (for the future)</h3> <ul><li>Prioritisation is not part of this ticket (requires fulltext support in DB or in external app such as Xapian) </li></ul> http://localhost/ticket/108 http://localhost/ticket/108 Report dread <p> Similar to <a class="closed ticket" href="http://localhost/ticket/108" title="enhancement: Package search in the REST API (closed: fixed)">ticket:108</a> but for tags: </p> <p> /api/search/tag?q </p> <p> For tags search is extremely simple since you can only search by name. </p> <h2 id="PossibleExtras">Possible Extras</h2> <ul><li>Allow "exotic" ordering of results e.g. ordering by number of packages with that tag. </li><li>Return this number with tag list. </li></ul> http://localhost/ticket/109 http://localhost/ticket/109 Report dread <ol><li>Package needs a groups property (backref Group.packages). </li><li>Put groups in the search api. </li><li>Display them on the package, but not editable. (Only editable from the group page.) </li></ol><p> Follow up to <a class="closed ticket" href="http://localhost/ticket/105" title="enhancement: Package groups (view) (closed: fixed)">ticket:105</a> (Groups). </p> <p> Cost: 1d </p> http://localhost/ticket/110 http://localhost/ticket/110 Report dread <p> Object properties: </p> <ul><li>id (uuid) </li><li>apikey (uuid) -- migrated from the apikey table </li><li>name (username = openid) </li></ul><p> /account is being renamed to /user in the wui. </p> http://localhost/ticket/111 http://localhost/ticket/111 Report dread <p> 'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass. </p> <p> Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work). </p> http://localhost/ticket/112 http://localhost/ticket/112 Report dread <p> Must not alter the db. </p> <p> Related to <a class="closed ticket" href="http://localhost/ticket/112" title="enhancement: Test db schema created via migration scripts (closed: wontfix)">ticket:112</a>. </p> http://localhost/ticket/113 http://localhost/ticket/113 Report dread <p> Create in the model basic operation of Access Control. </p> <p> roles table </p> <pre class="wiki">name | context | action ----------------------- admin| package | edit admin| package | edit-permissions admin| package | read editor| package | update editor| package | read reader| package | read This data is set-up on db init and will have no interface. </pre><p> user-roles table: </p> <pre class="wiki">username | context_type | objectid | role rgrp | system | n/a | admin visitor | package | * | reader bob | package | geonames | admin visitor | package | geonames | editor visitor | package | geonames | reader john | group | ukgov | admin dread | group | ukgov | editor visitor | group | ukgov | reader This data will be added when someone is given permissions for the system, a package or a group. </pre><p> Pseudo code: </p> <p> class Package </p> <blockquote> <p> def is_allowed(name, action): </p> <blockquote> <p> is_allowed(name, action, context=self) </p> </blockquote> </blockquote> <p> class Group </p> <blockquote> <p> def is_allowed(name, action): </p> <blockquote> <p> is_allowed(name, action, context=self) </p> </blockquote> </blockquote> <p> def is_allowed(name, action, context=None): <strong> name: string - a username or IP for 'visitor' </strong></p> <blockquote> <p> action: string - 'read', 'edit', 'delete', 'edit-permissions' context: object - a Group or a Package or None (which means system) </strong></p> </blockquote> <p> <strong> </p> <blockquote> <p> # look up user from name. # look up in user-roles table what roles this user has for this context. # for each roles, look up in roles table what actions are allowed. # return True if action is allowed, else False. </p> </blockquote> http://localhost/ticket/114 http://localhost/ticket/114 Report dread <p> Based on a section of <a class="wiki" href="http://localhost/wiki/AccessControl">AccessControl</a> design: Reading and writing to packages in the WUI and REST API are now dependent on the authz tables. </p> http://localhost/ticket/115 http://localhost/ticket/115 Report dread <p> Based on a section of <a class="wiki" href="http://localhost/wiki/AccessControl">AccessControl</a> design: WUI gives controls to user and administrator to change permissions on a package. </p> http://localhost/ticket/116 http://localhost/ticket/116 Report dread <p> Based on a section of <a class="wiki" href="http://localhost/wiki/AccessControl">AccessControl</a> design: Group reads and edits are controlled by access control. WUI and REST interfaces covered. </p> http://localhost/ticket/117 http://localhost/ticket/117 Report dread <p> Take out import of paginate in setup.py. Use paginate in webhelpers instead. Make changes to take account of any i/f changes. </p> http://localhost/ticket/118 http://localhost/ticket/118 Report dread <p> Ensure pending packages don't show up in search or browse </p> <p> cost: 4h </p> http://localhost/ticket/119 http://localhost/ticket/119 Report dread <p> Look for all places where model is accessed and check authorization is checked. </p> <p> Document holes (and, as necessary, suggestions for fixes) as new tickets. Likely areas that need looking at: </p> <ul><li>search i/f </li><li>package WUI commit </li></ul><p> Write holes are obviously much more significant to us than read holes. </p> http://localhost/ticket/120 http://localhost/ticket/120 Report dread <p> And associated page to browse group. </p> http://localhost/ticket/121 http://localhost/ticket/121 Report dread http://localhost/ticket/122 http://localhost/ticket/122 Report dread <p> Add Group editing page. </p> <p> If no permissions to change group can't edit group. Also cannot view edit page. </p> <p> Editable attributes: name, title, description </p> <p> No preview needed </p> http://localhost/ticket/123 http://localhost/ticket/123 Report rgrp <p> Split out from <a class="closed ticket" href="http://localhost/ticket/43" title="enhancement: Generic Attributes for Packages (closed: fixed)">ticket:43</a> </p> http://localhost/ticket/124 http://localhost/ticket/124 Report rgrp <p> Split out from <a class="closed ticket" href="http://localhost/ticket/43" title="enhancement: Generic Attributes for Packages (closed: fixed)">ticket:43</a> </p> http://localhost/ticket/125 http://localhost/ticket/125 Report dread <p> As a Package Admin I want to change the state of the package. In particular I wish to delete and undelete it. </p> <p> (NB: this is quite separate from "purging" objects which is the term we shall use for irrevocable removal of an object from the domain model). </p> <ul><li>Only Package Admins (and sysadmins) should be able to change state </li></ul><h2 id="ImplementationSuggestions">Implementation Suggestions</h2> <ul><li>'delete' action should be renamed to 'change-state' (NB: this requires a db migration ...) </li><li>Have new package formalchemy form (created via inheritance?) to incorporate state attribute. Suggest this is rendered as a dropdown (and may be simple object rendering of state, i.e. do NOT need to change it to a single name such 'active'). </li><li>This form should then be used when the user satisfies is_authorized(..., model.Action.CHANGE_STATE) instead of the usual fieldset </li></ul> http://localhost/ticket/126 http://localhost/ticket/126 Report dread <ul><li>group form: description field for needs to be a text area </li><li>group form: make title and name field wider </li><li>package view: need to move tags and license above notes </li></ul> http://localhost/ticket/127 http://localhost/ticket/127 Report rgrp <p> Add ckan_url attribute to REST JSON representation of a Package pointing to the (read) url of package on CKAN. </p> <p> Cost: 30m </p> http://localhost/ticket/128 http://localhost/ticket/128 Report dread <p> Controllers and templates should not access db objects directly - they should do all access via authz module giving username. They are handed by a query that has already been filtered by the packages they are authorized to read. </p> <p> (Additional idea to be discussed: When they request a package object, they are handed an copy of the db object - disconnected from the database - so it the db object can't be changed.) </p> <p> A couple of tests can be reenabled when this is done: ckan.tests.functional.test_authz.<a class="missing wiki">TestUsage?</a>.test_admin_list_deleted ckan.tests.functional.test_authz.<a class="missing wiki">TestUsage?</a>.test_search_deleted </p> http://localhost/ticket/129 http://localhost/ticket/129 Report rgrp <h2 id="Asa">As a</h2> <p> User </p> <h2 id="Iwantto">I want to</h2> <p> Create a new group via the WUI </p> <h2 id="Details">Details</h2> <ul><li>locate at /group/new/ </li><li>You must be logged in to create a group </li><li>Group creator is automatically given role of group admin </li><li>Edit screen is same as edit screen ... </li></ul> http://localhost/ticket/130 http://localhost/ticket/130 Report dread <p> Controlling Groups through a REST interface. </p> http://localhost/ticket/131 http://localhost/ticket/131 Report dread <p> Using REST interface you can list packages and groups without authorization being checked. </p> <p> Can be fixed using more advanced query to check authz. </p> http://localhost/ticket/132 http://localhost/ticket/132 Report dread <p> Using WUI or REST interface you can search packages and groups without authorization being checked. </p> <p> On the REST interface you can also read all the attributes of the packages using the 'all-fields' option. </p> <p> Can be fixed using more advanced query to check authz. </p> http://localhost/ticket/133 http://localhost/ticket/133 Report dread http://localhost/ticket/134 http://localhost/ticket/134 Report dread http://localhost/ticket/135 http://localhost/ticket/135 Report dread <p> A user's 'home page' is at: user/&lt;user.id&gt; e.g. user/28394723982-03849472 </p> <p> Step 1: </p> <ul><li>Current user home page with recent edits </li></ul><p> Step 2: Readonly </p> <ul><li>Number of edits </li><li>Number of Packages they are admin of </li></ul><p> How do we do a nicer URL for the home page - can we use their openid login? </p> <p> Follow on tickets: <a class="closed ticket" href="http://localhost/ticket/138" title="enhancement: User has editable home page (closed: fixed)">ticket:138</a> and <a class="closed ticket" href="http://localhost/ticket/142" title="defect: Update User object with additional fields such as 'nickname' (closed: fixed)">ticket:142</a> </p> http://localhost/ticket/136 http://localhost/ticket/136 Report dread <ul><li>Generic text box for markdown about the user 'About' </li></ul><p> Model's user table reflects these: </p> <ul><li>'about' attribute </li></ul> http://localhost/ticket/137 http://localhost/ticket/137 Report dread <ul><li>Generic text box for markdown about the user 'About' </li></ul><p> Model's user table reflects these: </p> <ul><li>'about' attribute </li><li>'created' attribute is a default=datetime.now() - so when user was created </li></ul><p> Need edit page for this. </p> <p> Only user should be able to edit it. </p> <p> Follows on from <a class="closed ticket" href="http://localhost/ticket/136" title="enhancement: User has publicly viewable page (closed: fixed)">ticket:136</a> and <a class="closed ticket" href="http://localhost/ticket/142" title="defect: Update User object with additional fields such as 'nickname' (closed: fixed)">ticket:142</a> </p> <p> Cost: 2h </p> http://localhost/ticket/138 http://localhost/ticket/138 Report rgrp <p> Seems some package have ended up with same tag twice: <a class="ext-link" href="http://www.ckan.net/package/fred2dot0"><span class="icon">​</span>http://www.ckan.net/package/fred2dot0</a> </p> <p> Check whether this is still possible and if it is fix it ... </p> <p> Cost: 2h </p> http://localhost/ticket/139 http://localhost/ticket/139 Report rgrp <p> Have a news section (suggest as a sidebar item). </p> <p> News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org. </p> <p> Details: </p> <ul><li>Suggest pulling via rss or similar. </li><li>Will want to cache this ... </li></ul><p> Cost: 4h? </p> http://localhost/ticket/140 http://localhost/ticket/140 Report rgrp <p> Write up search api documentation and put it in a template that shows up at api/search/. </p> <p> Cost: 2h </p> http://localhost/ticket/141 http://localhost/ticket/141 Report dread <p> Suggested fields: </p> <ul><li>id, name (username/profile name), openid, email, password, extras (<a class="missing wiki">JsonType?</a>) <ul><li>+ current items: apikey, about, created) </li><li>current contents of name is usually openid and should migrate to openid field (could match based on starting with <a class="ext-link" href="http://"><span class="icon">​</span>http://</a>) </li></ul></li><li>May also want a fullname field (or perhaps that can </li></ul><p> just go in extras - may be better to have its own field as useful to search on this ) </p> <p> Related changes (probably separate tickets): </p> <ul><li>Improve user home page to include this additional information (and have it be editable) </li><li>Use user 'name' (username) for home page links (rather than current user/&lt;id&gt;) </li><li>Where we show lists of revisions, each revision author is now a link to the author's user page. </li></ul><p> Related to <a class="closed ticket" href="http://localhost/ticket/136" title="enhancement: User has publicly viewable page (closed: fixed)">ticket:136</a> and <a class="closed ticket" href="http://localhost/ticket/138" title="enhancement: User has editable home page (closed: fixed)">ticket:138</a>. </p> http://localhost/ticket/142 http://localhost/ticket/142 Report dread <p> Display league of users' recent activity on homepage. </p> http://localhost/ticket/143 http://localhost/ticket/143 Report dread <p> Based on number of views. </p> <p> Related to <a class="assigned ticket" href="http://localhost/ticket/143" title="enhancement: Most active users listed on homepage (assigned)">ticket:143</a>. </p> http://localhost/ticket/144 http://localhost/ticket/144 Report dread <p> When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators) </p> http://localhost/ticket/145 http://localhost/ticket/145 Report rgrp <ol><li>Visitor starts editing a package (or creates a new package) </li><li>Visitor logs in before previewing or committing </li><li>Visitor is an admin (or sysadmin) </li><li>Preview or commit fails </li></ol><p> Reason (conjecture): the form we use for package editing is now dependent on whether you have admin privileges (we allow editing of state by admins). When you login the old form is now incorrect and this results in the formalchemy fieldset breaking. </p> http://localhost/ticket/146 http://localhost/ticket/146 Report dread http://localhost/ticket/147 http://localhost/ticket/147 Report dread <ul><li>parse like data4nr code </li></ul><p> Follow up tickets: <a class="closed ticket" href="http://localhost/ticket/149" title="enhancement: Loader for esw data recognises existing packages (closed: fixed)">ticket:149</a> and <a class="closed ticket" href="http://localhost/ticket/150" title="enhancement: Loader for esw data uses REST API (closed: fixed)">ticket:150</a> </p> http://localhost/ticket/148 http://localhost/ticket/148 Report dread <p> Follows on from <a class="closed ticket" href="http://localhost/ticket/148" title="enhancement: Parser and loader for esw.org data (closed: fixed)">ticket:148</a>. Sister to <a class="closed ticket" href="http://localhost/ticket/150" title="enhancement: Loader for esw data uses REST API (closed: fixed)">ticket:150</a>. </p> http://localhost/ticket/149 http://localhost/ticket/149 Report dread <p> (instead of sqlalchemy interface, to facilitate general usage of the loader) </p> <ul><li>use ckanclient </li></ul><p> Follows on from <a class="closed ticket" href="http://localhost/ticket/148" title="enhancement: Parser and loader for esw.org data (closed: fixed)">ticket:148</a> and <a class="closed ticket" href="http://localhost/ticket/149" title="enhancement: Loader for esw data recognises existing packages (closed: fixed)">ticket:149</a> </p> http://localhost/ticket/150 http://localhost/ticket/150 Report rgrp <p> User object should have a "created" attribute initialized to current datetime. </p> <p> Require a db migration but o/w very simple. </p> <p> Cost: 1.5h </p> http://localhost/ticket/151 http://localhost/ticket/151 Report dread <p> Package edit page has fields for extra key-value pairs. Don't use formalchemy. The form displays the existing ones and new ones as follows: </p> <pre class="wiki">Key Value delete [ country ] [ uk, spain ] [x] </pre><p> When presenting the form, existing pairs and 3 blank pairs are provided. </p> <p> When saving the data, keys that are not changed use the existing PackageExtra object (perhaps with new value). Deleted keys put the PackageExtra state to deleted. For new keys, only create a new PackageExtra if there isn't a state=deleted one to resurrect. </p> <p> Validators ensure no repeated keys. </p> http://localhost/ticket/152 http://localhost/ticket/152 Report dread <p> This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there. </p> <p> Suggested by Jonathan Gray </p> http://localhost/ticket/153 http://localhost/ticket/153 Report dread <p> Give guidance on what to put in 'author' and 'maintainer' fields (e.g. name? URL?) </p> <p> Suggestion from Jonathan Gray </p> http://localhost/ticket/154 http://localhost/ticket/154 Report dread <p> Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously) </p> <p> Use a bit of javascript to add more dropdowns. </p> <p> Suggested by Jonathan Gray </p> http://localhost/ticket/155 http://localhost/ticket/155 Report dread <p> Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed. </p> http://localhost/ticket/156 http://localhost/ticket/156 Report rgrp <p> Log message on package edit page is no longer persisted through a preview (no doubt due to recent form upgrades). Let's fix this. </p> <p> cost: 0.5h </p> http://localhost/ticket/157 http://localhost/ticket/157 Report dread <ul><li>Ensure you can add/change/delete extra fields from packages using the REST API. </li><li>Update API docs to cover this. </li></ul><p> On Package get, ensure separation of extra attributes from other attributes. </p> <p> What is the meaning of posting an incomplete dictionary (updating a package)? Option 1 - if field not present, leave field alone </p> <ul><li>Delete field if field value is None </li></ul><p> Option 2 - user must supply all fields </p> <ul><li>Delete field if field absent </li></ul><p> Prefer option 1. Document this. </p> <p> Follows on from <a class="closed ticket" href="http://localhost/ticket/152" title="enhancement: Package has editable 'Extra' fields (closed: fixed)">ticket:152</a> </p> http://localhost/ticket/158 http://localhost/ticket/158 Report rgrp <p> Search of form: tags: ... behaves differently depending on whether there is a leading space: </p> <ul><li>tags: postcode - tags found correctly but no packages found </li><li>tags:postcode - tags incorrectly found but correct packages found </li></ul><p> Let's fix this. </p> <p> Cost: 0.5h </p> http://localhost/ticket/159 http://localhost/ticket/159 Report rgrp <p> Suggest form is: key:value (with optional space after ':'. </p> <p> Keys would be for an object, i.e. 'package', 'tag', 'group'. </p> <p> Becomes a link. </p> <p> This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace. </p> <p> Cost: 4h </p> http://localhost/ticket/160 http://localhost/ticket/160 Report rgrp <p> E.g. <a class="ext-link" href="http://ckan.net/tag/read/postcodes"><span class="icon">​</span>http://ckan.net/tag/read/postcodes</a> this has no-postcode listed but when you look at that package 'postcodes' isn't a tag ... </p> <p> I suspect this may be to do with search using deleted tags as I think no-postcode had postcodes as a tag at some point earlier ... </p> http://localhost/ticket/161 http://localhost/ticket/161 Report rgrp <p> Rather than raising a validation error automatically lower-case tags on submission. </p> <p> Cost: 0.5h </p> http://localhost/ticket/162 http://localhost/ticket/162 Report dread <p> Related to: <a class="closed ticket" href="http://localhost/ticket/90" title="enhancement: Link to RDF versions of CKAN data (closed: fixed)">ticket:90</a> Link to RDF version of CKAN data on Talis Connected Commons <a class="closed ticket" href="http://localhost/ticket/164" title="enhancement: Creating RDF data from CKAN (closed: fixed)">ticket:164</a> Creating RDF data from CKAN </p> <p> Regularly re-upload the CKAN RDF data to Talis CC. </p> <p> In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required. </p> <p> Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is) </p> http://localhost/ticket/163 http://localhost/ticket/163 Report dread <p> Related to: <a class="closed ticket" href="http://localhost/ticket/90" title="enhancement: Link to RDF versions of CKAN data (closed: fixed)">ticket:90</a> Link to RDF version of CKAN data and <a class="closed ticket" href="http://localhost/ticket/163" title="enhancement: Regularly put CKAN RDF on Talis CC (closed: fixed)">ticket:163</a> (regularly convert CKAN data to RDF). </p> <p> Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net. </p> http://localhost/ticket/164 http://localhost/ticket/164 Report dread <p> Because people might try and do "tags:gov sweden" when "tag:gov sweden" makes more sense (sweden is searched for in all fields in these cases). </p> <p> But in REST i/f search allow "tag"="gov" OR "tags"="gov sweden" in parameters. </p> http://localhost/ticket/165 http://localhost/ticket/165 Report dread <p> Ensure logging is working on server. </p> http://localhost/ticket/166 http://localhost/ticket/166 Report dread <p> Only first tag is updated. </p> http://localhost/ticket/167 http://localhost/ticket/167 Report dread http://localhost/ticket/168 http://localhost/ticket/168 Report dread <p> A 'Derived' relationship can be applied from one package to another. </p> <p> e.g. sussex-demography is derived from census-2001 </p> <p> 'Derived' relationship is: </p> <ul><li>directional </li><li>many:many </li><li>stateful </li></ul><p> 'derived' table columns: </p> <ul><li>id (primary key) </li><li>source_package (foreign key) </li><li>result_package (foreign key) </li><li>description (markdown text) </li></ul><p> Further tickets: </p> <ul><li>WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -&gt; this package -&gt; y </li><li>WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate </li><li>REST if - expose reading and writing this property </li></ul> http://localhost/ticket/169 http://localhost/ticket/169 Report dread <p> e.g. </p> <p> <a class="ext-link" href="http://www.ckan.net/package/search?q=science&amp;search=Search+Packages+%C2%BB"><span class="icon">​</span>http://www.ckan.net/package/search?q=science&amp;search=Search+Packages+%C2%BB</a> </p> <p> Results are capped at 20, but can't see past first page. </p> http://localhost/ticket/170 http://localhost/ticket/170 Report rgrp <p> Create consolidated set of config by merging config from .datapkgrc and command line options and use that instead of mixture of command line options and other stuff ... (likely to be less buggy!) </p> http://localhost/ticket/171 http://localhost/ticket/171 Report rgrp <p> Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible. </p> <p> NB: improving the documentation is another matter (as is integrating e.g. existing api docs). </p> <p> Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net): </p> <ul><li><a class="ext-link" href="http://knowledgeforge.net/ckan/ckan/doc/"><span class="icon">​</span>http://knowledgeforge.net/ckan/ckan/doc/</a> </li><li>(Other options: <a class="ext-link" href="http://www.ckan.net/doc/"><span class="icon">​</span>http://www.ckan.net/doc/</a> <a class="ext-link" href="http://doc.ckan.net"><span class="icon">​</span>http://doc.ckan.net</a>) </li></ul> http://localhost/ticket/172 http://localhost/ticket/172 Report dread <p> Use vdm diff functionality (see <a class="closed ticket" href="http://localhost/ticket/82" title="enhancement: Support diffing of versioned objects (closed: fixed)">ticket:82</a>) to implement diffing of packages between versions in WUI. </p> <p> In package history you should be able to select to revisions and show the diff. </p> <p> Estimated cost: 12h </p> http://localhost/ticket/173 http://localhost/ticket/173 Report dread <p> the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'. </p> <p> Raised by Jonathan Gray </p> http://localhost/ticket/174 http://localhost/ticket/174 Report dread <p> 'created' attribute on User, Group and Rating </p> <p> datetime object: default=datetime.datetime.now() </p> <p> See example on revision </p> <p> Requires migrate script </p> http://localhost/ticket/175 http://localhost/ticket/175 Report dread <p> (Related to <a class="closed ticket" href="http://localhost/ticket/169" title="enhancement: Package derivations (closed: duplicate)">ticket:169</a> - Package derivations) </p> <p> A 'dependency' relationship can be applied from one package to another. It implies that a package requires the download or existence of another package which it 'depends on'. (Analogous to software package dependencies.) </p> <p> e.g. london-traffic-visualisation depends on road-map </p> <p> 'Dependency' relationship is: </p> <ul><li>directional </li><li>many:many </li><li>stateful </li></ul><p> 'dependency' table columns: </p> <ul><li>id (primary key) </li><li>dependent (foreign key) </li><li>dependency (foreign key) </li></ul><p> Further tickets: </p> <ul><li>WUI - package view - have list of dependencies (do not need to list packages which depend on this one) </li><li>WUI - package edit form - new option to say 'depends on' (no need for 'has dependent package') </li><li>REST api - expose reading and writing 'depends on' property. </li></ul><h3 id="Issues">Issues</h3> <ul><li>How do we deal with dependency at a particular version? </li></ul> http://localhost/ticket/176 http://localhost/ticket/176 Report dread <p> This is docs for users of a CKAN service (e.g. ckan.net), as opposed to someone installing / administering the software. (The latter has docs on knowledgeforge. </p> <p> Editability - wiki? </p> http://localhost/ticket/177 http://localhost/ticket/177 Report dread <p> /tool New 'tools' section of the WUI with 'Excel package import' page. </p> <p> /tool/import Import page allows you to upload a file. The format of the required Excel file is described here. List of licenses provided for copy&amp;paste. </p> <p> /tool/import/preview When you upload it, it gives you a simple preview of the file, offering import. Text as per Package Preview. 'import button' for completing process. </p> <p> File format: </p> <p> First row to have a cell value of 'name' or 'title' in it is the Title Row. It contains the field names, as per package. It includes: </p> <ul><li>license - name of license </li><li>tags - space separated </li><li>groups - space separated </li><li>anything - for extra key-values </li></ul><p> When importing, values are bound into the package fieldset, validated and displayed for the preview and synced to complete. </p> <p> User has to log in to use importer. Auth is checked for putting into groups. </p> http://localhost/ticket/178 http://localhost/ticket/178 Report rgrp <p> Cost: 0.5h (?) </p> <p> Conjecture this went missing in cset:a35db862a841 </p> http://localhost/ticket/179 http://localhost/ticket/179 Report jwyg <p> Create big tag cloud with all CKAN tags - perhaps weighting with size and colour... </p> http://localhost/ticket/180 http://localhost/ticket/180 Report jwyg <p> With up to date statistics such as: </p> <ul><li>total packages </li><li>total tags </li><li>total users </li><li>most active users </li><li>last update </li><li>etc. </li></ul><p> Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See <a class="closed ticket" href="http://localhost/ticket/184" title="enhancement: Stats page (closed: fixed)">ticket:184</a> </p> http://localhost/ticket/181 http://localhost/ticket/181 Report rgrp <p> Tick and cross aren't clear enough. Propose: </p> <ul><li>Icon for downloadability (nothing if absent ...(?)) </li><li>Icon for open license? (Need something if closed ...) </li></ul><p> Cost: 2h (?) </p> http://localhost/ticket/182 http://localhost/ticket/182 Report rgrp <p> At moment order packages by title. </p> http://localhost/ticket/183 http://localhost/ticket/183 Report rgrp <p> Create /stats/ page displaying main statistics, e.g: </p> <ul><li>Most highly rated packages </li><li>Most edited packages </li><li>Largest groups </li><li>Top tags (by packages) </li><li>Package addition rate </li><li>Users with most packages </li></ul><p> Related to <a class="closed ticket" href="http://localhost/ticket/181" title="enhancement: New 'stats' box on right hand sidebar (closed: invalid)">ticket:181</a> - Stats side-bar </p> <h2 id="Details">Details</h2> <ul><li>Should create a stats module and then use some/all of these features on stats page (likely over time that stats features much more extensive than what we display) </li><li>For graphs (e.g. additions over time) suggest use javascript graphing -- for examples of how this is done see <ul><li><a class="ext-link" href="http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py"><span class="icon">​</span>http://knowledgeforge.net/econ/hg/file/tip/econ/www/controllers/plot.py</a> </li><li><a class="ext-link" href="http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html"><span class="icon">​</span>http://knowledgeforge.net/econ/hg/file/tip/econ/www/templates/plot/chart_code.html</a> </li></ul></li></ul> http://localhost/ticket/184 http://localhost/ticket/184 Report dread <p> Make parts of the package form hidden by default, to make it look simpler. Users are invited to click a button to reveal the less important fields. </p> <p> Fields hidden by default: version, author, author_email, maintainer, maintainer_email </p> http://localhost/ticket/185 http://localhost/ticket/185 Report rgrp <p> (Follows on from <a class="closed ticket" href="http://localhost/ticket/107" title="enhancement: Provide information about uploading material (closed: invalid)">ticket:107</a>). We want to provide facility for users to automatically upload material. </p> <ul><li>Create an upload page </li><li>Store the material on archive.org using s3 - <a class="ext-link" href="http://www.archive.org/help/abouts3.txt"><span class="icon">​</span>http://www.archive.org/help/abouts3.txt</a> </li><li>Main issue is how we cache large datasets ... </li></ul> http://localhost/ticket/186 http://localhost/ticket/186 Report rgrp <p> Standard search should search notes field in addition to name, title and tags (discussed in <a class="closed ticket" href="http://localhost/ticket/108" title="enhancement: Package search in the REST API (closed: fixed)">ticket:108</a> but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results. </p> <p> If we do this we need: </p> <ol><li>To weight across fields in a sensible way </li><li>We can also use proper text search on title or ... </li></ol><p> Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: <a class="ext-link" href="http://www.postgresql.org/docs/8.3/static/textsearch.html"><span class="icon">​</span>http://www.postgresql.org/docs/8.3/static/textsearch.html</a> </p> <p> Using this with sqlalchemy: <a class="ext-link" href="http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/"><span class="icon">​</span>http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/</a> </p> <p> Issues with fulltext search: </p> <ul><li>tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking. </li><li>if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better. </li><li>can we split the name on dash or underscore before being indexed? </li><li>natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'. </li><li>do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres? </li><li>we want to weight name and title higher than other fields - achievable with custom trigger. </li></ul> http://localhost/ticket/187 http://localhost/ticket/187 Report rgrp <p> Propose change to tabular-like format showing these attributes (perhaps should be configurable?) </p> <ul><li>Openness status </li><li>Title (not sure name is needed) </li><li>Tags </li></ul><p> Cost: 4h </p> http://localhost/ticket/188 http://localhost/ticket/188 Report dread <p> See also: <a class="wiki" href="http://localhost/wiki/PackageResources">PackageResources</a> which defines a 'distribution' as a package's payload (perhaps with metadata). In our case, at least within CKAN, the representation of the distribution will just be the reference to get it (plus some minor metadata). We are expanding this functionality to be any kind of 'resource' associated with a package since it is likely in some cases people will want to link to a file containing a part, but not the whole, of a package's data. </p> <h2 id="Asa">As a</h2> <p> User (package editor and package user) </p> <h2 id="Iwantto">I want to</h2> <p> (package editor) Associate multiple distributions with a package. </p> <p> (package user) Easily have access to multiple distributions of a package should they exist. </p> <h2 id="Implementation">Implementation</h2> <p> There is an object called 'Resource' with the following attributes: </p> <ul><li>url (text) (required in UI, but not in db table) </li><li>format (text) </li><li>description (text) </li><li>package (association) - a resource is always associated with 1 package </li></ul><p> A package's list of resources will have an order which is preserved. The first in the list will be known as the 'default resource' (the presumption will always be that this is a 'distribution'). </p> <h3 id="Howtoreplacedownloadurl">How to replace download url</h3> <p> download_url removed and replaced as a proxy attribute. Read returns the url of the default distribution. Write changes the url of the default resource. Ensures backward compatibility. </p> <p> REST API has new attribute in package dictionary 'resouces' - an ordered list. </p> <p> In WUI, replace download url in package read with a table of the distributions. </p> <p> In the WUI package edit, be able to add/remove/edit/reorder distributions. </p> <h2 id="Example">Example</h2> <p> Example part of package: </p> <h3 id="Distributions:">Distributions:</h3> <table class="wiki"> <tr><td> http://some.com/pollution.pdf </td><td> PDF </td><td> Original report </td></tr><tr><td> http://some.com/pollution.csv </td><td> csv </td><td> Original data </td></tr><tr><td> http://some.com/pollution </td><td> sparql </td><td> Linked version of data </td></tr></table> <p> Existing data to be migrated to new structure. </p> <p> Cost: 4 days </p> http://localhost/ticket/189 http://localhost/ticket/189 Report dread <p> Cost 7 days </p> <p> When viewing a package, users can read user comments and leave their own. Users need to be logged in to leave a message. Comments appear immediately. A mechanism for deleting unwanted comments is provided to an authorized user. Comments are sorted with the most recent first. Comments are available for read, creation and deletion in both the Web UI and over the REST API. </p> <p> The admin for the package and a superuser can delete unwanted comments, both on the package page and a collation of all comments on their user page. Users can delete their own comments(?) Need to consider whether over the REST API we encourage the use of a 'frontend user' APIKEY which can be used to leave comments for another, actual user. </p> <p> Example at bottom of package page: </p> <p> <strong>Leave a comment:</strong> </p> <table class="wiki"> <tr><td>Subject</td><td style="text-align: center"> <span class="underline"></span><span class="underline"></span>_ </td></tr><tr><td>Comment</td><td style="text-align: center"> <span class="underline"></span><span class="underline"></span>_ </td></tr></table> <p> <br /> </p> <table class="wiki"> <tr><td>Submit button </td></tr></table> <p> <strong>Comments:</strong> </p> <table class="wiki"> <tr><td><strong>Explanation doc</strong> </td></tr><tr><td><em>Posted on 25h May 2009 by http://bertdavies.myopenid.com</em> </td></tr><tr><td>It says on the pollution data web page that not all the stations have a CO2 sensor, so you have to extrapolate from the ones that do. See my visualisation of CO2 across London for an idea of what you can do: bertdavies.com/pollution-2008.jpg </td></tr></table> <table class="wiki"> <tr><td><strong>More info</strong> </td></tr><tr><td><em>Posted on 24th May 2009 by http://ronsmith.myopenid.com</em> </td></tr><tr><td>Excellent data, but why is there no value in the CO2 column for some of the testing stations? </td></tr></table> <h1 id="Implementationdetails:">Implementation details:</h1> <p> Comments table is with columns: </p> <table class="wiki"> <tr><td style="text-align: left">id </td><td> package_id </td><td> date (date) </td><td style="text-align: right"> comment (multi-line text) </td></tr></table> http://localhost/ticket/190 http://localhost/ticket/190 Report dread <p> Cost - 2 days </p> <p> Search interface has new options to filter and sort the results by the date the package has been last modified in ckan. Search options are included in both Web UI and Search API. </p> <p> The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. </p> <p> Example search parameters: </p> <table class="wiki"> <tr><td> modification-range=5/4/09- </td><td> Exclude packages last modified earlier than 5/4/09 </td></tr><tr><td> modification-range=5/4/09-5/12/09 </td><td> Exclude packages last modified outside of 5/4/09-5/12/09 </td></tr><tr><td> order_by=mod </td><td style="text-align: left">Sort by metadata modification. Defaults to newest first. </td></tr><tr><td> order_by=mod-newest </td><td> Sort by metadata modification, newest first. </td></tr><tr><td> order_by=mod-oldest </td><td> Sort by metadata modification, oldest first. </td></tr></table> http://localhost/ticket/191 http://localhost/ticket/191 Report dread <p> Cost - 2 days </p> <p> The time period to which a package's data applies is stored in a new field called 'Relevant Date Range'. The value of this field is up to two calendar dates: i.e. either a point in time (e.g. date of a river map is 5/10/09) or a time period (e.g. pollution measurements 1/1/09-1/4/09). </p> <p> Example part of package: </p> <p> <strong>Date:</strong> 5/10/09 <br /> or <br /> <strong>Date:</strong> 1/1/09 – 1/4/09 </p> http://localhost/ticket/192 http://localhost/ticket/192 Report dread <p> Cost - 2 days </p> <p> Search interface has new options to filter and sort the results by the time-related field of the package. Search options are included in both Web UI and Search API. </p> <p> The filter specifies a range of dates. The results can be sorted by ascending or descending dates. The last modification date is surfaced in the package. Need to decide for a time-related field value that is date range, what date is used for the search. </p> <p> Example search parameters: </p> <table class="wiki"> <tr><td> reldate-range=5/4/09- </td><td> Exclude packages related to earlier than 5/4/09 </td></tr><tr><td> reldate-range=5/4/09-5/12/09 </td><td> Exclude packages related to date outside of 5/4/09-5/12/09 </td></tr><tr><td> order_by=reldate </td><td> Sort by date package is related to. Defaults to newest first. </td></tr><tr><td> order_by=reldate-newest </td><td> Sort by date package is related to, newest first. </td></tr><tr><td> order_by=reldate-oldest </td><td> Sort by date package is related to, oldest first. </td></tr></table> <p> Related to <a class="closed ticket" href="http://localhost/ticket/192" title="enhancement: Time-related package field (closed: fixed)">ticket:192</a> </p> http://localhost/ticket/193 http://localhost/ticket/193 Report dread <p> rel=nofollow or robots.txt ? </p> http://localhost/ticket/194 http://localhost/ticket/194 Report rgrp <p> Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name. </p> <p> Cost: 2h </p> http://localhost/ticket/195 http://localhost/ticket/195 Report dread <p> Provide RDF version of a package at the RDF URI: <a class="ext-link" href="http://ckan.net/package/rdf/32000-naples-florida-businesses-kml"><span class="icon">​</span>http://ckan.net/package/rdf/32000-naples-florida-businesses-kml</a> </p> <p> See email on OKFN help: </p> <p> For the information to be Linked Data, the following URI (from the api.talis.com SPARQL call) should be resolvable to RDF, or an RDFa enhanced HTML page. </p> http://localhost/ticket/196 http://localhost/ticket/196 Report dread <h2 id="Asa">As a</h2> <p> admin / user </p> <h2 id="Iwantto">I want to</h2> <p> Admin: export all HEAD data to JSON / CSV / RDF format dump. </p> <p> User: wants to get hold of this dump. </p> <h2 id="Details">Details</h2> <ul><li>Just do JSON &amp; CSV for now. </li><li>Put this in CLI. </li><li>Just do list of packages - as_dict info. CSV needs a bit of flattening. </li></ul><p> Cost: 2h </p> http://localhost/ticket/197 http://localhost/ticket/197 Report dread <p> See how we did it already for other things. </p> <p> Note: on ckan.net older <a class="missing wiki">PackageRevision?</a>.id might not be identical to Package.id but this may need sorting at this point. </p> http://localhost/ticket/198 http://localhost/ticket/198 Report casbon <p> As of 80:099bdc5b07a6 datapkg has 4 test fails and 1 test error. </p> <p> There is no documentation on how to run them, so I am using 'python setup.py test'. </p> <pre class="wiki">datapkg.tests.test_cli.TestCLI.test_ckan ... FAIL datapkg.tests.test_cli.TestCLI.test_walkthrough ... FAIL datapkg.tests.test_index.TestDbIndex.test_get ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/index.py:97: SADeprecationWarning: Use session.add() self.session.update(pkg) FAIL datapkg.tests.test_index.TestDbIndex.test_get_when_loaded_as_new_and_init_not_called ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/tests/test_index.py:57: SADeprecationWarning: Use session.expunge_all() self.index.session.clear() FAIL </pre> http://localhost/ticket/199 http://localhost/ticket/199 Report rgrp <p> Current write method just creates a bare bones package and does not write any of the available metadata other than name. </p> http://localhost/ticket/200 http://localhost/ticket/200 Report dread <p> Check: </p> <ul><li>high numbers of simultaneous users </li><li>incomplete transactions </li><li>etc. </li></ul> http://localhost/ticket/201 http://localhost/ticket/201 Report dread <p> Try i18n for one language (German) for ckan (de.ckan.net). </p> <p> Basic approach to i18n is well-covered in Pylons book: <a class="ext-link" href="http://pylonsbook.com/en/1.1/internationalization-and-localization.html"><span class="icon">​</span>http://pylonsbook.com/en/1.1/internationalization-and-localization.html</a> </p> <p> However, we need to integrate with our Genshi templates. Details on how to do this in comment: <a class="ext-link" href="http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization"><span class="icon">​</span>http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization</a> </p> <h2 id="Steps">Steps</h2> <ol><li>i18n templates <ul><li>Extract content using Babel </li><li>Get genshi translation working </li><li>Do the translation! </li></ul></li><li>i18n all strings in controllers (if any) </li><li>i18n forms </li></ol> http://localhost/ticket/202 http://localhost/ticket/202 Report dread <p> In two places in the WUI </p> <ul><li>list of groups </li><li>group page in title e.g. ukgov (5) </li></ul><p> Cost: 1h </p> http://localhost/ticket/203 http://localhost/ticket/203 Report dread <h2 id="Asa">As a</h2> <p> admin </p> <h2 id="Iwantto">I want to</h2> <p> input government data in a constrained way </p> <h2 id="sothat">so that</h2> <p> you get consistency in the db which is easily searched, discovered and automatically linked. </p> <h2 id="implementation">implementation</h2> <p> Form has fields suitable for government data. (related to custom excel importer ticket) </p> <p> In pylons config specify the form schema to use - whether the existing one or the government one. </p> <p> <strong>On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)</strong> </p> <h3 id="Simplefields">Simple fields</h3> <ul><li>all basic ckan fields </li><li>default license to OKD::Crown Copyright </li><li>co_id, precision, taxonomy url, agency </li></ul><h3 id="Datefields">Date fields</h3> <p> Date format: machine readable. Can be specific or vague: year or month/year or day/month/year. </p> <ul><li>date_released </li></ul><ul><li>date_updated </li></ul><ul><li>temporal_coverage_from, temporal_coverage_to - two date fields combined </li></ul><h3 id="Strictselection">Strict selection</h3> <p> Drop down combo </p> <ul><li>department </li></ul><h3 id="Selectionorfreetext">Selection or free text</h3> <p> Editable combo </p> <ul><li>update_frequency - selections: never, weekly, monthly, annually, discontinued </li></ul><ul><li>geographic_granularity - selections: national, regional, local authority, ward, point </li></ul><ul><li>temporal_granularity - selections: yes, months, weeks, days, hours, point </li></ul><h3 id="Specialisedfields">Specialised fields</h3> <ul><li>geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global </li></ul><p> (Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?) </p> <ul><li>national_statisic - UI is check box, store as 'Yes' or 'No' </li></ul><ul><li>resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets. </li></ul><ul><li>categories - multi values from strict selection of ONS categories. Stored as comma-separated. </li></ul><h3 id="Fielddataprocessed">Field data processed</h3> <ul><li>tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage &amp; categories. </li></ul><h3 id="Future-optional">Future - optional</h3> <p> Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views. </p> <p> When you edit a package it detects the schema based on the extra fields that are present. </p> http://localhost/ticket/204 http://localhost/ticket/204 Report dread <p> Importer function is exposed to logged in users. </p> <p> Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'. </p> <ul><li>Basic is as it is currently </li><li>Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords. </li></ul><p> All fields should be in step with the government custom input form. </p> <p> For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text. </p> <p> Estimate: </p> http://localhost/ticket/205 http://localhost/ticket/205 Report nickstenning <p> Investigate and list (on the wiki at <a class="wiki" href="http://localhost/wiki/UiReviewNotes">wiki:UiReviewNotes</a>) CKAN's UI issues, in particular: </p> <ul><li>Form usability <ul><li>best example: <a class="ext-link" href="http://ckan.net/package/new"><span class="icon">​</span>http://ckan.net/package/new</a> </li><li>"Extras" fields </li><li>Adding to groups (from group page or from package page?) </li></ul></li></ul><ul><li>Overview pages: reduction in visual noise and unnecessary elements. <ul><li>best example: package overview page </li><li>consider openness/downloadability icons </li><li>display common metadata fields in a more structured fashion? </li></ul></li></ul><p> Est cost: 2 hrs. </p> http://localhost/ticket/206 http://localhost/ticket/206 Report nickstenning <p> Remove the current dependency on Yahoo YUI toolkit for doing tag autocompletion on package edit forms. </p> <p> The (KForge-derived) behaviour files for CKAN can easily be switched to the new jQuery-based ones: see <a class="closed ticket" href="http://localhost/ticket/212" title="enhancement: Switch to using m.okfn.org &#34;kforge&#34; media (css and behaviour) (closed: fixed)">ticket:212</a>. Once this is done, it would make sense to implement the tag-selection code with jQuery. </p> <p> Relevant pages include: </p> <ul><li><a class="ext-link" href="http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/"><span class="icon">​</span>http://bassistance.de/jquery-plugins/jquery-plugin-autocomplete/</a> </li><li>e.g. <a class="ext-link" href="http://ckan.net/package/edit/acawiki"><span class="icon">​</span>http://ckan.net/package/edit/acawiki</a> </li></ul> http://localhost/ticket/207 http://localhost/ticket/207 Report rgrp <p> Obfuscate author email and maintainer email field. </p> <p> Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript') </p> <p> Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate? </p> http://localhost/ticket/208 http://localhost/ticket/208 Report rgrp <p> p:wui 2009-09-09 validate email address when creating an enquiry </p> http://localhost/ticket/209 http://localhost/ticket/209 Report johnbywater <p> Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid. </p> http://localhost/ticket/210 http://localhost/ticket/210 Report johnbywater <p> Native login scenario: Complete login form, require email address. Any session is resumed. </p> <p> Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from <a class="missing wiki">BlastOff?</a>. </p> <p> See CKAN or Microfacts for usage of repose with openid. </p> http://localhost/ticket/211 http://localhost/ticket/211 Report rgrp <p> We are already using standard kforge css though copied locally and with a few modifications. </p> <p> We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made. </p> <p> Cost: 2h </p> http://localhost/ticket/212 http://localhost/ticket/212 Report dread <p> fabric script to automate ckan deployment </p> http://localhost/ticket/213 http://localhost/ticket/213 Report dread <ul><li>Set hover names on navigation links. </li><li>Do we really need to display "About CKAN" everywhere? Just on home page. </li><li>Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions. </li></ul> http://localhost/ticket/214 http://localhost/ticket/214 Report dread <ul><li>'Login via OpenID' -&gt; 'Login with OpenID'? </li><li>Elements: <ul><li>Search </li><li>about CKAN </li><li>3 column Packages view (see <a class="closed ticket" href="http://localhost/ticket/219" title="enhancement: UI Review - Package (closed: wontfix)">ticket:219</a>) </li><li>Small tag cloud (RHS) </li></ul></li></ul><p> Also, what about <a class="assigned ticket" href="http://localhost/ticket/143" title="enhancement: Most active users listed on homepage (assigned)">ticket:143</a> ? </p> http://localhost/ticket/215 http://localhost/ticket/215 Report dread <ul><li>allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them &lt;label&gt;s. </li><li>cut 'only's -- use word 'filter' instead </li><li>doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package. </li><li>autocomplete package names &amp; tags </li></ul> http://localhost/ticket/216 http://localhost/ticket/216 Report dread <p> Package search results page <a href="http://localhost/package/search?q=something">package/search</a> </p> <ul><li>need clearer delineation between search entry form and results. </li><li>0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result. </li><li>"N tags found" -- why do I care? this should be presented as "filter results by tag" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search. </li><li>tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background) </li><li>filter out particular tags </li><li>tag cloud? </li><li>copywriting: "Packages - Search" -&gt; "Search packages" </li></ul> http://localhost/ticket/217 http://localhost/ticket/217 Report dread <ul><li>openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS. </li><li>maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence. </li><li>results data should be tabulated, with the project names and descriptions made most prominent. &lt;li&gt; blobs are unnecessary visual noise. </li><li>maybe links should go on the full titles? Get rid of package names. </li><li>Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS? </li><li>same comments as search listing (<a class="closed ticket" href="http://localhost/ticket/217" title="enhancement: UI Review - Search results (closed: fixed)">ticket:217</a>) </li></ul> http://localhost/ticket/218 http://localhost/ticket/218 Report dread <ul><li>Propose three-column package listing: two lists of highest-scorers on some metric and last column is an invitation to register a new package, with a little copy explaining what's involved and why you'd want to. </li><li>Add in search widget </li></ul> http://localhost/ticket/219 http://localhost/ticket/219 Report dread <p> Package page <a href="http://localhost/package/mypackagename">package/mypackagename</a> </p> <ul><li>Design <ul><li>In general: a huge amount of visual and typographic noise: "Package:" is redundant. </li><li>"{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these? </li><li>Openness/downloadability icons: need help, as discussed earlier </li><li>"Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget <ul><li>Rating - alternative to GET on set_rating. </li></ul></li><li>"Metadata:" technospeak, probably redundant. </li><li>&lt;li&gt; bullets add nothing. </li><li>Title, Version, URL can probably go in one line. </li><li>Resources can be made more prominent. </li><li>Author, author email and Maintainer, maintainer email can be merged. </li><li>Groups, tags in little boxes? </li><li>Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface? </li><li>Don't show extras unless there are any. </li></ul></li></ul> http://localhost/ticket/220 http://localhost/ticket/220 Report dread <p> Package edit page <a href="http://localhost/package/edit/mypackagename">package/edit/mypackagename</a> </p> <ul><li>Ability to add the package to groups on this page. </li><li>Fields could be grouped. </li><li>Inconsistent capitalisation (Url -&gt; URL). </li><li>Tag editor YUI script appears to break occasionally. </li><li>Perhaps move to two column forms to save vertical space? </li><li>Notes field to monospace? </li><li>Extras fields need some jQuery love -- shouldn't have a fixed number available. </li><li>Typography is a bit of a mess. </li></ul><p> </p> http://localhost/ticket/221 http://localhost/ticket/221 Report dread <p> Package history page <a href="http://localhost/package/history/mypackagename">package/history/mypackagename</a> </p> <ul><li>"Revisions" is redundant </li><li>"Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it! </li><li>Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links. </li></ul> http://localhost/ticket/222 http://localhost/ticket/222 Report dread <p> Tags landing page <a href="http://localhost/tag">tag</a> </p> <ul><li>Both search for tags *and* browse the list of tags right here </li><li>Big cloud of tags (sorted by popularity/usage?) filterable with a search box. </li><li>Tags search page <a href="http://localhost/tag/search">tag/search</a> and list <a href="http://localhost/tag/list">tag/list</a> are both redundant and should be merged straight into <a href="http://localhost/tag">tag</a>. </li></ul> http://localhost/ticket/223 http://localhost/ticket/223 Report dread <p> Groups landing page <a href="http://localhost/group">group</a> </p> <ul><li>Yeesh! What's up with the typography for "Just login..."? </li><li>Truncated description along with group name. </li><li>Explanation text about what groups are for (RHS) </li><li>Another case of &lt;li&gt; bullets adding nothing. </li></ul> http://localhost/ticket/224 http://localhost/ticket/224 Report dread <p> Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure. </p> http://localhost/ticket/225 http://localhost/ticket/225 Report dread <p> Repository History <a href="http://localhost/revision">revision</a> </p> <ul><li>What is this page called? "Recent Changes", "Repository History", "/revision": standardise. Between the link in the nav and the page &lt;title&gt; particularly, but the route is also important. Perhaps /changes or something similar? <ul><li>Will change page title on /revision/ to Revision History. Will not change route for the time being. </li></ul></li><li><del>needs copyediting.</del> </li><li><del>Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.</del> </li><li><del>Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say "Older"/"More recent" rather than "Previous"/"Next".</del> (wontfix: now have text saying we are showing most recent changes) </li><li>Table layout is pretty ugly (yes, I'm aware this is my fault). <ul><li>wontfix - nothing better at the moment </li></ul></li><li>Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail? </li><li>Can we link to an author page? Yes! </li><li><del>Atom feed should have a feed icon~~ </del></li><li>Why are we adding another &lt;link rel="alternate"&gt; to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page. </li></ul> http://localhost/ticket/226 http://localhost/ticket/226 Report dread <ul><li>There is huge inconsistency in the titles of pages across the site: to give one example "Edit Package:&#34; vs &#34;Package: mypkg [not linked]" vs "History - mypkg [linked to package page]" <ul><li>Normalize and use terminology: "Data Package" - also change in navbar (but nowhere else for the time being). So hvae Data Package - mypkg, Data Package - mypkg - Edit etc </li></ul></li><li>pagination is currently lost in amongst the results list. needs to be *much* more obvious, and should appear top and bottom, or at the very least at the bottom! </li></ul> http://localhost/ticket/227 http://localhost/ticket/227 Report rgrp <p> This needs to be thought out ... </p> http://localhost/ticket/228 http://localhost/ticket/228 Report dread <p> New field for resources - hash of the resource file. </p> <ul><li>CKAN itself will not calculate the hash value - user just pastes it in. </li><li>Display text field in resource table. </li><li>Requires migration script. </li></ul><p> Questions for the field's value: </p> <ol><li>Which hash to use? Restrict to python hashlib and other major hash libraries. </li></ol><ol start="2"><li>Should we use merkle trees? </li></ol><p> Thanks to Julien D'Assanges for the suggestion. </p> http://localhost/ticket/229 http://localhost/ticket/229 Report dread <ul><li>Requires Package Read to not use the side-bar </li><li>Need to sync params to a package object that is taken out of the session so that it doesn't go into the database. </li></ul> http://localhost/ticket/230 http://localhost/ticket/230 Report rgrp <p> Make group object revisioned/versioned. </p> <ul><li>Involves upgrade script both for group object and group_revision object </li><li>No changes required to WUI for present </li></ul> http://localhost/ticket/231 http://localhost/ticket/231 Report rgrp <p> Strapline (?): Sharing, discovery and reuse of data and content </p> <p> Need to consult on this. </p> <p> Cost: 1h </p> http://localhost/ticket/232 http://localhost/ticket/232 Report rgrp <p> Options: </p> <ol><li>Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works) </li></ol><ol><li>Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist). </li></ol><p> Can do this using: </p> <p> &lt;xi:include href=&#34;base.html&#34;&gt;&lt;xi:fallback /&gt;&lt;/xi:include&gt; </p> http://localhost/ticket/233 http://localhost/ticket/233 Report rgrp <p> Broken out from <a class="closed ticket" href="http://localhost/ticket/216" title="enhancement: UI Review - Search (closed: fixed)">ticket:216</a> </p> http://localhost/ticket/234 http://localhost/ticket/234 Report dread <p> Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (<a class="ext-link" href="https://bitbucket.org/pudo/dcat-tools"><span class="icon">​</span>https://bitbucket.org/pudo/dcat-tools</a>), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service. </p> <p> Sub-tasks: </p> <ul><li>Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -&gt; "text/csv" and "Name mappings", i.e. "text/csv" -&gt; "Comma-Separated Spreadsheet". </li><li>Collect and map surface forms from all CKANs </li><li>Access this via Swiss and apply, store as a <a class="missing wiki">PackageResource?</a> extra field pending <a class="closed ticket" href="http://localhost/ticket/826" title="enhancement: Resource 'extra' fields (closed: fixed)">#826</a> (Resource extras). </li><li>Add heuristics for format auto-detections: <ul><li>Map well-known file extensions </li><li>Recognize obvious magic (Zip, Tar) </li><li>Peek into <a class="missing wiki">Zipfile/Tarfiles?</a> </li></ul></li><li>Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar) </li><li>See also: <a class="closed ticket" href="http://localhost/ticket/816" title="enhancement: Autocomplete for the resource format field (closed: fixed)">#816</a> (Autocomplete for the resource format field) </li></ul> http://localhost/ticket/235 http://localhost/ticket/235 Report johnbywater http://localhost/ticket/236 http://localhost/ticket/236 Report johnbywater http://localhost/ticket/237 http://localhost/ticket/237 Report johnbywater http://localhost/ticket/238 http://localhost/ticket/238 Report dread <p> Integrate new test upgrade into testmigrate stuff. </p> <p> New "migrate.ini" config file programmed with the db to be used for migrate tests and a param saying where the db dump is. </p> http://localhost/ticket/239 http://localhost/ticket/239 Report rgrp <p> Currently it is possible (v0.11) to have a <a class="missing wiki">PackageResource?</a> object without a url. This should be impossible to do (either at creation time or via modification). </p> http://localhost/ticket/240 http://localhost/ticket/240 Report dread <p> 4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package. </p> http://localhost/ticket/241 http://localhost/ticket/241 Report rgrp <p> Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp </p> <ul><li>controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)? </li></ul><p> done: </p> <ul><li>lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have <a class="missing wiki">SearchPackage?</a>, <a class="missing wiki">SearchTag?</a> etc) </li></ul> http://localhost/ticket/242 http://localhost/ticket/242 Report dread <p> <a class="ext-link" href="http://ckan.net/group/publicdomain"><span class="icon">​</span>http://ckan.net/group/publicdomain</a> has: </p> <p> "There is 16 package in this group. There are 16 packages in this group." </p> http://localhost/ticket/243 http://localhost/ticket/243 Report dread <p> e.g. Text on ckan.net home page: </p> <p> "All material available under an /licenseopen license" </p> <p> Also seen on the package edit page: "/user/loginClick here to sign in" </p> <p> I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime. </p> http://localhost/ticket/244 http://localhost/ticket/244 Report rgrp <p> Problem here is that foreign key then becomes "complicated" (composite). </p> <ul><li>Could also deprecate continuity_id field in favour of the basic foreign key on ie </li></ul> http://localhost/ticket/245 http://localhost/ticket/245 Report rgrp <p> At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of <a class="closed ticket" href="http://localhost/ticket/245" title="enhancement: Support for composite primary keys (closed: duplicate)">ticket:245</a> (composite primary keys) </p> http://localhost/ticket/246 http://localhost/ticket/246 Report rgrp <p> "Alternative formats". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f). </p> <ul><li>How do we know if RDF available? Ans: see details in <a class="closed ticket" href="http://localhost/ticket/90" title="enhancement: Link to RDF versions of CKAN data (closed: fixed)">ticket:90</a>. </li></ul><p> Cost: 1h </p> http://localhost/ticket/247 http://localhost/ticket/247 Report rgrp <p> Need to do i18n of forms (not in templates but in code). Should consult with tonfa about this. </p> http://localhost/ticket/248 http://localhost/ticket/248 Report dread <p> Search of package name and title (and other fields) using regular expressions. </p> <p> Current example use-case: Wanting to specify packages with title beginning with 'B'. </p> <p> Issues: </p> <ul><li>Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields? </li></ul><p> Implementation: </p> <ul><li>Postgres reg ex searching detailed here: </li></ul><p> <a class="ext-link" href="http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE"><span class="icon">​</span>http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE</a> </p> http://localhost/ticket/249 http://localhost/ticket/249 Report dread <p> Add link to RDF representation of a package in our Atom feed. </p> http://localhost/ticket/250 http://localhost/ticket/250 Report johnbywater <p> Want to have: </p> <ul><li>/api/search/revision/ </li></ul><p> Support querying by "since time" and "since revision". Each individual revision should be available in the REST API (read-only): </p> <ul><li>/api/rest/revision/{revision ID} <ul><li>should include revision attributes plus revision.packages e.g. like { "id": "...", ... "packages": [ list of packages names from e.g. revision.packages ] } </li></ul></li></ul> http://localhost/ticket/251 http://localhost/ticket/251 Report johnbywater http://localhost/ticket/252 http://localhost/ticket/252 Report dread <h1 id="Overview">Overview</h1> <p> Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management. </p> <h1 id="Examples">Examples</h1> <ol><li>There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial. </li></ol><ol start="2"><li>In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.) </li></ol><ol start="3"><li>OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it. </li></ol><ol start="4"><li>IPCC collection of data linked / mirrored. Not sure if there are useful relationships here? </li></ol><ol start="5"><li>Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM. </li></ol><p> See more examples discussed here: <a class="ext-link" href="http://trac.ckan.org/ticket/253"><span class="icon">​</span>http://trac.ckan.org/ticket/253</a> </p> <h1 id="Implementation">Implementation</h1> <p> This is split into four tickets: </p> <ul><li>Model: <a class="closed ticket" href="http://localhost/ticket/254" title="enhancement: Package relationships - 1. model (closed: fixed)">ticket:254</a> </li><li>Read in WUI: <a class="closed ticket" href="http://localhost/ticket/255" title="enhancement: Package relationships - 2. Read in WUI (closed: fixed)">ticket:255</a> </li><li>Edit in WUI: <a class="assigned ticket" href="http://localhost/ticket/256" title="requirement: Package relationships - 3. Edit in WUI (assigned)">ticket:256</a> </li><li>API: <a class="closed ticket" href="http://localhost/ticket/257" title="enhancement: Package relationships - 4. Read in API (closed: fixed)">ticket:257</a> </li></ul><p> No need for write access to be provided API for the moment. </p> <p> This ticket also encompasses <a class="closed ticket" href="http://localhost/ticket/169" title="enhancement: Package derivations (closed: duplicate)">ticket:169</a> (Package derivations) and <a class="closed ticket" href="http://localhost/ticket/176" title="enhancement: Package dependencies (closed: duplicate)">ticket:176</a> (Package dependencies). </p> http://localhost/ticket/253 http://localhost/ticket/253 Report dread <p> New domain object: <a class="missing wiki">PackageRelationship?</a> (revisioned) </p> <p> Attributes: </p> <ul><li>subject (Package reference) </li><li>object (Package reference) </li><li>type (string) </li><li>comment </li></ul><p> Relationship type values: depends_on (dependency_of) derives_from (has_derivation) child_of (parent_of) </p> <p> Relationship type is stored as a string, as given in the first column. The reverse relationship (bracketed) is given just for display purposes only. </p> <p> Subject / Object - as in a sentence: "Dracos is derived from OSM" is &lt;subject&gt; &lt;predicate&gt; &lt;object&gt;. We'll use 'type' instead of 'predicate' as that is a more familiar word. </p> http://localhost/ticket/254 http://localhost/ticket/254 Report dread <p> WUI: </p> <ul><li>View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...) </li></ul> http://localhost/ticket/255 http://localhost/ticket/255 Report dread <p> WUI: </p> <ul><li>Editable as part of package or separately? (e.g. like authz) </li><li>Do we normalize to only one type name of the pair? </li><li>Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?) </li></ul> http://localhost/ticket/256 http://localhost/ticket/256 Report dread <p> API: </p> <ul><li>Appear in package listing Example: 'relationships': [{'is_dependency_of':'osm', comments:'Since version 0.2'}, {'is_parent_of':'bobs_maps'}] </li></ul> http://localhost/ticket/257 http://localhost/ticket/257 Report dread <p> Add uuids into package in REST interface. </p> http://localhost/ticket/258 http://localhost/ticket/258 Report dread <p> Pressing the 'make an enquiry' link on the package read page gives 404. (Also links should also be changed from isitopen.ckan.net to isitopendata.org) </p> <p> Example link is: </p> <p> <a class="ext-link" href="http://isitopen.ckan.net/enquiry/create/?ckan_package=zeno"><span class="icon">​</span>http://isitopen.ckan.net/enquiry/create/?ckan_package=zeno</a> </p> <p> which gets forwarded to: </p> <p> <a class="ext-link" href="http://www.isitopendata.org/enquiry/create/?ckan_package=zeno"><span class="icon">​</span>http://www.isitopendata.org/enquiry/create/?ckan_package=zeno</a> </p> <p> which gives 404 </p> http://localhost/ticket/259 http://localhost/ticket/259 Report rgrp <p> Currently use old "bullet-point" style. Should change to now standard style used on e.g. package search results page. </p> <p> Cost: 0.5h </p> http://localhost/ticket/260 http://localhost/ticket/260 Report rgrp <p> Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted). </p> <p> Using normal license field does not cause breakage but has no effect. </p> <p> (Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install). </p> <pre class="wiki"> ourjson = '''{ "maintainer" : "Applications Division", "version" : "1999-01-01", "license_id" : 9, "name" : "geogratiswaterfraction1km_v100", "author_email" : "...", "author" : "Dissemination Branch", "tags" : [ "GeoGratis", "Canada", "GCMD", "Hydrosphere", "Ground Water", "Surface Water", "Hydropattern", "Earth Science", "Land Surface", "Land Cover", "Landscape" ], "extras" : { "License" : "http://geogratis.ca/geogratis/en/licence.jsp", "Ministry" : "Natural Resources Canada", "Level of Government" : "Federal" }, "maintainer_email" : "richard.fernandes@nrcan.gc.ca", "notes" : "This product is a raster coverage", "url" : "http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67", "title" : "Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada" } ''' def test_15_random(self): offset = '/api/rest/package' import urllib tmpjson = simplejson.dumps(simplejson.loads(self.ourjson)) postparams = urllib.urlencode({tmpjson: 1}) res = self.app.post(offset, params=postparams, status=[200], extra_environ=self.extra_environ) model.Session.remove() </pre> http://localhost/ticket/261 http://localhost/ticket/261 Report rgrp <p> Need a way to mark packages as duplicate. </p> <p> <a class="missing wiki">Ideas/Questions?</a>: </p> <ul><li>Could work off a tag (meta-duplicate or somesuch) </li><li>What happens if we want to delete the package (to we leave a place-holder, do we merge content?) </li></ul> http://localhost/ticket/262 http://localhost/ticket/262 Report rgrp <p> At the moment the user has to figure out to login with providers -- this is not always obvious (e.g. google). </p> <p> We could improve this with a small bit of javascript. As an example see: <a class="ext-link" href="http://standalone.demo.civicrm.org/standalone/"><span class="icon">​</span>http://standalone.demo.civicrm.org/standalone/</a> </p> <p> Not sure exactly what library that is using (or if bespoke) so alternatives include (NB: we already use jquery): </p> <ul><li><a class="ext-link" href="http://code.google.com/p/openid-realselector/"><span class="icon">​</span>http://code.google.com/p/openid-realselector/</a> </li><li><a class="ext-link" href="http://code.google.com/p/openid-selector/"><span class="icon">​</span>http://code.google.com/p/openid-selector/</a> </li></ul><p> In addition to implementing this we also need to test logging in with main providers: google, wordpress -- as we have had reports of it not working with e.g. wordpress (not sure if this testing can be automated really -- best hope would be selenium I think -- so OK if done by hand). </p> <p> Cost: 3h (2h UI), (1h testing) </p> http://localhost/ticket/263 http://localhost/ticket/263 Report rgrp <p> Revision.packages should not include all changes so why are we getting revisions with no associated changes? See e.g. <a class="ext-link" href="http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb"><span class="icon">​</span>http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb</a>? </p> <p> Investigate and fix. </p> <p> NB: if this is bugfix this should go against stable or metastable branch if at all possible and then be merged into default. </p> <p> Cost: 1.5h (?) </p> http://localhost/ticket/264 http://localhost/ticket/264 Report dread <h2 id="Usecase">Use case</h2> <p> As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package. </p> <h2 id="Implementation">Implementation</h2> <p> Add into the Atom feed: </p> <ol><li>a package is a new one </li><li>the resources have been changed </li><li>the last_updated field has been changed </li></ol> http://localhost/ticket/265 http://localhost/ticket/265 Report dread <p> As a user I want to track changes to a particular package. </p> <p> Provide an Atom feed, linked from the package read page. </p> http://localhost/ticket/266 http://localhost/ticket/266 Report rgrp <p> Currently this raises an exception because there is no package/package_revision in existence at from_revision. </p> http://localhost/ticket/267 http://localhost/ticket/267 Report dread http://localhost/ticket/268 http://localhost/ticket/268 Report dread <p> If the notes field could use a WYSIWYG editor with word cleanup this would really help users who may well be pasting in text from Word (and I guess may have been approved or written by someone else) </p> <p> having auto-complete on tags would both make the system easier to use - and reduce the risk of synonymous tags being created inadvertently. </p> <p> Does the department drop down options list interact with user permissions - so that users who only have rights to one department only see that option ? </p> <p> Similarly for licenses - can this reduce to the set of allowed options for this user/ </p> <p> What level of validation is there on the fields? </p> http://localhost/ticket/269 http://localhost/ticket/269 Report johnbywater <p> There is code to create packages in-process (ckan/lib/importer) and via HTTP (ckanclient/loaders/base). There is also code to read different kinds of package source (Google Spreadsheets spreadsheet, other spreadsheet). </p> <p> Propose to refactor all this to have template method in base command class (subclasses may implement command line interface, graphical user interface, or Web interface), where the template method calls on aggregated strategy objects for: reading raw spreadsheet data from spreadsheet application; for inferring CKAN package entities from the raw spreadsheet data; and for writing package entities to CKAN service. </p> <p> Spreadsheet readers to be factored to allow for variations of spreadsheet application. </p> <p> Raw data inferrers to allow for variations in spreadsheet cell structure. </p> <p> CKAN writer strategies to allow for either CKAN client or direct linking. </p> <p> Actual cases then supported by specifying existing strategies (or defining new ones) in a script that invokes the command class. </p> http://localhost/ticket/270 http://localhost/ticket/270 Report johnbywater <p> See <a class="wiki" href="http://localhost/wiki/DistributingChanges">DistributingChanges</a> for analysis and functional requirements. </p> http://localhost/ticket/271 http://localhost/ticket/271 Report rgrp <p> I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group. </p> <ul><li>This would allow people to embed feed widgets in other places </li></ul> http://localhost/ticket/272 http://localhost/ticket/272 Report rgrp <p> Work already here: <a class="ext-link" href="http://knowledgeforge.net/ckan/trac/wiki/SearchEngine"><span class="icon">​</span>http://knowledgeforge.net/ckan/trac/wiki/SearchEngine</a> </p> <p> Add knowledge there or to: </p> <p> <a class="ext-link" href="http://wiki.okfn.org/SoftwareTools/Search"><span class="icon">​</span>http://wiki.okfn.org/SoftwareTools/Search</a> </p> http://localhost/ticket/273 http://localhost/ticket/273 Report rgrp <p> At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...) </p> <p> Cost: 1h </p> http://localhost/ticket/274 http://localhost/ticket/274 Report rgrp <p> This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type) </p> <p> Cost: 0.5h </p> <pre class="wiki">from ckan.model.types import JsonType forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer </pre> http://localhost/ticket/275 http://localhost/ticket/275 Report rgrp <p> Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'. </p> http://localhost/ticket/276 http://localhost/ticket/276 Report dread <h2 id="Usecase">Use case</h2> <p> As a ckan administrator I want to easily change options about the CKAN install. </p> <h2 id="Implementation">Implementation</h2> <h3 id="SettingstobeinDB">Settings to be in DB</h3> <p> Suggested: </p> <pre class="wiki">## Title of site (using in several places including templates and &lt;title&gt; tag ckan.site_title = CKAN ## Logo image to use (replaces site_title string on front page if defined) ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png ## Site tagline / description (used on front page) ckan.site_description = ## Used in creating some absolute urls (such as rss feeds, css files) and ## dump filenames ckan.site_url = ## Favicon (default is the CKAN software favicon) ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico ## An 'id' for the site (using, for example, when creating entries in a common search index) ## If not specified derived from the site_url # ckan.site_id = ckan.net ## API url to use (e.g. in AJAX callbacks) ## Enable if the API is at a different domain # ckan.api_url = http://www.ckan.net ## html content to be inserted just before &lt;/body&gt; tag (e.g. google analytics code) ## NB: can use html e.g. &lt;strong&gt;blah&lt;/strong&gt; ## NB: can have multiline strings just indent following lines # ckan.template_footer_end = </pre><p> NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ... </p> <h3 id="SettingsOptionsKeyValuesTable">Settings / Options / <a class="missing wiki">KeyValues?</a> Table</h3> <p> Columns: </p> <ul><li>[namespace]: ? only if <a class="missing wiki">KeyValues?</a> (for settings this would then always be settings) </li><li>key </li><li>label </li><li>value (json) </li><li>type (e.g. date and to specify in advance what type should be) </li><li>description </li><li>tags: ?? (for grouping ...) </li></ul><h3 id="LoadingsettingsfromDB">Loading settings from DB</h3> <p> Do this in ckan/config/environment.py </p> <h3 id="WUI">WUI</h3> <ul><li>/ckan-admin/settings </li><li>Show label, plus description plus text field </li></ul><h2 id="Depends">Depends</h2> <ul><li>Would be part of ckan-admin section and hence build on <a class="closed ticket" href="http://localhost/ticket/833" title="enhancement: [super] Administrative dashboard extension (closed: fixed)">ticket:833</a> (Administrative dashboard) </li></ul> http://localhost/ticket/277 http://localhost/ticket/277 Report dread <p> If the key to an extra field has a space in it then various form code doesn't work. Either need to disallow spaces or cope with them. </p> http://localhost/ticket/278 http://localhost/ticket/278 Report dread <p> With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost. </p> <p> So for this case we should add the package's license to the list displayed. </p> http://localhost/ticket/279 http://localhost/ticket/279 Report dread <p> When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear. </p> <p> Metastable branch. </p> http://localhost/ticket/280 http://localhost/ticket/280 Report dread http://localhost/ticket/281 http://localhost/ticket/281 Report dread <p> When a package's name is changed, references to the old name (in the WUI and REST) are redirected to the new name (assuming the old name is not being reused by another package). </p> http://localhost/ticket/282 http://localhost/ticket/282 Report dread <h2 id="Usecase">Use case</h2> <p> As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion. </p> <h2 id="Suggestedsolution">Suggested solution</h2> <p> In the package view side-bar, there is a note: "To have this package completely removed, contact the [ca.ckan.net administrators admin@…]." </p> <h2 id="Othersolutions">Other solutions</h2> <p> A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill. </p> <p> c.f. <a class="ext-link" href="http://en.wikipedia.org/wiki/Deleting_an_article"><span class="icon">​</span>http://en.wikipedia.org/wiki/Deleting_an_article</a> </p> http://localhost/ticket/283 http://localhost/ticket/283 Report rgrp <p> Also put in redirect from old locations (see examples with plurals in routing.py) </p> <p> Estimated cost: 20m </p> <p> Actual cost: 40m </p> http://localhost/ticket/284 http://localhost/ticket/284 Report rgrp <p> Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ... </p> http://localhost/ticket/285 http://localhost/ticket/285 Report dread <p> Use Nick's examples as a basis to revamp the formalchemy usage, principally in the package edit form. </p> <ul><li><a class="ext-link" href="http://static.whiteink.com/css/examples/forms.html"><span class="icon">​</span>http://static.whiteink.com/css/examples/forms.html</a> </li><li><a class="ext-link" href="http://whiteink.com/temp/ckan-forms/"><span class="icon">​</span>http://whiteink.com/temp/ckan-forms/</a> </li></ul> http://localhost/ticket/286 http://localhost/ticket/286 Report dread <p> Spec - see attached, which is a copy of the Pirate Pad from Lauren: <a class="ext-link" href="http://piratepad.net/2C2iwiLDhd"><span class="icon">​</span>http://piratepad.net/2C2iwiLDhd</a> </p> http://localhost/ticket/287 http://localhost/ticket/287 Report dread <p> These don't work. </p> http://localhost/ticket/288 http://localhost/ticket/288 Report dread http://localhost/ticket/289 http://localhost/ticket/289 Report dread <p> doc/api.rst needs to cover the new Revision REST interface. </p> http://localhost/ticket/290 http://localhost/ticket/290 Report dread <p> On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in. </p> <pre class="wiki">diff -r 813ad8b5de0b ckan/lib/search.py --- a/ckan/lib/search.py Mon Mar 01 22:23:36 2010 +0100 +++ b/ckan/lib/search.py Thu Apr 15 19:11:42 2010 +0200 @@ -63,6 +63,9 @@ def query(self, options): '''For the given search options, returns a query object.''' self._options = options + # temporary hack! + if self._options.q: + self._options.q = self._options.q.encode('utf8') general_terms, field_specific_terms = self._parse_query_string() if not general_terms and \ </pre> http://localhost/ticket/291 http://localhost/ticket/291 Report dread <p> When you edit a package, all of its resources are deleted and a fresh set created. This causes all history of them to be lost. </p> http://localhost/ticket/292 http://localhost/ticket/292 Report dread <p> This is caused by editing a package with resources over the REST interface. </p> <p> Exception: </p> <pre class="wiki">URL: http://de.ckan.net/api/rest/package/destatis Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module repoze.who.middleware:107 in __call__ &lt;&lt; wrapper = StartResponseWrapper(start_response) app_iter = app(environ, wrapper.wrap_start_response) # The challenge decider almost(?) always needs information from the &gt;&gt; app_iter = app(environ, wrapper.wrap_start_response) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:51 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckan.controllers.rest:145 in update &lt;&lt; fs = ckan.forms.group_fs_combined fs = fs.bind(entity, data=request_fa_dict) validation = fs.validate_on_edit(entity.name, entity.id) if not validation: response.status_int = 409 &gt;&gt; validation = fs.validate_on_edit(entity.name, entity.id) Module ckan.forms.package:385 in validate_on_edit &lt;&lt; temp_name = orig_pkg_name self.data['Package-%s-name' % record_id] = u'something_unique' validation = self.validate() if temp_name: # restore it &gt;&gt; validation = self.validate() Module formalchemy.forms:175 in validate &lt;&lt; if self.readonly: raise Exception('Cannot validate a read-only FieldSet') return AbstractFieldSet.validate(self) def sync(self): &gt;&gt; return AbstractFieldSet.validate(self) Module formalchemy.forms:104 in validate &lt;&lt; success = True for field in self.render_fields.itervalues(): success = field._validate() and success # run this _after_ the field validators, since each field validator # resets its error list. we want to allow the global validator to add &gt;&gt; success = field._validate() and success Module formalchemy.fields:763 in _validate &lt;&lt; try: value = self._deserialize() except validators.ValidationError, e: self.errors.append(e) &gt;&gt; value = self._deserialize() Module formalchemy.fields:1354 in _deserialize &lt;&lt; if self.is_composite_foreign_key: return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize())) return self.renderer.deserialize() &gt;&gt; return self.renderer.deserialize() Module formalchemy.fields:183 in deserialize &lt;&lt; if self.field.is_collection: return [self._deserialize(subdata) for subdata in self._serialized_value()] return self._deserialize(self._serialized_value()) def _deserialize(self, data): &gt;&gt; return self._deserialize(self._serialized_value()) Module formalchemy.fields:167 in _serialized_value &lt;&lt; if self.field.is_collection: return self._params.getall(self.name) return self._params.getone(self.name) def deserialize(self): &gt;&gt; return self._params.getone(self.name) Module formalchemy.base:63 in getone &lt;&lt; if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage): return v return v[0] def getall(self, key): v = dict.get(self, key) &gt;&gt; return v[0] TypeError: 'int' object is unsubscriptable </pre><p> Thanks to Friedrich for spotting this. </p> http://localhost/ticket/293 http://localhost/ticket/293 Report dread <p> Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources. </p> http://localhost/ticket/294 http://localhost/ticket/294 Report dread <p> In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add. </p> <p> Ticket is split off from <a class="closed ticket" href="http://localhost/ticket/221" title="enhancement: UI Review - Package edit (closed: fixed)">ticket:221</a> </p> <p> See also <a class="closed ticket" href="http://localhost/ticket/155" title="enhancement: Adding multiple packages to a group (closed: duplicate)">ticket:155</a> (duplicate). </p> http://localhost/ticket/295 http://localhost/ticket/295 Report johnbywater http://localhost/ticket/296 http://localhost/ticket/296 Report johnbywater http://localhost/ticket/297 http://localhost/ticket/297 Report johnbywater http://localhost/ticket/298 http://localhost/ticket/298 Report johnbywater http://localhost/ticket/299 http://localhost/ticket/299 Report dread <p> Failing test: ckan.tests.models.test_resource.<a class="missing wiki">TestResourceLifecycle?</a>.test_03_reorder_resources </p> <p> Not clear how visible this is to the user. </p> <p> Related to <a class="closed ticket" href="http://localhost/ticket/292" title="defect: PackageResources are created new on every package edit (closed: fixed)">ticket:292</a> </p> http://localhost/ticket/300 http://localhost/ticket/300 Report rgrp <p> A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package. </p> <p> It provides a way for people to make suggestions about a package without needing access to main package. </p> http://localhost/ticket/301 http://localhost/ticket/301 Report dread <p> Visual display of a package's feed in the Package Read (/package/xyz) and Package History (/package/history/xyz) pages. </p> <p> Motivation: each package's Atom feed is mentioned in the &lt;link rel="alternate" type="application/atom+xml" ...&gt; tag, which Firefox displays with a feed icon, but the other 3/4 of users may not discover it. </p> <p> The Repo History (/revision) page has a visual link the feed. </p> <p> PS It would be nice to display the feed icon in both these places too. </p> http://localhost/ticket/302 http://localhost/ticket/302 Report dread <p> Currently a package's history page (/package/history/xyz) and related feed only shows changes to the core package i.e. ones which create a <a class="missing wiki">PackageResource?</a>. This doesn't include changes to tags, extras and resources. These need to be added. </p> <p> There is the complication that the 'diff' function doesn't currently work beyond <a class="missing wiki">PackageResource?</a>. Unless this is quick and simple to fix, I suggest we let the diff remain slightly broken for now. </p> <p> Rufus suggests alongside the existing methods: </p> <pre class="wiki">pkg.all_revisions() pkg.diff() </pre><p> we have ones which also include the related objects (tags, extras, resources): </p> <pre class="wiki">pkg.all_revisions_full() pkg.diff_full() </pre><p> You could have a function which returned these related objects. Ideas: </p> <pre class="wiki">revobjlist = [ self, resources, extras ] list of tuples: (revobj, changed_revobjs) list of tuples: (revobj, {objtype: [changed_revobjs]}) tuple: (rev1, {Package: [ object_id, object_id], PackageTag: [object_id, object_id2] ...} </pre> http://localhost/ticket/303 http://localhost/ticket/303 Report johnbywater <p> Intended outcome: </p> <p> See resource url validation error when previewing package with a resources that has no url. </p> <p> Recurring Error: </p> <pre class="wiki">URL: http://127.0.0.1:5000/package/new File 'pyenv-ckan/lib/python2.5/site-packages/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File 'pyenv-ckan/lib/python2.5/site-packages/repoze/who/middleware.py', line 107 in __call__ app_iter = app(environ, wrapper.wrap_start_response) File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Routes-1.10.3-py2.5.egg/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File 'pyenv-ckan/src/ckan/ckan/lib/base.py', line 50 in __call__ return WSGIController.__call__(self, environ, start_response) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 216 in new clear_session=True) File 'pyenv-ckan/src/ckan/ckan/controllers/package.py', line 357 in _render_edit_form edit_form_html = fs.render() File 'pyenv-ckan/src/ckan/ckan/forms/builder.py', line 13 in render return render(self.form_template) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render format=format, namespace=kargs, **cache_args) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render **options) File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 533 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate File 'pyenv-ckan/src/ckan/ckan/templates/package/form_fields.html', line 52 in &lt;Expression u'h.literal(field.render())'&gt; ${h.literal(field.render())} File 'pyenv-ckan/lib/python2.5/site-packages/FormAlchemy-1.3.4-py2.5.egg/formalchemy/fields.py', line 1217 in render return self.renderer.render(**opts) File 'pyenv-ckan/src/ckan/ckan/forms/common.py', line 311 in render return render('package/form_resources') File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 663 in render format=format, namespace=kargs, **cache_args) File 'pyenv-ckan/lib/python2.5/site-packages/Pylons-0.9.7-py2.5.egg/pylons/templating.py', line 546 in render **options) File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/plugin.py', line 110 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 179 in render File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 60 in encode File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 425 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 592 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 698 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/output.py', line 532 in __call__ File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/core.py', line 283 in _ensure File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 569 in _include File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/markup.py', line 245 in _match File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 543 in _exec File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 510 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 520 in _eval File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/base.py', line 286 in _eval_expr File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 180 in evaluate File 'pyenv-ckan/src/ckan/ckan/templates/package/form_resources.html', line 14 in &lt;Expression u"res['id'] if res else ''"&gt; &lt;td&gt;&lt;input name="${c.id}-${i}-id" type="hidden" value="${res['id'] if res else ''}" /&gt;&lt;/td&gt; File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 335 in lookup_item File 'pyenv-ckan/lib/python2.5/site-packages/Genshi-0.5.1-py2.5-linux-i686.egg/genshi/template/eval.py', line 405 in undefined UndefinedError: {'url': u'a', 'hash': u'a', 'description': u'a', 'format': u'a'} has no member named "id" </pre><p> How to reproduce: </p> <ol><li>Go to package edit form, enter a package resource <strong>but without a URL</strong>. </li><li>Click preview. </li></ol> http://localhost/ticket/304 http://localhost/ticket/304 Report johnbywater <p> Expect to page through results. </p> <p> Only page 1 is shown, all other pages fail to display. </p> <p> Reproduce by searching for something common and browsing to the second page. </p> http://localhost/ticket/305 http://localhost/ticket/305 Report rgrp <p> Need to be able to build a distribution. Need: </p> <ul><li>new 'build' command </li><li>specify distribution format. Suggest at the moment a simple zip or tar.gz build in most straightforward way form distribution. </li></ul> http://localhost/ticket/306 http://localhost/ticket/306 Report rgrp <p> Support uploading a distribution somewhere. </p> <ul><li>New cli command: upload </li><li>Pluggable uploader framework. Suggest initial support </li></ul><p> Depends on <a class="closed ticket" href="http://localhost/ticket/306" title="enhancement: datapkg build command (closed: duplicate)">ticket:306</a> </p> http://localhost/ticket/307 http://localhost/ticket/307 Report rgrp <p> Extracted from <a class="closed ticket" href="http://localhost/ticket/216" title="enhancement: UI Review - Search (closed: fixed)">ticket:216</a>. </p> <p> Dubious of its merit. </p> http://localhost/ticket/308 http://localhost/ticket/308 Report dread <p> Deleted tags are visible on the /tag page and are searchable. </p> <p> Delete tags are ones that were in use once and then discarded, so this includes ones which are not used on any packages any more, including misspelling, mistakes, and deprecated tags. It would makes sense to remove them. </p> <p> Example tag that is viewed at /tag and searchable: <a class="ext-link" href="http://ckan.net/tag/2rand[0"><span class="icon">​</span>http://ckan.net/tag/2rand[0</a> </p> http://localhost/ticket/309 http://localhost/ticket/309 Report rgrp <p> Suggest move this below the label and make full width of screen and only 3/4 rows high (more like a wiki site). </p> <ul><li>Also change label to: Edit summary (Briefly describe the changes you have made) </li><li>Remove: you can markdown formatting here. </li><li>Move author: if you have not signed in smaller and closer (like markdown instructions are nwo). </li><li>Change commit -&gt; save </li><li>Remove "please save" just have the bullet points </li></ul> http://localhost/ticket/310 http://localhost/ticket/310 Report rgrp <p> Created a new package resource line and then moved it up above existing one and hit save. Result: 500 error. In logs have: </p> <pre class="wiki">[Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] Error - &lt;class 'sqlalchemy.exceptions.IntegrityError'&gt;: (IntegrityError) duplicate key value violates unique constraint "package_resource_revision_pkey", referer: http://ckan.net/package/edit/cofog [Sat May 08 21:55:41 2010] [error] [client 86.26.8.30] 'INSERT INTO package_resource_revision (id, package_id, url, format, description, hash, position, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(url)s, %(format)s, %(description)s, %(hash)s, %(position)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'hash': '', 'description': 'The Treasury record of COFOG functions. ', 'format': 'XLS', 'url': 'http://www.hm-treasury.gov.uk/d/cofog_definitions_coins250609.xls', 'package_id': '8482334d-fe2e-4285-9114-5243130f80c0', 'state': 'active', 'continuity_id': '8bf302db-8a80-47d3-b5dc-bc07512a3928', 'position': 3, 'revision_id': 'e4e2cb2d-4bd5-414e-b646-e484f174d9ab', 'id': '8bf302db-8a80-47d3-b5dc-bc07512a3928'}, referer: http://ckan.net/package/edit/cofog </pre> http://localhost/ticket/311 http://localhost/ticket/311 Report dread <p> On ckan.net there are a few packages which cause a 403 exception when you try to rate them: <a class="ext-link" href="http://www.ckan.net/package/rate/coins-data?rating=3"><span class="icon">​</span>http://www.ckan.net/package/rate/coins-data?rating=3</a> </p> <p> The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() <a class="missing wiki">InvalidRequestError?</a>: Multiple rows returned for one() </p> <p> It looks like this package got rated twice in quick succession, creating two similar <a class="missing wiki">PackageRating?</a> objects. This race condition needs to be solved and these particular rating objects repaired. </p> http://localhost/ticket/312 http://localhost/ticket/312 Report dread <p> If a package name changes, a simple CKAN client may not be aware of this (not monitoring the push notifications, revisions or feed), so it is preferable to refer to the package by its (invariant) ID. </p> <p> It is still useful to refer to a package by its name though, so both should be valid arguments in the REST interface. </p> http://localhost/ticket/313 http://localhost/ticket/313 Report johnbywater <p> Bug report regarding getting revisions: </p> <p> Getting revisons by ID (on the latest ID) GET "<a class="ext-link" href="http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff"><span class="icon">​</span>http://test-hmg.ckan.net/api/search/revision?since_revision=44aac9b6-ba24-43a8-87a1-f6923dc523ff</a>" </p> <p> Returns a whole load of stuff (it's also quite slow - about 10 seconds) </p> <p> I'm expecting it to return just an empty array - am I doing something wrong here - if so could you clarify correct use of the API? </p> <p> GET "<a class="ext-link" href="http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45"><span class="icon">​</span>http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-30T23:45</a>" Returns the empty string - I'd expect an empty array ie [] </p> <p> GET "<a class="ext-link" href="http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45"><span class="icon">​</span>http://test-hmg.ckan.net/api/search/revision?since_time=2010-04-31T23:45</a>" Returns an internal server error 500 - I think it should probably be "bad Request" 400 (the date is invalid) </p> http://localhost/ticket/314 http://localhost/ticket/314 Report rgrp <ol><li>Issues with quote in fields: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2010-May/000240.html</a> </li></ol><ol start="2"><li>Issues with package resource serialization into csv table. </li></ol><p> &lt;quote&gt; In the latest dump there were 116(!) sets of the three columns (“resource-[n]-url”, “resource-[n]-format”, and “resource-[n]-description”). However, these are an extract of the packed “resource” column and I’m not sure whether they’re needed. Also, they irritatingly don’t appear in order in the CSV serialisation. If the resource columns could be ordered in the file that would be great; if a second version without the unpacked resource data would be excellent. &lt;/quote&gt; </p> http://localhost/ticket/315 http://localhost/ticket/315 Report dread <p> If you search for unescaped characters such as '`' (backtick) in the URL in Chrome then you get a 500 error. </p> <p> e.g. <a class="ext-link" href="http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk"><span class="icon">​</span>http://www.ckan.net/package/search?q=fjdkf2B%C2%B4gfhgfkgf{gpk</a> fjdkf2B´gfhgfkgf{gpk </p> <p> returns this exception: </p> <pre class="wiki">URL: http://www.ckan.net/package/search?q=fjdkf%2B%C2%B4gfhgfkgf%7Bg%C2%B4pk&amp;search=Search+Packages+%C2%BB Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module repoze.who.middleware:107 in __call__ &lt;&lt; wrapper = StartResponseWrapper(start_response) app_iter = app(environ, wrapper.wrap_start_response) # The challenge decider almost(?) always needs information from the &gt;&gt; app_iter = app(environ, wrapper.wrap_start_response) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:50 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckan.controllers.package:52 in search &lt;&lt; collection=query, page=request.params.get('page', 1), items_per_page=50 ) # filter out ranks from the query result &gt;&gt; items_per_page=50 Module webhelpers.paginate:333 in __init__ &lt;&lt; self.item_count = item_count else: self.item_count = len(self.collection) # Compute the number of the first and last available page &gt;&gt; self.item_count = len(self.collection) Module webhelpers.paginate:204 in __len__ &lt;&lt; def __len__(self): return self.obj.count() # Since the items on a page are mainly a list we subclass the "list" type &gt;&gt; return self.obj.count() Module sqlalchemy.orm.query:1094 in count &lt;&lt; q = q.params(params) q = q._legacy_select_kwargs(**kwargs) return q._count() def _count(self): &gt;&gt; return q._count() Module sqlalchemy.orm.query:1103 in _count &lt;&lt; """ return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key)) def _col_aggregate(self, col, func, nested_cols=None): &gt;&gt; return self._col_aggregate(sql.literal_column('1'), sql.func.count, nested_cols=list(self.mapper.primary_key)) Module sqlalchemy.orm.query:1125 in _col_aggregate &lt;&lt; if self._autoflush and not self._populate_existing: self.session._autoflush() return self.session.scalar(s, params=self._params, mapper=self.mapper) def compile(self): &gt;&gt; return self.session.scalar(s, params=self._params, mapper=self.mapper) Module sqlalchemy.orm.session:635 in scalar &lt;&lt; engine = self.get_bind(mapper, clause=clause, instance=instance) return self.__connection(engine, close_with_result=True).scalar(clause, params or {}) def close(self): &gt;&gt; return self.__connection(engine, close_with_result=True).scalar(clause, params or {}) Module sqlalchemy.engine.base:834 in scalar &lt;&lt; """ return self.execute(object, *multiparams, **params).scalar() def statement_compiler(self, statement, **kwargs): &gt;&gt; return self.execute(object, *multiparams, **params).scalar() Module sqlalchemy.engine.base:844 in execute &lt;&lt; for c in type(object).__mro__: if c in Connection.executors: return Connection.executors[c](self, object, multiparams, params) else: raise exceptions.InvalidRequestError("Unexecutable object type: " + str(type(object))) &gt;&gt; return Connection.executors[c](self, object, multiparams, params) Module sqlalchemy.engine.base:895 in execute_clauseelement &lt;&lt; else: keys = None return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) &gt; 1), distilled_params=params) def _execute_compiled(self, compiled, multiparams=None, params=None, distilled_params=None): &gt;&gt; return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) &gt; 1), distilled_params=params) Module sqlalchemy.engine.base:907 in _execute_compiled &lt;&lt; context.pre_execution() self.__execute_raw(context) context.post_execution() self._autocommit(context) &gt;&gt; self.__execute_raw(context) Module sqlalchemy.engine.base:916 in __execute_raw &lt;&lt; self._cursor_executemany(context.cursor, context.statement, context.parameters, context=context) else: self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) def _execute_ddl(self, ddl, params, multiparams): &gt;&gt; self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) Module sqlalchemy.engine.base:958 in _cursor_execute &lt;&lt; self.engine.logger.info(repr(parameters)) try: self.dialect.do_execute(cursor, statement, parameters, context=context) except Exception, e: self._handle_dbapi_exception(e, statement, parameters, cursor) &gt;&gt; self.dialect.do_execute(cursor, statement, parameters, context=context) Module sqlalchemy.engine.default:133 in do_execute &lt;&lt; def do_execute(self, cursor, statement, parameters, context=None): cursor.execute(statement, parameters) def is_disconnect(self, e): &gt;&gt; cursor.execute(statement, parameters) UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 6: ordinal not in range(128) </pre> http://localhost/ticket/316 http://localhost/ticket/316 Report rgrp <p> Make lib/search.py pluggable so that we can plug in different search systems (e.g. SOLR). </p> <p> Suggest we define a base Search class from which specific search implementations inherit (e.g. SQLSearch, SOLRSearch etc). The specific one being used would then be set via a config variable. </p> http://localhost/ticket/317 http://localhost/ticket/317 Report wwaites <p> The CKAN instance on data.gov.uk serves invalid URIs out of its API. </p> <p> For example the following can be found, </p> <p> <a class="ext-link" href="http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&amp;ns_type=pdf&amp;ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls"><span class="icon">​</span>http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&amp;ns_type=pdf&amp;ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls</a>] </p> <p> In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396 </p> <p> Also URIs are not stripped of whitespace at the end. </p> <p> This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs. </p> <p> "Be liberal in what you accept and conservative in what you send" would seem apt. </p> <h2 id="Actions">Actions</h2> <ul><li>Validation of urls as part of form entry or data loading <ul><li>Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal </li></ul></li><li>This code should support analysis of existing data so we can go through existing database and find invalid urls <ul><li>Also useful to have this so we can do out of band validation </li></ul></li></ul> http://localhost/ticket/318 http://localhost/ticket/318 Report dread <p> Migration script 18 converts license "OKD Compliant::Local Authority Copyright with data.gov.uk rights" into "uklocalauthority-withrights" when it should be "localauth-withrights", according to the license service SoS 2. This causes clients difficulties looking it up. </p> http://localhost/ticket/319 http://localhost/ticket/319 Report rgrp <p> As a sysadmin I want to configure basic site title information for use in the site templates. </p> <p> Implementation: </p> <ul><li>ckan.site_title config variable </li><li>set this on g in app_globals.py e.g. <ul><li>from pylons import config; g.site_title = config.get('ckan.site_title, 'CKAN - Comprehensive Knowledge Archive Network') </li></ul></li><li>use in head title and in main site title/logo section (use it as alt on logo image) </li><li>Also all other pages (e.g. index, about) which talk about CKAN <ul><li>Is this needed? Would it not be better for people who want to customize the site to simply overwrite those templates? </li></ul></li></ul><p> Questions: </p> <ul><li>Do we want a site_logo variable whic his use for site title/logo section instead of site_title if site_logo defined? </li><li>Probably yes, but <strong>not</strong> part of this ticket. </li></ul> http://localhost/ticket/320 http://localhost/ticket/320 Report johnbywater <p> When CKAN is included in a Drupal front-end, CKAN edit pages are used in a slave-mode, such that authentication is delegated to the Drupal front-end user model. </p> <p> The Drupal front-end shall have: </p> <ol><li>Login page - fixed location, can authenticate users, on successful authentication sets auth cookie and redirects to HTTP_REFERER. </li></ol><ol start="2"><li>Access control resource - fixed location, can authorise users, on receipt of valid auth cookie return message listing account details and permitted actions. </li></ol><ol start="3"><li>Access denied page - fixed location, static resource, gently </li></ol><p> indicates what has happened, and how to ask for permission. </p> <p> The CKAN slave edit page shall: </p> <ol><li>Try to detect a Drupal session key (passed as cookie or as request param). </li></ol><ol start="2"><li>Redirect to Drupal login page if no session key. </li></ol><ol start="3"><li>Check authorisation if session key is found. </li></ol><ol start="4"><li>Redirect to access denied page if session key not authorised. </li></ol><ol start="5"><li>Present the Package edit page. </li></ol><ol start="6"><li>Reject unauthenticated or unauthorised edit submissions. </li></ol><ol start="7"><li>Snag invalid edit submissions from authenticated and authorised users. </li></ol><ol start="8"><li>Respond to valid edit submissions from authenticated and authorised users, by saving the new package state, and redirecting to Package read page in Drupal front-end. </li></ol> http://localhost/ticket/321 http://localhost/ticket/321 Report dread <h3 id="Usecases">Use cases</h3> <ul><li>Register for package changes </li><li>Register for all revisions </li><li>Notified of a package change </li><li>Notified of a revision </li><li>Deregistration </li><li>Configuration of port in pylons config </li></ul><h3 id="Design">Design</h3> <ul><li>Default port: 5672 (standard for AMQP) </li><li>Exchange name: 'ckan' </li><li>Exchange type: topic exchange (most flexible) </li><li>Routing keys: (see below) </li></ul><h3 id="Routingdetail">Routing detail</h3> <p> Routing key format: "OBJ_TYPE" (NB tags should be identified by their name, not ID) </p> <p> Example routing keys </p> <ul><li>'package' - Package edited/created </li><li>'resource' - Resource edited/created </li><li>'revision' - Any change </li><li>'db.clean' </li><li>'db.rebuild' </li></ul><p> Example queue bindings that clients may use: </p> <ul><li>* - no filtering - client receives all notifications </li><li>package - only changes to packages </li><li>revision - all revisions </li><li>db - all database operations </li></ul><h3 id="Versioning">Versioning</h3> <p> Since message payloads will be tied into the REST Entities, it makes sense to join up with the REST versioning. This could be achieved by providing new exchanges called 'ckan-1.1' perhaps? </p> <h3 id="Documentation">Documentation</h3> <ul><li>How to use </li><li>simple example of an external client? </li></ul> http://localhost/ticket/322 http://localhost/ticket/322 Report dread <h3 id="Whicheventstonotifyon">Which events to notify on</h3> <p> Listed by domain object, these are the notification message 'change types' that will be sent: </p> <ul><li>Package </li><li>PackageResource </li></ul><p> Also it is clear that it could be useful to know when db-wide maintenance is carried out: </p> <ul><li>db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration) </li></ul><h3 id="Ignoreddomainobjects">Ignored domain objects</h3> <p> These parts of the domain model will not carry notifications as no use case has been identified for them: </p> <ul><li>Revision </li><li>Group </li><li>Tag </li><li>Rating </li><li>User - list of users is sensitive info </li><li>Relationships - complicated </li><li>Authz - complicated and sensitive info </li><li>License - change of a license's metadata is a question for the 'license service' </li></ul><h3 id="Messageformat">Message format</h3> <p> A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity. </p> <p> For the 'db' notifications there shall be no payload. </p> http://localhost/ticket/323 http://localhost/ticket/323 Report dread <p> Currently search indexing is triggered directly using a Postgresql db callback. Now take advantage of the Notification system to register interest in all package changes and db changes to trigger this instead. </p> <p> The indexing shall run in a separate shell/process, managed by supervisord. </p> http://localhost/ticket/324 http://localhost/ticket/324 Report dread <h3 id="Asa">As a</h3> <p> CKAN client program </p> <h3 id="Iwantto">I want to</h3> <p> be notified when changes to the CKAN metadata occur. </p> <h3 id="Examplesofuse">Examples of use</h3> <ol><li>An external search engine needing to (re)index a package. (interest: Package) </li><li>A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: <a class="closed ticket" href="http://localhost/ticket/352" title="enhancement: Package notification worker - sends XML-RPC (closed: wontfix)">ticket:352</a> and previous iteration here: <a class="closed ticket" href="http://localhost/ticket/333" title="enhancement: CKAN front end requirements for package notifications (closed: wontfix)">ticket:333</a>. </li><li>A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource) </li><li>Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource) </li></ol><h3 id="Context">Context</h3> <p> The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling. </p> <h3 id="Design">Design</h3> <p> Split-off into two tickets: </p> <ul><li>Notification message - <a class="closed ticket" href="http://localhost/ticket/323" title="enhancement: Notification message (closed: fixed)">ticket:323</a> <ul><li>Which events to notify on </li><li>Message format </li></ul></li><li>Interface for Notifier Service - <a class="closed ticket" href="http://localhost/ticket/322" title="enhancement: Client interface for Notification Service (closed: fixed)">ticket:322</a> </li></ul><h3 id="Testing">Testing</h3> <p> To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine. </p> <h3 id="Related">Related</h3> <ul><li>Run CKAN search indexing using this system - <a class="closed ticket" href="http://localhost/ticket/324" title="enhancement: Search indexing using notifications (closed: fixed)">ticket:324</a> </li><li>Run SOLR indexing using this system - <a class="closed ticket" href="http://localhost/ticket/353" title="defect: SOLR search indexing (closed: fixed)">ticket:353</a> </li><li>Web hooks for notifications - <a class="closed ticket" href="http://localhost/ticket/327" title="defect: Create a web hook worker for CKAN (closed: wontfix)">ticket:327</a> </li></ul> http://localhost/ticket/325 http://localhost/ticket/325 Report dread <p> Later versions of python use json which is better than simplejson, but it must be kept as an option for compatibility. So centralise the import of json to ckan.lib.helpers. </p> http://localhost/ticket/326 http://localhost/ticket/326 Report pudo <blockquote> <p> This will be useful as an example consumer of the new queue notifications (<a class="closed ticket" href="http://localhost/ticket/325" title="enhancement: Event push notification (closed: fixed)">#325</a>). </p> </blockquote> <ul><li>A user can register any URL to be notified upon an event. </li><li>Possible extensions: GET with ID only vs. POST with serialized object </li><li>E-Mail notifications are also hooks, essentially. </li></ul><p> We need to have a UI module that allows for the hook CRUD and some feedback (e.g. non 200 status codes) </p> http://localhost/ticket/327 http://localhost/ticket/327 Report rgrp <p> Should prominently (at the top?) mention existing code libraries for working with ckan api. Have: </p> <ul><li>Python: ckanclient </li><li>Perl: luke closs wrote something </li><li>PHP: drupal library? </li><li>...? </li></ul> http://localhost/ticket/328 http://localhost/ticket/328 Report dread <h2 id="Reproduction">Reproduction</h2> <p> Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message. </p> <p> Affects all versions of CKAN. </p> <h2 id="Whyitshappening">Why it's happening</h2> <p> The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught. </p> http://localhost/ticket/329 http://localhost/ticket/329 Report dread <p> The python time module is supposed to recognise timezones such as 'UTC', 'GMT' and 'BST' using the %Z parameter. This works fine on British installs, but the buildbot (for example) gives this error: </p> <p> <a class="missing wiki">DateConvertError?</a>: Could not read date as ISO format "%a, %d %b %Y %H:%M:%S %Z". Date provided: "Mon, 04 Jan 2010 09:30:00 BST" </p> <p> It turns out that it only recognises local names of timezones. </p> <p> The ONS import doesn't care much about timezone, so we should just ignore it, avoiding these problems. </p> http://localhost/ticket/330 http://localhost/ticket/330 Report rgrp <p> Revisions are timestamped using the server's clock, which may not relate to the expected timezone for the site. e.g. the Norway site has a server on GMT. No timezone info is displayed either. </p> <p> Would like to set timezone for a CKAN instance to use in rendering revision timestamps. For example, use CET or EST timezone. </p> http://localhost/ticket/331 http://localhost/ticket/331 Report rgrp <p> During tag autocompletion, clicking on a suggested tag doesn't trigger completion in the text box. </p> http://localhost/ticket/332 http://localhost/ticket/332 Report dread <h2 id="Usecase:newpackage">Use case: new package</h2> <ol><li>An external front-end system provides a web page with a list of packages. Each package has the option to edit it or and there is also a button to create a new package. </li></ol><ol start="2"><li>User: clicks 'new package'. </li></ol><ol start="3"><li>CKAN presents the package/new form to the user. </li></ol><ol start="4"><li>(After a couple of previews) User: clicks 'commit'. </li></ol><ol start="5"><li>Notification message goes from CKAN to the front-end detailing the new package. </li></ol><ol start="6"><li>The user is redirected back to the front-end web page displaying the list of packages, which contains the new one. </li></ol><p> The notification message (step 5) has to get through to the front-end that the new package is created before the redirect (step 6). This suggests that the message sending needs to be *synchronous*, i.e. acknowledged by the front-end, before CKAN redirects the user to the front-end package listing page (step 6). </p> <p> In addition, this use case suggests the front-end listens for package notifications, to save another call to CKAN to get the package details, before the displaying the list of packages. If this isn't possible (see next use case) and it must listen for revision notifications instead, then perhaps it is worth including the full package details in the payload for the revision notification message. Would there be a problem with such a large message in the next use case, with 100 packages? </p> <h2 id="Usecase:CKANimportspackages">Use case: CKAN imports packages</h2> <ol><li>CKAN administrator runs a script that adds 100 new packages into CKAN. </li></ol><ol start="2"><li>CKAN sends notification message to front-end to report the new packages/revisions. </li></ol><ol start="3"><li>Knowing there are new revisions, the front-end queries the CKAN revision interface to get the list of new packages. </li></ol><ol start="4"><li>The front-end queries CKAN for each new package one-by-one. </li></ol><ol start="5"><li>A new user request to the front-end will include the info about the new packages. </li></ol><p> The package addition could be achieved in 1 revision, 100 revisions or some compromise: </p> <ul><li>If it is 1 revision then potentially there are problems displaying the long list of packages in the 'recent changes'. </li></ul><ul><li>If it is 100 revisions, then the notification webhook would be called 100 times, which creates unnecessary load on the front-end. Suppose each Webhook call-back (step 2) triggers the front-end to make a call to CKAN to get the latest revisions (step3), in this case it would make 100 calls, most of them fruitless, causing unnecessary load on CKAN. </li></ul><p> This use case suggests a bulk import of packages should go into one revision, and therefore generate one revision notification message and 100 package notification messages. The front-end client should listen to only revision messages. </p> http://localhost/ticket/333 http://localhost/ticket/333 Report dread <p> This is a problem with the package page e.g. <a class="ext-link" href="http://ckan.net/package/open-election-data-project"><span class="icon">​</span>http://ckan.net/package/open-election-data-project</a> . The feed icon links to <a class="missing changeset" title="No default repository defined">[1]</a> and the text next to it "Subscribe" links to <a class="missing changeset" title="No default repository defined">[2]</a>. Surely these should be the same? The second link seems to right one to me. </p> <ul><li><a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://ckan.net/package/open-election-data-project?format=atom"><span class="icon">​</span>http://ckan.net/package/open-election-data-project?format=atom</a> </li><li><a class="missing changeset" title="No default repository defined">[2]</a> <a class="ext-link" href="http://ckan.net/package/history/open-election-data-project?format=atom&amp;days=7"><span class="icon">​</span>http://ckan.net/package/history/open-election-data-project?format=atom&amp;days=7</a> </li></ul> http://localhost/ticket/334 http://localhost/ticket/334 Report dread <h2 id="Asa">As a</h2> <p> third-party interface to a CKAN instance </p> <h2 id="Iwantto">I want to</h2> <p> link to CKAN's package creation/editing pages. On 'commit', have the user redirected back to a URL in my interface that I can control. Also, when the package is created new, I need to be told what the new package's name is on return. </p> <h2 id="Design">Design</h2> <ol><li>The 'return URL' is passed as a parameter to CKAN. </li><li>CKAN substitutes the package name into the return URL. </li></ol><p> </p> <h2 id="Example">Example</h2> <p> Front-end links to: http://ca.ckan.net/package/new?return_to=http://datadotgc.ca/dataset/&lt;NAME&gt; (but with the parameter URL-encoded) </p> <p> When finished editing and the user commits, CKAN redirects the user to: http://datadotgc.ca/dataset/pollution_data </p> http://localhost/ticket/335 http://localhost/ticket/335 Report donovanhide <h2 id="Asa">As a</h2> <p> CKAN client such as ScraperWiki </p> <h2 id="Iwantto">I want to</h2> <p> search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL. </p> <h2 id="Proposedimplementation">Proposed implementation</h2> <p> Add resource search API at: </p> <p> /api/search/resource </p> <p> AND resource added to model API at: </p> <p> api/rest/resource </p> <blockquote> <p> (see <a class="closed ticket" href="http://localhost/ticket/358" title="enhancement: Resources in REST API (closed: duplicate)">ticket:358</a>) </p> </blockquote> <p> Functional differences from the ScraperWiki suggested patch: </p> <ul><li>URL is not normalised </li></ul><p> </p> <ul><li>URLs are not grouped </li></ul><ul><li>All fields of the resource object are returned, not just the URL </li></ul><ul><li>Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend '<a class="ext-link" href="http://ckan.net/package/"><span class="icon">​</span>http://ckan.net/package/</a>' to the package ID) </li></ul><p> This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok? </p> <h3 id="Examplesearch">Example search</h3> <p> POST </p> <pre class="wiki">{"url": "scraperwiki.com/", "all_fields": 1} </pre><p> to: /api/2/search/resource </p> <p> returns JSON: </p> <pre class="wiki"> [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028", "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe", "URL": "http://scraperwiki.com/lincolnshire-councillors", "format": "CSV", "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.", "hash": "", "position": 2 }] </pre><p> Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done <a class="closed ticket" href="http://localhost/ticket/341" title="enhancement: Web UI accepts package IDs in URLs (closed: fixed)">ticket:341</a> then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe </p> <h3 id="SearchParameters">Search Parameters</h3> <pre class="wiki">Key: q Description: Search all resource fields for the value Key: url / description / format / Description: Search particular field for the value Key: all_fields Value: 0 or 1 (0 is default) Description: If 1 (true), the full record of the package resource (and it's package reference) are returned, rather than just the PackageResource ID. </pre><p> May also choose to introduce 'offset' and 'limit' to page through a large number of results. </p> <p> JSONP achieved through API-wide parameter - see <a class="closed ticket" href="http://localhost/ticket/342" title="enhancement: JSONP parameter in API (closed: fixed)">ticket:342</a> </p> <p> Search is case insensitive. </p> <h2 id="Originalrequest">Original request</h2> <p> Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of <a class="closed ticket" href="http://localhost/ticket/388" title="task: Reply to &#34;two projects&#34; question from RS (closed: fixed)">#388</a>. </p> <p> Could also add a search version. Not sure what the best url would be for that though. </p> <p> Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere? </p> <p> Donovan </p> http://localhost/ticket/336 http://localhost/ticket/336 Report rgrp <p> target = _blank </p> <p> Cost: 30m </p> http://localhost/ticket/337 http://localhost/ticket/337 Report johnbywater http://localhost/ticket/338 http://localhost/ticket/338 Report johnbywater http://localhost/ticket/339 http://localhost/ticket/339 Report dread <p> An install of CKAN should be configurable without changing any of the installed files. This makes it clear to upgrade CKAN. Complete the changes in this wiki page to allow static files to be served from outside CKAN paths over CKAN versions and additional CSS file to be pulled in. </p> <p> <a class="ext-link" href="http://wiki.okfn.org/ckan/doc/theme"><span class="icon">​</span>http://wiki.okfn.org/ckan/doc/theme</a> </p> http://localhost/ticket/340 http://localhost/ticket/340 Report dread <p> (in the same way that we can refer to packages in the API by ID as well as name.) </p> <h2 id="Asa">As a</h2> <p> client of CKAN </p> <h2 id="Iwantto">I want to</h2> <p> link to a package page in the UI, referring to it by ID </p> http://localhost/ticket/341 http://localhost/ticket/341 Report dread <h2 id="Asa">As a</h2> <p> CKAN client using JQuery </p> <h2 id="Iwantto">I want to</h2> <p> call the CKAN API and instead of receiving back JSON I get JSONP. i.e. "%s(%s)" % (callback, json_content) </p> <h2 id="Suggestedimplementation">Suggested implementation</h2> <p> All API calls allow the JSONP 'callback' parameter to be specified in the request and this wraps the JSON response. See suggested patch to rest.py by Donovan Hide: </p> <p> <a class="ext-link" href="http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch"><span class="icon">​</span>http://knowledgeforge.net/ckan/trac/attachment/ticket/336/resource.patch</a> </p> <h2 id="Test">Test</h2> <p> import re import unittest </p> <p> def test_jsonp_callback(): </p> <blockquote> <p> response = self.app.get('/api/search/resource/?url=<a class="ext-link" href="http://www.scraperwiki.com&amp;callback=jsoncallback"><span class="icon">​</span>http://www.scraperwiki.com&amp;callback=jsoncallback</a>') match = re.match('jsoncallback\(.*\);',response) self.assertTrue(match) </p> </blockquote> <blockquote> <p> response = self.app.get('/api/search/resource/?url=<a class="ext-link" href="http://www.scraperwiki.com"><span class="icon">​</span>http://www.scraperwiki.com</a>') match = re.match('jsoncallback\(.*\);',response) self.assertFalse(match) </p> </blockquote> <p> I think the point needs to be made that JSONP only works for GET requests and not POST/PUT/DELETE, so there needs to be a check for that in the _finish_ok method. </p> <p> (thanks to Donovan Hide for test) </p> http://localhost/ticket/342 http://localhost/ticket/342 Report dread <p> When you do a Package Search or query a Package Relationship in the API version 2, the responses have relationships which refer to packages by name, not by ID. </p> http://localhost/ticket/343 http://localhost/ticket/343 Report dread <p> PUT to /rest/api/package of <tt> {"name": "name", "resources": ["someurl.com/data"] } </tt> (i.e. resource is a string, not a dictionary) gives 500 error, when it should give a 400 error and helpful error message. </p> <p> Exception: </p> <pre class="wiki">Module ckan.controllers.rest:154 in create &lt;&lt; if register == 'package' and not subregister: fs = ckan.forms.get_standard_fieldset() request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data) fs = fs.bind(model.Package, data=request_fa_dict, session=model.Session) elif register == 'package' and subregister in model.PackageRelationship.get_all_types(): &gt;&gt; request_fa_dict = ckan.forms.edit_package_dict(ckan.forms.get_package_dict(fs=fs), request_data) Module ckan.forms.package_dict:88 in edit_package_dict &lt;&lt; for res_dict in value: res_dict_str = {} for key, value in res_dict.items(): res_dict_str[str(key)] = value resources.append(res_dict_str) &gt;&gt; for key, value in res_dict.items(): AttributeError: 'unicode' object has no attribute 'items' </pre> http://localhost/ticket/344 http://localhost/ticket/344 Report johnbywater http://localhost/ticket/345 http://localhost/ticket/345 Report johnbywater <p> Whilst going through the API docs for the revision search API, it was noticed that the "Gdu" SoS doc doesn't match up. It returns revision IDs (perhaps this is useful to note in the spec?) so the format is probably not 'limitedstring'. Also, they appear to be ordered youngest first, not oldest as stated. </p> <p> And in the revision model, it refers to 'simplestring' which it doesn't define - I guess the names should be 'limitedstring'? </p> <p> Could this be checked out? </p> http://localhost/ticket/346 http://localhost/ticket/346 Report johnbywater http://localhost/ticket/347 http://localhost/ticket/347 Report johnbywater http://localhost/ticket/348 http://localhost/ticket/348 Report johnbywater http://localhost/ticket/349 http://localhost/ticket/349 Report dread <p> Need to research what can easily be done to improve CKAN packages in the search rankings. </p> <p> Comments from Glen Barnes: </p> <p> We've been pretty successful at SEO without even really trying (see <a class="ext-link" href="http://www.google.co.nz/search?client=safari&amp;rls=en&amp;q=auckland+google+transit+feed&amp;ie=UTF-8&amp;oe=UTF-8&amp;redir_esc=&amp;ei=dsYSTOzJLs2eceuZiI8I"><span class="icon">​</span>http://www.google.co.nz/search?client=safari&amp;rls=en&amp;q=auckland+google+transit+feed&amp;ie=UTF-8&amp;oe=UTF-8&amp;redir_esc=&amp;ei=dsYSTOzJLs2eceuZiI8I</a> as an example). This to me is key. If we are to make data available it has to be findable which is the main reason for a catalogue. There are probably things we should be doing on CKAN like using slugged urls (<a class="ext-link" href="http://www.ckan.net/package/ascoe"><span class="icon">​</span>http://www.ckan.net/package/ascoe</a> -&gt; <a class="ext-link" href="http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment"><span class="icon">​</span>http://www.ckan.net/package/ascoe/atmospheric-chemistry-studies-in-the-oceanic-environment</a>), setting the H1 tag correctly ("Atmospheric Chemistry Studies in the Oceanic Environment" on the example above). Some basic SEO 101 on page optimisations. </p> http://localhost/ticket/350 http://localhost/ticket/350 Report dread <p> Have a simpler list of exciting data, as opposed to the big revision list. </p> <p> For example: </p> <pre class="wiki">Hot data =========== New packages: package1, package2, package3 Updated resources: package1, package2, package3 Popular packages: </pre> http://localhost/ticket/351 http://localhost/ticket/351 Report dread <h2 id="Asan">As an</h2> <p> external front-end </p> <h2 id="Iwantto">I want to</h2> <p> be notified (by XML-RPC) about package creations and updates. </p> <h2 id="Implementation">Implementation</h2> <ol><li>A message queue worker waits for package update notifications </li><li>On reception, it constructs XML detailing the changes and PUSHes it to a configured URI. </li></ol><p> Rather than turning the package fields into XML fields, the JSON dump of the list of package dictionaries will become a single XML parameter. </p> <p> Config - in the CKAN config will be: </p> <ul><li>URI to callback to </li><li>API version to use (version 2 gives packages referred by ID not name) </li></ul> http://localhost/ticket/352 http://localhost/ticket/352 Report dread <h2 id="Asa">As a</h2> <p> SOLR instance </p> <h2 id="Iwantto">I want to</h2> <p> keep my search index of CKAN packages up-to-date </p> <h2 id="Implementation">Implementation</h2> <ul><li>Using asynchronous event notifications </li><li>Running in a separate process to CKAN </li></ul> http://localhost/ticket/353 http://localhost/ticket/353 Report johnbywater <p> Collect together requirements and top-level design for user/package 'groups': existing tickets, Rufus spec, Sean spec, meeting notes (dread) email, based on existing user authz stuff. </p> <p> <a class="ext-link" href="http://knowledgeforge.net/ckan/trac/wiki/AccessControl"><span class="icon">​</span>http://knowledgeforge.net/ckan/trac/wiki/AccessControl</a> </p> <p> Do we add these into user-role table somehow or new table? To present this to team </p> http://localhost/ticket/354 http://localhost/ticket/354 Report rgrp <p> Sort out how we deal with dashes versus underscores in package names. </p> http://localhost/ticket/355 http://localhost/ticket/355 Report rgrp <p> A small but useful ui improvement would be to have a search box at top right on every page. </p> <p> As an example see the one here on trac or on github.com or bitbucket.org. </p> <ul><li>It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration). </li></ul> http://localhost/ticket/356 http://localhost/ticket/356 Report johnbywater <p> Sean Burlington wrote: </p> <blockquote class="citation"> <p> I recently came across a documentation issue </p> <p> <a class="ext-link" href="http://knowledgeforge.net/ckan/doc/ckan/api.html"><span class="icon">​</span>http://knowledgeforge.net/ckan/doc/ckan/api.html</a> </p> <hr /> <p> The key should be passed in the API request header: Header Example value HTTP_AUTHORIZATION fde34a3c-b716-4c39-8dc4-881ba115c6d4 </p> <hr /> <p> But the header name is actually just 'AUTHORIZATION' </p> <p> For example to set it in PHP </p> <p> curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION: xxxxx-xxxx-xxxx-xxxx-xxxxxxx')); </p> </blockquote> http://localhost/ticket/357 http://localhost/ticket/357 Report dread <p> (spun out of <a class="closed ticket" href="http://localhost/ticket/336" title="defect: Resource Search API (closed: fixed)">ticket:336</a>) </p> <p> Resource added to model API at: </p> <p> api/rest/resource </p> <h3 id="Examplemodelrequest">Example model request</h3> <p> GET to: /api/2/rest/resource/a3dd8f64-9078-4f04-845c-e3f047125028 </p> <p> returns: </p> <pre class="wiki"> [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028", "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe", "URL": "http://scraperwiki.com/lincolnshire-councillors", "format": "CSV", "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.", "hash": "", "position": 2 }] </pre><h2 id="Authorization">Authorization</h2> <ol><li>Have it generic (ie. not per resource) and use an action/role on system </li><li>Require all resources to attach to packages an inherit their permissions (i.e. read/write etc if and only read/write on associated packages) </li><li>Introduce Resource in authorization system (requires migration) </li></ol><p> Mixed model </p> <pre class="wiki">Create / Edit: if resource associated to package: check_permissions(package, update) else: check_system_permissions(c.user, model.Action.Resouce Create/Update, model.System) </pre> http://localhost/ticket/358 http://localhost/ticket/358 Report rgrp <p> Storage represents something you can upload to (download is done in <a class="closed ticket" href="http://localhost/ticket/406" title="requirement: Download sub-system (closed: fixed)">ticket:406</a>). </p> <ul><li>storage sub package with defined interface (maybe just use OFS) </li><li>Only support uploading single files for the moment (though can do multiple files). <ul><li>Relationship to build (<a class="closed ticket" href="http://localhost/ticket/306" title="enhancement: datapkg build command (closed: duplicate)">ticket:306</a>) </li><li>No support for syncing e.g. hg/git repos -- that's up to you as a user </li></ul></li><li>Upload command </li></ul><p> At end of this we can deprecate Repository. </p> <p> Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication. </p> http://localhost/ticket/359 http://localhost/ticket/359 Report rgrp <p> In 0.5 and 0.6 have been more oriented to getting file indexes working to support the convenience of using packages on disk directly. However we now need to go back to having a better central/default index and the best model for this is a simple db index using sqlite (which is now supported in the standard library). </p> <p> NB: this is a retrospective ticket (been working on this for a couple of months) </p> http://localhost/ticket/360 http://localhost/ticket/360 Report rgrp http://localhost/ticket/361 http://localhost/ticket/361 Report dread <h2 id="Background">Background</h2> <p> In the Web UI, when you rate a package it simply links to something like: </p> <p> <a class="ext-link" href="http://ckan.net/package/rate/mke-liquor-licenses?rating=3"><span class="icon">​</span>http://ckan.net/package/rate/mke-liquor-licenses?rating=3</a> </p> <p> This creates a GET request. </p> <p> This is bad because: </p> <ul><li>Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this) </li><li>There are occasions when we want to make a CKAN instance read-only, so we put a &lt;<a class="missing wiki">LimitExcept?</a> GET&gt; Apache instruction in. But the database may still get written for these ratings. </li><li>Best practise for web requests is for GET to be a read-only request. </li></ul> http://localhost/ticket/362 http://localhost/ticket/362 Report dread <p> Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they? </p> <p> They appear on the 'Recently changed' list on the home page with an empty 'Packages' column. </p> http://localhost/ticket/363 http://localhost/ticket/363 Report dread <p> On ckan.net there are plenty of packages (and indeed their tags) with the word 'statistic' in them, but no packages turn up when you search for it: </p> <p> <a class="ext-link" href="http://ckan.net/package/search?q=statistic&amp;search=Search+Packages+%C2%BB"><span class="icon">​</span>http://ckan.net/package/search?q=statistic&amp;search=Search+Packages+%C2%BB</a> </p> <p> (Using Postgres full text search) </p> http://localhost/ticket/364 http://localhost/ticket/364 Report dread <p> If you change a resource then you not only get a <a class="missing wiki">PackageNotification?</a>, but also a <a class="missing wiki">ResourceNofication?</a>. </p> http://localhost/ticket/365 http://localhost/ticket/365 Report dread <p> You get redirected for a number of different reasons - often confuses me let alone the average user! </p> <p> Cost: 1h </p> http://localhost/ticket/366 http://localhost/ticket/366 Report dread <p> Runs on the commandline and prints out notifications. </p> http://localhost/ticket/367 http://localhost/ticket/367 Report anonymous <p> I've just started writing the importer from cap.open.org.nz and was running an initial import of just the package name and titles. The packages were created however I received a 500 error in response: </p> <p> "publish: New Zealand Coastline (new-zealand-coastline)" opening connection to nz.ckan.net... opened &lt;- "POST /api/rest/package HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nAuthorization: 81179ade-fa4a-4632-9b89-3d0c98bfc8b8\r\nContent-Length: 64\r\nHost: nz.ckan.net\r\n\r\n" &lt;- "{\"name\":\"new-zealand-coastline\",\"title\":\"New Zealand Coastline\"}" -&gt; "HTTP/1.1 500 Internal Server Error\r\n" -&gt; "Date: Tue, 20 Jul 2010 10:20:35 GMT\r\n" -&gt; "Server: Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2\r\n" -&gt; "Vary: Accept-Encoding\r\n" -&gt; "Content-Type: text/html; charset=iso-8859-1\r\n" -&gt; "Cache-Control: proxy-revalidate\r\n" -&gt; "Content-Length: 640\r\n" -&gt; "Connection: close\r\n" -&gt; "\r\n" reading 640 bytes... -&gt; "&lt;!DOCTYPE HTML PUBLIC \"-<em>IETF</em>DTD HTML 2.0<em>EN\"&gt;\n&lt;html&gt;&lt;head&gt;\n&lt;title&gt;500 Internal Server Error&lt;/title&gt;\n&lt;/head&gt;&lt;body&gt;\n&lt;h1&gt;Internal Server Error&lt;/h1&gt;\n&lt;p&gt;The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.&lt;/p&gt;\n&lt;p&gt;Please contact the server administrator,\n [no address given] and inform them of the time the error occurred,\nand anything you might have done that may have\ncaused the error.&lt;/p&gt;\n&lt;p&gt;More information about this error may be available\nin the server error log.&lt;/p&gt;\n&lt;hr&gt;\n&lt;address&gt;Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at nz.ckan.net Port 80&lt;/address&gt;\n&lt;/body&gt;&lt;/html&gt;\n" read 640 bytes Conn close </em></p> <p> The server should not respond with a 500 error if the package was created. </p> http://localhost/ticket/368 http://localhost/ticket/368 Report shudson@… <p> Currently there's a nice legend titled "Package Listing Key" that appears in right side of "Browse Packages" results. The same key should show on other search results like when searching for a tag. </p> http://localhost/ticket/369 http://localhost/ticket/369 Report shudson@… <p> The <a class="missing wiki">JavaScript?</a> email encryption used is not very reassuring. Google's <a class="missing wiki">MailHide?</a> is a much better solution that is easily implemented. </p> <p> <a class="ext-link" href="http://www.google.com/recaptcha/mailhide/"><span class="icon">​</span>http://www.google.com/recaptcha/mailhide/</a> </p> <p> Check on the Mailhide API where there are even some Python libraries already built. </p> http://localhost/ticket/370 http://localhost/ticket/370 Report johnbywater <p> Requested by DGU. </p> http://localhost/ticket/371 http://localhost/ticket/371 Report johnbywater <p> Set limits in /etc/security/limits.conf so that we can always ssh in at least. Requested by DGU. </p> http://localhost/ticket/372 http://localhost/ticket/372 Report johnbywater <p> Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally. </p> <p> Requested by DGU. </p> http://localhost/ticket/373 http://localhost/ticket/373 Report johnbywater http://localhost/ticket/374 http://localhost/ticket/374 Report johnbywater <p> Requested by DGU. </p> http://localhost/ticket/375 http://localhost/ticket/375 Report johnbywater <p> Requested by DGU. </p> http://localhost/ticket/376 http://localhost/ticket/376 Report glen@… <p> When the API returns any response it should always retunr JSON. For example a 404 response should return something like </p> <pre class="wiki">{'status': 404, 'message': 'package somename not found'} </pre><p> When errors are encountered when creating a package: </p> <pre class="wiki">{'status': nnn, 'message': 'Validation Errors', 'errors':['description':'name cannot be blank', 'description':'license code must be an integer']} </pre><p> Not authorised: </p> <pre class="wiki">{'status': 503, 'message': 'You are not authorised to create this package.'} </pre><p> The reason being that screeds of html coming back in the response just makes it harder to debug and understand. Having the proper error codes/messages means that you can see why your package was not created. </p> http://localhost/ticket/377 http://localhost/ticket/377 Report johnbywater http://localhost/ticket/378 http://localhost/ticket/378 Report johnbywater http://localhost/ticket/379 http://localhost/ticket/379 Report johnbywater http://localhost/ticket/380 http://localhost/ticket/380 Report johnbywater <p> please ignore </p> http://localhost/ticket/381 http://localhost/ticket/381 Report johnbywater <p> As a service administrator, I want to measure responsiveness, throughput, and availability of a CKAN service. </p> http://localhost/ticket/382 http://localhost/ticket/382 Report johnbywater http://localhost/ticket/383 http://localhost/ticket/383 Report johnbywater http://localhost/ticket/384 http://localhost/ticket/384 Report johnbywater http://localhost/ticket/385 http://localhost/ticket/385 Report johnbywater http://localhost/ticket/386 http://localhost/ticket/386 Report johnbywater http://localhost/ticket/387 http://localhost/ticket/387 Report johnbywater http://localhost/ticket/388 http://localhost/ticket/388 Report johnbywater http://localhost/ticket/389 http://localhost/ticket/389 Report johnbywater http://localhost/ticket/390 http://localhost/ticket/390 Report johnbywater http://localhost/ticket/391 http://localhost/ticket/391 Report johnbywater http://localhost/ticket/392 http://localhost/ticket/392 Report johnbywater http://localhost/ticket/393 http://localhost/ticket/393 Report johnbywater http://localhost/ticket/394 http://localhost/ticket/394 Report pudo <p> At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls. </p> <p> This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance. </p> <p> NB: We should also agree on a maximum request latency. </p> <p> TODO: Read up on all those QoS tickets to avoid overlapping efforts. </p> http://localhost/ticket/395 http://localhost/ticket/395 Report johnbywater http://localhost/ticket/396 http://localhost/ticket/396 Report johnbywater http://localhost/ticket/397 http://localhost/ticket/397 Report johnbywater http://localhost/ticket/398 http://localhost/ticket/398 Report johnbywater http://localhost/ticket/399 http://localhost/ticket/399 Report johnbywater <p> Have TSO configure data.ov.uk/api/catalogue. Have TSO register catalogue.data.gov.uk. </p> http://localhost/ticket/400 http://localhost/ticket/400 Report pudo <p> We should create and document a basic queue consumer and task processor for CKAN. It will connect to the CKAN update queue, read messages and dispatch them to a consumer function (such as a downloader or an analysis app) </p> http://localhost/ticket/401 http://localhost/ticket/401 Report pudo <p> Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system. </p> <ul><li>Naming scheme? <ul><li>Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? <ul><li>What happens if names change </li></ul></li><li>File: filename? hash? </li></ul></li><li>Store hash back on ckan instance? </li></ul><p> The caching worker will consumer update notifications and fetch packages. </p> <p> Extra points for: </p> <ul><li>Properly checking for source file modification (Last-modified, Etag) </li><li>Using PIP VCS Backends for retrieval </li><li>OFS/S3 Storage </li></ul> http://localhost/ticket/402 http://localhost/ticket/402 Report pudo <p> Create a thin web service that will allow users to upload files to OFS on S3 without knowing S3 credentials. This could also provide static urls for all stored resources. </p> <ul><li>relates to <a class="closed ticket" href="http://localhost/ticket/307" title="defect: datapkg upload command (closed: fixed)">#307</a> </li></ul> http://localhost/ticket/403 http://localhost/ticket/403 Report pudo <p> Allow using different indexes, including ckan, file, dcat, ... </p> http://localhost/ticket/404 http://localhost/ticket/404 Report rgrp <p> Download Command (was install command) should support the following modes: </p> <ul><li>Download only the first listed resource (current behaviour, slightly arbitrary) </li><li>Download resources by interactive selection </li><li>Download resources by MIME type (cf <a class="assigned ticket" href="http://localhost/ticket/235" title="enhancement: Resource format normalization and detection (assigned)">#235</a>) </li><li>Download all resources. </li></ul> http://localhost/ticket/405 http://localhost/ticket/405 Report rgrp <p> Factor out current downloader into dedicated download subpackage, abstract interface and make it pluggable so we can have downloaders loaded as per type of resource to download. </p> <ul><li>Also rename <a class="missing wiki">InstallCommand?</a> -&gt; <a class="missing wiki">DownloadCommand?</a> </li></ul><p> And we should support interactive retrieving of resources ... </p> <h2 id="ImplementationIdeas">Implementation Ideas</h2> <pre class="wiki">for resources in pkg.resources: # datapkg/hg load_storage_handler(resource.format) </pre> http://localhost/ticket/406 http://localhost/ticket/406 Report dread <h2 id="Asa">As a</h2> <p> user browsing a CKAN package which has a resource in a bad format </p> <h2 id="Iwantto">I want to</h2> <p> be alerted to the possibility of scraping it on scraperwiki </p> <h2 id="Implementation">Implementation</h2> <p> This should be done in a CKAN extension. It should appear in a side-bar on the package read page. </p> <p> If the package does not have a resource with scraperwiki.com in the url, then it displays this message: </p> <p> Title: Scrape it<br /> Image: (Scraperwiki dumper truck)<br /> Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can <a class="ext-link" href="http://scraperwiki.com/market/request/"><span class="icon">​</span>request</a> this data is scraped or even <a class="ext-link" href="http://scraperwiki.com/editor/"><span class="icon">​</span>write the scraper</a> yourself. </p> http://localhost/ticket/407 http://localhost/ticket/407 Report pudo <ul><li>As sysadmin/user of CKAN I want to integrate a CMS to support easy addition of additional content ... </li><li>I want to integrate with navigation </li><li>As a minimum I want to be able to add content pages (more stuff would be e.g. comments, sidebars, main menu ...) </li></ul><p> Why: CMS are great at creating content and have very good editors and theme support ... </p> <h3 id="Previouswork">Previous work</h3> <p> <a class="ext-link" href="http://groups.google.com/group/pylons-discuss/browse_thread/thread/f8969e28b0a007c7"><span class="icon">​</span>http://groups.google.com/group/pylons-discuss/browse_thread/thread/f8969e28b0a007c7</a> </p> <ul><li>deliverance: retheming on the fly </li></ul><p> <a class="ext-link" href="http://knowledgeforge.net/shakespeare/hg/file/677eef9402e6/shakespeare/controllers/template.py"><span class="icon">​</span>http://knowledgeforge.net/shakespeare/hg/file/677eef9402e6/shakespeare/controllers/template.py</a> </p> <ol><li>Rewrite urls for proxying </li><li>You need a theme to substitute into </li><li>pylons </li></ol><p> Extreme version: </p> <ul><li>2 sources -&gt; pylons/ckan app, cms </li><li>1 theme source: ckan or cms </li></ul><p> 1 frontend/proxy app which re-themes everything </p> <p> Owned by CKAN: </p> <ul><li>package/ </li><li>tag/ </li><li>group/ </li><li>api/ </li><li>/ </li></ul><p> Everything else proxied. </p> http://localhost/ticket/408 http://localhost/ticket/408 Report rgrp <ul><li><del>(Load cli commands from setuptools entry point name datapkg.cli)</del> </li><li>Document how other can write new commands </li><li>Refactor base Command class to support this pluggability (if necessary) </li></ul> http://localhost/ticket/409 http://localhost/ticket/409 Report pudo <p> This would be an extension to the disqus plugin that would patch itself into home/index. </p> http://localhost/ticket/410 http://localhost/ticket/410 Report pudo <ul><li>Need to figure out cache invalidation for CKAN </li><li>Evaluate against an internal cache (i.e. memcached) </li></ul> http://localhost/ticket/411 http://localhost/ticket/411 Report johnbywater <p> Requirements: </p> <ul><li>cater naturally for multiple packages with multiple resources </li><li>international encoding </li><li>large number of packages </li></ul> http://localhost/ticket/412 http://localhost/ticket/412 Report johnbywater http://localhost/ticket/413 http://localhost/ticket/413 Report dread <p> Ask Paul for a new machine for testing. Then one for varnish-live and one for varnish-test. </p> http://localhost/ticket/414 http://localhost/ticket/414 Report dread http://localhost/ticket/415 http://localhost/ticket/415 Report dread http://localhost/ticket/416 http://localhost/ticket/416 Report dread http://localhost/ticket/418 http://localhost/ticket/418 Report johnbywater http://localhost/ticket/419 http://localhost/ticket/419 Report dread http://localhost/ticket/420 http://localhost/ticket/420 Report dread http://localhost/ticket/421 http://localhost/ticket/421 Report dread <p> Requirements: </p> <ul><li>cater naturally for multiple packages with multiple resources </li><li>international encoding </li><li>large number of packages </li></ul> http://localhost/ticket/422 http://localhost/ticket/422 Report johnbywater http://localhost/ticket/423 http://localhost/ticket/423 Report dread http://localhost/ticket/424 http://localhost/ticket/424 Report johnbywater http://localhost/ticket/425 http://localhost/ticket/425 Report dread http://localhost/ticket/426 http://localhost/ticket/426 Report dread <ul><li>examples of posting a new package </li><li>example license_id - explain </li></ul> http://localhost/ticket/427 http://localhost/ticket/427 Report dread <p> End users want to get hold of a JSON &amp; CSV dump of the DGU data. </p> http://localhost/ticket/428 http://localhost/ticket/428 Report dread http://localhost/ticket/429 http://localhost/ticket/429 Report pudo <p> Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism. </p> <p> <a class="ext-link" href="http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py"><span class="icon">​</span>http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py</a> </p> <p> This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed. </p> http://localhost/ticket/430 http://localhost/ticket/430 Report pudo <p> There seem to be a few conditions under which either queue processing or the indexer fail in their current state. To get a more systematic picture of these failures, we should have a small testing tool to compare the index to the database of a live CKAN instance. </p> http://localhost/ticket/431 http://localhost/ticket/431 Report dread <p> This occurs when CKAN is run with mod_wsgi (not under paster). </p> <p> This is because of a unicode header being creating in rest.py. </p> http://localhost/ticket/432 http://localhost/ticket/432 Report wwaites <p> Still not sure if we shouldn't use the existing setuptools machinery to manage this -- there is already a way to get at the metadata. In any event, I've just made an addition to datapkg that makes it possible to put datapkg_sources as a dictionary in your package's setup.py. Afterwards it is possible to pull the metadata out of the egg. Of course this could easily be changed to save the information in whatever form, indeed if you pass it a string instead of a dictionary it will just write whatever you gave it into the datapkg_sources.spec. The point is, I think that the egg is a good place to stuff this information. </p> <p> For non-python users, it is always possible to simply put up the datapkg_sources.spec somewhere on the web so they can directly retrieve the data files. </p> <p> From the docstring:: </p> <pre class="wiki"> This is the implementation for an [egg_info.writers] entrypoint. Datapkg adds an argument to setuptools's setup() function called datapkg_sources. The argument should be a dictionary of the form: .. code-block:: python setup( ..., datapkg_sources = { "cra2009" : "http://www.hm-treasury.gov.uk/d/cra_2009_db.csv" } ) The result of this is that there will be a file in the egg called datapkg_sources.spec that looks like this:: [sources] cra2009=http://www.hm-treasury.gov.uk/d/cra_2009_db.csv How do you get at this data? Simple:: .. code-block:: python import pkg_resources dist = pkg_resources.get_distribution("ukgov_treasury_cra") spec = dist.get_metadata("datapkg_sources.spec") and 'spec' will be the contents of the file as a string. </pre> http://localhost/ticket/433 http://localhost/ticket/433 Report dread <p> Here's the trace: </p> <pre class="wiki">[Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Error - &lt;type 'exceptions.IndexError'&gt;: list index out of range [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] mod_wsgi (pid=6195): Exception occurred processing WSGI script '/home/okfn/var/srvc/si.ckan.net/py env/bin/si.ckan.net.py'. [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] Traceback (most recent call last): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/cascade.py", line 130, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.apps[-1](environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/paste/registry.py", line 375, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] app_iter = self.application(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/repoze/who/middleware.py ", line 107, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] app_iter = app(environ, wrapper.wrap_start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/middleware.py", l ine 214, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] self.app, new_environ, catch_exc_info=True) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/util.py", line 94 , in call_wsgi_application [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] app_iter = application(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/weberror/errormiddleware .py", line 156, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.application(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py", l ine 73, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.app(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/beaker/middleware.py", l ine 152, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return self.wrap_app(environ, session_start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/routes/middleware.py", l ine 130, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self.app(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py", line 125, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self.dispatch(controller, environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/wsgiapp.py", line 324, in dispatch [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return controller(environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py", line 73, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return WSGIController.__call__(self, environ, start_response) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py", line 221, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self._dispatch_call() [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py", line 172, in _dispatch_call [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] response = self._inspect_call(func) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py", line 107, in _inspect_call [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] result = self._perform_call(func, args) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/pylons/controllers/core. py", line 60, in _perform_call [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return func(**args) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/controllers/error.py", line 32, in doc ument [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] ckan_template = render('error_document_template.html') [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py", line 38, in render [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return render_template() [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/src/ckan/ckan/lib/base.py", line 36, in render_templ ate [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return literal(stream.render(method=method, encoding=None)) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py", line 18 3, in render [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return encode(generator, method=method, encoding=encoding, out=out) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 57, in encode [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] return _encode(''.join(list(iterator))) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 339, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 670, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 771, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in chain(stream, [(None, None, None)]): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/output.py", line 586, in __call__ [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for ev in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/core.py", line 288, in _ensure [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 618, in _include [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in tmpl.generate(ctxt, **vars): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 618, in _include [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in tmpl.generate(ctxt, **vars): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 605, in _include [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/markup.py", line 327, in _match [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/template/base.py", line 545, in _flatten [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for kind, data, pos in stream: [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py", line 177, in _generate [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] for event in msgbuf.translate(gettext(msgbuf.format())): [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py", line 1029, in translate [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] parts = parse_msg(string) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] File "/home/okfn/var/srvc/si.ckan.net/pyenv/lib/python2.5/site-packages/genshi/filters/i18n.py", line 1143, in parse_msg [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] parts.append((stack[-1], string)) [Wed Aug 18 11:10:19 2010] [error] [client 89.212.104.197] IndexError: list index out of range </pre><p> ckan: 1e403927f2bb (metastable) genshi: 0.6 lang: slovenian </p> http://localhost/ticket/434 http://localhost/ticket/434 Report dread <p> Analysis using pg_top reveals orphaed processes: </p> <p> <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/460#comment:6"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/460#comment:6</a> </p> <p> This followed switching ckan instances on the dgu machine. Do these hang around forever clogging up a db server? Can switching between instances be smoothed to avoid this? </p> http://localhost/ticket/435 http://localhost/ticket/435 Report dread <p> Here's the dump from 22:10 last night: </p> <p> URL: <a class="ext-link" href="http://ckan.net/api/search/resource?all_fields=1&amp;offset=0&amp;limit=20&amp;qjson=%3Cspan%20class="><span class="icon">​</span>http://ckan.net/api/search/resource?all_fields=1&amp;offset=0&amp;limit=20&amp;qjson=%3Cspan%20class=</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:152 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:130 in <span class="underline">call</span> &lt;&lt; environ<a class="missing wiki">SCRIPT_NAME?</a> = environ<a class="missing wiki">SCRIPT_NAME?</a>[:-1] </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:125 in <span class="underline">call</span> &lt;&lt; </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> if 'paste.testing_variables' in environ and hasattr(response, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.dispatch(controller, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: </p> <blockquote> <p> log.debug("Calling controller class with WSGI interface") </p> </blockquote> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> def load_test_env(self, environ): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:73 in <span class="underline">call</span> &lt;&lt; # available in environ<a class="missing wiki">pylons.routes_dict?</a> </p> <blockquote> <p> try: </p> <blockquote> <p> return WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> <p> finally: </p> <blockquote> <p> model.Session.remove() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:221 in <span class="underline">call</span> &lt;&lt; return response(environ, self.start_response) </p> <blockquote> <p> response = self._dispatch_call() if not start_response_called: </p> <blockquote> <p> self.start_response = start_response </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._dispatch_call() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ<a class="missing wiki">pylons.action_method?</a> = func </p> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> <blockquote> <p> else: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.<span class="underline">name</span>, args) </p> <blockquote> <p> try: </p> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> <p> except HTTPException, httpe: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = 'before_and_this' return func(<strong>args) </strong></p> </blockquote> <p> </p> <blockquote> <p> def _inspect_call(self, func): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return func(<strong>args) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.rest:400 in search &lt;&lt; response.status_int = 400 </p> <blockquote> <p> return gettext('Blank qjson parameter') </p> </blockquote> <blockquote> <p> params = json.loads(request.params<a class="missing wiki">qjson?</a>) </p> </blockquote> <blockquote> <p> elif request.params.values() and request.params.values() != [u<em>] and request.params.values() != [u'1']: </em></p> <blockquote> <p> params = request.params </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> params = json.loads(request.params<a class="missing wiki">qjson?</a>) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module simplejson:384 in loads &lt;&lt; parse_constant is None and object_pairs_hook is None </p> <blockquote> <p> and not use_decimal and not kw): </p> </blockquote> <blockquote> <p> return _default_decoder.decode(s) </p> </blockquote> <blockquote> <p> if cls is None: </p> <blockquote> <p> cls = JSONDecoder </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return _default_decoder.decode(s) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module simplejson.decoder:402 in decode &lt;&lt; """ </p> <blockquote> <p> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) end = _w(s, end).end() if end != len(s): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module simplejson.decoder:420 in raw_decode &lt;&lt; obj, end = self.scan_once(s, idx) </p> <blockquote> <p> except <a class="missing wiki">StopIteration?</a>: </p> <blockquote> <p> raise JSONDecodeError("No JSON object could be decoded", s, idx) </p> </blockquote> <p> return obj, end </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> raise JSONDecodeError("No JSON object could be decoded", s, idx) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/json' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_ACCEPT_LANGUAGE 'en-us,en;q=0.5' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE '<span class="underline">utma=27730403.1245320310.1281386803.1281386803.1282164955.2; </span>utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; <span class="underline">utmb=27730403.3.10.1282164955; </span>utmc=27730403' HTTP_HOST 'ckan.net' HTTP_KEEP_ALIVE '300' HTTP_REFERER '<a class="ext-link" href="http://jira/browse/PLATFORM-892"><span class="icon">​</span>http://jira/browse/PLATFORM-892</a>' HTTP_USER_AGENT 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/search/resource' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/search/resource' QUERY_STRING 'all_fields=1&amp;offset=0&amp;limit=20&amp;qjson=%3Cspan%20class=' REMOTE_ADDR '64.235.97.218' REMOTE_PORT '20720' REQUEST_METHOD 'GET' REQUEST_URI '/api/search/resource?all_fields=1&amp;offset=0&amp;limit=20&amp;qjson=%3Cspan%20class=' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI '<a class="ext-link" href="http://ckan.net/api/search/resource"><span class="icon">​</span>http://ckan.net/api/search/resource</a>' SCRIPT_URL '/api/search/resource' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0xa1c13ec&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0xa1c142c&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0xa1c12ac&gt;&gt; beaker.session {'_accessed_time': 1282165818.0880959, '_creation_time': 1282165818.0880959} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host <em> mod_wsgi.listener_port '80' mod_wsgi.process_group </em> mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (&lt;SimpleCookie: __utma='27730403.1245320310.1281386803.1281386803.1282164955.2' __utmb='27730403.3.10.1282164955' __utmc='27730403' __utmz='27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892'&gt;, '<span class="underline">utma=27730403.1245320310.1281386803.1281386803.1282164955.2; </span>utmz=27730403.1282164955.2.2.utmcsr=jira|utmccn=(referral)|utmcmd=referral|utmcct=/browse/PLATFORM-892; <span class="underline">utmb=27730403.3.10.1282164955; </span>utmc=27730403') paste.parsed_querystring ([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', '&lt;span class=')], 'all_fields=1&amp;offset=0&amp;limit=20&amp;qjson=%3Cspan%20class=') paste.registry &lt;paste.registry.Registry object at 0x130ed84c&gt; paste.throw_errors True pylons.action_method &lt;bound method <a class="missing wiki">RestController?</a>.search of &lt;ckan.controllers.rest.<a class="missing wiki">RestController?</a> object at 0xe9bbe0c&gt;&gt; pylons.controller &lt;ckan.controllers.rest.<a class="missing wiki">RestController?</a> object at 0xe9bbe0c&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0xe9bbe8c&gt; pylons.routes_dict {'action': u'search', 'controller': u'rest', 'register': u'resource'} repoze.who.logger &lt;logging.Logger instance at 0xa3cb0cc&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 170067148&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 171739788&gt;} routes.route &lt;routes.route.Route object at 0xa102fac&gt; routes.url &lt;routes.util.URLGenerator object at 0x13a5a3cc&gt; webob._parsed_query_vars (GET([('all_fields', '1'), ('offset', '0'), ('limit', '20'), ('qjson', '&lt;span class=')]), 'all_fields=1&amp;offset=0&amp;limit=20&amp;qjson=%3Cspan%20class=') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x12f53530&gt; wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x13a5a3cc&gt;, {'action': u'search', 'controller': u'rest', 'register': u'resource'}) </p> http://localhost/ticket/436 http://localhost/ticket/436 Report dread <p> On today's buildbot: <a class="ext-link" href="http://buildbot.okfn.org/builders/buildbot-test/builds/201"><span class="icon">​</span>http://buildbot.okfn.org/builders/buildbot-test/builds/201</a> </p> <p> 2 failures about ascii (ignore other 2) </p> http://localhost/ticket/437 http://localhost/ticket/437 Report dread <p> e.g. ONS importer </p> http://localhost/ticket/438 http://localhost/ticket/438 Report dread <p> Using use cases for security etc. Draw diagram to satisfy use cases. Communicate to Paul and therefore TSO. </p> http://localhost/ticket/439 http://localhost/ticket/439 Report dread <p> Run latest ckan on eu0. Automate some queries and searches. Check load and database connections / processes. </p> http://localhost/ticket/440 http://localhost/ticket/440 Report dread <p> When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages. </p> <p> Examples of use: </p> <ul><li>Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. </li><li>A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.: <ul><li>another (but writable) CKAN instance is connected to the same db </li><li>restoring database dumps from another CKAN db </li></ul></li><li>Should a security be breached, all editing could be stopped </li></ul> http://localhost/ticket/441 http://localhost/ticket/441 Report dread <p> Review what chunks of code we have and how to move forward with requirements. </p> <p> Requirements: </p> <ul><li>Bulk parsing is done away from the CKAN server </li></ul><p> Possible future requirements: </p> <ul><li>/importer for importing spreadsheets in WUI </li><li>validation-light - 'scoring' a package </li></ul><p> Current importers: </p> <ul><li>/importer controller uses: <a class="missing wiki">PackageImporter?</a>, form, <a class="missing wiki">PackageSaver?</a> </li><li>loader uses: package_importer -&gt; API </li><li>API uses: form, <a class="missing wiki">PackageSaver?</a> </li></ul><p> Modules </p> <ul><li>package_importer: <ul><li>xls/csv --<a class="missing wiki">SpreadsheetData?</a>--&gt; table </li><li>table --<a class="missing wiki">DataRecords?</a>--&gt; records </li><li>records --<a class="missing wiki">PackageImporter?</a>--&gt; pkg_dicts </li></ul></li></ul><ul><li>pkg_dict --form--&gt; fs &amp; validation </li><li>fs --<a class="missing wiki">PackageSaver?</a>--&gt; package </li></ul> http://localhost/ticket/442 http://localhost/ticket/442 Report dread <ul><li>Producing daily dump file </li><li>Publishing at ckan.net/dumps/ or better </li></ul> http://localhost/ticket/443 http://localhost/ticket/443 Report dread <ul><li>Create test data on visible ckan </li><li>Discuss with JF </li></ul> http://localhost/ticket/444 http://localhost/ticket/444 Report johnbywater <p> Thinking about putting something like this: </p> <p> Dumps Interface (this is NOT going in the API, it is another interface of the service): </p> <ul><li>resources: </li></ul><blockquote> <blockquote> <p> Daily Dump Folder Daily Dump File </p> </blockquote> </blockquote> <ul><li>locations: </li></ul><blockquote> <blockquote> <p> /dump/hmg.ckan.net-YYYY-MM-DD.[json|csv].zip </p> </blockquote> </blockquote> <ul><li>data format: </li></ul><blockquote> <blockquote> <p> Dump Folder is a HTML document containing a table of links to Dump Files Dump File has either JSON or CSV format. </p> </blockquote> </blockquote> <ul><li>status codes (200, 404, 500) </li></ul> http://localhost/ticket/445 http://localhost/ticket/445 Report dread <ul><li>Ensure use case is in design - "an admin writes to the CKAN API" (and see JB notes on paper) </li><li>Communicate how this is planned to be achieved in design to JF </li></ul> http://localhost/ticket/446 http://localhost/ticket/446 Report johnbywater <p> Follow up with NS and RP. </p> http://localhost/ticket/447 http://localhost/ticket/447 Report dread http://localhost/ticket/448 http://localhost/ticket/448 Report johnbywater http://localhost/ticket/449 http://localhost/ticket/449 Report dread <p> John did this a while ago. </p> http://localhost/ticket/450 http://localhost/ticket/450 Report dread <p> Use drupal api </p> http://localhost/ticket/451 http://localhost/ticket/451 Report dread http://localhost/ticket/452 http://localhost/ticket/452 Report johnbywater http://localhost/ticket/453 http://localhost/ticket/453 Report dread http://localhost/ticket/454 http://localhost/ticket/454 Report johnbywater <p> Either CKAN writes to SOLR and Drupal reads from SOLR, or CKAN writes to SOLR and Drupal reads SOLR via CKAN API (so search resource locations are unaffected). </p> http://localhost/ticket/455 http://localhost/ticket/455 Report dread http://localhost/ticket/456 http://localhost/ticket/456 Report dread http://localhost/ticket/457 http://localhost/ticket/457 Report johnbywater http://localhost/ticket/458 http://localhost/ticket/458 Report johnbywater http://localhost/ticket/459 http://localhost/ticket/459 Report johnbywater <p> This package: </p> <p> <a class="ext-link" href="http://ckan.net/package/dbtune-audioscrobbler"><span class="icon">​</span>http://ckan.net/package/dbtune-audioscrobbler</a> </p> <p> was: </p> <ol><li>created by Richard (logged-in) </li><li>edited by Richard (logged-in) </em></li></ol><p> (According to the logs, at this point the state was changed from 'active' to 'deleted') -- RP was it set to 'deleted' or just <em>? </p> <ol start="3"><li>pudo changed the state back to active </li></ol><p> Similarly an incident with bibbase package where field set to <em> (see <a class="ext-link" href="http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&amp;oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409"><span class="icon">​</span>http://ckan.net/revision/diff/bibbase?diff=702bb0a3-03b7-49ac-87ad-e489c414962f&amp;oldid=5447842d-b6ed-41d9-9cfd-8bb73b85c409</a>) </em></p> <p> Need to investigate how this got changed, fix if necessary and report back to Richard. Note that package 'admins' as well as sysadmins can change the state of a package (though note that bibbase did not appear to have an owner). </p> <p> Suggested solution (for setting to <em>): </em></p> <ul><li>Ensure in ckan/forms.py that there is a validator for state field that ensures only set to valid values. </li><li>Check that we do not allow state to be changed in the api except by package owner or sysadmin </li></ul> http://localhost/ticket/460 http://localhost/ticket/460 Report johnbywater <ul><li>Move script out to ckandgu repo </li><li>Change script to convert xml into package dicts </li><li>Test (against test.ckan.net, hmg.test.ckan.net) </li><li>Deploy </li></ul> http://localhost/ticket/461 http://localhost/ticket/461 Report johnbywater http://localhost/ticket/462 http://localhost/ticket/462 Report dread <p> Investigate exception: </p> <pre class="wiki"> URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:73 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckan.controllers.revision:119 in diff &lt;&lt; c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() &gt;&gt; diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff &lt;&lt; display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] &gt;&gt; display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=0' HTTP_COOKIE '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173' HTTP_HOST 'ckan.net' HTTP_REFERER 'http://ckan.net/package/history/dbtune-audioscrobbler' HTTP_USER_AGENT 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '92.226.62.168' REMOTE_PORT '55102' REMOTE_USER 'http://pudo.org/' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application &lt;beaker.middleware.CacheMiddleware object at 0xbb528cc&gt; beaker.cache &lt;beaker.cache.CacheManager object at 0xbb528ec&gt; beaker.get_session &lt;bound method SessionMiddleware._get_session of &lt;beaker.middleware.SessionMiddleware object at 0xbb5286c&gt;&gt; beaker.session {'_accessed_time': 1282252047.603267, '_creation_time': 1282252047.603267} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (&lt;SimpleCookie: __utma='27730403.1381387877.1281962773.1282248244.1282250173.11' __utmb='27730403.6.10.1282250173' __utmc='27730403' __utmz='27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan' auth_tkt='e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!'&gt;, '__utmz=27730403.1281000628.1.6.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=ckan; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; auth_tkt="e919c7bb40f890a0184ed7be95465e1a4c6d130bhttp://pudo.org/!"; __utma=27730403.1381387877.1281962773.1282248244.1282250173.11; __utmc=27730403; __utmb=27730403.6.10.1282250173') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry &lt;paste.registry.Registry object at 0xf702aec&gt; paste.throw_errors True pylons.action_method &lt;bound method RevisionController.diff of &lt;ckan.controllers.revision.RevisionController object at 0xf6eaf0c&gt;&gt; pylons.controller &lt;ckan.controllers.revision.RevisionController object at 0xf6eaf0c&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.PylonsContext object at 0xf6eaa8c&gt; pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.identity &lt;repoze.who identity (hidden, dict-like) at 280815340&gt; repoze.who.logger &lt;logging.Logger instance at 0xbcdfccc&gt; repoze.who.plugins {'openid': &lt;OpenIdIdentificationPlugin 196612236&gt;, 'auth_tkt': &lt;AuthTktCookiePlugin 198048140&gt;} routes.route &lt;routes.route.Route object at 0xbbc44cc&gt; routes.url &lt;routes.util.URLGenerator object at 0xf5e55ec&gt; webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0xfe0b0b0&gt; wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0xf5e55ec&gt;, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) </pre> http://localhost/ticket/463 http://localhost/ticket/463 Report dread http://localhost/ticket/464 http://localhost/ticket/464 Report johnbywater http://localhost/ticket/465 http://localhost/ticket/465 Report johnbywater http://localhost/ticket/466 http://localhost/ticket/466 Report johnbywater http://localhost/ticket/467 http://localhost/ticket/467 Report johnbywater http://localhost/ticket/468 http://localhost/ticket/468 Report johnbywater http://localhost/ticket/469 http://localhost/ticket/469 Report johnbywater http://localhost/ticket/470 http://localhost/ticket/470 Report johnbywater http://localhost/ticket/471 http://localhost/ticket/471 Report johnbywater http://localhost/ticket/472 http://localhost/ticket/472 Report johnbywater http://localhost/ticket/473 http://localhost/ticket/473 Report dread <p> Several questions to answer (and see file in email). To respond to the Finnish team and copy all Q&amp;A to the ckan-discuss list. </p> <p> 1 ) I have played with two different datasets to get the metadata from those imported to CKAN. The other dataset is in PCAXIS format and other one is in Excel format. Please find the scripts + sample files attached in a zip file that contains instructions too to run the scripts. I looked at import scripts written in object oriented manner and getdata samples in CKAN but it was easier for me too write these from scratch (copied the unicrap function from CKAN samples). If I run the PCAXIS import script I get Unicode decode error (see errors file). When I use the unicrap function then the error does not appear in logs. I assume this is some Python environment setting issue. Of course I would like to run the script without unicrap function to get scandic letter imported correctly to ckan(and displayed in WUI). In Excel case there is no scandic letter issue which is kind of strange but this probably relates how the Excel library handles encoding. </p> <p> To be honest the whole unicode/encoding/decoding issue has been quite difficult to deal with and debug. I hope I would get proper environment configuration to handle scandic letters. </p> <p> 2 ) It seem in CKAN by default the localisation file contains 462 strings. The transifex link Rufus sent contains 400 strings. Is transifex the one we should use as localisation file ? Where are the rest 62 strings then fetched from or are they needed at all. I have found that as I have been using the .po file coming in CKAN source (which has API translations too) this also causes similar unicode problems when an error in API occurs (the API error messages containing scandic letters provide the similar <a class="missing wiki">UnicodeDecoding?</a> errors as mentioned above). So, it seems the unicode issues appear in various cases. Maybe a proper environment settings will be able to prevent also these problems... If transifex is the one to use I can prepare it and maybe then you could upload it to fi.ckan.net as there will be also other people checking the translations. To run the test and see the API translation errors especially in pcaxis case I would recommend to install attached FI translations to your environment and include lang = fi configuration. It looks like the API errors does not come when running default lang = en. </p> <p> 3 ) Is it possible to change the license selection/options available in license field drop-down ? Where are these located and can these be modified locally ? </p> <p> 4) It looks like the values stored in package extra fields are not searchable. Is this intentional ? It looks more or less that all other fields can be searched using free-text search in CKAN. </p> <p> 5) How package dependencies are defined and managed ? For instance we want to import some packages in a manner that the other package is a child package of another. Does API support this and/or is this possible to do in WUI ? </p> <p> Best Regards, Henri </p> http://localhost/ticket/474 http://localhost/ticket/474 Report johnbywater http://localhost/ticket/475 http://localhost/ticket/475 Report johnbywater http://localhost/ticket/476 http://localhost/ticket/476 Report johnbywater http://localhost/ticket/477 http://localhost/ticket/477 Report johnbywater <p> We don't want /api to be an HTML page, but rather provide information about the API in a JSON structure. </p> http://localhost/ticket/478 http://localhost/ticket/478 Report johnbywater http://localhost/ticket/479 http://localhost/ticket/479 Report johnbywater <p> Common requirements for running CKAN behind a (e.g Wordpress or Drupal) front-end: </p> <ol><li>Unrestricted total read-only access to catalogue API for general public (e.g. voluntary organisation). <ul><li>monitored by API key </li><li>not monitored by API key </li></ul></li><li>Restricted total read-write access to catalogue API for authorized clients (e.g. front-end system, bulk upload clients). <ul><li>restricted by CKAN access controller </li><li>restricted by HTTP Auth </li><li>restricted by IP address </li></ul></li><li>Restricted total read-write access to catalogue Web UI for authorized users (e.g. site admins). <ul><li>restricted by CKAN access controller </li><li>restricted by HTTP Auth </li></ul></li><li>Restricted partial read-write access to catalogue Web UI for authorized users (e.g. group admins). <ul><li>restricted by CKAN access controller </li><li>restricted by HTTP Auth </li></ul></li></ol><p> <a style="padding:0; border:none" href="http://localhost/attachment/ticket/480/fig128130.png"><img src="http://localhost/raw-attachment/ticket/480/fig128130.png" alt="CKAN as a catalogue service" title="CKAN as a catalogue service" /></a> </p> http://localhost/ticket/480 http://localhost/ticket/480 Report johnbywater http://localhost/ticket/481 http://localhost/ticket/481 Report johnbywater http://localhost/ticket/482 http://localhost/ticket/482 Report johnbywater http://localhost/ticket/483 http://localhost/ticket/483 Report johnbywater http://localhost/ticket/484 http://localhost/ticket/484 Report johnbywater http://localhost/ticket/485 http://localhost/ticket/485 Report johnbywater http://localhost/ticket/486 http://localhost/ticket/486 Report johnbywater http://localhost/ticket/487 http://localhost/ticket/487 Report johnbywater http://localhost/ticket/488 http://localhost/ticket/488 Report johnbywater http://localhost/ticket/489 http://localhost/ticket/489 Report johnbywater http://localhost/ticket/490 http://localhost/ticket/490 Report johnbywater http://localhost/ticket/491 http://localhost/ticket/491 Report johnbywater http://localhost/ticket/492 http://localhost/ticket/492 Report johnbywater http://localhost/ticket/493 http://localhost/ticket/493 Report johnbywater http://localhost/ticket/494 http://localhost/ticket/494 Report johnbywater http://localhost/ticket/495 http://localhost/ticket/495 Report johnbywater http://localhost/ticket/496 http://localhost/ticket/496 Report johnbywater http://localhost/ticket/497 http://localhost/ticket/497 Report johnbywater http://localhost/ticket/498 http://localhost/ticket/498 Report johnbywater http://localhost/ticket/499 http://localhost/ticket/499 Report dread <p> Investigate exception occured occasionally in last couple of days on ckan.net: </p> <pre class="wiki">WebApp Error: &lt;type 'exceptions.AttributeError'&gt;: 'NoneType' object has no attribute 'key' URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:73 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckan.controllers.revision:119 in diff &lt;&lt; c.revision_to = model.Session.query(model.Revision).get( request.params.getone('diff')) diff = pkg.diff(c.revision_to, c.revision_from) c.diff = diff.items() c.diff.sort() &gt;&gt; diff = pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:340 in diff &lt;&lt; display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] &gt;&gt; display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'zh-cn,zh-tw' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Baiduspider+(+http://www.baidu.com/search/spider.htm)' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' REMOTE_ADDR '123.125.66.32' REMOTE_PORT '63767' REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application &lt;beaker.middleware.CacheMiddleware object at 0x9f603ec&gt; beaker.cache &lt;beaker.cache.CacheManager object at 0x9f6042c&gt; beaker.get_session &lt;bound method SessionMiddleware._get_session of &lt;beaker.middleware.SessionMiddleware object at 0x9f602ac&gt;&gt; beaker.session {'_accessed_time': 1282385101.4243281, '_creation_time': 1282385101.4243281} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (&lt;SimpleCookie: &gt;, '') paste.parsed_querystring ([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')], 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') paste.registry &lt;paste.registry.Registry object at 0x104552ec&gt; paste.throw_errors True pylons.action_method &lt;bound method RevisionController.diff of &lt;ckan.controllers.revision.RevisionController object at 0xfb17aec&gt;&gt; pylons.controller &lt;ckan.controllers.revision.RevisionController object at 0xfb17aec&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.PylonsContext object at 0x10286d4c&gt; pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.logger &lt;logging.Logger instance at 0xa16e0cc&gt; repoze.who.plugins {'openid': &lt;OpenIdIdentificationPlugin 167584972&gt;, 'auth_tkt': &lt;AuthTktCookiePlugin 169253516&gt;} routes.route &lt;routes.route.Route object at 0x9f3690c&gt; routes.url &lt;routes.util.URLGenerator object at 0xfd8d7cc&gt; webob._parsed_query_vars (GET([('diff', '66a47b9e-232a-4838-8674-66fa1a5c76e1'), ('oldid', 'a99c98be-767a-4e49-9025-2472b2d18b9c')]), 'diff=66a47b9e-232a-4838-8674-66fa1a5c76e1&amp;oldid=a99c98be-767a-4e49-9025-2472b2d18b9c') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x103a5bf0&gt; wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0xfd8d7cc&gt;, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) </pre> http://localhost/ticket/500 http://localhost/ticket/500 Report pudo <p> CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache. </p> http://localhost/ticket/501 http://localhost/ticket/501 Report rgrp <p> Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed. </p> http://localhost/ticket/502 http://localhost/ticket/502 Report rgrp <ol><li>Find some relationships. </li><li>Add them via API </li><li>Notify DGU of completion and point to exemplars </li></ol> http://localhost/ticket/503 http://localhost/ticket/503 Report rgrp <p> No need for requirement - closing in favour of <a class="closed ticket" href="http://localhost/ticket/505" title="enhancement: DGU fieldset supports v3 of DGU metadata specification (closed: fixed)">#505</a> </p> http://localhost/ticket/504 http://localhost/ticket/504 Report rgrp http://localhost/ticket/505 http://localhost/ticket/505 Report rgrp http://localhost/ticket/506 http://localhost/ticket/506 Report johnbywater http://localhost/ticket/507 http://localhost/ticket/507 Report dread <ul><li>Create a branch named ultrastable that represents latest version of 1.0 <em>as found on dgu</em> <ul><li>Going forward ultrastable will act as stable - 1 (and either ultrastable or stable will be on the rock-solid production systems such as dgu) </li></ul></li><li>Do <strong>not</strong> deprecate the stable or metastable branch for the present </li><li>Ask for recommendations on mailing list about best practice for branching (e.g. versioned based versus a simple HEAD versus stable) and how this affects standard operations such as applying bugfixes. <ul><li>For useful discussion of how to do backporting and applying bugfix patches see: <a class="ext-link" href="http://www.python.org/dev/peps/pep-0374/#backport"><span class="icon">​</span>http://www.python.org/dev/peps/pep-0374/#backport</a> </li><li>NB: in general do <strong>not</strong> always have to apply in stable/ultrastable and merge upwards for bugfixes -- can use transplant extension as well. </li></ul></li></ul> http://localhost/ticket/508 http://localhost/ticket/508 Report dread http://localhost/ticket/509 http://localhost/ticket/509 Report dread http://localhost/ticket/510 http://localhost/ticket/510 Report dread http://localhost/ticket/511 http://localhost/ticket/511 Report dread http://localhost/ticket/512 http://localhost/ticket/512 Report dread <p> When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK. </p> http://localhost/ticket/513 http://localhost/ticket/513 Report dread <p> When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all. </p> <p> I've removed the info about the header in the docs in the meantime. </p> http://localhost/ticket/514 http://localhost/ticket/514 Report dread <p> When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all. </p> <p> I've removed the info about the header in the docs in the meantime. </p> http://localhost/ticket/515 http://localhost/ticket/515 Report dread <p> At: <a class="ext-link" href="http://ckan.net/revision"><span class="icon">​</span>http://ckan.net/revision</a> you can see: "Next &amp;raquo;" on pager This doesn't seem to be a problem with earlier (1.0.1) version. </p> http://localhost/ticket/516 http://localhost/ticket/516 Report pudo <p> cf. Tariqs mail </p> <p> Updated version: <a class="ext-link" href="https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&amp;hl=en#gid=0"><span class="icon">​</span>https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&amp;hl=en#gid=0</a> </p> <h2 id="OriginalVersion">Original Version</h2> <pre class="wiki">Publishing Entity: Publishing Entity Type: (Donor, Recipient, Community Data..) Donor Country Activity period: Verification status: enumeration of statuses (checked, not checked etc) Resource links: to the actual IATI record Number of activities: ... Date record updated: Date data updated: License: Need this field even if it may be a standard license So naively mashing these together, we get something like: Identifier Title Abstract Donor Country Publisher Publisher Type Verification Status Department Contact Contact e-mail Licence Resource format Resource URL Resource ID Activity period Number of activities Date record updated Date data updated </pre> http://localhost/ticket/517 http://localhost/ticket/517 Report pudo <ul><li>Create custom form </li></ul> http://localhost/ticket/518 http://localhost/ticket/518 Report pudo <ul><li>Use what is on iatiregistry.org at the moment. </li></ul> http://localhost/ticket/519 http://localhost/ticket/519 Report pudo <ul><li>e.g. find/browse by country and by publishing entity and by donor </li></ul> http://localhost/ticket/520 http://localhost/ticket/520 Report pudo http://localhost/ticket/521 http://localhost/ticket/521 Report pudo <ul><li>Publishing entity emails system editors </li><li>System editor creates publishing entity account and emails publishing entity with details </li><li>Priority: 4/5 </li></ul> http://localhost/ticket/522 http://localhost/ticket/522 Report pudo <p> Read/write APIs for users and publishing entities to access and register data </p> <ul><li>Users have read access </li><li>Publishing entities have both (though can only write to 'their' records) </li><li>Priority: 4 for write (read access is 3) </li></ul> http://localhost/ticket/523 http://localhost/ticket/523 Report pudo <p> Limit registering data to authenticated users either via API or manually. </p> <ul><li>Edits may only be made by authenticated entities </li><li>Priority: 5 </li></ul> http://localhost/ticket/524 http://localhost/ticket/524 Report pudo <p> </p> <ul><li>So some kind of confirmation or approval by someone more senior? </li><li>Alternative: just have an account request. </li><li>Priority: 3 </li></ul> http://localhost/ticket/525 http://localhost/ticket/525 Report pudo <p> Assumes 8 is done: </p> <ol start="8"><li>Tool for validation an IATI XML record (referenced by a URL) for compliance with the IATI specification </li></ol><ul><li>Outside of scope of registry </li><li>Multiple validators with different levels of rigour </li></ul> http://localhost/ticket/526 http://localhost/ticket/526 Report rgrp http://localhost/ticket/527 http://localhost/ticket/527 Report rgrp http://localhost/ticket/528 http://localhost/ticket/528 Report rgrp http://localhost/ticket/529 http://localhost/ticket/529 Report pudo <p> From lopad: </p> <pre class="wiki">Publishing Entity (often/always? equal to the logged in user) * Domain logic: only can add entries for "yourself" Donor Country Activity period: period to which this record relates Verification status: enumeration of statuses (checked, not checked etc) Resource links: to the actual IATI record Number of activities: ... Date record updated: Date data updated: License: Need this field even if it may be a standard license </pre> http://localhost/ticket/530 http://localhost/ticket/530 Report rgrp <ul><li>Parse data </li><li>Load via API </li><li>Develop test for load via API (can be local or remote -- with paste webfixture may be easier) </li></ul> http://localhost/ticket/531 http://localhost/ticket/531 Report johnbywater http://localhost/ticket/532 http://localhost/ticket/532 Report johnbywater <p> CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses. </p> <p> How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware? </p> <p> Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases? </p> <p> Are there any other options for scaling the persistence mechanism? </p> http://localhost/ticket/533 http://localhost/ticket/533 Report dread <p> Configure buildbot to use postgres db which has been moved to eu5.okfn.org </p> <p> pudo said: Could you perhaps also comment on <a class="ext-link" href="http://knowledgeforge.net/okfn/tasks/ticket/466"><span class="icon">​</span>http://knowledgeforge.net/okfn/tasks/ticket/466</a> with a hint on how to update this properly? I only edited /home/buildslave/okfn/full/build/buildandsmoke/buildandsmoke.ini which was the only occurence of the DB credentials I could find there. </p> http://localhost/ticket/534 http://localhost/ticket/534 Report dread <p> Genshi exception when rendering the page whilst logged in to sl.ckan.net. </p> http://localhost/ticket/535 http://localhost/ticket/535 Report dread <p> Multiple ONS records can go into one package e.g. Stats for Jan, Stats for Feb, Stats for Mar etc. </p> <p> We already identify an existing package by 'external_id'. Now need to find package by title and an extra field (department in ONS case) and allow it to merge in resource and tags. Alert if rest of meta data is different. </p> http://localhost/ticket/536 http://localhost/ticket/536 Report wwaites <p> There are several places where performance is unacceptably slow. Even in places where it is not, the system could still be more responsive for read requests. </p> <p> Introducing caching has to be done carefully and should be done in a <em>standards compliant manner</em>. </p> <h2 id="Generalstrategy">General strategy</h2> <ul><li>Where possible, cache output within the pylons app (beaker). </li><li>Facilitate external caching in an end-user's web browser or a caching proxy </li><li>Slightly stale data is not necessarily much of a problem so allow the output to be cached for a relatively short period (e.g. 5-15 minutes). </li><li>When cache expiry has been reached, a request will be made to the server. The server should check if its internally cached data is still valid, and serve that, otherwise regenerate the data. </li></ul><h2 id="Tasks">Tasks</h2> <p> These tasks should be broken into sub-tickets: </p> <ul><li>caching of parts of templates that are expensive to render (package list, tag list, group list) </li><li>caching of entire output using beaker particularly for API read operations. </li><li>need to perform a check to see if the cache should be invalidated by checking if anything in the output would have changed -- i.e. checking timestamps on package modifications. this is a natural place to introduce the ETag which will help browsers and web caches. </li><li>cache infrastructure front end - varnish, squid, etc. To do this right, the controllers need to set the cache control headers appropriately (max-age, must-revalidate). This is a good resource: <a class="ext-link" href="http://www.mnot.net/cache_docs/#CACHE-CONTROL"><span class="icon">​</span>http://www.mnot.net/cache_docs/#CACHE-CONTROL</a> <ul><li>Deploy varnish on a host dedicated to this purpose for research. This will be useful for other sites as well </li><li>Do not configure varnish to ignore cache control headers or otherwise behave in a non HTTP/1.1 compliant manner </li></ul></li></ul><h2 id="FutureWork">Future Work</h2> <ul><li>Investigate ckanclient library maintaining a local cache as a web browser would </li><li>Investigate using a CDN like Google Storage or Amazon for serving cached data. </li></ul> http://localhost/ticket/537 http://localhost/ticket/537 Report johnbywater <p> Update report queries with more recent versions (below are report queries shipped with trac 0.11.7): </p> <p> NB: Report 1 has already been done. </p> <blockquote> <p> id | query </p> </blockquote> <hr /> <blockquote> <blockquote> <p> 1 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status &lt;&gt; 'closed' : ORDER BY CAST(p.value AS integer), milestone, t.type, time : </p> </blockquote> <p> 2 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, : version AS <span class="underline">group</span>, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status &lt;&gt; 'closed' : ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time : </p> </blockquote> <p> 3 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, </p> <table class="wiki"> <tr>: 'Milestone '<td style="text-align: left">milestone AS <span class="underline">group</span>, </td></tr></table> <p> : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status &lt;&gt; 'closed' : ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time : </p> </blockquote> <p> 4 | </p> <blockquote> <p> : : SELECT p.value AS <span class="underline">color</span>, : owner AS <span class="underline">group</span>, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time : </p> </blockquote> <p> 5 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, : owner AS <span class="underline">group</span>, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : description AS _description_, : changetime AS _changetime, reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time : </p> </blockquote> <p> 6 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, : t.milestone AS <span class="underline">group</span>, : (CASE status : WHEN 'closed' THEN 'color: <a class="closed ticket" href="http://localhost/ticket/777" title="task: Put 503-returning middleware into a module (closed: fixed)">#777</a>; background: #ddd; border-color: #ccc;' : ELSE : (CASE owner WHEN $USER THEN 'font-weight: bold' END) : END) AS <span class="underline">style</span>, : id AS ticket, summary, component, status, : resolution,version, t.type AS type, priority, owner, : changetime AS modified, : time AS _time,reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), : (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC : </p> </blockquote> <p> 7 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, : (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS <span class="underline">group</span>, : id AS ticket, summary, component, version, milestone, : t.type AS type, priority, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE t.status &lt;&gt; 'closed' AND owner = $USER : ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time : </p> </blockquote> <p> 8 | </p> <blockquote> <p> : SELECT p.value AS <span class="underline">color</span>, : (CASE owner : WHEN $USER THEN 'My Tickets' : ELSE 'Active Tickets' : END) AS <span class="underline">group</span>, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status &lt;&gt; 'closed' : ORDER BY (COALESCE(owner, <em>) = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time </em></p> </blockquote> </blockquote> </blockquote> http://localhost/ticket/538 http://localhost/ticket/538 Report rgrp http://localhost/ticket/539 http://localhost/ticket/539 Report rgrp http://localhost/ticket/540 http://localhost/ticket/540 Report rgrp <p> The simplest way to improve performance on specific pages is to implement full page caching. </p> <p> This, however, has challenges on pages with user-specific content. For example, on CKAN when a user is logged at the top of the page we have "You are logged in as: {username}". Similarly we change the visibility of some material depending on the status of users: </p> <ul><li>Visibility of Edit or Authz link on packages and groups </li></ul> http://localhost/ticket/541 http://localhost/ticket/541 Report rgrp <p> To allow for full page caching even on customized pages for logged in users we can use cookie+javascript techniques as described here: <a class="ext-link" href="http://www.ibm.com/developerworks/web/library/wa-rails2/"><span class="icon">​</span>http://www.ibm.com/developerworks/web/library/wa-rails2/</a> </p> <p> In the first place we just need to apply this to: </p> <ul><li>You are logged in as </li><li>Package/group edit/authz tabs </li></ul> http://localhost/ticket/542 http://localhost/ticket/542 Report rgrp <p> Edge-side includes or partial page caching are a standard way to deal with caching of pages in which some (usually small) part of the content cannot be cached or should be cached in a different manner (e.g. much more briefly) than the rest of the page. </p> <p> Edge-side includes have the advantage that they integrate with general 3rd-party caching systems such as varnish. </p> <p> Introducing either partial page or ESI will require some overhaul work to change the page render processing somewhat. </p> http://localhost/ticket/543 http://localhost/ticket/543 Report pudo <p> This is in IATI, would be nice to have in generic CKAN. </p> <ul><li><a class="ext-link" href="http://bitbucket.org/okfn/iati/changeset/378431974c76"><span class="icon">​</span>http://bitbucket.org/okfn/iati/changeset/378431974c76</a> </li></ul> http://localhost/ticket/544 http://localhost/ticket/544 Report johnbywater http://localhost/ticket/545 http://localhost/ticket/545 Report johnbywater http://localhost/ticket/546 http://localhost/ticket/546 Report johnbywater http://localhost/ticket/547 http://localhost/ticket/547 Report johnbywater http://localhost/ticket/548 http://localhost/ticket/548 Report johnbywater http://localhost/ticket/549 http://localhost/ticket/549 Report johnbywater http://localhost/ticket/550 http://localhost/ticket/550 Report johnbywater http://localhost/ticket/551 http://localhost/ticket/551 Report johnbywater http://localhost/ticket/552 http://localhost/ticket/552 Report johnbywater http://localhost/ticket/553 http://localhost/ticket/553 Report johnbywater http://localhost/ticket/554 http://localhost/ticket/554 Report johnbywater http://localhost/ticket/555 http://localhost/ticket/555 Report johnbywater http://localhost/ticket/556 http://localhost/ticket/556 Report pudo <p> Displaying siblings is not needed for link_to/from. </p> http://localhost/ticket/557 http://localhost/ticket/557 Report pudo http://localhost/ticket/558 http://localhost/ticket/558 Report pudo <p> Lookup is needed to add a package but won't show up at some times. </p> <p> Originally reported by Anja Jentzsch, re questions. </p> http://localhost/ticket/559 http://localhost/ticket/559 Report rgrp <p> E.g. <a class="ext-link" href="http://ckan.net/tag/lod"><span class="icon">​</span>http://ckan.net/tag/lod</a> contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins). </p> <p> Also suggested that these are showing up in the API (courtesy of Anja) </p> <p> Cost: 2h </p> http://localhost/ticket/560 http://localhost/ticket/560 Report pudo <p> Anja is reporting this, severe bug, I think. </p> http://localhost/ticket/561 http://localhost/ticket/561 Report pudo <p> reported by LOD folks. </p> <p> subsidary bug: error messages from OpenID providers are not shown on the form return page. </p> http://localhost/ticket/562 http://localhost/ticket/562 Report johnbywater http://localhost/ticket/563 http://localhost/ticket/563 Report johnbywater http://localhost/ticket/564 http://localhost/ticket/564 Report johnbywater http://localhost/ticket/565 http://localhost/ticket/565 Report johnbywater http://localhost/ticket/566 http://localhost/ticket/566 Report johnbywater http://localhost/ticket/567 http://localhost/ticket/567 Report johnbywater http://localhost/ticket/568 http://localhost/ticket/568 Report johnbywater http://localhost/ticket/569 http://localhost/ticket/569 Report johnbywater http://localhost/ticket/570 http://localhost/ticket/570 Report johnbywater http://localhost/ticket/571 http://localhost/ticket/571 Report johnbywater <p> The attributes we need to read in are being defined here: <a class="ext-link" href="http://okfnpad.org/uklii"><span class="icon">​</span>http://okfnpad.org/uklii</a> </p> <p> The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes. </p> <p> Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft. </p> http://localhost/ticket/572 http://localhost/ticket/572 Report johnbywater http://localhost/ticket/573 http://localhost/ticket/573 Report johnbywater http://localhost/ticket/574 http://localhost/ticket/574 Report johnbywater http://localhost/ticket/575 http://localhost/ticket/575 Report wwaites <p> Requires 0.4. 0.5 is a maintenance branch, 0.6 is current. </p> <p> VDM appears to work correctly (all the tests pass) with 0.5. CKAN does not -- requires more investigation to determine exactly why. </p> <p> It would actually be nice to be able to run with 0.6 or 0.5, though 0.6 will require some changes to VDM as well </p> http://localhost/ticket/576 http://localhost/ticket/576 Report johnbywater http://localhost/ticket/577 http://localhost/ticket/577 Report johnbywater http://localhost/ticket/578 http://localhost/ticket/578 Report johnbywater http://localhost/ticket/579 http://localhost/ticket/579 Report johnbywater http://localhost/ticket/580 http://localhost/ticket/580 Report johnbywater <p> Need to establish procedure for automatically generating package names from metadata documents. </p> http://localhost/ticket/581 http://localhost/ticket/581 Report johnbywater http://localhost/ticket/582 http://localhost/ticket/582 Report johnbywater http://localhost/ticket/583 http://localhost/ticket/583 Report johnbywater http://localhost/ticket/584 http://localhost/ticket/584 Report johnbywater http://localhost/ticket/585 http://localhost/ticket/585 Report johnbywater http://localhost/ticket/586 http://localhost/ticket/586 Report johnbywater http://localhost/ticket/587 http://localhost/ticket/587 Report johnbywater http://localhost/ticket/588 http://localhost/ticket/588 Report johnbywater http://localhost/ticket/589 http://localhost/ticket/589 Report johnbywater http://localhost/ticket/590 http://localhost/ticket/590 Report johnbywater http://localhost/ticket/591 http://localhost/ticket/591 Report johnbywater http://localhost/ticket/592 http://localhost/ticket/592 Report johnbywater http://localhost/ticket/593 http://localhost/ticket/593 Report johnbywater http://localhost/ticket/594 http://localhost/ticket/594 Report johnbywater http://localhost/ticket/595 http://localhost/ticket/595 Report johnbywater http://localhost/ticket/596 http://localhost/ticket/596 Report johnbywater http://localhost/ticket/597 http://localhost/ticket/597 Report johnbywater http://localhost/ticket/598 http://localhost/ticket/598 Report johnbywater http://localhost/ticket/599 http://localhost/ticket/599 Report johnbywater http://localhost/ticket/600 http://localhost/ticket/600 Report johnbywater http://localhost/ticket/601 http://localhost/ticket/601 Report johnbywater http://localhost/ticket/602 http://localhost/ticket/602 Report johnbywater http://localhost/ticket/603 http://localhost/ticket/603 Report johnbywater http://localhost/ticket/604 http://localhost/ticket/604 Report johnbywater http://localhost/ticket/605 http://localhost/ticket/605 Report johnbywater http://localhost/ticket/606 http://localhost/ticket/606 Report johnbywater http://localhost/ticket/607 http://localhost/ticket/607 Report johnbywater http://localhost/ticket/608 http://localhost/ticket/608 Report johnbywater http://localhost/ticket/609 http://localhost/ticket/609 Report johnbywater http://localhost/ticket/610 http://localhost/ticket/610 Report johnbywater http://localhost/ticket/611 http://localhost/ticket/611 Report johnbywater http://localhost/ticket/612 http://localhost/ticket/612 Report johnbywater http://localhost/ticket/613 http://localhost/ticket/613 Report johnbywater http://localhost/ticket/614 http://localhost/ticket/614 Report johnbywater <p> We "just" need a schematron.validate(metadata) method. :-) </p> http://localhost/ticket/615 http://localhost/ticket/615 Report johnbywater http://localhost/ticket/616 http://localhost/ticket/616 Report johnbywater http://localhost/ticket/617 http://localhost/ticket/617 Report johnbywater http://localhost/ticket/618 http://localhost/ticket/618 Report johnbywater http://localhost/ticket/619 http://localhost/ticket/619 Report johnbywater http://localhost/ticket/620 http://localhost/ticket/620 Report johnbywater http://localhost/ticket/621 http://localhost/ticket/621 Report johnbywater http://localhost/ticket/622 http://localhost/ticket/622 Report johnbywater http://localhost/ticket/623 http://localhost/ticket/623 Report johnbywater http://localhost/ticket/624 http://localhost/ticket/624 Report johnbywater http://localhost/ticket/625 http://localhost/ticket/625 Report johnbywater http://localhost/ticket/626 http://localhost/ticket/626 Report johnbywater http://localhost/ticket/627 http://localhost/ticket/627 Report johnbywater http://localhost/ticket/628 http://localhost/ticket/628 Report johnbywater http://localhost/ticket/629 http://localhost/ticket/629 Report johnbywater http://localhost/ticket/630 http://localhost/ticket/630 Report johnbywater http://localhost/ticket/631 http://localhost/ticket/631 Report johnbywater http://localhost/ticket/632 http://localhost/ticket/632 Report johnbywater http://localhost/ticket/633 http://localhost/ticket/633 Report johnbywater http://localhost/ticket/634 http://localhost/ticket/634 Report johnbywater http://localhost/ticket/635 http://localhost/ticket/635 Report johnbywater http://localhost/ticket/636 http://localhost/ticket/636 Report johnbywater http://localhost/ticket/637 http://localhost/ticket/637 Report johnbywater http://localhost/ticket/638 http://localhost/ticket/638 Report johnbywater http://localhost/ticket/639 http://localhost/ticket/639 Report johnbywater http://localhost/ticket/640 http://localhost/ticket/640 Report johnbywater http://localhost/ticket/641 http://localhost/ticket/641 Report johnbywater http://localhost/ticket/642 http://localhost/ticket/642 Report johnbywater http://localhost/ticket/643 http://localhost/ticket/643 Report johnbywater http://localhost/ticket/644 http://localhost/ticket/644 Report johnbywater http://localhost/ticket/645 http://localhost/ticket/645 Report johnbywater http://localhost/ticket/646 http://localhost/ticket/646 Report pudo <p> allow group-level authorization on all system entities </p> http://localhost/ticket/647 http://localhost/ticket/647 Report pudo <ul><li>copy exisiting tests, modify authz in setUp, adapt and extend tests <ul><li>Problem: default_role_actions is read by init_db </li><li>Solution: nuke db after monkey-patching </li><li>role = model.Role('Reader'), del role.actions[...] </li></ul></li></ul><blockquote> <blockquote> <p> self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions) mauthz.default_role_actions.remove((Role.READER, Action.CREATE)) #raise Exception(mauthz.default_role_actions) model.Session.remove() model.repo.rebuild_db() </p> </blockquote> </blockquote> <ol start="3"><li>Start from the functional <ul><li>Both the api controllers and wui controllers to do </li><li>tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. <a class="ext-link" href="http://bitbucket.org/pudo/ckan-authz2"><span class="icon">​</span>http://bitbucket.org/pudo/ckan-authz2</a> </li></ul></li></ol><p> is_authorized(user, Action.Create, model.Package) </p> <p> -&gt; Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards <a class="missing wiki">PackageRole?</a> in all queries. Since for class-level role assignments there never is a <a class="missing wiki">PackageRole?</a> join table entry, this will never return any results. </p> <ul><li>cf branch "classes": <a class="ext-link" href="http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66"><span class="icon">​</span>http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66</a> </li><li>No possibility of including "instance" column in inheritance decision. </li><li>Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling. </li></ul><ul><li>have a ckan install that would not allow visitors to either list packages or list groups <ul><li>two paths: create the listing, but for each group/pkg decide that you cannot show this </li><li>lock down the whole page (/package/list) <ul><li>this is class-based, not object-based </li></ul></li></ul></li></ul><p> is_authorized(user, Action.Package_Create, model.System()) is_authorized(user, Action.Group_Create, model.System()) </p> <p> [Separating Package and Group roles may be useful going forward: <a class="missing wiki">PackageEditor?</a>, <a class="missing wiki">GroupEditor?</a> etc] </p> <ol start="4"><li>Find a standard way to lock down classes <ul><li>possibly add default rows in user_object_role </li><li>introduce lock-down into controllers: group new, package new, REST equivalents, </li><li>confirm tests </li></ul></li></ol> http://localhost/ticket/648 http://localhost/ticket/648 Report pudo <p> Authorization Groups: </p> <ul><li>Rename <a class="missing wiki">UserObjectRole?</a> to <a class="missing wiki">SubjectObjectRole?</a> </li><li>Create database migration to do the rename in DB </li><li>Make sure all the tests are passing again </li></ul><blockquote> <p> (Doing this first seems more stable since we first want to "re-gain" exisiting functionality) </p> </blockquote> <p> </p> <ul><li>Create authz.py and model tests for authorization_group-based auth <ul><li>tests: functional: create fixtures, do actual requests </li></ul></li></ul><blockquote> <blockquote> <blockquote> <p> lib-tests: test for membership addition, removal, group-right additions affecting all members, group-right removal, group rights affecting non-members. </p> </blockquote> </blockquote> </blockquote> <p> # * Adapt model to get these passing </p> <ul><li>Integration into existing controllers should be negligible but may want ot have tests <ul><li>do not want that. would mean design is broken </li></ul></li><li>Need to create a new controller for editing authz groups (will have same permissions as any other object -- Reader, Editor, Administrator) </li></ul> http://localhost/ticket/649 http://localhost/ticket/649 Report pudo <ul><li>Copy &amp; adapt Group controller tests to suit <a class="missing wiki">AuthorizationGroups?</a> </li><li>Copy &amp; adapt Group controllers and templates for Authorization Groups </li></ul> http://localhost/ticket/650 http://localhost/ticket/650 Report pudo <p> This needs to be abstracted and we still need to be able to remove items before saving. </p> http://localhost/ticket/651 http://localhost/ticket/651 Report pudo <p> Much more natural place to do this, even at the risk of overloading the package edit form. </p> http://localhost/ticket/652 http://localhost/ticket/652 Report dread <p> When people link to a package, a track-back link is auto-created. (Similar system as for blogs). </p> <p> As suggested by Tim Davies: </p> <p> Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already. </p> <p> I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset. </p> http://localhost/ticket/653 http://localhost/ticket/653 Report johnbywater http://localhost/ticket/654 http://localhost/ticket/654 Report johnbywater http://localhost/ticket/655 http://localhost/ticket/655 Report johnbywater http://localhost/ticket/656 http://localhost/ticket/656 Report johnbywater http://localhost/ticket/657 http://localhost/ticket/657 Report johnbywater http://localhost/ticket/658 http://localhost/ticket/658 Report dread <p> Auto way to check web and API interface of a CKAN instance basically works. Several gotchas can be quickly determined, such as logging in, search not working. Needs to be configurable per site basis. </p> http://localhost/ticket/659 http://localhost/ticket/659 Report dread <p> As an admin I want to check a CKAN instance works having just upgraded it or configured it. </p> http://localhost/ticket/660 http://localhost/ticket/660 Report dread <p> From Rufus: </p> <pre class="wiki">'blackbox' testing command (read-only only would be fine) -- very useful for upgrade on sunday On blackbox thinking of something like: paster test-site http://hmg.ckan.net {package-name} Or (maybe better): paster test-site --site-spec {myconfig.json} my-config is a json file with info for testing (could overload --config rather than site-spec): site-url: http://hmg.ckan.net/ package-name: package-title: ...etc ... Then: Test front-page Test /revision/ Test /package/{page} -- check title Test /package/history Test /api/rest/package/ ... Test /api/search/package ... etc </pre> http://localhost/ticket/661 http://localhost/ticket/661 Report johnbywater <p> It's because Package carries several out-of-band values, which are snagged on the way back out. Entity get response also can't be posted. </p> <p> However, post response can be re-posted (because it isn't the same as the register-post/entity-get responses. </p> <p> An issue for CKAN too. </p> <p> Sub-ticket of <a class="closed ticket" href="http://localhost/ticket/961" title="enhancement: [super] Refactoring of forms, validation and model synchronization (closed: fixed)">#961</a> (form, validation, model sync meta-ticket) and depends on that work. </p> http://localhost/ticket/662 http://localhost/ticket/662 Report dread <p> Looking at: <a class="ext-link" href="http://ckan.net/package/rkb-explorer-acm"><span class="icon">​</span>http://ckan.net/package/rkb-explorer-acm</a> </p> <p> 'rkb-explorer-resex' gets display multiple times and shouldn't </p> http://localhost/ticket/663 http://localhost/ticket/663 Report pudo http://localhost/ticket/664 http://localhost/ticket/664 Report johnbywater <p> Discussion between John and Peter: </p> <blockquote class="citation"> <p> Given we can identify a document, does the disappearance of a document from a registered source imply the disappearance of the metadata (such that we delete packages once the documents disappear from the registered source)? </p> </blockquote> <p> I would expect a more explicit 'delete'. The UKLP Use Case Library describes this as "withdraw a dataset or service from publication" (part of UCD03 Maintain resources). </p> http://localhost/ticket/665 http://localhost/ticket/665 Report johnbywater http://localhost/ticket/666 http://localhost/ticket/666 Report dread <p> See message and script: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html</a> </p> <p> 1500 requests shouldn't take hours. </p> http://localhost/ticket/667 http://localhost/ticket/667 Report Colin Calnan <p> It seems like the API v1 on CKAN metastable (cset:ec21f8e1c87e) has some caching issues. </p> <p> Steps to test: </p> <ol><li>Modify a dataset on datadotgc.ca, redirects to CKAN </li></ol><ol start="2"><li>On save, redirects to <a class="ext-link" href="http://www.datadotgc.ca/update/geogratisnat_hydrography_v100"><span class="icon">​</span>http://www.datadotgc.ca/update/geogratisnat_hydrography_v100</a> which in turn redirects to <a class="ext-link" href="http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100"><span class="icon">​</span>http://www.datadotgc.ca/dataset/geogratisnat_hydrography_v100</a> </li></ol><ol start="3"><li>You can see that the Dataset has not updated correctly. Run a check on the API v1 - <a class="ext-link" href="http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100"><span class="icon">​</span>http://ca.ckan.net/api/1/rest/package/geogratisnat_hydrography_v100</a> the updates are not present </li></ol><ol start="4"><li>Check the v2 of the API - <a class="ext-link" href="http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100"><span class="icon">​</span>http://ca.ckan.net/api/rest/package/geogratisnat_hydrography_v100</a>, the updates are present. </li></ol><ol start="5"><li>Setting the headers to 'Cache-control: no-cache' or 'Pragma: no-cache' does not work either. </li></ol> http://localhost/ticket/668 http://localhost/ticket/668 Report pudo http://localhost/ticket/669 http://localhost/ticket/669 Report pudo http://localhost/ticket/670 http://localhost/ticket/670 Report pudo http://localhost/ticket/671 http://localhost/ticket/671 Report pudo <p> This has been done in IATI ckan but should be ported over to CKAN main and put into the UI in a nice fashion. </p> <p> IATI: <a class="ext-link" href="http://bitbucket.org/okfn/iati/changeset/378431974c76"><span class="icon">​</span>http://bitbucket.org/okfn/iati/changeset/378431974c76</a> (related to <a class="closed ticket" href="http://localhost/ticket/520" title="requirement: Browseable web interface onto the data (closed: fixed)">#520</a>) </p> <p> Requires SOLR search backend (for postgres should fall back gracefully -- just do not show them). </p> http://localhost/ticket/672 http://localhost/ticket/672 Report johnbywater http://localhost/ticket/673 http://localhost/ticket/673 Report johnbywater http://localhost/ticket/674 http://localhost/ticket/674 Report johnbywater http://localhost/ticket/675 http://localhost/ticket/675 Report johnbywater http://localhost/ticket/676 http://localhost/ticket/676 Report dread <ul><li>Uses API to change licenses of all packages in a CKAN instance to a particular one. </li></ul><ul><li>Run it on DGU. </li></ul> http://localhost/ticket/677 http://localhost/ticket/677 Report dread <ul><li>Create form </li><li>Deploy on test </li><li>Deploy on live </li></ul> http://localhost/ticket/678 http://localhost/ticket/678 Report dread <ul><li>Add help text, drawing from: <ul><li>ca.ckan.net </li><li><a class="ext-link" href="http://wiki.okfn.org/ckan/doc/package"><span class="icon">​</span>http://wiki.okfn.org/ckan/doc/package</a> </li><li><a class="ext-link" href="http://wiki.okfn.org/ckan/doc/faq"><span class="icon">​</span>http://wiki.okfn.org/ckan/doc/faq</a> </li></ul></li><li>Make longer text revealed on click </li></ul> http://localhost/ticket/679 http://localhost/ticket/679 Report dread http://localhost/ticket/680 http://localhost/ticket/680 Report dread <p> <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/613"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/613</a> </p> http://localhost/ticket/681 http://localhost/ticket/681 Report dread <p> Some work done on this to make tests run independently. </p> http://localhost/ticket/682 http://localhost/ticket/682 Report dread http://localhost/ticket/683 http://localhost/ticket/683 Report dread http://localhost/ticket/684 http://localhost/ticket/684 Report dread http://localhost/ticket/685 http://localhost/ticket/685 Report dread http://localhost/ticket/686 http://localhost/ticket/686 Report dread <p> Done. Added release process here: <a class="wiki" href="http://localhost/wiki/ReleaseProcess">wiki:ReleaseProcess</a> </p> http://localhost/ticket/687 http://localhost/ticket/687 Report johnbywater http://localhost/ticket/688 http://localhost/ticket/688 Report johnbywater http://localhost/ticket/689 http://localhost/ticket/689 Report johnbywater <p> Wish to fold service records into package resources. Trouble is that service records don't necessarily refer to a data record, so should this be a new package? </p> http://localhost/ticket/690 http://localhost/ticket/690 Report johnbywater http://localhost/ticket/691 http://localhost/ticket/691 Report johnbywater <p> To do this for the UKLP, the Ordnance Survey would need to add to their map service an operation to return image for a given bounding box, and COI would need to present the image on the search result page. </p> <p> Otherwise, it would be possible to use Google. </p> http://localhost/ticket/692 http://localhost/ticket/692 Report wwaites <p> ... and make cache-control header optional </p> http://localhost/ticket/693 http://localhost/ticket/693 Report dread <p> Database for all CKAN instances upgraded to Postgres 8.4, but none of the eu machines were upgraded with the tools necessary to administer them. </p> http://localhost/ticket/694 http://localhost/ticket/694 Report dread <p> e.g. searching for 'buddhist' or 'sanskrit', you don't get this newly created package: <a class="ext-link" href="http://ckan.net/package/digitalsanskritbuddhistcanon"><span class="icon">​</span>http://ckan.net/package/digitalsanskritbuddhistcanon</a> </p> http://localhost/ticket/695 http://localhost/ticket/695 Report johnbywater http://localhost/ticket/696 http://localhost/ticket/696 Report johnbywater <p> This seems to require doing something else with such a group's 'delete' button, such as showing it greyed, or perhaps not showing it at all. </p> http://localhost/ticket/697 http://localhost/ticket/697 Report thejimmyg <p> This proposal is to discuss adding a new API for proxying certain spreadsheet data via JSON-P to make it possible to build simple browser apps directly off the API. </p> <p> See the attached proposal for information. </p> http://localhost/ticket/698 http://localhost/ticket/698 Report TimDavies <p> When getting a search results page (e.g. <a class="ext-link" href="http://ckan.net/package/search?q=development&amp;open_only=1&amp;downloadable_only=1"><span class="icon">​</span>http://ckan.net/package/search?q=development&amp;open_only=1&amp;downloadable_only=1</a>) and then using the on-page links to page through results, the next link / numbered page links go to <a class="ext-link" href="http://ckan.net/package/search?q=development&amp;page=2"><span class="icon">​</span>http://ckan.net/package/search?q=development&amp;page=2</a> rather than <a class="ext-link" href="http://ckan.net/package/search?q=development&amp;open_only=1&amp;downloadable_only=1&amp;page=2"><span class="icon">​</span>http://ckan.net/package/search?q=development&amp;open_only=1&amp;downloadable_only=1&amp;page=2</a> meaning that the filter flags set (Download Only; Open Only) are not respected on the second page of listings. </p> http://localhost/ticket/699 http://localhost/ticket/699 Report dread <p> Editing groups in forms doesn't work for me, with latest code from this morning: </p> <ol><li>Clean db </li><li>paster create-test-data </li><li>paster sysadmin create <a class="ext-link" href="http://davidread.myopenid.com/"><span class="icon">​</span>http://davidread.myopenid.com/</a> </li><li>paster serve development.ini </li><li>In browser, log in to CKAN </li><li>Create new package: name=abc group=Roger's books </li><li>[Preview] - yes group appears </li><li>[Save] - shows package and group hasn't appeared ERROR </li><li>Check <a class="ext-link" href="http://127.0.0.1:5000/api/rest/package/abc"><span class="icon">​</span>http://127.0.0.1:5000/api/rest/package/abc</a> and reedit package it also doesn't appear here either. </li></ol> http://localhost/ticket/700 http://localhost/ticket/700 Report johnbywater http://localhost/ticket/701 http://localhost/ticket/701 Report johnbywater http://localhost/ticket/702 http://localhost/ticket/702 Report johnbywater http://localhost/ticket/703 http://localhost/ticket/703 Report johnbywater http://localhost/ticket/704 http://localhost/ticket/704 Report johnbywater http://localhost/ticket/705 http://localhost/ticket/705 Report johnbywater http://localhost/ticket/706 http://localhost/ticket/706 Report johnbywater http://localhost/ticket/707 http://localhost/ticket/707 Report johnbywater http://localhost/ticket/708 http://localhost/ticket/708 Report johnbywater http://localhost/ticket/709 http://localhost/ticket/709 Report johnbywater http://localhost/ticket/710 http://localhost/ticket/710 Report johnbywater <p> Previously, a package type attribute was intended to distinguish between package data that came from government depts. and publishers location information (GEMINI). Now there is an intention use a common set of attributes to support data from either source. </p> <p> Hence, there currently a v.3 of the government attributes, a document about extracting the attributes from Gemini document using XPath, and a comparison documents mapping between governemtn attributes and Gemini attributes. </p> <p> That means there are potentially four different list of attributes. I'm not sure how to go about reviewing the lists and converging any differences. </p> <p> But we do want to end up with a list of names, a description of their purpose, and then perhaps the XPath which would reveal the value from a Gemini doc? </p> http://localhost/ticket/711 http://localhost/ticket/711 Report johnbywater <p> Where said "v0.4" would be the result of converging four lists mentioned in requirement <a class="closed ticket" href="http://localhost/ticket/711" title="requirement: Packages shall support both UK government and location information with a ... (closed: fixed)">#711</a>. </p> http://localhost/ticket/712 http://localhost/ticket/712 Report johnbywater http://localhost/ticket/713 http://localhost/ticket/713 Report johnbywater <p> So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission. </p> http://localhost/ticket/714 http://localhost/ticket/714 Report johnbywater <p> Symptom <a class="closed ticket" href="http://localhost/ticket/1" title="enhancement: Visitor visits service (closed: fixed)">#1</a>: Groups without titles don't get a clickable link, because there is no text. Defaulting to the group name would solve this trouble. Symptom <a class="closed ticket" href="http://localhost/ticket/2" title="enhancement: Register as user on the service (closed: fixed)">#2</a>: Package groups drop-down list show's 'None' when a group doesn't have a title. </p> <p> Therefore, we need to centralize presentation of group title, so it defers to group name. And we need tests. </p> <p> Also please sort groups by title on group list page. </p> http://localhost/ticket/715 http://localhost/ticket/715 Report pudo http://localhost/ticket/716 http://localhost/ticket/716 Report pudo http://localhost/ticket/717 http://localhost/ticket/717 Report pudo http://localhost/ticket/718 http://localhost/ticket/718 Report pudo <p> ... probably by adapting aidtransparency.net </p> http://localhost/ticket/719 http://localhost/ticket/719 Report pudo <p> Perhaps using deliverance? </p> <ul><li>Move About to the right of the nav tabs and make it an about IATI page </li><li>Make a Help / FAQ section linked of main menu </li><li>Front page more dynamic - latest data registered and news (microblog thing) </li><li>make the front page blurb noticable / shorter etc. </li><li>link to iatistandard.org from the front page blurb </li></ul> http://localhost/ticket/720 http://localhost/ticket/720 Report pudo http://localhost/ticket/721 http://localhost/ticket/721 Report pudo <ul><li>Change "Publishing Entity" to "Publisher" </li></ul> http://localhost/ticket/722 http://localhost/ticket/722 Report pudo http://localhost/ticket/723 http://localhost/ticket/723 Report pudo http://localhost/ticket/724 http://localhost/ticket/724 Report pudo <p> With a pending state set ("unapproved") </p> http://localhost/ticket/725 http://localhost/ticket/725 Report pudo http://localhost/ticket/726 http://localhost/ticket/726 Report pudo http://localhost/ticket/727 http://localhost/ticket/727 Report johnbywater <p> Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please? </p> http://localhost/ticket/728 http://localhost/ticket/728 Report pudo <p> Move from hetzner box to s3. </p> http://localhost/ticket/729 http://localhost/ticket/729 Report rgrp <p> Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net. </p> <ul><li>Naming scheme? <ul><li>Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}? <ul><li>What happens if names change </li></ul></li><li>File: filename? hash? </li></ul></li><li>Store hash back on ckan instance? </li></ul> http://localhost/ticket/730 http://localhost/ticket/730 Report dread <p> Sometimes editing a package via api on dgu results in some countries being lost in geo coverage. </p> http://localhost/ticket/731 http://localhost/ticket/731 Report dread http://localhost/ticket/732 http://localhost/ticket/732 Report dread http://localhost/ticket/733 http://localhost/ticket/733 Report dread <p> Creation of duplicate packages named with trailing underscore </p> <p> ww 22/9/10: </p> <blockquote class="citation"> <blockquote class="citation"> <p> Need to ... merge some packages (the "find packages by attributes" is not 100% reliable because of inconsistent data already in there). </p> </blockquote> </blockquote> <p> The "find by tags" algorithm uses (title, department). Unfortunately some of the data already there has no department. This causes it to make a new package (with trailing underscore). </p> <p> dread 24/9/10: </p> <p> I've done a test for the missing department issue and I can't reproduce the problem. This may be due to my general tidy up and getting tests passing using the latest ckanclient. Anyway, can you check over my test to see if this is what you mean?: dgu/ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadMissingDept </p> <p> ww 20/10/10: </p> <p> you should uncover it if you run against (a copy of) the dgu database. </p> http://localhost/ticket/734 http://localhost/ticket/734 Report dread <p> ww 22/9/10: </p> <p> (discussion of <a class="closed ticket" href="http://localhost/ticket/509" title="story: ONS data imported daily (closed: invalid)">ticket:509</a>) This leads us to deduping which should be the next order of business oin this thread... </p> <p> ww 20/10/10: </p> <p> if a dataset is missing a department and the ons script makes a new version of it with a department we now have two reccords for the same dataset. these should be manually merged - need a merge tool where you can tell it that two datasets are the same. </p> http://localhost/ticket/735 http://localhost/ticket/735 Report johnbywater <p> Robust against slightly incorrect URLs (e.g. with trailing slash, or without /csw), and non-CWS service which also return &lt;ows:ExceptionReport ... /&gt;. </p> <p> Both cases would be resolved by checking capabilities (with CswClient's check_get_capabilities() method). </p> http://localhost/ticket/736 http://localhost/ticket/736 Report dread <p> I suggest we produce a quick Markdown cheat-sheet page, showing the key runes: e.g. create a title and quote some text. This page can link to the full Markdown docs for advanced users. </p> <p> A user going to the Markdown docs that we link will have to read a couple of pages of the raison-d'etre of Markdown before he gets to the syntax. And it's not very easy to read, and being white on black it looks like proper geek stuff. </p> http://localhost/ticket/737 http://localhost/ticket/737 Report johnbywater http://localhost/ticket/738 http://localhost/ticket/738 Report johnbywater <p> ...obviously, not all packages will have a harvested metadata document. </p> http://localhost/ticket/739 http://localhost/ticket/739 Report johnbywater http://localhost/ticket/740 http://localhost/ticket/740 Report ollyc <p> Create plugin system using &lt;<a class="ext-link" href="http://pypi.python.org/pypi/pyutilib.component.core/"><span class="icon">​</span>http://pypi.python.org/pypi/pyutilib.component.core/</a>&gt; </p> <p> Specification in CEP-003 (<a class="source" href="http://localhost/browser/doc/cep/cep-003.txt">browser:doc/cep/cep-003.txt</a>) and &lt;<a class="ext-link" href="http://okfnpad.org/ckan-plugins"><span class="icon">​</span>http://okfnpad.org/ckan-plugins</a>&gt; </p> http://localhost/ticket/741 http://localhost/ticket/741 Report dread <p> Need to make sure <a class="closed ticket" href="http://localhost/ticket/700" title="bug: Groups in package form (closed: fixed)">#700</a> is fixed. </p> http://localhost/ticket/742 http://localhost/ticket/742 Report johnbywater http://localhost/ticket/743 http://localhost/ticket/743 Report johnbywater http://localhost/ticket/744 http://localhost/ticket/744 Report johnbywater http://localhost/ticket/745 http://localhost/ticket/745 Report johnbywater http://localhost/ticket/746 http://localhost/ticket/746 Report johnbywater http://localhost/ticket/747 http://localhost/ticket/747 Report johnbywater http://localhost/ticket/748 http://localhost/ticket/748 Report johnbywater http://localhost/ticket/749 http://localhost/ticket/749 Report johnbywater http://localhost/ticket/750 http://localhost/ticket/750 Report johnbywater http://localhost/ticket/751 http://localhost/ticket/751 Report johnbywater http://localhost/ticket/752 http://localhost/ticket/752 Report johnbywater http://localhost/ticket/753 http://localhost/ticket/753 Report johnbywater <p> This has once again been ruled out for UKLP for the timebeing. </p> http://localhost/ticket/754 http://localhost/ticket/754 Report johnbywater http://localhost/ticket/755 http://localhost/ticket/755 Report johnbywater http://localhost/ticket/756 http://localhost/ticket/756 Report johnbywater http://localhost/ticket/757 http://localhost/ticket/757 Report johnbywater http://localhost/ticket/758 http://localhost/ticket/758 Report johnbywater http://localhost/ticket/759 http://localhost/ticket/759 Report johnbywater http://localhost/ticket/760 http://localhost/ticket/760 Report dread <p> Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done. </p> http://localhost/ticket/761 http://localhost/ticket/761 Report dread <p> A CKAN is used just for distributing metadata. Updates may still arrive through direct db manipulation, e.g.: </p> <ul><li>another (but writable) CKAN instance is connected to the same db </li><li>restoring database dumps from another CKAN db </li></ul> http://localhost/ticket/762 http://localhost/ticket/762 Report dread <p> Admin configures entering read-only mode in one of two places: </p> <ul><li>CKAN config file (e.g. ckan.ini) </li><li>environment variable from Apache config </li></ul><p> Once enabled, no writes can occur to the database (including user ratings and other usage stats). </p> http://localhost/ticket/763 http://localhost/ticket/763 Report dread <p> Whilse using CKAN web interface, you are not tempted to edit stuff: </p> <ul><li>You know at all times this CKAN is read-only </li><li>All editing facilities are still seen but greyed-out with an indication why it is. </li></ul> http://localhost/ticket/764 http://localhost/ticket/764 Report dread <p> All writes to the API are captured and you are returned an error explaining the reason. </p> <p> Possible errors: </p> <ul><li>503 temporary maintenance </li><li>403 forbidden (if server if permanently read-only) </li></ul> http://localhost/ticket/765 http://localhost/ticket/765 Report johnbywater <p> Perhaps 80% of the users of the UKLII may be using ArcGIS. </p> http://localhost/ticket/766 http://localhost/ticket/766 Report johnbywater http://localhost/ticket/767 http://localhost/ticket/767 Report johnbywater http://localhost/ticket/768 http://localhost/ticket/768 Report johnbywater http://localhost/ticket/769 http://localhost/ticket/769 Report johnbywater <p> Issue 1. Help command without arguments fails: </p> <p> $paster help --plugin ckan --config development.ini </p> <p> gives: </p> <p> EITHER </p> <p> Command help not recognized </p> <p> OR </p> <p> paster help: error: no such option: --plugin </p> <p> Issue 2: Harvester command with arguments fails. </p> <p> $ paster harvester Traceback (most recent call last): ... </p> <blockquote> <p> cmd = self.args<a class="missing changeset" title="No default repository defined">[0]</a> </p> </blockquote> <p> <a class="missing wiki">IndexError?</a>: list index out of range </p> http://localhost/ticket/770 http://localhost/ticket/770 Report johnbywater <p> Should return something sensible (currently reports an error). </p> http://localhost/ticket/771 http://localhost/ticket/771 Report johnbywater <p> Should return help for the harvester (currently crashes). </p> http://localhost/ticket/772 http://localhost/ticket/772 Report johnbywater http://localhost/ticket/773 http://localhost/ticket/773 Report johnbywater http://localhost/ticket/774 http://localhost/ticket/774 Report dread <p> We're getting this warning a great deal on live servers. Is these really a sign of the system not operating correctly or can we reduce the level to an INFO? </p> <p> e.g. on hmg.ckan.net: </p> <pre class="wiki">2010-10-29 17:12:08,262 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,333 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,375 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,406 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,480 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position 2010-10-29 17:12:08,613 WARNI [ckan.lib.search.common] NOOP Index: id,package_id,url,format,description,hash,position </pre> http://localhost/ticket/775 http://localhost/ticket/775 Report dread <p> We get a host of vdm warnings frequently in the CKAN logs. Are these genuine worries or can they be avoided? </p> <pre class="wiki">2010-10-29 17:10:06,003 WARNI [vdm] Skipping adding property Package.package_search (PackageSearch) to revisioned object 2010-10-29 17:10:06,015 WARNI [vdm] Skipping adding property Package.all_revisions_unordered (PackageRevision) to revisioned object 2010-10-29 17:10:06,024 WARNI [vdm] Skipping adding property PackageTag.all_revisions_unordered (PackageTagRevision) to revisioned object 2010-10-29 17:10:06,081 WARNI [vdm] Skipping adding property PackageExtra.all_revisions_unordered (PackageExtraRevision) to revisioned object 2010-10-29 17:10:06,105 WARNI [vdm] Skipping adding property PackageResource.all_revisions_unordered (PackageResourceRevision) to revisioned object 2010-10-29 17:10:06,129 WARNI [vdm] Skipping adding property PackageRelationship.all_revisions_unordered (PackageRelationshipRevision) to revisioned object </pre> http://localhost/ticket/776 http://localhost/ticket/776 Report dread <p> Also configure hmgckan, hmgapi, test-hmg and dev-hmg machines. </p> http://localhost/ticket/777 http://localhost/ticket/777 Report dread http://localhost/ticket/778 http://localhost/ticket/778 Report dread http://localhost/ticket/779 http://localhost/ticket/779 Report dread <p> This ticket refers to the form shown in Drupal. This doesn't have a preview function, but we don't have a requirement for it right now, so leaving ticket closed. </p> http://localhost/ticket/780 http://localhost/ticket/780 Report dread <p> (Just like resource table, but extra fields don't need up/down buttons) </p> <p> Nice to have: a blank field is added when you tab from the last filled-in field in the table. </p> http://localhost/ticket/781 http://localhost/ticket/781 Report rgrp <p> Need to be able to create (and remove) sysadmins easily (used to be able to do this via ini file but this functionality was removed a while back in authz system reorganization it seems). </p> <p> Suggest creating a new paster command as first step. </p> <pre class="wiki"># user identifier = openid (?) paster sysadmin add {user identifier} paster sysadmin remove {user identifier} </pre><p> Suggest this also be added to main CKAN docs. </p> <p> This should go into metastable as well as default so that it goes into the v1.2 release (can do this via a transplant from one branch to the other) </p> http://localhost/ticket/782 http://localhost/ticket/782 Report johnbywater <p> We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer. </p> http://localhost/ticket/783 http://localhost/ticket/783 Report johnbywater http://localhost/ticket/784 http://localhost/ticket/784 Report johnbywater <p> Scope of CSW guidance document has been broadened to include WAF. </p> http://localhost/ticket/785 http://localhost/ticket/785 Report johnbywater http://localhost/ticket/786 http://localhost/ticket/786 Report thejimmyg <p> Auth Proposal </p> <p> Use case: We'd like an authenticated and authorized Drupal user to be able to edit/delete packages using the API </p> <p> To do this a user would need a CKAN API key so we need some way of Drupal telling CKAN who a user is so that they can get their API key. </p> <p> Proposed Implementation </p> <hr /> <p> A user visits the CKAN API key page to get their key. Because CKAN is at catalogue.data.gov.uk (a subdomain of the Drupal site) it can read Drupal cookies. </p> <p> If there is no <tt></tt>DRXtrArgs<tt></tt> or <tt></tt>DRXtrArgs2<tt></tt> cookie, we know the user isn't signed in so we redirect them to Drupal to sign in. </p> <p> WISHLIST: It would be really nice if we could pass the URL to redirect back to Drupal so tha Druapl can send them back to the CKAN API key page </p> <p> Either way, they get back to CKAN and now the cookies exist. When the first HTTP request header is sent, CKAN will read the Drupal session ID and then call a Drupal API, server to server. </p> <p> TODO: Implement an API on the drupal server which accepts a Drupal session ID as an argument and returns the username and credentials, but only if the request if from the CKAN server (perhaps we specify an API key in the request) </p> <p> If the session is valid CKAN will set its own auth cookie and show them the page with the API key. CKAN only considers a user signed in if both the CKAN cookie *and* Drupal session ID are present. If at any time they sign out of Drupal the Drupal session disappears so they will be signed out of CKAN too. </p> <p> Now the user has an API key they can use the standard CKAN command line API tools. </p> <p> The API key is the only thing the user will need to use the CKAN API. This poses a problem. What if a user is removed from Drupal but still has a CKAN API key? </p> <p> Two solutions: </p> <ol><li>API keys will be set to only be valid for the length of a Drupal session so that a user will only be able to use the write API for a little time after credentials are revoked </li></ol><ol start="2"><li>CKAN provides an API to allow Drupal to revoke keys </li></ol><p> Option 1. seems easier to me. If we choose this we will write a <tt></tt>get_api_key<tt></tt> command line tool so that Drupal user can write things like this: </p> <p> :: </p> <blockquote> <p> datapkg list_packages --api_key <tt>get_api_key</tt> </p> </blockquote> <p> The <tt></tt>get_api_key<tt></tt> function will prompt for username and password and then perform the steps necessary to get an API key. </p> http://localhost/ticket/787 http://localhost/ticket/787 Report johnbywater <p> Needs to delete all harving jobs and harvested documents, and probably to purge all packages (perhaps just delete?). </p> http://localhost/ticket/788 http://localhost/ticket/788 Report johnbywater http://localhost/ticket/789 http://localhost/ticket/789 Report johnbywater http://localhost/ticket/790 http://localhost/ticket/790 Report johnbywater http://localhost/ticket/791 http://localhost/ticket/791 Report johnbywater http://localhost/ticket/792 http://localhost/ticket/792 Report johnbywater http://localhost/ticket/793 http://localhost/ticket/793 Report johnbywater <p> This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent). </p> <p> This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider. </p> http://localhost/ticket/794 http://localhost/ticket/794 Report johnbywater <p> TM wants to know how this will be "matched up". I don't know whether this always the data provider, or sometimes the publisher. </p> http://localhost/ticket/795 http://localhost/ticket/795 Report pudo <p> The <a class="missing wiki">AlphaPage?</a> implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead. </p> http://localhost/ticket/796 http://localhost/ticket/796 Report rgrp <p> WP twentyten has an excellent core html structure. Furthermore, using that structure makes us compatible with all the WP twentyten compatible themes. </p> <ul><li>Convert to wp twentyten html </li><li>Switch css to be based off twentyten css (and perhaps rework somewhat) </li></ul> http://localhost/ticket/797 http://localhost/ticket/797 Report dread <p> Move existing getdata/cospread.py into dgu repo and use the importer/loader mechanism. </p> http://localhost/ticket/798 http://localhost/ticket/798 Report johnbywater http://localhost/ticket/799 http://localhost/ticket/799 Report johnbywater http://localhost/ticket/800 http://localhost/ticket/800 Report johnbywater <p> Implies that last harvested time must be stored on source by harvesting job (see <a class="closed ticket" href="http://localhost/ticket/802" title="story: Save last harvested time on source (closed: duplicate)">#802</a>). </p> http://localhost/ticket/801 http://localhost/ticket/801 Report johnbywater http://localhost/ticket/802 http://localhost/ticket/802 Report johnbywater <p> As a system developer, I want to run 'create next migration version' script, so that I don't have to create migration version tests and files manually. </p> http://localhost/ticket/803 http://localhost/ticket/803 Report johnbywater <p> We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP. </p> http://localhost/ticket/804 http://localhost/ticket/804 Report dread http://localhost/ticket/805 http://localhost/ticket/805 Report rgrp <p> Packages should provide create and last modified attributes which are included in package dict exposed via api. </p> <p> These attributes will not be in db but are computed (depend not just on package but related objects -- package tags, package resources, package extras etc) </p> http://localhost/ticket/806 http://localhost/ticket/806 Report rgrp <p> We get a lot of robot traffic. Notice that robots are visiting package edit pages. Suggest: </p> <ul><li>Add rel=nofollow to various links </li><li>Throttle some robots (not all seem to respect robots.txt) </li></ul> http://localhost/ticket/807 http://localhost/ticket/807 Report rgrp <p> Allow choice of language on ckan instance. </p> <ul><li>Configuration variable listing language choices (could try doing this automatically but recommend against this) </li><li>Switcher option in UI (perhaps on user profile page) </li><li>Default it automatically based on browser settings (?) </li></ul><p> Think this is high value (and cost is low) </p> <p> Cost: low (4h) </p> http://localhost/ticket/808 http://localhost/ticket/808 Report pudo <p> This would allow for a community-driven definition of field values, conventions, etc. </p> <p> Plus: create wiki.ckan.net </p> http://localhost/ticket/809 http://localhost/ticket/809 Report pudo <p> Subticket of forms super ticket <a class="closed ticket" href="http://localhost/ticket/961" title="enhancement: [super] Refactoring of forms, validation and model synchronization (closed: fixed)">#961</a> </p> http://localhost/ticket/810 http://localhost/ticket/810 Report cygri <p> The layout of the editing section for extra fields breaks when a field name is slightly too long. Field names jump over to the right. See <a class="ext-link" href="http://ckan.net/package/edit/dbpedia"><span class="icon">​</span>http://ckan.net/package/edit/dbpedia</a> for examples. </p> http://localhost/ticket/811 http://localhost/ticket/811 Report cygri <h1 id="Rationale">Rationale</h1> <p> The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview). </p> <h1 id="Implementation">Implementation</h1> <p> A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has) </p> <p> Nice to have: a blank field is added when you tab from the last filled-in field in the table. </p> http://localhost/ticket/812 http://localhost/ticket/812 Report cygri <p> A link to the “register new package” page would be handy somewhere on the front page. As a frequent editor, I often go to <a class="ext-link" href="http://ckan.net"><span class="icon">​</span>http://ckan.net</a> in order to create a new package. I don't want to hunt around for the link to the “new package” form. </p> <p> The link could go into the bar next to “About / Statistics”. Or into the “More information” sidebar. Neither of these truly make sense, but it's a frequently accessed link and that's reason enough to put it somewhere onto the frontpage. </p> http://localhost/ticket/813 http://localhost/ticket/813 Report cygri <p> A link “About CKAN” pointing to <a class="ext-link" href="http://ckan.net/about"><span class="icon">​</span>http://ckan.net/about</a> should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is. </p> <p> I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar. </p> <p> But anywhere else would be fine as well. </p> http://localhost/ticket/814 http://localhost/ticket/814 Report cygri <p> The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page. </p> http://localhost/ticket/815 http://localhost/ticket/815 Report cygri <p> The field for entering the format of a resource should have autocomplete, populated from all the values that have already been entered in this field for any resource. </p> <p> This would help improve data quality and consistency. </p> http://localhost/ticket/816 http://localhost/ticket/816 Report cygri <p> There are some problems with the Resources table on package pages: </p> <ol><li>The column labelled “URL” doesn't contain URLs, but just the word “Download”. </li><li>The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links). </li></ol><p> Attached is a screenshot for a proposed redesign. The changes are: </p> <ol><li>Move the Format column to the right of the Description column (description is more important) </li><li>Make the description a clickable link </li><li>Drop the “URL/Download” column because it's now redundant </li><li>Rename “Description” to “Resource” (not that important) </li></ol> http://localhost/ticket/817 http://localhost/ticket/817 Report cygri <p> The semantics of the Author and Maintainer fields are really unclear at the moment. This leads to very inconsistent usage. Also, perhaps Name and Email are not the only fields that are needed for a contact. </p> <p> Here is a table that shows the current usage of these fields in CKAN: <a class="ext-link" href="http://richard.cyganiak.de/2010/ckan/ckan-ppl.html"><span class="icon">​</span>http://richard.cyganiak.de/2010/ckan/ckan-ppl.html</a> </p> <p> We note several problems: </p> <ul><li>Author and Maintainer are often the same </li><li>Author and Maintainer are often used interchangeably </li><li>People really want to specify URLs for the contacts and stick them into random places because there is no field for it </li><li>Multiple comma-separated names in a single field </li></ul><p> I'm not sure what to do about this, but a redesign is necessary in my opinion. </p> <p> Some ideas: </p> <ul><li>Remove the maintainer field? </li><li>Make really clear that Author doesn't refer to the metadata on CKAN, but to the original data </li><li>Add an “author URL” field? </li></ul> http://localhost/ticket/818 http://localhost/ticket/818 Report cygri <p> The widget for tag autocompletion is broken in various ways. </p> <p> For example, if I edit a package that is tagged "music", and just tab through the form fields to get to the Author field, then the widget changes to "industrial-music" as I tab through it. </p> <p> Or if the tag is "foo bar" and I hit alt-right to jump to the end of the text field (with the intent of adding a third tag), then the contents change to "foo barbecue". </p> http://localhost/ticket/819 http://localhost/ticket/819 Report dread <p> Should fail gracefully (not the existing 500 error) when you go to ckan.net/user/apikey (manually typing in the url) but haven't logged in. </p> <pre class="wiki">WebApp Error: &lt;type 'exceptions.AttributeError'&gt;: 'NoneType' object has no attribute 'apikey' URL: http://test-hmg.ckan.net/user/apikey </pre> http://localhost/ticket/820 http://localhost/ticket/820 Report dread <p> Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end. </p> <p> Here are the details: </p> <ul><li>in CKAN's package view (in web interface, API and dump) no resources display </li><li>but they WERE created in CKAN, as shown by the revision diffs. </li><li>they are in the Drupal front-end </li></ul><p> So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal. </p> <p> Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices </p> http://localhost/ticket/821 http://localhost/ticket/821 Report Javier de la Cueva <jdelacueva@…> <p> The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net </p> http://localhost/ticket/822 http://localhost/ticket/822 Report dread <ol><li>Create a package. </li><li>Edit package and add a resource. </li><li>View package in browser - no sign of the resource - ERROR. </li><li>View package in API - resource is there. </li></ol><p> Reason: hash doesn't change because package.revision only takes account of core fields. </p> http://localhost/ticket/823 http://localhost/ticket/823 Report dread <p> Data importers should use API instead of object model directly: </p> <ul><li>saves load on server </li><li>much better error checking and authentication </li><li>eating our own dogfood (tests the API well) </li></ul> http://localhost/ticket/824 http://localhost/ticket/824 Report dread http://localhost/ticket/825 http://localhost/ticket/825 Report dread <p> There are a number of extra properties we want to store in a resource: </p> <ul><li>resource size (in bytes, no. triples etc.) - <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673</a> </li><li>resource hash is 'static' (default) or 'changeable' - <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674</a> </li><li>alternative link (e.g. TNA Web Archive) to use the link is broken - <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173</a> </li></ul><p> i.e. this is a much needed enhancement </p> <p> Need to be able to read/write these values in web interface and API. </p> <p> (At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.) </p> <h1 id="Implementation">Implementation</h1> <p> I suggest these are arbitrary key/value pairs, just like package extras, reusing <a class="missing wiki">PackageExtra?</a> code as much as possible. (Other much more radical alternative is to move model to RDF.) </p> http://localhost/ticket/826 http://localhost/ticket/826 Report dread <p> Intermittent failure on ckan default branch. Looks like brittle test. </p> <pre class="wiki">====================================================================== FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/lib/python2.6/site-packages/nose/case.py", line 186, in runTest self.test(*self.arg) File "/home/buildslave/okfn/full/build/buildandsmoke/pyenv/src/ckan/ckan/tests/models/test_package.py", line 250, in test_02_metadata_created_and_modified assert out == exp, (out, exp) AssertionError: (datetime.datetime(2010, 11, 25, 4, 37, 52, 949380), datetime.datetime(2010, 11, 25, 4, 37, 52, 94938)) </pre><p> <a class="ext-link" href="http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio"><span class="icon">​</span>http://buildbot.okfn.org/builders/builder-ckan/builds/589/steps/shell/logs/stdio</a> </p> http://localhost/ticket/827 http://localhost/ticket/827 Report rgrp <p> Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...). </p> <p> This is especially frustrating in a development environment where is currently running into fact front page is being cached. </p> <p> Fix this. </p> http://localhost/ticket/828 http://localhost/ticket/828 Report dread <p> Browsing to the admin interface /admin (even logged in as a sysadmin) gives an exception. </p> http://localhost/ticket/829 http://localhost/ticket/829 Report rgrp <p> Already have the basic capacity using *_extras paths for templates and public directory (see <a class="ext-link" href="http://wiki.okfn.org/ckan/doc/theme"><span class="icon">​</span>http://wiki.okfn.org/ckan/doc/theme</a>). </p> <p> However, would be nice to have a proper 'theme' structure (e.g. would allow for theme switching) that also avoids directly polluting 'public'. Could be inspired by: <a class="ext-link" href="http://packages.python.org/Flask-Themes/"><span class="icon">​</span>http://packages.python.org/Flask-Themes/</a> </p> http://localhost/ticket/830 http://localhost/ticket/830 Report rgrp <p> Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc. </p> http://localhost/ticket/831 http://localhost/ticket/831 Report rgrp <p> Convert stats section to a ckan plugin. </p> <p> Stats need not be in core and making it a plugin would make it easier to extend and expand. </p> http://localhost/ticket/832 http://localhost/ticket/832 Report rgrp <p> Create an admin dashboard as /ckan-admin/ allowing for admin operations and overview. </p> <p> Possible features: </p> <ul><li>Purge revisions (or sets of revisions) and purge objects <a class="closed ticket" href="http://localhost/ticket/1076" title="enhancement: Improve revision and package purge system (closed: fixed)">#1076</a> </li><li>Set roles for users <a class="closed ticket" href="http://localhost/ticket/1075" title="enhancement: Administrative dashboard - Edit Authorization related to System object (closed: fixed)">#1075</a> </li><li>Put system into particular modes e.g. wiki mode (anyone can add, edit packages by default), data portal (only sysadmins or members of a special Editor group can create and edit packages) <ul><li>WONTFIX </li></ul></li><li>Overview of activity <ul><li>WONTFIX - already have revision log </li></ul></li></ul><p> Currently have an admin section using the formalchemy admin controller to provide basic editing of model objects. This can still be used but located at /admin/model/. </p> <p> <a class="ext-link" href="https://bitbucket.org/okfn/ckanext-admin"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-admin</a> </p> <h3 id="Tickets">Tickets</h3> <ul><li><a class="closed ticket" href="http://localhost/ticket/1031" title="enhancement: User lookup API (closed: fixed)">ticket:1031</a> - user autocomplete api in ckan </li></ul><h3 id="Notes">Notes</h3> <p> Here's putting into restricted mode (plus creating a dedicated authz group so that others can admin sysadmin simply through that group): </p> <pre class="wiki"># first remove permissions from roles # this is hacky but have to do it because we hardcode assignment of # role permissions on package on package create (see model/authz.py) paster roles deny editor edit paster roles deny editor create-authorization-group paster roles deny editor create-group paster roles deny editor create-package paster roles deny reader create-package # make superuser group # create authz group administrators / Administrators (if not exists) paster rights make agroup:administrators admin system </pre> http://localhost/ticket/833 http://localhost/ticket/833 Report Alexander <p> Hello. </p> <p> I've installed stable CKAN v1.1 from PyPI. </p> <p> I can't find any docs about using CKAN API in order to query packages. </p> <p> Query ./api/search/package?q=str works fine, but with extra parameters, such as limit, offset, fullinfo, order_by, search_notes, don't. Should I use new version for this? How can I perform this query via Ckanclient? Is it possible? </p> <p> Also I'm interested how to find open-licensed files? I tied URL ./api/search/package?q=str&amp;open_only=1&amp;downloadable_only=1 and Ckanclient: </p> <p> result = ckan.package_search('str', {'open_only': 1, 'downloadable_only': 1}) </p> <p> As result nothing found. </p> <p> Thanks. </p> http://localhost/ticket/834 http://localhost/ticket/834 Report pudo <p> As a sysadmin I want to take CKAN out of wiki-mode into normal mode. In normal mode visitor and logged in users cannot create or add packages. </p> <p> Suggest put this as an internal command in authz system and add to cli as paster command. </p> http://localhost/ticket/835 http://localhost/ticket/835 Report rgrp <p> A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html</a> </p> <p> Instead we should use the site_url config option where necessary. </p> <p> Cost: 1h </p> http://localhost/ticket/836 http://localhost/ticket/836 Report rgrp <p> Thread: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html</a> </p> <h3 id="Scenario1">Scenario 1</h3> <ol><li>User installs Refine and CKAN extension for refine </li><li>On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance) </li><li>They edit the dataset on Refine </li><li>On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package) </li></ol><p> NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required) </p> <h3 id="Scenario2">Scenario 2</h3> <ol><li>User visits a package on CKAN.net (or another CKAN instance) </li><li>There is a button on the page "View and edit this dataset in Google Refine" </li><li>Click button -- ask them if they have Google refine installed <ul><li>Yes: instructions for loading dataset into refine </li><li>No: load dataset in hosted version of google refine (we could run this) </li></ul></li><li>User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset. </li></ol> http://localhost/ticket/837 http://localhost/ticket/837 Report memespring <p> This is a meta ticket for a bunch of UI changes: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/839" title="enhancement: Package page redesign (closed: fixed)">#839</a> Package page redesign </li><li><a class="closed ticket" href="http://localhost/ticket/846" title="enhancement: merge style.css and ckan.css (closed: fixed)">#846</a> Merge css files </li><li><a class="closed ticket" href="http://localhost/ticket/850" title="enhancement: Show welcome message on first visit to package page (closed: fixed)">#850</a> Show welcome message </li><li><a class="closed ticket" href="http://localhost/ticket/863" title="enhancement: prompt users for missing package information (closed: wontfix)">#863</a> Prompt users to enter missing info </li><li><a class="closed ticket" href="http://localhost/ticket/864" title="enhancement: Search results UI changes (closed: fixed)">#864</a> Search results changes </li><li><a class="closed ticket" href="http://localhost/ticket/866" title="enhancement: Include download formats in search results (closed: fixed)">#866</a> Add download formats to search results </li></ul> http://localhost/ticket/838 http://localhost/ticket/838 Report anonymous <p> as per <a href="http://ckan.org/wiki/UIRedesignPackage">http://ckan.org/wiki/UIRedesignPackage</a> </p> http://localhost/ticket/839 http://localhost/ticket/839 Report dread <ul><li>Use config variable to switch ETags caching on or off. Consider joining in with 'cache_enabled'. </li><li>Default setting for (all) caching should be off. </li><li>Needs documenting in configuration.rst </li></ul> http://localhost/ticket/840 http://localhost/ticket/840 Report dread <p> Documentation article on caching / improving performance. (To complement configuration docs.) </p> <ul><li>Different sorts of cache - beaker style, etags, package_dict in search results(?) </li><li>How each one affects performance </li><li>How to turn them on/off and configure them </li><li>Is it possible to bypass each of them in the browser or with wget/curl? </li></ul> http://localhost/ticket/841 http://localhost/ticket/841 Report rgrp <p> An extension that provides a todo list feature on CKAN so that people can register and find things to do. </p> <p> Extension name: ckanext-todo </p> <h2 id="UserStory">User Story</h2> <h3 id="Packagepage">Package page</h3> <p> As a user I come to a package: </p> <ul><li>Have a todo count at that top that takes you down to the todo list (which may say nothing todo) </li><li>At the bottom is a section of the package display titled "<a class="missing wiki">ToDo?</a>" where I see a list of all toDos for the package most recent at the top <ul><li>If I am logged in <ul><li>See a form for "Add to do" at the top of the todo section and can add one straight away </li><li>I see a "now resolved" button next to each which goes green when you hover. </li></ul></li></ul></li></ul><blockquote> <blockquote> <blockquote> <blockquote> <p> When clicked the todo fades away. </p> </blockquote> </blockquote> </blockquote> </blockquote> <ul><li>Not logged in: I see a button that says "login to add todo" </li></ul><h3 id="Todolistpage">Todo list page</h3> <p> When a user comes to todo overview at /todo </p> <p> At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category. </p> <p> Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo. </p> <h2 id="Implementation">Implementation</h2> <p> The Todo form </p> <ul><li>One of the fields is category -&gt; autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed) </li><li>Add a description </li><li>Submit, the todo gets added via AJAX to the list at the top as the most recent todo </li></ul><p> Model: </p> <pre class="wiki">todo table id (autoincrement integer) package_id todo_category_id (required) description (required) created=NOW() resolved=null (unresolved) or a datetime (datetime of resolution) creator=user resolver=user todo_category table id name Prepopulate with: broken-resource-link, no-author, bad-format, add-description </pre><p> API at /api/2/todo </p> <ul><li>GET / POST / PUT ... </li></ul><p> /api/2/todo?package=package_id_or_name&amp;category=...&amp;resolved=0/1 </p> <ul><li>support limit (?) </li></ul><p> /api/2/todo/category -&gt; return list of todo categories </p> <ul><li>No GET / PUT / POST (these are auto-created by creation of todo) </li></ul><h2 id="OptionalExtrasWillnotbedoneatm">Optional Extras (Will not be done atm)</h2> <ul><li>Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ... </li></ul> http://localhost/ticket/842 http://localhost/ticket/842 Report wwaites <ul><li>perhaps we should show the openid as well to distinguish between users with the same name. </li></ul><ul><li>perhaps on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name. </li></ul><ul><li>also the list is much too long. can we make it work some other way? javascript is an option, but must be careful to fail gracefully when the browser does not support it. </li></ul> http://localhost/ticket/843 http://localhost/ticket/843 Report wwaites <p> ckan.net and ckan.org have no mx record and no listener on port 25. yet they send out mail. this means, when messages it sends bounce for whatever reason, the bounce messages stay in the queue on the remote mail server until they expire. for example: </p> <pre class="wiki">B30E6D9F 2873 Sat Dec 4 14:05:40 MAILER-DAEMON (delivery temporarily suspended: connect to ckan.net[46.51.189.76]:25: Operation timed out) no-reply@ckan.net </pre><p> both these domains should have an mx record pointing to mail.okfn.org and mail.okfn.org should be configured to handle these addresses correctly, e.g. by sending them to /dev/null </p> http://localhost/ticket/844 http://localhost/ticket/844 Report dread <p> Allow a form to specify fields that are required. Currently name and resource url are required. </p> <ul><li>Show required fields with asterisks by the input box. </li><li>Resource fields to be included in this. </li></ul> http://localhost/ticket/845 http://localhost/ticket/845 Report memespring <p> no need for separate files </p> http://localhost/ticket/846 http://localhost/ticket/846 Report rgrp <p> This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010). </p> <p> 3 key aspects of this: </p> <ul><li>Index package information in SOLR - <a class="closed ticket" href="http://localhost/ticket/353" title="defect: SOLR search indexing (closed: fixed)">ticket:353</a> - SOLR search indexing (index CKAN) <ul><li>Synchronously (generify Search Index update method to support SOLR or postgres?) - <a class="closed ticket" href="http://localhost/ticket/317" title="enhancement: Make search pluggable (closed: fixed)">#317</a>: Make search pluggable </li><li>Asynchroously: via worker with queue <ul><li><a class="closed ticket" href="http://localhost/ticket/324" title="enhancement: Search indexing using notifications (closed: fixed)">#324</a> Search indexing using notifications </li><li><a class="closed ticket" href="http://localhost/ticket/669" title="task: Refactor Solr to become a generic worker (closed: fixed)">#669</a>: Refactor Solr (indexer) to become a generic worker </li><li><a class="closed ticket" href="http://localhost/ticket/874" title="enhancement: Extract solr search backend into an extension (closed: fixed)">#874</a>: Extract Solr search backend into an extension </li></ul></li></ul></li><li>In core support using SOLR backend as well as postgres for queries <ul><li><a class="closed ticket" href="http://localhost/ticket/317" title="enhancement: Make search pluggable (closed: fixed)">#317</a>: Make search pluggable (?) </li></ul></li><li>Integrate into WUI (if changes needed) <ul><li><a class="closed ticket" href="http://localhost/ticket/672" title="enhancement: Facets in CKAN search results (closed: fixed)">#672</a> - Facets in CKAN search results </li></ul></li></ul><p> Extras: </p> <ul><li>Solr index testing tool <a class="closed ticket" href="http://localhost/ticket/431" title="task: Solr index testing tool (closed: wontfix)">#431</a> </li></ul> http://localhost/ticket/847 http://localhost/ticket/847 Report pudo <p> for helsinki regional infoshare. </p> http://localhost/ticket/848 http://localhost/ticket/848 Report pudo <p> The CSS @import directive at the bottom of the main CKAN css file is not understood in all browsers, move this to a seperate import. </p> http://localhost/ticket/849 http://localhost/ticket/849 Report memespring <p> Control using cookies. </p> http://localhost/ticket/850 http://localhost/ticket/850 Report wwaites <p> revisit link checker from <a class="ext-link" href="http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py"><span class="icon">​</span>http://knowledgeforge.net/ckan/ckanext/file/tip/ckanext/link_checker.py</a> </p> <p> revisit ollyc's parallel work </p> <p> look at how this gets used in practice (maybe deprecate in favour of curate tool (suitably wrapped to hide details from the user) </p> http://localhost/ticket/851 http://localhost/ticket/851 Report wwaites <p> 3 (4) use cases: (With and without access to auth keys.) </p> <ol><li>upload form on ckan (by default local system has auth key for storage) </li><li>datapkg upload (client) (by default without key) (plumbing done) </li><li>background job archiving downloads (by queue client) </li></ol><p> Most important are: </p> <ol><li>Upload form on ckan.net (with access to keys) </li><li>API for getting credentials for upload from ckan.net api (without access to keys). </li></ol><p> Aside: Once 2 was done we could deprecate storage.ckan.net </p> <h3 id="Sub-tickets">Sub-tickets</h3> <p> <strong> Required infrastructure </strong> </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/826" title="enhancement: Resource 'extra' fields (closed: fixed)">#826</a> - Resource 'extra' fields </li><li><a class="closed ticket" href="http://localhost/ticket/358" title="enhancement: Resources in REST API (closed: duplicate)">#358</a> - Resources in REST API </li><li><a class="assigned ticket" href="http://localhost/ticket/235" title="enhancement: Resource format normalization and detection (assigned)">#235</a> - Resource format normalization and detection </li></ul><p> <strong> User upload via Web UI </strong> </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/877" title="enhancement: File upload in WUI (extension) (closed: fixed)">#877</a> - File upload in WUI </li><li><a class="closed ticket" href="http://localhost/ticket/878" title="enhancement: Integrate file upload with workflow around package resources (closed: fixed)">#878</a> - Integrate file upload with workflow around package resources </li></ul><p> <strong> User upload via CLI/datapkg </strong> </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/853" title="enhancement: Client upload to storage without having primary storage keys (closed: fixed)">#853</a> - plumbing for credential-less rest upload </li><li><a class="closed ticket" href="http://localhost/ticket/879" title="enhancement: Storage auth API (closed: fixed)">#879</a> - Storage auth API </li></ul><p> Also related: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/405" title="task: Retrieval options for package resources (closed: fixed)">#405</a> - Retrieval options for package resources </li></ul><h3 id="MoreInfo">More Info</h3> <ul><li>Upload/storage discussion: <a class="ext-link" href="http://ckan.okfnpad.org/storage"><span class="icon">​</span>http://ckan.okfnpad.org/storage</a> <ul><li>For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/ </li></ul></li></ul> http://localhost/ticket/852 http://localhost/ticket/852 Report wwaites <p> Reverse engineer boto and work out how to get headers to support upload to google storage without holding api keys. </p> <p> This would lead to an extension to OFS. </p> <p> This analysis should inform (and go hand-in-hand) with the implementation of <a class="closed ticket" href="http://localhost/ticket/879" title="enhancement: Storage auth API (closed: fixed)">ticket:879</a> (Storage Auth API in CKAN). </p> <ul><li>master ticket <a class="closed ticket" href="http://localhost/ticket/852" title="enhancement: [super] Dataset upload and archiving (closed: fixed)">#852</a> </li></ul> http://localhost/ticket/853 http://localhost/ticket/853 Report dread <p> The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests. </p> http://localhost/ticket/854 http://localhost/ticket/854 Report dread <p> The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests. </p> http://localhost/ticket/855 http://localhost/ticket/855 Report dread <p> Test coverage for ckan.lib.cache is only 40%. Can that be usefully improved? </p> http://localhost/ticket/856 http://localhost/ticket/856 Report dread <p> Test coverage for ckan.lib.cli is 34%. Need to improve that. </p> http://localhost/ticket/857 http://localhost/ticket/857 Report dread <p> Test coverage for ckan.lib.diff is 24%. Need to improve </p> http://localhost/ticket/858 http://localhost/ticket/858 Report dread <p> Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement? </p> http://localhost/ticket/859 http://localhost/ticket/859 Report wwaites <p> ... </p> http://localhost/ticket/860 http://localhost/ticket/860 Report wwaites <p> <a class="ext-link" href="http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d"><span class="icon">​</span>http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d</a> </p> http://localhost/ticket/861 http://localhost/ticket/861 Report wwaites <p> handle various dgu extras, configure apache for autonegotiation, etc. </p> http://localhost/ticket/862 http://localhost/ticket/862 Report memespring <p> e.g. license, owner etc </p> http://localhost/ticket/863 http://localhost/ticket/863 Report memespring <p> as per <a href="http://ckan.org/wiki/UIRedesignSearch">http://ckan.org/wiki/UIRedesignSearch</a> </p> http://localhost/ticket/864 http://localhost/ticket/864 Report pudo <ul><li>Provide and include a "language.js" file. </li></ul> http://localhost/ticket/865 http://localhost/ticket/865 Report memespring <p> as above </p> http://localhost/ticket/866 http://localhost/ticket/866 Report dread <p> To be more pythonic, raise exceptions when ckanclient gets status which isn't 200. </p> http://localhost/ticket/867 http://localhost/ticket/867 Report thejimmyg <p> The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach. </p> <p> We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!) </p> <p> Some suggestions for achieving this include: </p> <ul><li>Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database </li><li>Not setting up and tearing down the database so frequently. </li></ul> http://localhost/ticket/868 http://localhost/ticket/868 Report rgrp <p> Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc). </p> <p> Allowing plugins to edit the config dictionary on load would allow this (and more!). </p> <p> Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?) </p> http://localhost/ticket/869 http://localhost/ticket/869 Report memespring <p> Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly. </p> <p> Specifically if Home / About or Add Package / Search are selected both menu items are highlighted. </p> <p> This is because the code assumes only one top-level menu item per controller. </p> http://localhost/ticket/870 http://localhost/ticket/870 Report nils.toedtmann <p> The infamous <a class="ext-link" href="http://www.exim.org/lurker/message/20101207.215955.bb32d4f2.en.html"><span class="icon">​</span>exim bug</a> only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers. </p> <p> We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not. </p> <p> BTW did we already run a rootkit checker like <a class="ext-link" href="http://rkhunter.sourceforge.net/"><span class="icon">​</span>Rootkit hunter</a> on eu1? If not we should maybe do it now - there was already an exploit out in the wild. <a class="missing wiki">ByteMark?</a> has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim&lt;4.70 (our EHLO banner contains the exim version), just as anyone could. </p> http://localhost/ticket/871 http://localhost/ticket/871 Report dread <p> Solutions: <a class="missing wiki">ResourceSeriesLoader?</a> allows synonyms for values. </p> http://localhost/ticket/872 http://localhost/ticket/872 Report dread <p> e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_. </p> http://localhost/ticket/873 http://localhost/ticket/873 Report pudo <p> Move the solr <a class="missing wiki">SearchBackend?</a> to a seperate code base, remove any mentions of the CKAN model. </p> http://localhost/ticket/874 http://localhost/ticket/874 Report rgrp <p> Consolidate search API interface (and backend) on solr (solrpy) type interface. </p> <ul><li>Support for standard query structure </li><li>Support for facet options </li></ul><p> Do <strong>not</strong> need to change response formats. (Or do we?) </p> <p> 2 options here for advanced features like facets in non-solr: </p> <ol><li>Disable (happens automatically) </li><li>Implement - suggest using group by etc </li></ol><h2 id="Extras">Extras</h2> <ul><li>Front-page tag cloud: change this to use facets <ul><li>Accept this means that if facets not functional in backend we have no tag cloud </li></ul></li></ul> http://localhost/ticket/875 http://localhost/ticket/875 Report rgrp <p> Among other things this will allow the tests to run much quicker. </p> http://localhost/ticket/876 http://localhost/ticket/876 Report rgrp <p> Support uploading files to a storage system from WUI with a simple upload form. </p> <h3 id="Implementation">Implementation</h3> <ul><li>Implement as an <a class="wiki" href="http://localhost/wiki/Extensions">Extension</a> </li><li>Locate at /upload </li><li>Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users </li><li>Use OFS + boto -&gt; generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists) </li><li>This wants to be tied in to general package workflow. See parent <a class="closed ticket" href="http://localhost/ticket/852" title="enhancement: [super] Dataset upload and archiving (closed: fixed)">ticket:852</a>. </li></ul> http://localhost/ticket/877 http://localhost/ticket/877 Report rgrp <p> Create/edit a package -&gt; add a resource -&gt; Oh, i haven't uploaded yet, 'upload here' -&gt; /upload -&gt; standard upload form </p> <p> Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata? </p> <p> If yes suggests (need to ticket): </p> <ul><li>'<a class="missing wiki">ResourceExtras?</a>' for extra metadata on resources </li><li>Makes 'Resource' into first class object (and make available in WUI)? </li></ul><h2 id="Depends">Depends</h2> <ul><li><a class="closed ticket" href="http://localhost/ticket/877" title="enhancement: File upload in WUI (extension) (closed: fixed)">#877</a> (Upload in WUI) - should be part of that extension </li></ul> http://localhost/ticket/878 http://localhost/ticket/878 Report rgrp <p> API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage. </p> <p> Implement as a CKAN extension. </p> http://localhost/ticket/879 http://localhost/ticket/879 Report dread <p> Deleted packages with the same title and extra fields as an active package trip up the loader. It finds them in the search, but due to ckanclient not using the apikey when you 'get' the package to examine it further, it causes an exception. </p> http://localhost/ticket/880 http://localhost/ticket/880 Report fccoelho <p> I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code. </p> http://localhost/ticket/881 http://localhost/ticket/881 Report pudo http://localhost/ticket/882 http://localhost/ticket/882 Report wwaites <h2 id="masterticket">master ticket</h2> <ul><li><a class="closed ticket" href="http://localhost/ticket/884" title="enhancement: harvest job delete operation (closed: fixed)">#884</a> implement harvest job delete command for an easier life </li><li><a class="closed ticket" href="http://localhost/ticket/885" title="defect: reimplement csw module to use owslib (closed: fixed)">#885</a> move csw code into dedicated ckanext plugin </li><li><a class="closed ticket" href="http://localhost/ticket/886" title="defect: move uklii forms api to dgu module (closed: duplicate)">#886</a> move forms api into dgu extension module </li><li><a class="closed ticket" href="http://localhost/ticket/887" title="enhancement: move harvesting to generic module (closed: fixed)">#887</a> move harvesting code to use generic harvesting module </li></ul><p> </p> http://localhost/ticket/883 http://localhost/ticket/883 Report wwaites <p> implement paster subcommand to delete harvest jobs </p> http://localhost/ticket/884 http://localhost/ticket/884 Report wwaites http://localhost/ticket/885 http://localhost/ticket/885 Report wwaites <p> doesn't belong in ckan core </p> http://localhost/ticket/886 http://localhost/ticket/886 Report wwaites <p> in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat </p> http://localhost/ticket/887 http://localhost/ticket/887 Report rgrp <p> First version of dataproxy and data API working (<a class="closed ticket" href="http://localhost/ticket/698" title="task: CKAN Data API v1 (closed: fixed)">ticket:698</a>) but have identified a variety of important improvements. (Should split these into sub-tickets ...): </p> <p> For dataproxy: </p> <ul><li>Testing for dataproxy <ul><li>Can start by using known good remote urls (moving forward could switch to providing/mocking these locally) </li></ul></li><li>Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum) </li><li>Google docs style row/column selections </li><li>Use the swiss library - <a class="ext-link" href="https://bitbucket.org/okfn/swiss"><span class="icon">​</span>https://bitbucket.org/okfn/swiss</a> <ul><li>Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet) </li></ul></li><li>Handle redirects for content-length? </li><li>Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?) </li></ul><p> For dataapi: </p> <ul><li>Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -&gt; {dataproxy}?url={resource-url}&amp;type={resource-type} </li></ul> http://localhost/ticket/888 http://localhost/ticket/888 Report rgrp <p> Add a config option 'template_footer_end' which is inserted in layout_template just before &lt;/body&gt;. </p> <p> This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics. </p> <p> Aside: going forward may want to turn this into a extension. </p> <p> Cost: 1h </p> http://localhost/ticket/889 http://localhost/ticket/889 Report pudo <p> The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows: </p> <ul><li>routing_key: Package </li><li>operation: daily </li><li>payload: pkg.as_dict() </li></ul><p> Where operation is one of daily, weekly, monthly or any other interval. </p> http://localhost/ticket/890 http://localhost/ticket/890 Report pudo <p> Superticket: <a class="closed ticket" href="http://localhost/ticket/1397" title="enhancement: [super] Resource archiving (closed: fixed)">#1397</a> </p> <p> Write a worker daemon to download all resources from a CKAN instance to a local repository. </p> <h2 id="Questions">Questions</h2> <ul><li>Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues) </li><li>Should we have clever ways to dump APIs? ANS: no. </li><li>Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving) </li><li>Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache. <ul><li>Complete support for ETags </li><li>Expires, Max-Age etc. </li></ul></li><li>Check </li></ul><h2 id="Functionality">Functionality</h2> <ul><li>Download files via HTTP, HTTPS (will not do FTP) </li></ul><p> Process: </p> <ol><li>[Archiver.Update checks queue (automated as part of celery)] </li><li>Open url and get any info from resource on cache / content-length etc <ol><li>If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery </li><li>Check headers for content-length and content-type ... <ul><li>IF: content-length &gt; max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?) </li><li>ELSE: check content-type. <ul><li>IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource) </li><li>ELSE: archive it (compute md5 hash etc) </li></ul></li><li>IF: get content-length and content-length unchanged GOTO step 4 </li></ul></li></ol></li><li>Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext <ul><li>Add cache url to resource and updated date </li><li>Add other relevant info to resource such as md5, content-type etc </li></ul></li><li>Update task_status </li></ol><p> </p> <h2 id="Optionalfunctionality">Optional functionality</h2> <ul><li>If result object is HTML, search for references to "proper data" (CSV download pages etc.) </li><li>Download from POST forms (accepting licenses or weird proprietary systems) </li><li>Support running on Google Apps Engine to save traffic costs. </li></ul><h2 id="Existingwork">Existing work</h2> <ul><li><a class="ext-link" href="https://bitbucket.org/okfn/ckanext-qa/overview"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-qa/overview</a> </li><li>out of date: <a class="ext-link" href="https://bitbucket.org/pudo/ckanextarchive"><span class="icon">​</span>https://bitbucket.org/pudo/ckanextarchive</a> - Old archiver extension, largely experimental. </li><li>out of date: <a class="ext-link" href="https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65"><span class="icon">​</span>https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65</a> - Openness scores by ollyc </li></ul> http://localhost/ticket/891 http://localhost/ticket/891 Report pudo <p> Once we have storage, make the data available in the following ways: </p> <ul><li>Now have a cached_url field can show in the frontend ... </li><li>Add a [&lt;a href="${cached_url}"&gt;cached&lt;/a&gt;] link to right of real url on resource listing on dataset page. </li><li>On resource page: will not add it yet. <ul><li>At the moment no clear place to pu this given nice big download button (could put in list of items on left but that does not seem right and note that it will turn up in big list of info at bottom) </li></ul></li><li>Add test (?) </li><li>Deploy </li></ul> http://localhost/ticket/892 http://localhost/ticket/892 Report pudo <p> At the moment, <a class="missing wiki">ExtrasField?</a> cannot be used in conjunction with more specific extra fields, such as <a class="missing wiki">TextExtraField?</a> or <a class="missing wiki">SuggestTextExtraField?</a>. </p> http://localhost/ticket/893 http://localhost/ticket/893 Report wwaites http://localhost/ticket/894 http://localhost/ticket/894 Report memespring <p> Updates to css after a new deploy don't come through without a hard refresh. Adding the version number to the include urls will solve this e.g. </p> <p> mycssfile.css?v=12345678 </p> http://localhost/ticket/895 http://localhost/ticket/895 Report thejimmyg <p> Based on the ideas in: </p> <ul><li><a class="wiki" href="http://localhost/wiki/DistributingChanges">DistributingChanges</a> </li><li><a class="wiki" href="http://localhost/wiki/SyncingInstances">SyncingInstances</a> </li></ul><p> Merging the tickets <a class="closed ticket" href="http://localhost/ticket/296" title="enhancement: Commit CKAN revisions to changeset system (closed: duplicate)">#296</a>, <a class="closed ticket" href="http://localhost/ticket/297" title="enhancement: Update CKAN repository from changeset system (closed: duplicate)">#297</a>, <a class="closed ticket" href="http://localhost/ticket/298" title="enhancement: Pull changesets from remote CKAN instance (closed: duplicate)">#298</a>, <a class="closed ticket" href="http://localhost/ticket/299" title="enhancement: Merge diverging lines of changesets (closed: duplicate)">#299</a> into this one. </p> <p> Related ideas include: </p> <ul><li>Using dcat to exchange data </li><li>Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance </li></ul><p> This should be considered a wishlist item for the timebeing. </p> http://localhost/ticket/896 http://localhost/ticket/896 Report dread <p> On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted. </p> http://localhost/ticket/897 http://localhost/ticket/897 Report dread <p> Every change to every object is being stored in memory, which could add up to quite a lot of memory. </p> <p> This fixes it by making sure the objects are in a weakref. <a class="ext-link" href="https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f"><span class="icon">​</span>https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f</a> </p> http://localhost/ticket/898 http://localhost/ticket/898 Report dread <p> There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. <a class="ext-link" href="https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e"><span class="icon">​</span>https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e</a> </p> <p> From David Raznick </p> http://localhost/ticket/899 http://localhost/ticket/899 Report dread <p> There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. <a class="ext-link" href="https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e"><span class="icon">​</span>https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e</a>. </p> <p> From David Raznick </p> http://localhost/ticket/900 http://localhost/ticket/900 Report dread http://localhost/ticket/901 http://localhost/ticket/901 Report dread <p> On a package page there are three buttons to the right - View, Edit, History. Each of these buttons contain an icon and the text, but you can only activate the button by clicking on the icon. When you click on the part of the button where the text is (about 50% of the button) then nothing happens. </p> <p> Using: Chrome, Firefox </p> http://localhost/ticket/902 http://localhost/ticket/902 Report pudo <p> Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr. </p> http://localhost/ticket/903 http://localhost/ticket/903 Report Stiivi <p> What's bad at the moment? </p> <ul><li>lack of documentation e.g. config (very poorly documented) </li><li>too many sources of documentation </li><li>no common theming </li></ul><p> Sources: </p> <ul><li><a href="http://ckan.org/">http://ckan.org/</a> </li><li><a class="ext-link" href="http://packages.python.org/ckan"><span class="icon">​</span>http://packages.python.org/ckan</a> </li><li><a class="ext-link" href="http://wiki.okfn.org/ckan/"><span class="icon">​</span>http://wiki.okfn.org/ckan/</a> - user </li><li>Deprecate: <a class="ext-link" href="http://knowledgeforge.net/ckan/doc/ckan/"><span class="icon">​</span>http://knowledgeforge.net/ckan/doc/ckan/</a> </li></ul><p> Resulting meta-ticket with things to do: <a class="closed ticket" href="http://localhost/ticket/927" title="enhancement: [super] Improvements to CKAN documentation (closed: fixed)">ticket:927</a> </p> http://localhost/ticket/904 http://localhost/ticket/904 Report Stiivi <p> I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results. </p> <p> CKAN setup: </p> <ul><li>psql DB with UTF-8 tables </li></ul><ul><li>psql search backend </li></ul><ul><li>ckan version: 1.3.1b </li></ul><p> Example: </p> <p> <a class="ext-link" href="http://verejnedata.sk/package/agenda-cestovnch-pasov"><span class="icon">​</span>http://verejnedata.sk/package/agenda-cestovnch-pasov</a> </p> <p> try to search for the first word in the package name. </p> http://localhost/ticket/905 http://localhost/ticket/905 Report Stiivi <p> In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results. </p> <p> This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents. </p> <p> For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCII<em>TRANSLIT). </em></p> http://localhost/ticket/906 http://localhost/ticket/906 Report rgrp <p> As a user I want to preview a dataset ('resource') before downloading it. </p> <h2 id="Implementation">Implementation</h2> <p> This will be implemented as an <a class="wiki" href="http://localhost/wiki/Extensions">Extension</a> </p> <p> NB: This ticket requires a way to get data out of a remote url (the dataset). We do <strong>not want to put this data load/streaming into ckan core</strong> due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file) </p> <ol><li>Need a 'data proxy' - see <a class="closed ticket" href="http://localhost/ticket/698" title="task: CKAN Data API v1 (closed: fixed)">ticket:698</a> and <a class="closed ticket" href="http://localhost/ticket/888" title="enhancement: Improvements to the dataproxy and the data API (closed: fixed)">ticket:888</a> </li><li>Preview using javascript in WUI - do this in an extension <a class="ext-link" href="http://bitbucket.org/okfn/ckanext-dataapi"><span class="icon">​</span>http://bitbucket.org/okfn/ckanext-dataapi</a> </li></ol> http://localhost/ticket/907 http://localhost/ticket/907 Report rgrp <p> Some extensions import pylons features (such as request) at root. </p> <p> Have encountered following issues: </p> <ol><li>You need to import ckanext before any ckan stuff in your wsgi script (extension stats and dataapi) </li></ol><ol start="2"><li>Running under wsgi daemon mode (normal mode is fine!) the front page (front page only) will not work (does not occur with disqus or synchronous_search but does with stats and dataapi extensions). You get errors like this in the log: </li></ol><pre class="wiki">[Mon Jan 17 09:01:19 2011] [error] Exception KeyError: KeyError(-1218594160,) in &lt;module 'threading' from '/usr/lib/python2.6/threading.pyc'&gt; ignored [Mon Jan 17 09:01:20 2011] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in &lt;bound method PluginEnvironment.__del__ of Services for Environment '&lt;default&gt;' </pre> http://localhost/ticket/908 http://localhost/ticket/908 Report pudo <p> Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported: </p> <ul><li>CKANrdf generated exports </li><li>opengov.se RDF (not really DCat) </li><li>Sunlight Nationdatacatalog as harvested by dcat-tools. </li></ul> http://localhost/ticket/909 http://localhost/ticket/909 Report pudo <p> Extra fields should be represented in some manner. </p> http://localhost/ticket/910 http://localhost/ticket/910 Report pudo http://localhost/ticket/911 http://localhost/ticket/911 Report pudo <ul><li>Should license go in the bottom line? </li><li>Formats should be styled as in CKAN </li></ul> http://localhost/ticket/912 http://localhost/ticket/912 Report pudo <ul><li>Create a list of all valid predicates where licenses are stored. </li><li>Create a license normalization registry in GSpread </li><li>Fill spreadsheet from OKD/OSI lists. </li></ul> http://localhost/ticket/913 http://localhost/ticket/913 Report pudo <ul><li>Create a list of valid predicates for format information (accessURL et al) </li><li>Execute <a class="assigned ticket" href="http://localhost/ticket/235" title="enhancement: Resource format normalization and detection (assigned)">#235</a> via Google Spreadsheets and include in DCat enrichment </li></ul> http://localhost/ticket/914 http://localhost/ticket/914 Report pudo <p> We should present something there, why not start with a meta-search. </p> http://localhost/ticket/915 http://localhost/ticket/915 Report pudo <p> We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG. </p> http://localhost/ticket/916 http://localhost/ticket/916 Report kindly <p> David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior. </p> <p> I have a patch in <a class="ext-link" href="https://bitbucket.org/kindly/ckan/"><span class="icon">​</span>https://bitbucket.org/kindly/ckan/</a> on branch feature_switch_autoflush_off_by_default. </p> <p> This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet). </p> <p> Things to know about using flush: </p> <ul><li>Most of the time nothing will change. A commit implies flush. </li><li>You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query. </li><li>A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated. </li><li>You can use them as a limited form of nested transaction. </li></ul><p> It increased the performance on the tests by about 5%. There are less flushes altogether so is faster. </p> <p> I do not see any downsides. </p> http://localhost/ticket/917 http://localhost/ticket/917 Report dread <ol><li>Edit any package </li><li>Hit 'preview' </li></ol><p> Notice that the preview at the bottom doesn't show what tags or groups the package is in. </p> <p> This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview. </p> <p> One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision. </p> http://localhost/ticket/918 http://localhost/ticket/918 Report dread <ol><li>Edit a package </li><li>Hit 'preview'. </li></ol><p> The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too? </p> http://localhost/ticket/919 http://localhost/ticket/919 Report hellmann@… <p> <a class="ext-link" href="http://ckan.net/tag/linguistsic"><span class="icon">​</span>http://ckan.net/tag/linguistsic</a> doesn't have a single entry, but is still kept and displayed. </p> <p> It should be purged !!! </p> http://localhost/ticket/920 http://localhost/ticket/920 Report thejimmyg <p> Refactor harvesting code to simplify. </p> http://localhost/ticket/921 http://localhost/ticket/921 Report kindly <p> Why? </p> <ul><li>Will make it easier to give resources their own form, independent of packages. </li><li>Makes uploading, harvesting and previewing data cleaner as its separate from the packages. </li><li>Gives the option of sharing a resource between packages. </li><li>Leads the way to supporting storing richer semantic information against them. </li></ul> http://localhost/ticket/922 http://localhost/ticket/922 Report dread <ol><li>Go to: <a class="ext-link" href="http://ckan.net/stats/leaderboard#content"><span class="icon">​</span>http://ckan.net/stats/leaderboard#content</a> </li><li>In the far top-right of the browser, select the search box in 'water'. </li><li>Press enter to search. Nothing happens. </li></ol><p> Tried in: chrome, firefox </p> http://localhost/ticket/923 http://localhost/ticket/923 Report dread <p> The search box at the top-right of CKAN's page doesn't have a 'go' button. I feel that a larger percentage of users expect a 'go' or 'search' button on the right-hand side of the box to press to start searching. Techies tend to know the keyboard shortcut of pressing 'carriage-return' but it might be better to follow standard practise on this. </p> <p> Examples with 'search' button: Internet Explorer, Firefox, Google, Amazon, trac Examples without: ? </p> http://localhost/ticket/924 http://localhost/ticket/924 Report dread <p> Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed? </p> http://localhost/ticket/925 http://localhost/ticket/925 Report sebbacon <p> The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise. </p> <p> Review existing (and potentially non-existing) frameworks with a view to porting forms over to something more explicit and lightweight. </p> <p> Implement the current Package forms as an example of how this would work. </p> <p> Document and circulate. </p> <p> Sub-ticket of <a class="closed ticket" href="http://localhost/ticket/961" title="enhancement: [super] Refactoring of forms, validation and model synchronization (closed: fixed)">#961</a> (form, validation, model sync meta-ticket) </p> http://localhost/ticket/926 http://localhost/ticket/926 Report rgrp <p> List of improvements to CKAN documentation (result of <a class="closed ticket" href="http://localhost/ticket/904" title="task: Review CKAN documentation (closed: fixed)">ticket:904</a>). Should convert most items into sub-tickets as we go along. </p> <h2 id="Major">Major</h2> <ul><li><del>(Make blog.ckan.org into ckan.org and move ckan.org to trac.ckan.org/)</del> </li><li>Document caching system - <a class="closed ticket" href="http://localhost/ticket/841" title="enhancement: Caching docs (as a whole) (closed: duplicate)">ticket:841</a> </li><li>Howto guide for writing an extension - <em>started on this at <a class="ext-link" href="http://wiki.ckan.net/Extending_CKAN"><span class="icon">​</span>http://wiki.ckan.net/Extending_CKAN</a></em> </li></ul><h2 id="Miscellaneous">Miscellaneous</h2> <ul><li>Contributors / credits page: <a class="closed ticket" href="http://localhost/ticket/928" title="enhancement: Documentation - CKAN credits / contributors page (closed: fixed)">ticket:928</a> </li><li><del>(Data proxy documentation: move it to packages.python.org/dataproxy)</del> </li><li><del>(List dataapi extensions on <a href="http://ckan.org/wiki/Extensions">http://ckan.org/wiki/Extensions</a> (Name, summary string + point to pypi page ...))</del> </li><li>Document license configuration and license system - <a class="closed ticket" href="http://localhost/ticket/973" title="enhancement: Document license configuration and license system. (closed: fixed)">ticket:973</a> </li><li>Refactor trac frontpage moving general feature overview to ckan.org </li><li>Create proper roadmap information on <a class="ext-link" href="http://trac.ckan.org/"><span class="icon">​</span>http://trac.ckan.org/</a> </li><li>Add link to wiki.ckan.net to ckan.org </li></ul> http://localhost/ticket/927 http://localhost/ticket/927 Report rgrp <ol><li>Remove contributors list from about page (out-of-date and does not belong there) <ul><li>A "Powered by CKAN" link in footer is enough </li></ul></li><li>Add contributors / team / credits page to ckan.org <ul><li>May want to separate credits (e.g. software use), from contributors (also is team different from contributors) </li><li>(perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket) </li></ul></li><li>? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info. </li></ol><p> Cost: 1h </p> http://localhost/ticket/928 http://localhost/ticket/928 Report rene.kapusta <p> I'm using v 1.3.1a and get a lot of "<a class="missing wiki">WebApp?</a> Error: &lt;type 'exceptions.Exception'&gt;: Couldn't connect to licenses service: &lt;urlopen error (104, 'Connection reset by peer')&gt;" error messages. </p> <pre class="wiki">Module ckan.forms.package:87 in build_package_form &lt;&lt; # Options/settings builder.set_field_option('name', 'validate', package_name_validator) builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()}) builder.set_field_option('state', 'dropdown', {'options':model.State.all}) builder.set_field_option('notes', 'textarea', {'size':'60x15'}) &gt;&gt; builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()}) Module ckan.model.package:283 in get_license_options &lt;&lt; @classmethod def get_license_options(self): register = self.get_license_register() return [(l.title, l.id) for l in register.values()] &gt;&gt; register = self.get_license_register() Module ckan.model.package:278 in get_license_register &lt;&lt; def get_license_register(self): if not hasattr(self, '_license_register'): self._license_register = LicenseRegister() return self._license_register &gt;&gt; self._license_register = LicenseRegister() Module ckan.model.license:46 in __init__ &lt;&lt; from licenses.service import LicensesService2 self.service = LicensesService2(group_url) entity_list = self.service.get_licenses() else: from licenses import Licenses &gt;&gt; entity_list = self.service.get_licenses() Module licenses.service:44 in get_licenses &lt;&lt; except Exception, inst: msg = "Couldn't connect to licenses service: %s" % inst raise Exception, msg try: licenses = loads(response_body) &gt;&gt; raise Exception, msg Exception: Couldn't connect to licenses service: &lt;urlopen error (104, 'Connection reset by peer')&gt; </pre> http://localhost/ticket/929 http://localhost/ticket/929 Report dread <p> On a production server we produce literally millions of little files saying how long every single request took to process. </p> <p> They are here: </p> <pre class="wiki">hmg.ckan.net.2/pylonsdata/call_timing </pre><p> They were added by ckan/lib/base.py in cset:da438a9085d3. </p> <p> I don't believe anyone uses these stats. </p> http://localhost/ticket/930 http://localhost/ticket/930 Report dread <p> ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically. </p> http://localhost/ticket/931 http://localhost/ticket/931 Report dread <p> Basic steps from kindly: </p> <ul><li>changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions. </li><li>running the migrate script on the repository (or changing the repository scripts manually). </li><li>checking in both these changes at once. </li></ul> http://localhost/ticket/932 http://localhost/ticket/932 Report kindly <p> use pylint </p> http://localhost/ticket/933 http://localhost/ticket/933 Report kindly <p> We need to allow plugins to store information. </p> <p> Each plug-in should have its own name space and allow any data to be set. </p> <p> There are two options. </p> <ul><li>make a key value table in ckan that will hold random information. </li><li>use redis (or other key value store) </li></ul><p> If in sql suggest schema of form: </p> <ul><li>(namespace, obj_id, key, value [, value_type]) </li><li>value is json </li></ul> http://localhost/ticket/934 http://localhost/ticket/934 Report pudo <p> An internal server error was reported when trying to create a new package (not when editing): </p> <blockquote> <p> <a class="missing wiki">WebApp?</a> Error: &lt;class 'sqlalchemy.exceptions.<a class="missing wiki">IntegrityError?</a>'&gt;: (<a class="missing wiki">IntegrityError?</a>) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'} </p> </blockquote> <p> The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server. </p> <p> Possible causes include: </p> <ul><li>Stale pyc file </li><li>Threading issue </li><li>Dependency issue </li></ul><p> As the problem is gone for now, it cannot be further investigated atm. </p> http://localhost/ticket/935 http://localhost/ticket/935 Report rgrp <p> As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis). </p> <p> NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'. </p> <p> Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). <strong>Decision: use follow</strong> </p> <h2 id="Implementation">Implementation</h2> <h3 id="Interface">Interface</h3> <p> Become a follower: </p> <ul><li>Follow button on packages (if already watching say 'unfollow') </li></ul><p> Package-related changes: </p> <ul><li>Show number of followers on a package </li><li>List followers of a package at /package/{name}/followers <ul><li>On a separate page </li></ul></li></ul><p> a package User-related changes: </p> <ul><li>List followed packages <ul><li>Either on user's page on a separate 'following' page. (NB: called 'following') </li></ul></li><li>Does watching involve notifications (by email) <ul><li>Probably not: you can already subscribe to RSS feed after all and email not that necessary (?) </li></ul></li><li>[Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page </li></ul><h3 id="Nitty-Gritty">Nitty-Gritty</h3> <ul><li>Want to do this in ajax-y manner </li><li>API endpoint: /api/2/follower </li><li>Store data in a new follower table </li></ul><h4 id="API">API</h4> <p> /api/2/follower </p> <pre class="wiki">follow =&gt; PUT / POST { user_id object_type object_id } </pre><p> If this is submitted by a user with user.id != user_id =&gt; error (401) </p> <pre class="wiki">unfollow =&gt; DELETE /api/2/follower/package/{id} =&gt; list of followers [ { safe dictized user } ] </pre><p> NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method. </p> <h4 id="Table">Table</h4> <p> Called 'follower' </p> <pre class="wiki">user_id, table, object_id, created xxx, package, yyy, ... xxx, user, yyy, ... [future] </pre><p> </p> <h3 id="RandomExtras">Random Extras</h3> <ul><li>What about following users as well </li></ul> http://localhost/ticket/936 http://localhost/ticket/936 Report rgrp <p> As a User (especially as a Package <a class="missing wiki">Owner/Maintainer?</a>) I want to know how many times a resource has been downloaded (and when). </p> <p> So let's record download stats (as in clicks on the link for a resource). </p> <h2 id="Implementation">Implementation</h2> <ul><li>Use the existing support for this feature in google analytics or piwiki </li><li>For google analytics see: <ul><li>Integrate to record: <a class="ext-link" href="http://www.google.com/support/analytics/bin/answer.py?answer=55529"><span class="icon">​</span>http://www.google.com/support/analytics/bin/answer.py?answer=55529</a> </li><li>Accessing analytics data via API (for showing download counts): <a class="ext-link" href="http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html"><span class="icon">​</span>http://code.google.com/apis/analytics/docs/gdata/gdataDeveloperGuide.html</a> </li></ul></li><li>Limitations only record downloads by browsers with js turned on. <ul><li>This is OK I think (in any case machine download via e.g. datapkg does not get downloaded) </li></ul></li></ul><h3 id="OldSpecdoitinCKAN">Old Spec (do it in CKAN)</h3> <ul><li>Record info of form: resource id (or url?), timestamp </li><li>Do this via javascript capturing of onclick event talking to an api </li><li>API: /api/resource/{id}/download <ul><li>POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js) </li><li>GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] } </li></ul></li></ul><h2 id="Questions">Questions</h2> <ul><li>Do we record ip addresses (to handle de-botting etc)? </li><li>Do we count preview clicks as well? </li></ul> http://localhost/ticket/937 http://localhost/ticket/937 Report rgrp <p> Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see <a class="ext-link" href="http://flask.pocoo.org/docs/patterns/flashing/"><span class="icon">​</span>http://flask.pocoo.org/docs/patterns/flashing/</a>). </p> <p> We already do some of this (slightly poorly), e.g. on adding an authorization the app says: "X authorization added". </p> <h2 id="Implementation">Implementation</h2> <ul><li>Standardize this process and add section to html for this purpose. </li><li>Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit </li><li>Testing: suggest keeping this very simple </li></ul><h2 id="Estimate">Estimate</h2> <ul><li>Cost: 0.5h (html etc) + 1h to do some integration </li></ul> http://localhost/ticket/938 http://localhost/ticket/938 Report rgrp <p> On <a class="ext-link" href="http://getthedata.org/"><span class="icon">​</span>http://getthedata.org/</a> or stackoverflow there is a pop-up bar for new users that give them some simple instructions. </p> <p> Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better. </p> <h2 id="Implementation">Implementation</h2> <ul><li>Pop-up bar (based on a cookie or just being not logged in?) - uses <a class="closed ticket" href="http://localhost/ticket/938" title="enhancement: Message flashing / notification in WUI (closed: fixed)">ticket:938</a> (message flashing) </li><li>FAQ/instructions page (use the new wiki?) </li></ul><h2 id="Depends">Depends</h2> <ul><li><a class="closed ticket" href="http://localhost/ticket/938" title="enhancement: Message flashing / notification in WUI (closed: fixed)">ticket:938</a> </li></ul><h2 id="Estimate">Estimate</h2> <ul><li>Cost: 1h </li></ul> http://localhost/ticket/939 http://localhost/ticket/939 Report rgrp <p> This has been noted twice in the last couple of weeks. </p> <p> Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See: </p> <ul><li><a class="ext-link" href="http://groups.google.com/group/google-federated-login-api/web/the-most-important-technical-issue-in-using-the-google-accounts-api?pli=1"><span class="icon">​</span>http://groups.google.com/group/google-federated-login-api/web/the-most-important-technical-issue-in-using-the-google-accounts-api?pli=1</a> </li><li><a class="ext-link" href="http://stackoverflow.com/questions/3549953/how-to-get-unique-identifier-from-openid-google"><span class="icon">​</span>http://stackoverflow.com/questions/3549953/how-to-get-unique-identifier-from-openid-google</a> </li></ul><p> To fix this we should set the openid.realm (not sure how atm). </p> <p> If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache). </p> <p> Cost: 1h </p> http://localhost/ticket/940 http://localhost/ticket/940 Report rgrp <p> The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted. </p> http://localhost/ticket/941 http://localhost/ticket/941 Report pudo <p> It only passes "q", but the whole query must be serialized. </p> http://localhost/ticket/942 http://localhost/ticket/942 Report pudo <p> Set up a Mediawiki to contain info on CKAN.net (community) conventions etc. </p> http://localhost/ticket/943 http://localhost/ticket/943 Report pudo <ul><li>Move code base, fix any breakage. </li><li>Deploy to eu8 </li><li>Test </li><li>Deploy to eu14 </li></ul><h2 id="Why">Why</h2> <p> Later CKAN codebase has a bunch of improvements / fixes that it would be good to have. </p> <h2 id="Estimate">Estimate</h2> <ul><li>8h </li></ul> http://localhost/ticket/944 http://localhost/ticket/944 Report kindly <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1032" title="enhancement: [super] Resources in WUI (closed: fixed)">#1032</a> </p> <p> This is a meta ticket for changes that are going to happen in resources. </p> <ul><li>New resource group table. <a class="closed ticket" href="http://localhost/ticket/956" title="enhancement: Add a resource group table. (closed: fixed)">#956</a> </li><li>New kind field in resource. <a class="closed ticket" href="http://localhost/ticket/957" title="enhancement: new &#34;kind &#34; field added to resources (closed: wontfix)">#957</a> </li><li>UI for new kind field. <a class="closed ticket" href="http://localhost/ticket/958" title="enhancement: ui for new kind field (closed: duplicate)">#958</a> </li><li>Resources in REST API <a class="closed ticket" href="http://localhost/ticket/358" title="enhancement: Resources in REST API (closed: duplicate)">ticket:358</a> </li><li>Resources in WUI <a class="closed ticket" href="http://localhost/ticket/1445" title="enhancement: Resource View page in WUI (closed: fixed)">#1445</a> </li><li>Make Resources first class entity. <a class="closed ticket" href="http://localhost/ticket/922" title="enhancement: Make Resources first class entity. (closed: duplicate)">#922</a> (duplicate?) </li></ul><p> Background on this change can be found at: </p> <ul><li><a class="ext-link" href="http://ckan.okfnpad.org/resources"><span class="icon">​</span>http://ckan.okfnpad.org/resources</a> - discussion etherpad <ul><li>Contains commented version of James Gardner's main email: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html</a> </li></ul></li><li><a class="wiki" href="http://localhost/wiki/UseCasesResources">UseCasesResources</a> </li></ul> http://localhost/ticket/945 http://localhost/ticket/945 Report dread <p> CKAN currently comes out of the box being uneditable. We'd like to encourage people to have open CKANs and therefore by default init the db with this right: </p> <pre class="wiki">visitor -&gt; is editor on -&gt; System system </pre> http://localhost/ticket/946 http://localhost/ticket/946 Report rgrp <p> At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded. </p> <p> This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see <a class="closed ticket" href="http://localhost/ticket/833" title="enhancement: [super] Administrative dashboard extension (closed: fixed)">ticket:833</a>). </p> <p> An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own). </p> http://localhost/ticket/947 http://localhost/ticket/947 Report dread <p> When a customer logs in as a sysadmin then he/she see all packages, including deleted and pending ones. These are hidden to the average user, but the sysadmin has no idea of this until he clicks on the package and sees at the bottom 'state: deleted'. </p> <p> It should be more obvious than that on the search view - an icon, message or crossed-out name to packages are deleted. </p> http://localhost/ticket/948 http://localhost/ticket/948 Report pudo <p> based on repoze.who-friendlyForms </p> http://localhost/ticket/949 http://localhost/ticket/949 Report dread <p> ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN. </p> <p> In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext. </p> http://localhost/ticket/950 http://localhost/ticket/950 Report adrian.pohl@… <p> I can't add a package (e.g. <a class="ext-link" href="http://ckan.net/package/ub-konstanz"><span class="icon">​</span>http://ckan.net/package/ub-konstanz</a>) to a group (e.g. <a class="ext-link" href="http://ckan.net/group/bibliographic"><span class="icon">​</span>http://ckan.net/group/bibliographic</a>). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page. </p> http://localhost/ticket/951 http://localhost/ticket/951 Report pudo <p> second part of <a class="closed ticket" href="http://localhost/ticket/908" title="defect: Issues deploying extensions with modwsgi / uwsgi (closed: fixed)">#908</a> </p> http://localhost/ticket/952 http://localhost/ticket/952 Report rgrp <p> Have a tagline / description are under main title and set if from ckan.site_description config variable. </p> http://localhost/ticket/953 http://localhost/ticket/953 Report rgrp <p> Child tickets: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1107" title="refactor: Move package autocomplete from package controller and move to API (closed: fixed)">#1107</a> Move package autocomplete from package controller and move to API </li><li><a class="closed ticket" href="http://localhost/ticket/1087" title="enhancement: version and contact info api call (closed: fixed)">#1087</a> version and contact info api call </li></ul><p> Move to a format that has a separate responseHeader and response. </p> <h3 id="Astandardpackageresponse">A standard package response</h3> <pre class="wiki">{ responseHeader: { status: 0, } response: {package-dict} } </pre><p> On error: </p> <pre class="wiki">{ responseHeader: { status: {err-code}, error: 'message' } response: none } </pre><h3 id="Asearchquery">A search query</h3> <p> Based directly on solr. </p> <pre class="wiki">{ responseHeader: { status: 0, } response: { numFound: 5, start: 0 docs: [ ] } } </pre><h2 id="Issues">Issues</h2> <p> This is a breaking change for clients </p> <h2 id="References">References</h2> <ul><li><a class="ext-link" href="http://wiki.apache.org/solr/SolJSON"><span class="icon">​</span>http://wiki.apache.org/solr/SolJSON</a> </li><li><a class="ext-link" href="http://www.flickr.com/services/api/response.json.html"><span class="icon">​</span>http://www.flickr.com/services/api/response.json.html</a> </li></ul> http://localhost/ticket/954 http://localhost/ticket/954 Report dread <ul><li>Removal of init_db from every setup etc. </li><li>Get them working </li></ul> http://localhost/ticket/955 http://localhost/ticket/955 Report kindly <p> We need a way to group resources by various different factors. The most important of which is timeseries. </p> <p> The model will look like </p> <p> Package &lt;o2m&gt; resource_group &lt;o2m&gt; resource </p> <p> The use cases to this are outlined in this email. <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html</a> </p> http://localhost/ticket/956 http://localhost/ticket/956 Report kindly <p> The new field was outlined in </p> <p> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html</a> </p> <p> it says </p> <p> <strong>Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.</strong> </p> http://localhost/ticket/957 http://localhost/ticket/957 Report kindly <p> Ui for related new field in resource (<a class="closed ticket" href="http://localhost/ticket/957" title="enhancement: new &#34;kind &#34; field added to resources (closed: wontfix)">#957</a>) </p> http://localhost/ticket/958 http://localhost/ticket/958 Report rgrp <p> Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way. </p> <p> Should do this not just here but everywhere we can find in the code base. </p> <pre class="wiki">Module ckan.controllers.package:302 in edit &lt;&lt; am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg) if not am_authz: abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id))) auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg) &gt;&gt; abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) </pre><h2 id="Estimate">Estimate</h2> <p> Cost: 1h </p> http://localhost/ticket/960 http://localhost/ticket/960 Report rgrp <p> This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN. </p> <p> ckan-dev thread: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html</a> </p> <h2 id="TheIssue">The Issue</h2> <p> From <a class="closed ticket" href="http://localhost/ticket/926" title="enhancement: Pick a simpler form framework (closed: fixed)">#926</a>: </p> <p> The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise. </p> <p> From <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html</a>: </p> <p> ... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing. </p> <p> At least 3 functions involved [in this area]: </p> <ol><li>Generating (or just filling) a form template with 'form data' (and errors) </li><li>Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization' </li><li>Converting form data to model data (and validating) (inverse of previous step) </li></ol><h2 id="RelatedTickets">Related Tickets</h2> <ul><li><a class="closed ticket" href="http://localhost/ticket/926" title="enhancement: Pick a simpler form framework (closed: fixed)">#926</a> - Pick a simpler form framework </li><li><a class="closed ticket" href="http://localhost/ticket/1046" title="enhancement: Dictization and the new logic layer (closed: fixed)">#1046</a> 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict <ul><li><a class="closed ticket" href="http://localhost/ticket/1079" title="enhancement: Refactor API to use new logic layer and dictization (closed: fixed)">#1079</a> Refactor API to use new logic layer and dictization </li><li><a class="closed ticket" href="http://localhost/ticket/1078" title="enhancement: Refactors WUI controllers and forms to use logic layer (closed: fixed)">#1078</a> Refactor WUI controllers and forms to use logic layer </li><li>cf existing dumper and importer code </li><li>This will fix <a class="closed ticket" href="http://localhost/ticket/662" title="defect: Can't put entity that is returned by posting to package register (closed: fixed)">#662</a> </li></ul></li><li>[not ticketed yet] - validation layer (should work on serialized objects?) </li><li><a class="closed ticket" href="http://localhost/ticket/662" title="defect: Can't put entity that is returned by posting to package register (closed: fixed)">#662</a> - Can't put entity that is returned by posting to package register (Defect) </li><li><a class="closed ticket" href="http://localhost/ticket/972" title="enhancement: Merge 'extras' into main package dict rather than having separate key (closed: wontfix)">#972</a> - Merge 'extras' into main package dict rather than having separate key </li><li><a class="closed ticket" href="http://localhost/ticket/1035" title="enhancement: Form impressions given an ID (closed: wontfix)">#1035</a> - Form impressions are given IDs </li><li><a class="closed ticket" href="http://localhost/ticket/810" title="enhancement: Move &#34;add packages&#34; field up in group form (closed: wontfix)">#810</a> - Move "add packages" field up in group form (easier to do this once forms are done) </li></ul> http://localhost/ticket/961 http://localhost/ticket/961 Report rgrp <p> Basic implementation done (and deployed): </p> <p> <a class="ext-link" href="https://bitbucket.org/okfn/ckanext-datapreview"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-datapreview</a> </p> <p> However plenty to improve, e.g. </p> <ul><li>Support more formats (use external systems for preview?) <ul><li>json (!) </li><li>html (trivial!) </li><li>sparql </li><li>... </li></ul></li><li>Do not display preview if no preview </li></ul><p> Also suggest reworking to use external services rather than doing preview 'in house' (doing in house places heavy reliance on data proxy service and on converting data to a standard format). </p> http://localhost/ticket/962 http://localhost/ticket/962 Report rgrp <p> Package CKAN as a debian package so it is easier to install. </p> http://localhost/ticket/963 http://localhost/ticket/963 Report rgrp <p> In downloading packages we need to download resources and we need a <a class="missing wiki">ResourceDownloader?</a> object for this. these should be pluggable so that we can add support for different types of resources. </p> http://localhost/ticket/964 http://localhost/ticket/964 Report kindly <p> In order to test migrations properly, we need to test the upgraded database against *all* the tests. </p> <p> To do this a special flag to nosetests should be added. nosetests --migrations </p> <p> This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db. </p> http://localhost/ticket/965 http://localhost/ticket/965 Report sebbacon <p> I need to completely munge the rendered data coming from the app. There's not currently a plugin hook for this, where I get access to the entire environment. The most generic form of plugin would be WSGI Middleware. I propose adding an IMiddleware plugin hook for inserting abritrary middleware via our plugin system. </p> http://localhost/ticket/971 http://localhost/ticket/971 Report rgrp <p> Rather than have a separate 'extras' key all the extras fields should be consolidated into the main package dict when presenting the package internally or e.g. via the API. </p> <p> Why? Extras are really just an artefact of our internal storage model. Clients of the system (both internal and external) should just see a set of key/values with no distinction between extras and non-extras. </p> <h2 id="Issues">Issues</h2> <ul><li>Possible breaking change to the API (could enforce backwards compatibility by keepings extras for the time being) </li></ul><p> Possible subticket of forms refactoring: <a class="closed ticket" href="http://localhost/ticket/961" title="enhancement: [super] Refactoring of forms, validation and model synchronization (closed: fixed)">#961</a> </p> http://localhost/ticket/972 http://localhost/ticket/972 Report rgrp <p> Document how license system works and specifically how licenses can be configured. </p> <p> Cost: 2h </p> http://localhost/ticket/973 http://localhost/ticket/973 Report dread <p> Configuration option 'ckan.site_description' isn't documented </p> http://localhost/ticket/974 http://localhost/ticket/974 Report rgrp <p> Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE). </p> <p> This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search. </p> <p> We should fix this by finding a way to display placeholder attribute in all browsers. </p> http://localhost/ticket/975 http://localhost/ticket/975 Report dread http://localhost/ticket/976 http://localhost/ticket/976 Report dread <ul><li>db_upgrade (in ckan/model/<span class="underline">init</span>) makes call to validate_authorization_setup but forgets about rest of initialisation. </li></ul><ul><li>it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?) </li></ul> http://localhost/ticket/977 http://localhost/ticket/977 Report dread <p> Part of super-ticket <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a>. </p> <p> Follows on from <a class="closed ticket" href="http://localhost/ticket/826" title="enhancement: Resource 'extra' fields (closed: fixed)">#826</a>. </p> <p> We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis. </p> http://localhost/ticket/978 http://localhost/ticket/978 Report dread <p> Follows on from <a class="closed ticket" href="http://localhost/ticket/826" title="enhancement: Resource 'extra' fields (closed: fixed)">#826</a>. We can now edit resource extras in the WUI (to some extent - see <a class="closed ticket" href="http://localhost/ticket/978" title="enhancement: Edit Resource including Extras in Web UI (closed: fixed)">#978</a> for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API. </p> http://localhost/ticket/979 http://localhost/ticket/979 Report pudo <p> We've fixed a few things in CKAN that were discovered and tracked by HRI in their own issue tacker. This includes: </p> <ul><li>add authz checks to package edit links (cset:0752316cd2fe) </li><li>replace gettext with _ in controllers, to support unicode error msgs (cset:822340e6077e) </li><li>handle broken html in notes field without crashing on package read (cset:4b6be037dda0) </li><li>update i18n (cset:37d57dc3c492, cset:ea03173f5e77, cset:f16f4ee40fe7) </li></ul> http://localhost/ticket/980 http://localhost/ticket/980 Report kindly <p> This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up. </p> http://localhost/ticket/981 http://localhost/ticket/981 Report rgrp <p> We can just use the branch name to pull down specific versions of pip requirements so we don't need all the different versions in head. </p> <p> Remove them and update any client systems. </p> <p> Cost: 20m </p> http://localhost/ticket/982 http://localhost/ticket/982 Report dread <p> (should leave priviledges alone) </p> http://localhost/ticket/983 http://localhost/ticket/983 Report kindly <p> The migration scripts have not been inline with the upgrade scripts. There are lots of discrepancies. </p> http://localhost/ticket/984 http://localhost/ticket/984 Report pudo <ul><li>data.london.gov.uk (1d, ping DERI) </li><li>catalogues from Paris and Vienna (1d) </li></ul><p> Others? </p> <ul><li>digitaliser.dk (1d) </li><li>data-publica.com (2d) </li><li>data.suomi.fi (2d) </li><li>dati.piedmonte.it (1d) </li><li>portalu.de (2d) </li></ul> http://localhost/ticket/985 http://localhost/ticket/985 Report thejimmyg <p> Should have the following features: </p> <ul><li>A list of all packages with broken links on the resource </li><li>Under each packages a description of any resources with broken links to include: <ul><li>Link </li><li>Description </li><li>Format </li><li>First found to be broken date </li></ul></li></ul><p> [If you can't do the last one yet, don't worry] </p> http://localhost/ticket/986 http://localhost/ticket/986 Report pudo <p> We are now harvesting metadata from other sources in various places around CKAN. Such harvesting can include: </p> <ul><li>CSW/WFS for INSPIRE/UKLII (yields CKAN packages) </li><li>Catalogue scraping for LOD2 experiments (yields RDF graphs) </li><li>Atom/DCat for LOD2 production (yields RDF graphs) </li><li>OAI-PMH for <a class="ext-link" href="http://datadryad.org/"><span class="icon">​</span>http://datadryad.org/</a> and other dspace (yields CKAN packages) </li></ul><p> We should aim to consolidate the harvesting clients into a common system that is easy to extend when needed and can be re-used in different scenarios. </p> <p> In general, such a system would have the following stages: </p> <ul><li>Source selection: find what to download/scrape/harvest/parse </li><li>Index retrieval (i.e. package index) </li><li>Item retrieval (i.e. package entity) </li><li>(Optional: Serialization) </li><li>Normalisation </li><li><a class="missing wiki">Loading/Merging?</a> into CKAN </li></ul><p> Exisiting harvesters are at: </p> <ul><li>CSW: <a class="ext-link" href="https://bitbucket.org/okfn/ckanext-csw/src/"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-csw/src/</a> </li><li>Scraper+CKAN: <a class="ext-link" href="https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/"><span class="icon">​</span>https://bitbucket.org/pudo/dcat-tools/src/d5d96b06ec9a/dcat/crawl/</a> </li></ul> http://localhost/ticket/987 http://localhost/ticket/987 Report thejimmyg http://localhost/ticket/988 http://localhost/ticket/988 Report pudo <p> We need to support extending the model from plugins. This could involve: </p> <ul><li>Adding a plugin hook to extend the mapper </li><li>Adding an upgrade hook for plugin schema migrations </li><li>Documenting how this is to be done </li><li>Find a way to avoid conflicts </li></ul> http://localhost/ticket/989 http://localhost/ticket/989 Report rene.kapusta <p> It's not possible to use umlauts (like "CKAN Österreich") in the title / text. </p> http://localhost/ticket/990 http://localhost/ticket/990 Report dread <p> Form for new package has <a class="missing wiki">CheckboxExtraField?</a> checked, when the value is False. (as used in ckanext-dgu package v3 form) </p> http://localhost/ticket/991 http://localhost/ticket/991 Report rgrp <p> At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache. </p> <p> We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available. </p> <p> Cost: 10m </p> http://localhost/ticket/992 http://localhost/ticket/992 Report jason.kitcat@… <p> The link for About &gt; 'Project Home Page' on the CKAN.net footer should be changed to point to: </p> <p> <a href="http://ckan.org/">http://ckan.org/</a> </p> <p> The current link is broken. </p> <p> Thanks, Jason </p> http://localhost/ticket/993 http://localhost/ticket/993 Report kindly <p> It is too easy to make a mistake with the migrations at the moment. A more systematic way of doing them is needed and this needs to be documented. </p> http://localhost/ticket/994 http://localhost/ticket/994 Report kindly <p> Try and minimise the amount of different caching methodologies, especially for api calls. </p> http://localhost/ticket/995 http://localhost/ticket/995 Report kindly <p> This will involve running a sample of real requests synchronously against real data. </p> http://localhost/ticket/996 http://localhost/ticket/996 Report kindly <p> When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. </p> http://localhost/ticket/997 http://localhost/ticket/997 Report kindly <p> When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way. </p> http://localhost/ticket/998 http://localhost/ticket/998 Report dread <p> Rather than polluting the ckan core code base on ckan.net install with ckan.net specific changes these adaptations should be moved out into a dedicated ckan.net theme. </p> http://localhost/ticket/999 http://localhost/ticket/999 Report rgrp <p> Not used and extra overhead. Very useful to strip this out and low cost. </p> <p> Places where it seems to be: </p> <ul><li>lib/base.py: c.time_call_started </li><li>.... </li></ul> http://localhost/ticket/1000 http://localhost/ticket/1000 Report rgrp <p> When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key. </p> <p> The key change appears to be to change _get_user_for_apikey method in lib/base.py <a class="missing wiki">BaseController?</a> to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...). </p> <p> This is critical to incorporating any ajax editing into the frontend. </p> <p> As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author). </p> http://localhost/ticket/1001 http://localhost/ticket/1001 Report rgrp <p> This code is unused and tests are disabled i belive. I suggest stripping this out (can always be re-added in an extension later). If we remove we should probably delete the relevant tables (and so a migration is needed). </p> http://localhost/ticket/1002 http://localhost/ticket/1002 Report rgrp <p> A plain javascript library for interfacing with CKAN would be very useful (why? see below!). It would also be nice to have a pure html + javascript web interface to CKAN both for its own sake and to act as a demonstrator for the library. </p> <p> Why? </p> <ul><li>Development of bespoke interfaces -- much easier to edit html + javascript than to change ckan core <ul><li>E.g. for specific communities e.g. geodata, science </li><li>Specialized tasks - multi-package editing </li></ul></li><li>Very easy deployment and integration (e.g. can drop in to getthedata or other sites) </li></ul> http://localhost/ticket/1003 http://localhost/ticket/1003 Report dread <p> Need instructions on the group page to tell people they need to login to create a group. Someone must have deleted this. </p> http://localhost/ticket/1004 http://localhost/ticket/1004 Report dread <p> Package edit form at hmg.ckan.net has no publishers any more. </p> http://localhost/ticket/1005 http://localhost/ticket/1005 Report rgrp <p> Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time). </p> <p> Cost: 10m (giving high priority because of low cost) </p> <p> (Assigning to dread as he has been managing the stable branch). </p> http://localhost/ticket/1006 http://localhost/ticket/1006 Report pudo <p> Needed to allow extensions to be internationalized in the code and in templates. Requires in-template i18n domain switching which is broken between Genshi and Pylons, thus needs this hackish class. </p> <p> cf. <a class="ext-link" href="https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16"><span class="icon">​</span>https://bitbucket.org/okfn/wdmmg/src/198841e770e5/wdmmg/config/environment.py#cl-16</a> </p> http://localhost/ticket/1007 http://localhost/ticket/1007 Report pudo <p> We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer. </p> <p> We should also have page fragment caching in Genshi, which is not currently implmented. </p> <p> cf. <a class="ext-link" href="http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html"><span class="icon">​</span>http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html</a> </p> http://localhost/ticket/1008 http://localhost/ticket/1008 Report rgrp <ul><li>Forgot password (email a new password) </li><li>Confirm email </li><li>Do not show register page if you are logged in (redirect to home page) </li><li><a class="closed ticket" href="http://localhost/ticket/1010" title="enhancement: List CKAN users in WUI (closed: fixed)">ticket:1010</a> - listing of users. <ul><li>Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount) </li></ul></li></ul> http://localhost/ticket/1009 http://localhost/ticket/1009 Report rgrp <p> Should have user listing at /user/ rather than user account page. </p> <ul><li>list users, sorted by number of packages contributed/edited </li><li>Move user home page to /user/{user-id} </li><li>Paginated </li></ul> http://localhost/ticket/1010 http://localhost/ticket/1010 Report sebbacon <p> I have a requirement to allow an external source to define the groups of which a user is a member. </p> <p> I propose to create an IAuthzUserGroups plugin interface that allows an extension to arbitrarily extend the list of <a class="missing wiki">AuthzGroups?</a> that a user is in. </p> http://localhost/ticket/1011 http://localhost/ticket/1011 Report kindly <p> Revision history information is not accessible, dgu want this. </p> http://localhost/ticket/1012 http://localhost/ticket/1012 Report sebbacon <p> The Wordpresser extension seems to be working well for DataGM, but needs caching and tests. </p> http://localhost/ticket/1013 http://localhost/ticket/1013 Report sebbacon <p> The current oAuth extension tightly couples repoze.who, ckan, and data.no. These components should be decoupled; they also need more tests </p> http://localhost/ticket/1014 http://localhost/ticket/1014 Report dread <p> Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?) </p> <p> Example page: <a class="ext-link" href="http://ckan.net/group/edit/civil-society"><span class="icon">​</span>http://ckan.net/group/edit/civil-society</a> </p> <p> Exception: </p> <pre class="wiki">WebApp Error: &lt;class 'sqlalchemy.exc.IntegrityError'&gt;: (IntegrityError) duplicate key value violates unique constraint "group_revision_pkey" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u"A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09 </pre> http://localhost/ticket/1015 http://localhost/ticket/1015 Report sebbacon <p> Against ckan-1.3.1, when I create an authz group called "administrators" and visit /authorizationgroup/edit/administrators, I am unable to add more than one user to it. </p> <p> Each time I add additional users, the existing user on the list is replaced with the new one. </p> http://localhost/ticket/1017 http://localhost/ticket/1017 Report dread <p> (It has been moved into ckanext-dgu) </p> http://localhost/ticket/1018 http://localhost/ticket/1018 Report pudo <p> We propose the following push approach using individual packages: </p> <ol><li>New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form: </li></ol><blockquote> <p> { </p> <blockquote> <p> payload: The entity data as available from the REST API entity-type: 'Package', operation-type: 'create'|'update'|'delete' # one of these options </p> </blockquote> <p> } </p> </blockquote> <p> It will be sent as the body of the request with content-type set to application/json. </p> <ol start="2"><li>The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review. </li></ol><ol start="3"><li>Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model) </li></ol><ol start="4"><li>[optional] CKAN side will support configuring authorization for basic authentication if applied on WP side. </li></ol><ol start="5"><li>[optional] List queue status (including failures) on ckan adminstrative dashboard. </li></ol> http://localhost/ticket/1019 http://localhost/ticket/1019 Report kindly <p> add revisioning to harvested document table </p> http://localhost/ticket/1020 http://localhost/ticket/1020 Report pudo <p> HRI don't like federation, want to login normal way only. Make this a config option and perhaps even mess with runtime repoze config </p> http://localhost/ticket/1021 http://localhost/ticket/1021 Report pudo <p> Report on errors by any worker daemons, send them out via E-Mail </p> http://localhost/ticket/1022 http://localhost/ticket/1022 Report pudo <p> Schedule worker calls to be ran several times, with certain intervals. </p> http://localhost/ticket/1023 http://localhost/ticket/1023 Report pudo <p> When a user visits the site, he/she may only see a login box. </p> http://localhost/ticket/1024 http://localhost/ticket/1024 Report dread <p> Currently the default authz for a package is hard-coded to: </p> <pre class="wiki"> &lt;PackageRole user="visitor" role="editor" context="Package"&gt;, &lt;PackageRole user="logged_in" role="editor" context="Package"&gt;, </pre><p> This should be configurable in the config, so that you can have a more locked down instance etc. </p> http://localhost/ticket/1025 http://localhost/ticket/1025 Report dread <p> It's handy to be able to create users using the cli (e.g. dgu migration) </p> http://localhost/ticket/1026 http://localhost/ticket/1026 Report pudo <p> We want to have authz checks on all controller actions so that we can lock down CKAN to a login-only mode. </p> http://localhost/ticket/1027 http://localhost/ticket/1027 Report dread http://localhost/ticket/1028 http://localhost/ticket/1028 Report kindly <p> Errors are caused when harvesting documents. This also makes the count not show up correctly on the ckan search page. </p> http://localhost/ticket/1029 http://localhost/ticket/1029 Report thejimmyg http://localhost/ticket/1030 http://localhost/ticket/1030 Report rgrp <p> Add an api for searching users. This is needed for any kind of ajax autocomplete (needed for anywhere we want to add users). </p> <ul><li>API location: /api/util/user/lookup?q=querystr&amp;limit=10 </li><li>Return json objects containing {id: ..., name: ..., fullname: ...} </li><li>Put in a module called controllers/apiv2/user.py </li></ul> http://localhost/ticket/1031 http://localhost/ticket/1031 Report rgrp <p> Add resources into Web User Interface. </p> <ul><li>Locate at: /dataset/{dataset}/resource/{id} </li><li>CRUD </li><li>Authorization </li></ul><p> Done: </p> <ul><li><del><a class="closed ticket" href="http://localhost/ticket/945" title="enhancement: [super] Richer resources - Resource Groups, new fields, improved UI (closed: fixed)">#945</a> - Richer resources - Resource Groups, new fields, improved UI</del> </li><li><del><a class="closed ticket" href="http://localhost/ticket/1445" title="enhancement: Resource View page in WUI (closed: fixed)">#1445</a> - Resource View page in WUI</del> </li><li><del><a class="closed ticket" href="http://localhost/ticket/1450" title="enhancement: Simplify dataset view page (closed: fixed)">#1450</a> - Dataset view pages to match</del> </li></ul><p> Moved to superticket <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a>: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/978" title="enhancement: Edit Resource including Extras in Web UI (closed: fixed)">#978</a> - Edit Resource Extras in Web UI </li></ul> http://localhost/ticket/1032 http://localhost/ticket/1032 Report dread <p> Go to <a class="ext-link" href="http://ckan.net/user/register"><span class="icon">​</span>http://ckan.net/user/register</a> and fill in all the fields apart from password. On submit you get 500 error. </p> http://localhost/ticket/1033 http://localhost/ticket/1033 Report dread <ol><li>Login (shows ckan home page with flash message "Welcome back xyz") </li><li>Click away, to "Add package" say </li><li>Click back to ckan "Home". It shows "Welcome back xyz" again - it shouldn't, I assume? </li></ol><p> </p> http://localhost/ticket/1034 http://localhost/ticket/1034 Report dread <p> To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form. </p> <p> (Something to consider for the new form mechanism) </p> http://localhost/ticket/1035 http://localhost/ticket/1035 Report johnlawrenceaspden <p> create a new test file ckan/model/test_user.py </p> <p> add tests for the following three functions in ckan/model/user.py </p> <p> number_of_edits, number_administered_packages, search </p> <p> merged in in changeset 0046f83aedcf </p> http://localhost/ticket/1036 http://localhost/ticket/1036 Report thejimmyg <p> CKAN's harvesting facility is now live on DGU but there are some major improvements that could be made to make it more robust and better fit the generic CKAN harvesting framework proposed in <a class="closed ticket" href="http://localhost/ticket/987" title="defect: Common harvesting framework (closed: duplicate)">#987</a>. </p> <p> Some of the key issues: </p> <ul><li>Error reports do not currently contain the ID or title of the document with the error. </li><li>We only have "added" and "error" logging on jobs when we really need a report of "added", "updated", "not changed" and "errors" with the items in each referencing a real metadata document for which harvesting was attempted </li><li>We need deletion and editing of sources, without deleting the harvested documents or packages </li><li>We need a more robust harvesting mechanism than a cron job or we need to deal with the case of multiple cron jobs running at once. </li><li>We need to know the last time a list of documents was scheduled for harvest and the last time each one was fetched. </li></ul> http://localhost/ticket/1037 http://localhost/ticket/1037 Report dread <p> Add 'package:all' to authz tool to allow mass changes of authz. </p> http://localhost/ticket/1038 http://localhost/ticket/1038 Report dread <p> (instead of being hard coded) </p> http://localhost/ticket/1039 http://localhost/ticket/1039 Report dread <p> Public file path code adds a trailing , and thus adds / as a static file app in the Pylons middleware cascade. </p> http://localhost/ticket/1040 http://localhost/ticket/1040 Report thejimmyg <p> For example, I will document the following: </p> <ul><li>A specific guide on using mercurial when working with our branching and merging policy <ul><li>DONE - <a class="ext-link" href="http://wiki.ckan.net/Becoming_a_CKAN_Developer"><span class="icon">​</span>http://wiki.ckan.net/Becoming_a_CKAN_Developer</a> </li></ul></li><li>Policy document on how we use tickets, plan sprints and make releases <ul><li>DONE - <a class="ext-link" href="http://wiki.ckan.net/Becoming_a_CKAN_Developer"><span class="icon">​</span>http://wiki.ckan.net/Becoming_a_CKAN_Developer</a> </li></ul></li><li>How to package software as .deb file for project deployment </li></ul><p> I'd love if someone else would write: </p> <ul><li>An authorisation tutorial covering the core model, the command line tools and examples of every possible way of using the system </li><li>A HOWTO guide with screenshots for adding a package </li></ul> http://localhost/ticket/1041 http://localhost/ticket/1041 Report dread <p> It's not good to have ckanext doing lots of different things with different dependencies. Split it off into: </p> <ul><li>ckanext-importlib </li></ul><p> etc. </p> <p> And then deprecate the ckanext repo itself. </p> http://localhost/ticket/1042 http://localhost/ticket/1042 Report kindly <p> see summary. </p> http://localhost/ticket/1043 http://localhost/ticket/1043 Report dread <p> The problem is that in ckan/controllers/rest.py the <a class="missing wiki">BaseApiController?</a> has this method: </p> <pre class="wiki"> def __before__(self, action, **env): BaseController.__before__(self, action, **env) if not self.authorizer.am_authorized(c, model.Action.SITE_READ, model.System): abort(401, _('Not authorized to see this page')) </pre><p> which works on the basis of your c.user, rather than your apikey. All API users are treated as visitors (since API users don't get a login cookie) and even a sysadmin's apikey is blocked unless there is a right for a Visitor to SITE_READ. </p> <p> Also needs tests. </p> <p> (Also, why is this restriction only on the API, package search, group index and tags and agroup index? I'm guessing SITE_READ is only for places where other authz don't apply, but maybe it should not be called 'SITE_READ' but 'OTHER_READ' or something?) </p> http://localhost/ticket/1044 http://localhost/ticket/1044 Report dread <pre class="wiki">http://ckan.net/api/2/rest/group </pre><p> returns group IDs but I can only reach a group by name: </p> <pre class="wiki">http://ckan.net/api/2/rest/group/economics </pre><p> when I also want to get a group by ID: </p> <pre class="wiki">http://ckan.net/api/2/rest/group/04fb43d2-8ddf-4485-9bf5-66d47d3672f3 </pre> http://localhost/ticket/1045 http://localhost/ticket/1045 Report thejimmyg <p> The stages involved with doing this. </p> <ul><li>Convert model objects to standard dict format (DONE) </li><li>Convert standard dicts to current api formats (DONE) </li><li>Make standard dicts savable (DONE) </li><li>Validate standard dict format. (DONE) </li><li>Authorize actions </li></ul> http://localhost/ticket/1046 http://localhost/ticket/1046 Report sebbacon <p> At the bottom, it says something like </p> <p> Author: Bob Bumgardner Since you have not signed in this will just be your IP address. Click here to sign in before saving (opens in new window). </p> http://localhost/ticket/1047 http://localhost/ticket/1047 Report dread <ul><li>Deleting a group changes state to 'deleted' rather than purging it </li><li>Adding authz tests for deleted groups </li></ul> http://localhost/ticket/1048 http://localhost/ticket/1048 Report dread <p> We're getting these sorts of exceptions from <a class="ext-link" href="http://offener.datenkatalog.at/"><span class="icon">​</span>http://offener.datenkatalog.at/</a> like we had from ckan.net, which was due to small errors in migration scripts. Is it worth fixing this database, and in fact all the ckans on eu3 at once? </p> <pre class="wiki">WebApp Error: &lt;class 'sqlalchemy.exceptions.IntegrityError'&gt;: (IntegrityError) duplicate key value violates unique constraint "group_revision_pkey" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, ... </pre><p> Thoughts James and Rufus? </p> http://localhost/ticket/1049 http://localhost/ticket/1049 Report johnlawrenceaspden <p> Refactor ckan/lib/authztool.py so that the relevant methods are independent of the command line interface. </p> <p> The extracted methods should live in a new file ckan/authz.py. authztool.py should probably move into cli.py and will just do command line parsing and printing and use ckan/authz.py. The updated web gui for authz will also use this code. </p> <p> Tests should be made. There's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests. </p> <p> all to go on a branch feature-1050-refactor-authtoolz </p> <h2 id="Optionalextras">Optional extras</h2> <ul><li>Rename ckan/authz.py to ckan/lib/authz.py or even ckan/logic/authz.py </li></ul> http://localhost/ticket/1050 http://localhost/ticket/1050 Report sebbacon <p> Allow deployers to set a URL pointing to their own favicon </p> http://localhost/ticket/1051 http://localhost/ticket/1051 Report dread <p> No authz on: </p> <ul><li>Group creation/edit/listing </li><li>Package relationship create/edit/delete </li></ul> http://localhost/ticket/1052 http://localhost/ticket/1052 Report dread <p> Currently in the API if you DELETE a package/group/user (and you have the required permissions) then it purges the object, when it should probably just set the state to deleted. </p> <p> There is no way to delete objects at the moment - changes to 'state' are ignored in the API. </p> <p> Do we need an alternative way to purge objects in the API? </p> http://localhost/ticket/1053 http://localhost/ticket/1053 Report dread <p> Changing an old resource and creating a new resource on the end results in the old resource moving to the end in the ordering. </p> <p> This breaks tests: </p> <ul><li>(ckanext-dgu) ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadBasic.test_fields </li><li>(ckanext-importlib) ckanext/importlib/tests/test_loader.py:TestLoaderInsertingResources.test_0_reload </li></ul><p> You can make a ckan test break with this patch: </p> <pre class="wiki">diff -r e6643cf1324c ckan/tests/models/test_resource.py --- a/ckan/tests/models/test_resource.py Wed Mar 23 13:25:52 2011 +0000 +++ b/ckan/tests/models/test_resource.py Wed Mar 23 19:22:35 2011 +0000 @@ -297,6 +297,8 @@ 'url':self.urls[1], 'format':u'OTHER FORMAT', 'description':self.description, 'hash':self.hash, 'id':original_res_ids[2]}, + { #new + 'url':'new'}, ] pkg.update_resources(res_dicts) model.repo.commit_and_remove() </pre><p> There seems to be a problem with vdm creating a replacement Resource for the old resource - because it has a duplicate position it is put to the end by the SQLAlchemy ordering_list function. </p> http://localhost/ticket/1054 http://localhost/ticket/1054 Report dread <p> Tests marked decorated "@search_related" should only be run against postgresql, but in fact they don't get run at all. </p> http://localhost/ticket/1055 http://localhost/ticket/1055 Report pudo <p> Use case: </p> <ul><li>Login using OpenID </li><li>Click on 'My account' - results in 404 </li></ul><p> Solutions: </p> <ul><li>User user.id instead of their name </li><li>Escape the URL properly. </li></ul> http://localhost/ticket/1056 http://localhost/ticket/1056 Report dread <pre class="wiki">$ curl "http://127.0.0.1:5000/api/rest/package/annakarenina?callback=&lt;script&gt;jsoncallback" </pre><p> gives: </p> <pre class="wiki">&lt;script&gt;jsoncallback({"id": "c10ebd31-5b45-4f6f-885d-dca9b18caec4", "name": "annakarenina", "title": "A Novel By Tolstoy", </pre><p> which could run script code in the client who made the call. </p> <p> One idea for filtering: <a class="ext-link" href="http://tav.espians.com/sanitising-jsonp-callback-identifiers-for-security.html"><span class="icon">​</span>http://tav.espians.com/sanitising-jsonp-callback-identifiers-for-security.html</a> Maybe just better to have a restricted whitelist of characters to be even more sure. </p> <p> Same as: <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/906"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/906</a> </p> http://localhost/ticket/1057 http://localhost/ticket/1057 Report dread <p> Examples which prompt annoying exception emails: </p> <pre class="wiki">http://ckan.net/locale?locale=ja Module ckan.i18n:21 in set_session_locale assert locale in _KNOWN_LOCALES </pre><p> A bot has caused these: </p> <pre class="wiki">http://ca.ckan.net/package/new?package_form=gov Module ckan.forms.registry:32 in get_fieldset raise ValueError('Could not find package_form name %r in those found: \n%r' % (package_form, [en.name for en in entrypoints])) ValueError: Could not find package_form name u'gov)' in those found: ['gov', 'standard', 'ca'] </pre> http://localhost/ticket/1058 http://localhost/ticket/1058 Report dread <p> Loader currently checks for same name, but also should check for name_, name<span class="underline"> etc. </span></p> http://localhost/ticket/1059 http://localhost/ticket/1059 Report dread <p> e.g. we just added notes_rendered and that is read in as an extra field. Tests failing in ckanext-importlib </p> <p> Also related: we are missing lost metadata_created and metadata_modified in the dumps. </p> http://localhost/ticket/1060 http://localhost/ticket/1060 Report dread <p> No links to home/license and it contains out of date references to knowledgeforge. Remove it. </p> http://localhost/ticket/1061 http://localhost/ticket/1061 Report sebbacon <p> The preview of "Species Misc Turtle Download" at <a class="ext-link" href="http://ckan.net/package/taxonconcept"><span class="icon">​</span>http://ckan.net/package/taxonconcept</a> results in the following error: </p> <p> Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte </p> http://localhost/ticket/1062 http://localhost/ticket/1062 Report sebbacon <p> I've been asked if we can do something about the overflow of the Group name in the right hand column on this page: </p> <blockquote> <p> <a class="ext-link" href="http://register.data.overheid.nl/package/europese-aanbestedingen"><span class="icon">​</span>http://register.data.overheid.nl/package/europese-aanbestedingen</a> </p> </blockquote> <p> The reason is that the list display for groups is in the form "group_tltie (group_name)", and of course group_name can't have spaces and so can't wrap nicely. </p> <p> I was wondering if there's a good reason why we don't only display group_title (if it exists) and group_name only when there's not a title? </p> http://localhost/ticket/1063 http://localhost/ticket/1063 Report amercader <p> The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message). This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly. </p> http://localhost/ticket/1064 http://localhost/ticket/1064 Report johnlawrenceaspden <p> Child tickets </p> <ul><li><a class="new ticket" href="http://localhost/ticket/1198" title="enhancement: Publisher hierarchy (new)">#1198</a> Publisher hierarchy </li><li><a class="closed ticket" href="http://localhost/ticket/1050" title="enhancement: Authz lib improvement and refactor of ckan/lib/authztool.py (closed: invalid)">#1050</a> Authz lib improvement and refactor of ckan/lib/authztool.py </li><li><a class="closed ticket" href="http://localhost/ticket/1004" title="defect: Group creation instructions missing (closed: fixed)">#1004</a> Group creation instructions missing </li><li><a class="closed ticket" href="http://localhost/ticket/1099" title="defect: strange interactions between two browsers while playing with authz groups (closed: wontfix)">#1099</a> Strange interactions between two browsers while playing with authz groups </li><li><a class="closed ticket" href="http://localhost/ticket/1115" title="defect: can have two authzgroups with the same name (closed: wontfix)">#1115</a> can have two authzgroups with the same name </li><li><a class="closed ticket" href="http://localhost/ticket/1133" title="defect: command line rights manipulation doesn't work (closed: worksforme)">#1133</a> command line rights manipulation doesn't work </li><li><a class="closed ticket" href="http://localhost/ticket/1138" title="enhancement: minor navigations behave inconsistently (closed: invalid)">#1138</a> minor navigations behave inconsistently </li></ul><p> Old ticket description: </p> <ol><li>Change name of <a class="missing wiki">AuthzGroup?</a> to <a class="missing wiki">UserGroup?</a> to reflect what it is for </li></ol><ol start="2"><li>Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones. <ul><li>Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system. </li><li>Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages) </li></ul></li></ol><ol start="3"><li>Change <a class="missing wiki">UserGroups?</a> so that they can have a hierarchical structure, </li></ol><h2 id="MoreinfoonHierarchychange">More info on Hierarchy change</h2> <p> e.g. <a class="missing wiki">UserGroup?</a> NHS contains the User nhsysadmin, as well as the <a class="missing wiki">UserGroups?</a> SURREY and BERKS, which themselves contain users. </p> <p> One user in SURREY is Simon the Sysadmin, who has permissions on the whole system. His permissions should not leak out to other users or groups, and user permissions generally should not. </p> <p> Each Group has permissions over various objects. </p> <p> A user has permissions in his own right, and also has the permissions of his own group, and of all the groups contained in his group, and so on recursively. </p> <p> Algorithm: </p> <p> possible(user, action, package): </p> <blockquote> <p> if user has permission for action on package </p> <blockquote> <p> or any of have that permission </p> <blockquote> <p> or any of his groups group-children (but not user-children), and so on recursively have the permission. </p> </blockquote> </blockquote> </blockquote> http://localhost/ticket/1065 http://localhost/ticket/1065 Report dread <p> The definition of the 'reader' role includes creating packages, which is too permissive for some CKAN instances (e.g. DGU). 'Reader' suggests only reading, so I think this role should avoid creating and editing. </p> <p> All projects so far want all roles to be able to create users, so this stays as a Reader action for now, as a convenience. </p> <p> Implementation: </p> <ul><li>Action.PACKAGE_CREATE removed from reader's default_role_actions </li><li>Visitor has a new default role, called 'anon_editor' which can edit packages, but not groups / auth groups - you have to log in for that. </li><li>Migration script not needed? </li><li>Code comments written, to make clear the suggested policy </li></ul> http://localhost/ticket/1066 http://localhost/ticket/1066 Report dread <p> Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options. </p> http://localhost/ticket/1067 http://localhost/ticket/1067 Report dread <p> This test has been failing since the clocks changed: </p> <pre class="wiki">====================================================================== FAIL: ckan.tests.models.test_package.TestPackageRevisions.test_02_metadata_created_and_modified ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/dread/hgroot/pyenv-ckan2/lib/python2.6/site-packages/nose-0.11.3-py2.6.egg/nose/case.py", line 186, in runTest self.test(*self.arg) File "/home/dread/hgroot/ckan2/ckan/tests/models/test_package.py", line 283, in test_02_metadata_created_and_modified assert out == exp, (out, exp) AssertionError: (datetime.datetime(2011, 4, 1, 10, 45, 50, 875509), datetime.datetime(2011, 4, 1, 9, 45, 50, 875509)) ---------------------------------------------------------------------- </pre> http://localhost/ticket/1068 http://localhost/ticket/1068 Report rgrp <p> Idea is to have stubs for datasets that someone wants but don't yet exist (or haven't been discovered) in the way one has stub pages on a wiki. </p> <p> We could do this within the existing model by a slight 'abuse' - create a dataset and mark it with a special tag e.g. todo.does-not-yet-exist or similar ... </p> <p> (Just as we have datasets listed that exist but aren't available ...) </p> <p> Alternative would be to have a request for datasets subsystem. </p> <p> I prefer the stub dataset model because it's simpler, provides a simple workflow (as a dataset is found or comes into existence), and the package page provides a natural space in which to accumulate information about what is wanted and what exists. </p> <h2 id="Implementation">Implementation</h2> <ul><li>Agree a new dedicated tag. e.g. todo.does-not-exist </li></ul><h2 id="Related">Related</h2> <ul><li>This ticket has clear links to <a class="ext-link" href="http://getthedata.org/"><span class="icon">​</span>http://getthedata.org/</a> </li></ul> http://localhost/ticket/1069 http://localhost/ticket/1069 Report rgrp <p> See <a class="ext-link" href="http://wiki.ckan.net/Domain_Model"><span class="icon">​</span>http://wiki.ckan.net/Domain_Model</a> especially section on v2. </p> <ul><li>New domain model is planned but not yet finally agreed. </li><li>Layer architecture is complete and implemented </li></ul> http://localhost/ticket/1070 http://localhost/ticket/1070 Report dread <p> api/rest/package_history is not RESTful or follow API naming conventions. Therefore move it to /api/rest/package/revisions </p> <p> Also, API docs incomplete. </p> http://localhost/ticket/1071 http://localhost/ticket/1071 Report dread <p> It takes several minutes to print the 'rights' on DGU, which is annoying when you only want to grep for a few lines. Much quicker than grepping is to filter in the query. </p> http://localhost/ticket/1072 http://localhost/ticket/1072 Report dread <p> Tool that checks which packages have not been indexed. </p> <p> Required for DGU: <a class="ext-link" href="https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940"><span class="icon">​</span>https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940</a> </p> http://localhost/ticket/1073 http://localhost/ticket/1073 Report rgrp <p> Currently repeat the same template and code across Package Authz, Group Authz, and Authz Group authz. </p> <p> Having now implemented a new, cleaner setup in ckanext-admin we should port this back into core. </p> <ul><li>Common template code (checkbox template) </li><li>Logic code (or just common code) for wiring into authz system </li><li>Look for all places thoroughout the system where usernames, authzgroups or groups need to be typed into boxes, and make sure that they auto-complete appropriately. </li></ul><p> Will also deliver a significant improvement in the form of ajax user lookup. </p> http://localhost/ticket/1074 http://localhost/ticket/1074 Report rgrp <p> Roles on System object are important because admin role on system equates to being a 'sysadmin' (i.e. able to do anything). </p> <ul><li>Make users sysadmin (either as separate action or as part of editing roles on system object) </li><li>/authz subpage for editing roles on system object <ul><li>Add and update user roles </li><li>Add and update authz group roles </li><li>List actions associated to roles at top of page (extra points for checkbox table with editability) </li></ul></li><li>Document on <a class="ext-link" href="http://wiki.ckan.net/Authorization"><span class="icon">​</span>http://wiki.ckan.net/Authorization</a> what roles on System object 'mean' esp sysadmin role on System </li></ul><h2 id="RelatedTickets">Related Tickets</h2> <p> </p> <ul><li>super ticket: <a class="closed ticket" href="http://localhost/ticket/833" title="enhancement: [super] Administrative dashboard extension (closed: fixed)">#833</a> </li><li>authz lib improvement and authztool refactor <a class="closed ticket" href="http://localhost/ticket/1050" title="enhancement: Authz lib improvement and refactor of ckan/lib/authztool.py (closed: invalid)">#1050</a> </li></ul> http://localhost/ticket/1075 http://localhost/ticket/1075 Report rgrp <h2 id="PurgingRevisions">Purging Revisions</h2> <ul><li>Delete button displayed on: <ul><li>/revision/list </li><li><del>(/package/history)</del> <ul><li>/package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison. </li></ul></li><li>/revision/{id} </li></ul></li><li>Delete button submits to delete action on revision and changes revision state to 'deleted'. <ul><li>undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?) </li></ul></li><li>Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state. </li></ul><h2 id="PurgingPackages">Purging Packages</h2> <ul><li>Put into deleted state. </li><li>Listed on /ckan-admin/trash </li><li>Separate Empty trash button which deletes all associated revisions. <ul><li>Should be separate from Empty trash for revisions </li></ul></li></ul><h2 id="Currentsystem">Current system</h2> <ul><li>Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!) </li></ul> http://localhost/ticket/1076 http://localhost/ticket/1076 Report rgrp <h2 id="Option1:ChangesetModel">Option 1: 'Changeset' Model</h2> <p> See <a class="assigned ticket" href="http://localhost/ticket/1135" title="enhancement: Changeset model for vdm (assigned)">ticket:1135</a> for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this. </p> <p> Have developed a new "changeset" based model for revisioning in vdm. </p> <h3 id="Implementation">Implementation</h3> <ul><li>The main challenge with this change is schema and data migration </li></ul><p> Every revisioned object has a revision_id and revision attribute. </p> <p> Approximate algorithm: </p> <pre class="wiki">Revision -&gt; Changeset for revtype in [PackageRevision, ...]: for pkgrev in package_revision: changeset = lookupchangeset(package_revision) ChangeObject(cset, (table, id), dictize(pkgrev)) </pre><p> Question: </p> <ul><li>does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter. </li></ul><h2 id="Option2:SimplifyRevisionObjectModel">Option 2: Simplify Revision Object Model</h2> <p> Just use a simpler vdm, see <a class="assigned ticket" href="http://localhost/ticket/1136" title="enhancement: Move to SessionExtension in vdm (assigned)">ticket:1136</a> (move to SessionExtension) and <a class="assigned ticket" href="http://localhost/ticket/1137" title="enhancement: Remove need for statefulness in vdm (assigned)">ticket:1137</a> (remove need for statefulness in vdm). </p> <h2 id="Discussion">Discussion</h2> <p> Advantage of Option 1 versus 2: </p> <ul><li>Easier support for pending state and similar behaviour </li><li>No need to introduce new tables (and hence migrations) when making something revisioned (or not). </li></ul><h3 id="Disadvantages">Disadvantages</h3> <ul><li>Migration is required </li><li>More difficult to query revision history. <ul><li>Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult. </li></ul></li><li>Performance (?) <ul><li>Have one big ChangeObject table to query when looking at changed objects rather than many revision tables. <ul><li>Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table </li></ul></li></ul></li></ul><h2 id="Conclusion">Conclusion</h2> <p> Implement Option 2 and leave Option 1 for present. </p> <p> Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2). </p> <p> Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date. </p> http://localhost/ticket/1077 http://localhost/ticket/1077 Report rgrp <ul><li>Deserialize forms to new dict format. </li><li>Replace controllers/forms to use dictization. </li></ul> http://localhost/ticket/1078 http://localhost/ticket/1078 Report rgrp <ul><li>Convert current api saves to the new standard dict format. </li></ul> http://localhost/ticket/1079 http://localhost/ticket/1079 Report johnlawrenceaspden <p> I've found that if I make an authorization group I sometimes can't remove myself from it. I've no idea why. I can add and remove other users. I'll investigate, just making a note of it here. </p> http://localhost/ticket/1081 http://localhost/ticket/1081 Report johnlawrenceaspden <p> Set language to Greek, flash message says 'Language set to: English', but page is now about half in Greek. </p> <p> Set language back to English causes server error: </p> <p> <a class="missing wiki">AttributeError?</a>: '<a class="missing wiki">NoneType?</a>' object has no attribute 'path' </p> <p> Module ckan.controllers.error:29 in document view </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> if original_request.path.startswith('/api'): </p> </blockquote> </blockquote> </blockquote> <p> However going to a new page reveals that it's back to English </p> http://localhost/ticket/1082 http://localhost/ticket/1082 Report johnlawrenceaspden <p> the add_user_to_role/remove_user_from_role functions are asymmetrical in that the add function is happy to add the same role twice but the remove asserts that it's only in the table once and crashes if that's not true. </p> <p> an attempt has been made to guard against this, but fails, I think because the add functions rely on the caller committing the change to the db. </p> <p> same problem affects corresponding authorization_group functions </p> <p> I'll try to sort this out. Making a note here. </p> http://localhost/ticket/1083 http://localhost/ticket/1083 Report wwaites <p> need to make some changes for the links to semantic.ckan.net. it should use <a class="ext-link" href="http://semantic.ckan.net/record/"><span class="icon">​</span>http://semantic.ckan.net/record/</a>&lt;package_id&gt; now </p> <p> append .rdf, .ttl, .nt, .dot, .json (even .html for an ugly table) to taste (or just leave off the suffix and let content negotiation take care of it) </p> <p> the base url is changed, but it now uses id not name. </p> <p> see for example: </p> <ul><li><a class="ext-link" href="http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33"><span class="icon">​</span>http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33</a> </li><li><a class="ext-link" href="http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33.ttl"><span class="icon">​</span>http://semantic.ckan.net/record/6058c017-607b-48d9-b3cd-72106ad96e33.ttl</a> </li></ul> http://localhost/ticket/1084 http://localhost/ticket/1084 Report johnlawrenceaspden <p> ckan.net appears to have either gone down or be running ultra slowly. </p> <p> this means that ckan copies running locally on my machine run very slowly indeed. </p> <p> is this behaviour desirable? </p> <p> This command finds lots of <a class="ext-link" href="http://~~~ckan.net"><span class="icon">​</span>http://~~~ckan.net</a> references in python, html and javascript files: </p> <p> find ~/pyenv/src \( -name "*.py" -or -name "*.html" -or -name "*.js" \) -print0 | xargs -0 -e grep --color -nH -e "<a class="ext-link" href="http://.*ckan.net"><span class="icon">​</span>http://.*ckan.net</a>" </p> <p> output for reference: </p> <hr /> <p> /home/okfn/pyenv/src/ckan/ckan/<span class="underline">init__.py:5:Network (CKAN) site: <a class="ext-link" href="http://www.ckan.net"><span class="icon">​</span>http://www.ckan.net</a>. /home/okfn/pyenv/src/ckan/ckan/lib/create_test_data.py:346:&lt;<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>&gt; /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:3:DOMAIN = '<a class="ext-link" href="http://ckan.net"><span class="icon">​</span>http://ckan.net</a>' /home/okfn/pyenv/src/ckan/ckan/lib/rdf.py:4:CKAN_NAMESPACE = '<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>#' /home/okfn/pyenv/src/ckan/ckan/lib/talis.py:60: 'ckan':'<a class="ext-link" href="http://ckan.net/ns"><span class="icon">​</span>http://ckan.net/ns</a>#', /home/okfn/pyenv/src/ckan/ckan/public/scripts/bookmarklet.js:2: f='<a class="ext-link" href="http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title"><span class="icon">​</span>http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;title='+encodeURIComponent(document.title</a>); /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:16: addtockan.src='<a class="ext-link" href="http://ckan.net/scripts/bookmarklet.js"><span class="icon">​</span>http://ckan.net/scripts/bookmarklet.js</a>'; /home/okfn/pyenv/src/ckan/ckan/public/scripts/test_bookmarklet.html:27: &lt;p&gt;&lt;strong&gt;Proper bookmarklet (compressed -- need to escape &amp;amp;):&lt;/strong&gt; &lt;a href="javascript:(function(){f='<a class="ext-link" href="http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;amp;&amp;amp;(d=n.content)){f+='&amp;amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a"><span class="icon">​</span>http://ckan.net/package/new?url='+encodeURIComponent(window.location.href)+'&amp;amp;title='+encodeURIComponent(document.title);if((n=document.getElementsByName('description')[0])&amp;amp;&amp;amp;(d=n.content)){f+='&amp;amp;notes='+encodeURIComponent(d);}a=function(){if(!window.open(f)){location.href=f;}};if(/Firefox/.test(navigator.userAgent)){setTimeout(a,0)}else{a</a>()}})()"&gt;Add to CKAN&lt;/a&gt; /home/okfn/pyenv/src/ckan/ckan/templates/home/license.html:31: For convenience, all material - including all package, tag and revision information - is available in bulk, in the form of a full dump of the CKAN database. This (gzipped) dump file is updated daily and can be downloaded from &lt;a href="<a class="ext-link" href="http://www.ckan.net/dump/"><span class="icon">​</span>http://www.ckan.net/dump/</a>"&gt;<a class="ext-link" href="http://www.ckan.net/dump/"><span class="icon">​</span>http://www.ckan.net/dump/</a>&lt;/a&gt;. /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:71: 'notes': u'Some test notes\n\n### A 3rd level heading\n\n<strong>Some bolded text.</strong>\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow &lt;\n\n&lt;<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>&gt;\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:137: 'notes': u'Some test notes\n\n### A 3rd level heading\n\n<strong>Some bolded text.</strong>\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow &lt;\n\n&lt;<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>&gt;\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:447: 'notes': u'Some test notes\n\n### A 3rd level heading\n\n<strong>Some bolded text.</strong>\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow &lt;\n\n&lt;<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>&gt;\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/dictization.py:458: 'notes': u'Some test notes\n\n### A 3rd level heading\n\n<strong>Some bolded text.</strong>\n\n*Some italicized text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow &lt;\n\n&lt;<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>&gt;\n\n', /home/okfn/pyenv/src/ckan/ckan/tests/functional/api/base.py:178: assert '"ckan_url": "<a class="ext-link" href="http://test.ckan.net/package/annakarenina"><span class="icon">​</span>http://test.ckan.net/package/annakarenina</a>"' in msg, msg /home/okfn/pyenv/src/ckanclient/ckanclient/</span>init__.py:116: api e.g. <a class="ext-link" href="http://ckan.net/api"><span class="icon">​</span>http://ckan.net/api</a> rather than <a class="ext-link" href="http://ckan.net/api/rest"><span class="icon">​</span>http://ckan.net/api/rest</a>) /home/okfn/pyenv/src/ckanclient/ckanclient/<span class="underline">init__.py:261: :param base_location: default *<a class="ext-link" href="http://www.ckan.net/api"><span class="icon">​</span>http://www.ckan.net/api</a>* /home/okfn/pyenv/src/ckanclient/ckanclient/</span>init__.py:267: base_location = '<a class="ext-link" href="http://www.ckan.net/api"><span class="icon">​</span>http://www.ckan.net/api</a>' </p> <hr /> http://localhost/ticket/1085 http://localhost/ticket/1085 Report johnlawrenceaspden <p> as title. </p> http://localhost/ticket/1086 http://localhost/ticket/1086 Report wwaites <p> a simple api call that returns data like this: </p> <pre class="wiki">{ "version": ckan_software_version, "contact": { "name": "Some Admin", "mbox": "admin@exmaple.org" }, "description": "Site Description", "url": "http://canonical.name.ckan.net/" </pre> http://localhost/ticket/1087 http://localhost/ticket/1087 Report wwaites <p> in ckan/controllers/package.py around line 130 it does some strange things... </p> <p> perhaps replace with <a class="ext-link" href="https://github.com/wwaites/autoneg"><span class="icon">​</span>https://github.com/wwaites/autoneg</a> </p> <p> and handle redirection of these content types: </p> <pre class="wiki">application/rdf+xml application/turtle text/plain text/x-graphviz </pre> http://localhost/ticket/1088 http://localhost/ticket/1088 Report dread <p> (because if you forget, you get difficult to understand errors, and more than one person has tripped up on this) </p> http://localhost/ticket/1089 http://localhost/ticket/1089 Report dread <p> Default visitor roles in default config is reader, not anon_editor. </p> <p> Problem caused by changes in <a class="closed ticket" href="http://localhost/ticket/1066" title="enhancement: Default reader role too permissive (closed: fixed)">#1066</a> (released in 1.3.3) </p> <p> New installs will be affected, although simple to just increase permissions when the installer realises a visitor can't create packages. </p> <p> The solution to the config getting out of sync with the code like this is to not have the default_roles in the config - refer to the code in the configuration instructions. </p> http://localhost/ticket/1090 http://localhost/ticket/1090 Report johnlawrenceaspden <p> If I use my gmail openID to log into a CKAN instance, then my username is: </p> <p> <a class="ext-link" href="https://www.google.com/accounts/o8/id?id=AItOawnduohQ5RgXdPJKHiq-SIPbvCBqUaERuEQ"><span class="icon">​</span>https://www.google.com/accounts/o8/id?id=AItOawnduohQ5RgXdPJKHiq-SIPbvCBqUaERuEQ</a> </p> <p> This seems a bit odd. </p> http://localhost/ticket/1091 http://localhost/ticket/1091 Report kindly <p> The logic layer is a bit too api centric. Make the reusable parts separate in preparation for the wui refactor. </p> http://localhost/ticket/1092 http://localhost/ticket/1092 Report dread <p> CKAN gets its license list from a license service, which can be a local file, but is often the <a class="ext-link" href="http://licenses.opendefinition.org/2.0/ckan_original"><span class="icon">​</span>http://licenses.opendefinition.org/2.0/ckan_original</a> server. This server is currently flakey, but I think we only request the list on start up. The problem is we query it much more often than required. It is queried for every request to api/rest/licenses, and we are returning lots of 500 errors when the license server is timing out. </p> http://localhost/ticket/1093 http://localhost/ticket/1093 Report thejimmyg <p> Here are some proposed changes related to CKAN's authorization system - they aren't very big, but should provide for some forthcoming use cases including <a class="closed ticket" href="http://localhost/ticket/787" title="task: Auth API (closed: fixed)">#787</a>. </p> <p> Two man reasons for the changes are: </p> <ul><li>We have a completely refactored architecture now which introduces a logic layer. These Auth changes are designed to better support the way we work with that layer. </li></ul><ul><li>Different CKAN extension apps may need radically different authentication/authorisation so we need to allow whatever we have to be override-able. </li></ul><p> The first two changes revolve around the is_authorized method, which is called by the logic layer to ask whether a particular user (e.g. Bob) is allowed to do a certain action (e.g. edit) on a certain object (e.g. Package). </p> <ol><li>The first thing the is_authorized method is a hook to a plugin </li></ol><p> which *overrides* the current call with its own implementation (note: in previous discussions we have considered allowing a chain of plugins, no longer!) </p> <p> Reason: authorization can be completely delegated to another system (or partially) </p> <ol start="2"><li>is_authorized method currently takes (username, action, object) </li></ol><p> but for action=create_package, the object supplied is System, and for action=edit the object supplied is the package. Instead action should always be the string name of a function in the logic layer and object should always be the object passed to that function. This means our auth system is based around the actual actions we are performing (rather than a model them) and with the actual data that forms the action (rather than a related object). You never need a System object in this model. </p> <ol start="3"><li>Rename these two classes to better reflect what they are <ul><li><a class="missing wiki">AuthorizationGroup?</a> -&gt; <a class="missing wiki">UserGroup?</a> </li><li>Group -&gt; <a class="missing wiki">PackageGroup?</a> </li></ul></li></ol><ol start="4"><li>Rename the Editor role to <a class="missing wiki">PriveledgeUser?</a> since Editors sometimes can't edit. </li></ol><p> Although this sounds a bit radical we already have auth extensions. </p> <h2 id="Read-onlyCKANWebUI">Read-only CKAN Web UI</h2> <p> (Additional requirement from <a class="closed ticket" href="http://localhost/ticket/764" title="enhancement: Read-only CKAN Web UI (closed: duplicate)">#764</a>) </p> <p> Whilse using CKAN web interface, you are not tempted to edit stuff: </p> <ul><li>You know at all times this CKAN is read-only </li><li>All editing facilities are still seen but greyed-out with an indication why it is. </li></ul> http://localhost/ticket/1094 http://localhost/ticket/1094 Report kindly <p> We need a way to pass in schemas to the logic layer to deal with edge cases. </p> http://localhost/ticket/1095 http://localhost/ticket/1095 Report pudo <p> Many users of CKAN want to have their own instance without much effort. Setting these up in separate places is a maintenance nightmare, we should much rather have some tenant separation in core CKAN. Some ideas: </p> <ul><li>introduce model.Site and c.site <ul><li>site has: custom CSS, extra_template_path, title, languages list, package_form, group_form (all configured via web UI) </li></ul></li><li>Subdomain detector to activate sites. </li><li>use site in Authorizer instead of System, have a <a class="missing wiki">NullSite?</a> for global things </li><li>allow cross-site search </li><li>packages are in a list of sites, m:n rather than 1:n <ul><li>list of sites is string-based, can contain sites not in site table to express harvested external material which is not editable locally. </li></ul></li></ul> http://localhost/ticket/1096 http://localhost/ticket/1096 Report dread <p> The web interface has a sidebar (#primary) which should be hidden in some pages. This is for QA extension and useful for package new and edit pages. Must be compatible with DGU theme. </p> http://localhost/ticket/1097 http://localhost/ticket/1097 Report dread <p> Only tests with failing --ckan-migration fail, due to authz not being initialised. </p> http://localhost/ticket/1098 http://localhost/ticket/1098 Report johnlawrenceaspden <p> While playing with the authorization groups, trying to design tests, I found that it was necessary to log in as two different users with two different browsers. Often actions of one user would cause server errors in the other user's browser. </p> <p> I don't have a reproducible test case, but it happens fairly often so it shouldn't be too difficult to get one. </p> http://localhost/ticket/1099 http://localhost/ticket/1099 Report dread <p> Need some changes to pip-requirements files in release branches. </p> http://localhost/ticket/1100 http://localhost/ticket/1100 Report sebbacon <p> There's a stats plugin (e.g. at <a class="ext-link" href="http://trac.ckan.org/ticket/832"><span class="icon">​</span>http://trac.ckan.org/ticket/832</a>). </p> <p> Output from the googleanalytics plugin should append to that page, if the stats plugin is present. </p> <p> Possibly the stats plugin and the googleanalytics plugin should be merged? </p> <p> Finally, if the stats plugin is active, then a link to the stats page should be added to the main site footer. </p> http://localhost/ticket/1101 http://localhost/ticket/1101 Report johnlawrenceaspden <p> With the default test data created by </p> <p> paster db clean paster db init paster create-test-data </p> <p> going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy </p> <p> But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits. </p> <p> That isn't true on ckan.net, where searching seems to work. </p> http://localhost/ticket/1102 http://localhost/ticket/1102 Report johnlawrenceaspden <p> With the default test data created by </p> <p> paster db clean paster db init paster create-test-data </p> <p> going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy </p> <p> But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits. </p> <p> That isn't true on ckan.net, where searching seems to work. </p> http://localhost/ticket/1103 http://localhost/ticket/1103 Report johnlawrenceaspden <p> With the default test data created by </p> <p> paster db clean paster db init paster create-test-data </p> <p> going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy </p> <p> But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits. </p> <p> That isn't true on ckan.net, where searching seems to work. </p> http://localhost/ticket/1104 http://localhost/ticket/1104 Report nils.toedtmann <p> . </p> http://localhost/ticket/1105 http://localhost/ticket/1105 Report rgrp <p> Various bugs I've been encountering: </p> <ul><li>Autocomplete of tag names no longer works (no longer works on <a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>). Appears to be due to no longer have a routes for apiv2 (i'm seeing failing calls to: <a class="ext-link" href="http://ckan.net/apiv2/package/autocomplete?callback=callback&amp;incomplete=a"><span class="icon">​</span>http://ckan.net/apiv2/package/autocomplete?callback=callback&amp;incomplete=a</a>) </li><li>Incorrect url generated for API in page footer (e.g. <a class="ext-link" href="http://ckan.net/rest/get_api"><span class="icon">​</span>http://ckan.net/rest/get_api</a>) due to use of old 'rest' rather than new 'api' </li></ul><p> Latter issue was masked by existence of 'default' routes: </p> <pre class="wiki"> map.connect('/{controller}', action='index') map.connect('/:controller/{action}') map.connect('/{controller}/{action}/{id}') </pre><p> Having these is, I think, bad practice as it is better to be explicit and we should therefore remove asap. </p> <p> In addition I think we should be cautious about 'default' routes in core such as: </p> <pre class="wiki"> map.connect('/api/rest/:register', controller='api', action='list', conditions=dict(method=['GET']) ) </pre><p> As it makes it harder for extensions to introduce their own APIs (here one could perhaps add something at /api/rest/{my-object} but only by using before_map rather than after_map). </p> http://localhost/ticket/1106 http://localhost/ticket/1106 Report rgrp <p> Currently autocomplete method on package controller. This method should be in API (like other autocomplete methods). </p> <p> Will need to update client code (just forms atm I think). </p> http://localhost/ticket/1107 http://localhost/ticket/1107 Report pudo <p> CKAN looks a bit aged, it should be styled more modernly and some elements could be re-arranged: </p> <ul><li>Collect user info in top bar </li><li>re-add the logo to ckan.net </li><li>Remove tags from main menu, replace with /sitemap.xml </li></ul><p> Inpiration: </p> <p> quora.com, github.com, Google Projects, Google Refine, etc. </p> <h2 id="CKAN.netorCKANgeneraltheme">CKAN.net or CKAN general theme?</h2> <p> To be decided. Suggest we start with ckan.net specific and then backwards integrate (?). Existing ckan.net theme repo: </p> <p> <a class="ext-link" href="https://bitbucket.org/okfn/ckan-ckan.net"><span class="icon">​</span>https://bitbucket.org/okfn/ckan-ckan.net</a> </p> http://localhost/ticket/1108 http://localhost/ticket/1108 Report kindly <p> This is a regression that happened after refactoring the api. </p> <p> It was shown by </p> <p> <a class="ext-link" href="http://pastebin.com/2v7QasZy"><span class="icon">​</span>http://pastebin.com/2v7QasZy</a> </p> http://localhost/ticket/1109 http://localhost/ticket/1109 Report kindly <p> We need to see what areas of ckan are slow. </p> http://localhost/ticket/1110 http://localhost/ticket/1110 Report lucychambers <p> Write CKAN FAQ (Basis can be: <a class="ext-link" href="http://wiki.ckan.net/FAQ"><span class="icon">​</span>http://wiki.ckan.net/FAQ</a>) </p> <p> Post preliminary questions on: </p> <p> <a class="ext-link" href="http://notebook.okfn.org/"><span class="icon">​</span>http://notebook.okfn.org/</a> </p> http://localhost/ticket/1111 http://localhost/ticket/1111 Report dread <pre class="wiki">GET api/search/package?q= </pre><p> returns all packages. This is so you can filter them e.g. by openness, which is not currently possible. </p> http://localhost/ticket/1112 http://localhost/ticket/1112 Report kindly <p> Lists are being converted to unicode and then translated into a json when getting from the api. </p> http://localhost/ticket/1113 http://localhost/ticket/1113 Report dread <p> To see what lexemes are generated for debug purposes. </p> http://localhost/ticket/1114 http://localhost/ticket/1114 Report johnlawrenceaspden <p> If you've got edit permission on an authzgroup, then you can change its name to be the same as another existing authzgroup. </p> <p> This causes some strange UI effects at worst, and probably causes worse problems somewhere else. </p> <p> Is there any reason why changing the names of existing authzgroups should be allowed? And if so, name collisions should presumably be guarded against in both the name-changing and creation functions </p> http://localhost/ticket/1115 http://localhost/ticket/1115 Report dread <p> filter_by_openness and filter_by_downloadable options don't work when specified as URI parameters. (They do work in qjson parameters) </p> http://localhost/ticket/1116 http://localhost/ticket/1116 Report nils.toedtmann <p> ... otherwise the scripts fails. </p> http://localhost/ticket/1117 http://localhost/ticket/1117 Report johnlawrenceaspden <p> I'm finding that if I try to take an action with insufficient credentials from a test then I often (but not always) get a 401 error, whereas in the browser I get redirected to the login page. </p> <p> It's a bit worrying that the program in its test environment doesn't behave like it does in the browser. </p> http://localhost/ticket/1118 http://localhost/ticket/1118 Report rgrp <p> Previous work in <a class="closed ticket" href="http://localhost/ticket/877" title="enhancement: File upload in WUI (extension) (closed: fixed)">#877</a> and <a class="closed ticket" href="http://localhost/ticket/879" title="enhancement: Storage auth API (closed: fixed)">#879</a> + <a class="closed ticket" href="http://localhost/ticket/853" title="enhancement: Client upload to storage without having primary storage keys (closed: fixed)">#853</a> (storage API). In this ticket: </p> <ul><li>Improve authorization </li><li>Establish convention for laying out files on disk </li><li>Add documentation </li><li>Fix bugs with <a class="closed ticket" href="http://localhost/ticket/879" title="enhancement: Storage auth API (closed: fixed)">#879</a> (does not currently work -- boto may have changed) </li></ul> http://localhost/ticket/1119 http://localhost/ticket/1119 Report tsm <p> Tags could/should have an Atom feed. This would mean that every edit to relevant packages could be easily monitored. See <a class="missing changeset" title="No default repository defined">[1]</a>. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001162.html</a> </p> http://localhost/ticket/1120 http://localhost/ticket/1120 Report dread <p> It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is displayed in the package edit form. </p> <p> Example: Package <a class="ext-link" href="http://ckan.net/package/hbz_unioncatalog"><span class="icon">​</span>http://ckan.net/package/hbz_unioncatalog</a> in the API the extra value as a list: </p> <pre class="wiki">"extras": {"publishingInstitution": ["http://lobid.org/organisation/DE-605", "http://lobid.org/organisation/DE-290"... </pre><p> yet when you edit the package it loses all the quotes and brackets: <a class="ext-link" href="http://ckan.net/package/edit/hbz_unioncatalog"><span class="icon">​</span>http://ckan.net/package/edit/hbz_unioncatalog</a> {{{<a class="ext-link" href="http://lobid.org/organisation/DE-605http://lobid.org/organisation/DE-290"><span class="icon">​</span>http://lobid.org/organisation/DE-605http://lobid.org/organisation/DE-290</a>... }}} so when you save the package, the list is mangled into a bad string. </p> http://localhost/ticket/1121 http://localhost/ticket/1121 Report dread <p> It is possible to use the CKAN API to insert JSON format data into package extra values, but this data is not found on searching. </p> <p> Full text from Pascal: </p> <pre class="wiki">we encountered a Problem concerning accessing Arrays/Lists. curl -XGET 'http://ckan.net/api/rest/package/hbz_unioncatalog' will get you amongst others: "extras": {"publishingInstitution": "[u'http://lobid.org/organisation/DE-605', u'http://lobid.org/organisation/DE-290', u'http://lobid.org/organisation/DE-38M', u'http://lobid.org/organisation/DE-98', u'http://lobid.org/organisation/DE-38', u'http://lobid.org/organisation/DE-Kn41', u'http://lobid.org/organisation/DE-82', u'http://lobid.org/organisation/DE-107', u'http://lobid.org/organisation/DE-929', u'http://lobid.org/organisation/DE-Zw1', u'http://lobid.org/organisation/DE-832']"} but if I try to query this: wget 'http://ckan.net/api/search/package?q=lobid&amp;publishingInstitution="http://lobid.org/organisation/DE-605"' I get only two packages, among the package "hbz_unioncatalog" is missing. (These two packages have only one value for "publishingInstitution"). The "extra/publishingInstitution"-Array was uploaded through a "curl -XPUT ... "extras": { "publishingInstitution":[ "http://lobid.org/organisation/DE-605", "http://lobid.org/organisation/DE-290", "http://lobid.org/organisation/DE-38M", "http://lobid.org/organisation/DE-98", "http://lobid.org/organisation/DE-38", "http://lobid.org/organisation/DE-Kn41", "http://lobid.org/organisation/DE-82", "http://lobid.org/organisation/DE-107", "http://lobid.org/organisation/DE-929", "http://lobid.org/organisation/DE-Zw1", "http://lobid.org/organisation/DE-832" ] }, ... </pre> http://localhost/ticket/1122 http://localhost/ticket/1122 Report nils.toedtmann <p> ... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too. </p> <p> Currently, <a class="ext-link" href="http://apt-alpha.ckan.org/debian"><span class="icon">​</span>http://apt-alpha.ckan.org/debian</a> only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386. </p> <p> We would need this in order to migrate the community instances to a packaged based CKAN. </p> <p> Rufus, pls prioritise. </p> http://localhost/ticket/1123 http://localhost/ticket/1123 Report nils.toedtmann <p> python-ckanext-solr is already available in <a class="ext-link" href="http://apt-alpha.ckan.org/datanl-dev"><span class="icon">​</span>http://apt-alpha.ckan.org/datanl-dev</a>, but not yet in <a class="ext-link" href="http://apt-alpha.ckan.org/debian"><span class="icon">​</span>http://apt-alpha.ckan.org/debian</a> (that is why we had to [pip-install it for DataGM). Please push into main repo. </p> http://localhost/ticket/1124 http://localhost/ticket/1124 Report nils.toedtmann <p> The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs. </p> <p> Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package. </p> http://localhost/ticket/1125 http://localhost/ticket/1125 Report dread <p> I'm not completely clear what the use case is for loading the error page in this way, but somehow original_request is None and that creates an unnecessary exception with the logic refactor. </p> <pre class="wiki">http://ckan.net/error/document?__cache=39020485 ... Module ckan.controllers.error:29 in document &lt;&lt; original_response = request.environ.get('pylons.original_response') # Bypass error template for API operations. if original_request.path.startswith('/api'): return original_response.body # Otherwise, decorate original response with error template. &gt;&gt; if original_request.path.startswith('/api'): AttributeError: 'NoneType' object has no attribute 'path' </pre> http://localhost/ticket/1126 http://localhost/ticket/1126 Report sebbacon <p> <strong>Proposer:</strong> Seb Bacon<br /> <strong>Seconder:</strong> Rufus Pollock<br /> <br /> </p> <h2 id="Abstract">Abstract</h2> <p> When adding major new features to CKAN, a longer, more formal discussion will improve software design quality and documentation, better engage the wider community, and ensure the core team are up to date with latest developments. </p> <p> I propose a formal process (CREP -- CKAN Revision and Enhancement Proposal) for making this happen. </p> <h2 id="TheProblem">The Problem</h2> <p> The current workflow for introducing major new features into CKAN is very informal, typically based around one person's great idea, which they've discussed with one or two other people in the team. The originator of the idea is typically the only person with access to all the input they've had through such discussions. Often, the only location of this information is in that person's head. </p> <p> However, there is a lot of experience embodied in the CKAN community which should be drawn on before making large design decisions. This will lead to better software. Additionally, building consensus in the community around a proposal before implementation ensures positive community engagement and buy-in to new features, making them more likely to be a success. </p> <p> We aren't great at documenting new features. Documentation after coding is complete is an unrewarding experience for most programmers. Requiring skeleton documentation before code is written is a good discipline that can form the basis of better documentation in the future (e.g. by a writer rather than a programmer). </p> <h2 id="Specification">Specification</h2> <p> Minor features don't require a CREP, and can just be entered in the issue tracking system as a bug or feature. As a rule of thumb, a feature is major if it will take more than a day to implement, or is likely to involve matters of opinion in its design. </p> <p> A developer may decide that a CREP is too formal and long-winded. The decision to write a CREP is at at their discretion; however, new features MUST always be proposed via email, even if this is just a couple of sentences. </p> <p> If a feature requires a CREP, the proposer should find a seconder for their idea. This sanity check step happens before a CREP is written to ensure at least the possibility of consensus on the CREP. </p> <p> Next the proposer should write a CREP, starting by copying and pasting the <a class="ext-link" href="http://wiki.ckan.net/CKAN_Revision_and_Enhancement_Proposals"><span class="icon">​</span>template on the wiki</a> into a new Trac ticket. This will be with a status of "new" and Type of "CREP". The proposer should notify the <a class="ext-link" href="http://lists.okfn.org/mailman/listinfo/ckan-dev"><span class="icon">​</span>ckan-dev</a> mailing list, and possibly the <a class="ext-link" href="http://lists.okfn.org/mailman/listinfo/ckan-discuss"><span class="icon">​</span>ckan-discuss</a> list for less technical CREPs. </p> <p> The draft can be discussed via email, verbally, or via the trac ticket. In any case, it is the proposer's responsibility to keep the CREP updated to reflect the current consensus. </p> <p> Once consensus has been reached, the ticket should be marked with the "accepted" status and assigned to a CKAN release milestone. </p> <p> When an accepted CREP has been implemented, it should be resolved as "fixed". </p> <p> If no consensus can be reached on a draft CREP, or for some reason an accepted CREP doesn't get completed, it should be marked as or "wontfix". </p> <p> If a completed CREP becomes obsolete, it should be marked as "invalid", with a note pointing to the obsoleting ticket(s) </p> <h2 id="Whydoitthisway">Why do it this way</h2> <p> Given the distributed nature of the core team plus other volunteers, some kind of written procedure is necessary to ensure a fully documented and discussed proposal. </p> <p> The idea of "Enhancement Proposals" which can be semi-formally proposed and discussed prior to implementation is common in the Open Source world (<a class="ext-link" href="http://www.python.org/dev/peps/pep-0001/"><span class="icon">​</span>PEPs</a>, <a class="ext-link" href="http://dep.debian.net/"><span class="icon">​</span>DEPs</a>, <a class="ext-link" href="http://plone.org/documentation/glossary/plip"><span class="icon">​</span>PLIPs</a>, to name three). </p> <p> Existing historic proposals exist, called CEPs. The proposed system is called CREP (CKAN revision or enhancement proposal) to disambiguate it from the legacy proposals, and from the delicious fungus <em>Boletus Edulis</em>. </p> <p> Giving a formal structure to the proposal is useful as it gives the community a means to identify a CREP that's not had sufficient thought or discussion. An informal email thread can easily be lost and important questions (such as backwards compatibility) overlooked. The use of the proposed template empowers any community member to ask the proposer to expand on rationale, deliverables, etc. </p> <p> The structure chosen is somewhere between Debian's and Plone's. It aims to give a structure to the debate, a clear start at documentation, and also prompt some thinking about implementation and timescales. </p> <p> All this policy about structure should not be construed as mandatory. In particular, the later fields in the CREP template regarding Implementation Plan may be omitted if the author doesn't find them helpful. </p> <p> Some projects (e.g. Debian) keep their enhancement proposals in a versioning repository; others (e.g. Plone) keep them in an issue tracking system. Trac is proposed for CKAN because we already use it for small feature proposals and for team planning. It seems unlikely that change tracking on an individual CREP will be useful; a CREP that changes sufficiently from its original form should probably be marked "obselete" and a new CREP started. Using an issue tracking system also means we can easily track CREPs by state. </p> <h2 id="BackwardsCompatibility">Backwards Compatibility</h2> <p> Some [<a class="ext-link" href="https://bitbucket.org/okfn/ceps/src/76b274888bcf/cep/"><span class="icon">​</span>https://bitbucket.org/okfn/ceps/src/76b274888bcf/cep/</a> legacy enhancement proposals], called CEPs, have previously been started. </p> <p> They are currently all marked as "active". Any which require discussion should be altered by the proposer to match the new CREP specification and submitted to trac. The original CEP should be updated with a banner at the top pointing a reader to the new CREP. </p> <p> Any that are now obselete should be clearly marked as such in a banner at the top, pointing a reader to the trac for new CREPs. </p> <h2 id="Implementationplan">Implementation plan</h2> <h3 id="Deliverables">Deliverables</h3> <ul><li>This CREP, agreed </li><li>Support for proposed statuses in Trac </li><li>Canned reports for listing CREPs in Trac </li></ul><h3 id="Risksandmitigations">Risks and mitigations</h3> <ul><li>That this CREP is agreed, but rarely acted on. This risk can be mitigated by nominating a CREP champion in the community or core team, whose job it is to say "where's the CREP for that?" and generally own the quality of CREPS </li></ul><h3 id="Participants">Participants</h3> <p> Seb Bacon: as current Documentation Czar (May 2011), responsible for ensuring CREPs are up to date. </p> <h3 id="Progress">Progress</h3> <p> This document is the entire proposal. </p> http://localhost/ticket/1127 http://localhost/ticket/1127 Report dread <p> Upload to ckan.net: <a class="ext-link" href="https://sites.google.com/site/scotlandsdata/"><span class="icon">​</span>https://sites.google.com/site/scotlandsdata/</a> </p> http://localhost/ticket/1128 http://localhost/ticket/1128 Report kindly <p> <strong>Proposer:</strong> David Raznick<br /> </p> <h2 id="Abstract.">Abstract.</h2> <p> We are trying to achieve these goals. </p> <ul><li>To get people involved with making edits to CKAN metadata. </li><li>To have an ownership model as to who can moderate and validate these changes </li><li>To not put too huge a burden on these owners. </li></ul><p> In order to achieve this, a feature which lets anyone edit a package but only let the moderator/owner accept it. The moderator should be able to look at a list of changes and accept the ones that </p> <p> This cep is not about 'if' we need such a feature, it is about 'how' we go about implementing it. Another cep may needed for the 'if' case. </p> <h2 id="TheProblem">The Problem</h2> <p> We need the following to be possible. </p> <ul><li>Storing revision of objects that are not the current active one. </li><li>A way of the user viewing past revisions. </li><li>Accessing not only the history of a particular object but also of related objects at that time. i.e If a resource related to a package changes we need a way to see this when looking at the package. </li><li>A robust way of doing this in the face of database schema changes. </li><li>Make sure database queries are quick. </li></ul><h2 id="Solutions.">Solutions.</h2> <ol><li> Store the whole dictization of the package and all its related objects every time you change anything in its dictized representation and only save to the database proper if accepted. </li></ol><blockquote> <p> Pros </p> </blockquote> <ul><li>Easy to implement, we already have a preview which makes the dictized form of a package without actually saving it. This will just need to be persisted in some way. </li><li>Fast retrieval. </li><li>Potential to store a branching revision tree of changes. </li></ul><blockquote> <p> Cons </p> </blockquote> <ul><li>No easy way to remake the dictized packages historically or if there is an there a change in the way we represent packages, i.e schema changes. </li><li>Will only work for the particular objects we decide to store these changes for. </li><li>Stores a lot of repeated information </li></ul><ol start="2"><li> Write specialized queries for every read of the database looking only at the revision tables. </li></ol><p> This method requires there to be a change in the way we use VDM, so that we manage statefulness ourselves. We will need to add other states such as 'waiting for approval'. </p> <blockquote> <p> Pros </p> </blockquote> <ul><li>No specialized storage required </li><li>Only need to change queries when schema changes </li><li>Can be made to work easily for other objects </li></ul><blockquote> <p> Cons </p> </blockquote> <ul><li>Slower query time on read, as even looking at the last active package will need to do a fairly complicated query. </li></ul><h2 id="Implementationdetails.">Implementation details.</h2> <p> 1. </p> <blockquote> <p> A new table with columns id, user, package_id, timestamp, revision_id, parent_id, dictized_package. revision_id should be null unless it is actually persisted to the database. parent_id is the id that this package_dict was changed from. </p> </blockquote> <blockquote> <p> We could store only the diffs of the dictized_package as long as we assure that everything inside the json is stably sorted, this will make getting the historical data out slower. </p> </blockquote> <blockquote> <p> Getting out the history of the dictized packages is an intensive task, as it will require replaying the whole history of all the changes and creating the dict for each change. This re-caching will need to be redone for every change we make to dictized representation of a package. </p> </blockquote> <p> 2. </p> <blockquote> <p> Every normal packages read needs to look at the revision table to see the last accepted change in the dictized representation of the package. We also need to way to get what the dictized representation of the package was like at any point of its revision history. This querying is non-trivial in sql. </p> </blockquote> <h3 id="Participants">Participants</h3> <p> David Raznick to do it. </p> <h2 id="Progress.">Progress.</h2> <p> Decided to go with option 2. However we will change the revisioning system to be like the schema attached. This gets rid of difficult querying problems caused by querying the revision tables by adding an end date, meaning you can do range queries. </p> <p> The better and more normalized version of a revisioning system is outlined <a class="ext-link" href="https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&amp;authkey=CJfd8vsB"><span class="icon">​</span>https://docs.google.com/drawings/d/1Y7nMgVsrs081Pame2RdbZHlCAlV33ddTZ8VAsab1j-0/edit?hl=en_GB&amp;authkey=CJfd8vsB</a>. We will be a step closer to that, with this change, but we will keep the current vdm more or less, intact. </p> http://localhost/ticket/1129 http://localhost/ticket/1129 Report lucychambers <p> Send users to FAQ first time on CKAN </p> http://localhost/ticket/1130 http://localhost/ticket/1130 Report dread <p> Example request: </p> <pre class="wiki">http://nl.ckan.net/api/2/search/package?q=delft&amp;order_by=&amp;offset=&amp;limit=&amp;tags= </pre><p> Gives 500 error: </p> <pre class="wiki">&lt;type 'exceptions.ValueError'&gt;: invalid literal for int() with base 10: '' </pre> http://localhost/ticket/1131 http://localhost/ticket/1131 Report johnlawrenceaspden <p> Trying to run the tests in test_authz.py with </p> <p> $ nosetests --ckan ckan/tests/functional/test_authz.py results in no tests being run: </p> <hr /> <p> Ran 0 tests in 0.840s </p> <p> OK (SKIP=3) </p> http://localhost/ticket/1132 http://localhost/ticket/1132 Report johnlawrenceaspden <p> It appears that the command </p> <p> $ paster rights add russianfan admin warandpeace </p> <p> has no effect, even though </p> <p> $ paster rights remove russianfan admin warandpeace </p> <p> works fine. This may be specific to something I've done, could someone confirm? </p> <p> If it's the case more generally, then I'm assuming this behaviour is untested? Tests should probably be added. </p> http://localhost/ticket/1133 http://localhost/ticket/1133 Report amercader <p> <strong>Proposer</strong>: Adrià Mercader </p> <h2 id="Abstract">Abstract</h2> <p> The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to: </p> <ul><li>Expose their type and other details so they can be used internally and on the UI. </li><li>Define configuration settings for particular harvester instances. </li></ul><h2 id="TheProblem">The Problem</h2> <h3 id="Harvesterdescription">Harvester description</h3> <p> The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current <tt>get_type</tt> method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. "CSW Server"), making it error prone. </p> <h3 id="Arbitraryconfiguration">Arbitrary configuration</h3> <p> In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the <tt>IHarvester</tt> interface and calls the appropiate methods for the current stage (<tt>gather_stage</tt>,<tt>fetch_stage</tt>,<tt>import_stage</tt>). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code). </p> <h2 id="Specification">Specification</h2> <h3 id="Harvesterdescription1">Harvester description</h3> <p> Harvesters will need to provide the following information so the UI form can be built: </p> <ul><li>name: machine-readable name (e.g. "waf"). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester. </li><li>title: human-readable name (e.g. "Web Accessible Folder (WAF)"). This will appear in the form's select box. </li><li>description: a description of what the harvester does (e.g. "A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents"). This will appear on the form as a guidance to the user. </li></ul><p> The way to provide it will be an <tt>info</tt> method that all harvesters must implement, which will return a dictionary with the previous elements: </p> <pre class="wiki"> { 'name': 'csw', 'title': 'CSW Server', 'description': 'A server that implements OGC's Catalog Service for the Web (CSW) standard' } </pre><h3 id="Arbitraryconfiguration1">Arbitrary configuration</h3> <p> As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called <tt>config</tt> in the database) (Maybe using <a class="missing wiki">JsonType?</a>?). </p> <p> This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an "Add a configuration flag" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an "Advance options" section. </p> <h2 id="Whydoitthisway">Why do it this way</h2> <h3 id="Harvesterdescription2">Harvester description</h3> <p> The <tt>info</tt> method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface. </p> <h3 id="Arbitraryconfiguration2">Arbitrary configuration</h3> <p> There is an already existing <tt>config</tt> field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime. </p> <h2 id="Implementationplan">Implementation plan</h2> <h3 id="Deliverables">Deliverables</h3> <h3 id="Risksandmitigations">Risks and mitigations</h3> <p> The highest risk on the harvesters <tt>info</tt> method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI. </p> <h3 id="Participants">Participants</h3> <p> Adrià Mercader to do it. </p> <h3 id="Progress">Progress</h3> <p> None yet. </p> http://localhost/ticket/1134 http://localhost/ticket/1134 Report rgrp <p> Move to Changeset model for vdm. </p> <p> A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned. </p> <p> This change would also incorporate significant simplication of vdm. </p> http://localhost/ticket/1135 http://localhost/ticket/1135 Report rgrp <p> When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that <a class="missing wiki">SessionExtension?</a> exists we should use it. We can also follow the existing SQLAlchemy recipe: &lt;<a class="ext-link" href="http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects"><span class="icon">​</span>http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects</a>&gt; </p> http://localhost/ticket/1136 http://localhost/ticket/1136 Report rgrp <p> Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is <em>required</em> because, atm, revision objects have FKs to continuity objects. </p> <p> This ticket proposes the following changes: </p> <p> NB: this could be limited just to case of join tables (leaving state stuff on other tables) </p> <ul><li>Remove FKs from revision to continuity (or allow for them to be nullable). <ul><li>We could just limit this to m2m stuff </li></ul></li><li>Delete of an object leads to: <ul><li>Deletion of continuity object </li><li>Adding an entry in revision table with state set to deleted (we retain state on revision table) </li></ul></li></ul><p> If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain "active" relationships. </p> <p> If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state). </p> <p> The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects). </p> http://localhost/ticket/1137 http://localhost/ticket/1137 Report johnlawrenceaspden <p> For Authorization Groups, if you have admin privileges you see view, edit and authz tabs, and if you don't have the necessary privileges you only see the view tab. </p> <p> For Packages, you see all tabs whatever your permissions, so there's a link you can click on which will redirect you to the login page. </p> http://localhost/ticket/1138 http://localhost/ticket/1138 Report lucychambers <p> Take screenshots of existing ckan instances esp those mentioned &lt;<a class="ext-link" href="http://wiki.ckan.net/Theming"><span class="icon">​</span>http://wiki.ckan.net/Theming</a>&gt; and put on flickr in ckan or ckan-theme group so we can create a gallery ... (both to illustrate theming but also to show ckan instances that are around -- could add to <a class="ext-link" href="http://wiki.ckan.net/Instances"><span class="icon">​</span>http://wiki.ckan.net/Instances</a>) </p> http://localhost/ticket/1139 http://localhost/ticket/1139 Report dread <p> To reproduce: </p> <ol><li><tt>paster db init</tt> </li><li><tt>paster create-test-data</tt> </li><li>In Web UI: create new group 'tilo', which includes package 'annakarenina' </li><li><tt>curl http://localhost:5000/api/search/package?groups=tilo</tt> results in 0 results (WRONG) </li><li><tt>paster search-index rebuild</tt> </li><li><tt>curl http://localhost:5000/api/search/package?groups=tilo</tt> results in 1 result </li></ol> http://localhost/ticket/1140 http://localhost/ticket/1140 Report johnglover <p> <strong>Proposer</strong>: John Glover<br /> <strong>Seconder</strong>: James Gardner </p> <h2 id="Abstract">Abstract</h2> <p> We are trying to achieve these goals: </p> <ul><li>To get people involved with making edits to CKAN metadata. </li><li>To have an ownership model as to who can moderate and validate these changes </li><li>To not put too huge a burden on these owners. </li></ul><p> This feature allows anyone to edit a package and create a new revision, but requires an owner/moderator to approve a revision before it is are made "official". </p> <p> There have been a lot of discussions around the revisioning system side of this ticket (CREP 0002) and I think these are now largely resolved. We now want to discuss the user interface. </p> <h2 id="TheProblem">The Problem</h2> <p> We require the following functionality: </p> <ul><li>Allow a group of changes to be stored as a new revision. </li><li>Allow a linear stack of "community" revisions. </li><li>Provide a way for the editor and moderator to compare previous revisions to the current one. </li><li>When a moderator approves a change it creates a new revision flagged "moderated" (this is analogous to a merge commit) </li><li>Provide a way for the editor and moderator comment on revisions if necessary. </li></ul><p> Extra features: </p> <ul><li>Need a way to summarise the changes (as part of the preview perhaps) </li><li>Sysadmin needs to purge a revision completely </li></ul><h2 id="Specification">Specification</h2> <h3 id="UIUX">UI/UX</h3> <p> UI Mockup: </p> <p> <a style="padding:0; border:none" href="http://localhost/attachment/ticket/1141/ModeratedEdits2.png"><img src="http://localhost/raw-attachment/ticket/1141/ModeratedEdits2.png" /></a> </p> <p> Revisions: </p> <ul><li>Revisions are per package rather than per field. </li><li>Internally CKAN has separate revisions for resources, extras and package metadata. From a user's point of view this could be confusing to expose, so everything that they see on a package form when they hit save is a single revision. </li></ul><p> On the Edit page: </p> <ul><li>We have a panel on the right, listing all the revisions with the current moderated one selected. Moderated revisions are highligted in some way (red and bold?). </li><li>The values displayed in the form are by default populated from the latest revision (whether community or moderated) </li><li>Under each field is a "shadow", showing the value of the field in the revision selected in the panel, if it is different from the value in the field. By default the shadow values are populated from the latest moderated revision which is the one selected in the revision panel by default too. </li><li>When you change the value of a field, a shadow may appear or disappear accordingly. If they disappear a box saying that they are the same replaces it </li><li>If you want to edit values from a previous revision, you first select that revision to get the shadows populated. There is a button named "Replace fields with values from this revision" under the revision list. You click this, a warning pops up and then you say "Yes". You then select the moderated revision again. </li><li>We also allow package comments the same way as the todo extension works at the moment. Additionally, we need to be able to differentiate between what the moderator wrote and what a community member wrote, and so we may need to make a small change to the todo extension to facilitate this. </li><li>In addition to package comments, each revision will have a revision log (analogous to a commit message). </li></ul><h3 id="TechnicalDetails">Technical Details</h3> <ul><li>This CREP will result in a new CKAN extension. </li><li>It depends heavily on the new revisioning system (CREP0002), some of the details of which are yet to be finalised. </li><li>This CREP therefore requires working closely with David Raznick to come up with an API that the UI AJAX calls can use. </li><li>We will then use suitable test data to mimic these API calls until CREP0002 is ready. </li></ul><h2 id="Whydoitthisway">Why do it this way</h2> <p> This hopefully provides a clear and consistent mechanism allowing both a community member to make new revisions and a moderator to view and approve revisions, with largely the same UI/UX. </p> <h2 id="Implementationplan">Implementation plan</h2> <h3 id="Deliverables">Deliverables</h3> <p> A new CKAN extension, consisting of: </p> <ul><li>Code: Python, HTML, CSS, Javascript </li><li>Unit tests </li><li>Localization </li><li>Documentation </li></ul><h3 id="Participants">Participants</h3> <p> John Glover to do it. </p> <h3 id="Progress">Progress</h3> <p> John has implemented the bulk of this UI. Just some things to tidy up before it is complete: </p> <ul><li>Genshi stream filters to be updated with CKAN 1.5 / 1.5.1 templates </li><li>history_ajax / read_ajax to be replaced with calls to Action API (or Util REST API) </li></ul><p> I've split these two off into a new ticket <a class="new ticket" href="http://localhost/ticket/1604" title="enhancement: Get ckanext-moderatededits working with CKAN 1.5+ templates (new)">#1604</a>. </p> <h3 id="RelatedProgress">Related Progress</h3> <p> The Todo extension is written and available at: <a class="ext-link" href="https://bitbucket.org/johnglover/ckanext-todo"><span class="icon">​</span>https://bitbucket.org/johnglover/ckanext-todo</a>. </p> <p> In the section 'The Problem', under extra features, we mention a need for the sysadmin to be able to purge a revision already. This is already done. </p> <h3 id="Seealso">See also</h3> <p> <a class="closed ticket" href="http://localhost/ticket/1129" title="CREP: CREP0002: Moderated Edits (closed: fixed)">#1129</a> Backend work </p> http://localhost/ticket/1141 http://localhost/ticket/1141 Report rgrp <p> Child tickets: </p> <ul><li><a class="assigned ticket" href="http://localhost/ticket/1041" title="enhancement: Start Using the CKAN Wiki for Tutorial-style documentation (assigned)">#1041</a> Start Using the CKAN Wiki for Tutorial-style documentation </li><li><a class="closed ticket" href="http://localhost/ticket/1192" title="task: Convert CKAN Sphinx docs into admin/reference manual (closed: fixed)">#1192</a> Convert CKAN Sphinx docs into admin/reference manual </li></ul><p> Previous super ticket (from 3m ago): <a class="ext-link" href="http://trac.ckan.org/ticket/927"><span class="icon">​</span>http://trac.ckan.org/ticket/927</a> </p> <ul><li>CKAN 1-page overview (for enterprise and for data hackers) </li><li>Administrator's Guide (including install) </li><li>Extensions Guide </li><li>Separate CKAN.net info from Software Documentation (?) </li></ul><p> Also now a wiki page with more detail: <a class="ext-link" href="http://wiki.ckan.net/Documentation_Plans"><span class="icon">​</span>http://wiki.ckan.net/Documentation_Plans</a> </p> <h2 id="MinorItems">Minor Items</h2> http://localhost/ticket/1142 http://localhost/ticket/1142 Report dread <ul><li>Ensure we don't include deleted packages in the stats </li><li>Some visual improvements: <ul><li>Number of packages: <ul><li>fix x axis to start at first revision </li><li>fix y axis to start at zero </li></ul></li><li>Fix problem with legend to 'Revisions to packages' graph. </li><li>Hide sidebar in the template, so it isn't right-cropped in the DGU theme </li></ul></li><li>Add some testing of the stats lib - results of basic stats </li></ul> http://localhost/ticket/1143 http://localhost/ticket/1143 Report timmcnamara <p> DSPL, the Dataset Publishing Language, is being promoted by Google for its "Google Public Data Explorer" system. It is an XML format with metadata. </p> <p> The format is described on the <a class="ext-link" href="http://code.google.com/apis/publicdata/docs/developer_guide.html"><span class="icon">​</span>developer docs ofthe Google Code site</a>. </p> <p> Google provides a <a class="ext-link" href="http://code.google.com/apis/publicdata/docs/dsplgen.html"><span class="icon">​</span>Python script which reads CSV data and generates DSPL</a> </p> <p> Sample from <a class="ext-link" href="http://code.google.com/apis/publicdata/docs/dspl_sample.html"><span class="icon">​</span>http://code.google.com/apis/publicdata/docs/dspl_sample.html</a>: </p> <div class="code"><pre> <span class="cp">&lt;?xml version="1.0" encoding="UTF-8"?&gt;</span> <span class="nt">&lt;dspl</span> <span class="na">xmlns=</span><span class="s">"http://schemas.google.com/dspl/2010"</span> <span class="na">xmlns:geo=</span><span class="s">"http://www.google.com/publicdata/dataset/google/geo"</span> <span class="na">xmlns:geo_usa=</span><span class="s">"http://www.google.com/publicdata/dataset/google/geo/us"</span> <span class="na">xmlns:time=</span><span class="s">"http://www.google.com/publicdata/dataset/google/time"</span> <span class="na">xmlns:quantity=</span><span class="s">"http://www.google.com/publicdata/dataset/google/quantity"</span> <span class="na">xmlns:entity=</span><span class="s">"http://www.google.com/publicdata/dataset/google/entity"</span><span class="nt">&gt;</span> <span class="nt">&lt;import</span> <span class="na">namespace=</span><span class="s">"http://www.google.com/publicdata/dataset/google/time"</span><span class="nt">/&gt;</span> <span class="nt">&lt;import</span> <span class="na">namespace=</span><span class="s">"http://www.google.com/publicdata/dataset/google/quantity"</span><span class="nt">/&gt;</span> <span class="nt">&lt;import</span> <span class="na">namespace=</span><span class="s">"http://www.google.com/publicdata/dataset/google/entity"</span><span class="nt">/&gt;</span> <span class="nt">&lt;import</span> <span class="na">namespace=</span><span class="s">"http://www.google.com/publicdata/dataset/google/geo"</span><span class="nt">/&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>My statistics<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value&gt;</span>Some very interesting statistics about countries<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;url&gt;</span> <span class="nt">&lt;value&gt;</span>http://www.stats-bureau.com/mystats/info.html<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/url&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;provider&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>Bureau of Statistics<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;url&gt;</span> <span class="nt">&lt;value&gt;</span>http://www.stats-bureau.com<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/url&gt;</span> <span class="nt">&lt;/provider&gt;</span> <span class="nt">&lt;topics&gt;</span> <span class="nt">&lt;topic</span> <span class="na">id=</span><span class="s">"geography"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;&lt;value&gt;</span>Geography<span class="nt">&lt;/value&gt;&lt;/name&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;/topic&gt;</span> <span class="nt">&lt;topic</span> <span class="na">id=</span><span class="s">"social_indicators"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;&lt;value&gt;</span>Social indicators<span class="nt">&lt;/value&gt;&lt;/name&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;topic</span> <span class="na">id=</span><span class="s">"population_indicators"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;&lt;value&gt;</span>Population indicators<span class="nt">&lt;/value&gt;&lt;/name&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;/topic&gt;</span> <span class="nt">&lt;topic</span> <span class="na">id=</span><span class="s">"poverty_and_income"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;&lt;value&gt;</span>Poverty <span class="err">&amp;</span> income<span class="nt">&lt;/value&gt;&lt;/name&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;/topic&gt;</span> <span class="nt">&lt;topic</span> <span class="na">id=</span><span class="s">"health"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;&lt;value&gt;</span>Health<span class="nt">&lt;/value&gt;&lt;/name&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;/topic&gt;</span> <span class="nt">&lt;/topic&gt;</span> <span class="nt">&lt;/topics&gt;</span> <span class="nt">&lt;concepts&gt;</span> <span class="c">&lt;!-- As noted in the tutorial, this concept should extend quantity:amount.--&gt;</span> <span class="nt">&lt;concept</span> <span class="na">id=</span><span class="s">"population"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>Population<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value&gt;</span>Size of the resident population.<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;topic</span> <span class="na">ref=</span><span class="s">"population_indicators"</span><span class="nt">/&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"integer"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/concept&gt;</span> <span class="c">&lt;!-- This country concept is defined for educational purposes only. A country concept exists in the Google geo dataset. See: http://code.google.com/apis/publicdata/docs/canonical/geo.html --&gt;</span> <span class="nt">&lt;concept</span> <span class="na">id=</span><span class="s">"country"</span> <span class="na">extends=</span><span class="s">"geo:location"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>Country<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value&gt;</span>My list of countries<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;property</span> <span class="na">id=</span><span class="s">"name"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;&lt;value</span> <span class="na">xml:lang=</span><span class="s">"en"</span><span class="nt">&gt;</span>Country name<span class="nt">&lt;/value&gt;&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value</span> <span class="na">xml:lang=</span><span class="s">"en"</span><span class="nt">&gt;</span>The official name of the country<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;table</span> <span class="na">ref=</span><span class="s">"countries_table"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/concept&gt;</span> <span class="c">&lt;!-- This US state concept is defined for educational purposes only. A US state concept exists in the Google geo US dataset. See: http://code.google.com/apis/publicdata/docs/canonical/geo.us.html --&gt;</span> <span class="nt">&lt;concept</span> <span class="na">id=</span><span class="s">"state"</span> <span class="na">extends=</span><span class="s">"geo:location"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>State<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value&gt;</span>US states<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;property</span> <span class="na">concept=</span><span class="s">"country"</span> <span class="na">isParent=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;table</span> <span class="na">ref=</span><span class="s">"states_table"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/concept&gt;</span> <span class="nt">&lt;concept</span> <span class="na">id=</span><span class="s">"gender"</span> <span class="na">extends=</span><span class="s">"entity:entity"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>Gender<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value&gt;</span>Gender, Male or Female<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;pluralName&gt;&lt;value&gt;</span>Genders<span class="nt">&lt;/value&gt;&lt;/pluralName&gt;</span> <span class="nt">&lt;totalName&gt;&lt;value&gt;</span>Both genders<span class="nt">&lt;/value&gt;&lt;/totalName&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;table</span> <span class="na">ref=</span><span class="s">"genders_table"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/concept&gt;</span> <span class="nt">&lt;concept</span> <span class="na">id=</span><span class="s">"unemployment_rate"</span> <span class="na">extends=</span><span class="s">"quantity:rate"</span><span class="nt">&gt;</span> <span class="nt">&lt;info&gt;</span> <span class="nt">&lt;name&gt;</span> <span class="nt">&lt;value&gt;</span>unemployment rate<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;</span> <span class="nt">&lt;value&gt;</span>The percent of the labor force that is unemployed, not seasonally adjusted.<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;url&gt;&lt;value&gt;</span>http://www.bls.gov/cps/cps_htgm.htm<span class="nt">&lt;/value&gt;&lt;/url&gt;</span> <span class="nt">&lt;/info&gt;</span> <span class="nt">&lt;topic</span> <span class="na">ref=</span><span class="s">"social_indicators"</span><span class="nt">/&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"float"</span><span class="nt">/&gt;</span> <span class="nt">&lt;attribute</span> <span class="na">id=</span><span class="s">"is_percentage"</span><span class="nt">&gt;</span> <span class="nt">&lt;type</span> <span class="na">ref=</span><span class="s">"boolean"</span><span class="nt">/&gt;</span> <span class="nt">&lt;value&gt;</span>true<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/attribute&gt;</span> <span class="nt">&lt;/concept&gt;</span> <span class="nt">&lt;/concepts&gt;</span> <span class="nt">&lt;slices&gt;</span> <span class="nt">&lt;slice</span> <span class="na">id=</span><span class="s">"countries_slice"</span><span class="nt">&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"country"</span><span class="nt">/&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"time:year"</span><span class="nt">/&gt;</span> <span class="nt">&lt;metric</span> <span class="na">concept=</span><span class="s">"population"</span><span class="nt">/&gt;</span> <span class="nt">&lt;table</span> <span class="na">ref=</span><span class="s">"countries_slice_table"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/slice&gt;</span> <span class="nt">&lt;slice</span> <span class="na">id=</span><span class="s">"states_slice"</span><span class="nt">&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"state"</span><span class="nt">/&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"time:year"</span><span class="nt">/&gt;</span> <span class="nt">&lt;metric</span> <span class="na">concept=</span><span class="s">"population"</span><span class="nt">/&gt;</span> <span class="nt">&lt;metric</span> <span class="na">concept=</span><span class="s">"unemployment_rate"</span><span class="nt">/&gt;</span> <span class="nt">&lt;table</span> <span class="na">ref=</span><span class="s">"states_slice_table"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/slice&gt;</span> <span class="nt">&lt;slice</span> <span class="na">id=</span><span class="s">"countries_gender_slice"</span><span class="nt">&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"country"</span><span class="nt">/&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"gender"</span><span class="nt">/&gt;</span> <span class="nt">&lt;dimension</span> <span class="na">concept=</span><span class="s">"time:year"</span><span class="nt">/&gt;</span> <span class="nt">&lt;metric</span> <span class="na">concept=</span><span class="s">"population"</span><span class="nt">/&gt;</span> <span class="nt">&lt;table</span> <span class="na">ref=</span><span class="s">"countries_gender_slice_table"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/slice&gt;</span> <span class="nt">&lt;/slices&gt;</span> <span class="nt">&lt;tables&gt;</span> <span class="nt">&lt;table</span> <span class="na">id=</span><span class="s">"countries_table"</span><span class="nt">&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"country"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"name"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"latitude"</span> <span class="na">type=</span><span class="s">"float"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"longitude"</span> <span class="na">type=</span><span class="s">"float"</span><span class="nt">/&gt;</span> <span class="nt">&lt;data&gt;</span> <span class="nt">&lt;file</span> <span class="na">format=</span><span class="s">"csv"</span> <span class="na">encoding=</span><span class="s">"utf-8"</span><span class="nt">&gt;</span>countries.csv<span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/data&gt;</span> <span class="nt">&lt;/table&gt;</span> <span class="nt">&lt;table</span> <span class="na">id=</span><span class="s">"countries_slice_table"</span><span class="nt">&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"country"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"year"</span> <span class="na">type=</span><span class="s">"date"</span> <span class="na">format=</span><span class="s">"yyyy"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"population"</span> <span class="na">type=</span><span class="s">"integer"</span><span class="nt">/&gt;</span> <span class="nt">&lt;data&gt;</span> <span class="nt">&lt;file</span> <span class="na">format=</span><span class="s">"csv"</span> <span class="na">encoding=</span><span class="s">"utf-8"</span><span class="nt">&gt;</span>country_slice.csv<span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/data&gt;</span> <span class="nt">&lt;/table&gt;</span> <span class="nt">&lt;table</span> <span class="na">id=</span><span class="s">"states_table"</span><span class="nt">&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"state"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"name"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"country"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>US<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/column&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"latitude"</span> <span class="na">type=</span><span class="s">"float"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"longitude"</span> <span class="na">type=</span><span class="s">"float"</span><span class="nt">/&gt;</span> <span class="nt">&lt;data&gt;</span> <span class="nt">&lt;file</span> <span class="na">format=</span><span class="s">"csv"</span> <span class="na">encoding=</span><span class="s">"utf-8"</span><span class="nt">&gt;</span>states.csv<span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/data&gt;</span> <span class="nt">&lt;/table&gt;</span> <span class="nt">&lt;table</span> <span class="na">id=</span><span class="s">"states_slice_table"</span><span class="nt">&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"state"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"year"</span> <span class="na">type=</span><span class="s">"date"</span> <span class="na">format=</span><span class="s">"yyyy"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"population"</span> <span class="na">type=</span><span class="s">"integer"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"unemployment_rate"</span> <span class="na">type=</span><span class="s">"float"</span><span class="nt">/&gt;</span> <span class="nt">&lt;data&gt;</span> <span class="nt">&lt;file</span> <span class="na">format=</span><span class="s">"csv"</span> <span class="na">encoding=</span><span class="s">"utf-8"</span><span class="nt">&gt;</span>state_slice.csv<span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/data&gt;</span> <span class="nt">&lt;/table&gt;</span> <span class="nt">&lt;table</span> <span class="na">id=</span><span class="s">"genders_table"</span><span class="nt">&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"gender"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"name"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;data&gt;</span> <span class="nt">&lt;file</span> <span class="na">format=</span><span class="s">"csv"</span> <span class="na">encoding=</span><span class="s">"utf-8"</span><span class="nt">&gt;</span>genders.csv<span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/data&gt;</span> <span class="nt">&lt;/table&gt;</span> <span class="nt">&lt;table</span> <span class="na">id=</span><span class="s">"countries_gender_slice_table"</span><span class="nt">&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"country"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"gender"</span> <span class="na">type=</span><span class="s">"string"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"year"</span> <span class="na">type=</span><span class="s">"date"</span> <span class="na">format=</span><span class="s">"yyyy"</span><span class="nt">/&gt;</span> <span class="nt">&lt;column</span> <span class="na">id=</span><span class="s">"population"</span> <span class="na">type=</span><span class="s">"integer"</span><span class="nt">/&gt;</span> <span class="nt">&lt;data&gt;</span> <span class="nt">&lt;file</span> <span class="na">format=</span><span class="s">"csv"</span> <span class="na">encoding=</span><span class="s">"utf-8"</span><span class="nt">&gt;</span>gender_country_slice.csv<span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/data&gt;</span> <span class="nt">&lt;/table&gt;</span> <span class="nt">&lt;/tables&gt;</span> <span class="nt">&lt;/dspl&gt;</span> </pre></div> http://localhost/ticket/1144 http://localhost/ticket/1144 Report timmcnamara <p> The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot. </p> <p> Its documentation is here: <a class="ext-link" href="http://www.handle.net/"><span class="icon">​</span>http://www.handle.net/</a>. Servers running CKAN could host a "Local Handle Service", which redirects a hash of a resource to an actual URL. </p> <p> Some suggested use cases: </p> <ul><li>Researcher would like to cite where data came from </li><li>Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform </li></ul> http://localhost/ticket/1145 http://localhost/ticket/1145 Report kindly <p> Logic layer should not use any vdm defined state and should manage it itself. </p> http://localhost/ticket/1146 http://localhost/ticket/1146 Report kindly <p> Revision tables need expired_id to make querying history AND pending changes more efficient. </p> <p> This involves making a session extension and a large table migration. </p> http://localhost/ticket/1147 http://localhost/ticket/1147 Report kindly <p> The tests have been running slower recently and need fixing. They also could do with a bit more consistency to them. </p> http://localhost/ticket/1148 http://localhost/ticket/1148 Report kindly <p> This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1. </p> http://localhost/ticket/1149 http://localhost/ticket/1149 Report timmcnamara <p> Characters outside of ASCII range are not supported within data previews. </p> <p> Steps to reproduce: </p> <ol><li>Visit <a class="ext-link" href="http://ckan.net/package/kivele2010"><span class="icon">​</span>http://ckan.net/package/kivele2010</a> </li><li>Click on [preview] for any of the resources </li></ol> http://localhost/ticket/1150 http://localhost/ticket/1150 Report timmcnamara <p> Data viewer / previewer for a resource that has a KML file, or others, such as GeoRSS and GeoJSON, we should provide a map. </p> <ul><li>Support for KML files: <a class="assigned ticket" href="http://localhost/ticket/1458" title="enhancement: Support previewing kml files in data viewer (assigned)">#1458</a> </li></ul><p> This is non-trivial for external files as we need a way to jsonify. For files stored locally this is more of a recline issue (and will require a bit of work to either guess columns or allow user to specify them). </p> http://localhost/ticket/1151 http://localhost/ticket/1151 Report amercader <p> The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested. </p> <p> The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are: </p> <ul><li>Handling the validators (right now are hardcoded in the harvester </li></ul><p> code). This probably involves issues discussed in the CREP 3 (ticket <a class="new ticket" href="http://localhost/ticket/1134" title="CREP: CREP0003: Description and Configuration of Harvesters (new)">#1134</a>) </p> <ul><li>Changes in the model to adapt the specification to ISO 19139 </li></ul><ul><li>Renaming objects and classes which are now Gemini-centric </li></ul><p> List of CSW servers tested: </p> <p> <a class="ext-link" href="https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&amp;hl=en_US&amp;authkey=CNu4hsEB#gid=0"><span class="icon">​</span>https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&amp;hl=en_US&amp;authkey=CNu4hsEB#gid=0</a> </p> http://localhost/ticket/1152 http://localhost/ticket/1152 Report lucychambers <p> Update CKAN wiki front page - a la <a class="missing wiki">OpenSpending?</a>: <a class="ext-link" href="http://wiki.openspending.org/Main_Page"><span class="icon">​</span>http://wiki.openspending.org/Main_Page</a> </p> <p> Sections should relate to different types of people using the site: </p> <p> Developers, Users etc.. </p> http://localhost/ticket/1153 http://localhost/ticket/1153 Report nils.toedtmann <p> According to pudo, a ckan with activated solr extension throws a 5xx when solr is unreachable. Instead, it should behave more like a ckan without ckanext-solr when this happens. </p> http://localhost/ticket/1154 http://localhost/ticket/1154 Report pudo <p> Write a harvester for data.london.gov.uk to import catalogue metadata into PDEU. API (or at least documentation) is available at: <a class="ext-link" href="http://sourceforge.net/projects/londondatastore/files/"><span class="icon">​</span>http://sourceforge.net/projects/londondatastore/files/</a> </p> http://localhost/ticket/1155 http://localhost/ticket/1155 Report pudo <p> Import metadata from both sources into PDEU via the Harvesting framework but by scraping their respective catalogue pages. </p> <ul><li><a class="ext-link" href="http://data.wien.gv.at/katalog/"><span class="icon">​</span>http://data.wien.gv.at/katalog/</a> </li><li><a class="ext-link" href="http://opendata.paris.fr/opendata/jsp/site/Portal.jsp"><span class="icon">​</span>http://opendata.paris.fr/opendata/jsp/site/Portal.jsp</a> </li></ul> http://localhost/ticket/1156 http://localhost/ticket/1156 Report pudo <p> Change the visual style of CKAN to be more like these sites: </p> <ul><li><a class="missing wiki">GitHub?</a> </li><li>Quora </li><li>Google Projects </li></ul><p> This does not include major UX work. </p> http://localhost/ticket/1157 http://localhost/ticket/1157 Report pudo <p> PDEU should be in a mode where: </p> <ul><li>Users cannot edit or create packages </li><li>Users cannot sign up on their own </li><li><a class="missing wiki">AuthorizationGroups?</a> and <a class="missing wiki">RevisionHistory?</a> is completely gone from the UI </li><li>Groups have been repurposed as Subjects </li></ul> http://localhost/ticket/1158 http://localhost/ticket/1158 Report pudo <p> RDFa can be used as a simple way to expose linked data or at least sameAs the API version of the data but we need to make sure we do not expose anything different from the "official" representation. </p> http://localhost/ticket/1159 http://localhost/ticket/1159 Report amercader <p> It could be interesting for the publicdata.eu project to add some CSW datasources. Thanks to the INSPIRE directive, there are quite a few national catalogs of geodata that could be worth harvesting. Work on this should include: </p> <ul><li>Compose Endpoint List (1d) </li><li>Harvest on staging system and evaluate which sources have useful material (2d) </li></ul><p> First point is being performed in this GDoc: <a class="ext-link" href="https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&amp;authkey=CNu4hsEB&amp;hl=en_US#gid=0"><span class="icon">​</span>https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&amp;authkey=CNu4hsEB&amp;hl=en_US#gid=0</a> </p> http://localhost/ticket/1160 http://localhost/ticket/1160 Report amercader <p> We need to offer a strongly simplified version, read-only of CKAN under publicdata.eu, with a focus on its role as search engine instead of a data catalogue. </p> <p> This ticket relates to work on the PDEU theme only! </p> http://localhost/ticket/1161 http://localhost/ticket/1161 Report amercader <p> We need a common extra across the different harvested sources that defines the extent of the data. We will have mainly national level catalogues, but also at a lower level (region, city). </p> <p> Relevant discussion: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001245.html</a> </p> http://localhost/ticket/1162 http://localhost/ticket/1162 Report rgrp <p> Storage is now working but there are </p> <ul><li>Integrate with Resources (e.g. create a resource for each file upload and give option to associate with a package) <ul><li>Should we introduce rule that files *not* associated with a Resource are periodically deleted? </li></ul></li><li>Allow setting of a file name/path before upload </li><li>Allow for file overwriting/deleting etc (how should this work -- do we want to allow this sort of thing) </li><li>Integrate local file upload stuff in api/auth/* </li></ul><h2 id="DifferentBackendIssues">Different Backend Issues</h2> <p> Local file store is rather different from 'remote' storage in various ways: </p> <ul><li>For remote you don't want to use many buckets as there are bucket limits while for local you want to. Should we there have a single path that users provide which we then partition differently for different backends. </li></ul> http://localhost/ticket/1163 http://localhost/ticket/1163 Report amercader <p> A nice map in the homepage showing the availability of data across Europe </p> http://localhost/ticket/1164 http://localhost/ticket/1164 Report nils.toedtmann <p> Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3). </p> <p> That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "&lt;<a class="missing wiki">VirtualHost?</a>&gt;", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey. </p> <p> I see two ways to constitute the map hostname-to-ini-file map: </p> <ul><li>ckan reads a set of ini files, and each ini file declares which servers names it is responsible for </li><li>In a global ini file, there are directives mapping servernames to ini files. </li></ul><p> In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc. </p> http://localhost/ticket/1165 http://localhost/ticket/1165 Report amercader <p> Right now, for the "View available WMS layers" link to be displayed, the package needs to have an "INSPIRE=True" extra, a "resource-type=service" extra and a resource of type "WMS". </p> <p> This is due to DGU (the project this was developed for) related reasons, but for a generic support, the resource type should probably be enough. </p> <p> (The repository for this is ckanext-spatial, but it's not on the list) </p> http://localhost/ticket/1166 http://localhost/ticket/1166 Report nils.toedtmann <p> We should create a public AMI with CKAN pre-installed and configured such that users can easily create their own EC2 machine with a running CKAN to play with. </p> <p> There are three phases: </p> <ol><li>[nils] Deploy an empty EC2 instance to become the CKAN image master instance </li><li>Install a CKAN and give it a standard configuration. </li><li>[nils] Create a AMI from the CKAN image master instance and publish it. </li></ol><p> I am happy to do first and last. Who is installing and configuring CKAN? </p> <p> Unfortunately AMIs are specific to region, architecture and storage type. We cannot maintain too many images, so a number of choices have to be made: </p> <ul><li>Which distribution/version? <strong>Ubuntu 10.04 LTS</strong> </li><li>Which architecture/instance-type? I suggest 64-bit/t1.micro </li><li>Which region? I suggest us-east-1 and maybe eu-west-1 </li><li>Which storage type? <strong>EBS</strong> (way easier to make an AMI from than instance-store) </li><li>Install CKAN from deb packages via mercurial/virtualenv? I assume the latter because the AMI is targeted to developers? </li></ul><hr /> <p> <a class="ext-link" href="http://alestic.com/2009/06/ec2-ami-bundle"><span class="icon">​</span>1</a> <a class="ext-link" href="http://uec-images.ubuntu.com/releases/10.04/release/"><span class="icon">​</span>2</a> </p> http://localhost/ticket/1167 http://localhost/ticket/1167 Report dread <p> Get buildbot to: </p> <ul><li>build the deb packages </li><li>install them into a fresh virtual machine </li><li>run smoke tests on the installed ckan </li></ul> http://localhost/ticket/1168 http://localhost/ticket/1168 Report dread http://localhost/ticket/1169 http://localhost/ticket/1169 Report dread <p> Need to cover pip-requirements.txt on different branches in the doc/deployment.rst. </p> <p> Also could do with tidying up upgrade.txt into this document too. </p> http://localhost/ticket/1170 http://localhost/ticket/1170 Report dread <p> Some sort of citation helper. Something small on the dataset and resource page that would show how to cite. </p> <p> wwaites: Some related thoughts on this from opb: <a class="ext-link" href="http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf"><span class="icon">​</span>http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf</a> </p> <p> timclicks: I'm looking at Dataverse for the first time<a class="missing changeset" title="No default repository defined">[0]</a>. It seems very popular in the social sciences. I noticed that there is a recommended citation for each dataset. For example, <a class="missing changeset" title="No default repository defined">[1]</a> is has this one: "Targeted Input Programme (TIP) 2000-01", <a class="ext-link" href="http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1"><span class="icon">​</span>http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1</a> V1 [Version]" </p> <h2 id="Implementation">Implementation</h2> <p> Add a small box at bottom of dataset / resource page (or in sidebar on dataset page) with title "Cite this" with contents like: </p> <p> %title. %author. Retrieved %date. %site_title. </p> <p> For resource: %title = %dataset_title. %resource_name. </p> <p> Could also add export to ref managers (e.g. to bibtex) but that is for later. </p> http://localhost/ticket/1171 http://localhost/ticket/1171 Report thejimmyg <p> The current codebase has one or two try: except; blocks that don't catch specific exceptions. Under no circumstances should any broad try: except: blocks be allowed unless the exceptions they catch are immediately re-raised. Uncaught exceptions are wasting us quite a lot of time when trying to track down problems. </p> http://localhost/ticket/1172 http://localhost/ticket/1172 Report amercader <p> /api/rest/package/foo.rdf should return a DCat representation of the package. To create it, we will use the functions in ckanext-rdf. </p> http://localhost/ticket/1173 http://localhost/ticket/1173 Report pudo <p> As CKAN grows, more and more optional representations of packages will become used. Besides RDF (which is the motivation for this ticket), support for DSPL, home-grown XML, or CSV listings is imaginable. </p> <p> To properly support this CKAN should have an extensible representation registry that can be extended when new output types become available (and without changing the API in the process). This needs to integrate in two places: </p> <ul><li>We need to add support for file format extensions to the package handler and REST API. </li><li>We need to add HTTP content type negotiation (Accept headers). </li></ul><p> To support this we should have a registry with two registers: </p> <ul><li>to map format extensions to mime types (e.g. "json" -&gt; "application/json") </li><li>to map mime types and entity types to converter functions (e.g. 8"application/json", ckan.model.Package) -&gt; func(obj, mime_type) ) </li></ul><p> This should be set up on load_environment so that IConfigurer plugins can feed into it. </p> <p> The registry should then first be added to _finish in the REST API (which needs to be refactored to be passed the {format} part of the URL if one is given. Based on the format part and HTTP headers, an appropriate representation can be generated by the registry and then be returned to the user. </p> <p> Forwarding of requests to the regular WUI controllers with Accept headers set or a format specified can be implemented in a separate effort. </p> http://localhost/ticket/1174 http://localhost/ticket/1174 Report fccoelho@… <p> Hi, I get a 500, Internal server error when I enable ckanext-stats. Flavio </p> http://localhost/ticket/1175 http://localhost/ticket/1175 Report dread <p> We seem to occasionally get this exception on ckan.net. </p> <p> User agents causing this include: </p> <ul><li>Alexa ia_archiver (3/6/11) </li><li>Maxthon browser (17/5/11) </li></ul><pre class="wiki">Module ckan.controllers.error:32 in document &lt;&lt; return original_response.body # Otherwise, decorate original response with error template. c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', '')) c.prefix=request.environ.get('SCRIPT_NAME', ''), c.code=cgi.escape(request.GET.get('code', str(original_response.status_int))), &gt;&gt; c.content = literal(original_response.unicode_body) or cgi.escape(request.GET.get('message', '')) AttributeError: 'NoneType' object has no attribute 'unicode_body' </pre> http://localhost/ticket/1176 http://localhost/ticket/1176 Report fccoelho <blockquote> <p> Hello, </p> </blockquote> <p> I have done a recent update to mu ckan install, and I am now getting this error for all my packages: </p> <pre class="wiki">URL: http://ckan.emap.fgv.br/package/dengue-net Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:118 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ?:2 in read Module ckan.lib.cache:167 in wrapper &lt;&lt; log = __import__("logging").getLogger("proxy_cache") def wrapper(func, *args, **kwargs): result = func(*args, **kwargs) pylons = get_pylons(args) &gt;&gt; result = func(*args, **kwargs) Module ckan.controllers.package:208 in read &lt;&lt; #render the package PackageSaver().render_package(c.pkg) return render('package/read.html') def comments(self, id): &gt;&gt; return render('package/read.html') Module ckan.lib.base:74 in render &lt;&lt; return cached_template(template_name, render_template, cache_key=cache_key, cache_type=cache_type, cache_expire=cache_expire) #, ns_options=('method'), method=method) &gt;&gt; cache_type=cache_type, cache_expire=cache_expire) Module pylons.templating:249 in cached_template &lt;&lt; return content else: return render_func() &gt;&gt; return render_func() Module ckan.lib.base:63 in render_template &lt;&lt; stream = item.filter(stream) return literal(stream.render(method=method, encoding=None)) if 'Pragma' in response.headers: &gt;&gt; return literal(stream.render(method=method, encoding=None)) Module genshi.core:183 in render &lt;&lt; method = self.serializer or 'xml' generator = self.serialize(method=method, **kwargs) return encode(generator, method=method, encoding=encoding, out=out) def select(self, path, namespaces=None, variables=None): &gt;&gt; return encode(generator, method=method, encoding=encoding, out=out) Module genshi.output:57 in encode &lt;&lt; _encode = lambda string: string if out is None: return _encode(''.join(list(iterator))) for chunk in iterator: out.write(_encode(chunk)) &gt;&gt; return _encode(''.join(list(iterator))) Module genshi.output:339 in __call__ &lt;&lt; for filter_ in self.filters: stream = filter_(stream) for kind, data, pos in stream: cached = cache_get((kind, data)) if cached is not None: &gt;&gt; for kind, data, pos in stream: Module genshi.output:670 in __call__ &lt;&lt; _gen_prefix = _gen_prefix().next for kind, data, pos in stream: output = cache_get((kind, data)) if output is not None: &gt;&gt; for kind, data, pos in stream: Module genshi.output:771 in __call__ &lt;&lt; push_text = textbuf.append pop_text = textbuf.pop for kind, data, pos in chain(stream, [(None, None, None)]): if kind is TEXT: &gt;&gt; for kind, data, pos in chain(stream, [(None, None, None)]): Module genshi.output:586 in __call__ &lt;&lt; def __call__(self, stream): prev = (None, None, None) for ev in stream: if prev[0] is START: if ev[0] is END: &gt;&gt; for ev in stream: Module genshi.core:288 in _ensure &lt;&lt; # unchanged yield event for event in stream: yield event &gt;&gt; for event in stream: Module genshi.template.base:618 in _include &lt;&lt; tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: &gt;&gt; for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include &lt;&lt; tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: &gt;&gt; for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include &lt;&lt; tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: &gt;&gt; for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:605 in _include &lt;&lt; from genshi.template.loader import TemplateNotFound for event in stream: if event[0] is INCLUDE: href, cls, fallback = event[1] &gt;&gt; for event in stream: Module genshi.template.markup:378 in _match &lt;&lt; for event in self._match(self._flatten(template, ctxt, **vars), ctxt, start=idx + 1, **vars): yield event &gt;&gt; ctxt, start=idx + 1, **vars): Module genshi.template.markup:327 in _match &lt;&lt; break for event in stream: # We (currently) only care about start and end events for matching &gt;&gt; for event in stream: Module genshi.template.base:565 in _flatten &lt;&lt; elif kind is EXPR: result = _eval_expr(data, ctxt, vars) if result is not None: # First check for a string, otherwise the iterable test &gt;&gt; result = _eval_expr(data, ctxt, vars) Module genshi.template.base:277 in _eval_expr &lt;&lt; if vars: ctxt.push(vars) retval = expr.evaluate(ctxt) if vars: ctxt.pop() &gt;&gt; retval = expr.evaluate(ctxt) Module genshi.template.eval:178 in evaluate &lt;&lt; __traceback_hide__ = 'before_and_this' _globals = self._globals(data) return eval(self.code, _globals, {'__data__': data}) &gt;&gt; return eval(self.code, _globals, {'__data__': data}) Module ?:16 in &lt;Expression u"h.subnav_link(c, h.icon('comments') + _('Comments &amp; Questions'), controller='package', action='comments', id=c.pkg.name)"&gt; &lt;&lt; &lt;/li&gt; &lt;li py:if="g.has_commenting"&gt; ${h.subnav_link(c, h.icon('comments') + _('Comments &amp;amp; Questions'), controller='package', action='comments', id=c.pkg.name)} &lt;/li&gt; &lt;li&gt;${h.subnav_link(c, h.icon('page_white_stack') + _('History'), controller='package', action='history', id=c.pkg.name)}&lt;/li&gt; &gt;&gt; ${h.subnav_link(c, h.icon('comments') + _('Comments &amp;amp; Questions'), controller='package', action='comments', id=c.pkg.name)} Module ckan.lib.helpers:126 in subnav_link &lt;&lt; return link_to( text, url_for(action=action, **kwargs), class_=('active' if c.action == action else '') ) &gt;&gt; url_for(action=action, **kwargs), Module routes.util:280 in url_for &lt;&lt; raise GenerationException( "url_for could not generate URL. Called with args: %s %s" % \ (args, kargs)) return url &gt;&gt; (args, kargs)) GenerationException: url_for could not generate URL. Called with args: () {'action': 'comments', 'controller': 'package', 'id': 'dengue-net'} CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; __utmc=4669863' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_REFERER 'http://ckan.emap.fgv.br/' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/package/dengue-net' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/package/dengue-net' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '47624' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/package/dengue-net' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application &lt;beaker.middleware.CacheMiddleware object at 0x7f051c802690&gt; beaker.cache &lt;beaker.cache.CacheManager object at 0x7f051c802750&gt; beaker.get_session &lt;bound method SessionMiddleware._get_session of &lt;beaker.middleware.SessionMiddleware object at 0x7f051c8026d0&gt;&gt; beaker.session {'locale': u'pt_BR', '_accessed_time': 1307358102.6266389, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script '' mod_wsgi.input_chunked '0' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio'&gt;, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode";... __utmc=4669863') paste.registry &lt;paste.registry.Registry object at 0x7f051d7e5710&gt; paste.throw_errors True pylons.action_method &lt;bound method PackageController.read of &lt;ckan.controllers.package.PackageController object at 0x7f051d7e5b10&gt;&gt; pylons.controller &lt;ckan.controllers.package.PackageController object at 0x7f051d7e5b10&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.PylonsContext object at 0x7f051d7e5a50&gt; pylons.routes_dict {'action': u'read', 'controller': u'package', 'id': u'dengue-net'} repoze.who.identity &lt;repoze.who identity (hidden, dict-like) at 139659946193344&gt; repoze.who.logger &lt;logging.Logger instance at 0x7f051c801128&gt; repoze.who.plugins {'openid': &lt;OpenIdIdentificationPlugin 139659931643152&gt;, 'friendlyform': &lt;FriendlyFormPlugin 139659929725584&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f051c9e7610&gt;, 'auth_tkt': &lt;AuthTktCookiePlugin 139659931643280&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f051c9dcfd0&gt;} routes.route &lt;routes.route.Route object at 0x7f051c69fd50&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f051d7e57d0&gt; webob._parsed_cookies ({'ckan': 'd4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06', '__utmz': '4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'auth_tkt': 'c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode', 'ckan_display_name': 'Fl\xc3\xa1vio Code\xc3\xa7o Coelho', 'ckan_apikey': 'c139718d-918f-4a2b-b219-e33cb05cbe23', 'ckan_user': 'flavio', '__utma': '4669863.1260802706.1295364409.1306935693.1306960483.86', '__utmc': '4669863', 'SESS61faf7f7d2406929b2f9eb4cbfff17d2': 'a3306f5d1194cf0d8a51dbf7b42c1259'}, '__utma=4669863.1260802706.1295364409.1306935693.1306960483.86; __utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode"&lt; wbr&gt;; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!... __utmc=4669863') webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f051cc1ad50&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f051d7e57d0&gt;, {'action': u'read', 'controller': u'package', 'id': u'dengue-net'}) </pre> http://localhost/ticket/1177 http://localhost/ticket/1177 Report fccoelho <p> I have followed the instruction for installation a configuration of ckanext-storage, but I am getting this error when I try to access the /storage/upload url: </p> <p> URL: <a class="ext-link" href="http://ckan.emap.fgv.br/storage/upload"><span class="icon">​</span>http://ckan.emap.fgv.br/storage/upload</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:152 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:130 in <span class="underline">call</span> &lt;&lt; environ<a class="missing wiki">SCRIPT_NAME?</a> = environ<a class="missing wiki">SCRIPT_NAME?</a>[:-1] </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:124 in <span class="underline">call</span> &lt;&lt; return <a class="missing wiki">% paste.registry.restorer.get_request_id(environ)?</a> </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> controller = self.resolve(environ, start_response) </p> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:263 in resolve &lt;&lt; if self.log_debug: </p> <blockquote> <p> log.debug("Resolved URL to controller: %r", controller) </p> </blockquote> <blockquote> <p> return self.find_controller(controller) </p> </blockquote> <p> </p> <blockquote> <p> def find_controller(self, controller): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.find_controller(controller) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.config.environment:39 in find_controller &lt;&lt; # Check to see if its a dotted name </p> <blockquote> <p> if '.' in controller or ':' in controller: </p> <blockquote> <p> mycontroller = pkg_resources.<a class="missing wiki">EntryPoint?</a>.parse('x=%s' % controller).load(False) self.controller_classes[controller] = mycontroller return mycontroller </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> mycontroller = pkg_resources.<a class="missing wiki">EntryPoint?</a>.parse('x=%s' % controller).load(False) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pkg_resources:1954 in load &lt;&lt; def load(self, require=True, env=None, installer=None): </p> <blockquote> <p> if require: self.require(env, installer) entry = <span class="underline">import</span>(self.module_name, globals(),globals(), <a class="missing wiki">__name__?</a>) for attr in self.attrs: </p> <blockquote> <p> try: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> entry = <span class="underline">import</span>(self.module_name, globals(),globals(), <a class="missing wiki">__name__?</a>) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ?:87 in &lt;module&gt; &lt;&lt; class StorageAPIController(<a class="missing wiki">BaseController?</a>): </p> <blockquote> <p> ofs = get_ofs() </p> </blockquote> <p> </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> class StorageAPIController(<a class="missing wiki">BaseController?</a>): </p> </blockquote> </blockquote> </blockquote> <p> Module ?:88 in StorageAPIController &lt;&lt; class StorageAPIController(<a class="missing wiki">BaseController?</a>): </p> <blockquote> <p> ofs = get_ofs() </p> </blockquote> <p> </p> <blockquote> <p> @jsonpify </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> ofs = get_ofs() </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ?:69 in get_ofs &lt;&lt; continue </p> <blockquote> <p> kw[k[4:]] = v </p> </blockquote> <blockquote> <p> ofs = get_impl(storage_backend)(<strong>kw) return ofs </strong></p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> ofs = get_impl(storage_backend)(<strong>kw) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ofs.local.pairtreestore:26 in <span class="underline">init</span> &lt;&lt; self.hashing_type = hashing_type </p> <blockquote> <p> self.shorty_length = shorty_length self._open_store() </p> </blockquote> <p> </p> <blockquote> <p> def _open_store(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> self._open_store() </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ofs.local.pairtreestore:30 in _open_store &lt;&lt; def _open_store(self): </p> <blockquote> <p> if self.hashing_type: </p> <blockquote> <p> self._store = <a class="missing wiki">PairtreeStorageClient?</a>(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type) </p> </blockquote> <p> else: </p> <blockquote> <p> self._store = <a class="missing wiki">PairtreeStorageClient?</a>(self.uri_base, self.storage_dir, shorty_length=shorty_length) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> self._store = <a class="missing wiki">PairtreeStorageClient?</a>(self.uri_base, self.storage_dir, shorty_length=self.shorty_length, hashing_type=self.hashing_type) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pairtree.pairtree_client:91 in <span class="underline">init</span> &lt;&lt; self._decode = re.compile(r"\<sup>(..)", re.U) </sup></p> <blockquote> <p> self._init_store() </p> </blockquote> <p> </p> <blockquote> <p> def <span class="underline">char2hex(self, m): </span></p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> self._init_store() </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pairtree.pairtree_client:264 in _init_store &lt;&lt; self.uri_base = prefix </p> <blockquote> <p> else: </p> <blockquote> <p> raise NotAPairtreeStoreException </p> </blockquote> </blockquote> <p> </p> <blockquote> <p> if not os.path.isdir(self.store_dir): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> raise NotAPairtreeStoreException </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> NotAPairtreeStoreException: CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE '<span class="underline">utma=4669863.1260802706.1295364409.1306935693.1306960483.86; </span>utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; <span class="underline">utmc=4669863; hide_welcome_message=1' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/storage/upload' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '57426' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [<em>] REQUEST_METHOD 'GET' REQUEST_URI '/storage/upload' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7fc2999a6cd0&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7fc2999a6d90&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7fc2999a6d10&gt;&gt; beaker.session {'locale': u'pt_BR', '_accessed_time': 1307373458.4790759, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script </em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host <em> mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'&gt;, '</em></span><em>utma=4669863.1260802706.1295364409.1306935693.1306960483.86; <span class="underline">utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1') paste.registry &lt;paste.registry.Registry object at 0x7fc299de4f50&gt; paste.throw_errors True pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7fc299e27d50&gt; pylons.routes_dict {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'} repoze.who.identity &lt;repoze.who identity (hidden, dict-like) at 140473773131872&gt; repoze.who.logger &lt;logging.Logger instance at 0x7fc2999a3b48&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 140473779397008&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 140473777417040&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7fc299b93a50&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 140473779397136&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0&gt;} routes.route &lt;routes.route.Route object at 0x7fc299864090&gt; routes.url &lt;routes.util.URLGenerator object at 0x7fc299e27ed0&gt; webob._parsed_query_vars (GET([]), </span></em><span class="underline">) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc299dae4e0&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7fc299e27ed0&gt;, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'}) Reply Reply to all Forward Reply to all paste@localhost to fccoelho show details 10:17 (2 hours ago) </span></p> <ul><li>Show quoted text - </li></ul><p> CGI Variables AUTH_TYPE 'cookie' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE '<span class="underline">utma=4669863.1260802706.1295364409.1306935693.1306960483.86; </span>utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flaviouserid_type:unicode"; ckan_user="flavio"; ckan_display_name="Fl\xc3\xa1vio Code\xc3\xa7o Coelho"; ckan_apikey="c139718d-918f-4a2b-b219-e33cb05cbe23"; <span class="underline">utmc=4669863; hide_welcome_message=1' HTTP_HOST 'ckan.emap.fgv.br' HTTP_KEEP_ALIVE '115' HTTP_USER_AGENT 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1' PATH_INFO '/storage/upload' PATH_TRANSLATED '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py/storage/upload' REMOTE_ADDR '10.250.48.110' REMOTE_PORT '57429' REMOTE_USER u'flavio' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [<em>] REQUEST_METHOD 'GET' REQUEST_URI '/storage/upload' SCRIPT_FILENAME '/home/flavio/var/srvc/ckan.emap.fgv.br/pyenv/bin/ckan.emap.fgv.br.py' SERVER_ADDR '10.252.2.60' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.emap.fgv.br' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.16 (Debian) Server at ckan.emap.fgv.br Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.16 (Debian)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7fc2999a6cd0&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7fc2999a6d90&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7fc2999a6d10&gt;&gt; beaker.session {'locale': u'pt_BR', '_accessed_time': 1307373466.085686, '_creation_time': 1302605470.0245121} mod_wsgi.application_group 'dck093|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script </em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host <em> mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: SESS61faf7f7d2406929b2f9eb4cbfff17d2='a3306f5d1194cf0d8a51dbf7b42c1259' __utma='4669863.1260802706.1295364409.1306935693.1306960483.86' __utmc='4669863' __utmz='4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid_type:unicode' ckan='d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06' ckan_apikey='c139718d-918f-4a2b-b219-e33cb05cbe23' ckan_display_name='Fl\xc3\xa1vio Code\xc3\xa7o Coelho' ckan_user='flavio' hide_welcome_message='1'&gt;, '</em></span><em>utma=4669863.1260802706.1295364409.1306935693.1306960483.86; <span class="underline">utmz=4669863.1295364409.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESS61faf7f7d2406929b2f9eb4cbfff17d2=a3306f5d1194cf0d8a51dbf7b42c1259; ckan=d4951177ef995d92f0f987d481bd9180aa0f89e4518921db9cd9b6ac75fdd3fab4a25d06; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221flavio!userid _type:unicode"; auth_tkt="c6c3c3e0b3fe7bb765356659992760b24da43221fla...lcome_message=1') paste.registry &lt;paste.registry.Registry object at 0x7fc29a9da850&gt; paste.throw_errors True pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7fc29a9ef950&gt; pylons.routes_dict {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'} repoze.who.identity &lt;repoze.who identity (hidden, dict-like) at 140473790084352&gt; repoze.who.logger &lt;logging.Logger instance at 0x7fc2999a3b48&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 140473779397008&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 140473777417040&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7fc299b93a50&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 140473779397136&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7fc299b934d0&gt;} routes.route &lt;routes.route.Route object at 0x7fc299864090&gt; routes.url &lt;routes.util.URLGenerator object at 0x7fc29a9efcd0&gt; webob._parsed_query_vars (GET([]), </span></em><span class="underline">) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fc29a9d7d50&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7fc29a9efcd0&gt;, {'action': u'index', 'controller': u'ckanext.storage.controller:StorageController'}) </span></p> http://localhost/ticket/1178 http://localhost/ticket/1178 Report timmcnamara <p> A small number of tags are near-duplicates of each other. </p> <p> Perhaps we could support word stemming from NLTK and/or manual tag aliases: </p> <blockquote> <p> statistics statistik ... survey surveying surveys </p> </blockquote> http://localhost/ticket/1179 http://localhost/ticket/1179 Report dread <p> User can insert bad anchor tags into the User-About and Package-Notes fields and when you view them (web interface) it causes a 500 error. </p> <p> Need to improve filtering for anchors in markdown. </p> <pre class="wiki">&lt;a href="http://xxxsex.com&gt;nasty/website </pre><p> Also check this related exception: </p> <pre class="wiki">Module ckan.controllers.user:59 in read &lt;&lt; c.is_myself = user.name == c.user c.api_key = user.apikey c.about_formatted = self._format_about(user.about) revisions_q = model.Session.query(model.Revision ).filter_by(author=user.name) &gt;&gt; c.about_formatted = self._format_about(user.about) Module ckan.controllers.user:167 in _format_about &lt;&lt; def _format_about(self, about): about_formatted = ckan.misc.MarkdownFormat().to_html(about) return genshi.HTML(about_formatted) def _get_form_password(self): &gt;&gt; return genshi.HTML(about_formatted) WebApp Error: &lt;class 'genshi.input.ParseError'&gt;: junk characters in start tag: u'\u201dhttp://www.settingu': line 1, column 3 </pre> http://localhost/ticket/1180 http://localhost/ticket/1180 Report dread <p> When viewing a user and a package, the about/notes fields contain Markdown, which may have links. These should have rel="nofollow" to discourage link spam. </p> http://localhost/ticket/1181 http://localhost/ticket/1181 Report timmcnamara <p> When a package has been deleted, say for spam moderation, comments still appear in the recent comments section. </p> <p> This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link. </p> <p> At CKAN.net currently, this affects the most recent comment. </p> http://localhost/ticket/1182 http://localhost/ticket/1182 Report nickstenning <p> The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file. </p> <p> This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of <tt>application/x-some-junk-here</tt> in order to force a download. </p> http://localhost/ticket/1183 http://localhost/ticket/1183 Report timmcnamara <p> Most of CKANs storage options are tied to the USA. This brings concerns of data security for some organisations who may wish to adopt the system. <a class="ext-link" href="http://wua.la"><span class="icon">​</span>Wuala</a> is a distributed file system that stores data in a peer-to-peer manner. The company behind it, <a class="missing wiki">LaCie?</a> sells storage for a fee. However, they also enable clients to have 'free' storage space when machines act as a storage node. </p> <p> In order to be a storage node, a machine needs to be online for more than 14% of the time - roughly 4h per day. Most CKAN servers are likely to have a far greater uptime than this. </p> <p> Supporting Wuala would go some way to enabling CKAN to be used in a secure manner. That is, CKAN could be promoted for organisational use where there is lots of data to be stored and large geographic distances to be managed. There is a <a class="ext-link" href="http://www.wuala.com/Wuala%20API/Sample%20Clients/Python/wualaAPI/"><span class="icon">​</span>Python client available</a> and a fairly <a class="ext-link" href="http://www.youtube.com/watch?v=3xKZ4KGkQY8"><span class="icon">​</span>long Google Tech Talk</a> that overviews the system. </p> http://localhost/ticket/1184 http://localhost/ticket/1184 Report timmcnamara <p> Administrators have "View", "Edit" and "History" tabs. However, I can't see a way to delete a package from the web UI. </p> <p> Version: CKAN.net as of today </p> http://localhost/ticket/1185 http://localhost/ticket/1185 Report dread <p> You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password. </p> http://localhost/ticket/1186 http://localhost/ticket/1186 Report dread <p> When you edit a user, you should be stopped from putting links in the 'about' field, because this is a common tactic by spammers. </p> http://localhost/ticket/1187 http://localhost/ticket/1187 Report nickstenning <p> Currently the history page only allows diffing between different versions of a package, but there doesn't appear to be any easy way to see the changes introduced by the first version of a package. </p> <p> I'm requesting the ability to diff against a "blank slate" initial state of a project, so I can see the content of the first project commit. </p> <p> Not sure if this is a vdm feature, so I'm putting this ticket in against ckan. </p> http://localhost/ticket/1188 http://localhost/ticket/1188 Report timmcnamara <p> <a class="ext-link" href="http://spideroak.com"><span class="icon">​</span>Spideroak</a> is a bulk storage platform by a company that releases quite a lot of <a class="ext-link" href="https://spideroak.com/code"><span class="icon">​</span>free software</a>, has an extensive <a class="ext-link" href="https://spideroak.com/diy/"><span class="icon">​</span>developer API</a> with pricing of $10/100GB/month. </p> <p> The data store is optimised for bulk data storage and retrieval. This is the kind of use case that CKAN packages require. We don't need low latency. We are after low cost, high reliability solution. </p> http://localhost/ticket/1189 http://localhost/ticket/1189 Report rgrp <p> For some time (e.g. 1y+!) we have known that we want to integrate some kind of datastore / data processing system with CKAN. We've had a CREP in progress on this for some months (may copy that here at some point): </p> <p> <a class="ext-link" href="http://wiki.ckan.org/CEP0004"><span class="icon">​</span>http://wiki.ckan.org/CEP0004</a> </p> <p> We can distinguish 3 modules that are needed: </p> <ol><li>"Webstore": A datastore with dataapi - <a class="closed ticket" href="http://localhost/ticket/1208" title="enhancement: Webstore: a datastore with web API (closed: fixed)">#1208</a> </li></ol><p> Suggestion is this would be sqlite based with a simple sql based API. <a class="ext-link" href="http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query"><span class="icon">​</span>http://ckan.net/api/data/{user|org}/{datastore_name}?q={some-read-sql-query</a>} </p> <ol start="2"><li>Automated conversion of suitable resources into datastore upon resource creation so that e.g. they are accessible via the API. <a class="closed ticket" href="http://localhost/ticket/1398" title="enhancement: Automated conversion of resource data into webstore (closed: fixed)">#1398</a> </li></ol><ol start="3"><li>A data processing system which utilizes this datastore. One could </li></ol><p> get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) - <a class="new ticket" href="http://localhost/ticket/1432" title="enhancement: [super] Data processing system for CKAN and Webstore (new)">#1432</a> </p> <h2 id="Moreinfo">More info</h2> <ul><li>Overview diagram: <a class="ext-link" href="https://docs.google.com/drawings/d/1XK7dcpFXNlMzVFgLPYPZUXOsFRPzgCEvS-w7FcvydEQ/edit?hl=en_GB"><span class="icon">​</span>https://docs.google.com/drawings/d/1XK7dcpFXNlMzVFgLPYPZUXOsFRPzgCEvS-w7FcvydEQ/edit?hl=en_GB</a> </li></ul> http://localhost/ticket/1190 http://localhost/ticket/1190 Report dread <p> Create/register a user with a unicode character in the password. It creates the user, but the redirect to the user page doesn't work and results in 500 error. </p> http://localhost/ticket/1191 http://localhost/ticket/1191 Report annapowellsmith <p> As part of the general documentation overhaul (<a class="closed ticket" href="http://localhost/ticket/1142" title="enhancement: [super] Major Overhaul and Extension of CKAN Documentation (closed: fixed)">ticket:1142</a>) we (APS and RGRP) want to convert the current Sphinx docs into: </p> <ul><li>An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install. </li><li>A Reference Manual which is the primary source of reference for CKAN software - this includes API docs. </li></ul><p> The current chapters of the Sphinx docs should be moved as follows: </p> <ul><li>index.rst - copy some stuff from README.txt, keep short </li><li>README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst </li><li>[NEW] install.rst - new file on installation </li><li>[NEW] theming.rst - move over from wiki </li><li>api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide) </li><li>i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file </li><li>design.rst - REMOVE (can copy some over if you can be bothered to <a class="ext-link" href="http://wiki.ckan.net/Vision"><span class="icon">​</span>http://wiki.ckan.net/Vision</a> (all philosophical stuff should go!)) </li><li>loaders.rst: move to wiki.ckan.net/Using_Loaders </li><li>feeds.rst: move to User Manual (CKAN_Feeds or just Feeds) </li><li>importer.rst: remove (don't even copy) </li><li>forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) <a class="ext-link" href="http://wiki.ckan.net/Customizing_Forms"><span class="icon">​</span>http://wiki.ckan.net/Customizing_Forms</a> </li><li>forms-integration.rst: remove </li><li>model.rst: leave but move to reference (only for core devs) </li><li>load-testing.rst -&gt; move to <a class="ext-link" href="http://wiki.ckan.net/Load_Testing"><span class="icon">​</span>http://wiki.ckan.net/Load_Testing</a> </li><li>distributed.rst -&gt; remove </li><li>admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) &lt;-- high priority </li><li>deb.html -&gt; go into reference (query james gardner on list about moving this to wiki?) </li></ul> http://localhost/ticket/1192 http://localhost/ticket/1192 Report kindly <p> The new formalchemy stores dict in a new format. </p> http://localhost/ticket/1193 http://localhost/ticket/1193 Report dread <ol><li>Register a new user (/user/register) </li><li>It redirects you to user page and displays "Welcome back" even though you are a new user. </li></ol> http://localhost/ticket/1194 http://localhost/ticket/1194 Report dread <p> We've seen exceptions caused by people getting the file storage link wrong, adding a slash to the end. </p> <p> e.g. <a class="ext-link" href="http://ckan.net/storage/f/file/e696d857-e997-41c8-be08-408697168ca8/"><span class="icon">​</span>http://ckan.net/storage/f/file/e696d857-e997-41c8-be08-408697168ca8/</a> </p> <p> We should aim to handle all requests better than causing exception. 404 (or even 302 in this case) is better than 500. </p> http://localhost/ticket/1195 http://localhost/ticket/1195 Report dread <p> Markdown produces Debug logging, even when the pylons logging config is set to level of Info or Warning. This clogs up the logfile. </p> http://localhost/ticket/1196 http://localhost/ticket/1196 Report timmcnamara <p> A new library, <a class="ext-link" href="http://blog.optimizely.com/introducing-guidersjs-an-open-source-guider-e"><span class="icon">​</span>guiders.js</a>, has been open sourced that seems to be great at unobtrusively introducing new users to features of websites. </p> <p> The library drives Optimizely's website. The GitHub repo is here: <a class="ext-link" href="https://github.com/jeff-optimizely/Guiders-JS"><span class="icon">​</span>https://github.com/jeff-optimizely/Guiders-JS</a> </p> <p> Some examples: </p> <p> <a style="padding:0; border:none" href="http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png"><img src="http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png" alt="http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png" title="http://posterous.com/getfile/files.posterous.com/temp-2011-06-22/BnlgHDvhjpFfqbwvprsIaklqrpdgJyebfuhxajbDhmvnFgzdzzCJdvluJFki/Screen_shot_2011-06-22_at_6.42.21_PM.png.scaled500.png" /></a> </p> <p> <a style="padding:0; border:none" href="http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png?AWSAccessKeyId=AKIAJFZAE65UYRT34AOQ&amp;Expires=1308801019&amp;Signature=DJWDgj7fmyq7K3ve177H7DPwKdw%3D"><img src="http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png?AWSAccessKeyId=AKIAJFZAE65UYRT34AOQ&amp;Expires=1308801019&amp;Signature=DJWDgj7fmyq7K3ve177H7DPwKdw%3D" alt="http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png" title="http://s3.amazonaws.com/files.posterous.com/temp-2011-06-22/CpvlcinkrxhtnAHfGtEHhxyGkraqlFmCjcnztFuugagBGxmBjCdwuquHpvEB/promo_gmail.png" /></a> </p> http://localhost/ticket/1197 http://localhost/ticket/1197 Report dread <p> 'Publisher' entities in the model. They are hierarchical. </p> <p> 'User-Publisher' connections with one or more roles (e.g. drafter, moderator). </p> <p> Authorization settings can control who can set what values in a 'published by' type field. </p> <p> Publishers and User-Publishers available to read in the API. </p> <p> Future tickets will provide: </p> <ul><li>API to write Publishers and User-Publishers </li><li>UI to edit Publishers and User-Publishers </li></ul><p> (This feature deprecates authorization groups) </p> http://localhost/ticket/1198 http://localhost/ticket/1198 Report pudo <p> We need an email function in CKAN to accept messages sent to users. The basic signature will be: </p> <ul><li>mail_user(user_obj, subject, body, mime_type='text/plain', headers={}) </li></ul><p> This has a number of use cases: </p> <ul><li>Retrieval of lost passwords </li><li>E-Mail confirmation </li></ul><p> Finally, the mail function should be exposed in the API for sysadmin clients. This way we can have scripts traverse CKAN for 404s, invalid data or missing fields and ping users about that automatically (requires traversal by revision, not package, to get the associated users). </p> <h2 id="Implementation">Implementation</h2> <p> Note we have already written code like this (*and* tested it) in isitopen: </p> <ul><li><a class="ext-link" href="https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/lib/mailer.py"><span class="icon">​</span>https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/lib/mailer.py</a> </li><li><a class="ext-link" href="https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/tests/lib/test_mailer.py"><span class="icon">​</span>https://bitbucket.org/okfn/isitopen/src/bb2cbd146fa5/isitopen/tests/lib/test_mailer.py</a> </li></ul> http://localhost/ticket/1199 http://localhost/ticket/1199 Report pudo <p> PDEU has a newer layout which could be adapted to also be used on ckan.net. For this, things should be cleaned up, copied to the ckanext-ckan.net repo as needed and re-colored the core CKAN color scheme. </p> <p> Maybe we could have this coincide with a CKAN rename? </p> http://localhost/ticket/1200 http://localhost/ticket/1200 Report kindly <p> Simplify the logic in the atom feed an make all feeds use logic layer to return lists. </p> http://localhost/ticket/1201 http://localhost/ticket/1201 Report rolf <p> In the presentation of data packages, the link in "The information on this page and the downloads / resources are also available using the datapkg command line utility." (<a class="ext-link" href="http://knowledgeforge.net/ckan/doc/datapkg/install.html"><span class="icon">​</span>http://knowledgeforge.net/ckan/doc/datapkg/install.html</a>) ends up at the home page of ckan.org </p> <p> Perhaps link it to <a class="ext-link" href="https://bitbucket.org/okfn/datapkg/overview"><span class="icon">​</span>https://bitbucket.org/okfn/datapkg/overview</a> for now, to save some searching for people who want to find out more? </p> http://localhost/ticket/1202 http://localhost/ticket/1202 Report rolf <p> I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML. </p> <p> The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &amp;#8212 and in the rendered table that is an &amp;mdash; </p> http://localhost/ticket/1203 http://localhost/ticket/1203 Report timmcnamara <p> When renaming a package, and doing other changes at the same time an error is reported. However, the changes seem to go through fine. </p> <p> ENV: </p> <ul><li>datacatalogs.org as of 29 June 2011 </li></ul> http://localhost/ticket/1204 http://localhost/ticket/1204 Report dread <p> When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below). </p> <p> cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan. </p> <p> In the meantime I've reverted the db to the old version (pre-upgrade). </p> <p> cz.ckan.net is not working properly until this is fixed. </p> <pre class="wiki">okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://&lt;user&gt;:&lt;pass&gt;@&lt;host&gt;/&lt;dbname&gt; module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects 2011-06-29 09:07:41,294 INFO [migrate.versioning.api] 28 -&gt; 29... /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname ret = fn(self, con, *args, **kw) /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname ret = fn(self, con, *args, **kw) /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname ret = fn(self, con, *args, **kw) /usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname ret = fn(self, con, *args, **kw) Traceback (most recent call last): File "/usr/bin/paster", line 18, in &lt;module&gt; command.run() File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run result = self.command() File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command model.repo.upgrade_db() File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db mig.upgrade(self.metadata.bind, self.migrate_repository, version=version) File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "&lt;string&gt;", line 2, in _migrate File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine return f(*a, **kw) File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate schema.runchange(ver, change, changeset.step) File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange change.run(self.engine, step) File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run script_func(engine) File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade migrate_engine.execute(q) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute return connection.execute(statement, *multiparams, **params) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute params) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement return self.__execute_context(context) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context context.parameters[0], context=context) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute context) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute context) File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey" DETAIL: Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group". 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'} </pre> http://localhost/ticket/1205 http://localhost/ticket/1205 Report dread <p> Compare these two requests to create a package: </p> <pre class="wiki">curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'Content-Type: application/json' -H 'X-CKAN-API-KEY: tester' curl http://test.ckan.net/api/rest/package -d '{name:"test"}' -H 'X-CKAN-API-KEY: tester' </pre><p> The second one gets the payload through (ckan log): </p> <pre class="wiki">Retrieving request params: UnicodeMultiDict([('{name:"test"}', u'')]) </pre><p> But the first one causes a <a class="missing wiki">ServerError?</a> because the payload (name:&#34;test&#34;) doesn't make it to request.POST or request.params: </p> <pre class="wiki">Retrieving request params: UnicodeMultiDict([]) </pre><p> The only difference is the "<a class="missing wiki">ContentType?</a>: application/json" header, which seems a reasonable thing to include. Javascript lib backbone.js (for example) inserts this automatically. </p> <p> So why does this header cause the payload to not get through to the request object? </p> http://localhost/ticket/1206 http://localhost/ticket/1206 Report dread <p> When package does not exist in ckan catalogue, ckanclient.package_entity_get should raise more specific exception, such as <a class="missing wiki">CkanNotFoundError?</a> instead of generic <a class="missing wiki">CkanApiError?</a>. </p> http://localhost/ticket/1207 http://localhost/ticket/1207 Report rgrp <ul><li>SQLite based </li><li>Use Cyclone for async http server on top with auth </li><li>Designed to not be specific to CKAN or anything else </li></ul><p> Current code is here: <a class="ext-link" href="https://github.com/okfn/webstore"><span class="icon">​</span>https://github.com/okfn/webstore</a>. </p> <h2 id="APISpec">API Spec</h2> <h4 id="Read">Read</h4> <p> Two basic ways to query:: </p> <pre class="wiki"> GET: /{owner}/{db-name}/?sql=... GET: /{owner}/{db-name}/?table=...&amp;attr=value&amp;attr=value&amp;limit=... </pre><p> Returns:: </p> <pre class="wiki"> { u'keys': [u'id', u'name'], u'data': [ [1, u'jones'], [u'aaa', u'jones'] ] } </pre><h4 id="Write">Write</h4> <p> POST to:: </p> <blockquote> <p> /{owner/{database}/{table} </p> </blockquote> <p> Payload is json data structured as follows:: </p> <blockquote> <p> { </p> <blockquote> <p> unique_keys: [list of key attributes] data: {dict of values} </p> </blockquote> <p> } </p> </blockquote> <h3 id="AuthenticationandAuthorization">Authentication and Authorization</h3> <p> Authentication: use basic auth header. </p> <p> Authorization: </p> <ul><li>Default: all read, owner can write </li><li>Restricted: owner can read and write, everyone can do nothing </li></ul><p> Possible future: config file can specify a python method (TODO: method signature) </p> <h3 id="IntegrationwithOtherSystems">Integration with Other Systems</h3> <p> TODO: Specify how to delegate authenatication to user database in some other system. </p> http://localhost/ticket/1208 http://localhost/ticket/1208 Report kindly <p> Got broken due to the moderated edits changes <a class="closed ticket" href="http://localhost/ticket/1141" title="CREP: [super] Moderated Edits User Interface (closed: fixed)">#1141</a>. </p> <p> Need to make sure preview works. </p> http://localhost/ticket/1209 http://localhost/ticket/1209 Report dread <p> <a class="missing wiki">WebOb?</a> from v1.0.7 has some interesting new behaviour with reading request data for different Content-Types: </p> <ul><li>It seems that looking at request.body mangles request.POST. See fix in ckan cset:e83bad9caa1b (problem with <a class="missing wiki">WebOb?</a> v1.0.7 but may go back to v1.0.4?) </li></ul><ul><li>A particular request caused an <a class="missing wiki">AssertionError?</a> during looking at request.body, so we need to enclose this in a try/except. I couldn't recreate it in a test unfortunately (problem with <a class="missing wiki">WebOb?</a> v1.0.7 but may go back to v1.0.2?) </li></ul><p> Example: </p> <pre class="wiki">Module ckan.controllers.api:206 in create &lt;&lt; log.debug('create: %s' % (context)) try: request_data = self._get_request_data() except ValueError, inst: response.status_int = 400 &gt;&gt; request_data = self._get_request_data() Module ckan.lib.base:149 in _get_request_data &lt;&lt; cls.log.debug('Retrieving request params: %r' % request.params) cls.log.debug('Retrieving request POST: %r' % request.POST) cls.log.debug('Retrieving request POST body: %r' % request.body) if request.POST: try: &gt;&gt; cls.log.debug('Retrieving request POST body: %r' % request.body) Module paste.registry:137 in __getattr__ &lt;&lt; def __getattr__(self, attr): return getattr(self._current_obj(), attr) def __setattr__(self, attr, value): &gt;&gt; return getattr(self._current_obj(), attr) Module webob.request:470 in _body__get &lt;&lt; Return the content of the request body. """ self.make_body_seekable() # we need this to have content_length r = self.body_file.read(self.content_length) self.body_file.seek(0) &gt;&gt; self.make_body_seekable() # we need this to have content_length Module webob.request:697 in make_body_seekable &lt;&lt; self.body_file_raw.seek(0) else: self.copy_body() &gt;&gt; self.copy_body() Module webob.request:714 in copy_body &lt;&lt; self.body = self.body_file_raw.read(length) elif self.is_body_readable: self.body = self.body_file_raw.read() self._copy_body_tempfile() else: &gt;&gt; self.body = self.body_file_raw.read() Module webob.request:1190 in read &lt;&lt; def read(self, size=-1): body = self._get_body() if size &lt; 0: v = body[self.position:] &gt;&gt; body = self._get_body() Module webob.request:1207 in _get_body &lt;&lt; self._body = _encode_multipart(self.vars, self.content_type) else: assert 0, ('Bad content type: %r' % self.content_type) return self._body &gt;&gt; assert 0, ('Bad content type: %r' % self.content_type) AssertionError: Bad content type: '; charset=utf-8' </pre> http://localhost/ticket/1210 http://localhost/ticket/1210 Report kindly <p> Sub-tasks / tickets: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1335" title="enhancement: Action API - basic (closed: fixed)">#1335</a> Action API - access to any logic layer function </li><li>Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict. </li><li>Add package_create_validate &amp; package_update_validate logic functions </li><li>Allow extensions to provide logic layer functions (IActions) </li></ul> http://localhost/ticket/1211 http://localhost/ticket/1211 Report dread <p> Notes on the CKAN API from Aron: </p> <ul><li>Server returns text/html for errors even when "Accept: application/json" header is set. </li><li>DELETE package request requires a Content-Length? Should be 405 Method Not Allowed? </li><li>Tag returned as a JSON object when updating but as a string when requesting. Same with "extras" content. </li><li>How to remove key from "extras" object? Passing None as described in the docs is invalid JSON and using null doesn't work. </li><li>Tags return package names rather than ids when querying GET /tag/{id} </li><li>Doesn't return rendered_text property on package update. </li><li>Relationships use "object" key rather than id or package_id. </li></ul> http://localhost/ticket/1214 http://localhost/ticket/1214 Report dread <p> I'm not sure if this was reported earlier, but I have found a minor bug in the "Add new package" form. As far as I can see this bug exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). This is a problem with adding new resources in "Resources" section. To reproduce the bug do the following: </p> <ol><li>Go to "Add new package", for instance <a class="ext-link" href="http://ckan.net/package/new"><span class="icon">​</span>http://ckan.net/package/new</a> </li><li>Scroll down to "Resources" section. </li><li>Click on "Remove this row" link (red circle). </li><li>Say "Yes" to "Are you sure you wish to remove this row?" </li><li>Now there should be no rows. Only the header "URL* Format Description" </li><li>Click on "Add row to table" </li><li>All you get now is another row with table header "URL* Format Description". You will not be able to add any resources now. Solution is to reload the page. </li></ol><p> The simplest solution to this bug will be to hide the "Remove this row" link if there is only one row. </p> http://localhost/ticket/1215 http://localhost/ticket/1215 Report dread <p> Exception raised doing this particular diff. </p> <p> <a class="ext-link" href="http://ckan.net/revision/diff/sudocfr?diff_entity=package&amp;oldid=a18b2520-09e0-44be-80dd-4fea94ffe7d1&amp;diff=3934ff16-a3a7-4481-b390-72944903b945"><span class="icon">​</span>http://ckan.net/revision/diff/sudocfr?diff_entity=package&amp;oldid=a18b2520-09e0-44be-80dd-4fea94ffe7d1&amp;diff=3934ff16-a3a7-4481-b390-72944903b945</a> </p> <pre class="wiki">... Module ckan.controllers.revision:142 in diff &lt;&lt; if c.diff_entity == 'package': c.pkg = model.Package.by_name(id) diff = c.pkg.diff(c.revision_to, c.revision_from) elif c.diff_entity == 'group': c.group = model.Group.by_name(id) &gt;&gt; diff = c.pkg.diff(c.revision_to, c.revision_from) Module ckan.model.package:449 in diff &lt;&lt; display_id = to_obj_rev.tag.name elif obj_class.__name__ == 'PackageExtra': display_id = to_obj_rev.key else: display_id = related_obj_id[:4] &gt;&gt; display_id = to_obj_rev.key AttributeError: 'NoneType' object has no attribute 'key' </pre><p> Seems to be to do with this particular extra key: </p> <pre class="wiki">[&lt;PackageExtraRevision id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 package_id=febe4da9-5874-4411-a7da-b497fdebf739 key=namespace value=http://www.sudoc.fr/ state=active revision_id=5ec8e730-c73d-4f1c-989b-2e015eb3d315 continuity_id=ba1f3b3e-f61f-4924-bae0-ec3dd2163ae4 expired_id=None revision_timestamp=2011-07-08 13:48:11.027260 expired_timestamp=9999-12-31 00:00:00 current=True&gt;] </pre><p> Is it because it was created between the from_revision and to_revision? </p> http://localhost/ticket/1217 http://localhost/ticket/1217 Report minspamboks@… <p> Change the color of the "History" tab icon to yellowish, like the rest of the icons in the other tabs ("View" and "Edit", "Authorization"). </p> <h2 id="Reasoning">Reasoning</h2> <p> When you view a data package, for instance <a class="ext-link" href="http://ckan.net/package/thesaurus-w"><span class="icon">​</span>http://ckan.net/package/thesaurus-w</a>, you will see "View", "Edit", "History" tabs on the top. "History" tab has a black-and-white icon which makes it look like an inactive/disabled tab (since the text is also grayed out when the tab is not selected). This is not a major issue, but it is a little bit confusing for the users. This icon exists in v1.3.2 and also in v1.4.1a (that runs on ckan.net). </p> <p> The simple solution would be to change the color of the "History" tab icon and give it the same yellowish color like the rest of the icons in the other tabs (e.g. "View" and "Edit"). </p> http://localhost/ticket/1218 http://localhost/ticket/1218 Report timmcnamara <p> As reported on <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2011-July/001043.html"><span class="icon">​</span>ckan-dev</a>: </p> <blockquote class="citation"> <p> items in the footer of CKAN ("Packages", "Groups &amp; Tags", "About", "Language", etc.) are shown vertically instead of horizontally in IE7. This works fine in later browsers like IE8, IE9, FF4, and latest Opera and Chrome. </p> </blockquote> <blockquote class="citation"> <p> This seems to exist in all recent CKAN versions up to 1.4.1a. </p> </blockquote> http://localhost/ticket/1219 http://localhost/ticket/1219 Report dread <ul><li>No tests. </li><li>500 error given when user is not found. </li></ul> http://localhost/ticket/1221 http://localhost/ticket/1221 Report dread http://localhost/ticket/1222 http://localhost/ticket/1222 Report pudo <p> StaticURLParser can have caching - use it </p> http://localhost/ticket/1223 http://localhost/ticket/1223 Report dread <p> Using ckanapi </p> http://localhost/ticket/1225 http://localhost/ticket/1225 Report dread <p> <a class="missing wiki">IndexError?</a> exception in genshi.filters.i18n keeps popping up daily. </p> <p> e.g. </p> <pre class="wiki">... Module ckan.controllers.authorization_group:28 in index &lt;&lt; items_per_page=20 ) return render('authorization_group/index.html') def read(self, id): &gt;&gt; return render('authorization_group/index.html') Module ckan.lib.base:74 in render &lt;&lt; return cached_template(template_name, render_template, cache_key=cache_key, cache_type=cache_type, cache_expire=cache_expire) #, ns_options=('method'), method=method) &gt;&gt; cache_type=cache_type, cache_expire=cache_expire) Module pylons.templating:249 in cached_template &lt;&lt; return content else: return render_func() &gt;&gt; return render_func() Module ckan.lib.base:63 in render_template &lt;&lt; stream = item.filter(stream) return literal(stream.render(method=method, encoding=None)) if 'Pragma' in response.headers: &gt;&gt; return literal(stream.render(method=method, encoding=None)) Module genshi.core:183 in render &lt;&lt; method = self.serializer or 'xml' generator = self.serialize(method=method, **kwargs) return encode(generator, method=method, encoding=encoding, out=out) def select(self, path, namespaces=None, variables=None): &gt;&gt; return encode(generator, method=method, encoding=encoding, out=out) Module genshi.output:57 in encode &lt;&lt; _encode = lambda string: string if out is None: return _encode(''.join(list(iterator))) for chunk in iterator: out.write(_encode(chunk)) &gt;&gt; return _encode(''.join(list(iterator))) Module genshi.output:339 in __call__ &lt;&lt; for filter_ in self.filters: stream = filter_(stream) for kind, data, pos in stream: cached = cache_get((kind, data)) if cached is not None: &gt;&gt; for kind, data, pos in stream: Module genshi.output:670 in __call__ &lt;&lt; _gen_prefix = _gen_prefix().next for kind, data, pos in stream: output = cache_get((kind, data)) if output is not None: &gt;&gt; for kind, data, pos in stream: Module genshi.output:771 in __call__ &lt;&lt; push_text = textbuf.append pop_text = textbuf.pop for kind, data, pos in chain(stream, [(None, None, None)]): if kind is TEXT: &gt;&gt; for kind, data, pos in chain(stream, [(None, None, None)]): Module genshi.output:586 in __call__ &lt;&lt; def __call__(self, stream): prev = (None, None, None) for ev in stream: if prev[0] is START: if ev[0] is END: &gt;&gt; for ev in stream: Module genshi.core:288 in _ensure &lt;&lt; # unchanged yield event for event in stream: yield event &gt;&gt; for event in stream: Module genshi.core:288 in _ensure &lt;&lt; # unchanged yield event for event in stream: yield event &gt;&gt; for event in stream: Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): &gt;&gt; for mark, event in stream: Module genshi.filters.transform:1145 in __call__ &lt;&lt; :param stream: The marked event stream to filter """ for mark, event in stream: yield mark, event if mark is ENTER: &gt;&gt; for mark, event in stream: Module genshi.filters.transform:714 in __call__ &lt;&lt; stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event &gt;&gt; for mark, event in stream: Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): for event in stream: yield OUTSIDE, event &gt;&gt; for event in stream: Module genshi.core:288 in _ensure &lt;&lt; # unchanged yield event for event in stream: yield event &gt;&gt; for event in stream: Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): &gt;&gt; for mark, event in stream: Module ckanext.googleanalytics.plugin:62 in download_adder &lt;&lt; (downloaded %s times)&lt;/span&gt;''' count = None for mark, (kind, data, pos) in stream: if mark and kind == START: href = data[1].get('href') &gt;&gt; for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ &lt;&lt; stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event &gt;&gt; for mark, event in stream: Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): for event in stream: yield OUTSIDE, event &gt;&gt; for event in stream: Module genshi.core:288 in _ensure &lt;&lt; # unchanged yield event for event in stream: yield event &gt;&gt; for event in stream: Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): &gt;&gt; for mark, event in stream: Module genshi.filters.transform:1175 in __call__ &lt;&lt; """ callable_value = hasattr(self.value, '__call__') for mark, (kind, data, pos) in stream: if mark is ENTER: if callable_value: &gt;&gt; for mark, (kind, data, pos) in stream: Module genshi.filters.transform:714 in __call__ &lt;&lt; stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event &gt;&gt; for mark, event in stream: Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): for event in stream: yield OUTSIDE, event &gt;&gt; for event in stream: Module genshi.core:288 in _ensure &lt;&lt; # unchanged yield event for event in stream: yield event &gt;&gt; for event in stream: Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): for mark, event in stream: kind = event[0] if not (kind is None or kind is ATTR or kind is BREAK): &gt;&gt; for mark, event in stream: Module genshi.filters.transform:1145 in __call__ &lt;&lt; :param stream: The marked event stream to filter """ for mark, event in stream: yield mark, event if mark is ENTER: &gt;&gt; for mark, event in stream: Module genshi.filters.transform:714 in __call__ &lt;&lt; stream = iter(stream) next = stream.next for mark, event in stream: if mark is None: yield mark, event &gt;&gt; for mark, event in stream: Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): for event in stream: yield OUTSIDE, event &gt;&gt; for event in stream: Module genshi.template.base:618 in _include &lt;&lt; tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: &gt;&gt; for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include &lt;&lt; tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: &gt;&gt; for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include &lt;&lt; tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: &gt;&gt; for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:605 in _include &lt;&lt; from genshi.template.loader import TemplateNotFound for event in stream: if event[0] is INCLUDE: href, cls, fallback = event[1] &gt;&gt; for event in stream: Module genshi.template.markup:378 in _match &lt;&lt; for event in self._match(self._flatten(template, ctxt, **vars), ctxt, start=idx + 1, **vars): yield event &gt;&gt; ctxt, start=idx + 1, **vars): Module genshi.template.markup:327 in _match &lt;&lt; break for event in stream: # We (currently) only care about start and end events for matching &gt;&gt; for event in stream: Module genshi.template.base:545 in _flatten &lt;&lt; while 1: for kind, data, pos in stream: if kind is START and data[1]: &gt;&gt; for kind, data, pos in stream: Module genshi.filters.i18n:177 in _generate &lt;&lt; msgbuf.append(*previous) previous = None for event in msgbuf.translate(gettext(msgbuf.format())): yield event if previous: &gt;&gt; for event in msgbuf.translate(gettext(msgbuf.format())): Module genshi.filters.i18n:1029 in translate &lt;&lt; ) parts = parse_msg(string) parts_counter = {} for order, string in parts: &gt;&gt; parts = parse_msg(string) Module genshi.filters.i18n:1143 in parse_msg &lt;&lt; if string: parts.append((stack[-1], string)) return parts &gt;&gt; parts.append((stack[-1], string)) IndexError: list index out of range </pre> http://localhost/ticket/1226 http://localhost/ticket/1226 Report timmcnamara <p> In when displaying search results, it would be useful to also display the tags of a package. Sometimes it's difficult to infer the scope of what the package does from the title and the first sentence of the description. Tags are quite concise way to display rich information. </p> <p> ENV=datacatalos.org, with CKAN 1.4.2a </p> http://localhost/ticket/1227 http://localhost/ticket/1227 Report dread <p> When there are errors when logging into OpenID, these don't get displayed. You just get the login screen again. </p> http://localhost/ticket/1228 http://localhost/ticket/1228 Report thejimmyg <p> Progress can be tracked in this document: </p> <p> <a class="ext-link" href="https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&amp;hl=en_US#gid=0"><span class="icon">​</span>https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdFQzRnVoUHV6ckoyblI5TUdEcnJLVlE&amp;hl=en_US#gid=0</a> </p> http://localhost/ticket/1229 http://localhost/ticket/1229 Report kindly <p> Dgu tests failed as harvesting table are not created at the correct point. </p> http://localhost/ticket/1230 http://localhost/ticket/1230 Report thejimmyg <p> Child tickets: </p> <ul><li><a class="new ticket" href="http://localhost/ticket/1101" title="enhancement: Integrate googlanalytics into site nav (new)">#1101</a> Integrate stats and googlanalytics into site nav </li></ul><p> We have a spreadsheet from UKLP of statistics we'd like to generate </p> http://localhost/ticket/1231 http://localhost/ticket/1231 Report thejimmyg <p> Child tickets: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1194" title="defect: &#34;Welcome back&#34; message for newly registered user (closed: fixed)">#1194</a> "Welcome back" message for newly registered user </li><li><a class="closed ticket" href="http://localhost/ticket/1202" title="enhancement: Links to datapkg utility don't lead to info about it (closed: invalid)">#1202</a> Links to datapkg utility don't lead to info about it </li><li><a class="closed ticket" href="http://localhost/ticket/925" title="defect: Change the search box icon to remove the down arrow (closed: fixed)">#925</a> Change the search box icon to remove the down arrow </li><li><a class="closed ticket" href="http://localhost/ticket/923" title="defect: Search box doesn't work in leaderboard page (closed: worksforme)">#923</a> Search box doesn't work in leaderboard page in stats extension </li><li><a class="closed ticket" href="http://localhost/ticket/1034" title="defect: Flash message cached (closed: duplicate)">#1034</a> Flash message cached </li><li><a class="new ticket" href="http://localhost/ticket/737" title="enhancement: Markdown syntax summary page (new)">#737</a> Markdown syntax summary page </li><li><a class="new ticket" href="http://localhost/ticket/811" title="defect: Extra field editing form layout breaks when there are long field names (new)">#811</a> Extra field editing form layout breaks when there are long field names </li></ul> http://localhost/ticket/1232 http://localhost/ticket/1232 Report thejimmyg <p> At the moment we have a good revisioning system but a poor history interface. We need to improve this in a number of areas: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/191" title="enhancement: Searching by modification date (closed: fixed)">#191</a> Searching by modification date </li><li><a class="closed ticket" href="http://localhost/ticket/193" title="enhancement: Searching by time-related field (closed: wontfix)">#193</a> Searching by time-related field </li><li><a class="assigned ticket" href="http://localhost/ticket/301" title="enhancement: Package discussion pages (assigned)">#301</a> Package discussion pages </li><li><a class="closed ticket" href="http://localhost/ticket/1236" title="enhancement: View a package at a given revision (closed: fixed)">#1236</a> Package history page should provide links to pages at particular revisions, similar to the wikipedia pages </li><li><a class="closed ticket" href="http://localhost/ticket/1236" title="enhancement: View a package at a given revision (closed: fixed)">#1236</a> Viewing old revisions or unmoderated changes should have a message at the top of the package page </li><li>Other improvements as per my word doc. </li></ul> http://localhost/ticket/1233 http://localhost/ticket/1233 Report dread <p> Needs tests for error conditions for bad locale and redirect parameters. </p> http://localhost/ticket/1234 http://localhost/ticket/1234 Report thejimmyg <p> Child tickets: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/234" title="enhancement: UI Review - Autocomplete package names &amp; tags in search (closed: wontfix)">#234</a> UI Review - Autocomplete package names &amp; tags in search </li><li><a class="closed ticket" href="http://localhost/ticket/193" title="enhancement: Searching by time-related field (closed: wontfix)">#193</a> Searching by time-related field </li><li><a class="closed ticket" href="http://localhost/ticket/191" title="enhancement: Searching by modification date (closed: fixed)">#191</a> Searching by modification date </li><li><a class="closed ticket" href="http://localhost/ticket/905" title="defect: Unable to search with accented characters in package names (closed: worksforme)">#905</a> Unable to search with accented characters in package names </li><li><a class="closed ticket" href="http://localhost/ticket/906" title="enhancement: Ability to search without accents for accented words (closed: fixed)">#906</a> Ability to search without accents for accented words </li><li><a class="new ticket" href="http://localhost/ticket/924" title="enhancement: Search box has no search button (new)">#924</a> Search box has no search button </li></ul><p> Broadly speaking though we need to choose PostgreSQL, Solr or something else. We don't want to invest our time maintaining two search backends with a limited abstraction layer between the two. </p> http://localhost/ticket/1235 http://localhost/ticket/1235 Report dread <p> As a user I want to view a package at a given revision: </p> <ul><li>When I visit /package/osm@yyyy-mm-dd or /package/osm@rrrrrrr I should be shown package at date yyyy-mm-dd or revision rrrrrrr. </li><li>viewing this page, it should say it is the old revision, just like wikipedia </li><li>package history page should provide links to these pages </li></ul> http://localhost/ticket/1236 http://localhost/ticket/1236 Report dread <p> There's a problem with getting old versions of packages using @date. The query looks for the <a class="missing wiki">PackageRevision?</a> related to the date, but ignores <a class="missing wiki">TagRevisions?</a> etc. </p> <p> So for example if you compare this table: <a class="ext-link" href="http://ckan.net/package/history/osm"><span class="icon">​</span>http://ckan.net/package/history/osm</a> with this view: <a class="ext-link" href="http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830"><span class="icon">​</span>http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830</a> you actually see the 2010-01-13 11:13 revision - the wrong revision. </p> http://localhost/ticket/1238 http://localhost/ticket/1238 Report rgrp <p> See reports on list at: </p> <ul><li><a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2011-July/001103.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2011-July/001103.html</a> </li><li><a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2011-July/001107.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2011-July/001107.html</a> </li></ul><p> Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix. </p> http://localhost/ticket/1239 http://localhost/ticket/1239 Report rgrp <p> (Just creating this ticket as somewhere to keep notes) </p> <ul><li>Decide on REST api versus action API <ul><li>Do we want to support both? </li></ul></li><li>Tidying <ul><li>Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...) </li><li>extras merged into normal field list in package </li><li>Get rid of /rest/ so just have api/v1/package </li><li>Get rid of separation of search api from 'rest' api <ul><li>Propose that GET on REST index is search e.g. /package/?q=... <ul><li>This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned) </li></ul></li></ul></li></ul></li><li>Resource read/write in API (separate from package) <ul><li>Does this need authorization work? </li></ul></li><li>user/account API - read/write </li><li>Remove autocomplete -- can just use search <ul><li>Do not worry about backwards compat as should only be used in our js (if others using it too bad!) </li></ul></li></ul> http://localhost/ticket/1240 http://localhost/ticket/1240 Report timmcnamara <p> Produce a video explaining some of the things which can be done on the datahub.org </p> http://localhost/ticket/1241 http://localhost/ticket/1241 Report timmcnamara <p> The right hand side of <a class="ext-link" href="http://thedatahub.org/user/login"><span class="icon">​</span>thedatahub.org login screen</a> is a little confusing: </p> <pre class="wiki"> Not a member? Join CKAN to contribute packages under your own name. - Register - Reset your password - Privacy Policy </pre><p> Suggested improvements to make: </p> <ul><li>replace CKAN with thedatahub.org </li><li>in general, edit the copy to be more inviting </li><li>remove "Reset your password" from that bubble </li><li>move the privacy policy link to somewhere less prominent </li></ul> http://localhost/ticket/1242 http://localhost/ticket/1242 Report timmcnamara <p> If someone who is not logged in clicks on "Upload Data", they are presented with </p> <pre class="wiki"> Not authorized to upload files. </pre><p> This is in emboldened red text. That is, the system is telling a user that they're doing something illegitimate, even though they're actually following an invitation from the main menu. </p> <p> I propose two options for resolving this: </p> <ol><li>only display the "Upload Data" tab if someone is authorised to upload data </li><li>if someone isn't logged int, provide a nice welcome message thanking them for wanting to upload data, but we only allow registered users to do so </li></ol><p> ENV: thedatahub.org, CKAN 1.4.2b </p> http://localhost/ticket/1243 http://localhost/ticket/1243 Report dread <p> When you edit a package in the web form, if the notes field had <tt>\n</tt> as the End Of Line symbol, it gets lost when you preview or save the package, and the notes field is displayed all on one line. </p> <p> This can be seen when editing annakarenina (as created by 'paster create-test-data'). The diff shows for example: </p> <pre class="wiki">- Some test notes + Some test notes ? + </pre><p> but it would more clearly be shown as: </p> <pre class="wiki">- Some test notes\n + Some test notes ? ++ </pre><p> This is a significant problem with DGU, since a lot comes in via the API. </p> <p> It's not clear what we should do about it. We could standardise on \n or \r\n when the form submission comes in. Do different browsers on different platforms do different things with EOLs? </p> <h2 id="Analysis">Analysis</h2> <p> Displaying the package: the Markdown processor respects both EOLs when displaying the field, putting each line in a &lt;p&gt; tag. </p> <p> Creating the package edit form: placed into &lt;textfield&gt;. </p> <p> Browser displaying package edit form: &lt;textfield&gt; displays \n and \r\n as EOL. But \n\n gets compressed to one EOL. But on submission, both are returned as \r\n. </p> <p> Receiving the edited package: Somewhere along the line the EOL gets converted to \n\n. </p> http://localhost/ticket/1244 http://localhost/ticket/1244 Report rgrp <p> Here is Rufus' plan for ckan.org reorganisation from <a class="ext-link" href="http://ckan.okfnpad.org/documentation"><span class="icon">​</span>http://ckan.okfnpad.org/documentation</a> (22/7/11) (ticket made by dread): </p> <pre class="wiki">/solutions /overview &lt;--- current 'about' page (the software) /customers &lt;-- say "in addition to our clients..." /features &lt;-- needs most work (pictures etc) /pricing/ &lt;-- simplest thing possible ... talk with Ira (cover pricing and support) /developers &lt;--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact /blog </pre><p> Tasks: </p> <ul><li>add tagline in header - "the data portal software" </li><li>make and add glossy PDF </li><li>change "data hub" to "data portal" passim (if only using one term) </li><li>add info@… </li><li>image banner on home page - redo screenshots without menus </li></ul> http://localhost/ticket/1245 http://localhost/ticket/1245 Report adrian.pohl@… <p> All package search results on ckan.net are labelled as 'open' even when their license is closed or unknown: <a class="ext-link" href="http://ckan.net/package"><span class="icon">​</span>http://ckan.net/package</a> </p> http://localhost/ticket/1246 http://localhost/ticket/1246 Report dread <p> Put docs on <a class="missing wiki">ReadTheDocs?</a> instead of eu13. </p> http://localhost/ticket/1247 http://localhost/ticket/1247 Report dread <p> Exception received on test.ckan.net at 10.12 on 29/7/11 </p> <pre class="wiki">http://test.ckan.net/package?__cache=2759373&amp;page=4 Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckan.controllers.package:153 in search &lt;&lt; } query = get.package_search(context,data_dict) c.page = h.Page( &gt;&gt; query = get.package_search(context,data_dict) Module ckan.logic.action.get:514 in package_search &lt;&lt; for package in query.results: result_dict = table_dictize(package, context) result_dict = _extend_package_dict(result_dict,context) results.append(result_dict) &gt;&gt; result_dict = _extend_package_dict(result_dict,context) Module ckan.logic.action.get:537 in _extend_package_dict &lt;&lt; license_id = package_dict['license_id'] if license_id: isopen = model.Package.get_license_register()[license_id].isopen() package_dict['isopen'] = isopen else: &gt;&gt; isopen = model.Package.get_license_register()[license_id].isopen() Module ckan.model.license:59 in __getitem__ &lt;&lt; return default else: raise KeyError, "License not found: %s" % key def get(self, key, default=None): &gt;&gt; raise KeyError, "License not found: %s" % key KeyError: u'License not found: None' </pre><p> and a similar one: </p> <pre class="wiki"> http://test.ckan.net/tag/barbecue Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:118 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckan.controllers.tag:2 in read Module ckan.lib.cache:167 in wrapper &lt;&lt; log = __import__("logging").getLogger("proxy_cache") def wrapper(func, *args, **kwargs): result = func(*args, **kwargs) pylons = get_pylons(args) &gt;&gt; result = func(*args, **kwargs) Module ckan.controllers.tag:64 in read &lt;&lt; data_dict = {'id':id} try: c.tag = get.tag_show(context,data_dict) except NotFound: abort(404, _('Tag not found')) &gt;&gt; c.tag = get.tag_show(context,data_dict) Module ckan.logic.action.get:341 in tag_show &lt;&lt; extended_packages = [] for package in tag_dict['packages']: extended_packages.append(_extend_package_dict(package,context)) tag_dict['packages'] = extended_packages &gt;&gt; extended_packages.append(_extend_package_dict(package,context)) Module ckan.logic.action.get:537 in _extend_package_dict &lt;&lt; license_id = package_dict['license_id'] if license_id: isopen = model.Package.get_license_register()[license_id].isopen() package_dict['isopen'] = isopen else: &gt;&gt; isopen = model.Package.get_license_register()[license_id].isopen() Module ckan.model.license:59 in __getitem__ &lt;&lt; return default else: raise KeyError, "License not found: %s" % key def get(self, key, default=None): &gt;&gt; raise KeyError, "License not found: %s" % key KeyError: u'License not found: CreativeCommonsAttributionCCBY25' </pre> http://localhost/ticket/1248 http://localhost/ticket/1248 Report rgrp <p> Currently have this script section put in for i18n. It shouldn't be. </p> <pre class="wiki">&lt;script type="text/javascript"&gt; //&lt;![CDATA[ (function($){ $.fn.ajaxCreateSlug = function(name, url) { var title = this; var updater = { init: function(title, name) { // Add a new element where the validity of the package name can be displayed this.name_field = name; this.title_field = title; this.name_field.parent().append('&lt;div id="package_name_valid_msg"&gt;&lt;/div&gt;'); this.title_field.blur(this.title_change_handler()) this.title_field.keyup(this.title_change_handler()) this.name_field.keyup(this.name_change_handler()); this.name_field.blur(this.name_blur_handler()); this.url = url; }, .... </pre> http://localhost/ticket/1249 http://localhost/ticket/1249 Report rgrp <p> At the moment we sort search results alphabetically. While this is useful for doing 'browse' case where no search bad for all other cases. </p> <p> Adopt default sort order of 'score' though may wish to keep alphabetical for no search term (i.e. wildcard). </p> <p> Options: </p> <ul><li>Default this in solr (no need to touch code) but fragile and affects everything ... </li><li>Do it in code and default to score </li><li>Do it in code and have alphabetical (on name or title?) when no criteria otherwise score </li></ul><p> Aside: may also wish to support search in query api but that is for later! </p> http://localhost/ticket/1250 http://localhost/ticket/1250 Report rgrp <p> Inline with general naming policy going forward. </p> http://localhost/ticket/1251 http://localhost/ticket/1251 Report rgrp <p> Deprecate ckan.net in favour of thedatahub.org </p> <ul><li>Announce </li><li>Create thedatahub.org and have it run same data as ckan.net </li><li>Announce closing down of ckan.net with 302 </li><li>Implement 302 permanent </li></ul><p> Also redirect wiki.ckan.net to wiki.ckan.org </p> http://localhost/ticket/1252 http://localhost/ticket/1252 Report amercader <p> We agreed on a system that allows overriding the authz checks at a logic function level (e.g. package_create, user_show) and supports the old (current) authz system. </p> http://localhost/ticket/1253 http://localhost/ticket/1253 Report dread <p> Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers. </p> http://localhost/ticket/1254 http://localhost/ticket/1254 Report kindly <p> Make a robust way to make sure the drupal database is consistent with the ckan data. </p> http://localhost/ticket/1255 http://localhost/ticket/1255 Report kindly <p> Solr is not working for moderation, pending changes are not being indexed. </p> http://localhost/ticket/1256 http://localhost/ticket/1256 Report dread <p> We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible. </p> <p> Spam: </p> <ul><li>Package creation </li><li>Package edit </li><li>User creation </li><li>Group creation </li></ul><p> Systems to consider: </p> <ul><li>Automatic <ul><li>captcha </li><li>bayesian scoring </li></ul></li><li>Sysadmin <ul><li>a tool to quickly analyse and remove packages, edits and users </li></ul></li><li>Helpful users <ul><li>button to click indicating spam, leading to any/all of: <ul><li>add TODO item (which emails package admin and sysadmins) </li><li>quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it </li></ul></li></ul></li></ul><p> General thoughts: </p> <ul><li>We should learn from the wikipedia tools, as they are the experts </li><li>This is a serious problem that is only going to get worse and is a threat to the whole site. </li></ul> http://localhost/ticket/1257 http://localhost/ticket/1257 Report kindly <p> In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory. </p> http://localhost/ticket/1258 http://localhost/ticket/1258 Report pudo <p> The default package form offers 4 empty extras fields. Like the resource section, it should have an "add more" button to add another row. </p> http://localhost/ticket/1259 http://localhost/ticket/1259 Report pudo <p> There seems to be both a list view for dictized and non dictized data structures for all entities in _util.html at the moment. Probably in the back of someone's mind already, but cleanup here would be nice. </p> http://localhost/ticket/1260 http://localhost/ticket/1260 Report pudo <p> It seems that searching for extras_foo:value works with solr, but extras_foo.bar:value doesn't. No theory on why. </p> http://localhost/ticket/1261 http://localhost/ticket/1261 Report pudo <p> This does not seem to have any implications at the moment, it should lock down registration and remove all related links. </p> http://localhost/ticket/1262 http://localhost/ticket/1262 Report dread <p> A bad action name: e.g. curl <a class="ext-link" href="http://test.ckan.net/api/action/user_showaa"><span class="icon">​</span>http://test.ckan.net/api/action/user_showaa</a> -d '{}' causes 500 response. </p> <p> Occurs only on 1.4.3 beta. </p> http://localhost/ticket/1263 http://localhost/ticket/1263 Report rgrp <p> This extension should be in core now -- functionality is basic and very useful. </p> <p> Copy across from here: <a class="ext-link" href="http://bitbucket.org/okfn/ckanext-admin"><span class="icon">​</span>http://bitbucket.org/okfn/ckanext-admin</a> </p> <p> Suggest this no longer even be an extension. </p> http://localhost/ticket/1264 http://localhost/ticket/1264 Report dread <p> We get exceptions on these occasions: </p> <ul><li>user/edit if not logged in </li><li>user/edit/bob if bob is not a known user </li></ul><p> These are a problem only on 1.4.3b. </p> <p> Introduced in <a class="closed ticket" href="http://localhost/ticket/1229" title="enhancement: Refactor all database code out of the controllers and into the logic layer (closed: fixed)">#1229</a> - merge of branch feature-1229-db-out-of-controllers </p> http://localhost/ticket/1265 http://localhost/ticket/1265 Report dread <p> I'm not sure why some timestamps are missing microseconds in the ckan.net data, but plenty seem to. This causes this exception when viewing the package rss feed: </p> <pre class="wiki">URL: http://ckan.net/package/history/sfk_publishing?format=atom&amp;days=7 In Module ckan.lib.helpers:227 in date_str_to_datetime WebApp Error: &lt;type 'exceptions.ValueError'&gt;: time data '2008-04-13T20:40:20' does not match format '%Y-%m-%dT%H:%M:%S.%f' </pre><p> Only seen with ckan 1.4.3b </p> http://localhost/ticket/1266 http://localhost/ticket/1266 Report dread <p> Occasionally see this problem when producing an error page. Here's an example </p> <pre class="wiki">WebApp Error: &lt;type 'exceptions.TypeError'&gt;: You cannot set the body to a unicode value without a charset URL: http://at.ckan.net/authorizationgroup/new Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:117 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:284 in __call__ &lt;&lt; if log_debug: log.debug("Calling Response object to return WSGI data") return response(environ, self.start_response) if log_debug: &gt;&gt; return response(environ, self.start_response) Module webob.exc:248 in __call__ &lt;&lt; return [] if not self.body and not self.empty_body: return self.generate_response(environ, start_response) return Response.__call__(self, environ, start_response) &gt;&gt; return self.generate_response(environ, start_response) Module webob.exc:239 in generate_response &lt;&lt; status=self.status, headerlist=headerlist, content_type=content_type ) return resp(environ, start_response) &gt;&gt; content_type=content_type Module webob.response:94 in __init__ &lt;&lt; if charset is None: raise TypeError( "You cannot set the body to a unicode value without a charset") body = body.encode(charset) self._body = body &gt;&gt; "You cannot set the body to a unicode value without a charset") TypeError: You cannot set the body to a unicode value without a charset </pre> http://localhost/ticket/1267 http://localhost/ticket/1267 Report rgrp <p> At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance. </p> <p> Would this also speed up user page where we recent changes? </p> http://localhost/ticket/1268 http://localhost/ticket/1268 Report rgrp <p> This overlaps with Sean's follow support (for activity streams) and so will be dealt with there. </p> http://localhost/ticket/1269 http://localhost/ticket/1269 Report dread <p> The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters: </p> <pre class="wiki">In [1]: import ckanclient In [2]: c = ckanclient.CkanClient() In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api') In [4]: c.package_search('', search_options={'groups':'openspending'}) Out[4]: {u'count': 2102, u'results': &lt;generator object _result_generator at 0x10168b1e0&gt;} In [5]: c2.package_search('', search_options={'groups':'openspending'}) Out[5]: {u'count': 29, u'results': &lt;generator object _result_generator at 0x10168b410&gt;} </pre><p> Thanks to borior for finding this and raising it. </p> http://localhost/ticket/1270 http://localhost/ticket/1270 Report rgrp <p> CORS - <a class="ext-link" href="http://www.w3.org/TR/cors/"><span class="icon">​</span>http://www.w3.org/TR/cors/</a> - support. </p> <p> This is what you do in Apache. Should do this in lib/base.py or similar. </p> <pre class="wiki"> Header always set Access-Control-Allow-Origin "*" Header always set Access-Control-Allow-Methods "POST, PUT, GET, OPTIONS" Header always set Access-Control-Allow-Headers "X-CKAN-API-KEY, Content-Type" # Respond to all OPTIONS requests with 200 OK # This could be done in the webapp # This is need for pre-flighted requests (POSTs/PUTs) RewriteEngine On RewriteCond %{REQUEST_METHOD} OPTIONS RewriteRule ^(.*)$ $1 [R=200,L] </pre> http://localhost/ticket/1271 http://localhost/ticket/1271 Report amercader <p> Consolidate the extra <tt>spatial</tt> as the field for providing information about the geographic extent of the package. This will integrate automatically with the spatial search and give developers a single point to develop geospatial features. </p> http://localhost/ticket/1272 http://localhost/ticket/1272 Report amercader http://localhost/ticket/1273 http://localhost/ticket/1273 Report johnglover <p> Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend. </p> http://localhost/ticket/1274 http://localhost/ticket/1274 Report johnglover http://localhost/ticket/1275 http://localhost/ticket/1275 Report johnglover <p> For example: </p> <ul><li>name and title should have a higher weight than the notes field </li></ul> http://localhost/ticket/1276 http://localhost/ticket/1276 Report johnglover <p> See <a class="ext-link" href="http://ckan.okfnpad.org/search"><span class="icon">​</span>http://ckan.okfnpad.org/search</a> for details/discussion </p> http://localhost/ticket/1277 http://localhost/ticket/1277 Report amercader <p> There are some functions that still use the Auhtorizer().authorized_query method: </p> <pre class="wiki">./ckan/controllers/authorization_group.py:24: query = ckan.authz.Authorizer().authorized_query(c.user, model.AuthorizationGroup) ./ckan/lib/base.py:237: groups = ckan.authz.Authorizer.authorized_query(c.user, model.Group, ./ckan/lib/search/sql.py:55: q = authz.Authorizer().authorized_query(username, model.Group) ./ckan/lib/search/sql.py:118: q = authz.Authorizer().authorized_query(self.options.get('username'), model.Package) ./ckan/logic/action/get.py:154: query = Authorizer().authorized_query(user, model.Group, model.Action.EDIT) ./ckan/tests/test_authz.py:158: q = self.authorizer.authorized_query(self.notadmin.name, model.Package) ./ckan/tests/test_authz.py:353: q = self.authorizer.authorized_query(self.notmember.name, model.Package) ./ckan/tests/test_authz.py:357: q = self.authorizer.authorized_query(self.member.name, model.Package) ./ckan/tests/functional/test_authorization_group.py:44: group_count = Authorizer.authorized_query(u'russianfan', model.AuthorizationGroup).count() </pre> http://localhost/ticket/1278 http://localhost/ticket/1278 Report amercader <p> Authorization Groups are going to be deprecated in 1.8 </p> <p> It is still unclear how Authorization Groups fit in the new authz model, so they have been left out of the logic and auth refactoring (Tickets <a class="closed ticket" href="http://localhost/ticket/1229" title="enhancement: Refactor all database code out of the controllers and into the logic layer (closed: fixed)">#1229</a> and <a class="closed ticket" href="http://localhost/ticket/1253" title="enhancement: Authz refactor (closed: fixed)">#1253</a>) </p> http://localhost/ticket/1279 http://localhost/ticket/1279 Report kindly <p> After the ckan migrate process, sqlalcehmy reflect does not work and causes an error when reflecting indexes. A bug report needs to be put into sqlalchemy. </p> http://localhost/ticket/1280 http://localhost/ticket/1280 Report pudo <p> When you change the locale, it displays a flash message "Language has been set to: English" in the original language, not the one we have just changed to. </p> http://localhost/ticket/1281 http://localhost/ticket/1281 Report rgrp <p> Consolidate js. Remove cruft. </p> <ul><li>Move stuff to vendor <ul><li>Have all vendor libs local (?) </li><li>jquery, jquery-ui etc </li></ul></li><li>Consolidate autocomplete to use jquery-ui <ul><li>package autocomplete (groups) </li><li>tag autocomplete (package) </li><li>user autocomplete (authz group create) <ul><li>turns out that authz group edit had autocomplete disabled so refactoring meant enabling ... </li></ul></li></ul></li><li>Move forms.css into main </li><li>Remove tag cloud from front page (can be example going forward) </li><li>Remove bookmarklet (since in wiki at: <a class="ext-link" href="http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F"><span class="icon">​</span>http://wiki.ckan.net/Managing_Data_Packages#How_Do_I_Install_the_CKAN_Bookmarklet.3F</a>) </li><li>Move name slug generation into standard js and clean up </li></ul><p> Possible: </p> <ol><li>JS unit tests for the given pages ... </li><li>Option to use external site to serve js and css (e.g. CDN) </li></ol><p> </p> http://localhost/ticket/1282 http://localhost/ticket/1282 Report dread <p> A package in deleted state doesn't show to general users, but admins DO see them. The idea is that they can resurrect them if they want. But now we have the trash can that lists deleted packages. So we don't want admins to see the deleted packages anywhere else. </p> <p> (But an admin should still be able to READ and EDIT a deleted package, so if he clicks on it in the trash list he can see it to decide whether to resurrect and actually be able to change its state.) </p> <p> Same in the Web interface and API. </p> <p> This ticket ties in with <a class="assigned ticket" href="http://localhost/ticket/948" title="enhancement: Highlight (to a sysadmin) which packages are deleted (assigned)">#948</a> highlighting a package being deleted (when viewing or editing) and follows on from the introduction of the trash can <a class="closed ticket" href="http://localhost/ticket/1076" title="enhancement: Improve revision and package purge system (closed: fixed)">#1076</a>. </p> http://localhost/ticket/1283 http://localhost/ticket/1283 Report rgrp <p> Revisions aren't assigned to packages. Where are they coming from? This seems to have been occurring since the upgrade to 1.4.3b a couple of weeks ago? </p> http://localhost/ticket/1284 http://localhost/ticket/1284 Report dread <p> Currently a sysadmin gets an email when an exception is not caught. But there are occasions when we DO want to catch an exception so we can fail nicely for the user, but the sysadmin STILL gets an email to know to fix something. e.g. if there is an exception when search indexes a package. You want to catch the exception and still run any other notify calls. </p> http://localhost/ticket/1285 http://localhost/ticket/1285 Report rgrp <p> Stuff I've spotted: </p> <ul><li>forms/* </li><li>template/group/edit_form.html </li><li>template/package/edit_form.html </li></ul><p> This can go once new DGU form is in. </p> http://localhost/ticket/1286 http://localhost/ticket/1286 Report dread <p> When you create a package, but specify a key that is not allowed (e.g. 'relationships') then you get error message: </p> <pre class="wiki">{"__junk": ["The input field __junk was not expected."]} </pre><p> It should mention the actual key which is not expected. e.g. </p> <pre class="wiki">{"relationships": ["The input field 'relationships' was not expected."]} </pre><p> Kindly said that James' version of NAVL was better in this respect, so this might be best solved by moving to that. </p> http://localhost/ticket/1287 http://localhost/ticket/1287 Report dread <p> When you create or edit a package (via the API), you aren't able to specify the relationships it has. (If you do you get 409 <tt> {"__junk": ["The input field __junk was not expected."]} </tt>) </p> <p> The normal way to create relationships is via /api/rest/relationships/ and this works. But when you GET a package, the dictionary lists all relationship details. So this bug creates a problem for editing a package that has relationships - you want to GET it, make any edits and then PUT it back. The work-around is to delete the 'relationships' key from the dict before you PUT it back. </p> <h2 id="Options">Options</h2> <p> Ideally, CKAN would read the 'relationships' key and make the necessary changes. This is a chunk of work. </p> <p> Another good option is to allow an unchanged 'relationships' value, but barf it is edited. This is also a chunk of work. </p> <p> A bad option would be to just ignore the 'relationships' value, since users will get frustrated changing this value and wonder why it never saves, not understanding it is different to all the rest, without error message. </p> <p> A final option is to get rid of relationships altogether. </p> http://localhost/ticket/1288 http://localhost/ticket/1288 Report dread <h2 id="Abstract">Abstract</h2> <p> Package Relationships have not taken off in the 18 months we've had them in the API. There are some issues with them and we need to spend more time improving them or consider getting rid of them. </p> <h2 id="TheProblem">The Problem</h2> <p> Original use cases are expressed here: <a class="assigned ticket" href="http://localhost/ticket/253" title="enhancement: Package relationships (assigned)">#253</a> Here are comments about how we could handle these specific examples better: </p> <ol><li>groups of packages - maybe better with a custom tag? </li><li>fragment resources - soon to be covered by 'kind' resource field <a class="closed ticket" href="http://localhost/ticket/957" title="enhancement: new &#34;kind &#34; field added to resources (closed: wontfix)">#957</a> </li></ol><p> 3&amp;5. derived resource - better to have some sort of resource relationship perhaps? </p> <ol start="4"><li>linked resource - again better to have some sort of resource relationship perhaps? </li></ol><p> Outstanding issues needing serious effort to fix: </p> <ul><li><a class="assigned ticket" href="http://localhost/ticket/256" title="requirement: Package relationships - 3. Edit in WUI (assigned)">#256</a> Editing them in Web UI (not done yet) </li><li><a class="new ticket" href="http://localhost/ticket/1288" title="defect: Package edit/creation can't include 'relationships' field (new)">#1288</a> Package edit/creation can't include 'relationships' field </li></ul><h2 id="Specification">Specification</h2> <p> Remove relationships from model, API, tests, Web UI. Data migration to remove from db. </p> <h2 id="Whydoitthisway">Why do it this way</h2> <p> Getting frustrated having problems with the code, when it's not used much. Often asked about what it's for, but rarely used. Seems an overly complicated design. </p> <h2 id="BackwardsCompatibility">Backwards Compatibility</h2> <p> n/a </p> <h2 id="Implementationplan">Implementation plan</h2> <h3 id="Deliverables">Deliverables</h3> <p> See Specification </p> <h3 id="Risksandmitigations">Risks and mitigations</h3> <p> Risk: a customer suddenly wants this, and the new ways to relate resources are not in place yet. </p> <p> Mitigation: discuss this decision thoroughly to make sure we are confident the use cases are not important. Discuss with team, ckan-discuss and specifically the LOD people who have some related packages on thedatahub.org. </p> <h3 id="Participants">Participants</h3> <p> David Read </p> <h3 id="Progress">Progress</h3> <p> Not yet. </p> http://localhost/ticket/1289 http://localhost/ticket/1289 Report dread <p> When installing CKAN, when doing "paster serve development.ini", lots of users encounter the error for every request: </p> <pre class="wiki">ProgrammingError: (ProgrammingError) relation "user" does not exist </pre><p> This is because the database tables have not been created - they have forgotten or missed the "paster db init" step. </p> <p> Can we provide a better error to say that the database is not initialised yet? </p> <h2 id="Implementationoptions">Implementation options</h2> <ol><li>At the start of every request we reflect the database tables and check they are there. This is rather expensive! </li></ol><ol start="2"><li>Uncached requests to the home page start with a cheap database query. If there is an exception then return this error about database setup. I really like Drupal's page for this that has in large letters that the site is currently off-line. Below a line, in small letters, there are developer suggestions on what is wrong and where to look to fix it. </li></ol> http://localhost/ticket/1290 http://localhost/ticket/1290 Report kindly <p> Add tables described in <a class="ext-link" href="http://ckan.okfnpad.org/notifications"><span class="icon">​</span>http://ckan.okfnpad.org/notifications</a> </p> http://localhost/ticket/1291 http://localhost/ticket/1291 Report dread <ul><li>Review process of releases and updating strings (pot) and translations (po) files <ul><li>c.f. <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html</a> </li></ul></li><li>Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?) </li><li>Review long list of strings for improvement <ul><li>Consolidate any? </li><li>Get rid of i18n for strings in api </li><li>Try and remove javascript </li><li>Add in extensions </li></ul></li><li><a class="closed ticket" href="http://localhost/ticket/1374" title="defect: Can't switch to English if default is non-English (closed: fixed)">#1374</a> Fix switching to en from other default language </li><li><a class="closed ticket" href="http://localhost/ticket/1417" title="defect: Browser language detection doesn't work (closed: fixed)">#1417</a> Fix browser language detection </li></ul> http://localhost/ticket/1292 http://localhost/ticket/1292 Report rgrp <h2 id="Changes">Changes</h2> <ul><li>All templates </li><li>[DONE] Routing: /package/ -&gt; /dataset/ (including API) </li><li>[DONE] Tests: self.app.get('/dataset/...') rather than package ... <ul><li>One test to check redirect works? </li></ul></li><li>Docs </li></ul><h2 id="Leave">Leave</h2> <ul><li>Stay with Package in code e.g. stay with '<a class="missing wiki">PackageController?</a>' <ul><li>also leave stuff like pkg = .... (gradual migration) </li></ul></li><li>Leave extensions </li></ul><h2 id="Notes">Notes</h2> <ul><li>Package -&gt; Dataset </li><li>Data Package -&gt; Dataset </li><li>respect capitalization </li></ul> http://localhost/ticket/1293 http://localhost/ticket/1293 Report rgrp <p> Largely an integration of work from ckanjs </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1295" title="enhancement: Simplify package create form (closed: fixed)">#1295</a> Simplify package create form </li><li>Improvements to edit form <ul><li>Split edit form into sections </li><li>Remove preview from edit form (introduce inline preview on description/notes field) </li></ul></li><li><a class="closed ticket" href="http://localhost/ticket/1296" title="enhancement: Improved resource adding/editing on dataset/page page (closed: fixed)">#1296</a> Improved resource adding/editing on dataset/page page </li><li><a class="closed ticket" href="http://localhost/ticket/878" title="enhancement: Integrate file upload with workflow around package resources (closed: fixed)">#878</a> File upload integrated into resource creation </li><li><a class="closed ticket" href="http://localhost/ticket/1297" title="enhancement: In-place preview of edited description markdown on edit dataset form (closed: fixed)">#1297</a> In-place editing of notes field for existing packages </li></ul> http://localhost/ticket/1294 http://localhost/ticket/1294 Report rgrp <p> Sub ticket of <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a> </p> <ul><li>Initial screen just title, name, description (notes), license (and tags?). </li><li>State that you can add data and more information on next screen. This takes you to Package 'view' where you can then do more stuff. <ul><li>Flash some instructions? e.g. that resources are missing? </li></ul></li><li>Name slug should by default look uneditable (cf wordpress way to do the slug) </li></ul> http://localhost/ticket/1295 http://localhost/ticket/1295 Report rgrp <p> Subticket of <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a>. </p> <ul><li>Nice way to edit all resource attributes </li><li>Nice way to add new resources <ul><li>Including file upload (cf <a class="closed ticket" href="http://localhost/ticket/878" title="enhancement: Integrate file upload with workflow around package resources (closed: fixed)">#878</a>). <ul><li>Cf work in ckanjs (this can be a straight integration). </li></ul></li></ul></li></ul> http://localhost/ticket/1296 http://localhost/ticket/1296 Report rgrp <p> Subticket of <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a>. </p> <p> Can integrate from working system on ckanjs. </p> http://localhost/ticket/1297 http://localhost/ticket/1297 Report kindly <p> This should be done from the logic layer or automatically from a session extension. </p> http://localhost/ticket/1298 http://localhost/ticket/1298 Report kindly <p> Migrate tables for activity streams </p> http://localhost/ticket/1299 http://localhost/ticket/1299 Report rgrp <p> Core changes to theme in order to make it easier to re-theme: </p> <ul><li>page_heading </li><li>side bar menu must be switchable (left to right) (through config or css) </li><li>Add optional_footer to complement optional_head <ul><li>And adopt rule for template writers that all extra js must go in optional footer </li><li>That way we could move optional js to the bottom of the page to improve page load times. </li></ul></li><li>Re-add support for body-class ... (seems to be removed by <a class="closed ticket" href="http://localhost/ticket/1108" title="enhancement: Create a more modern theme for CKAN (closed: fixed)">#1108</a>) </li><li>more divs. </li><li>better labelling for css (and javascript) </li></ul> http://localhost/ticket/1300 http://localhost/ticket/1300 Report amercader <p> This was removed during the auth refactoring, but it's needed in the old controllers </p> http://localhost/ticket/1301 http://localhost/ticket/1301 Report johnglover <p> Resource and Tag search is currently still using the old ckan query parser which is unnecessary. Refactor this out and move to the logic layer. </p> http://localhost/ticket/1302 http://localhost/ticket/1302 Report amercader <p> Only happens if the user does not update the password. Most probably introduced during <a class="closed ticket" href="http://localhost/ticket/1229" title="enhancement: Refactor all database code out of the controllers and into the logic layer (closed: fixed)">#1229</a>. </p> http://localhost/ticket/1303 http://localhost/ticket/1303 Report amercader <p> Sending emails is failing in some cases, and the error message just says 'Could not send reset link'. </p> http://localhost/ticket/1304 http://localhost/ticket/1304 Report amercader <p> The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one. </p> <p> Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},) </p> <p> As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org </p> http://localhost/ticket/1305 http://localhost/ticket/1305 Report rgrp <p> AFAICT there are no tests for the logic layer at the moment. I imagine this is an issue (if it is not please explain and close as wontfix). </p> http://localhost/ticket/1306 http://localhost/ticket/1306 Report rgrp <ul><li>Major issue with spam signup on thedatahub.org </li><li>Suggest using recaptcha </li></ul><p> Est cost: 1d </p> http://localhost/ticket/1307 http://localhost/ticket/1307 Report dread <p> On the CKAN home page there is no suggestion that you can add or register a dataset. The 'Register a dataset' option has been removed at some point! (only visible to users that have registered - why would they do that when the benefits aren't described?) </p> http://localhost/ticket/1308 http://localhost/ticket/1308 Report zephod <p> Registering a new user directs you to that user's profile page but does not log you in. There is an empty set of "Recent changes" and no ability to modify the page. Users must follow the login link in the top-right corner to continue. </p> <p> Registering a new user should immediately log you in. </p> http://localhost/ticket/1309 http://localhost/ticket/1309 Report dread <p> e.g. this particular revision on ckan.net: <a class="ext-link" href="http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c"><span class="icon">​</span>http://ckan.net/api/rest/revision/9601fbcb-03a7-4ebc-981d-5aa16df4852c</a> (most others have not been approved) Causes 500 error. </p> http://localhost/ticket/1310 http://localhost/ticket/1310 Report rgrp <p> Subticket of: <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a> </p> <p> Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...). </p> <h2 id="Why">Why</h2> <ul><li>It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...) </li><li>It allows for quicker and easier logging in </li></ul><h2 id="Implementation">Implementation</h2> <ul><li>See Friedrich's work on the datahub </li></ul> http://localhost/ticket/1311 http://localhost/ticket/1311 Report florian.marienfeld@… <p> From Florian: </p> <pre class="wiki">I am having trouble with the characters "=&amp;;". if they occur e.g. in the title of a package, I get a 400-Bad-Request when I try to register a package by POSTing to ckan/rest/api/package However, when I PUT the same package as an edit to ckan/rest/api/package/existing_package, it works and the title shows correctly on the front end and API. </pre><h2 id="Reproduced">Reproduced</h2> <pre class="wiki">(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{"name": "test3", "title": "Test &amp;"}' -H "Authorization:tester" HTTP/1.0 400 Bad Request Server: PasteWSGIServer/0.5 Python/2.6.5 Date: Tue, 06 Sep 2011 14:09:31 GMT Pragma: no-cache Cache-Control: no-cache Content-Type: application/json;charset=utf-8 Content-Length: 87 "Bad request - JSON Error: Unterminated string starting at: line 1 column 27 (char 27)" </pre> http://localhost/ticket/1312 http://localhost/ticket/1312 Report dread <p> Awesome to have the captcha implementation and running on thedatahub.org! Here are some further suggestions concerning the config options, before this goes into a release: </p> <ul><li>Document "ckan.recaptcha.publickey" and "ckan.recaptcha.privatekey" in doc/configuration.rst </li><li>Which one config option defines whether recaptcha is enabled? There is disagreement between the template and the lib. Maybe better to have a third option for this. When setting the syntax for this, I can see us wanting to use captcha for other operations, like package edit, package create, group create etc. so perhaps we'd want to be able to turn each one on/off separately, using the name of the logic action. e.g. <tt>ckan.recaptcha.enable = user_create, group_create</tt> </li><li>We got this exception last night from the template: <pre class="wiki">Module ?:37 in &lt;Expression u'g.recaptcha_publickey'&gt; &lt;&lt; &lt;dd&gt;&lt;input type="password" name="password2" value="" /&gt;&lt;/dd&gt; &lt;dd py:if="g.recaptcha_publickey"&gt; &lt;script type="text/javascript" src="http://www.google.com/recaptcha/api/challenge?k=${g.recaptcha_publickey}"&gt; &gt;&gt; &lt;dd py:if="g.recaptcha_publickey"&gt; Module genshi.template.eval:324 in lookup_attr &lt;&lt; val = obj[key] except (KeyError, TypeError): val = cls.undefined(key, owner=obj) return val &gt;&gt; val = cls.undefined(key, owner=obj) Module genshi.template.eval:410 in undefined &lt;&lt; """Raise an ``UndefinedError`` immediately.""" __traceback_hide__ = True raise UndefinedError(key, owner=owner) &gt;&gt; raise UndefinedError(key, owner=owner) UndefinedError: &lt;ckan.lib.app_globals.Globals object at 0xb8f1182c&gt; has no member named "recaptcha_publickey" </pre></li></ul><p> I'm not sure why - might be something to do with the maintenance on the server, but worth double checking, since we don't want exceptions, say if one or other config variable is not defined. </p> http://localhost/ticket/1313 http://localhost/ticket/1313 Report dread <p> Apparently the search generator always makes two requests, even if you don't want to see the search results, which might be slow. Can this be optimised? </p> <p> Maybe we should also provide a second search function that doesn't use the generator - the original simple search function (that leaves the user to deal with limit &amp; offset). </p> http://localhost/ticket/1314 http://localhost/ticket/1314 Report dread <p> We need some documentation for the /api/action. Even though it's beta and subject to change it would be good to have a guide. </p> http://localhost/ticket/1315 http://localhost/ticket/1315 Report dread <pre class="wiki">curl http://ckan.net/api/action/package_show -d '{"name": "uk-quango-data"}' </pre><p> This gives 500 error due to <a class="missing wiki">KeyError?</a> (it looked for the 'id' param), rather than 400. </p> http://localhost/ticket/1316 http://localhost/ticket/1316 Report dread <p> In password reset, it gets confused if you have two similar users. This is because with the string the user provides, it searches several fields, not just name but also fullname and email address, allowing you to search for these. But only name is unique. </p> <p> Specific problem: Ira searches for "Irina" then it finds both: <tt>&lt;User name=irina fullname=Irina email=irina.bolychevsky@okfn.org ]</tt> and <tt>&lt;User name=shevski fullname=Ira email=&gt;</tt> (I think) </p> <p> Maybe need to choose which field it searches? </p> http://localhost/ticket/1317 http://localhost/ticket/1317 Report dread <p> When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected. </p> http://localhost/ticket/1318 http://localhost/ticket/1318 Report dread <p> Superticket: <a class="new ticket" href="http://localhost/ticket/1343" title="enhancement: [super] User related improvements (login, user pages etc) (new)">#1343</a> </p> <p> This avoids difficulties when they forget their password. </p> <ul><li>(?) Email validation (navl does not have one!) </li><li>Require email confirmation to be activated (?) (Answer: no, not at present -- see discussion below) </li></ul> http://localhost/ticket/1319 http://localhost/ticket/1319 Report rgrp <p> It is very annoying that you have to wrap links in &lt;...&gt; and most people don't bother. Extend h.markdown in code and markdown in js to fix this. </p> <p> (We should also support 'package' links etc in javascript?) </p> <p> *ADDITIONAL* </p> <ul><li>Note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package). </li><li>Move the markdown api to /util/markdown </li></ul> http://localhost/ticket/1320 http://localhost/ticket/1320 Report dread <p> When you reset a password, the flash message saying "check your email" sometimes doesn't display on the next page. You then click on the link in your email and it displays on that page (i.e. too late). </p> <p> Seen this on thedatahub.org. To do with the proxy caching? </p> http://localhost/ticket/1321 http://localhost/ticket/1321 Report dread <p> Focusing on improving Action API as the v3 API: </p> <ul><li>have an optional parameter of the data_dict called "options". Options would contain items that would get passed into the context. e.g. <tt>{"options" : {"ref_package_by": "id"}}. </tt> </li><li>instead of using API version to change the way packages are referenced, use the ref_package_by. </li><li>All package_show, group_show etc. to accept an object 'name' as an alternative to object 'id'. </li><li>Action API is v3 of api, replacement for v1 &amp; v2. Default for most urls is still v1, but if url is /api/action then default to v3. </li></ul><p> Next steps: </p> <ul><li>Add search API (package, resource, </li><li>Add Util API </li><li>Clarify JSONP still applies </li><li>Add doc strings, clarifying parameters </li></ul> http://localhost/ticket/1322 http://localhost/ticket/1322 Report dread <p> on thedatahub.org </p> http://localhost/ticket/1323 http://localhost/ticket/1323 Report kindly <p> add extra fields to resource as outlined in </p> <p> <a class="ext-link" href="http://wiki.ckan.net/Domain_Model/Resource"><span class="icon">​</span>http://wiki.ckan.net/Domain_Model/Resource</a> </p> <p> Specifically (all unicode strings except for those noted): </p> <ul><li><em>name</em>: a name for this resource (could be used in a ckan url) </li><li><em>type</em>: the type of the resource. One of: file | api | service | listing </li><li><em>mimetype</em>: standard mimetype (e.g. for zipped csv would be application/zip) </li><li><em>mimetype_inner</em>: mimetype of innermost object (so for example would be text/csv) </li><li><em>size</em>: [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file. </li><li><em>last_modified</em>: [datetime] the date when this resource's data was last modified (NB: <em>not</em> the date when the metadata was modified). </li><li>cache_url: url for cache of object in ckan storage </li><li>cache_last_updated: [datetime] timestamp when cached version was created </li><li>webstore_url: </li><li>webstore_last_updated: [datetime] timestamp when webstore was last updated </li></ul> http://localhost/ticket/1324 http://localhost/ticket/1324 Report dread <p> We claim to support Python 2.5 (useful for Lenny), yet a few problems have crept in: </p> <ul><li>'with' statement requires <tt>from __future__ import with_statement</tt> </li><li>import json from ckan.lib.helpers, rather than directly, since it copes with reverting to simplejson, used in python 2.5. </li></ul> http://localhost/ticket/1325 http://localhost/ticket/1325 Report thejimmyg <p> Ticket <a class="closed ticket" href="http://localhost/ticket/787" title="task: Auth API (closed: fixed)">#787</a> described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor. </p> http://localhost/ticket/1326 http://localhost/ticket/1326 Report rgrp <p> Split out of <a class="closed ticket" href="http://localhost/ticket/852" title="enhancement: [super] Dataset upload and archiving (closed: fixed)">#852</a>. Automated archiving of datasets (related to QA). </p> <p> <strong> Automated archiving using worker process </strong> </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/890" title="enhancement: Introduce timed actions into ckanext-queue (closed: invalid)">#890</a> - Timed actions in ckanext-queue </li><li><a class="closed ticket" href="http://localhost/ticket/891" title="task: Resource download worker daemon (closed: fixed)">#891</a> - Resource download worker daemon </li><li><a class="closed ticket" href="http://localhost/ticket/892" title="enhancement: Make stored data available in WUI - 0.5d (closed: fixed)">#892</a> - Make stored data available in WUI </li></ul> http://localhost/ticket/1327 http://localhost/ticket/1327 Report minspamboks@… <p> A possible bug in CKAN when I tried deleting users using "paster --plugin=ckan user delete" command. </p> <p> To reproduce the bug do the following: </p> <ol><li>Create a user with an ID (which in my case was a user's full name) </li></ol><p> that contains non-unicode caracters like Norwegian "æ", "ø", or "å". </p> <ol start="2"><li>Make sure that you can see something like the example below: </li></ol><p> (pyenv) rm@mycomputer:$ paster --plugin=ckan user Users: name=Rustæm </p> <ol start="3"><li>Then try deleting the user with following command: </li></ol><p> (pyenv) rm@mycomputer:$ paster --plugin=ckan user delete "Rustæm" </p> <p> You should now get a python encoding error. I know that this is quite rare case, but in our case it caused some trouble. Could you guys have a look at this bug? </p> <p> CKAN ver. 1.3.3. </p> http://localhost/ticket/1328 http://localhost/ticket/1328 Report amercader <p> Just a tip for developers regarding translations: </p> <ul><li>When writing templates, mark &lt;p&gt;, &lt;div&gt;, etc elements with the </li></ul><p> i18:msg attribute, otherwise the messages don't get extracted properly. </p> <p> Good: <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/home/about.html#cl-40</a> </p> <pre class="wiki">&lt;p i18n:msg=""&gt;More about CKAN, and the background to its creation can be found in &lt;a href="http://m.okfn.org/files/talks/opentech_20080705/"&gt;these slides&lt;/a&gt;.&lt;/p&gt; </pre><p> gives this: <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1238</a> </p> <p> Bad: <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/templates/package/new_package_form.html#cl-211</a> </p> <pre class="wiki">&lt;p class="hints"&gt; &lt;strong&gt;Important:&lt;/strong&gt; By submitting content, you agree to release your contributions under the &lt;a href="http://opendatacommons.org/licenses/odbl/1.0/"&gt;Open Database License&lt;/a&gt;. Please &lt;strong&gt;refrain&lt;/strong&gt; from editing this page if you are &lt;strong&gt;not&lt;/strong&gt; happy to do this. &lt;/p&gt; </pre><p> gives this: <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/1996b0439e81/ckan/i18n/ca/LC_MESSAGES/ckan.po#cl-1384</a> </p> <p> The message is split in different parts that not necessarily match the grammar or syntax of the translated language. </p> http://localhost/ticket/1329 http://localhost/ticket/1329 Report rgrp <p> test_authz.py appears to test in great detail some very specific additional authz (related to total site lock-down it seems -- introduced I think for hri project). </p> <p> I think there are simpler ways to get total site lockdown (use external auth!) and this test is slow and delicate (e.g. depends on specific words in templates). Suggest removing. If we don't remove we should at least refactor tests for access to certain pages to use a proper method of testing (e.g. agreed html comments in each page) rather than being depending on the presence of absence of specific wording. </p> http://localhost/ticket/1330 http://localhost/ticket/1330 Report dread <p> To reproduce: </p> <ol><li>Create a package with two tags the same: "bulk bulk" </li><li>Click 'save' </li><li>500 ERROR - 'Server Error' </li></ol> http://localhost/ticket/1331 http://localhost/ticket/1331 Report dread <p> We get this i18n error for the URL <a class="ext-link" href="http://no.ckan.net/authorizationgroup"><span class="icon">​</span>http://no.ckan.net/authorizationgroup</a> (which runs 1.4.2 with Norwegian translation). I'm not sure why it occurs. Fixing it is low importance though, because this site doesn't use Authorization Group feature </p> <p> Usually caused by bots. The user does not appear to be logged in. </p> <pre class="wiki">WebApp Error: &lt;type 'exceptions.IndexError'&gt;: list index out of range ... Module genshi.filters.i18n:177 in _generate &lt;&lt; msgbuf.append(*previous) previous = None for event in msgbuf.translate(gettext(msgbuf.format())): yield event if previous: &gt;&gt; for event in msgbuf.translate(gettext(msgbuf.format())): Module genshi.filters.i18n:1029 in translate &lt;&lt; ) parts = parse_msg(string) parts_counter = {} for order, string in parts: &gt;&gt; parts = parse_msg(string) Module genshi.filters.i18n:1143 in parse_msg &lt;&lt; if string: parts.append((stack[-1], string)) return parts &gt;&gt; parts.append((stack[-1], string)) IndexError: list index out of range </pre> http://localhost/ticket/1332 http://localhost/ticket/1332 Report amercader <p> Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core. </p> http://localhost/ticket/1333 http://localhost/ticket/1333 Report amercader <p> On same cases this exception is thrown when logging in: </p> <pre class="wiki">File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in user_dict = get_action('user_show')(context,data_dict) File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show revision_dict = revision_show(context,{'id':revision.id}) File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show ref_package_by=ref_package_by) File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict for pkg in revision.packages] AttributeError: 'NoneType' object has no attribute 'name' </pre> http://localhost/ticket/1334 http://localhost/ticket/1334 Report dread <p> Provide an API to the logic layer at /api/action. </p> http://localhost/ticket/1335 http://localhost/ticket/1335 Report dread <p> cset:4b59ab34137d ckan/logic/action/get.py: </p> <pre class="wiki">- isopen = model.Package.get_license_register()[license_id].isopen() - result_dict['isopen'] = isopen + try: + isopen = model.Package.get_license_register()[license_id].isopen() + result_dict['isopen'] = isopen + except KeyError: + # TODO: create a log message this error? + result_dict['isopen'] = False </pre><p> This change hides problems with the license server and returns potentially incorrect values for openness. </p> <p> This has been noted as 'temporary fix' but seems to be forgotten about, since it has been merged to default and gone into release 1.4.3. </p> <p> I suggest the licenses are cached (I thought this was already the case when CKAN first requests them after start-up?). I suggest failure would return 503. </p> http://localhost/ticket/1336 http://localhost/ticket/1336 Report amercader <p> Instead of this: </p> <pre class="wiki">extras: { type: ""Primary source"" license_id: ""localauth-withrights"" publisher_frequency: ""2Pre\u00f1o \u20ac $ \u00e0dri\u00e0"" } </pre><p> we should return this: </p> <pre class="wiki">extras: { type: "Primary source", license_id: "localauth-withrights", publisher_frequency: "2Preño € $ àdrià", } </pre> http://localhost/ticket/1337 http://localhost/ticket/1337 Report amercader <p> Both with the package and group controllers, when overriding the default _db_to_form_schema method (which does nothing) from an extension, you get an exception because the context is not passed to the validators. </p> <pre class="wiki">File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/group.py', line 140 in edit old_data, errors = validate(old_data, schema) File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 215 in validate converted_data, errors = _validate(flattened, schema, context) File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 270 in _validate convert(converter, key, converted_data, errors, context) File '/home/adria/dev/envs/iati/src/ckan/ckan/lib/navl/dictization_functions.py', line 178 in convert converter(key, converted_data, errors, context) File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/validators.py', line 108 in group_name_validator model = context['model'] KeyError: 'model' </pre> http://localhost/ticket/1338 http://localhost/ticket/1338 Report rgrp <p> I ran into a bug with the size field on resources. </p> <ul><li>It would not accept an empty value from form (IMO this clearly equates to null/None) </li><li>This could be fixed via using ignore_empty instead of ignore_missing </li><li>However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value) </li><li>similar issues could arise around other fields (such as last_modified ...) <ul><li>cf cset:645031d07b60 </li></ul></li></ul><p> To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works. </p> http://localhost/ticket/1339 http://localhost/ticket/1339 Report dread <p> e.g. <a class="ext-link" href="http://ckan.net/group/edit/history"><span class="icon">​</span>http://ckan.net/group/edit/history</a> </p> <pre class="wiki">Module ckan.controllers.group:135 in edit &lt;&lt; try: old_data = get_action('group_show')(context, data_dict) c.grouptitle = old_data.get('title') c.groupname = old_data.get('name') &gt;&gt; old_data = get_action('group_show')(context, data_dict) Module ckan.logic.action.get:347 in group_show &lt;&lt; if group is None: raise NotFound check_access('group_show',context, data_dict) &gt;&gt; raise NotFound </pre> http://localhost/ticket/1340 http://localhost/ticket/1340 Report kindly <p> Spam users where added to thedatahub and we need to clean them. </p> http://localhost/ticket/1341 http://localhost/ticket/1341 Report amercader <p> If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting <a class="ext-link" href="http://localhost:5000/dataset"><span class="icon">​</span>http://localhost:5000/dataset</a>). </p> <p> Steps to reproduce: </p> <ul><li>Change the solr_url parameter in the ini file to something non-existant </li><li>Visit <a class="ext-link" href="http://localhost:5000"><span class="icon">​</span>http://localhost:5000</a> </li></ul> http://localhost/ticket/1342 http://localhost/ticket/1342 Report rgrp <ul><li>Disallow account creation via openid - <a class="closed ticket" href="http://localhost/ticket/1386" title="enhancement: Disallow account creation via openid (closed: fixed)">#1386</a> </li><li>Require email field - <a class="closed ticket" href="http://localhost/ticket/1319" title="enhancement: Require an email address on User registration (closed: fixed)">#1319</a> <ul><li>Require email confirmation to be activated (?) </li></ul></li><li>Improvements to user page (e.g. show activity and more info about user) - <a class="closed ticket" href="http://localhost/ticket/1396" title="enhancement: Improvements to user page UX - 1d (closed: fixed)">#1396</a> </li><li>Modal user login - <a class="new ticket" href="http://localhost/ticket/1311" title="enhancement: Modal user register and login form (new)">#1311</a> </li></ul> http://localhost/ticket/1343 http://localhost/ticket/1343 Report kindly <p> Json decoding error on search, due to date in resources. </p> http://localhost/ticket/1344 http://localhost/ticket/1344 Report nils.toedtmann <p> There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions: </p> <p> When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (<a class="ext-link" href="http://trac.okfn.org/ticket/882"><span class="icon">​</span>ticket</a>) we experienced extraordinary memory usage peaks (<a class="ext-link" href="http://trac.okfn.org/ticket/904"><span class="icon">​</span>ticket</a>). Here are the observed value with Apache default settings: </p> <ul><li>eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB </li><li>s055, mpm-prefork: base level ~1GB, peaks up to 4GB </li><li>s055, mpm-worker: base level ~1.5GB, peaks up to 6GB </li></ul><p> William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (<a class="ext-link" href="https://bitbucket.org/okfn/sysadmin/changeset/126290aead28"><span class="icon">​</span>changeset</a>). This (together with two other tweaks) changed the situation drastically: </p> <ul><li>s055, mpm-event: base level ~1.4GB, no peaks </li></ul><p> This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak. </p> <p> To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to <em>processes=3</em>), and then observing the relation between <em>maximum-requests=25...500</em> and memory consumption. </p> <p> On 14/09/11 17:49, David Read wrote: </p> <blockquote class="citation"> <p> Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin. </p> </blockquote> <p> +1 </p> http://localhost/ticket/1345 http://localhost/ticket/1345 Report dread <p> This stems from a feature to allow you to prefill fields in the authorizationgroup creation form. </p> <p> e.g. <a class="ext-link" href="http://ckan.net/authorizationgroup/new?name=science"><span class="icon">​</span>http://ckan.net/authorizationgroup/new?name=science</a> </p> <p> It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it. </p> http://localhost/ticket/1346 http://localhost/ticket/1346 Report dread <p> Exception caused by bots POSTing 'new group' form, but with bad (old-style) parameters. </p> <pre class="wiki">&lt;type 'exceptions.ValueError'&gt;: invalid literal for int() with base 10: 'cache' ... Module ckan.controllers.group:102 in new &lt;&lt; if context['save'] and not data: return self._save_new(context) data = data or {} &gt;&gt; return self._save_new(context) Module ckan.controllers.group:152 in _save_new &lt;&lt; try: data_dict = clean_dict(unflatten( tuplize_dict(parse_params(request.params)))) context['message'] = data_dict.get('log_message', '') group = create.group_create(data_dict, context) &gt;&gt; tuplize_dict(parse_params(request.params)))) Module ckan.logic:57 in tuplize_dict &lt;&lt; for num, key in enumerate(key_list): if num % 2 == 1: key_list[num] = int(key) tuplized_dict[tuple(key_list)] = value return tuplized_dict &gt;&gt; key_list[num] = int(key) ValueError: invalid literal for int() with base 10: 'cache' </pre> http://localhost/ticket/1347 http://localhost/ticket/1347 Report zephod <p> Tweaking UIs and polishing the work done in <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a>. </p> <p> ## Edit Page ## </p> <ul><li>[Done; cset:49e0b059b6f5] Hide upload if storage not enabled </li><li>[Done; cset:81ff13b42dad] Make 'Add resource' links on edit page clearer and more button-like. </li><li>[Done; cset:03904fc50cbf] Move edit form tabs to the left </li><li>[Done; cset:fde26b87d6ce] API placeholder text should not link to a CSV file </li><li>[Done] We used to autocomplete the format field - restore this functionality <ul><li>Works on live. My local db has an empty 'resource_revision' table; maybe 'paster create-test-data' should insert something here? </li></ul></li><li>[Done; cset:1aab499d73db] Summary should contain 'name' (or, if unavailable, use description. Long term: description is to be deprecated) </li><li>[Done] Remove/move instructions where possible (a lot could just be placeholder text on the elements) (other stuff could only show on hover) <ul><li>NB: reveal_instructions.js already disabled on edit page </li><li>[RP] have now done this on create -- think we can leave on edit </li></ul></li><li>[Done; cset:755b1a622199] We have broken 'error' reporting for resource issues. Either need to move that into error summary or somehow not destroy it when we rewrite table on page load ... </li><li>[Done; cset:7ad090cb7855] Add slideDown to Resource expanded view </li><li>[Done; cset:d3595143e506] Need ability to delete resources <ul><li>No thought yet given to uploaded resources. Long term re-think probable. </li></ul></li><li>[Done; cset:2430a68a59fc] Empty resource table to resemble view page ("no resources" text) </li></ul><p> ## General ## </p> <ul><li>[Done; cset:cc0ad47293e5] Highlight the package tab I'm on (view/edit/history) <ul><li>More generally, the presence of the #minornavigation bar is very inconsistent throughout the site. </li></ul></li><li>[Done; cset:d5babc3629cf] Remove 'license' filtered search from backend </li><li>[Done] Move script stuff to footer of page </li></ul> http://localhost/ticket/1348 http://localhost/ticket/1348 Report zephod <p> Various things in templates can be tidied up after work has finished on <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a>: </p> <ul><li>remove styling in page in group/index.html (put in css main) <ul><li>ditto for package/search.html </li><li>ditto for home/index.html </li></ul></li><li>Forms.css - to merge into style.css; remove explicit imports elsewhere <ul><li>@import at the top of style.css @import url(forms.css) </li></ul></li><li>Make a note: Success/failure controllers clash CSS with success/failure flash banners </li></ul> http://localhost/ticket/1349 http://localhost/ticket/1349 Report zephod <p> New js code running produced in the work following <a class="closed ticket" href="http://localhost/ticket/1294" title="enhancement: [super] Package creation and editing UX improvements (closed: fixed)">#1294</a> contains various string literals, and needs proper internationalisation. </p> <p> See files: </p> <ul><li>application.js </li><li>templates.js </li></ul><p> Furthermore, efforts can be made to tidy up strings as described in <a class="closed ticket" href="http://localhost/ticket/1329" title="defect: i18n poor in some templates - missing i18n tags (closed: fixed)">#1329</a>. </p> http://localhost/ticket/1350 http://localhost/ticket/1350 Report zephod <p> The template for editing a user's profile seems to output the same form twice in one page. Furthermore, the page is made needlessly complex by the preview functionality. The neat inline preview used in package will make this page a lot neater. </p> <ul><li>Fix the duplicate form </li><li>Strip the preview functionality from the front-end, replacing it with js editor </li><li>Strip the preview functionality from the back-end and test </li></ul> http://localhost/ticket/1351 http://localhost/ticket/1351 Report amercader <p> The current search implementation uses the output of the the as_dict method of the domain Package object to update the index </p> <p> <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48</a> </p> <p> It also uses package_to_api1 in the <a class="missing wiki">SynchronousSearch?</a> plugin: </p> <p> <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93</a> </p> <p> This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model). </p> <p> The search should use the logic function to get the package properties: </p> <pre class="wiki">get_action('package_show')(context,data_dict) </pre> http://localhost/ticket/1352 http://localhost/ticket/1352 Report nickstenning <p> I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org. </p> http://localhost/ticket/1353 http://localhost/ticket/1353 Report amercader <p> Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit). </p> http://localhost/ticket/1354 http://localhost/ticket/1354 Report amercader <p> The extras in the package object sent to the extensions after editing (<a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/01efd5649c10/ckan/logic/action/update.py#cl-226</a>) do not include the newly added. </p> http://localhost/ticket/1355 http://localhost/ticket/1355 Report amercader <p> If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine). </p> http://localhost/ticket/1356 http://localhost/ticket/1356 Report rgrp <p> We have an all new and much improved data preview system developed as part of QA in July. Need to integrate this. </p> <ul><li>Is pure javascript </li><li>Will have this in CKAN core but considered to be a plugin (so need datapreview in ckan.plugins list to work -- will enable this by default) </li><li>Will depend on the <a class="ext-link" href="http://wiki.ckan.org/DataExplorer"><span class="icon">​</span>http://wiki.ckan.org/DataExplorer</a> </li></ul> http://localhost/ticket/1357 http://localhost/ticket/1357 Report rgrp <p> At the moment documentation of config options is duplicated between source (deployment_ini_tmpl in ckan/config which is used to generate user ini file) and the docs. </p> <p> Suggest we write a script that automatedly generates reference documentation for the config from the source. </p> <p> May be obsoleted by <a class="assigned ticket" href="http://localhost/ticket/277" title="enhancement: Set some config options / settings in WUI (extension) (assigned)">#277</a> (some config in db) </p> http://localhost/ticket/1358 http://localhost/ticket/1358 Report rgrp <p> Major UX grip and incredibly simple to do. May also want to have basic autocomplete support within the list (trivial to do as well I believe). </p> <p> (Talk to RP about this re how we define lists of licenses). </p> http://localhost/ticket/1359 http://localhost/ticket/1359 Report rgrp <p> Deliver little user benefit and easier just to search and then see what is listed as open and with resources. </p> <h2 id="ImplementationDetails">Implementation Details</h2> <p> Areas to fix: </p> <ul><li>UI </li><li>lib/search/... </li><li>tests </li><li>...? </li></ul><h2 id="Cost">Cost</h2> <p> Est: 2h </p> http://localhost/ticket/1360 http://localhost/ticket/1360 Report rgrp <p> Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install). </p> <p> Propose: </p> <ul><li>ckan.simple_search config option </li><li>If set: <ul><li>query via simple query to database backend </li><li>do no specialized indexing </li></ul></li></ul><h2 id="Extras">Extras</h2> <p> Remove </p> <ul><li><a class="missing wiki">TagSearchQuery?</a> from lib/search (just do a search directly)? </li><li><a class="closed ticket" href="http://localhost/ticket/1360" title="enhancement: Remove support for filter by download and filter by openness in search (closed: fixed)">#1360</a>: filter by downloadable and filter by open search options </li></ul><h2 id="PossibleExtras">Possible Extras</h2> <p> (Probably future improvements) </p> <ul><li>Re-introduce full-text search indexing where supported in e.g. postgres and use for querying </li></ul> http://localhost/ticket/1361 http://localhost/ticket/1361 Report johnglover <p> When paster db clean is run, the search index should also be cleared. </p> http://localhost/ticket/1362 http://localhost/ticket/1362 Report kindly <pre class="wiki">table: task_status id entity_id entity_type task_type key value last_updated state unique(entity_id, task_type, key) example keys: last_task_id </pre> http://localhost/ticket/1363 http://localhost/ticket/1363 Report kindly <p> We should ignore these but log the packages that are causing this to investigate. </p> http://localhost/ticket/1364 http://localhost/ticket/1364 Report dread <p> Lots of things rely on the old (i.e. pre-SOLR) search param formats, including ckanclient / DGU. Need a translation layer to convert these params (in api v1&amp;2) to SOLR syntax. </p> <p> This functionality was broken by the changes in <a class="closed ticket" href="http://localhost/ticket/1277" title="enhancement: Use solr query parser for search instead of ckan query parser (closed: fixed)">#1277</a> which went into 1.5a </p> http://localhost/ticket/1365 http://localhost/ticket/1365 Report dread <p> SOLR search doesn't support searching for part of an extra field, but it does for other fields. </p> <p> i.e. title="One Two Three" matches q=one AND q=title:one and geographic_coverage="England Scotland" matches q=England BUT NOT q=geographic_coverage:England </p> <p> This problem emerged when we went to SOLR in <a class="closed ticket" href="http://localhost/ticket/1275" title="enhancement: Move solr search extension to ckan core (closed: fixed)">#1275</a> (CKAN 1.5a). Tests were skipped. </p> <p> This is could be a problem for DGU and maybe elsewhere. </p> http://localhost/ticket/1366 http://localhost/ticket/1366 Report dread <p> manifest.cfg was not included in the pypi package, causing <a class="missing wiki">InvalidRepositoryError?</a> when you do 'paster db init'. </p> <p> Appears to affect most CKAN PyPI releases up to and including 1.4.3. </p> http://localhost/ticket/1367 http://localhost/ticket/1367 Report zephod <p> From Ira: PDF of bugs and suggestions is attached. </p> <p> This ticket is included in release v1.5. Post-v1.5 UX improvements are collected in <a class="closed ticket" href="http://localhost/ticket/1404" title="enhancement: Search Page UI improvements (closed: wontfix)">#1404</a>. </p> <p> <a class="missing wiki"> DONE ?</a> </p> <ul><li>[dataset/edit]: Deletions are confirmed with a dialog box </li><li>[dataset/edit]: Long URLs break the edit resources tab. Show name (&amp; 'no name') with a pencil icon. </li><li>[dataset/edit]: Rearrange the delete resources mechanism; <ul><li>Use a little minus icon on the far left, followed by a pencil icon for editing. </li><li>The 'delete resource' button can become 'hide' </li><li>Deletions are confirmed with a dialog box </li></ul></li><li>[dataset/edit]: Rearrange the 'you have unsaved changes' warning; <ul><li>Delete the little green plus sign on the right; </li><li>Try a flash banner instead (more general purpose &amp; beautiful) </li></ul></li><li>[dataset/edit]: Try a pencil icon (+'hide' button) rather than the +/- buttons which look a bit like you're adding a dataset. </li><li>[dataset/view]: Subscribe button (&amp; subscribe to history) doesn't work as expected. Modal dialog carrying instructions? </li><li>[dataset/view]: Drop the 'add to a group&gt;&gt;' and 'add tag&gt;&gt;'. Needs inline tag-adder. </li><li>[markdown editor]: Tab colours confuse the user. Inactive button appears to be greyed out. </li><li>[dataset/edit]: The upload file UI mutates and breaks across lines; make it a little more rigid. Verify at different zoom levels. </li><li>[groups/add]: Hide unimportant fields </li><li>[navigation] Add 3D bevel (@see github) to current tab highlight. </li><li>[dataset/create and groups/create]: Slug generator can be tidied. Default: Not editable. @see <a class="ext-link" href="http://groupspaces.com/a/group/"><span class="icon">​</span>http://groupspaces.com/a/group/</a> </li><li>[groups/view]: Show description and owner. To match dataset section. </li><li>[groups/authz]: Tidy to match admin/authz (pretty buttons &amp; spacing). Add navigation bar. </li><li>[groups]: Entire section to use pretty buttons please. </li><li>[about]: Get the about page working on test.ckan.org. (Works locally and on live). </li></ul><p> <a class="missing wiki"> WONTFIX ?</a> </p> <ul><li><del>[dataset/edit]: Make 'tags' a top level form section; it is too important to go below the fold.</del> (wontfix) </li><li><del>[groups]: Clicking a column heading should sort by that column. Default should be 'size', not 'name'.</del> (ticket <a class="new ticket" href="http://localhost/ticket/1403" title="enhancement: Refactor groups index page (new)">#1403</a>) </li><li><del>([groups]: (Investigate) Is it easy to implement a little search-groups feature?)</del> (ticket <a class="new ticket" href="http://localhost/ticket/1403" title="enhancement: Refactor groups index page (new)">#1403</a>) </li><li><del>[*/authz]: Add tooltips to explain column headers. (Not required if mechanism is about to be refactored...)</del> (wontfix: we need to rethink this mechanism; authorization refactor is coming) </li><li><del>[general]: 'Authorization' refactor to 'Permissions' on nav-bars.</del> (wontfix, further discussion. authorization might be less techy...) </li><li><del>[search]: (Investigate) Can we quickly &amp; easily reintegrate the faceted search-by-license but present it in the sidebar like the other facets?</del> (wontfix: high cost and maintenance overhead; need compelling user story) </li></ul> http://localhost/ticket/1368 http://localhost/ticket/1368 Report zephod <p> ckan.js can be rearchitected to avoid instantiating showdown.js in the root namespace. That way, layout_base.html doesn't need to import the script (because we don't use the showdown-dependant parts of ckan.js). </p> http://localhost/ticket/1369 http://localhost/ticket/1369 Report rgrp <p> Add a way to share datasets and see mentions of datasets on twitter (and elsewhere) </p> <p> As a visitor I want to share a link to a dataset I have found. I also want to see how many others have shared this (or mentioned it). </p> <ul><li>Sharing platform: simplest option is probably twitter but could generalize to e.g. sharethis system which supports, facebook, twitter etc etc. <ul><li>Will need a config option for relevant API key e.g. ckan.sharethis.apikey </li></ul></li><li>Location: Put this at top of sidebar on dataset view (a better suggsetion?) </li><li>Should show how many mentions / shares there are. (very important!) </li></ul><h2 id="Discussion">Discussion</h2> <ul><li>(Tom and Ira and ...): do we really want this, versus e.g. proper follow extension? </li><li>What about bringing conversation back in to CKAN. E.g. show all times this dataset (i.e. its url) was mentioned on twitter. If we do this I think this should be definite +1. </li><li>Also, given our users, I think just doing twitter (identica) may be sufficient (how many people want to share links to datasets on facebook?) </li></ul> http://localhost/ticket/1370 http://localhost/ticket/1370 Report johnglover <p> The background tasks will use the logic API to access the task_status table so new logic layer functions are needed. </p> http://localhost/ticket/1371 http://localhost/ticket/1371 Report johnglover <p> Take the resource archiver (currently part of ckanext-qa) and make it into a standalone extension that works as a celery task. </p> http://localhost/ticket/1372 http://localhost/ticket/1372 Report shevski <p> Either: thedatahub.org does not display 'add a dataset' top navigation link or the 'create dataset' main central link to logged in user - instead showing 'sign up' link in the central box. It does however display the 'my account' and 'logout' link in the top right correctly. </p> <p> Navigating to another part of the site (e.g. search or about or my account) does bring the 'add a dataset' navigation link back and the functionality works. </p> <p> Works fine on test.ckan.net so only on thedatahub. </p> <p> Not obviously a caching problem, since I did try clearing cache. </p> <p> OR: it displays as if you're logged in even when you log out. The links doesn't work and prompt you to log in though. </p> http://localhost/ticket/1373 http://localhost/ticket/1373 Report dread <p> e.g. cz.ckan.net defaults to Czech (config option lang=cs_CZ) but it fails when you try to switch to English. </p> http://localhost/ticket/1374 http://localhost/ticket/1374 Report amercader <p> When creating a package, the package_create logic function sends the newly created package object to all plugins implementing the IPackageController interface </p> <p> <a class="ext-link" href="https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64"><span class="icon">​</span>https://bitbucket.org/okfn/ckan/src/93194933ea75/ckan/logic/action/create.py#cl-64</a> </p> <p> At this point, the changes have not yet been commited and the id of the package is not set (It has a default value of make_uuid, which is set when commited). For this reason, plugins have no way of knowing the package id. E.g. the spatial query plugin needs to know this id to link the package extent to the actual package. </p> <p> One possible solution would be to set the package id in the data dict received by package_create (if it's not defined): </p> <pre class="wiki">from ckan.model.types import make_uuid if not 'id' in data or not data['id']: data['id'] = make_uuid() </pre> http://localhost/ticket/1375 http://localhost/ticket/1375 Report kindly <p> Regression found by updating datacatalogs. </p> http://localhost/ticket/1376 http://localhost/ticket/1376 Report zephod <p> Integrating ckanext-admin into core has thrown up a number of problems: </p> <ul><li>Look &amp; feel does not match the rest of the site </li><li>Tests are not passing </li><li>On the trash page, clicking 'undelete' triggers a purge </li><li>Using the purge functionality is dangerous; deleting and purging the latest revision will corrupt a dataset (&amp; several corrupt datasets have been found on thedatahub.org) </li><li>Trash page can contain nested form tags in certain cases (breaking test harness &amp; form redirection) </li></ul> http://localhost/ticket/1377 http://localhost/ticket/1377 Report zephod <p> Implement a banner across the top of the screen giving newbies a link to the about page, and making them feel welcome. See stackoverflow for reference? </p> http://localhost/ticket/1378 http://localhost/ticket/1378 Report zephod <p> eg. <a class="ext-link" href="http://thedatahub.org/dataset/us-gov-tarp"><span class="icon">​</span>http://thedatahub.org/dataset/us-gov-tarp</a> </p> <p> [This Paper][my-paper] is very interesting. </p> <p> [my-paper]: <a class="ext-link" href="http://somelink.com/paper"><span class="icon">​</span>http://somelink.com/paper</a> </p> <p> Write a test to check this markdown feature works - if not, use external markdown lib rather than webhelpers.markdown. Probably need to add markdown lib to list of requirements. </p> http://localhost/ticket/1379 http://localhost/ticket/1379 Report zephod <p> While integrating the new theme and making a large number of UX tweaks, I've allowed the CSS file to bloat out of control with some very specific rules and subtle hacks, as well as a couple of clearfix &lt;div&gt; objects in the markup which isn't really good practice. </p> <p> In general we want a logical CSS structure which is as easy to modify as possible. You should not find yourself being overwritten by highly specific rules all the time! </p> <h2 id="Actions">Actions</h2> <ol><li>Clean up css </li><li>Add a config option ckan.template_head_end - arbitrary string inserted at end of &lt;head&gt; tag so you can add custom css etc (may also want to update theming docs to reflect possibility of using this) </li><li>(While we're at it): Add ckan.template_footer_end config option to deployment.ini_template. (This already exists, we just haven't told anybody) </li><li>The /users index page isn't linked anywhere. Clean it up and link it in the footer next to Groups etc. </li><li>package/new, package/edit, group/new and group/edit all include their forms differently. Use h.literal in all cases. </li><li>Fix the padding on minornavigation. </li></ol><h2 id="Background">Background</h2> <p> David Read sent me the following as background: </p> <ol class="upperalpha" start="22"><li>basic theming is described here: <a class="ext-link" href="http://docs.ckan.org/en/latest/theming.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/theming.html</a> </li></ol><p> Deep customisation of the templates is done by creating an extension, with this as a template: <a class="ext-link" href="https://bitbucket.org/okfn/ckanext-exampletheme/overview"><span class="icon">​</span>https://bitbucket.org/okfn/ckanext-exampletheme/overview</a> </p> <p> That's what Augusto and the Brazilians have done <a class="ext-link" href="http://alpha.dados.gov.br/dados/"><span class="icon">​</span>http://alpha.dados.gov.br/dados/</a> and their code is here: hg clone <a class="ext-link" href="http://dev.dados.gov.br/codigo/dev/tema-ckan"><span class="icon">​</span>http://dev.dados.gov.br/codigo/dev/tema-ckan</a> &lt;snip&gt; Another thing of interest was us becoming compatible with Wordpress themes last December. Richard Pope worked with Rufus on this. I believe the base theme is 'twentyten' and I'm guessing that one would swap that css file with another to change theme, but I don't know. I don't know if this feature has been dropped since then. <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html</a> </p> <p> Dave </p> http://localhost/ticket/1380 http://localhost/ticket/1380 Report pudo <p> It is not currently possible to assign or remove groups by adding or removing their group name from the "groups" list on the REST API. This should be allowed, as it is easier than first adding a package and then editing each group it must belong to. </p> http://localhost/ticket/1381 http://localhost/ticket/1381 Report thejimmyg <p> Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate. </p> http://localhost/ticket/1382 http://localhost/ticket/1382 Report kindly <p> Need to add way for extensions to trigger based on resource change and just resource url change. </p> http://localhost/ticket/1383 http://localhost/ticket/1383 Report shevski <p> Most articles still refer and link to ckan.net, wiki.ckan.net and to packages </p> http://localhost/ticket/1384 http://localhost/ticket/1384 Report dread <p> Currently there is a problem because the docs guide us to set the sqlalchemy url to use 'localhost' i.e. loopback, whereas paster commands don't specify '-h localhost' so use unix sockets (you need to do 'sudo -u postgres'). These should be the same. </p> <p> Also do we need to tell people to add a line to their postgres authentication config /etc/postgresql/8.4/main/pg_hba.conf to help things? Florian suggests: </p> <pre class="wiki">local   std         std                          md5 </pre><p> /etc/postgresql/8.4/main/pg_hba.conf </p> http://localhost/ticket/1385 http://localhost/ticket/1385 Report rgrp <p> Superticket: <a class="new ticket" href="http://localhost/ticket/1343" title="enhancement: [super] User related improvements (login, user pages etc) (new)">#1343</a> </p> <p> Creation of accounts from OpenID causes large number of problems: </p> <ul><li>No guarantee of other profile info </li><li>Poor username (just from openid) </li></ul><p> We therefore will: </p> <ol><li>Require creation of user account via register (no auto-creation of accounts via openid) <ul><li>sidebar of register page should point to login page rather than openid signin </li></ul></li></ol><p> </p> <ol start="2"><li>Permit association of an openid with an account ... <ul><li>How: paste in your openid url (requires change to edit form to have this) <ul><li>won't work for google with their weird openid urls </li></ul></li><li>[future] Nicer way is to have login via openid while logged in (which association then happening in background) </li></ul></li></ol><ol start="3"><li>Migration (for 1+2): <ul><li>(?) Generate decent usernames for all existing accounts </li><li>As this is nontrivial suggest instead we allow editing of usernames (by account owner and sysadmins). This is useful in its own right and is a reasonable 80/20 solution. <ul><li>This does present the problem of re-associating commits with the new username. Best solution to this would be to switch revisions to point to userid rather than username. See <a class="new ticket" href="http://localhost/ticket/1534" title="enhancement: Change revisions to record userid rather than username (new)">#1534</a> </li></ul></li></ul></li></ol><ol start="4"><li>[optional] Change display_name to always be username <ul><li>Change My Account to Username at top right of all pages (once we know usernames are short ...) <ul><li>could do this straight away by truncating long usernames (e.g. truncate at 20 chars ...) </li><li>Also should we lose the icon? </li></ul></li></ul></li></ol> http://localhost/ticket/1386 http://localhost/ticket/1386 Report johnglover <p> If a user enters a url with no protocol (eg: www.data.gov.uk) in the 'url' field of a dataset, a broken link is created on the view page. </p> <p> CKAN should probably either put in a <a class="ext-link" href="http://"><span class="icon">​</span>http://</a> if no protocol is given, or validation should fail. </p> http://localhost/ticket/1387 http://localhost/ticket/1387 Report dread <p> Needs to update on: </p> <ol><li>language - It's broken on thedatahub.org if you change to German and then click on the CKAN icon top-left to go to thedatahub.org again. </li></ol><p> AND </p> <ol start="2"><li>whether signed in or not - <a class="closed ticket" href="http://localhost/ticket/1373" title="defect: home page view does not react to logging in / out (closed: fixed)">#1373</a> </li></ol><p> BUT not latest revisions (which is what it was) </p> <p> Or get rid of etag caching on this page and others? </p> http://localhost/ticket/1388 http://localhost/ticket/1388 Report johnglover http://localhost/ticket/1389 http://localhost/ticket/1389 Report johnglover http://localhost/ticket/1390 http://localhost/ticket/1390 Report johnglover http://localhost/ticket/1391 http://localhost/ticket/1391 Report johnglover http://localhost/ticket/1392 http://localhost/ticket/1392 Report dread <p> Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break. </p> http://localhost/ticket/1393 http://localhost/ticket/1393 Report dread <p> (Editing a dataset) If the second resource contains any validation error then it says "Resources: Package resource(s) incomplete" and "Resource 1:". </p> http://localhost/ticket/1394 http://localhost/ticket/1394 Report seanh <p> At the <em>paster db init</em> command when installing ckan from source I get the error <em>"Import Error: cannot import name <a class="missing wiki">UnicodeMultiDict?</a>"</em> (happens with both ckan 1.4.2 and today's latest bleeding edge code, on Ubuntu 10.04.3). </p> <p> <a class="missing wiki">UnicodeMultiDict?</a> has been removed in a recent version of python-webob, and the <em>pip install ... lucid_missing.txt</em> causes a too-new version of python-webob to be installed into ckan's virtualenv (the new webob gets installed as a dependency of formalchemy). </p> <p> I manually did <em>pip uninstall webob</em> and then ran <em>paster db init</em> again and it worked. </p> http://localhost/ticket/1395 http://localhost/ticket/1395 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a> </p> <ul><li>Show member since </li><li>Display email (in obfuscated form) </li><li>List all owned datasets (in standard manner) <ul><li>Will need change to logic layer to show datasets on user dict (this is useful for the API as well ...) </li></ul></li><li>(Re)move list of changes <ul><li>Should have activity (once activity available) </li></ul></li></ul><p> WONTFIX: </p> <ul><li>?? Allow users to provide a home page, organization (requires migration) </li></ul> http://localhost/ticket/1396 http://localhost/ticket/1396 Report rgrp <p> We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...) </p> <p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/queue"><span class="icon">​</span>http://ckan.okfnpad.org/queue</a> (most relevant parts inlined here) </p> <h2 id="Preliminaries">Preliminaries</h2> <ul><li>Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - <a class="closed ticket" href="http://localhost/ticket/1363" title="enhancement: Add task_status table to store qa/archiever/webstore information that does ... (closed: fixed)">#1363</a> (and <a class="closed ticket" href="http://localhost/ticket/1371" title="enhancement: Make logic layer functions for working with task_status table (closed: fixed)">#1371</a> - related logic functions) </li></ul><h3 id="Configurationsetupfordaemons">Configuration setup for daemons</h3> <p> Pass config through to workers i.e site_url, user, api_key. Need to make site user account. <a class="closed ticket" href="http://localhost/ticket/1408" title="enhancement: Make site user, so tasks can post information back to ckan. (closed: fixed)">#1408</a> </p> <p> celeryd config: </p> <p> All providers of tasks will add an item to the following entry point: </p> <pre class="wiki">[ckan.tasks] name = ckanext.{name}.tasks:.... </pre><p> celeryconfig.py </p> <pre class="wiki">from pkg_resources import iter_entry_points for entry in iter_entry_points: celeryimports.appen(....) CELERY_IMPORTS = celeryimports </pre><h2 id="WorkItems">Work Items</h2> <ul><li>Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - <a class="closed ticket" href="http://localhost/ticket/1383" title="enhancement: Add IResourceChange IResourceUrlChange Interfaces. (closed: fixed)">#1383</a> </li><li>Generate archiving request on resource url change [0.25d][0.25d] - <a class="closed ticket" href="http://localhost/ticket/1399" title="enhancement: Generate archiving request on resource url change (closed: fixed)">#1399</a> </li><li>Make site user account. </li><li>Make entry point system for celery config </li><li>Archiver daemon <a class="closed ticket" href="http://localhost/ticket/891" title="task: Resource download worker daemon (closed: fixed)">#891</a> <ol><li>implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d] </li><li>Rewrite archiver to use external storage. (decide how!)[3d][~2d] </li></ol></li><li>Write to resource and task status table.[1d][0.75d] </li><li>[Required?] Make archived data available in WUI - <a class="closed ticket" href="http://localhost/ticket/892" title="enhancement: Make stored data available in WUI - 0.5d (closed: fixed)">#892</a> </li><li>Documentation - <a class="closed ticket" href="http://localhost/ticket/1400" title="enhancement: Document Archiver Extension (closed: fixed)">#1400</a> </li></ul> http://localhost/ticket/1397 http://localhost/ticket/1397 Report rgrp <ul><li>Possible sibling / requirement: <a class="closed ticket" href="http://localhost/ticket/1397" title="enhancement: [super] Resource archiving (closed: fixed)">#1397</a> (resource archiving) </li></ul><ul><li>Setup test environment. 1d </li><li>Make task to upload files to webstorer. 3d <ul><li>This includes writing back webstore_url and updated date. 0.5d </li><li>Type guessing of file format. 1d </li><li>Make cron to upload all un webstored resources. 0.5d </li></ul></li><li>Testing for errors by running cron 2d </li><li>Make nice way to deploy celery 1d. </li><li>Documentation. 1d </li><li>Deployment. 1d </li></ul> http://localhost/ticket/1398 http://localhost/ticket/1398 Report rgrp <p> Superticket: <a class="closed ticket" href="http://localhost/ticket/1397" title="enhancement: [super] Resource archiving (closed: fixed)">#1397</a> </p> <p> Generate archiving request on resource url change by implementing IResourceUrlChange and sending tasks to celery. [0.25d][0.25d] </p> <p> Will be part of of ckanext-archiver extension. </p> <p> Process: </p> <ol class="arabiczero"><li>A resource is added to CKAN </li><li>IResourceCreate event generated </li><li>IF: resource url points to ckan storage or falls within some other set of exclusion conditions then END else continue </li><li>Generate a Archiver.Update task with resource.id </li></ol> http://localhost/ticket/1399 http://localhost/ticket/1399 Report rgrp <ul><li>Create page on <a class="ext-link" href="http://wiki.ckan.org/"><span class="icon">​</span>http://wiki.ckan.org/</a> about setting up a queue (this will merge into core docs when this is stable) - having a queue will now become a generic requirement for a bunch of extensions </li><li>README with install instructions in ckanext-archiver </li><li>Create an entry in <a class="ext-link" href="http://wiki.ckan.org/Extensions"><span class="icon">​</span>http://wiki.ckan.org/Extensions</a> </li><li>Write a blog post </li></ul> http://localhost/ticket/1400 http://localhost/ticket/1400 Report flofokus <p> if you want your ckan to reside not at <a class="ext-link" href="http://yourhost.org/"><span class="icon">​</span>http://yourhost.org/</a> but rather e.g. at <a class="ext-link" href="http://yourhost.org/ckan/"><span class="icon">​</span>http://yourhost.org/ckan/</a> you can use this hint <a class="ext-link" href="http://wiki.pylonshq.com/display/pylonsdocs/Configuration+Files"><span class="icon">​</span>http://wiki.pylonshq.com/display/pylonsdocs/Configuration+Files</a> and set ckan.site_url accordingly in your .ini. However, there are some instances where internal links are created without the required prefix. </p> http://localhost/ticket/1401 http://localhost/ticket/1401 Report rgrp <p> Plan to migrate from mercurial git </p> <ul><li>Convert from mercurial to git <ul><li>Map named branches to normal branches </li><li>Also, if possible map usernames for RP: rgrp &lt;<a class="ext-link" href="http://rufuspollock.org/"><span class="icon">​</span>http://rufuspollock.org/</a>&gt; (mercurial) -&gt; rgrp &lt;rufus.pollock@…&gt; </li></ul></li><li>Place at <a class="ext-link" href="http://github.com/okfn/ckan"><span class="icon">​</span>http://github.com/okfn/ckan</a> (can delete current repo there as an out of date mirror) </li></ul><p> Process: </p> <ul><li>Do trial run </li><li>Announce conversion date / time </li><li>Require everyone to have pushed <strong>all</strong> outstanding changes at that time </li><li>Do conversion </li><li>Test </li><li>Announce on list and ckan.org/ </li></ul> http://localhost/ticket/1402 http://localhost/ticket/1402 Report zephod <p> Groups are listed alphabetically with paging - not an ideal user experience. We would like to list groups in order of 'popularity': The number of datasets they contain. </p> <p> Following this chain of thought, then, it would be nice to rearrange the groups table by clicking on column headers and having it sort by that column. </p> <p> Furthermore, then, we'd like to implement a full-fledged groups search feature (if this is at all feasible). </p> <p> The forthcoming groups refactor will probably have some bearing on this task. </p> http://localhost/ticket/1403 http://localhost/ticket/1403 Report zephod <p> [Refactored] :: Follows on from <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a> UX changes. </p> <ul><li>Declutter the sidebar. No yellow box. </li><li>Facets to go on the left, rather than the right. More logical flow. </li><li>Did you know you can search by author? Probably not. Find a nice way of presenting extended search options. </li><li>Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results). <ul><li>Update that look-and-feel to include the new resource icons created in <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a> </li></ul></li></ul> http://localhost/ticket/1404 http://localhost/ticket/1404 Report jonathan.gray@… <p> To reproduce (from Jonathan Gray): </p> <ul><li>I go to add an item to the group </li><li>As I type, I get autocomplete options (cool!) </li><li>When I click on one, it adds this, and copies the same autocomplete option in the box </li><li>When I click submit, it seems to add the item twice </li><li>On the main group page I think it just shows up once, but when I click edit it shows two identical boxes, both ticked - and the count includes two copies of the same item </li></ul><p> Basically the problem is allowing the same package associated multiple times with a group. </p> <p> This is an issue on CKAN 1.5b and probably earlier. </p> http://localhost/ticket/1405 http://localhost/ticket/1405 Report zephod <p> RSS 'subscribe' buttons appeared in many places on the site but were not very helpful. They took (confused) users pointed to the raw feed code, and Google Reader could not understand the feed. Safari, however, could interpret it correctly. </p> <p> Their presentation needs to be clear and consistent; the RSS feed really needs testing in a variety of readers; and we need to decide exactly which items should get a feed. (Package updates? Groups?) </p> http://localhost/ticket/1406 http://localhost/ticket/1406 Report dread <p> Graphs don't show any more at <a class="ext-link" href="http://thedatahub.org/stats"><span class="icon">​</span>http://thedatahub.org/stats</a> </p> http://localhost/ticket/1407 http://localhost/ticket/1407 Report kindly <p> Give site user system administrative rights. </p> http://localhost/ticket/1408 http://localhost/ticket/1408 Report zephod <p> Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?) </p> <p> [Reopened: Additional] Use gravatars in the HTML generated by helpers.py to create lists of users. Eg. on user/list and dataset/history. </p> http://localhost/ticket/1409 http://localhost/ticket/1409 Report zephod <p> Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?) </p> http://localhost/ticket/1410 http://localhost/ticket/1410 Report rgrp <p> Format should be lowercase. Automatically lower case (for extra points have a bit of javascript to force lower case when entering). </p> <ul><li>Even more points: do a update on thedatahub repo to make all format lower case (or script this as an update?) </li></ul> http://localhost/ticket/1411 http://localhost/ticket/1411 Report seanh <p> Create a new dataset, don't add any resources to the dataset yet, view the dataset in ckan, if you have firebug enabled you'll see the error "resource is undefined" at line 699 of application.js. </p> http://localhost/ticket/1412 http://localhost/ticket/1412 Report shevski <p> I'd like us to display a one-off banner that prompts users who log into thedatahub.org to go and update &amp; their profile &amp; fill in their email address (if we could do it just for those who don't have an email address, then even better) That is, once they log in, they see a banner which says: Please update your profile [here] and add your email address, so you can receive notifications, be able to reset your password and get updates </p> <p> Then when we build notifications or if we decide to survey people, we can actually contact them. Email address should be required as standard </p> <p> DR: Also, mention their Full Name too - v. useful for selecting correct user in group curation </p> http://localhost/ticket/1413 http://localhost/ticket/1413 Report shevski <p> Set up tracking for user logins so that we have stats about how many active users of thedatahub exist want to be able to see who logged in the the last x months </p> http://localhost/ticket/1414 http://localhost/ticket/1414 Report nils.toedtmann <p> This is a scratch pad ticket with some comments on the current status of our ckan deb packages. I know that some of it is the deb packaging roadmap anyway, please forgive me if i mention them here again. </p> <p> Rufus and me re-deployed some community ckan instances onto s022 (see <a class="ext-link" href="http://trac.okfn.org/ticket/926"><span class="icon">​</span>http://trac.okfn.org/ticket/926</a>). We followed the documentation <a class="ext-link" href="http://docs.ckan.org/en/latest/install-from-package.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/install-from-package.html</a> </p> <ul><li>Deb package version number: the version of the deb package is "python-ckan 1309471251~149be76faabc+lucid-1", and it's hard to guess from there that it contains a ckan 1.4.2a </li><li>When is 1.4.3/1.5.x expected as deb package? </li><li>There was a bug in the DB upgrade script /usr/share/pyshared/ckan/migration/versions/029_version_groups.py (line 150) which looks like it was fixed 1.4.1==&gt;1.4.2 but was nevertheless present in this deb package. </li><li>The current script /usr/bin/ckan-std-install <ul><li>does not set the Apache <a class="missing wiki">ServerName?</a> according to the $INSTANCE variable </li><li>automatically configures a ckan extension named after $INSTANCE </li><li>depends on local postgres </li><li>could be replaced with "/usr/bin/ckan-deploy --name=ckan-std --domain=ckan-std.localhost (see next point) </li></ul></li><li>(i think this is exactly James' plan): have more generic deployment script /usr/bin/ckan-deploy as part of python-ckan which takes arguments like <ul><li>--domain=cc.ckan.net </li><li>--aliases=$list-of-domains </li><li>--name=cc (defaults to "domain") </li><li>--no-db (does not configure a DB) </li><li>--sql-alchemy=$DB_CONFIG_STRING (also runs "paster --plugin ckan db upgrade --config") </li><li>--extension $list-of-extesions </li><li>... </li></ul></li></ul> http://localhost/ticket/1415 http://localhost/ticket/1415 Report dread <p> This might well be a bot, but not POSTing the email field should not cause an exception - it should be 400 and then the sysadmin doesn't get these nags. </p> <pre class="wiki">http://www.publicdata.eu/user/register ... Module ckan.controllers.user:79 in register &lt;&lt; c.login = request.params.getone('login') c.fullname = request.params.getone('fullname') c.email = request.params.getone('email') if not model.User.check_name_available(c.login): h.flash_error(_("That username is not available.")) &gt;&gt; c.email = request.params.getone('email') Module webob.multidict:335 in getone &lt;&lt; values were found. """ return self._decode_value(self.multi.getone(self._encode_key(key))) def mixed(self): &gt;&gt; return self._decode_value(self.multi.getone(self._encode_key(key))) Module webob.multidict:99 in getone &lt;&lt; v = self.getall(key) if not v: raise KeyError('Key not found: %r' % key) if len(v) &gt; 1: raise KeyError('Multiple values match %r: %r' % (key, v)) &gt;&gt; raise KeyError('Key not found: %r' % key) KeyError: "Key not found: 'email'" </pre><pre class="wiki">HTTP_USER_AGENT 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 4.3 (build 01218); .NET CLR 1.1.4322)' webob._parsed_post_vars (MultiDict([('login', 'ahusernames9'), ('fullname', 'ahusernames9'), ('password1', '******'), ('password2', '******'), ('s', 'Sign up')]), &lt;FakeCGIBody at 0xb9d7c5ec viewing MultiDict([('lo...p')])&gt;) </pre> http://localhost/ticket/1416 http://localhost/ticket/1416 Report dread <p> In Firefox: </p> <ul><li>Edit | Preferences | Content | Languages - put 'de' at top language </li><li>Clear cookies </li><li>Load thedatahub.org </li><li>it should be in German, but it's in the default language - English. </li></ul> http://localhost/ticket/1417 http://localhost/ticket/1417 Report dread <p> e.g. cz.ckan.net has lang=cz but when you click on Home page link "English" it flashes message "Language has been set to: English" but the rest of the page is still Czech. </p> http://localhost/ticket/1418 http://localhost/ticket/1418 Report dread <p> I couldn't log into theDataHub with OpenID today. I tried both Google ID and MyOpenID. Both times the login on the remote auth server went fine, but when it returns you to theDataHub you get error "Login failed. Bad username or password." </p> http://localhost/ticket/1419 http://localhost/ticket/1419 Report johnglover <p> Currently can run as a celery task launched by paster command. </p> <p> Still have to add a CKAN plugin so that QA will run when a resource is added or URL changed, and then update the extension controller/templates to look for QA data from the task_status table instead of directly on the resource. </p> http://localhost/ticket/1420 http://localhost/ticket/1420 Report dread <p> home page doesn't have c.facets setup. </p> <p> Only affects 1.5 branch - ok 1.4.3.1. Was broken in cset:05f60f2d26e8. </p> http://localhost/ticket/1421 http://localhost/ticket/1421 Report dread <p> Rufus decided this was more trouble than it was worth. </p> <p> Removes ckan/lib/cache.py </p> http://localhost/ticket/1422 http://localhost/ticket/1422 Report markbrough <ul><li>Description vs Name - Edit Resources view is showing the name of the package rather than the description, and a lot (all?) of the packages before the upgrade don't have names, so might be good to swap this round again, e.g.: <a class="ext-link" href="http://thedatahub.org/dataset/edit/iati-registry"><span class="icon">​</span>http://thedatahub.org/dataset/edit/iati-registry</a> </li><li>Moving resources - Moving them up or down the list used to be quite useful if you had a lot of resources that you might want to leave on the resources page, but only one or two that were actually current and that you wanted to draw attention to. This doesn't exist any more on CKAN but I think it would be good to add it back in. </li></ul> http://localhost/ticket/1423 http://localhost/ticket/1423 Report dread <p> ckan-discuss discussion suggests changes to the 'openness' indicator ( <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-October/001786.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-October/001786.html</a> ) </p> <p> Dataset view page: </p> <ul><li>If there is an explicit but non-OKD compliant license, such as CC-BY-NC, then this should be stated explicitly, perhaps: “This dataset is Not Open. License: Creative Commons Attribution Noncommerical. This is not an open license as it does not meet the Open Knowledge Definition.” </li><li>If the license is marked as “Other::License Not Specified”, then this should be stated explicitly, perhaps: “This dataset is Not Open. It is published without an explicit license, the publisher reserves all rights to the dataset.” </li><li>3. If the license field was left empty by the contributor of the Data Hub record, then again this should be stated explicitly, perhaps: “This dataset is Not Open. The license of this dataset is unknown or unspecified. Start an enquiry on <a class="missing wiki">IsItOpenData?</a> » </li><li>There is a bug so that non-open licenses doesn't have an openness notice. </li><li>If downloadable resources are not available, this should not affect 'openness' - check this has been removed. </li></ul> http://localhost/ticket/1424 http://localhost/ticket/1424 Report dread <p> From Jonathan gray: </p> <p> At the moment to delete a package you have to go to 'state' and change from 'active' to 'deleted'. </p> <p> For me this was not very obvious, and someone recently wrote to me about this too. </p> <p> Its a small issue - but unless you know what 'state: active' refers to, its not obvious what this means, nor that this is where to go if you want to delete a package. </p> <p> From UI perspective this could be replaced by either a trash can or some text that says 'delete', perhaps with some prompt that explains what happens when you delete a package - explaining that non-admins won't be able to see it, and how you purge, etc. </p> http://localhost/ticket/1425 http://localhost/ticket/1425 Report dread <p> Validation error is incorrect arguments, so should be a "400 Bad Request" response. "409 Conflict" is for when there is a conflict creating two resources with the same identifier. Yet we get 409 for all validation errors. </p> http://localhost/ticket/1426 http://localhost/ticket/1426 Report dread <p> The tag name is limited in length in the db table to 100 characters. If you try and create a longer one you get an exception. This should cause a <a class="missing wiki">ValidationError?</a> and tell the user this sensibly. </p> http://localhost/ticket/1427 http://localhost/ticket/1427 Report dread <p> Need to ensure config options in the deployment.ini_tmpl match up with doc/configuration.rst. Remove any unused ones too. </p> http://localhost/ticket/1428 http://localhost/ticket/1428 Report rgrp <p> DOI = digital object identifier = <a class="ext-link" href="http://www.doi.org/"><span class="icon">​</span>http://www.doi.org/</a> </p> <p> As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way. </p> <h2 id="Details">Details</h2> <ul><li>Probably implement as extension rather than core </li></ul> http://localhost/ticket/1429 http://localhost/ticket/1429 Report amercader <p> On some occasions (apparently random), the documents indexed in a specific SOLR core get mixed with different site_ids. </p> <p> E.g: We look for all documents in the testing.iatiregistry.org core, faceted by site_id. We would expect all documents to have site_id = iati_testing, but some of them have site_id = iatiregistry.org </p> <p> <a class="ext-link" href="http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&amp;version=2.2&amp;q=*:*&amp;fq=+state:active&amp;facet=true&amp;facet.field=site_id"><span class="icon">​</span>http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&amp;version=2.2&amp;q=*:*&amp;fq=+state:active&amp;facet=true&amp;facet.field=site_id</a> </p> <pre class="wiki">&lt;lst name="facet_fields"&gt; &lt;lst name="site_id"&gt; &lt;int name="iati_testing"&gt;265&lt;/int&gt; &lt;int name="iatiregistry.org"&gt;255&lt;/int&gt; &lt;/lst&gt; &lt;/lst&gt; </pre><p> If we compare one of the records which disappeared from the "iati_testing" site_id in both the production and testing SOLR cores of the server, the records are exactly the same, including the indexed_ts property: </p> <p> <a class="ext-link" href="http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&amp;version=2.2&amp;q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&amp;facet=true&amp;facet.field=site_id"><span class="icon">​</span>http://okfn-solr.fry-it.com:8080/solr/iatiregistry.org/select?indent=on&amp;version=2.2&amp;q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&amp;facet=true&amp;facet.field=site_id</a> </p> <p> <a class="ext-link" href="http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&amp;version=2.2&amp;q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&amp;facet=true&amp;facet.field=site_id"><span class="icon">​</span>http://okfn-solr.fry-it.com:8080/solr/testing.iatiregistry.org/select?indent=on&amp;version=2.2&amp;q=id:97d1ab0d-b203-4757-8f4e-a0c84d2f759f&amp;facet=true&amp;facet.field=site_id</a> </p> <p> Note that the response from the URLs shown may vary, as the testing site could have been reindexed. </p> http://localhost/ticket/1430 http://localhost/ticket/1430 Report dread <p> During registering a user, the user inputs foreign chars into the captcha field. </p> <pre class="wiki">URL: http://thedatahub.org/user/register ... Module ckan.lib.captcha:22 in check_recaptcha &lt;&lt; remoteip=client_ip_address, challenge=recaptcha_challenge_field, response=recaptcha_response_field)) f = urllib2.urlopen(recaptcha_server_name, params) data = f.read() &gt;&gt; response=recaptcha_response_field)) Module urllib:1267 in urlencode &lt;&lt; for k, v in query: k = quote_plus(str(k)) v = quote_plus(str(v)) l.append(k + '=' + v) else: &gt;&gt; v = quote_plus(str(v)) UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 0: ordinal not in range(128) </pre> http://localhost/ticket/1431 http://localhost/ticket/1431 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1190" title="enhancement: [super] CREP 0004 Data API and Data Processing System (closed: fixed)">#1190</a> </p> <p> A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules) </p> http://localhost/ticket/1432 http://localhost/ticket/1432 Report rgrp <p> Why: current stable version of sqlalchemy. geoalchemy stuff required 0.7 and likely that some other things will require it soon. </p> <p> Probably requires work on vdm <a class="ext-link" href="https://bitbucket.org/okfn/vdm"><span class="icon">​</span>https://bitbucket.org/okfn/vdm</a> </p> <p> NB: should have discussion before making 0.7 the default required version in CKAN core. </p> http://localhost/ticket/1433 http://localhost/ticket/1433 Report dread <p> Decide which extensions are important. </p> http://localhost/ticket/1434 http://localhost/ticket/1434 Report rgrp http://localhost/ticket/1435 http://localhost/ticket/1435 Report johnglover http://localhost/ticket/1436 http://localhost/ticket/1436 Report dread <p> Action API needs JSONP support - be able to return responses encapsulated in a function of a supplied name. This is important for remote sites running javascript to interact with a CKAN site. </p> <p> Specifying the callback parameter is the way we've achieved JSONP with the RESTful and Search APIs. It should work like this: </p> <pre class="wiki">curl http://test.ckan.net/api/action/package_show?callback=jsoncallback -d '{"id": "fd788e57-dce4-481c-832d-497235bf9f78"}' </pre><p> Or maybe the callback should be specified in the payload in the context or data_dict? </p> <p> (My understanding is that CORS is similar - when more browsers support it, can we drop JSONP?) </p> http://localhost/ticket/1437 http://localhost/ticket/1437 Report dread <p> Many actions in the Action API require parameters. What params are needed should be listed and checked. Because currently, if you get them wrong you simply get a useless 500 error. </p> <p> Currently they are listed in the docs, extracted from the code manually. </p> <p> So you could GET /action/api/package_list to receive not only the help text, but a list of arguments. </p> <p> And if you send an extra or missing argument then an intelligent error message can be returned. </p> <h1 id="implementation">implementation</h1> <p> How about some sort of decorator on the action function: </p> <pre class="wiki">@logic_params(id, offset, limit) def get_package_list(context, data_dict): ... </pre><p> This would do the param checking, and is there a way to extract these params from the function? Or do a registration of the logic function? </p> <p> I'd certainly like to keep the list of the list of params for the function with the function, for ease of reading the code. </p> <p> Another good thing would be to pass in the params named as themselves, rather than having them contained in the data_dict. </p> http://localhost/ticket/1438 http://localhost/ticket/1438 Report dread <p> A good service API needs to be discoverable, so you are not always having to refer to the documentation html. </p> <p> Maybe /api/action should return a list of actions available? (Currently this returns a 404.) </p> <ul><li>It would be nice to sort these into get/create/update/delete. </li><li><a class="new ticket" href="http://localhost/ticket/1438" title="enhancement: Action API - parameter discovery/checking (new)">#1438</a> Parameters for each of the actions must be discoverable too </li></ul><p> /api/action/{action_name} should also return the help text / parameters allowable. (Currently this returns 400 error) </p> http://localhost/ticket/1439 http://localhost/ticket/1439 Report dread <p> <tt>paster db load ckan.pgdump</tt> will load in the database but not update the search index, no we're using SOLR. </p> <h1 id="implementation">implementation</h1> <p> We can't tack on 'rebuild' to this command, because it needs a 'db upgrade' first. I'm tempted to make 'db load' do the upgrade and rebuild all in one command. </p> http://localhost/ticket/1440 http://localhost/ticket/1440 Report dread <p> Using local image rather than the same image at assets.okfn.org reduces dependency on another server that may be down, slow or remote. James agrees. </p> http://localhost/ticket/1441 http://localhost/ticket/1441 Report dread <p> If you try to GET a dataset, purge it then POST it again, it fails checking authz for the group: </p> <pre class="wiki">Module ckan.controllers.api:278 in create &lt;&lt; (register, subregister)) try: response_data = action(context, data_dict) location = None if "id" in data_dict: &gt;&gt; response_data = action(context, data_dict) Module ckan.logic.action.create:240 in package_create_rest &lt;&lt; api = context.get('api_version') or '1' check_access('package_create_rest', context, data_dict) dictized_package = package_api_to_dict(data_dict, context) &gt;&gt; check_access('package_create_rest', context, data_dict) Module ckan.logic:129 in check_access &lt;&lt; # log.debug('Valid API key needed to make changes') # raise NotAuthorized logic_authorization = new_authz.is_authorized(action, context, data_dict) if not logic_authorization['success']: msg = logic_authorization.get('msg','') &gt;&gt; logic_authorization = new_authz.is_authorized(action, context, data_dict) Module ckan.new_authz:16 in is_authorized &lt;&lt; auth_function = _get_auth_function(action) if auth_function: return auth_function(context, data_dict) else: raise ValueError(_('Authorization function not found: %s' % action)) &gt;&gt; return auth_function(context, data_dict) Module ckan.logic.auth.create:115 in package_create_rest &lt;&lt; return {'success': False, 'msg': _('Valid API key needed to create a package')} return package_create(context, data_dict) def group_create_rest(context, data_dict): &gt;&gt; return package_create(context, data_dict) Module ckan.logic.auth.create:15 in package_create &lt;&lt; else: check2 = check_group_auth(context,data_dict) if not check2: return {'success': False, 'msg': _('User %s not authorized to edit these groups') % str(user)} &gt;&gt; check2 = check_group_auth(context,data_dict) Module ckan.logic.auth.create:90 in check_group_auth &lt;&lt; groups = set() for group_dict in group_dicts: id = group_dict.get('id') if not id: continue &gt;&gt; id = group_dict.get('id') AttributeError: 'unicode' object has no attribute 'get' </pre><p> (using RESTful Model API) </p> <p> The reason is the POST is expecting the group to be expressed as a dict, not the list of names that GET returns. </p> http://localhost/ticket/1442 http://localhost/ticket/1442 Report dread <p> If language detected doesn't match any of the offered CKAN languages, you get the default language, but see an error flash up for every page, saying: </p> <pre class="wiki">Could not change language to 'None': IOError: [Errno 2] No translation file found for domain: 'ckan' </pre><p> Affects 1.5b only. </p> http://localhost/ticket/1443 http://localhost/ticket/1443 Report dread <p> Using curl you always get English, even if ckan.locale_default=de. Find out why. (1.5b) </p> <p> Maybe we should disable locale detection for this release if lots of people's browsers don't have it set correctly (e.g. chinese with us internet explorer) </p> http://localhost/ticket/1444 http://localhost/ticket/1444 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1032" title="enhancement: [super] Resources in WUI (closed: fixed)">#1032</a> </p> <ul><li>Locate at: /dataset/{dataset}/resource/{id} </li></ul><p> See: <a class="ext-link" href="http://wiki.ckan.org/Dataset_View_Page"><span class="icon">​</span>http://wiki.ckan.org/Dataset_View_Page</a> </p> <p> Implemented in branch feature-1445-resource-view. </p> <p> Still to do: </p> <ul><li>Add inline data explorer to page </li></ul> http://localhost/ticket/1445 http://localhost/ticket/1445 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1602" title="enhancement: [super] Data Previewer / Viewer v2 (closed: fixed)">#1602</a> (Data Previewer v2) </p> <p> We already have first pass of Data Explorer that was released as part of <a class="closed ticket" href="http://localhost/ticket/1357" title="enhancement: Integrate new version of data preview (closed: fixed)">#1357</a>. </p> <ul><li>Merging <a class="ext-link" href="https://github.com/okfn/dataexplorer"><span class="icon">​</span>https://github.com/okfn/dataexplorer</a> with <a class="ext-link" href="http://github.com/okfn/recline"><span class="icon">​</span>http://github.com/okfn/recline</a> and developing it <ul><li><a class="ext-link" href="https://github.com/okfn/recline/issues"><span class="icon">​</span>https://github.com/okfn/recline/issues</a> </li></ul></li></ul><p> Tickets include (* indicates improvement over current explorer) </p> <ul><li>Core Backbone Models representing Dataset and Tabular data: <a class="ext-link" href="https://github.com/okfn/recline/issues/10"><span class="icon">​</span>https://github.com/okfn/recline/issues/10</a> </li><li>New theme: <a class="ext-link" href="https://github.com/okfn/recline/issues/22"><span class="icon">​</span>https://github.com/okfn/recline/issues/22</a> </li><li>Read-only mode: <a class="ext-link" href="https://github.com/okfn/recline/issues/17"><span class="icon">​</span>https://github.com/okfn/recline/issues/17</a> </li><li>Introduce hash navigation / state support (*): <a class="ext-link" href="https://github.com/okfn/recline/issues/19"><span class="icon">​</span>https://github.com/okfn/recline/issues/19</a> </li><li>Re-enable editing in <a class="missing wiki">DataTable?</a> (*): <a class="ext-link" href="https://github.com/okfn/recline/issues/13"><span class="icon">​</span>https://github.com/okfn/recline/issues/13</a> </li><li>[super] <a class="missing wiki">DataTable?</a> view (in Backbone) (*): <a class="ext-link" href="https://github.com/okfn/recline/issues/14"><span class="icon">​</span>https://github.com/okfn/recline/issues/14</a> </li><li><a class="missing wiki">DataExplorer?</a> parent view (*): <a class="ext-link" href="https://github.com/okfn/recline/issues/12"><span class="icon">​</span>https://github.com/okfn/recline/issues/12</a> </li><li>Simple graph widget using flot: <a class="ext-link" href="https://github.com/okfn/recline/issues/11"><span class="icon">​</span>https://github.com/okfn/recline/issues/11</a> </li></ul><p> Est: 10-15d (should be broken down -- partly is in recline issues) </p> http://localhost/ticket/1446 http://localhost/ticket/1446 Report dread <p> Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it. </p> <p> One problem in the past has been file handles running out when creating lots of tiny files in the data directory. </p> <p> Another problem has been several enourmous backups being created every day - pdeu on eu25. </p> http://localhost/ticket/1447 http://localhost/ticket/1447 Report kindly <p> Celery is awkward to deply, need to find a way to do it more simply. i.e using celery-pylons and supervisor. A modified version of celery-pylons may be the best solution. 1d </p> http://localhost/ticket/1448 http://localhost/ticket/1448 Report johnglover <p> Should list resources in the form of search results rather than as a table. </p> http://localhost/ticket/1449 http://localhost/ticket/1449 Report johnglover <p> See <a class="ext-link" href="http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal"><span class="icon">​</span>http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal</a> </p> <ul><li>Create a collapsible section for the README </li><li>Reduce the number of things on the page </li><li>Tidy up groups and tags in the sidebar </li><li>Resource view becomes a list of search results </li></ul><p> (additional) </p> <ul><li>Resource navbar item links to first resource </li><li>Resource navbar has a dropdown menu of all resources in the dataset </li><li>Edit and Authz navbar items should be in a separate logical group to the right </li></ul><p> Very rough summary: create more info section and move stuff out (relayout page e.g. README, tags etc) </p> http://localhost/ticket/1450 http://localhost/ticket/1450 Report johnglover <ul><li>css class: resource-url-analytics </li><li>Assign to the link tag (a) everywhere we want to count (dataset, resource view) - 0.25d </li><li>Display counts in same place ... </li></ul><p> And deploy on <a class="ext-link" href="http://thedatahub.org/"><span class="icon">​</span>http://thedatahub.org/</a> - 0.25d </p> <p> Possible: Also move analytics extension into core (decided not to). </p> http://localhost/ticket/1451 http://localhost/ticket/1451 Report dread <p> There are issues with setting the site's language according to the browser detection: </p> <ul><li>Many browsers have the language setting incorrectly set. e.g. people who download US versions of Mozilla, pirated US copies in the far east. </li></ul><ul><li>Although thedatahub.org has a worldwide audience, many CKAN instances are designed for a particular country's citizens. So Open Berlin should probably be in German as default for all. </li></ul><p> I suggest the site should have a default language saved in the config. The browser language *should* be detected, and that prompts a flash message offering to change to that language. And if you change language that is saved in the cookie (as we currently do). </p> <p> Then do some testing to see if this suits people. </p> http://localhost/ticket/1452 http://localhost/ticket/1452 Report icmurray <p> Allowing more flexible tag names: </p> <ul><li>allowing spaces </li><li>allow capital letters (search is case in-sensitive) </li><li>allow all punctuation except for commas and double-quotes '"' </li><li>allow unicode </li><li>commas delimit tag names in the package create/edit form </li></ul><p> Effects: </p> <ul><li>package creation/edit form. </li><li>/tag/{tagname} uri </li><li>search action </li><li>api controller (search/package-create/edit) </li><li>web controller (search/package-create/edit) </li><li>search api documentation </li><li>autocomplete for tag names </li></ul> http://localhost/ticket/1453 http://localhost/ticket/1453 Report dread <p> We made SOLR the main and only supported way to do CKAN search three months ago (<a class="closed ticket" href="http://localhost/ticket/1275" title="enhancement: Move solr search extension to ckan core (closed: fixed)">#1275</a>), but I notice that we still have remnants of the postgres full-text search hanging around: </p> <ul><li>model code (ckan/model/search_index.py) </li><li>table in the database (package_search) </li></ul><p> The presence of these still create sqlalchemy warnings and add to our db, (&amp; db backups etc), so it would be good to get rid of them if we are not using them. John says this can probably be removed </p> http://localhost/ticket/1454 http://localhost/ticket/1454 Report dread <p> When you do a search like this: </p> <pre class="wiki">http://thedatahub.org/api/search/package?q=tauberer+census&amp;all_fields=1 </pre><p> the "extra" fields (e.g. "triples", "shortname") get missed off the results. The docs say it should be a "full record" and I don't see any reason why this is missed off. </p> <p> This is a problem because search all_fields is the only way for clients and front-ends to get packages in bulk. They end up (like lodcloud) doing thousands of requests to get packages individually. </p> <p> The full record is: </p> <pre class="wiki">http://thedatahub.org/api/rest/dataset/2000-us-census-rdf {"count": 1, "results": [{"res_description": ["Download", "XML Sitemap", "SPARQL enpdoint", "Example (RDF/XML)"], "name": "2000-us-census-rdf", "license": "Non-OKD Compliant::Creative Commons Non-Commercial (Any)", "author": "Joshua Tauberer", "author_email": "http://razor.occams.info/", "ckan_url": "http://thedatahub.org/dataset/2000-us-census-rdf", "notes": "2000 U.S. Census converted into over a billion RDF triples.\n\nPopulation statistics at various geographic levels, from the U.S. as a whole, down through states, counties, sub-counties (roughly, cities and incorporated towns)\n\nNotes: also found in the of SPARQL Endpoints.\n\nFrom home page:\n\n&gt; * For the detailed Census statistics, you'll have to download the raw Census data files from the Census Bureau, my Perl script and the patch file below and run it yourself because the files are too big for me to offer as a download!\n&gt; \n&gt; * The data and scripts can be reused under Creative Commons Attribution-NonCommercial-ShareAlike.\n", "entity_type": "package", "site_id": "www.ckan.net", "download_url": "http://www.rdfabout.com/demo/census/", "indexed_ts": "2011-11-01T12:52:36.034Z", "url": "http://www.rdfabout.com/demo/census/", "state": "active", "title": "2000 U.S. Census in RDF (rdfabout.com)", "groups": ["lod", "lodcloud"], "res_format": ["", "meta/sitemap", "api/sparql", "example/rdf+xml"], "license_id": "cc-nc", "revision_id": "fcbad0de-79ea-41bd-8e01-eb832a05b732", "res_url": ["http://www.rdfabout.com/demo/census/", "http://www.rdfabout.com/sitemap.xml", "http://www.rdfabout.com/sparql", "http://www.rdfabout.com/rdf/usgov/geo/us/ny"], "id": "551ec435-f198-4d52-9b56-ec0b0be6aec9", "tags": ["census", "data", "demographics", "deref-vocab", "format-dc", "format-geonames", "format-politico", "format-rdf", "geographic", "linkeddata", "lod", "lodcloud.nolinks", "no-license-metadata", "no-provenance-metadata", "no-vocab-mappings", "population", "published-by-third-party", "rdf", "statistics", "us"]}]} </pre> http://localhost/ticket/1455 http://localhost/ticket/1455 Report amercader <p> If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those. </p> http://localhost/ticket/1456 http://localhost/ticket/1456 Report jilly.mathews <blockquote> <p> "when logging into <a class="ext-link" href="http://register.data.overheid.nl/"><span class="icon">​</span>http://register.data.overheid.nl/</a> with OpenID, the /user/me page gives a 404. CKAN version 1.3.4." </p> </blockquote> <p> n the manual it says an API key kan be created via <a class="ext-link" href="http://test.ckan.net/user/me"><span class="icon">​</span>http://test.ckan.net/user/me</a> /. However, when I try the corresponding <a class="ext-link" href="http://register.data.overheid.nl/user/me"><span class="icon">​</span>http://register.data.overheid.nl/user/me</a>, I get a 404 error (not found). </p> http://localhost/ticket/1457 http://localhost/ticket/1457 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1151" title="enhancement: Preview for geographic data should be a map (closed: wontfix)">#1151</a> (viewing geo data) </p> <p> E.g. preview of <a class="ext-link" href="http://thedatahub.org/dataset/louisvillecrime"><span class="icon">​</span>http://thedatahub.org/dataset/louisvillecrime</a> should bring up a map </p> http://localhost/ticket/1458 http://localhost/ticket/1458 Report rgrp <p> Provide way to mark a dataset as featured. Featured database show up on the front page. </p> <p> TODO: detail this more. </p> http://localhost/ticket/1459 http://localhost/ticket/1459 Report rgrp <p> Current extensions documentation needs some work: <a class="ext-link" href="http://docs.ckan.org/en/latest/plugins.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/plugins.html</a> </p> <ul><li>Queue extension section may now be out of date (?) </li><li>Think about how it integrates with <a class="ext-link" href="https://github.com/okfn/ckanext-example"><span class="icon">​</span>https://github.com/okfn/ckanext-example</a> (especially tutorial and example extension) </li><li>Document all plugin points (auto-extract from CKAN source??) </li></ul> http://localhost/ticket/1460 http://localhost/ticket/1460 Report pudo <p> e.g. package_register_get. </p> http://localhost/ticket/1461 http://localhost/ticket/1461 Report thejimmyg <p> With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix. </p> <p> I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page. </p> <p> We need to decide how to do this upgrade without affecting existing CKAN solr indexes. </p> http://localhost/ticket/1462 http://localhost/ticket/1462 Report thejimmyg <p> The extension needs upgrading: </p> <blockquote> <p> No module named webstore.database </p> </blockquote> <p> The other option is to delay this functionality and merge with the publisher dashboards after the group refactor. </p> <p> John, David Raznick, David Read, Ira, what do you think? </p> http://localhost/ticket/1463 http://localhost/ticket/1463 Report thejimmyg <p> The current harvesting implementation can only have one instance per server. </p> <p> We could either: </p> <ul><li>Accept this </li><li>Change the RabbitMQ code to support multiple instances </li><li>Switch the entire harvesting to use Celery </li><li>Wait for James's redis based live feedback based system </li></ul> http://localhost/ticket/1464 http://localhost/ticket/1464 Report thejimmyg <p> At the moment most of the harvesting functionality is only available to sysadmins. This is fine for most cases but going forward we will need to grant each publisher access to their own sources, harvest objects and harvest object errors. This will probably be done based on the groups refactor. </p> <p> One more pressing concern is giving DGU publishers the ability to make calls to /harvest/object/id. We want them to be able to see documents that faild validation and also to see *all* errors in each document. </p> http://localhost/ticket/1465 http://localhost/ticket/1465 Report thejimmyg http://localhost/ticket/1466 http://localhost/ticket/1466 Report thejimmyg <p> Pawel knows about this so David Read, Pawel and I need to find time to discuss it. </p> http://localhost/ticket/1467 http://localhost/ticket/1467 Report thejimmyg <p> When I delete a package from the pylons shell I need to remove that package's entry from the Solr search index. That doesn't happen at the moment. </p> <p> It would be useful if the search-index rebuild command could take a list of package IDs to rebuild. </p> http://localhost/ticket/1468 http://localhost/ticket/1468 Report thejimmyg <p> Now that we have a geospatial search API, we should build a front-end that integrates with the existing search for displaying results. </p> http://localhost/ticket/1469 http://localhost/ticket/1469 Report amercader http://localhost/ticket/1470 http://localhost/ticket/1470 Report dread <p> Nils flagged up that Ubuntu launches another LTS in April which we should probably move to. This ticket is to get the ball rolling with what we do. We can always include in our package the exact package versions of deps that we want, but it is good to be in line with the release if possible. </p> <p> Here's how to see the packages included with it: <a class="ext-link" href="http://packages.ubuntu.com/search?keywords=python-pylons&amp;searchon=names&amp;suite=all&amp;section=all"><span class="icon">​</span>http://packages.ubuntu.com/search?keywords=python-pylons&amp;searchon=names&amp;suite=all&amp;section=all</a> </p> <p> Pylons 0.9.7 -&gt; 1.0-2 Requires a small amount of work SQLAlchemy 0.7.3-2 OK Babel 0.9.6-1 OK Genshi 0.6-2 Perfect </p> <p> Another alternative is to move to another web framework, especially minding that Pylons is not being further developed. </p> http://localhost/ticket/1471 http://localhost/ticket/1471 Report rgrp http://localhost/ticket/1472 http://localhost/ticket/1472 Report dread <p> Max length of package name and version field is 100 characters, but the validators seem to enforce this, although in the web interface the javascript limits typing in these fields, so it just affects the API really. </p> http://localhost/ticket/1473 http://localhost/ticket/1473 Report kindly <p> navl tests are being skipped unskip them! </p> http://localhost/ticket/1474 http://localhost/ticket/1474 Report kindly http://localhost/ticket/1475 http://localhost/ticket/1475 Report kindly <p> This is the implementation of the final draft in </p> <p> <a class="ext-link" href="http://wiki.ckan.org/Groups_Refactor"><span class="icon">​</span>http://wiki.ckan.org/Groups_Refactor</a> </p> <p> Steps to complete this will follow (somewhat) in the following order. </p> <p> Change package_group table to membership. (<a class="closed ticket" href="http://localhost/ticket/1478" title="enhancement: change package_group to members table (closed: fixed)">#1478</a>) 3d </p> <ul><li>model migration. (done) </li><li>simplify revisioning (done) </li><li>Remove any sqlalchemy releations from groups. (done) </li><li>change logic functions (done) </li></ul><p> </p> <p> Single type per group (<a class="closed ticket" href="http://localhost/ticket/1531" title="enhancement: Update group create/update so you can add capacities and group types. (closed: fixed)">#1531</a>) </p> <ul><li>migration add type to group table. (done) 1d </li></ul><p> Add capacity table (<a class="closed ticket" href="http://localhost/ticket/1522" title="enhancement: Add capacity to member table. (closed: fixed)">#1522</a>) </p> <ul><li>model migration. (done) </li><li>add to logic functions. </li></ul><p> Allow other objects to be added to groups. (<a class="closed ticket" href="http://localhost/ticket/1531" title="enhancement: Update group create/update so you can add capacities and group types. (closed: fixed)">#1531</a>) </p> <ul><li>change logic functions (done) </li></ul><p> Adapt thedatahub.org to use new model. </p> <ul><li>move auth groups over to groups/members </li><li>change user_object roles to use groups? keep user object roles for time being. </li><li>make create update package/group put the correct items in the group/members/capacity tables. </li><li>work out how ui works. </li></ul><p> --not doing the below as part of this super ticker now <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a>. </p> <p> Create model for dgu. </p> <ul><li>adapt auth functions to use auth. </li><li>adapt package_create/update group_create/update to fill in the correct capacties/members. </li></ul><p> Adapt Iati, and potentially other custom instances. </p> <p> Sub-tickets: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1279" title="enhancement: Refactor or deprecate Authorization Groups (closed: duplicate)">#1279</a> Deprecate Authorization Groups </li><li><a class="new ticket" href="http://localhost/ticket/1198" title="enhancement: Publisher hierarchy (new)">#1198</a> Publisher hierarchy </li></ul> http://localhost/ticket/1477 http://localhost/ticket/1477 Report kindly <p> This will comprise of the following steps. </p> <ul><li>model migration. </li><li>simplify revisioning </li><li>Remove any sqlalchemy relations from groups. </li><li>change logic functions </li></ul><p> estimate 2d </p> http://localhost/ticket/1478 http://localhost/ticket/1478 Report dread <ol><li>Register User with an email address with a unicode char (e.g. u'\u044e') </li><li>View the User in the UI (/user/) or with 'user_show' Action API </li></ol><p> Exception: </p> <pre class="wiki">Module ckan.controllers.user:98 in read &lt;&lt; try: user_dict = get_action('user_show')(context,data_dict) except NotFound: h.redirect_to(controller='user', action='login', id=None) &gt;&gt; user_dict = get_action('user_show')(context,data_dict) Module ckan.logic.action.get:488 in user_show &lt;&lt; check_access('user_show',context, data_dict) user_dict = user_dictize(user_obj,context) if not (Authorizer().is_sysadmin(unicode(user)) or user == user_obj.name): &gt;&gt; user_dict = user_dictize(user_obj,context) Module ckan.lib.dictization.model_dictize:189 in user_dictize &lt;&lt; result_dict['display_name'] = user.display_name result_dict['email_hash'] = user.email_hash result_dict['number_of_edits'] = user.number_of_edits() result_dict['number_administered_packages'] = user.number_administered_packages() &gt;&gt; result_dict['email_hash'] = user.email_hash Module ckan.model.user:59 in email_hash &lt;&lt; if self.email: e = self.email.strip().lower() return hashlib.md5(e).hexdigest() def get_reference_preferred_for_uri(self): &gt;&gt; return hashlib.md5(e).hexdigest() UnicodeEncodeError: 'ascii' codec can't encode character u'\u044e' in position 17: ordinal not in range(128) </pre> http://localhost/ticket/1479 http://localhost/ticket/1479 Report dread <p> munge_title_to_name, munge_tag v. useful for client software importing things into CKAN. I want it for ONS. </p> http://localhost/ticket/1480 http://localhost/ticket/1480 Report dread <p> The license is not displayed any more, unless there are resources e.g. <a class="ext-link" href="http://test.ckan.net/dataset/dtest#section-basic-information"><span class="icon">​</span>http://test.ckan.net/dataset/dtest#section-basic-information</a> is OGL, but that's not shown. </p> http://localhost/ticket/1481 http://localhost/ticket/1481 Report zephod <p> Three JS bugs have appeared: </p> <ul><li>The slug editor is not updating the text field again. [Fixed; commit:fadda17383] </li></ul><ul><li>The View Dataset page will stop executing javascript if there are no resources. [Fixed; commit:e54b089f6e] </li></ul><ul><li>The Add-Dataset and Add-Group URL editor shows the wrong URL for datasets and groups. Broken in this commit: </li></ul><p> <a class="ext-link" href="https://github.com/okfn/ckan/commit/5a01e67"><span class="icon">​</span>https://github.com/okfn/ckan/commit/5a01e67</a> by 'map.explicit=true'. JohnG: Can you look into this? [Fixed; commit:88e88380f6] </p> http://localhost/ticket/1482 http://localhost/ticket/1482 Report johnglover <p> The error reports do not have a "value". The original task in the task status table is not made and it tries to make a new task, but can't. </p> <p> Issue: why is the original task_status entry not made? </p> <p> kindly: there is a chance of course that the task is quicker than the plugin to get to the point where it needs to update the table. </p> <p> kindly: so we may have to find a way to update the task status before we run the task. </p> <p> kindly: I think you can supply a task_id to celery. i.e we make our own guid as the task id. So we can use that one to populate the task status. </p> http://localhost/ticket/1483 http://localhost/ticket/1483 Report dread <p> e.g. </p> <pre class="wiki">curl http://localhost:5000/api/action/package_create -d '{"name":"test5", "tags":[{"name": "tag6"}, {"name": "tag6"}]}' -H 'Authorization:tester' </pre><p> Also broken in Web UI (adding a package). </p> <p> Affects API and Web UI in 1.5 and I think all versions from 1.4.1 </p> http://localhost/ticket/1484 http://localhost/ticket/1484 Report kindly <p> We want to be able to change form depending on package type or group type. </p> <p> This is dependent on a type field being added to the Package and the Group. </p> http://localhost/ticket/1485 http://localhost/ticket/1485 Report seanh <ol><li>Call ckan.logic.action.update:package_update() </li><li>Pass in a context dict with "allow_partial_update":True </li><li>Pass in a partial data dict containing only the package's ID and those fields that you want to update (e.g. title). </li></ol><p> =&gt; All of the package's resource will have their statuses changed to "deleted". </p> <p> The "allow_partial_update" option should apply to the resource lists of packages, but it does not. </p> http://localhost/ticket/1486 http://localhost/ticket/1486 Report kindly <p> ordering on homepage by name instead of group count </p> http://localhost/ticket/1487 http://localhost/ticket/1487 Report dread <p> Add the list of extensions install (config's ckan.plugins) to status_show logic action (See <a class="closed ticket" href="http://localhost/ticket/1087" title="enhancement: version and contact info api call (closed: fixed)">#1087</a>) </p> http://localhost/ticket/1488 http://localhost/ticket/1488 Report dread <p> ckanext-example needs updating for CKAN 1.5: </p> <ul><li>theme changes </li><li>new forms </li></ul><p> About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look &amp; operation of CKAN. This moved to github and renamed 'ckanext-example'. </p> http://localhost/ticket/1489 http://localhost/ticket/1489 Report amercader <p> Right now, the two logic functions that return a list of packages (package_search <a class="missing changeset" title="No default repository defined">[1]</a> and group_packages_list <a class="missing changeset" title="No default repository defined">[2]</a>)use custom functions to generate the output dict. That's suboptimal because: </p> <ul><li>The template function that renders the package listing (package_list_from_dict) could end up receiving different dicts. </li><li>Not all package properties are available at the template level (e.g extras, tags...). These can be really useful to customize the listing from an extension. </li></ul><p> In general only the functions present in lib/dictization/model_dictize.py should be used to build the output of a logic function, in that case package_dictize. If necessary, they can be modified to include missing properties, like on this particular case the "isopen" property, needed by the template renderer. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685"><span class="icon">​</span>https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L685</a> <a class="missing changeset" title="No default repository defined">[2]</a> <a class="ext-link" href="https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442"><span class="icon">​</span>https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L442</a> </p> http://localhost/ticket/1490 http://localhost/ticket/1490 Report dread <p> Sean spotted that some strings need internationalising, such as flash messages. Generally all strings that appear on the web front-end should be internationalised. </p> <p> Particular places that need i18n: </p> <ul><li>ckan/controllers/admin </li><li>ckan/lib/base </li></ul><p> Note: there are some exceptions, such as in i18n.py, very obscure error messages and stuff that only appears on the API. </p> <p> NB: there is a cost in making a string internationalisable (all our volunteers have to translate it), so we should not be too zealous. We should also look at the i18n/ckan.pot to see if any of the existing strings can be reused. </p> http://localhost/ticket/1491 http://localhost/ticket/1491 Report dread <p> From Jan Kučera &lt;elquenor@…&gt; </p> <blockquote> <p> I have also tried to use extras fields to store information about relationships between datasets. I found out that if you first add relationship using CKAN API into dataset relationships attribute and than you add extras field for the same relationship with key in format "parent_of:dataset_name" or "child_of:dataset_name", storing this extras field will clear all relationships in relationships attribute stored earlier. </p> </blockquote> <p> </p> <blockquote> <p> I might be some kind of bug but I suppose you will know better. Is it possible to use relationships attribute and extras field for relationships at the same time? </p> </blockquote> http://localhost/ticket/1492 http://localhost/ticket/1492 Report dread <p> You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank. </p> <p> Should assert that one of these are set up. </p> http://localhost/ticket/1493 http://localhost/ticket/1493 Report seanh <p> This could be implemented as a separate API call, or the rendered text versions of the activities could be added into the JSON returned by the existing API call. </p> <p> This requires setting up templates for rendering activity streams items and detail items as nice, human-readable text. </p> <p> There are some open questions, e.g.: Do we want the entire activity stream rendered as a block of plain text? As HTML? Or do we want a list of JSON objects, where each object contains its textual and/or HTML representations as fields? </p> <p> Activity stream items and their related detail items are separate objects that each have their own textual representations. </p> <p> For a mockup of the kind of text messages we want, see: </p> <p> <a class="ext-link" href="http://datahub.pudo.org/pudo"><span class="icon">​</span>http://datahub.pudo.org/pudo</a> </p> <p> but note that this ticket is just for creating the text snippets themselves, not rendering then in an HTML page or RSS feed. Also the mockup only show activity items and not their detail items. </p> http://localhost/ticket/1494 http://localhost/ticket/1494 Report seanh <p> An API call that returns a user's public activity stream in JSON format. </p> http://localhost/ticket/1495 http://localhost/ticket/1495 Report seanh http://localhost/ticket/1496 http://localhost/ticket/1496 Report dread <p> We really shouldn't have lots of benign stuff printed on stderr when we run every command. It confuses users and hides important stuff. </p> http://localhost/ticket/1497 http://localhost/ticket/1497 Report amercader <p> At some point the SOLR schema used by CKAN will need to be changed and the changes won't be always backward compatible, so we need a mechanism to support instances that use older versions of the schema. This process will involve: </p> <ul><li>Better documentation on how to deploy SOLR, as single instance or multicore </li><li>A convention on how to tag schema versions and SOLR endpoints to ensure that CKAN instances point to the correct SOLR URL. </li><li>(If possible) Checks in CKAN core that deal with different schema versions. </li></ul> http://localhost/ticket/1498 http://localhost/ticket/1498 Report dread <p> It would be useful to be able to manipulate packages using the paster command-line interface. In particular listing, reading and deleting packages. Seeing which packages were deleted too. </p> http://localhost/ticket/1499 http://localhost/ticket/1499 Report dread <ul><li>API call documented but don't exist any more: dataset create_slug </li><li>API call not documented but exists: is_slug_valid </li><li>API call not documented, badly sited in API and using custom code in controller (not logic layer): /package/autocomplete </li></ul> http://localhost/ticket/1500 http://localhost/ticket/1500 Report amercader <p> Most of the different pagination links are broken and point to wrong locations. e.g: </p> <ul><li>Dataset listings in group page: </li></ul><p> <a class="ext-link" href="http://test.ckan.net/group/bibliographic"><span class="icon">​</span>http://test.ckan.net/group/bibliographic</a> </p> <ul><li>Tag letters in tag listing: </li></ul><p> <a class="ext-link" href="http://test.ckan.net/tag"><span class="icon">​</span>http://test.ckan.net/tag</a> </p> <ul><li>User listing: </li></ul><p> <a class="ext-link" href="http://test.ckan.net/user"><span class="icon">​</span>http://test.ckan.net/user</a> </p> <ul><li>Revision listing: </li></ul><p> <a class="ext-link" href="http://test.ckan.net/revision"><span class="icon">​</span>http://test.ckan.net/revision</a> </p> <p> This is most probably related with: </p> <p> <a class="ext-link" href="https://github.com/okfn/ckan/commit/1772a5c"><span class="icon">​</span>https://github.com/okfn/ckan/commit/1772a5c</a> </p> http://localhost/ticket/1501 http://localhost/ticket/1501 Report johnglover http://localhost/ticket/1502 http://localhost/ticket/1502 Report markbrough <p> Reasons: more people familiar with Git, bigger Open Source community on Github. </p> <p> Keeping track of the move here: <a class="ext-link" href="https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&amp;hl=en_GB#gid=0"><span class="icon">​</span>https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&amp;hl=en_GB#gid=0</a> </p> http://localhost/ticket/1503 http://localhost/ticket/1503 Report dread <p> This is what caused the exception: </p> <pre class="wiki">curl http://localhost:5000/api/action/package_search -d='{"q": "osm"}' </pre><p> The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary. </p> http://localhost/ticket/1504 http://localhost/ticket/1504 Report dread <p> This query caused ckan to except because ckan/controllers/api.py doesn't catch <a class="missing wiki">SearchError?</a> and <a class="missing wiki">SearchQueryError?</a>: </p> <pre class="wiki">curl http://localhost:5000/api/action/package_search -d '{"sort": "metadata_modified"}' </pre> http://localhost/ticket/1505 http://localhost/ticket/1505 Report rgrp <h2 id="HistoricalParentTickets">Historical Parent Tickets</h2> <ul><li><del>Parent-ticket: <a class="closed ticket" href="http://localhost/ticket/1450" title="enhancement: Simplify dataset view page (closed: fixed)">#1450</a></del> </li><li><del>Parent-ticket: <a class="closed ticket" href="http://localhost/ticket/1032" title="enhancement: [super] Resources in WUI (closed: fixed)">#1032</a></del> </li></ul><h2 id="StepZero">Step Zero</h2> <ul><li><del>COMPLETELY REDESIGN resource edit process</del> </li></ul><h2 id="DatasetEdit-Resources">Dataset Edit-Resources</h2> <ul><li><del>Force format field to be lowercase</del> <ul><li><del>Modify backend to serve it up in that format</del> </li></ul></li><li><del>Subticket <a class="closed ticket" href="http://localhost/ticket/978" title="enhancement: Edit Resource including Extras in Web UI (closed: fixed)">#978</a>: Add extra fields to resources in Web UI</del> </li><li><del>Fix display of large datasets (<a class="ext-link" href="http://test.ckan.net/dataset/europeana-lod"><span class="icon">​</span>http://test.ckan.net/dataset/europeana-lod</a>)</del> </li><li><del>When submitting forms with errors, open the correct resource pane.</del> </li><li><del>No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10</del> </li><li><del>Resource description field to become the markdown editor we use everywhere else.</del> </li></ul><h2 id="DatasetSettings">Dataset Settings</h2> <ul><li><del>Subticket <a class="closed ticket" href="http://localhost/ticket/1648" title="enhancement: Clarify that additional info = extra fields + add guidance (closed: fixed)">#1648</a>: Clarify edit terminology</del> </li><li><del>Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).</del> </li><li><del>Fix (i.e. allow / show) name editing on dataset edit.</del> <ul><li><del>Fix the slugify so it does not overwrite if i change title after fixing name.</del> </li><li><del>Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'</del> </li></ul></li><li><del>Autocomplete the author/maintainer box (requires API extension?)</del> [WONTFIX what? it's not like that field is linked to users of the site or anything] </li></ul><h2 id="DatasetView">Dataset View</h2> <ul><li><del>Fix animation glitch in expanding description</del> </li><li><del>Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?</del> </li><li><del>Verify that there are no more (none) placeholder &lt;td&gt; objects (i believe they've been refactored out)</del> </li></ul><h2 id="DatasetCreate">Dataset Create</h2> <ul><li><del>"Upload or link data files, APIs and other materials related to your dataset."</del> </li><li><del>Include the description markdown editor again (keep it small)</del> </li><li><del>Licence should maybe be restored</del> <ul><li><del>"(Don't worry if you don't know what your licence is)"</del> </li></ul></li><li><del>Show the log message field</del> <ul><li><del>Beautify</del> </li></ul></li><li><del>Pre-populate the log message field?</del> </li><li><del>? Show Home Page field when creating dataset</del> [WONTFIX don't push it] </li><li><del>? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation?</del> [WONTFIX doesnt make sense with new resource editor awesomeness] </li></ul><h2 id="Userpages">User pages</h2> <ul><li><del>Improvements to user page UX - 1d - <a class="closed ticket" href="http://localhost/ticket/1396" title="enhancement: Improvements to user page UX - 1d (closed: fixed)">#1396</a></del> </li><li><del>Broken links for non-Gravatar use icons - 0.25d - <a class="closed ticket" href="http://localhost/ticket/1719" title="defect: Broken links for non-Gravatar use icons - 0.25d (closed: fixed)">#1719</a></del> </li></ul><h2 id="Etc">Etc</h2> <ul><li>Take before and after screenshots </li></ul> http://localhost/ticket/1506 http://localhost/ticket/1506 Report rgrp <p> Group edit dataset add form needs some work </p> <ul><li>Dataset name is not cleared when you add </li><li>No way to remove item from list of datasets to be added if I make a mistake </li><li>(2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...) </li></ul> http://localhost/ticket/1507 http://localhost/ticket/1507 Report rgrp <p> This ticket is about finalizing, then deploying (to <a class="ext-link" href="http://thedatahub.org/"><span class="icon">​</span>http://thedatahub.org/</a>), and finally announcing some of recent major new features such as: </p> <ul><li>Deploy wordpresser extension for thedatahub (so we can start blogging) - <a class="closed ticket" href="http://localhost/ticket/1533" title="defect: Make sure ckanext-wordpresser works on CKAN 1.5.1 (closed: wontfix)">#1533</a> - WONTFIX </li><li><del>(Deploy new Resource and Dataset pages (invite comments))</del> </li><li>Get thedatahub geo-ified i.e. ckanext-spatial - <a class="closed ticket" href="http://localhost/ticket/1601" title="enhancement: [super] Well integrated Geo-spatial support including search (closed: fixed)">#1601</a> (?) <ul><li>Any additional dev that needs to be done? (e.g. geo in search ...?) </li></ul></li><li><del>(Deploy / re-enable ckanext-googleanalytics on thedatahub - <a class="closed ticket" href="http://localhost/ticket/1451" title="enhancement: Reintegrate download stats on dataset and resource view page - 0.5d (closed: fixed)">#1451</a> (more stuff <a class="closed ticket" href="http://localhost/ticket/1566" title="enhancement: [super] Finalize Google analytics extension (closed: fixed)">#1566</a>))</del> <ul><li>Rename the extension to ckanext-analytics? </li></ul></li><li>Custom forms (and use of 'dataset types') - Ticket (??) </li><li><del>(Get QA deployed and running - <a class="closed ticket" href="http://localhost/ticket/1582" title="enhancement: Deploy QA for thedatahub - 0.5d (closed: fixed)">#1582</a>)</del> </li></ul><p> It would be best if this was done by someone *unfamiliar* with the extensions so feedback could be provided on quality of install docs etc etc. </p> http://localhost/ticket/1508 http://localhost/ticket/1508 Report dread <p> e.g. <a class="ext-link" href="http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358"><span class="icon">​</span>http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358</a> gives: </p> <pre class="wiki">This is the current revision of this dataset, as edited at 2011-07-12 12:16. </pre><p> but it should say the date 2011-07-12 (which is the data being displayed). </p> <p> The problem is looking at <a class="missing wiki">PackageRevision?</a>, when it might be a tag or extra. </p> http://localhost/ticket/1509 http://localhost/ticket/1509 Report seanh <p> See '### Task 2' in this etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/notifications"><span class="icon">​</span>http://ckan.okfnpad.org/notifications</a> </p> <p> I have no idea yet about how to do this or how long it might take me. </p> http://localhost/ticket/1510 http://localhost/ticket/1510 Report seanh <p> Add a function to logic.action.get that returns a user's public activity stream as a list of dicts. </p> http://localhost/ticket/1511 http://localhost/ticket/1511 Report rgrp <p> If you modify your user name, it looks like you are logged out, but you are actually logged in and therefore can't login. (Reported by Pablo) </p> <p> As a work around, you need to logout and log back in. </p> <h1 id="implementation">implementation</h1> <ul><li>We should force this by auto-logging you out (DR: maybe!) </li></ul> http://localhost/ticket/1512 http://localhost/ticket/1512 Report dread <p> Richard Cyganiak: I often see a problem where half the system thinks that I'm logged in, while the other half insists I'm not. Logout then login again may fix it. </p> <p> David Read: I think I've seen it several times that the "My Account" link is given (suggesting you're logged in) but when you click it it asks you to log in, which means you weren't. </p> http://localhost/ticket/1513 http://localhost/ticket/1513 Report dread <p> If you edit your user name, the number of revisions you made returns to 0. This is because in the Revision object, the user's name is stored, rather than the user's ID. </p> <p> rgrp: We can reconnect that pretty easily (and have a longer term solution that involves not using the usernames but the userids in in the Revision objects so we don't hvae this problem in future!) </p> http://localhost/ticket/1514 http://localhost/ticket/1514 Report seanh <p> We want to add "activity streams" (chronological lists of activities) to CKAN. </p> <p> Activity streams standard: <a class="ext-link" href="http://activitystrea.ms/"><span class="icon">​</span>http://activitystrea.ms/</a> </p> <p> Super branch where the feature was initially developed (this has now been merged into master): <a class="ext-link" href="https://github.com/okfn/ckan/tree/feature-1515-activity-streams"><span class="icon">​</span>https://github.com/okfn/ckan/tree/feature-1515-activity-streams</a> </p> <p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/notifications"><span class="icon">​</span>http://ckan.okfnpad.org/notifications</a> (most relevant parts inlined here) </p> <p> Friedrich's mockup: <a class="ext-link" href="http://datahub.pudo.org/pudo"><span class="icon">​</span>http://datahub.pudo.org/pudo</a> </p> <h2 id="Tasks">Tasks</h2> <p> <del>Strikethrough</del> means the task is done. </p> <ul><li><del>New domain objects Activity and <a class="missing wiki">ActivityStream?</a> [2d] <a class="closed ticket" href="http://localhost/ticket/1298" title="enhancement: Generate activites to be put into activities table. (closed: fixed)">#1298</a></del> </li></ul><ul><li>Migrate existing data into activity streams model [3.5d] <a class="closed ticket" href="http://localhost/ticket/1510" title="enhancement: Migrate existing data into activity streams database tables (closed: wontfix)">#1510</a> </li></ul><ul><li><del>Generate activities when objects are saved [4d] <a class="closed ticket" href="http://localhost/ticket/1298" title="enhancement: Generate activites to be put into activities table. (closed: fixed)">#1298</a></del> </li></ul><ul><li><del>Get the activity stream for a given user as a JSONifiable list of dictionaries [1d] <a class="closed ticket" href="http://localhost/ticket/1511" title="enhancement: Logic function for getting a user's public activity stream (closed: fixed)">#1511</a></del> </li></ul><ul><li><del>API call for getting a user's public activity stream as JSON [0.5d] <a class="closed ticket" href="http://localhost/ticket/1495" title="enhancement: API call for getting a user's public activity stream (closed: fixed)">#1495</a></del> </li></ul><ul><li><del>API call for getting a user's public activity stream as rendered text, HTML, etc. [5d] <a class="closed ticket" href="http://localhost/ticket/1494" title="enhancement: API call for getting a user's public activity stream as rendered rtext (closed: wontfix)">#1494</a></del> </li></ul><ul><li><del>Render a user's activity stream on her user page [3d] <a class="closed ticket" href="http://localhost/ticket/1496" title="enhancement: Render a user's activity stream on her user page (closed: fixed)">#1496</a></del> (needs review/page design work) </li></ul><ul><li><del>Generate activities for other types of objects besides users (e.g. packages, groups, tags, extras, and save them in the database.</del> </li></ul><ul><li><del>Logic functions for getting activity streams for packages, groups, and users.</del> </li></ul><ul><li><del>API calls for getting activity streams of packages, groups, and users</del> as JSON. </li></ul><ul><li><del>Rendering activity streams for packages and groups etc. into their pages.</del> (done but disabled pending page design work) </li></ul><ul><li><del>API calls for inserting events into activity streams.</del> (Still need to handle activity details in the API, ticketed). </li></ul><ul><li>Conform to activitystrea.ms standard </li></ul><ul><li>Enable users to subscribe to and unsubscribe from activity streams, see an aggregated activity stream on their dashboard page. </li></ul><ul><li>Generate aggregated activity streams for arbitrary sets of multiple activity sources/streams </li></ul><ul><li>Email notifications </li></ul><ul><li>Extension point for extensions to register themselves as renderers for different types of activity </li></ul><ul><li>Deploy to UAT [0.5d] </li></ul><ul><li>Docs and announce [1d] </li></ul> http://localhost/ticket/1515 http://localhost/ticket/1515 Report amercader <p> CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like: </p> <p> <tt>http://&lt;solr-server&gt;/solr/schema-&lt;version&gt;</tt> </p> <p> Different scenarios: </p> <ul><li>Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine): <ul><li>Update CKAN </li><li>Update SOLR schema (symlink to suitable version in CKAN source) </li><li>Rebuild search index </li></ul></li></ul><ul><li>Single SOLR instance used by multiple CKAN sites. <ul><li>Update CKAN source </li><li>Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source) </li><li>Update solr_url on each of the CKAN sites with the suitable core </li><li>Rebuild search index </li></ul></li></ul><p> These are the SOLR and CKAN instances that need to be updated (Please add any missing ones): </p> <ul><li>Same machine as the CKAN site: <ul><li><del>test.ckan.net</del> </li><li><del>iati.test.ckan.net</del> </li><li>data.gov.uk/ catalogue.data.gov.uk (confirm) </li></ul></li></ul><ul><li><del>s046.okserver.org</del> <ul><li><del>iatiregistry.org</del> </li></ul></li></ul><ul><li><del>s052.okserver.org</del> <ul><li><del>datacatalogs.org</del> </li></ul></li></ul><ul><li><del>s004.okserver.org / eu4.okfn.org / solr.okfn.org</del> <ul><li>thedatahub.org (=www.ckan.net) </li><li>datagm.org.uk </li><li><del>publicdata.eu</del> </li><li>hri.fi (dev.fvh.fi?) </li><li>it.ckan.net </li><li>ie.ckan.net </li><li>cz.ckan.net </li><li>register.data.overheid.nl (nl.ckan.net?) </li><li>no.ckan.net / datakilder.no </li><li>br.ckan.net </li><li>colorado.ckan.net </li><li>at.ckan.net </li></ul></li></ul><p> TODO: which SOLR server are using these instances? </p> <ul><li>data.norge.no </li><li>nederland.ckan.net </li><li>lt.ckan.net </li><li>pl.ckan.net </li><li>datadotmontreal.ca/ </li><li>ca.ckan.net / datadotgc.com </li></ul> http://localhost/ticket/1516 http://localhost/ticket/1516 Report zephod <p> Clicking any of the links to a resource page in the dataset viewer seems to load the page and immediately start downloading the resource file. </p> <p> For example, being here: <a class="ext-link" href="http://test.ckan.net/dataset/europeana-lod"><span class="icon">​</span>http://test.ckan.net/dataset/europeana-lod</a> and clicking on any of the links to the first resource. </p> http://localhost/ticket/1517 http://localhost/ticket/1517 Report markbrough <p> E.g. uploading a file with spaces in it: OECD Monthly Exchange Rates.zip </p> <p> Gives a 404 Not Found response to the following file: <a class="ext-link" href="http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip"><span class="icon">​</span>http://test.ckan.org/api/storage/metadata/2011-12-05T193046/OECD%20Monthly%20Exchange%20Rates.zip</a> </p> <p> The upload wheel keeps spinning and the user is not informed that the upload has failed. </p> <p> Analysis: turns out that google storage (possibly s3 but not checked) replaces with ' ' in keys with '+' on upload. This breaks things because we try and look up metadata about upload using the filename/key we put in but of course that does not exist because google has changed name. </p> <p> Fix is trivial: replace ' ' in keys / filenames with '-'. </p> http://localhost/ticket/1518 http://localhost/ticket/1518 Report shevski <p> Makes more sense to only have one comprehensive stats/analytics extension, so when people are looking to add a stats extension they won't have to add two which may be confusing (is one an old version of the other? why do I have to add two? what's the difference? etc) </p> <p> User-wise we need a way to display our stats with google analytics in the same place </p> http://localhost/ticket/1519 http://localhost/ticket/1519 Report dread <p> Because of <a class="closed ticket" href="http://localhost/ticket/1514" title="defect: Modifying user name loses connection with revisions (closed: duplicate)">#1514</a> we should just disable name changing, until <a class="closed ticket" href="http://localhost/ticket/1514" title="defect: Modifying user name loses connection with revisions (closed: duplicate)">#1514</a> is done. </p> http://localhost/ticket/1520 http://localhost/ticket/1520 Report shevski <p> Various UX and UI improvements to groups. </p> <p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/feature-1521-groups-ux"><span class="icon">​</span>http://ckan.okfnpad.org/feature-1521-groups-ux</a> </p> <h2 id="Improvedgroups">Improved groups</h2> <p> From Richard Cyganiak. </p> <p> This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as <a class="ext-link" href="http://ie.ckan.net/"><span class="icon">​</span>http://ie.ckan.net/</a> could be turned into a simple group on the <a class="ext-link" href="http://thedatahub.org"><span class="icon">​</span>Data Hub</a> without losing their separate identity. </p> <ul><li><del><strong>Group homepages:</strong> Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.</del> </li><li><strong>Subdomains for groups:</strong> Enable subdomains such as <tt>ie.thedatahub.org</tt> as aliases for <tt>thedatahub.org/group/ireland</tt> </li><li><del><strong>Search within the group:</strong> Have a big search box that searches just within the group on the group page, with a look like on <a class="ext-link" href="http://thedatahub.org/dataset"><span class="icon">​</span>http://thedatahub.org/dataset</a></del> </li><li><del>Have an <strong>Add a dataset to this group</strong> link on the group page, perhaps in the sidebar.</del> </li><li><del><strong>Navigation within the group:</strong> Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from <a class="ext-link" href="http://thedatahub.org/dataset"><span class="icon">​</span>http://thedatahub.org/dataset</a></del> </li><li><del><strong>Group documentation:</strong> Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.</del> <ul><li><del>The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.</del> </li><li>Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines <em>About</em>, <em>Frequently Asked Questions</em> and <em>How to get involved</em>, we'd then get three links in the sidebar that point to <tt>datahub.org/group/ireland/about</tt>, <tt>datahub.org/group/ireland/frequently-asked-questions</tt>, <tt>datahub.org/group/ireland/how-to-get-involved</tt> <ul><li>RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was. </li></ul></li></ul></li></ul><h2 id="Additional">Additional</h2> <ul><li><del>Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)</del> </li><li>Better group index page (sort by number of dataset, allow searching ...) - <a class="new ticket" href="http://localhost/ticket/1403" title="enhancement: Refactor groups index page (new)">#1403</a> <ul><li>Add search to groups: <a class="ext-link" href="http://thedatahub.org/group"><span class="icon">​</span>http://thedatahub.org/group</a> </li><li>Way to filter by activity, number of datasets </li></ul></li><li><a class="closed ticket" href="http://localhost/ticket/1597" title="enhancement: Tag search - filter by group (closed: fixed)">#1597</a> <strong>Tag search, filtered by group</strong> <ul><li>RP: not worth it in terms of benefit/cost IMO </li></ul></li></ul><h2 id="Steps">Steps</h2> <ul><li><del>"Add Dataset" button on action bar</del> <ul><li><del>Make it a dropdown? -&gt; New Dataset / Existing Dataset</del> </li><li><del>Redirect Add Existing -&gt; Edit</del> </li><li><del>Check permissions</del> </li></ul></li><li><del>Expandable "about this group" box</del> </li><li><del>Implement simplest search-this-group box</del> <ul><li><del>turn the list of datasets into a dynamic list of search results</del> <ul><li><del>pagination</del> </li></ul></li><li><del>Matching set of facets in the sidebar (matches 'real' search page)</del> <ul><li><del>Deleteable, displayed facets</del> </li></ul></li></ul></li></ul><h2 id="Nice-To-Have">Nice-To-Have</h2> <ul><li>Get rid of the old group_package_show action (confirm with team) </li><li>Tabbed "about this group" box (extract headers?) </li><li><del>Scrollable group members widget in the sidebar</del> </li></ul> http://localhost/ticket/1521 http://localhost/ticket/1521 Report kindly <p> Need to add capacities to member tables. </p> http://localhost/ticket/1522 http://localhost/ticket/1522 Report icmurray <p> Form refactor. </p> <p> New create/edit package form wizard for DGU. See attached spec. </p> http://localhost/ticket/1523 http://localhost/ticket/1523 Report thejimmyg http://localhost/ticket/1524 http://localhost/ticket/1524 Report icmurray <p> Form refactor. New create/edit package form wizard for DGU. </p> <p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/dgu-package-form"><span class="icon">​</span>http://ckan.okfnpad.org/dgu-package-form</a> </p> <p> Tasks: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1526" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1526</a> : Simple one-page form. </li><li><a class="closed ticket" href="http://localhost/ticket/1527" title="enhancement: Break DGU package edit form into sections (closed: fixed)">#1527</a> : Tab-ify the form using JS. No validation between steps. </li><li><a class="closed ticket" href="http://localhost/ticket/1541" title="task: Setup server for the DGU form-refactor. (closed: fixed)">#1541</a> : Setup server for the DGU form-refactor. </li><li><a class="closed ticket" href="http://localhost/ticket/1537" title="enhancement: Package create form wizard (closed: fixed)">#1537</a> : Package create form wizard </li><li><a class="closed ticket" href="http://localhost/ticket/1538" title="enhancement: Store partially-filled in package form data as a draft. (closed: wontfix)">#1538</a> : <del>Store partially-filled in package form data as a draft.</del> </li><li><a class="closed ticket" href="http://localhost/ticket/1539" title="enhancement: Package forms auto-save (closed: wontfix)">#1539</a> : <del>Package forms auto-save</del> </li></ul><p> How the rest of the work is carried out is up for discussion, as it depends on the approach taken. Whether to continue with validation using AJAX queries, or whether to break the form down into multi pages. <a class="ext-link" href="https://docs.google.com/spreadsheet/ccc?key=0Av2RfDDpj67zdGpSdFlKVjNrQmFXV3lHbGM3ZFlDdnc&amp;hl=en_US#gid=1"><span class="icon">​</span>Google doc outlining design(s)</a> </p> http://localhost/ticket/1525 http://localhost/ticket/1525 Report icmurray <p> Re-create behaviour of old form without using form-alchemy. And unit test. </p> <ul><li>[X] write high-level functional tests for the form. Creation / edition / validation. </li></ul><blockquote> <blockquote> <p> Original estimate: 6 days (3 days x 2) </p> <blockquote> <p> Time spent: 3 days </p> </blockquote> <p> Time remaining: 0 days </p> </blockquote> </blockquote> <ul><li>[X] write the html </li></ul><blockquote> <blockquote> <p> Original estimate: 2 days (1 day x 2) </p> <blockquote> <p> Time spent: 1.3 days </p> </blockquote> <p> Time remaining: 0 days </p> </blockquote> </blockquote> <ul><li>[X] modify the existing validation schema </li></ul><blockquote> <blockquote> <p> Original estimate: 2 days (1 day x 2) </p> <blockquote> <p> Time spent: 1.3 days </p> </blockquote> <p> Time remaining: 1 days </p> </blockquote> </blockquote> <ul><li>[X] split resources into types: individual, time-series, and additional. </li></ul><blockquote> <blockquote> <p> Original estimate: 2 days </p> <blockquote> <p> Time spent: 2.5 days </p> </blockquote> <p> Time remaining: 0 days </p> </blockquote> </blockquote> <ul><li>[X] contact information is derived from Publisher, but can be altered for each dataset. <a class="closed ticket" href="http://localhost/ticket/1526#1617" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1617</a> </li></ul><blockquote> <blockquote> <p> Original estimate: 4 days </p> <blockquote> <p> Time spent: 0 days </p> </blockquote> <p> Time remaining: 4 days </p> </blockquote> <p> (Waiting on publisher integration) </p> </blockquote> <ul><li>[X] foi contact information is derived from user, but can be altered for each dataset. <a class="closed ticket" href="http://localhost/ticket/1526#1617" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1617</a> </li></ul><blockquote> <blockquote> <p> Original estimate: 1 days </p> <blockquote> <p> Time spent: 0 days </p> </blockquote> <p> Time remaining: 1 days </p> </blockquote> <p> (Waiting on publisher integration) </p> </blockquote> <ul><li>[X] provide stubs for the theme and sub-themes <a class="closed ticket" href="http://localhost/ticket/1526#1618" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1618</a> </li></ul><blockquote> <blockquote> <p> Original estimate: 1 days </p> <blockquote> <p> Time spent: 0 days </p> </blockquote> <p> Time remaining: 0 days </p> </blockquote> </blockquote> <ul><li>[X] provide stubs for the publisher hierarchy <a class="closed ticket" href="http://localhost/ticket/1526#1619" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1619</a> </li></ul><blockquote> <blockquote> <p> Original estimate: 2 days </p> <blockquote> <p> Time spent: 0.5 days </p> </blockquote> <p> Time remaining: 1.5 days </p> </blockquote> <p> Note - the groups refactor is essentially ready to be used. And this would be a good test case for it. </p> </blockquote> <ul><li>[X] don't allow package-type selection when editing a package </li></ul><blockquote> <blockquote> <p> Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. (<a class="closed ticket" href="http://localhost/ticket/1620" title="enhancement: Update and test existing DGU package form : don't allow package-type ... (closed: fixed)">#1620</a>) </p> <blockquote> <p> Original estimate: 0.5 days </p> <blockquote> <p> Time spent: 0 days </p> </blockquote> <p> Time remaining: 0 days </p> </blockquote> </blockquote> </blockquote> http://localhost/ticket/1526 http://localhost/ticket/1526 Report icmurray <ul><li>use javascript to selectively hide/show parts of the form </li><li>there's no validation between steps at this stage. It's still a "big save button at the end". </li></ul> http://localhost/ticket/1527 http://localhost/ticket/1527 Report zephod <p> Rather than the anonymous 'user' icon in the top-right corner, we could display your gravatar on all pages when you are logged in. </p> <p> This was suggested by Richard Cyganiak: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-November/001825.html</a> </p> http://localhost/ticket/1528 http://localhost/ticket/1528 Report dread <p> Currently when you log in it says "logged-in". Most sites show your user-name and this is helpful when you have more than one account or human using the computer. </p> http://localhost/ticket/1529 http://localhost/ticket/1529 Report lucychambers <p> Confusion between entering values in the title and name/url fields in datacatalogs.org. </p> <p> The help text "Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" applies to the name/url field, yet in Lucy's browser the text that is actually hovering over the title field in my browser. </p> http://localhost/ticket/1530 http://localhost/ticket/1530 Report kindly <p> The new members table needs a way so you can add arbitrary domain objects against them. </p> <p> We need to extend the group schema to accept types, and instead of just being able to add packages to groups add members with their capacities that associated with different table rows. </p> <p> 4d </p> http://localhost/ticket/1531 http://localhost/ticket/1531 Report dread <p> The log-in page says "Login using Open ID" and gives instructions for signing up. YET this is only available to users who've already added openid to their account. If you have not done this and then sign-in via OpenID (which is successful from the OpenID end) then you are told "Login failed. Bad username or password." in a flash message. </p> <p> Proposed solution (i don't know if this is possible): </p> <ul><li>When you log-in for the first time via OpenID, it doesn't actually log you in in CKAN. It just sends you to the 'Create User' page with the OpenID field pre-filled, and puts up a flash message "This OpenID account is not yet registered on thedatahub. Please complete your details.". This allows you to complete the registration and logs you in, and allows you to log-in directly with OpenID in the future. </li></ul><p> Alternatives: </p> <ul><li>Just change the error message to be 'You need to register in CKAN first. Quote your OpenID in the registration form to use it in future.' </li><li>Remove OpenID altogether </li></ul> http://localhost/ticket/1532 http://localhost/ticket/1532 Report amercader <p> Superceded by <a class="assigned ticket" href="http://localhost/ticket/2234" title="enhancement: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN ... (assigned)">#2234</a> </p> <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1508" title="enhancement: [super] Finalize and deploy some of our major (new) features as of Dec ... (closed: fixed)">#1508</a> </p> <p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/wordpresser"><span class="icon">​</span>http://ckan.okfnpad.org/wordpresser</a> </p> <p> The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded. </p> <p> Looks like this changeset may be involved <a class="ext-link" href="https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1"><span class="icon">​</span>https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1</a> </p> <p> I think that even before that changeset, the page was shown without any margins. </p> <p> In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version </p> http://localhost/ticket/1533 http://localhost/ticket/1533 Report rgrp <p> The use of username is problematic because username's can change. </p> <ul><li>Change all revision creation code to use user id (simplest is to change c.author field in lib/base.py (?)) <ul><li>(?) Add a field ipaddr for ip address of anonymous users? (or just keep putting this in author field on Revision and then acception that those won't match when we do a look up against user table) </li></ul></li><li>Change user view page to look up against user id rather than name </li><li>Perform migration on existing Revision objects <ul><li>Match should probably be against both openid and username when searching Revisions' author field (especially true on CKAN where some people have already changed their username from being their openid) </li></ul></li></ul> http://localhost/ticket/1534 http://localhost/ticket/1534 Report dread <p> When using the API, the apikey needs to be supplied in a header called 'Authorization'. Because some proxys / deployments use this header for other things, a configurable header was provided as an alternative, with default "X-CKAN-API-KEY". </p> <p> Rufus suggests having *one* way for this. a) making this not configurable any more b) making X-CKAN-API-KEY the default </p> <p> (keep Authorization allowed, but not documented, for backwards compatibility) </p> http://localhost/ticket/1535 http://localhost/ticket/1535 Report dread <p> In the Web interface and API, an administrator can set the 'state' field of a group to 'deleted' but this change is ignored in the save/submit and it doesn't get deleted. </p> http://localhost/ticket/1536 http://localhost/ticket/1536 Report icmurray <p> Create the form wizard for the package-new form. </p> <p> Each section of the form will be a separate page as this was decided to be simpler than the alternative of making AJAX calls for validation at each stage. (*) </p> <ul><li>separate pages for each section of the form </li><li>validation carried out at each stage against the whole schema. Each section/page declares a list of schema keys that need to validate for that section to validate, and thus move onto the next section. </li><li>no draft saving to be performed in this ticket. </li></ul><p> (*) - although the javascript alternative will probably provide better UX (each step would require a page-load in the wizard approach), it was decided that: </p> <ul><li>with the javascript approach it would be harder to test the workflow. </li><li>with the javascript approach there would be additional work displaying validation correctly. Although not that complicated, it was felt to add another point of failure. </li><li>the multi-page wizard is quicker and easier to implement, and if it provided poor UX, then the javascript approach would be used instead. </li><li>the multi-page wizard wouldn't preclude a javascript-tabbing create-form for other cases (where the wizard workflow wasn't such a good match, eg on the hedatahub.org) </li><li>the multi-page wizard wouldn't preclude a javascript-tabbing edit-form. </li></ul> http://localhost/ticket/1537 http://localhost/ticket/1537 Report icmurray <p> Allow a user to save a partially-complete and probably invalid package-create form in order to come back to it later. </p> <p> We decided on blob storage over a state field on the package: </p> <ul><li>prevents a history of the drafts being stored in the revision history. This is a particular problem if we start auto-saving, say every 20s. </li><li>the blob can store arbitrary extra data, such as how far through the wizard we are. </li></ul> http://localhost/ticket/1538 http://localhost/ticket/1538 Report icmurray <p> Using the saved-draft feature, we enable auto-save for the package-new and package-edit forms. </p> <p> An AJAX call is made every N seconds, and the auto-saved blob is updated. </p> <p> note - we don't want to overwrite a previously save draft, but we will want to overwrite the last autosave. </p> http://localhost/ticket/1539 http://localhost/ticket/1539 Report amercader <p> Both in 1.5.1b: </p> <p> <a class="ext-link" href="http://thedatahub.org/api/search/dataset?groups=lodcloud&amp;title="><span class="icon">​</span>http://thedatahub.org/api/search/dataset?groups=lodcloud&amp;title=</a> </p> <p> and 1.5.2a (current master): </p> <p> <a class="ext-link" href="http://test.ckan.net/api/search/dataset?groups=lodcloud&amp;title="><span class="icon">​</span>http://test.ckan.net/api/search/dataset?groups=lodcloud&amp;title=</a> </p> <p> Although the error message in 1.5.2a is more verbose: </p> <p> "Bad request - Bad search option: HTTP code=400, reason=org.apache.lucene.queryParser.<a class="missing wiki">ParseException?</a>: Cannot parse 'groups:lodcloud title:': Encountered \"&lt;EOF&gt;\" at line 1, column 22. Was expecting one of: \"(\" ... \"*\" ... &lt;QUOTED&gt; ... &lt;TERM&gt; ... &lt;PREFIXTERM&gt; ... &lt;WILDTERM&gt; ... \"[\" ... \"{\" ... &lt;NUMBER&gt; ..." </p> <p> Some parameter validation before sending it to Solr should do the trick </p> http://localhost/ticket/1540 http://localhost/ticket/1540 Report icmurray <p> To enable us to show DGU work in progress, for feedback. </p> http://localhost/ticket/1541 http://localhost/ticket/1541 Report dread <p> A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it. </p> <p> The existing two-stage revision delete is currently unreliable and perhaps too laborious. </p> <p> Olav and Richard have needs along this line. </p> http://localhost/ticket/1542 http://localhost/ticket/1542 Report amercader <p> E.g. Pagination links on this page don't include groups=lodcloud <a class="ext-link" href="http://thedatahub.org/dataset?groups=lodcloud"><span class="icon">​</span>http://thedatahub.org/dataset?groups=lodcloud</a> </p> <p> Not sure if related to <a class="closed ticket" href="http://localhost/ticket/1501" title="defect: Pagination links broken (closed: fixed)">#1501</a> (probably not) </p> http://localhost/ticket/1543 http://localhost/ticket/1543 Report dread <p> We have 150 odd branches (<tt>git branch -a</tt>) - most of them old - we should prune them. At very least, branches that have been merged in should be deleted. Look at old branches that haven't been merged in and wonder why. </p> <p> May be of some use: </p> <pre class="wiki">git branch --merged git branch --no-merged </pre> http://localhost/ticket/1544 http://localhost/ticket/1544 Report amercader <p> CKAN is pulling a number of resources from external locations. This causes problems when connectivity is limited and you have to work locally. Maybe some of them cold be moved to CKAN source to avoid external requests. </p> <p> Quick search: </p> <pre class="wiki"> ./ckan/templates/layout_base.html: &lt;img src="http://assets.okfn.org/images/logo/okf_logo_white_and_green_tiny.png" id="footer-okf-logo" /&gt; ./ckan/templates/layout_base.html: &lt;a href="http://opendefinition.org/"&gt;&lt;img alt="This Content and Data is Open" src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" style="border: none ; margin-bottom: -4px;"/&gt;&lt;/a&gt; ./ckan/templates/package/resource_read.html: &lt;img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /&gt; ./ckan/templates/package/read.html: &lt;img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /&gt;&lt;/a&gt; ./ckan/templates/_util.html: &lt;img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /&gt; ./ckan/templates/_util.html: &lt;img src="http://assets.okfn.org/images/ok_buttons/od_80x15_blue.png" alt="[Open Data]" /&gt; ./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js: this.$dialog.html('&lt;h2&gt;Loading results...&lt;/h2&gt;&lt;img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" /&gt;'); ./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js: self.setMessage('Uploading file ... &lt;img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" /&gt;'); ./ckan/public/scripts/vendor/ckanjs/1.0.0/ckanjs.js: self.setMessage('Checking upload permissions ... &lt;img src="http://assets.okfn.org/images/icons/ajaxload-circle.gif" class="spinner" /&gt;'); Binary file ./ckan/lib/app_globals.pyc matches ./ckan/lib/app_globals.py: 'http://assets.okfn.org/p/ckan/img/ckan.ico') ./ckan/config/deployment.ini_tmpl:ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico </pre> http://localhost/ticket/1545 http://localhost/ticket/1545 Report johnglover <p> metadata_created seems to be returning the date of the earliest revision (for all packages), rather than the earliest revision for a specific package. </p> http://localhost/ticket/1546 http://localhost/ticket/1546 Report amercader <p> Time estimate: 1d </p> <p> Add a two search method to the IPackageController interface so extensions can: </p> <ul><li>Modify the query being sent to Solr </li></ul><pre class="wiki"> def before_search(search_params): return search_params </pre><ul><li>Modify the results returned (<tt>after_search</tt>) </li></ul><pre class="wiki"> def after_search(search_results, search_params): return search_results </pre><p> Please see this pad for implementation details: </p> <p> <a class="ext-link" href="http://ckan.okfnpad.org/search"><span class="icon">​</span>http://ckan.okfnpad.org/search</a> </p> http://localhost/ticket/1547 http://localhost/ticket/1547 Report dread <p> It's hard to understand what's going on with the functional tests because there is so much going on in inherited classes. Would be better to be explicit in constructors, even if it involves cut &amp; paste. </p> http://localhost/ticket/1548 http://localhost/ticket/1548 Report ross <p> It would be great to have a CKAN extension that allowed users (or CKAN itself) to generate short links to other URIs (both internal and external). Once created, shortlinks made by CKAN should be changeable. This would allow uploaded content to be moved without the user's link changing at all. The tool itself might also be of use as a general link-shortener to users other than the CKAN system itself. </p> <p> Another useful feature would be for this to also collect some simple analytics such as the referrer and client IP for future reference. I'm not yet sure what we would do with the analytics other than some sort of popularity metric. </p> <p> Questions: </p> <ul><li>Core, or Extension, or Self-hosted? </li></ul> http://localhost/ticket/1549 http://localhost/ticket/1549 Report ross <p> It should be possible to pass userid/username and api key and obtain a response from CKAN for external services that use CKAN auth. Those services shouldn't be talking to the DB directly. </p> http://localhost/ticket/1550 http://localhost/ticket/1550 Report ross <p> Webstore should use auth api ( <a class="assigned ticket" href="http://localhost/ticket/1550" title="enhancement: Allow simple auth via the API (assigned)">#1550</a> ) for authenticating users accessing webstore rather than talking directly to the CKAN database. We also need it to suppose /user/ urls and /userid/ for accessing databases. </p> http://localhost/ticket/1551 http://localhost/ticket/1551 Report dread <p> (instead of using the prompt) - needed for deployment </p> http://localhost/ticket/1552 http://localhost/ticket/1552 Report dread <p> When you install ckanext-store (Storage extension) then CKAN displays 'Upload a file' button in the dataset/resource edit tab. </p> <p> If you are not logged in, then this feature doesn't work, but there is nothing to indicate this. The button should be disabled. </p> <p> Furthermore, it lets you think you are starting the upload process, the spinner spins, it says 'Uploading...' and stays like that for a long while. </p> <p> This is a problem CKAN 1.4.3(ish) - 1.5.1 (current version) when you install the current ckanext-storage. </p> http://localhost/ticket/1553 http://localhost/ticket/1553 Report jilly mathews <p> Finish any work out standing on web store preview extension to be able to package and release. </p> <p> Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted. </p> http://localhost/ticket/1557 http://localhost/ticket/1557 Report jilly mathews <p> Summarise final set of requirements for this and finish development and test. Estimated 10 working days. </p> http://localhost/ticket/1558 http://localhost/ticket/1558 Report jilly mathews <p> Polish off comments extension dev and test. estimate 2 days. </p> http://localhost/ticket/1559 http://localhost/ticket/1559 Report jilly mathews <p> Estimate 2 days to finish dev and test. </p> <p> David can you add any info needed here? </p> http://localhost/ticket/1560 http://localhost/ticket/1560 Report jilly mathews <p> Can we finish this ready for release on data hub and CKAN Hosted. </p> http://localhost/ticket/1561 http://localhost/ticket/1561 Report jilly mathews <p> Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients. </p> http://localhost/ticket/1562 http://localhost/ticket/1562 Report jilly mathews <p> Unsure what needs to happen here. Need to list outstanding tasks and implement. </p> http://localhost/ticket/1563 http://localhost/ticket/1563 Report jilly mathews <p> Basic websotre exists but this may be not what is described yet. </p> <p> CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first. </p> http://localhost/ticket/1564 http://localhost/ticket/1564 Report jilly mathews <p> Is testing complete and ready for release? </p> http://localhost/ticket/1565 http://localhost/ticket/1565 Report jilly mathews <ul><li><del>(Deploy it)</del> </li><li>Integrate with stats plugin (now that is in core) - <a class="new ticket" href="http://localhost/ticket/1101" title="enhancement: Integrate googlanalytics into site nav (new)">#1101</a> </li><li>Blog about it <a class="new ticket" href="http://localhost/ticket/1581" title="enhancement: Blog post about Google Analytics extension for CKAN (new)">#1581</a> </li><li>(Other tickets re upgrading for v1.5.1 - done but where are tickets?) </li></ul> http://localhost/ticket/1566 http://localhost/ticket/1566 Report jilly mathews <p> Requires change to celeryd. Estimated 4 weeks. </p> http://localhost/ticket/1567 http://localhost/ticket/1567 Report jilly mathews <p> Can this be released as a standard CKAN feature? </p> http://localhost/ticket/1568 http://localhost/ticket/1568 Report jilly mathews <p> How much effort will this be to be ready to use? </p> http://localhost/ticket/1569 http://localhost/ticket/1569 Report jilly mathews <p> Is this ready for release? What needs to be done? </p> http://localhost/ticket/1570 http://localhost/ticket/1570 Report jilly mathews <p> "Proper" issues extension. </p> <ul><li>Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/issues"><span class="icon">​</span>http://ckan.okfnpad.org/issues</a> </li><li>Code: <a class="ext-link" href="https://github.com/okfn/ckanext-issues"><span class="icon">​</span>https://github.com/okfn/ckanext-issues</a> </li></ul><p> Overview </p> <ul><li>This would be an extension and improvement of existing todo extension - <a class="closed ticket" href="http://localhost/ticket/842" title="enhancement: Todo list CKAN extension (closed: fixed)">#842</a> (<a class="ext-link" href="https://github.com/okfn/ckanext-issues"><span class="icon">​</span>https://github.com/okfn/ckanext-issues</a>) <ul><li>Current extension does work but lacks polish </li><li>Should we polish or do significant improvements? </li></ul></li><li>Functionality - see etherpad </li></ul><h2 id="Steps">Steps</h2> <ul><li><del>(Rename todo(s) -&gt; issue(s) in existing extension)</del> </li></ul> http://localhost/ticket/1571 http://localhost/ticket/1571 Report jilly mathews <p> Need to write custom harvesters for each client. Is it worth having one for data hub? </p> http://localhost/ticket/1572 http://localhost/ticket/1572 Report jilly mathews <p> Estimate 2 weeks for someone to finish and test. </p> http://localhost/ticket/1573 http://localhost/ticket/1573 Report ross <p> Allow uploading of files to push the data into webstore. Initially we were going to suggest changes to ckanext-storage but after further analysis we arrived at the conclusion that this should be implemented by ckanext-archiver as it already handles archiving of data from various sources and would be the best place to 'archive' to webstore. </p> <ol><li>A user wants to upload a file to CKAN, and so chooses the file upload option as they do currently with ckanext-storage. The file upload itself should be handled by this code internal to core (see <a class="closed ticket" href="http://localhost/ticket/1608" title="enhancement: Merge storage into Core (closed: fixed)">#1608</a>) but still using ofs [5d] </li></ol><ol start="2"><li>The user is provided the link to the file as currently. </li></ol><ol start="3"><li>The archiver's update task will check configuration to determine if it should upload some mime-types (tabular formats) to webstore. Configuration should also determine where webstore is. (see <a class="closed ticket" href="http://localhost/ticket/1609" title="enhancement: Celery task for ckanext-archiver to write to webstore. (closed: fixed)">#1609</a>) [5d] - took 6 </li></ol><ul><li>It should be stored in webstore using {username}/{resource-id} where {username} is the name of the user uploading the original file. </li></ul><ol start="4"><li>The archiver will continue to move the file into permanent storage as per current practice. </li></ol><h2 id="Tickets">Tickets</h2> <ul><li><a class="closed ticket" href="http://localhost/ticket/1608" title="enhancement: Merge storage into Core (closed: fixed)">#1608</a> Merge storage into Core [5d] </li><li><a class="closed ticket" href="http://localhost/ticket/1609" title="enhancement: Celery task for ckanext-archiver to write to webstore. (closed: fixed)">#1609</a> Celery task for ckanext-archiver to write to webstore. [5d] (took 6) </li><li><a class="closed ticket" href="http://localhost/ticket/1687" title="enhancement: Deploy new storage changes from master to datahub (closed: invalid)">#1687</a>: Deploy to datahub [1d] </li><li><a class="closed ticket" href="http://localhost/ticket/1681" title="enhancement: Core storage document (closed: fixed)">#1681</a>: Core storage documentation [1d?] <ul><li>Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc </li><li>External: here's how to configure it and here's what you get ... (goes in main sphinx docs) </li></ul></li></ul> http://localhost/ticket/1574 http://localhost/ticket/1574 Report dread <p> Last week I imported ca.ckan.net datasets into thedatahub.org, but the tags seem to have lost their dashes, underscores and dots. </p> http://localhost/ticket/1575 http://localhost/ticket/1575 Report rgrp <ul><li>Move into core (keeping as an extension) and default enable </li><li>Do we need tests for controller? (No: already there) </li><li>Documentation (?) - e.g. of basic config options </li><li>Remove lib/stats.py which is still in core <ul><li>dread moved it to extension in <a class="ext-link" href="https://github.com/okfn/ckanext-stats/commit/293b2544737d8ab7610884bd5e999ee37f1ab813"><span class="icon">​</span>https://github.com/okfn/ckanext-stats/commit/293b2544737d8ab7610884bd5e999ee37f1ab813</a> but did not remove from core </li></ul></li><li>Deprecation notice on ckanext-stats repo and update extension registry </li></ul><p> Est: 0.5d. </p> <p> Questions: </p> <ul><li>Why do this? <ul><li>tiny extension with few dependencies - and really nice to have out of the box </li><li>trial for doing this on larger scale </li></ul></li><li>Do we keep as extension (even if in core)? <ul><li>Ans: Yes, keep as extension because: <ul><li>Already set up that way </li><li>Cleaner </li><li>Easier to disable / enable </li></ul></li></ul></li><li>How do we integrate with main them (e.g. have stats link) <ul><li>Ans: not sure (this is part of more general issue of how we update theme for varying changes elsewhere). Best answer is to have some widgetization in theme. </li></ul></li><li>Hide ratings section (at least until we reinstate ratings <a class="new ticket" href="http://localhost/ticket/1598" title="enhancement: Reinstate Ratings (new)">#1598</a>) <ul><li>Ans: no, let's not bother (and having ratings there encourages us to do <a class="new ticket" href="http://localhost/ticket/1598" title="enhancement: Reinstate Ratings (new)">#1598</a> and/or find out whether people are interested in ratings) </li></ul></li></ul> http://localhost/ticket/1576 http://localhost/ticket/1576 Report dread <p> Looks like uploading a file with foreign characters fails due to encoding reasons. </p> <pre class="wiki">URL: http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf Module weberror.errormiddleware:162 in __call__ &lt;&lt; __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: &gt;&gt; app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ &lt;&lt; self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) &gt;&gt; return self.app(environ, start_response) Module beaker.middleware:152 in __call__ &lt;&lt; headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): &gt;&gt; return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ &lt;&lt; environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already &gt;&gt; response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ &lt;&lt; controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, &gt;&gt; response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: log.debug("Calling controller class with WSGI interface") return controller(environ, start_response) def load_test_env(self, environ): &gt;&gt; return controller(environ, start_response) Module ckan.lib.base:123 in __call__ &lt;&lt; # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() &gt;&gt; return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ &lt;&lt; return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response &gt;&gt; response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: &gt;&gt; response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: &gt;&gt; result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): &gt;&gt; return func(**args) Module ckanext.storage.controller:2 in auth_form Module ckan.lib.jsonp:26 in jsonpify &lt;&lt; Very much modelled after pylons.decorators.jsonify . """ data = func(*args, **kwargs) return to_jsonp(data) &gt;&gt; data = func(*args, **kwargs) Module ckanext.storage.controller:301 in auth_form &lt;&lt; method = 'POST' authorize(method, bucket, label, c.userobj, self.ofs) data = self._get_form_data(label) return data &gt;&gt; authorize(method, bucket, label, c.userobj, self.ofs) Module ckanext.storage.controller:79 in authorize &lt;&lt; if method != 'GET': # do not allow overwriting if ofs.exists(bucket, key): abort(409) # now check user stuff &gt;&gt; if ofs.exists(bucket, key): Module ofs.remote.botostore:53 in exists &lt;&lt; if bucket is None: return False return (label is None) or (label in bucket) def claim_bucket(self, bucket): &gt;&gt; return (label is None) or (label in bucket) Module boto.s3.bucket:87 in __contains__ &lt;&lt; def __contains__(self, key_name): return not (self.get_key(key_name) is None) def startElement(self, name, attrs, connection): &gt;&gt; return not (self.get_key(key_name) is None) Module boto.s3.bucket:144 in get_key &lt;&lt; response = self.connection.make_request('HEAD', self.name, key_name, headers=headers, query_args=query_args) # Allow any success status (2xx) - for example this lets us # support Range gets, which return status 206: &gt;&gt; query_args=query_args) Module boto.s3.connection:388 in make_request &lt;&lt; if isinstance(key, Key): key = key.name path = self.calling_format.build_path_base(bucket, key) boto.log.debug('path=%s' % path) auth_path = self.calling_format.build_auth_path(bucket, key) &gt;&gt; path = self.calling_format.build_path_base(bucket, key) Module boto.s3.connection:88 in build_path_base &lt;&lt; def build_path_base(self, bucket, key=''): return '/%s' % urllib.quote(key) class SubdomainCallingFormat(_CallingFormat): &gt;&gt; return '/%s' % urllib.quote(key) Module urllib:1222 in quote &lt;&lt; safe_map[c] = (c in safe) and c or ('%%%02X' % i) _safemaps[cachekey] = safe_map res = map(safe_map.__getitem__, s) return ''.join(res) &gt;&gt; res = map(safe_map.__getitem__, s) KeyError: u'\xed' CGI Variables AUTH_TYPE 'cookie' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.3' HTTP_ACCEPT_ENCODING 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE 'en-US,en;q=0.8' HTTP_CACHE_CONTROL 'max-age=259200' HTTP_CONNECTION 'keep-alive' HTTP_COOKIE 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; ckan_user=elenaibp; ckan_display_name="Elena Mondo"; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; __utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10.1324298316; __utmc=119670349; __utmz=119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' HTTP_HOST 'thedatahub.org' HTTP_REFERER 'http://thedatahub.org/dataset/edit/budget-library-czeck-republic' HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7' HTTP_VIA '1.1 localhost (squid/3.0.STABLE19)' HTTP_X_FORWARDED_FOR '87.114.74.190' HTTP_X_REQUESTED_WITH 'XMLHttpRequest' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' REMOTE_ADDR '193.34.146.142' REMOTE_PORT '55419' REMOTE_USER u'elenaibp' REMOTE_USER_DATA 'userid_type:unicode' REMOTE_USER_TOKENS [''] REQUEST_METHOD 'GET' REQUEST_URI '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ%C3%AD-%C4%8Cesk%C3%A9-republiky-_-P%C5%99%C3%ADprava-rozpo%C4%8Dtu.pdf' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://thedatahub.org/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SCRIPT_URL '/api/storage/auth/form/2011-12-19T124447/Ministerstvo-financ\xc3\xad-\xc4\x8cesk\xc3\xa9-republiky-_-P\xc5\x99\xc3\xadprava-rozpo\xc4\x8dtu.pdf' SERVER_ADDR '193.34.146.146' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.14 (Ubuntu) Server at thedatahub.org Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.14 (Ubuntu)' WSGI Variables application &lt;beaker.middleware.CacheMiddleware object at 0x7f22601c7dd0&gt; beaker.cache &lt;beaker.cache.CacheManager object at 0x7f22601c7b50&gt; beaker.get_session &lt;bound method SessionMiddleware._get_session of &lt;beaker.middleware.SessionMiddleware object at 0x7f22601c7a90&gt;&gt; beaker.session {'_accessed_time': 1324298703.071357, '_creation_time': 1324293077.4139669} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group 'ckan.net' mod_wsgi.reload_mechanism '1' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 8) paste.cookies (&lt;SimpleCookie: __utma='119670349.1809834699.1323782464.1324293066.1324298316.4' __utmb='119670349.3.10.1324298316' __utmc='119670349' __utmz='119670349.1323782464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' auth_tkt='f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode' ckan_apikey='decd48b1-49ee-4250-bff4-98ccca9c02a5' ckan_display_name='Elena Mondo' ckan_user='elenaibp' hide_welcome_message='1' thedatahub_net='27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e'&gt;, 'thedatahub_net=27a7f095fcca1ea6b36df996d595e3278b16f4538862bf7f88d49e2000b9246547c8fd0e; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; auth_tkt="f9c6ab2b0d9fcd71c4c2408bc12fab544eef1c45elenaibp!userid_type:unicode"; ckan_user=elenaibp; ckan_display_name="Elena Mondo"; ckan_apikey=decd48b1-49ee-4250-bff4-98ccca9c02a5; hide_welcome_message=1; _ _utma=119670349.1809834699.1323782464.1324293066.1324298316.4; __utmb=119670349.3.10...)|utmcmd=(none)') paste.registry &lt;paste.registry.Registry object at 0x7f226194df50&gt; paste.throw_errors True pylons.action_method &lt;bound method StorageAPIController.auth_form of &lt;ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990&gt;&gt; pylons.controller &lt;ckanext.storage.controller.StorageAPIController object at 0x7f2261dad990&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.PylonsContext object at 0x7f2261daddd0&gt; pylons.routes_dict {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'} repoze.who.identity &lt;repoze.who identity (hidden, dict-like) at 139785645747120&gt; repoze.who.logger &lt;logging.Logger instance at 0x7f225e23c098&gt; repoze.who.plugins {'openid': &lt;OpenIdIdentificationPlugin 139785625065680&gt;, 'friendlyform': &lt;FriendlyFormPlugin 139785618095248&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f2260874c10&gt;, 'auth_tkt': &lt;AuthTktCookiePlugin 139785625065808&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f2260874c90&gt;} routes.route &lt;routes.route.Route object at 0x7f22601a1090&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f2261dadf50&gt; webob._parsed_query_vars (GET([]), '') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multiprocess' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f2261da9af8&gt; wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f2261dadf50&gt;, {'action': u'auth_form', 'controller': u'ckanext.storage.controller:StorageAPIController', 'label': u'2011-12-19T124447/Ministerstvo-financ\xed-\u010cesk\xe9-republiky-_-P\u0159\xedprava-rozpo\u010dtu.pdf'}) </pre> http://localhost/ticket/1577 http://localhost/ticket/1577 Report rgrp http://localhost/ticket/1578 http://localhost/ticket/1578 Report ross <p> We should allow full text search within databases for individual databases, although we may want to wait for any pending ports to postgres. </p> http://localhost/ticket/1579 http://localhost/ticket/1579 Report johnglover <ul><li>Write up: task status table in <a class="ext-link" href="http://wiki.ckan.org/Domain_Model"><span class="icon">​</span>http://wiki.ckan.org/Domain_Model</a> </li><li>Write up: <a class="ext-link" href="http://wiki.ckan.org/Data_Quality"><span class="icon">​</span>http://wiki.ckan.org/Data_Quality</a> to reflect current state of play (move all old material to "Discussion / Talk" page) </li><li>Ping ckan-discuss re new Data Quality page </li></ul> http://localhost/ticket/1580 http://localhost/ticket/1580 Report johnglover <p> The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post. </p> <p> Can ping John Glover in January for any details required. </p> <p> Key link is: <a class="ext-link" href="http://thedatahub.org/analytics/dataset/top"><span class="icon">​</span>http://thedatahub.org/analytics/dataset/top</a> though this should probably move to be under stats (e.g. <a class="ext-link" href="http://thedatahub.org/stats/usage"><span class="icon">​</span>http://thedatahub.org/stats/usage</a>) </p> http://localhost/ticket/1581 http://localhost/ticket/1581 Report johnglover <ul><li>deploy celery </li><li>deploy QA and archiver tasks </li><li>write up a blog post announcing QA on thedatahub </li></ul> http://localhost/ticket/1582 http://localhost/ticket/1582 Report johnglover <p> Tasks: </p> <ul><li><del>On dataset view page, add star rating after each resource view url. Probably similar to current format label after url.</del> </li><li><del>Add star rating to resource view page, along with a way of displaying the reason for the rating and tips to improve. Should be unobtrusive.</del> </li><li><del>Fix the table at <a class="ext-link" href="http://thedatahub.org/qa/dataset/broken_resource_links"><span class="icon">​</span>http://thedatahub.org/qa/dataset/broken_resource_links</a> - URLs should be truncated to a fixed not-too-long length and the other fields line up (eg by using a table)~ </del></li><li><del>Handle 404s by showing that 'we think this resource is 404ing'.</del> </li><li><del>Handle the absence of QA data on a resource (currently nothing is displayed. Maybe that's fine.</del> </li><li><del>Do we need to integrate the 'reason for rating' string into the UI?</del> </li></ul><p> Next Steps: </p> <ul><li><del>Use db dump of thedatahub.org for final testing.</del> </li></ul> http://localhost/ticket/1583 http://localhost/ticket/1583 Report johnglover <p> Super: <a class="closed ticket" href="http://localhost/ticket/1594" title="enhancement: [super] QA Improvements (closed: wontfix)">#1594</a> </p> <ul><li>qa/{username} </li><li>qa/{groupname} </li><li>paginate QA results </li><li>search / filter QA results </li><li><del>list organisation report by default, but can disable via config option</del> (done) </li><li>UX tidy up of report pages - hide border if no sidebar, etc </li></ul> http://localhost/ticket/1584 http://localhost/ticket/1584 Report dread <p> (details embargoed until 31/1/2012) </p> http://localhost/ticket/1585 http://localhost/ticket/1585 Report dread <p> We default to sort packages search results by "search rank", but we can change it to be "name", "title", "metadata_modified" date, "metadata_created" date. It works in the API now, so let's have (say) a drop-down in the Web UI for this. </p> <p> Follows on from <a class="closed ticket" href="http://localhost/ticket/191" title="enhancement: Searching by modification date (closed: fixed)">#191</a>. </p> http://localhost/ticket/1586 http://localhost/ticket/1586 Report johnglover <p> Super: <a class="closed ticket" href="http://localhost/ticket/1594" title="enhancement: [super] QA Improvements (closed: wontfix)">#1594</a> </p> <p> From Richard Cyganiak on the CKAN discuss list: </p> <p> According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license. </p> http://localhost/ticket/1587 http://localhost/ticket/1587 Report johnglover <p> Super: <a class="closed ticket" href="http://localhost/ticket/1594" title="enhancement: [super] QA Improvements (closed: wontfix)">#1594</a> </p> <p> From Richard Cyganiak on the CKAN Discuss list: </p> <p> Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type "api/sparql". A few more are recorded with the format "SPARQL". I suggest that datasets with such resources should also be considered for the fourth star. </p> http://localhost/ticket/1588 http://localhost/ticket/1588 Report johnglover <p> Super: <a class="closed ticket" href="http://localhost/ticket/1594" title="enhancement: [super] QA Improvements (closed: wontfix)">#1594</a> </p> <p> From Richard Cyganiak on the CKAN Discuss list: </p> <p> Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN <a class="missing changeset" title="No default repository defined">[1]</a>, as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field. </p> http://localhost/ticket/1589 http://localhost/ticket/1589 Report amercader <p> We need a way to track changes on the registry (datasets edited or updated), globally and on a per country/publisher/etc. basis. RSS and Atom feeds are really popular, and after closing <a class="closed ticket" href="http://localhost/ticket/191" title="enhancement: Searching by modification date (closed: fixed)">#191</a> and <a class="closed ticket" href="http://localhost/ticket/1498" title="enhancement: Support multiple SOLR schema versions (closed: fixed)">#1498</a> crating them from the search results should be fairly easy. </p> <p> The following URLs are pretty self-explanatory: </p> <pre class="wiki">http://localhost:5000/feed/registry.rss http://localhost:5000/feed/country/AF.rss http://localhost:5000/feed/publisher/worldbank.rss http://localhost:5000/feed/organisation_type/10.rss </pre><p> As we need to implement custom wrappers for countries, publishers, etc, we might as well offer a fully customizable feed, e.g.: </p> <pre class="wiki">http://localhost:5000/feed/custom.rss?q=activity_count:[* TO 100] http://localhost:5000/feed/custom.rss?publishertype=primary_source </pre><p> Apart from the actual feeds, there will be a small amount of work at the template level to add the links to the suitable pages (and maybe a generic page showing all available feeds) </p> http://localhost/ticket/1590 http://localhost/ticket/1590 Report dread <p> CKAN 1.5 installations need to be upgrade to 1.5.1. </p> <p> Working methodically through the CKANs on <a class="ext-link" href="http://datacatalogs.org/group/ckan"><span class="icon">​</span>http://datacatalogs.org/group/ckan</a> : </p> <p> At 1.5: </p> <ul><li>ckan-brazil <a class="ext-link" href="http://br.ckan.net/"><span class="icon">​</span>http://br.ckan.net/</a> s057 - DONE </li><li>ckan_ireland <a class="ext-link" href="http://ie.ckan.net/"><span class="icon">​</span>http://ie.ckan.net/</a> s057 - DONE </li><li>cz-ckan-net <a class="ext-link" href="http://cz.ckan.net/"><span class="icon">​</span>http://cz.ckan.net/</a> s057 - DONE </li><li>datakilder-no <a class="ext-link" href="http://no.ckan.net/"><span class="icon">​</span>http://no.ckan.net/</a> s057 - DONE </li><li>data_gov_uk <a class="ext-link" href="http://catalogue.data.gov.uk/"><span class="icon">​</span>http://catalogue.data.gov.uk/</a> </li><li>datacatalogs-org <a class="ext-link" href="http://datacatalogs.org/"><span class="icon">​</span>http://datacatalogs.org/</a> s054 - DONE </li></ul><p> Other versions: </p> <ul><li>berlin-open-data <a class="ext-link" href="http://daten.berlin.de/"><span class="icon">​</span>http://daten.berlin.de/</a> CKAN hidden </li><li>ckan-italia <a class="ext-link" href="http://it.ckan.net/"><span class="icon">​</span>http://it.ckan.net/</a> 1.3.3b </li><li>ckan-poland <a class="ext-link" href="http://pl.ckan.net/"><span class="icon">​</span>http://pl.ckan.net/</a> Down </li><li>data-norge-no <a class="ext-link" href="http://data.norge.no/"><span class="icon">​</span>http://data.norge.no/</a> CKAN hidden </li><li>datadotgc-ca <a class="ext-link" href="http://datadotgc.ca/"><span class="icon">​</span>http://datadotgc.ca/</a> <a class="ext-link" href="http://ca.ckan.net/"><span class="icon">​</span>http://ca.ckan.net/</a> 1.4.2a </li><li>datagm <a class="ext-link" href="http://www.datagm.org.uk/"><span class="icon">​</span>http://www.datagm.org.uk/</a> 1.3.3 </li><li>datagov-ru <a class="ext-link" href="http://datagov.ru/"><span class="icon">​</span>http://datagov.ru/</a> CKAN hidden </li><li>datospublicos-org <a class="ext-link" href="http://datospublicos.org/"><span class="icon">​</span>http://datospublicos.org/</a> 1.4.2 </li><li>iati-registry <a class="ext-link" href="http://iatiregistry.org/"><span class="icon">​</span>http://iatiregistry.org/</a> 1.5.1c </li><li>katalog-offene-daten-oesterreich <a class="ext-link" href="http://offener.datenkatalog.at/"><span class="icon">​</span>http://offener.datenkatalog.at/</a> 1.3.3b </li><li>nosdonnees_fr <a class="ext-link" href="http://www.nosdonnees.fr/"><span class="icon">​</span>http://www.nosdonnees.fr/</a> 1.3.1a </li><li>opengov-fi <a class="ext-link" href="http://opengov.fi/"><span class="icon">​</span>http://opengov.fi/</a> CKAN hidden </li><li>opengov_es <a class="ext-link" href="http://opengov.es/"><span class="icon">​</span>http://opengov.es/</a> 1.2 </li><li>publicdata-eu <a class="ext-link" href="http://publicdata.eu/"><span class="icon">​</span>http://publicdata.eu/</a> 1.4.2a </li><li>the-data-hub <a class="ext-link" href="http://thedatahub.org/"><span class="icon">​</span>http://thedatahub.org/</a> 1.5.1 </li><li>helsinki_finland <a class="ext-link" href="http://hri.fi/"><span class="icon">​</span>http://hri.fi/</a> CKAN hidden </li></ul><p> Now deleted from datacatalogs.org: </p> <ul><li>ckan-belgium <a class="ext-link" href="http://be.ckan.net/"><span class="icon">​</span>http://be.ckan.net/</a> Migrated </li><li>si-ckan-net <a class="ext-link" href="http://si.ckan.net/"><span class="icon">​</span>http://si.ckan.net/</a> migrated </li><li>gr-ckan-net <a class="ext-link" href="http://gr.ckan.net/"><span class="icon">​</span>http://gr.ckan.net/</a> Migrated </li><li>hu-ckan-net <a class="ext-link" href="http://hu.ckan.net/"><span class="icon">​</span>http://hu.ckan.net/</a> Migrated </li><li>ckan-lithuania <a class="ext-link" href="http://lt.ckan.net/"><span class="icon">​</span>http://lt.ckan.net/</a> Migrated </li></ul> http://localhost/ticket/1591 http://localhost/ticket/1591 Report amercader <p> The dict returned by package_dictize does not include metadata_modified and metadata_created. These are really useful properties, so it's worth having them on the standard package dict representation, which is used in several places, like at the template level. </p> http://localhost/ticket/1592 http://localhost/ticket/1592 Report amercader <p> Recent changes (see <a class="closed ticket" href="http://localhost/ticket/191" title="enhancement: Searching by modification date (closed: fixed)">#191</a> and <a class="closed ticket" href="http://localhost/ticket/1498" title="enhancement: Support multiple SOLR schema versions (closed: fixed)">#1498</a>) have made really easy to get search results ordered chronologically, and creating RSS or Atom feeds from them is relatively easy. </p> <p> Feeds are extremely popular and there are several tools for consuming them, so it would be a great way to allow users to follow datasets related with their topics of interest. </p> <p> Apart from the general feed with changes on the whole instance, other different "pre-set" feed endpoints could be implemented. The most obvious choices for custom feeds could be groups and tags: </p> <pre class="wiki">http://&lt;ckan_server&gt;/feed/group/{id}.{format:rss|atom} http://&lt;ckan_server&gt;/feed/tag/{id}.{format:rss|atom} </pre><p> These could be shown on the group/tag page, both on the &lt;head&gt; section, so browsers can autodiscover them, and on the UI, with a message like "Subscribe to this group/tag". </p> <p> Another powerful option is to have a custom feed builder that accepts the same parameters as the search page, so users can subscribe to whatever query they are interested on </p> <pre class="wiki">http://&lt;ckan_server&gt;/feed/custom.{format:rss|atom}?res_format=api%2Fsparql http://&lt;ckan_server&gt;/feed/custom.{format:rss|atom}?q=tags:animal* </pre><p> On this case a message "Subscribe to these search results" would appear. </p> <p> Note that a lot of work on this front needs to be done for the IATI registry project (see <a class="closed ticket" href="http://localhost/ticket/1590" title="enhancement: Create customized feeds for the IATI Registry (closed: fixed)">#1590</a>) </p> <p> Other tickets related: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1592" title="enhancement: Add metadata_modified and metadata_created to package_dictize output (closed: fixed)">#1592</a>: metadata_modified and metadata_created are not included in the search results, so an extra query for each package is needed right now. </li></ul> http://localhost/ticket/1593 http://localhost/ticket/1593 Report johnglover <p> Now that QA has been updated for Celery and CKAN 1.5.1, there are a number of improvements that we should probably implement. </p> <ul><li>Integrate QA into frontend - 1d - <a class="closed ticket" href="http://localhost/ticket/1583" title="enhancement: Integrate QA into frontend - 1d (closed: fixed)">#1583</a> </li><li>QA report improvements - 2.5d - <a class="new ticket" href="http://localhost/ticket/1584" title="enhancement: QA report improvements - 2.5d (new)">#1584</a> </li><li>QA - Check license as part of 5 star rating - <a class="closed ticket" href="http://localhost/ticket/1587" title="enhancement: QA - Check license as part of 5 star rating (closed: fixed)">#1587</a> </li><li>QA - Give SPARQL endpoints a 4 star rating - <a class="new ticket" href="http://localhost/ticket/1588" title="enhancement: QA - Give SPARQL endpoints a 4 star rating (new)">#1588</a> </li><li>QA - Give 5 star rating to datasets with link metadata - <a class="new ticket" href="http://localhost/ticket/1589" title="enhancement: QA - Give 5 star rating to datasets with link metadata (new)">#1589</a> </li></ul><p> Main pad: <a class="ext-link" href="http://ckan.okfnpad.org/qa"><span class="icon">​</span>http://ckan.okfnpad.org/qa</a> </p> <p> Some QA discussion (relating to Celery in particular) at <a class="ext-link" href="http://ckan.okfnpad.org/queue"><span class="icon">​</span>http://ckan.okfnpad.org/queue</a> </p> http://localhost/ticket/1594 http://localhost/ticket/1594 Report rgrp <p> Or at least it appears that way from revision log which displays all Group datasets against the changelog, see <a class="ext-link" href="http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217"><span class="icon">​</span>http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217</a> -- not apparent from that page but see my user page: <a class="ext-link" href="http://thedatahub.org/user/rufuspollock"><span class="icon">​</span>http://thedatahub.org/user/rufuspollock</a> </p> <p> This may turn out to be a UX bug rather than a real bug ... </p> <p> Assigning to kindly for review and comment. </p> http://localhost/ticket/1595 http://localhost/ticket/1595 Report dread <p> Suggestions from rgrp: </p> <ul><li>Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones. </li><li>Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system. </li><li>Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages) </li></ul><p> (This ticket is split off from <a class="closed ticket" href="http://localhost/ticket/1065" title="enhancement: [super] Change Authorization System (closed: fixed)">#1065</a>) </p> http://localhost/ticket/1596 http://localhost/ticket/1596 Report dread <p> I want to browse tags, but filtered for a particular group. Currently our tag API doesn't allow for filtering by group. </p> <p> This is important for improving groups as communities within a site <a class="closed ticket" href="http://localhost/ticket/1521" title="enhancement: [super] Groups UI and UX improvements in v1.6 from Cygri (closed: fixed)">#1521</a>. It would be easy to do this by adding an option to filter by a group. BUT are there any other use cases that would warrant a more complete faceted tag search? </p> <p> -- </p> <p> BTW I can currently draw a tag cloud for a group - I can get the top tags used in a group like this: </p> <p> curl <a class="ext-link" href="http://thedatahub.org/api/action/package_search"><span class="icon">​</span>http://thedatahub.org/api/action/package_search</a> -d '{"q":"groups:country-ca", "facet.field":"tags", "rows":"0"}' </p> <p> but it only contains the top 20 tags. </p> http://localhost/ticket/1597 http://localhost/ticket/1597 Report rgrp <p> Ratings were disabled approximately a year ago because: </p> <ul><li>Unclear purpose and UX. What did ratings tell you? How useful were they? </li><li>Spamming (esp by bots: you could submit an anonymous rating via a GET request which caused problems) </li></ul><p> Both problems are solvable and it would be nice to have this feature reinstated. </p> <ul><li>Purpose: can make this more purposable by limiting to logged in users (or at least distinguishing logged in from non-logged in users) <ul><li>Even better we could allow ratings to be made public (I'm interested in what someone else I respect finds important) </li></ul></li><li>Spamming: limit to logged in users and / or use AJAX over an API to submit ... </li></ul> http://localhost/ticket/1598 http://localhost/ticket/1598 Report rgrp <p> Simple embeddable widget for use on 3rd party sites showing dataset counts for a given search query. Have a specific version just for groups. </p> <ul><li>Simple group count widget in JS for embedding in wordpress and elsewhere <ul><li>Requested by several people (e.g Guo Xu from Econ working group). Already have something like this in CKAN JS for doing an embeddable search box. </li></ul></li><li>All you need to do is do a dataset query over the API e.g. <a class="ext-link" href="http://thedatahub.org/api/search/dataset?groups=economics"><span class="icon">​</span>http://thedatahub.org/api/search/dataset?groups=economics</a> and then embed in some html! </li></ul><p> Estimate: 30m (for someone who knows their jquery). </p> http://localhost/ticket/1599 http://localhost/ticket/1599 Report rgrp <p> Provide help to users in the WUI, with this linking to a detailed user guide. </p> <ul><li>User guide (suggest at help.thedatahub.org (or possibly help.ckan.org) </li><li>Snippets in the WUI with links to full user guide. <ul><li>Provide helper methods for creating these links with config options for setting path to user guide so that the link location can vary on install by install basis </li></ul></li><li>General info about publishing/working with/etc data (where we don't duplicate elsewhere) </li></ul><p> Est: (?) 5-10d </p> <h2 id="Topics">Topics</h2> <p> See <a class="ext-link" href="http://ckan.okfnpad.org/datahub-help"><span class="icon">​</span>http://ckan.okfnpad.org/datahub-help</a> </p> <ul><li>Step-by-step for adding a dataset including tips <ul><li>Reuse: <a class="ext-link" href="http://datapatterns.org/datahub.html"><span class="icon">​</span>http://datapatterns.org/datahub.html</a> </li><li>Quick overview of relevant parts of domain model (dataset and resource) </li></ul></li><li>API (perhaps link to main docs) </li><li>Command line interface and dpm </li><li>Using groups </li></ul><h2 id="Whydothis">Why do this</h2> <p> We already have wiki.ckan.org and docs.ckan.org. Why do this? </p> <p> Answer for docs: </p> <ul><li>This is administrative guide and focused on general CKAN software. </li><li>Hard to provide completely generic help. </li><li>Integratable with the <a class="missing wiki">DataHub?</a> site </li></ul><p> Answer for wiki: </p> <ul><li>No spam </li><li>General feeling that wikis are a little bit loose and untidy. We want this to be smart, clean and useful. </li><li>Integratable with the <a class="missing wiki">DataHub?</a> </li></ul> http://localhost/ticket/1600 http://localhost/ticket/1600 Report rgrp <p> Need thoughts from Adria and James. RP thoughts </p> <ul><li>Integration into WUI search: <a class="closed ticket" href="http://localhost/ticket/1469" title="enhancement: Geo-spatial search on the front end (closed: fixed)">#1469</a> </li><li>JS Editor for adding/editing spatial attribute: <a class="new ticket" href="http://localhost/ticket/1818" title="enhancement: Spatial metadata editor (new)">#1818</a> </li><li>Theme/template hook for adding the map e.g. on dataset page (or elsewhere) </li><li>Clear Documentation </li><li>Deployment </li><li>Announce post </li><li>(?) Map page for stats section showing location of datasets </li></ul><p> Extras (probably separate): </p> <ul><li>csw support ... </li></ul> http://localhost/ticket/1601 http://localhost/ticket/1601 Report rgrp <p> We already have first pass of Data Preview that was done as part of <a class="closed ticket" href="http://localhost/ticket/1357" title="enhancement: Integrate new version of data preview (closed: fixed)">#1357</a>. This is about v2 of this. Two main components: </p> <ul><li>A new version of the separate Data Explorer library (now called Recline Data Explorer) - <a class="closed ticket" href="http://localhost/ticket/1446" title="enhancement: Data Explorer v2 (closed: fixed)">#1446</a> <strong>Est: 10-15d</strong> <ul><li>Why: <ul><li>Cleaner, backbone based solution. </li><li>Provides editing support if wanted. </li><li>Collaboration with Max Ogden </li></ul></li></ul></li><li>Integrate this into CKAN (into resource view page) - see <a class="closed ticket" href="http://localhost/ticket/1630" title="enhancement: Integrate Data Explorer v2 into Data Viewer (closed: fixed)">#1630</a> <strong>Est: 1d</strong> </li><li>Improvements to Data Viewer / Previewer system in CKAN itself - <strong>Est: 2-3d</strong> <ul><li><del>(Update Data Preview-er (View-er) to support images ...)</del> </li><li><del>(WONTFIX here - (?) Update Data Preview-er to support geodata - <a class="closed ticket" href="http://localhost/ticket/1151" title="enhancement: Preview for geographic data should be a map (closed: wontfix)">#1151</a>)</del> </li></ul></li></ul><p> </p> http://localhost/ticket/1602 http://localhost/ticket/1602 Report rgrp <p> Super ticket: <a class="new ticket" href="http://localhost/ticket/1745" title="enhancement: Dataset search UX improvements as of Jan 2012 (new)">#1745</a> </p> <p> Ability to build up search query using a nice javascript-y interface. </p> <ul><li>Add facets by selecting attribute and adding -&gt; search facet options in dropdown -&gt; added to search (with 'x' to remove -- as we currently do). <ul><li>(a bit like the data.hri.fi) </li></ul></li><li>Some improvements to css <ul><li>See <a class="ext-link" href="http://eutr.ep.io/"><span class="icon">​</span>http://eutr.ep.io/</a> </li></ul></li><li>Improvements to faceting <ul><li>Ability to configure faceting and number of items to show (?) </li></ul></li><li>Pure JS search implementation to make it easy to reuse across site </li></ul> http://localhost/ticket/1603 http://localhost/ticket/1603 Report dread <p> ckanext-moderatededits requires an old and possibly development version of CKAN. It would be good to update it for later CKAN versions. </p> <p> According to the README, you need CKAN from branch feature-1141-moderated-edits-ajax but the changelog suggests this branch went into version 1.4.2. So it possibly works with 1.4.2 and 1.4.3(.1). But CKAN 1.5 has revamped templates, so the genshi stream filters definitely don't work. </p> <p> BTW history_ajax/read_ajax calls have been deprecated in CKAN since 1.5.2a and will need fixing up to use the Action API too as part of this. </p> http://localhost/ticket/1604 http://localhost/ticket/1604 Report rgrp <p> Multi-language: </p> <ul><li>dataset and resource metadata (and other objects such as groups?) <ul><li>Options for implementation in <a class="ext-link" href="http://ckan.okfnpad.org/multilingual"><span class="icon">​</span>http://ckan.okfnpad.org/multilingual</a> </li></ul></li><li>Field values in taxonomy (e.g. country names - Eurovoc) <ul><li><a class="closed ticket" href="http://localhost/ticket/1665" title="task: Begin doing research into eurovoc (closed: fixed)">#1665</a> Research into Eurovoc </li><li>Display Taxonomy in different languages </li></ul></li><li>Field values not in taxonomy (e.g. title &amp; description) <ul><li>Use extra fields e.g. _i18n_title_fr = Le data. </li><li>(If lots of fields would need translating then would consider having a new package for each language, linked together with <a class="missing wiki">PackageRelationships?</a>. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.) </li></ul></li><li>EC extension (templates, form) <ul><li>Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions? </li></ul></li></ul> http://localhost/ticket/1605 http://localhost/ticket/1605 Report dread <p> Add a config option to choose the metadata licence. Set default to Open Database License. </p> <p> Currently the dataset edit form says "Important: By submitting content, you agree to release your contributions under the Open Database License." This is hard-coded, but not suitable for when DGU uses the CKAN form - they use the OGL. </p> http://localhost/ticket/1606 http://localhost/ticket/1606 Report dread <p> A ticket to collect all of the changes needed for CKAN to become the 'data' tab on DGU. </p> <p> Design doc: <a class="ext-link" href="https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB"><span class="icon">​</span>https://docs.google.com/document/d/19h9bA1G4cQkv031m8jNCu6FEB3a8qpXUmdPAguM-Ofs/edit?hl=en_GB</a> The design doc is the authoritative source of tasks. This ticket acts as a synopsis. </p> <p> Current sprint (2012-02-20): </p> <ul><li>7a. <del>Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1</del> </li><li>6. <del>Publisher Registration Improvements MUST PHASE 1</del> </li><li>3. <del><a class="missing wiki">Themes/Taxonomy?</a> as well as tags WONT PHASE 2</del> </li><li>32. <del>Browse by Tags PHASE 1 MUST</del> </li><li>12. <del><a class="missing wiki">Edit/Delete?</a> for Harvested Datasets MUST URGENT PHASE 1</del> </li><li>13. <del>Provider labelling MUST PHASE 1 [UKLP <a class="closed ticket" href="http://localhost/ticket/14" title="enhancement: Update a release (closed: invalid)">#14</a> MUST]</del> </li><li>17. <del>Filter “UK Location Records” MUST PHASE 1 [UKLP <a class="closed ticket" href="http://localhost/ticket/21" title="enhancement: Undelete a package (closed: fixed)">#21</a> SHOULD]</del> </li><li>22. <del>Mixed Licenses MUST PHASE 2</del> </li></ul><p> Plus, brought forward from last sprint: </p> <ul><li><del>Publisher hierarchy support</del> </li><li><del>Improved dataset creation tools (form) - <a class="closed ticket" href="http://localhost/ticket/1525" title="enhancement: [super] DGU package form refactor (closed: fixed)">#1525</a></del> </li><li><del>Edit/delete for harvested datasets</del> </li></ul><p> Overview of tasks: see the design doc. </p> http://localhost/ticket/1607 http://localhost/ticket/1607 Report ross <p> As part of <a class="closed ticket" href="http://localhost/ticket/1608#1574" title="enhancement: Merge storage into Core (closed: fixed)">Storage Changes</a> and <a class="ext-link" href="http://ckan.okfnpad.org/newstorage"><span class="icon">​</span>http://ckan.okfnpad.org/newstorage</a> it was decided to merge ckanext-storage back into core. </p> <p> Other changes: </p> <ul><li>Remove /storage/upload page (we have frontend integration elsewhere now and we should also ensure that upload items relate to a resource or other domain object). </li><li>Provide a local path as well as remote path for form upload requests in API so that frontend can see url to CKAN path rather than cloud storage path ... </li></ul> http://localhost/ticket/1608 http://localhost/ticket/1608 Report ross <p> From super Storage changes - <a class="closed ticket" href="http://localhost/ticket/1574" title="enhancement: [super] Storage changes including automated save to Webstore (closed: fixed)">#1574</a> - and <a class="ext-link" href="http://ckan.okfnpad.org/newstorage"><span class="icon">​</span>http://ckan.okfnpad.org/newstorage</a> we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore </p> <h2 id="Analysis">Analysis</h2> <p> When I upload a file to CKAN: </p> <ul><li>End up with file in permanent storage </li><li>IF file is ot type ... csv,xls,xlsx,sqlite,.sql <ul><li>End up with new db in webstore <ul><li>Where? {username}/{resource-id}/... <ul><li>If a single table: name it after the file name (appropriately slugified) </li></ul></li><li>A resource *always* corresponds to a 'database' in webstore ... </li><li>In Data Explorer have "Sheets" tab ... </li></ul></li></ul></li><li>Resource url = /dataset/{x}/resource/{y}/link -&gt; cached_url ... </li></ul> http://localhost/ticket/1609 http://localhost/ticket/1609 Report ross <p> The default backing store for webstore should be Postgres and not sqlite. This was agreed as part of the <a class="closed ticket" href="http://localhost/ticket/1574" title="enhancement: [super] Storage changes including automated save to Webstore (closed: fixed)">#1574</a> storage changes and on <a class="ext-link" href="http://ckan.okfnpad.org/newstorage"><span class="icon">​</span>http://ckan.okfnpad.org/newstorage</a> </p> http://localhost/ticket/1610 http://localhost/ticket/1610 Report ross <p> As part of <a class="closed ticket" href="http://localhost/ticket/1574" title="enhancement: [super] Storage changes including automated save to Webstore (closed: fixed)">#1574</a> we decided that it would be better for webstore (and future external services) to be able to authenticate simply with CKAN-Core. </p> <p> Currently webstore access the CKAN database to obtain the key for the user but it would be better if this connection was not so tightly bound and that webstore used an API as any other external service might. </p> <p> Need to discuss further with dread </p> http://localhost/ticket/1611 http://localhost/ticket/1611 Report kindly <p> Group show that lists packages is slow due to not using query in pagination. </p> http://localhost/ticket/1612 http://localhost/ticket/1612 Report dread <ol><li>Viewing a dataset, hit 'Edit' </li><li>Click on the "Basic Information" tab (note: URL has suffix #section-basic-information </li><li>Click 'Save' </li><li>URL still has suffix #section-basic-information </li></ol><p> Affects 1.5, 1.5.1, 1.5.2a </p> http://localhost/ticket/1613 http://localhost/ticket/1613 Report kindly <p> Its a pain to see the difference between branches as there are normally a lot of po file transaction. Make the default be see that they have changed without actually show the diffs themselves. </p> http://localhost/ticket/1614 http://localhost/ticket/1614 Report thejimmyg <p> This would allow deployment via Nginx or Apache using proxy to Paster, uWSGI. At the moment CKAN isn't aware of the proxy's IP address so when you perform an action which does a redirect (such as adding a package), CKAN redirects you to the *internal IP* not the external *proxy IP*. </p> <p> We would like this work to facilitate testing within VMs as part of our new build infrastructure. </p> <p> It would also be nice if CKAN worked when mounted at a path other than /. That could be dealt with in another ticket because it isn't a problem at the moment. </p> http://localhost/ticket/1615 http://localhost/ticket/1615 Report amercader <p> Right now if an exception is found while reindexing, the whole process stops and the remaining datasets are left out of the index. The process should continue after logging the exception. If more than a certain number of exceptions occur in a row, the process should stop. </p> http://localhost/ticket/1616 http://localhost/ticket/1616 Report icmurray <p> [Child of <a class="closed ticket" href="http://localhost/ticket/1526" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1526</a>] </p> <p> Contact/FOI information is derived from <a class="missing wiki">Publisher/User?</a>, but can be altered for each dataset. </p> <blockquote> <p> Original estimate: 4 days </p> <blockquote> <p> Time spent: 0 days </p> </blockquote> <p> Time remaining: 4 days </p> </blockquote> http://localhost/ticket/1617 http://localhost/ticket/1617 Report icmurray <p> [Child of <a class="closed ticket" href="http://localhost/ticket/1526" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1526</a>] </p> <p> Provide the interface for the themes and subthemes hierarchy. </p> http://localhost/ticket/1618 http://localhost/ticket/1618 Report icmurray <p> [Child of <a class="closed ticket" href="http://localhost/ticket/1526" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1526</a>] </p> <p> Create the interface used to retrieve publisher and authz information from the publisher hierarchy. </p> http://localhost/ticket/1619 http://localhost/ticket/1619 Report icmurray <p> [Child of <a class="closed ticket" href="http://localhost/ticket/1526" title="enhancement: [super] Update and test existing DGU package form (closed: fixed)">#1526</a>] </p> <p> Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa. </p> http://localhost/ticket/1620 http://localhost/ticket/1620 Report seanh <p> A test case is currently failing for me on master: </p> <pre class="wiki">ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py", line 342, in test_user_create_unicode res = res.follow() File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 603, in follow return self.test_app.get(location, **kw) File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 208, in get return self.do_request(req, status=status) File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 389, in do_request **req.environ) File "/usr/lib/pymodules/python2.6/paste/wsgilib.py", line 343, in raw_interactive app_iter = application(basic_environ, start_response) File "/usr/lib/pymodules/python2.6/paste/lint.py", line 170, in lint_app iterator = application(environ, start_response_wrapper) File "/usr/lib/pymodules/python2.6/paste/cascade.py", line 130, in __call__ return self.apps[-1](environ, start_response) File "/usr/lib/pymodules/python2.6/paste/registry.py", line 350, in __call__ app_iter = self.application(environ, start_response) File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 69, in __call__ auth_ids = self.authenticate(environ, classification, ids) File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 201, in authenticate userid = plugin.authenticate(environ, identity) File "/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py", line 29, in authenticate if user.validate_password(identity.get('password')): File "/home/seanh/pyenv/src/ckan/ckan/model/user.py", line 113, in validate_password hashed_pass = sha1(password_8bit + self.password[:40]) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128) </pre> http://localhost/ticket/1621 http://localhost/ticket/1621 Report johnglover <ul><li>Update CKAN on DGU UAT to 1.5.1 </li><li>Deploy Celery </li><li>Deploy QA extension </li></ul> http://localhost/ticket/1622 http://localhost/ticket/1622 Report dread <p> The database dump currently contains all Packages and their associated objects, even those that have been set to state=deleted. We should exclude these from the dump now. </p> <p> Dump = paster db simple-dump-csv/json </p> <h2 id="reasoning">reasoning</h2> <p> The dumps are designed for end-users to be able to run scripts on the mass of data. Since end-users don't see state=deleted packages then they shouldn't need them in the dump. In fact their presence in the dump probably confuses them. </p> <p> Admins get the full database anyway in the backup pg_dump. </p> <p> We only included them in the user dump because it was designed before use of state=deleted was established. </p> <p> Time estimate: 2h </p> http://localhost/ticket/1623 http://localhost/ticket/1623 Report pudo <p> Futher Information -&gt; Further Information </p> http://localhost/ticket/1624 http://localhost/ticket/1624 Report seanh <p> This requires adding a logic function for emitting an activity stream event, and then editing the logic functions for creating or updating users and making them call the new emit event function. This same emit event function can later be used to emit activity stream events for other types of object as well. </p> http://localhost/ticket/1625 http://localhost/ticket/1625 Report dread <p> thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data. </p> http://localhost/ticket/1626 http://localhost/ticket/1626 Report dread <p> On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab. </p> <p> Original ticket <a class="closed ticket" href="http://localhost/ticket/48" title="enhancement: Create favicon (closed: fixed)">#48</a> </p> http://localhost/ticket/1627 http://localhost/ticket/1627 Report dread <p> johnglover said: I can confirm that even with the mapping fix, the ckanext-dgu dgu_form plugin template does not work properly with 1.5.1, so should probably not be installed at present. The edit page is ugly but should work (eg: <a class="ext-link" href="http://dgu-os.okfn.org/dataset/edit/abandoned-vehicles"><span class="icon">​</span>http://dgu-os.okfn.org/dataset/edit/abandoned-vehicles</a>), but the 'add a dataset' page is broken (eg: <a class="ext-link" href="http://dgu-os.okfn.org/dataset/new"><span class="icon">​</span>http://dgu-os.okfn.org/dataset/new</a>) </p> http://localhost/ticket/1628 http://localhost/ticket/1628 Report dread <p> This was seen on datacatalogs. When we upgraded it 1.5 to 1.5.1 we saw some permissions being reset so that it could be spammed. Anonymous and logged in users were given anon_editor and editor permissions. I don't know what the previous config was. Permissions for sysadmins remain unaffected. </p> http://localhost/ticket/1629 http://localhost/ticket/1629 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1602" title="enhancement: [super] Data Previewer / Viewer v2 (closed: fixed)">#1602</a> </p> <p> Steps: </p> <ul><li>Include (or git submodule) data explorer into vendor </li><li>Update existing js integration code </li><li>Fix css and any bugs </li></ul><p> Est: 1d </p> <p> Actual: 1d </p> http://localhost/ticket/1630 http://localhost/ticket/1630 Report seanh http://localhost/ticket/1631 http://localhost/ticket/1631 Report seanh <p> Add logic functions for getting the activity stream for a group in JSON and rendered HTML formats, add HTML activity stream into group page. </p> http://localhost/ticket/1632 http://localhost/ticket/1632 Report seanh <p> Add logic functions for getting the activity stream for a dataset in JSON and rendered HTML formats, add HTML activity stream into dataset page. </p> http://localhost/ticket/1633 http://localhost/ticket/1633 Report seanh <p> This is a fairly big feature to add. Analysis: <a class="ext-link" href="http://ckan.okfnpad.org/27"><span class="icon">​</span>http://ckan.okfnpad.org/27</a> </p> <p> This ticket is related to <a class="new ticket" href="http://localhost/ticket/1635" title="enhancement: Email notifications (e.g. for activity streams) (new)">#1635</a> (email notifications for activity streams) which is about managing a list of activity streams for which the user receives email notifications. There are decisions to be made about how the two will work together. </p> http://localhost/ticket/1634 http://localhost/ticket/1634 Report seanh <p> CKAN should be able to send email notifications to users. </p> <p> Maybe have a notifications table in the db, and a server-side job that runs periodically and consumes rows from this table, mailing them to the users. </p> <p> One thing that we may want to send users notifications of is activity stream events. So the activity streams code would have to add rows to the notifications table for the mailer job to consume. But remember that email notifications feature is separate from activity streams - we may want to send notifications of other things as well. </p> <p> Need to implement (at least some of) <a class="closed ticket" href="http://localhost/ticket/1634" title="enhancement: Allow users to follow/unfollow activity streams of other users, datasets ... (closed: duplicate)">#1634</a> before this can be implemented, in order to have something to send notifications about. </p> <p> Analysis here: <a class="ext-link" href="http://ckan.okfnpad.org/27"><span class="icon">​</span>http://ckan.okfnpad.org/27</a> </p> http://localhost/ticket/1635 http://localhost/ticket/1635 Report seanh http://localhost/ticket/1636 http://localhost/ticket/1636 Report seanh <p> (and add test cases for it) </p> http://localhost/ticket/1637 http://localhost/ticket/1637 Report seanh <p> Dump and load JSON explicitly instead. </p> http://localhost/ticket/1638 http://localhost/ticket/1638 Report seanh <p> The HTML and CSS for the activity streams templates needs to be tidied up. Currently the CSS is inlined in the templates themselves, and there is much unnecessary repetition of CSS. </p> <p> Perhaps put this one in hold until there has been some design discussion about how the activity streams and the user, dataset and group pages with activity streams should look? </p> http://localhost/ticket/1639 http://localhost/ticket/1639 Report dread <p> Set-up publicdata.eu to harvest datasets at rs.ckan.net (Serbian community CKAN). </p> http://localhost/ticket/1640 http://localhost/ticket/1640 Report amercader <p> The download task in ckanext-archiver performs a HEAD request on the resource URL and checks if the "Content-Type" and "Content-Length" headers differ from the values stored to see if the resource needs to be updated <a class="missing changeset" title="No default repository defined">[1]</a>. </p> <p> The "Content-Length" header, although widely used, is not mandatory and some servers don't provide it, e.g.: </p> <pre class="wiki">$ curl -I http://portfolio.theglobalfund.org/en/IATI/Activities?countryCode=AFG HTTP/1.1 200 OK Cache-Control: private Transfer-Encoding: chunked Content-Type: text/xml Vary: Accept-Encoding Server: Microsoft-IIS/7.5 Set-Cookie: ASP.NET_SessionId=3qhqekddgmre0kmk5cynq0sy; path=/; HttpOnly X-AspNetMvc-Version: 3.0 content-disposition: attachment; filename=AFG_IATI_12012012.xml X-AspNet-Version: 4.0.30319 X-Powered-By: ASP.NET Date: Thu, 12 Jan 2012 12:36:43 GMT </pre><p> Also worth noting that <a class="ext-link" href="http://docs.python-requests.org/"><span class="icon">​</span>requests</a>, the python library that uses ckanext-archiver, sets an "Accept-Encoding: gzip" header by default, which depending on the configuration of the remote web server, may prevent the "Content-Length" server from being sent, e.g.: </p> <pre class="wiki">$ curl -H "Accept-Encoding: gzip" -I http://iatistandard.org/published-temp/adb-activities.xml HTTP/1.1 200 OK Date: Thu, 12 Jan 2012 12:12:46 GMT Server: Apache Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Type: application/xml curl -I http://iatistandard.org/published-temp/adb-activities.xml HTTP/1.1 200 OK Date: Thu, 12 Jan 2012 11:56:23 GMT Server: Apache Last-Modified: Mon, 28 Nov 2011 15:55:35 GMT Accept-Ranges: bytes Content-Length: 2686720 Vary: Accept-Encoding Content-Type: application/xml </pre><p> All this can lead to some resources never getting updated, and of course the size property of the resource not being set. </p> <p> As we need to download the resource anyway, it would be better to check if the real length of the data has been modified (and store it). </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72"><span class="icon">​</span>https://github.com/okfn/ckanext-archiver/blob/0a189262dca4ab5b286fb6a02b4ab8a201f639f3/ckanext/archiver/tasks.py#L72</a> </p> http://localhost/ticket/1641 http://localhost/ticket/1641 Report pudo <p> I had a package descriptions with URLs that contain "group:foo". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link. </p> <p> Need to fix the parser. </p> <p> Text: </p> <p> Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als <a class="missing wiki">WebMapTileService?</a> (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5). </p> <p> Metadateneinträge zu den Daten im PortalU: </p> <ul><li>[apps4d_DOP40](<a class="ext-link" href="http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&amp;plugid=/kug-group:kug-iplug-udk-db_hh"><span class="icon">​</span>http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&amp;plugid=/kug-group:kug-iplug-udk-db_hh</a>) </li><li>[apps4d_DISK](<a class="ext-link" href="http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&amp;plugid=/kug-group:kug-iplug-udk-db_hh"><span class="icon">​</span>http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&amp;plugid=/kug-group:kug-iplug-udk-db_hh</a>) </li><li>[apps4d_DIRK](<a class="ext-link" href="http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&amp;plugid=/kug-group:kug-iplug-udk-db_hh"><span class="icon">​</span>http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&amp;plugid=/kug-group:kug-iplug-udk-db_hh</a>) </li><li>[apps4d_DK5](<a class="ext-link" href="http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&amp;plugid=/kug-group:kug-iplug-udk-db_hh"><span class="icon">​</span>http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&amp;plugid=/kug-group:kug-iplug-udk-db_hh</a>) </li></ul><p> One fix is quoting the URLs </p> http://localhost/ticket/1642 http://localhost/ticket/1642 Report shevski <p> Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags </p> http://localhost/ticket/1643 http://localhost/ticket/1643 Report shevski <p> Instead of alphabetically as we do currently, alternatively by most viewed datasets </p> <p> for <a class="ext-link" href="http://thedatahub.org/dataset"><span class="icon">​</span>http://thedatahub.org/dataset</a> </p> http://localhost/ticket/1644 http://localhost/ticket/1644 Report icmurray <p> Theme the DGU form. </p> <p> Doesn't need to be an exact replica of DGU, but just enough to show it's possible. </p> http://localhost/ticket/1645 http://localhost/ticket/1645 Report dread <p> When viewing a dataset, the "Resources" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu. </p> <p> <a class="ext-link" href="http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f"><span class="icon">​</span>http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f</a> </p> <p> <a class="missing wiki">Chrome/Linux?</a> </p> http://localhost/ticket/1646 http://localhost/ticket/1646 Report shevski <p> In the footer as well as more clearly &amp; directly on the About page </p> http://localhost/ticket/1647 http://localhost/ticket/1647 Report shevski <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a> </p> <p> Need to decide which term to use and then have the same for editing as well as viewing a dataset. </p> <p> In creating/editing a dataset, want more explanation about adding extra fields (probably as a tooltip or similar).. i.e. that this let's you add extra custom metadata such as 'location: uk' which is then searchable etc </p> http://localhost/ticket/1648 http://localhost/ticket/1648 Report ross <ul><li>Make sure that the extension works as it should and isn't broken on current releases [3d] </li></ul><ul><li>Make it work with the EC schema, including extras [2d] </li></ul> http://localhost/ticket/1649 http://localhost/ticket/1649 Report ross <p> Datasets should have a field that specifies the language used in the dataset, this should be a two character ISO 639-1 code (see <a class="ext-link" href="http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes"><span class="icon">​</span>http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes</a> for list) . Also make sure all tests and migrations are resolved. [2d] </p> <p> Determine the best approach to having 'translation_of' references to show which dataset the current one is a translation of. This may be a translation of field, or a 'copied from' field or another form of package relationship. [2d] </p> http://localhost/ticket/1650 http://localhost/ticket/1650 Report dread <p> In this commit <a class="ext-link" href="https://github.com/okfn/ckan/commit/1772a5c"><span class="icon">​</span>https://github.com/okfn/ckan/commit/1772a5c</a> John Glover set map.explicit=True in ckan/config/routing.py. </p> <p> The reason this was done was to avoid links collecting parameters. e.g. if you were on page /dataset/{id}/resource/{resource_id} then by default all the links on that page generated by url_for (Routes) would include the id and resource_id parameters as well. To avoid this, you had to go through all the links and add id=None and resource_id=None to the url_for parameters. </p> <p> When map.explicit was changed to True, the value of the controller, action, id and any parameters were no longer automatically carried over into the generated links for the page. So previously links within the same controller didn't need to specify the controller (for example), but now they did. So when we did this we also had to fix up links that weren't explicit: </p> <ul><li><a class="ext-link" href="https://github.com/okfn/ckan/commit/88e88380f61965f58fb105095e76a78673ce3e2c"><span class="icon">​</span>https://github.com/okfn/ckan/commit/88e88380f61965f58fb105095e76a78673ce3e2c</a> </li><li><a class="closed ticket" href="http://localhost/ticket/1501" title="defect: Pagination links broken (closed: fixed)">#1501</a> Pagination links </li><li><a class="ext-link" href="https://github.com/okfn/ckan/commit/52d71102d9b07693ea240d4a66e8adb43e515414"><span class="icon">​</span>https://github.com/okfn/ckan/commit/52d71102d9b07693ea240d4a66e8adb43e515414</a> </li></ul><p> John made the config change on 5/11/2011 which was merged to master <a class="ext-link" href="https://github.com/okfn/ckan/commit/5a01e67"><span class="icon">​</span>https://github.com/okfn/ckan/commit/5a01e67</a> 21/11/2011. The related fixes mentioned were in within the same week. This all went into release 1.5.1. The requirement </p> http://localhost/ticket/1651 http://localhost/ticket/1651 Report ross <ul><li>Analysis of how/where we can integrate with Drupal Multilingual </li></ul><ul><li>Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/... </li></ul><ul><li>Check with Ian for other issues. </li></ul> http://localhost/ticket/1652 http://localhost/ticket/1652 Report ross <p> To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session. </p> <ul><li>Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs. </li></ul><ul><li>Write Middleware + update url_for to take account of the language. [2d] </li></ul><p> </p> <ul><li>Document the language setup, and how to replicate it. [1d] </li></ul> http://localhost/ticket/1653 http://localhost/ticket/1653 Report amercader <p> Tasks include: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1813" title="task: Upadate ckanext-pdeu to work with CKAN 1.6 (closed: fixed)">#1813</a> Update ckanext-pdeu (4d) </li><li><a class="closed ticket" href="http://localhost/ticket/1814" title="enhancement: Check publicadata.eu harvesters (closed: fixed)">#1814</a> Update harvesters (2d) </li><li><a class="closed ticket" href="http://localhost/ticket/1649" title="enhancement: Verify RDF Extension (ckanext-rdf) works (closed: fixed)">#1649</a> Verify ckanext-rdf works with latest CKAN (3d) </li><li><a class="new ticket" href="http://localhost/ticket/1815" title="task: Reenable Sparql endpoint on publicdata.eu (new)">#1815</a> Reenable Sparql endpoint (?) </li><li><a class="closed ticket" href="http://localhost/ticket/1816" title="enhancement: Update ckanext-apps to work with CKAN 1.5.1 - 1.6 (closed: fixed)">#1816</a> Update ckanext-apps (2-3d) </li></ul> http://localhost/ticket/1654 http://localhost/ticket/1654 Report amercader <p> Time estimate: 2d </p> <ul><li>Fix logs (apache, ckan, harvest): rotate, set suitable levels </li><li>Fix harvesting jobs: supervisord for gather consumer, cron job </li><li>Fix backups </li></ul><p> Also it may be worth setting up a test instance ( on s023 ?) </p> http://localhost/ticket/1655 http://localhost/ticket/1655 Report ross <p> Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME) </p> <ul><li>Analysis of the best solution [1d] </li></ul><ul><li>Implement reverse proxying in tandem with <a class="closed ticket" href="http://localhost/ticket/1653" title="enhancement: Change URLs for multilingual site (closed: fixed)">#1653</a> [2d] </li></ul><ul><li>Document and store the configuration files. [1d] </li></ul> http://localhost/ticket/1656 http://localhost/ticket/1656 Report ross <p> Provide configuration for reverse proxying that will correctly handle the mapping of a URL to a sub-folder (using X-SCRIPT-NAME) </p> <ul><li>Analysis of the best solution [1d] </li></ul><ul><li>Implement reverse proxying in tandem with <a class="closed ticket" href="http://localhost/ticket/1653" title="enhancement: Change URLs for multilingual site (closed: fixed)">#1653</a> [2d] </li></ul><ul><li>Document and store the configuration files. [1d] </li></ul> http://localhost/ticket/1657 http://localhost/ticket/1657 Report ross <p> Implement proxy caching for performance gains </p> <ul><li>Analysis of best solution and what we can deliver in a reasonable time [2d] </li></ul><ul><li>Checks for presence of cookie 'auth_tkt' and appropriate setup for media [2d] </li></ul><ul><li>Remove session requirements for user who isn't logged in [3d] </li></ul><p> </p> <ul><li>Document the configuration and setup so that it can be easily modified and replicated [1d] </li></ul> http://localhost/ticket/1658 http://localhost/ticket/1658 Report dread <p> If you set WSGIScriptAlias to mount CKAN at a URL other than / then you cannot logout without adjusting the OpenID logged_out_url to match in who.ini config. e.g. </p> <blockquote> <p> [plugin:openid] ... logged_out_url = /sub/dir/user/logged_out </p> </blockquote> <p> Note: all the other URLs in who.ini should not have the /sub/dir/ - it is just this one that doesn't take account of the mounting point. </p> <p> The solution is to fix-up the repoze.who OpenID plugin to take account of the mounting point. </p> http://localhost/ticket/1659 http://localhost/ticket/1659 Report lucychambers <p> This CSV resource used to preview but now the format appears to be unsupported: "We are unable to preview this type of resource: x-osdata-csv" </p> <p> <a class="ext-link" href="http://thedatahub.org/dataset/lbhf-spending-2010/resource/9661abbd-2816-4d58-8b20-3cb0eb770c69"><span class="icon">​</span>http://thedatahub.org/dataset/lbhf-spending-2010/resource/9661abbd-2816-4d58-8b20-3cb0eb770c69</a> </p> <p> This is used as an example by the <a class="missing wiki">OpenSpending?</a> team all the time. </p> http://localhost/ticket/1660 http://localhost/ticket/1660 Report dread <p> Jaakko Louhio reported that the wrong Routes version got installed during CKAN's package install. </p> <p> He is using Ubuntu 10.04 and he believes it install Routes 1.12.3 instead of what we use which is 1.11. </p> http://localhost/ticket/1661 http://localhost/ticket/1661 Report dread <p> Mounting CKAN at a non-root URL was made to work properly here: <a class="closed ticket" href="http://localhost/ticket/1659" title="defect: Cannot logout if CKAN mounted at non-root url (closed: fixed)">#1659</a> </p> <p> Unfortunately OpenID doesn't play nicely and would require some work to get working. </p> http://localhost/ticket/1662 http://localhost/ticket/1662 Report seanh <p> The create_activity() logic function already exists, but validation and authorization need to be added before it's hooked up to an API call. </p> http://localhost/ticket/1663 http://localhost/ticket/1663 Report seanh <p> We already have simple activity streams for groups that have new group or group updated events (see <a class="closed ticket" href="http://localhost/ticket/1631" title="enhancement: Add activity stream events for new/changed groups (closed: fixed)">#1631</a>). Smarter/more useful group activity streams would also contain the events for things related to the group: users, datasets, other groups, etc. </p> <p> This should just mean using a more complex SQL query in the group_activity_list() logic function. </p> <p> Things might get tricky when a user or dataset joins a group, this may show up as two events (the user or dataset was modified, and the group was modified). We may want to have two separate events for this (one to go in the user or dataset activity stream and one for the group), or maybe we just want one 'user/dataset joined group' event. </p> http://localhost/ticket/1664 http://localhost/ticket/1664 Report seanh <p> How big is it? How are we going to store it? etc. </p> http://localhost/ticket/1665 http://localhost/ticket/1665 Report seanh <p> ...and show them on the tag pages. </p> <p> This means adding tag_activity_list() and tag_activity_list_html() logic functions, adding the templating to render tags as HTML, and adding the tag HTML into the tag pages. </p> <p> What exactly should go into a tag's activity stream? As with datasets we may want a more complex SQL query, where events from objects related to the tag (datasets, users, etc.) appear in the tag's stream. </p> http://localhost/ticket/1666 http://localhost/ticket/1666 Report seanh <p> Currently rendering of activity stream events to HTML works by looking up a rendering function in a dictionary that maps activity types ('new package', 'changed group', etc.) to rendering functions that take an activity stream event and return the rendered HTML. </p> <p> There needs to be an extension point where extensions can register their own rendering functions for particular activity types. </p> http://localhost/ticket/1667 http://localhost/ticket/1667 Report dread <p> There's a discrepency in repoze.who versions between the source and package installs: </p> <ul><li>repoze.who - package 1.0.18 vs source 1.0.19 </li><li>repoze.who-friendlyform - package 1.0b3 vs source 1.0.8 </li></ul><p> We get a test failure <a class="missing changeset" title="No default repository defined">[1]</a> with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who. </p> <p> The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package. </p> <p> James suggests we 'do something horrible like dynamically patch repoze on CKAN import'. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio"><span class="icon">​</span>http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio</a> ERROR: ckan.tests.functional.test_user.<a class="missing wiki">TestUserController?</a>.test_user_create_unicode </p> http://localhost/ticket/1668 http://localhost/ticket/1668 Report ross <p> Implement the necessary backend changes to build profile for publishers using the group refactor. </p> <p> See <a class="ext-link" href="http://ckan.okfnpad.org/publisherprofiles"><span class="icon">​</span>http://ckan.okfnpad.org/publisherprofiles</a> </p> <p> Subtickets are at: </p> <ul><li>Publisher profile analysis <a class="closed ticket" href="http://localhost/ticket/1670" title="enhancement: Publisher profile documentation (closed: fixed)">#1670</a> [2d] </li><li>Publisher profile sysadmin authz <a class="closed ticket" href="http://localhost/ticket/1671" title="enhancement: Publisher profile sysadmin authz (closed: fixed)">#1671</a> [1d] </li><li>Publisher profile user authz <a class="closed ticket" href="http://localhost/ticket/1672" title="enhancement: Publisher profile user authz (closed: fixed)">#1672</a> [1d] </li><li>Test auth logic hooks for publisher profile <a class="closed ticket" href="http://localhost/ticket/1674" title="enhancement: Test auth logic hooks for publisher profile (closed: fixed)">#1674</a> [3d] </li><li>Publisher admin authz <a class="closed ticket" href="http://localhost/ticket/1675" title="enhancement: Publisher admin authz (closed: fixed)">#1675</a> [3d] </li><li>Publisher editor authz <a class="closed ticket" href="http://localhost/ticket/1676" title="enhancement: Publisher editor authz (closed: fixed)">#1676</a> [2d] </li><li>Add approval status field to groups <a class="closed ticket" href="http://localhost/ticket/1673" title="enhancement: Add approval status field to groups (closed: fixed)">#1673</a> [1d] </li></ul> http://localhost/ticket/1669 http://localhost/ticket/1669 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> Analysis of all permissions for users given their publishers to ensure that we have covered all of the bases and haven't missed any use-cases. </p> http://localhost/ticket/1670 http://localhost/ticket/1670 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> Implement authz for sysadmins in publisher profile </p> http://localhost/ticket/1671 http://localhost/ticket/1671 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> Authz for normal users in publisher profile as per sysadmin </p> http://localhost/ticket/1672 http://localhost/ticket/1672 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> As groups need to be authorised in some cases, particularly publisher profiles, we need a field to denote how this is managed. </p> <p> We should add an 'approval_status' field that will be used to denote its status. By default this will be "approved" but this will allow us to override it with a "pending" or "denied" status. The latter should also change the main status of the Group. </p> http://localhost/ticket/1673 http://localhost/ticket/1673 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> Test all of the new auth logic hooks for publisher profile to make sure they work as expected. Also make sure that we have a test to compare the naming of the core auth funcs with the profile hooks (so we can see what may have changed in one but not the other). </p> http://localhost/ticket/1674 http://localhost/ticket/1674 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> Authz for the publisher admin (who has permission to administer the publisher group themselves). </p> http://localhost/ticket/1675 http://localhost/ticket/1675 Report ross <p> Super <a class="closed ticket" href="http://localhost/ticket/1669" title="enhancement: [super] Publisher profile (closed: fixed)">#1669</a> </p> <p> Editor authz for publisher groups need to be implemented and tested. </p> http://localhost/ticket/1676 http://localhost/ticket/1676 Report amercader <p> Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour. </p> http://localhost/ticket/1677 http://localhost/ticket/1677 Report amercader <p> Right now you need to explicitly load the synchronous_search plugin in your ini file, when this is probably the behaviour that all users expect by default. We could keep a config flag to deactivate it, but synchronous search should be the default behaviour. </p> http://localhost/ticket/1678 http://localhost/ticket/1678 Report dread <p> The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py). </p> <p> The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different. </p> <p> We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views? </p> http://localhost/ticket/1679 http://localhost/ticket/1679 Report ross <p> The new group refactor allows for a hierarchy of groups where each group has a type - to be able to implement a tree of groups. </p> <p> It will need a flag within the group to denote that it is a top level group, to circumvent the need to determine whether the group is a child of a parent node (for a specific use case - a publisher representing the department that only contains publishers). </p> <p> Suggest 'is_top_level' </p> http://localhost/ticket/1680 http://localhost/ticket/1680 Report ross <p> Document all of the changes and usage for <a class="closed ticket" href="http://localhost/ticket/1574" title="enhancement: [super] Storage changes including automated save to Webstore (closed: fixed)">#1574</a> so that we can mark it 'done done'. </p> <p> Make sure we document any setup issues/processes and the settings.py setup required for CKAN so that Mark can announce next week and it should be easily deployable/testable. </p> <p> Internal (perhaps in README somewhere or wiki): how does process work, where does data end up and in what order etc </p> <p> External: here's how to configure it and here's what you get ... (goes in main sphinx docs) </p> http://localhost/ticket/1681 http://localhost/ticket/1681 Report dread <p> We need to be able to translate: </p> <ul><li>EC extension (templates, form) <ul><li>Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions? </li></ul></li><li>Field values in taxonomy (e.g. country names - Eurovoc) <ul><li><a class="closed ticket" href="http://localhost/ticket/1665" title="task: Begin doing research into eurovoc (closed: fixed)">#1665</a> Research into Eurovoc </li><li>Display Taxonomy in different languages </li></ul></li><li>Field values not in taxonomy (e.g. title &amp; description) <ul><li>Use extra fields e.g. _i18n_title_fr = Le data. </li><li>(If lots of fields would need translating then would consider having a new package for each language, linked together with <a class="missing wiki">PackageRelationships?</a>. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.) </li></ul></li></ul><p> Timescale to complete this - <a class="missing wiki">Mar/Apr?</a>. </p> <p> Related work to interface with: </p> <ul><li>Eurovoc - Sean </li><li>i18n in search index - Adria </li></ul> http://localhost/ticket/1682 http://localhost/ticket/1682 Report dread <p> On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect. </p> <p> This is a known issue: <a class="ext-link" href="https://issues.apache.org/jira/browse/SOLR-1777"><span class="icon">​</span>https://issues.apache.org/jira/browse/SOLR-1777</a> affecting SOLR 1.4 only (which comes with Ubuntu 10.04) </p> <p> It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently). </p> http://localhost/ticket/1683 http://localhost/ticket/1683 Report ross <p> ckanext-archiver currently has a settings file (and a default) and it should be passed in all relevant information from the context. </p> <p> Remove all settings (ARCHIVE_DIR and MAX_CONTENT_LENGTH and others) and pass them in from CKAN. </p> http://localhost/ticket/1684 http://localhost/ticket/1684 Report ross <p> Archiver is included in some places just for access to download(), we should move this to lib in core. </p> http://localhost/ticket/1685 http://localhost/ticket/1685 Report seanh http://localhost/ticket/1686 http://localhost/ticket/1686 Report ross <p> We need to deploy the new storage changes (and webstore) to thedatahub for further testing </p> http://localhost/ticket/1687 http://localhost/ticket/1687 Report dread <p> We need an API for changing the authorization (permissions) on a model object. </p> <p> This was neglected in <a class="closed ticket" href="http://localhost/ticket/1253" title="enhancement: Authz refactor (closed: fixed)">#1253</a> because of various changes to authz were looming. But six months later it still hasn't been done. </p> <p> Suggest this is just moving _add_user_object_role from ckan/lib/base.py to the logic layer, but no doubt it will be more complicated. </p> <p> Estimate: 3 days </p> http://localhost/ticket/1688 http://localhost/ticket/1688 Report dread <p> The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets. </p> <p> 'state' could be a parameter on the package_list call </p> <ul><li>Easy to call and implement </li><li>But getting permissions right for the different state values may not be easy - maybe a separate logic function is better? </li></ul> http://localhost/ticket/1689 http://localhost/ticket/1689 Report ross <p> Missed the ckan prefix on the storage settings names so this needs to be fixed. </p> http://localhost/ticket/1690 http://localhost/ticket/1690 Report rgrp <p> Needed to support automated deployment more easily. </p> <p> Est: 15m </p> http://localhost/ticket/1691 http://localhost/ticket/1691 Report rgrp <p> Add image attribute for Dataset and Group which would be shown in prominent place. </p> <p> For dataset this could be a relevant image and for Group this would be more of a logo. </p> <p> I think this has very high value from a UX point of view. </p> <h2 id="Issues">Issues</h2> <ul><li>Do we need to support uploading and storing the image (or do users store elsewhere) <ul><li>Could use something like transloadit service </li><li>Use our own storage facility with a reserved directory </li></ul></li><li>Do we need to do resizing etc? <ul><li>Maybe. Perhaps can prefer this in reasonably small size and then just size down in css for thumbnails etc. </li></ul></li></ul> http://localhost/ticket/1692 http://localhost/ticket/1692 Report ross <p> We should try and move register_pluggable_behaviour into environment and out of routing as it is causing import issues when using simple search. </p> <p> These functions already take a map and so they should be fine any time after the call to make_map </p> http://localhost/ticket/1693 http://localhost/ticket/1693 Report seanh <p> The activity streams feature is already thoroughly tested via the logic layer and API calls, but there needs to be a quick test of the rendering code, e.g. fetch a user's page (thus testing that ckan doesn't crash while rendering it) and then assert that the expected activity streams HTML is in the fetched HTML. </p> http://localhost/ticket/1694 http://localhost/ticket/1694 Report dread <p> Suggested by Richard Cyganiak in <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2012-January/001977.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2012-January/001977.html</a> </p> <ul><li>new relationship type links_to / linked_from <ul><li>Maybe with extra 'count' property? </li></ul></li><li>creating a new relationship should be post to the relationship register, not the source entity. </li><li>add missing doc about creating a new relationship </li><li>'Model Formats' section seems to be missed out of the docs since release 1.4.2. See: <a class="ext-link" href="http://readthedocs.org/docs/ckan/en/ckan-1.4.2/api/version2.html#model-formats"><span class="icon">​</span>http://readthedocs.org/docs/ckan/en/ckan-1.4.2/api/version2.html#model-formats</a> </li></ul> http://localhost/ticket/1695 http://localhost/ticket/1695 Report johnglover <p> To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function. </p> <p> This behaviour is now deprecated however. </p> http://localhost/ticket/1696 http://localhost/ticket/1696 Report rgrp <p> Currently have 'active' and 'deleted' suggest also: </p> <ul><li>'draft' </li><li>'hidden' </li></ul><p> (Do we need both). Also write out workflows related to these. </p> http://localhost/ticket/1697 http://localhost/ticket/1697 Report seanh <p> Add drupal-like "taxonomies" to CKAN. </p> <p> Etherpad with user stories, feature list, design and implementation discussion: </p> <p> <a class="ext-link" href="http://ckan.okfnpad.org/22"><span class="icon">​</span>http://ckan.okfnpad.org/22</a> </p> <p> Branch where this is being developed is feature-1698-tag-taxonomies: </p> <p> <a class="ext-link" href="https://github.com/okfn/ckan/compare/master...feature-1698-tag-taxonomies"><span class="icon">​</span>https://github.com/okfn/ckan/compare/master...feature-1698-tag-taxonomies</a> </p> <p> Tickets related to this have <a class="ext-link" href="http://trac.ckan.org/query?status=accepted&amp;status=assigned&amp;status=closed&amp;status=new&amp;status=reopened&amp;group=owner&amp;max=1000&amp;order=milestone&amp;col=id&amp;col=summary&amp;col=status&amp;col=owner&amp;col=milestone&amp;col=keywords&amp;keywords=~taxonomies"><span class="icon">​</span>keyword taxonomies</a>. </p> http://localhost/ticket/1698 http://localhost/ticket/1698 Report kindly <p> Need to setup server in tenforce test server. </p> http://localhost/ticket/1699 http://localhost/ticket/1699 Report kindly <p> Document deployment on tenforce test server. </p> http://localhost/ticket/1700 http://localhost/ticket/1700 Report kindly <p> Make sure accented characters are normalized when indexed and when searched for. </p> http://localhost/ticket/1701 http://localhost/ticket/1701 Report kindly <p> Make sure accented characters are normalized when indexed and when searched for. </p> http://localhost/ticket/1702 http://localhost/ticket/1702 Report kindly <p> The prototype portal gave us a list of classifiers that are expected for the initial launch. Add then to ckanext-ecportal. </p> http://localhost/ticket/1703 http://localhost/ticket/1703 Report kindly <p> Eurostat datasets mainly. </p> http://localhost/ticket/1704 http://localhost/ticket/1704 Report seanh <p> Add Vocabulary domain class, add logic functions for creating, updating, listing, getting, deleting vocabularies, add tests. </p> http://localhost/ticket/1705 http://localhost/ticket/1705 Report seanh <p> Enable ckanext's to add new vocabularies to default_package_schema(), use the new vocabularies in templates for dataset view, create, edit pages, helper function for extensions to create a vocabulary schema for a given vocabulary name (must match a vocab already in the db) and add it to default package schema. </p> http://localhost/ticket/1706 http://localhost/ticket/1706 Report johnglover <p> From Mark on the ckan-dev list: </p> <p> In the table of broken links at &lt;<a class="ext-link" href="http://thedatahub.org/qa/dataset/broken_resource_links"><span class="icon">​</span>http://thedatahub.org/qa/dataset/broken_resource_links</a>&gt;, the links as printed should be abridged to a fixed length (but link to the full URL). Otherwise the 'reason' column vanishes off the screen. Also a table would probably be better since as things stand the 'reasons' are not lined up </p> http://localhost/ticket/1707 http://localhost/ticket/1707 Report dread <p> The SOLR url, user and password defined in the CKAN config file are ignored and the default SOLR url is used. </p> <p> This causes: </p> <ul><li>"0 datasets" displayed on the home page </li><li>Dataset searches result in 0 results and a small message "There was an error while searching". (Nothing about it in the logs) </li></ul><h2 id="Toreproduce">To reproduce</h2> <p> This bug is only visible if your SOLR instance is not at the default place. To quickly reproduce this problem, setup your machine as a SOLR multicore instance and run: "paster db clean &amp;&amp; paster create-test-data &amp;&amp; paster serve development.ini". It quits with the error: "solr.core.<a class="missing wiki">SolrException?</a>: HTTP code=400, reason=Missing solr core name in path" </p> <h2 id="Codeaffected">Code affected</h2> <ul><li>Does not affect any released versions (1.5.1 is clean). </li><li>Introduced to master: <a class="ext-link" href="https://github.com/okfn/ckan/commit/627382e6ff575146da221efc50321a41152d8b0f"><span class="icon">​</span>https://github.com/okfn/ckan/commit/627382e6ff575146da221efc50321a41152d8b0f</a> by Ross </li></ul> http://localhost/ticket/1708 http://localhost/ticket/1708 Report dread <p> The home page always shows the total number of packages as 0, because the query is '*:*' and SQL search doesn't know how to handle it. </p> <p> Also, simple_search configuration is not documented. </p> <p> Also, no tests for sql search. </p> http://localhost/ticket/1709 http://localhost/ticket/1709 Report dread <p> We want to make email announcements to users of thedatahub.org. </p> <p> The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list. </p> <p> Different ways to do it: </p> <ul><li>'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough? </li><li>CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default). </li></ul><p> Something more advanced in the future could be: </p> <ul><li>CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle. </li><li>Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine. </li><li>CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach. </li></ul> http://localhost/ticket/1710 http://localhost/ticket/1710 Report icmurray <ul><li>On the resources tab, there's a "Check Resources" button which, when clicked makes an ajax request with the list of URLs entered by the user. </li></ul><p> </p> <ul><li>The server checka each link for errors and header information about the linked resource. (Using ckanext/archiver/tasks.py:link_checker()). </li></ul><p> </p> <ul><li>The server returns a list of dicts (json), containing information about the linked resource, and the client uses that to: </li></ul><ul><li>populate the format field of each resource </li></ul><p> </p> <blockquote> <blockquote> <blockquote> <p> The (guessed) 'file_extension' populates the '<span class="underline">format' field. If it's 'htm' or 'html', then we assume it's a listing page, and so don't populate the format field with 'htm' or 'html'. </span></p> </blockquote> </blockquote> </blockquote> <p> </p> <ul><li>provide feedback if a URL appears to be invalid </li></ul><p> </p> <blockquote> <blockquote> <blockquote> <p> If the URL doesn't appear to be a URL at all, or returns a HTTP error, or times-out, then URL field is highlighted in red to indicate it's a bad URL. A tooltip shows the error message to the user. </p> </blockquote> </blockquote> </blockquote> <p> </p> <ul><li>[Optional] provide feedback if a URL appears to point to "Additional Information" - ie the Resource should be entered under "Additional Resources", rather than "Timeseries" or "Individual" datasets. </li></ul><p> Analysis and further description on etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/dgu-package-form"><span class="icon">​</span>http://ckan.okfnpad.org/dgu-package-form</a>? [Section I] </p> http://localhost/ticket/1711 http://localhost/ticket/1711 Report icmurray <p> CKAN provides the /data page page of data.gov.uk </p> <p> Breakdown of tasks: </p> <ul><li>[x] Analysis / refinement of spec. </li><li>[x] Log-in / register as publisher </li></ul><blockquote> <blockquote> <p> (Waiting on publisher form) </p> </blockquote> </blockquote> <ul><li>[x] Population of "browse by publisher" </li></ul><blockquote> <blockquote> <p> list groups ordered by most datasets (Waiting on publisher integration) </p> </blockquote> </blockquote> <ul><li>[ ] <del>Browse by nation</del> </li><li>[ ] <del>Featured datasets</del> </li></ul><blockquote> <blockquote> <p> Now a possible integration point with drupal </p> </blockquote> </blockquote> <ul><li>[X] Tag cloud </li><li>[X] Developers section </li></ul> http://localhost/ticket/1712 http://localhost/ticket/1712 Report icmurray <p> Search results page of DGU. </p> <p> Breakdown: </p> <blockquote> <p> [X] ensure faceting by group (publisher) works [X] allow faceting by UKLP dataset type [X] allow faceting by INSPIRE </p> </blockquote> http://localhost/ticket/1713 http://localhost/ticket/1713 Report icmurray <p> Dataset read page for DGU </p> <p> Breakdown: </p> <ul><li>[X] Fix missing title </li><li>[X] Links to the social media stuff <ul><li>"Share your app" </li><li>"Share your idea" </li><li>"Request new data" </li></ul></li><li>[ ] <del>Presentation of additional information should be broken into sections</del> </li></ul><blockquote> <blockquote> <blockquote> <p> Content is there. Don't do anything until theming/design work is undertaken. </p> </blockquote> </blockquote> </blockquote> <ul><li>[ ] <del>"More like this": links to similar datasets.</del> </li></ul><blockquote> <blockquote> <p> Dropping </p> </blockquote> </blockquote> <ul><li>[N] <del>Stars</del> (I think these are being removed) </li><li>[ ] <del>Comments</del> </li></ul><blockquote> <blockquote> <p> Possible integration point with Drupal. Needs it own ticket. </p> </blockquote> </blockquote> <ul><li>[X] Give feedback to department </li><li>[X] Tidy up the QA stars </li></ul> http://localhost/ticket/1714 http://localhost/ticket/1714 Report kindly <p> Need to make solr schema work for many languages. Get stopwords and choose correct analysis for each. </p> http://localhost/ticket/1715 http://localhost/ticket/1715 Report shevski <p> To allow navigation between sections to still work, also potentially rephrase to avoid confusion that you have to save every time to add a tag or resource and can't say add description, tag and resource in one go </p> http://localhost/ticket/1716 http://localhost/ticket/1716 Report shevski <ol><li>Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search) </li></ol><ol start="2"><li>Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page). </li></ol><ol start="3"><li>Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it. </li></ol><ol start="4"><li>More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu </li></ol><ol start="5"><li>Blue search button should be displayed in line with the search bar, not underneath </li></ol><ol start="6"><li>Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name </li></ol> http://localhost/ticket/1717 http://localhost/ticket/1717 Report dread <p> I get this warning when viewing the home page: </p> <pre class="wiki">event.layerX and event.layerY are broken and deprecated in WebKit. They will be removed from the engine in the near future. </pre><p> CKAN: latest master Browser: Chrome Ubuntu </p> <p> Minor, but worth checking? </p> http://localhost/ticket/1718 http://localhost/ticket/1718 Report dread <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a> </p> <p> e.g. <a class="ext-link" href="http://thedatahub.org/user"><span class="icon">​</span>http://thedatahub.org/user</a> The users with Gravatar have their nice user icons, but the majority of users (without Gravatars) have 'broken link' symbols. Same problem wherever users are shown, such as the dataset history pages. </p> <p> Broken on 1.5.2b - currently on thedatahub.org. </p> http://localhost/ticket/1719 http://localhost/ticket/1719 Report johnglover <p> Part of <a class="closed ticket" href="http://localhost/ticket/1698" title="enhancement: [super] Tag Taxonomies (closed: fixed)">#1698</a> </p> http://localhost/ticket/1720 http://localhost/ticket/1720 Report seanh http://localhost/ticket/1721 http://localhost/ticket/1721 Report seanh <p> See the etherpad for notes: <a class="ext-link" href="http://ckan.okfnpad.org/22"><span class="icon">​</span>http://ckan.okfnpad.org/22</a> </p> http://localhost/ticket/1722 http://localhost/ticket/1722 Report seanh <p> In the tag-taxonomies branch tags now (optionally) belong to vocabularies and where tags previously had unique names, a CKAN instance can now have two tags with the same name as long as they belong to different vocabularies. The tests for the tags model should probably be updated to reflect this. </p> <p> Other tests might also need to be updated. </p> http://localhost/ticket/1723 http://localhost/ticket/1723 Report seanh <p> <tt>ckan/logic/action/get.py:tag_list()</tt> Currently (I think) just returns all tags in the CKAN instance that the user has access to. Needs an optional argument to get only tags from a particular vocabulary? When called with no argument, should only return free tags? Also add an optional dataset argument, so you can get tags for a given dataset and/or vocabulary? </p> <p> <tt>ckan/logic/action/get.py:tag_autocomplete()</tt> Add optional argument to get only tags for a given vocab? When called with no argument, should get only free tags? </p> <p> <tt>ckan/logic/action/get.py:tag_search()</tt> Again, optional arguments for restricting by vocabulary and/or dataset. </p> http://localhost/ticket/1724 http://localhost/ticket/1724 Report dread <p> If you use tag autocompletion, it automatically adds a comma after the tag. This is fine if you are going to add another tag, but not fine if it is the last one. If you don't manually remove the final comma than you get form validation error "Tag string: Tag "" length is less than minimum 2" </p> <p> Affects only version 1.5.2a. </p> http://localhost/ticket/1725 http://localhost/ticket/1725 Report amercader <p> We need: </p> <ul><li>"current" column in harvest_object </li><li>"harvest_source_id" column in harvest_object, foreign key to harvest_source </li><li>"title" column in harvest_source </li></ul><p> The extension should check on startup if the db needs updating and run the necessary scripts. </p> http://localhost/ticket/1726 http://localhost/ticket/1726 Report amercader <p> Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes: </p> <ul><li>Model update (See <a class="closed ticket" href="http://localhost/ticket/1726" title="enhancement: Update harvesting model (closed: fixed)">#1726</a>) </li><li>Update queries for getting harvest sources' datasets </li><li>Update ckanext-csw queries for getting the objects </li><li>Update inspire and ckan harvesters </li><li>Document new field </li></ul> http://localhost/ticket/1727 http://localhost/ticket/1727 Report amercader <p> Three small changes: </p> <ul><li>Set Resource type to None instead of 'Undefined' </li><li>Set extra 'UKLP' = True instead of 'INSPIRE' = True </li></ul><p> These two changes will require manual update of the database. </p> <ul><li>Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed: </li></ul><pre class="wiki"> if last_harvested_object.metadata_modified_date is None \ or last_harvested_object.metadata_modified_date &lt; self.obj.metadata_modified_date \ or self.force_import \ or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False: </pre> http://localhost/ticket/1728 http://localhost/ticket/1728 Report seanh <p> Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation. </p> http://localhost/ticket/1729 http://localhost/ticket/1729 Report seanh <p> A function that takes a vocabulary name or ID as argument and returns a nice select box for selecting items from that given vocabulary. Meant to be used by form templates, to make it easy for them to integrate custom vocabularies. Could use <a class="ext-link" href="http://harvesthq.github.com/chosen/"><span class="icon">​</span>http://harvesthq.github.com/chosen/</a> </p> http://localhost/ticket/1730 http://localhost/ticket/1730 Report amercader <p> Estimate: 4-5d </p> <p> Only sysadmins can manage harvest sources right now, we need to allow publishers to handle their own sources. </p> <p> We need to decide if there will be two different profiles on ckanext-harvest, the current one with only sysadmins allowed to manage the harvesting and a new one supporting publishers (ckan core can be configured to not use the publisher model) </p> <p> Tasks include: </p> <ul><li>Auth checks on the different controller actions (and the command line interface?) </li><li>Modify the index page to just show sources the logged user has permissions on. </li></ul><p> (Talk with Ross about the specific functions to use) </p> <ul><li>Create/edit form to show a dropdown with the user available publishers. </li></ul> http://localhost/ticket/1731 http://localhost/ticket/1731 Report seanh <p> Initially we'll just add all tags from all vocabularies into the existing tags search facet (this should happen automatically). Later we want to enable templates to use separate search facets for different vocabularies. </p> <p> Also need to make sure that search terms are matched against the texts of the tag names themselves (should happen already) and maybe also against the texts of the vocabulary names? </p> http://localhost/ticket/1732 http://localhost/ticket/1732 Report amercader <p> Right now harvest sources can be inactivated (only via the cli), but we need to make sure: </p> <ul><li>Expose this on the web interface (only for sysadmins, see <a class="closed ticket" href="http://localhost/ticket/1731" title="enhancement: Publisher support for harvesting (closed: fixed)">#1731</a>) </li><li>Inactive sources are not shown in the web interface </li></ul><p> Also for deleted packages: </p> <blockquote> <p> a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins </p> </blockquote> <p> Done in <a class="closed ticket" href="http://localhost/ticket/1425" title="enhancement: Dataset delete button (closed: fixed)">#1425</a>. Not sure if the button is only shown to sysadmins. </p> <blockquote> <p> b) No undelete: If you reharvest a deleted package, it remains deleted </p> </blockquote> <ul><li>Publisher should create a new record with new identifier and same/different content as applicable. </li><li>If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated. </li><li>You won't get the state where a package is updated in its deleted state </li><li>If they change the ID and reharvest, a new record appears on DGU. </li></ul><blockquote> <p> c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API. </p> </blockquote> http://localhost/ticket/1733 http://localhost/ticket/1733 Report amercader <p> Some changes are needed in ckanext-harvest to support the new piece of work in the UKLP harvesters, mainly related to the deletion of existing sources. </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1726" title="enhancement: Update harvesting model (closed: fixed)">#1726</a> Update harvesting model </li><li><a class="closed ticket" href="http://localhost/ticket/1727" title="enhancement: Add &#34;current&#34; property to harvest objects (closed: fixed)">#1727</a> Add "current" field to harvesting objects </li><li><a class="closed ticket" href="http://localhost/ticket/1728" title="enhancement: Update UKLP harvesters (closed: fixed)">#1728</a> Update UKLP harvesters </li><li><a class="closed ticket" href="http://localhost/ticket/1733" title="enhancement: Delete (inactivate) harvest sources (closed: fixed)">#1733</a> Delete (inactivate) harvest sources </li></ul><p> At some point we also want to support a publisher model for the harvesting: </p> <ul><li><a class="closed ticket" href="http://localhost/ticket/1731" title="enhancement: Publisher support for harvesting (closed: fixed)">#1731</a> Publisher support for harvesting </li></ul> http://localhost/ticket/1734 http://localhost/ticket/1734 Report dread <p> Super ticket <a class="closed ticket" href="http://localhost/ticket/1744" title="enhancement: UX improvements to dataset edit as of Dec 2011 (closed: fixed)">#1744</a> </p> <p> If you edit a dataset, simply add a tag and click on "Save" then you get the message "You have unsaved changed. Make sure to click 'Save Changes' below before leaving this page." inserted whilst the submission takes place. (This is because the click moves focus away from the tag field). This (reasonably common) edge-case could usefully be squashed. </p> <p> Also note typo: "unsaved changed" </p> http://localhost/ticket/1735 http://localhost/ticket/1735 Report kindly <p> Translate all areas of metatdata to selected language. This includes finding datasets in all languages. </p> <p> Multilingal Metadata: Tasks: </p> <ul><li>Add translation table. 2d </li><li>Api to add translation 3d </li><li>Add extention point to dataset view. 3d </li><li>Add extension point to search index to add translations. 2d. </li><li>Add all eurovoc data to translations. 1d. </li><li>Make multilingual solr schema 2d <a class="closed ticket" href="http://localhost/ticket/1715" title="enhancement: Search index multilingual (closed: fixed)">#1715</a> </li></ul> http://localhost/ticket/1736 http://localhost/ticket/1736 Report rgrp <p> Super ticket: <a class="new ticket" href="http://localhost/ticket/1745" title="enhancement: Dataset search UX improvements as of Jan 2012 (new)">#1745</a> </p> <ul><li>Convert /api/rest/dataset to be search query (i.e. take arguments in ?....) </li><li>Directly expose solr though may want to override limit. See <a class="ext-link" href="https://github.com/okfn/openspending/blob/master/openspending/ui/controllers/api.py#L48"><span class="icon">​</span>https://github.com/okfn/openspending/blob/master/openspending/ui/controllers/api.py#L48</a> </li></ul><p> Required for some improvements to UX (such as autocomplete and better search). </p> http://localhost/ticket/1737 http://localhost/ticket/1737 Report kindly <p> It is useful for extensions to add or change items that go into the search index. Add an extension point for this. </p> http://localhost/ticket/1738 http://localhost/ticket/1738 Report kindly <p> Test failures in master due to tom and rufus not running tests. 0.3d. </p> http://localhost/ticket/1739 http://localhost/ticket/1739 Report seanh <p> It's really bad to do <tt>from module import *</tt> and CKAN has a lot of them. I suggest a three-pronged approach: </p> <ol><li>Don't add any more of them. </li></ol><ol start="2"><li>When you're programming if you see an easy opportunity to remove one then do so. </li></ol><ol start="3"><li>At some point we should task someone to go through the code and remove them all (which is what this ticket is for), but this will be a big job and may break things. </li></ol><p> We should also get rid of most or all of the <tt>from module import foo</tt> and <tt>from module import foo, bar</tt> statements. </p> <p> I think the right thing to do is just <tt>import module</tt> and then use <tt>module.foo</tt> in your code. But if you find yourself doing <tt>module.foo.bar</tt> then you may have a code smell. </p> <p> See: <a class="ext-link" href="http://docs.python.org/dev/howto/doanddont.html"><span class="icon">​</span>http://docs.python.org/dev/howto/doanddont.html</a> </p> http://localhost/ticket/1740 http://localhost/ticket/1740 Report kindly <p> Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown. </p> <p> Tasks include: </p> <ul><li>Fix up data_dict and templates so that the pkg objects do not get used in the templates. This is to assure that all data passed is modifiable. (2d) (complete) </li><li>Add extension point to Ipackage controller (1d) </li><li>Test (1d) </li></ul> http://localhost/ticket/1741 http://localhost/ticket/1741 Report ross <p> It is possible to create a group or a dataset called new, or edit, which will then be inaccessible as it is blocked by the existing routes with those names. </p> <p> We should maintain a list of invalid names in the verification that happens behind the scenes in the UI. </p> http://localhost/ticket/1742 http://localhost/ticket/1742 Report seanh <p> For example, after deleting a dataset old activity stream events will still hyperlink to that dataset. </p> http://localhost/ticket/1743 http://localhost/ticket/1743 Report rgrp <p> Super ticket: <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a> </p> <ul><li><del>Show edit summary form again, at least on edit pages (and prefill it for create to something sensible like: Creating dataset {title} {name})</del> [Done <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a>] </li><li><del>Fix drop down for resource types in resource add/edit to limit to fixed set of options (as per <a class="ext-link" href="http://wiki.ckan.org/Domain_Model/Resource"><span class="icon">​</span>http://wiki.ckan.org/Domain_Model/Resource</a>)</del> <ul><li><del>See also <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-December/001860.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-December/001860.html</a></del> </li></ul></li><li><del>Allow reordering of resources (or: and this is somewhat more complex but possibly simpler: allow editing of a resource_order field and we use that to sort resources (this is more like wordpress page_order -- these two options could become the same if we just hijack the existing position field on the join table and get rid of ordereredlist extension))</del> [DONE <a class="closed ticket" href="http://localhost/ticket/1506" title="enhancement: [super] UX improvements to dataset edit, create and search + group + user ... (closed: fixed)">#1506</a>] </li><li><del>Radically simplify set of licenses</del> [DONE <a class="closed ticket" href="http://localhost/ticket/1359" title="enhancement: Radically simplify standard list of licenses (closed: fixed)">#1359</a>] </li><li><del>"Unsaved changes" warning briefly when you click "Save" <a class="closed ticket" href="http://localhost/ticket/1735" title="enhancement: &#34;Unsaved changes&#34; warning briefly when you click &#34;Save&#34; (closed: wontfix)">#1735</a></del> </li><li><del>"Unsaved changed" typo on dataset edit page</del> </li></ul> http://localhost/ticket/1744 http://localhost/ticket/1744 Report rgrp <p> Changes to make search both more exploratory and more satisfying to use </p> <ul><li>Search query build - <a class="closed ticket" href="http://localhost/ticket/1603" title="enhancement: Search query builder (closed: duplicate)">#1603</a> <ul><li>Ability to add new facet fields "live" </li><li>That is add fields which then contain faceted options (a bit like data.hri.fi) </li></ul></li><li>(??) Autocomplete / drop down on search (i.e. search while you type) <ul><li>Dubious about value / cost ratio here </li></ul></li></ul><p> Probably would involve to pure JS and HTML implementation. </p> <h2 id="Implementation">Implementation</h2> <p> Probably require </p> <ul><li> API changes to expose solr style API directly <a class="closed ticket" href="http://localhost/ticket/1737" title="enhancement: Expose solr-based search API (closed: fixed)">#1737</a> </li></ul> http://localhost/ticket/1745 http://localhost/ticket/1745 Report seanh <p> Currently user, package and group activity streams only return the most recent 15 activities, even though all activities are kept in the db. Do we want to add pagination - to both the API and the HTML pages - to support retrieving older activities? </p> http://localhost/ticket/1746 http://localhost/ticket/1746 Report seanh <p> Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits? </p> http://localhost/ticket/1747 http://localhost/ticket/1747 Report seanh <p> At the very least we should make our JSON output conform to their spec: </p> <p> <a class="ext-link" href="http://activitystrea.ms/"><span class="icon">​</span>http://activitystrea.ms/</a> </p> http://localhost/ticket/1748 http://localhost/ticket/1748 Report seanh <p> Currently the activity_create() logic action function only lets you create top-level activity stream items, and not their related activity details. It should handle activity details via nested dicts. </p> http://localhost/ticket/1749 http://localhost/ticket/1749 Report seanh <p> Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers. </p> http://localhost/ticket/1750 http://localhost/ticket/1750 Report icmurray <p> A collection of improvements to the dataset creation/edition form (feedback form DGU). </p> <ul><li>[x] validation error formatting </li><li>[x] geograohic coverage: add local authority </li><li>[x] resource format field </li><li>[x] resource format auto-complete </li><li>[x] renamed data tab -&gt; files tab </li><li>[x] integrate Adris'a INSPIRE changeset, and migration scripts as part of build </li><li>[x] remove "discontinued" from the update frequency </li><li>[x] disable the disabling of save </li><li>[x] Remove ability to edit url on the edit-form (and remove "url not available") </li></ul> http://localhost/ticket/1751 http://localhost/ticket/1751 Report icmurray <ul><li>[x] hide black stars on the data-view page </li><li>[x] Edit on view form (if you have the right permission) </li><li>[x] RES_FORMAT ~&gt; Resource Format </li><li>[x] Map Based Search -&gt; UK Location Map Based Search </li><li>[x] Note at the top right expalining that these are seach filters (if they are) -&gt; this requires "add a dataset" and "other access" to be moved to a different place (just some whitespace is fine) </li><li>[x] Display publisher and provider (UKLP records only). </li><li>[x] analysis of filter by OGL / non-OGL </li><li>[x] provide /dump on the build server </li></ul> http://localhost/ticket/1752 http://localhost/ticket/1752 Report rgrp <ul><li>Upgrade to new <a class="missing wiki">DataStore?</a> system </li><li>Better support for unicode etc </li><li>? Rename to <a class="missing wiki">DataStorer?</a> </li><li>Deploy </li></ul> http://localhost/ticket/1753 http://localhost/ticket/1753 Report seanh <p> When an extra is added to or removed from a dataset or is updated, an activity should go in the dataset's and user's activity streams. </p> http://localhost/ticket/1754 http://localhost/ticket/1754 Report ross <p> The publisher index page should show an alpha paged list of publishers. </p> http://localhost/ticket/1755 http://localhost/ticket/1755 Report ross <p> The publisher index page should show a treeview of the publishers within the system. Probably behind a tab. </p> http://localhost/ticket/1756 http://localhost/ticket/1756 Report ross <p> On publisher view page - hide administrators from public view - only logged in admins of group should see them. </p> http://localhost/ticket/1757 http://localhost/ticket/1757 Report ross <p> Ensure publisher edit page only allows admins to manage users. <a class="closed ticket" href="http://localhost/ticket/1675" title="enhancement: Publisher admin authz (closed: fixed)">#1675</a> is related. </p> http://localhost/ticket/1758 http://localhost/ticket/1758 Report ross <p> Only sysadmin can access create publisher form. </p> http://localhost/ticket/1759 http://localhost/ticket/1759 Report ross <blockquote> <p> Get access to live Drupal MySQL database to get latest publisher list for migration. </p> </blockquote> <p> Write a one-off script for import of the data. </p> http://localhost/ticket/1760 http://localhost/ticket/1760 Report ross <p> History on user profile should only be visible to a user who is a member of at least one group. </p> http://localhost/ticket/1761 http://localhost/ticket/1761 Report ross <p> A new form for create publisher -&gt; Add publisher access to your account' New wizard Autocomplete publisher name if linked from publisher view. Email address used to publish should be activated/validated on first use. Original email still used to log in. </p> http://localhost/ticket/1762 http://localhost/ticket/1762 Report seanh http://localhost/ticket/1763 http://localhost/ticket/1763 Report seanh <p> Which is where the other API tests have now been moved. </p> http://localhost/ticket/1764 http://localhost/ticket/1764 Report seanh <p> Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique. </p> http://localhost/ticket/1765 http://localhost/ticket/1765 Report johnglover http://localhost/ticket/1766 http://localhost/ticket/1766 Report johnglover <p> Need to test: </p> <ul><li>API call to create new vocab </li><li>API call to add tags with a given vocab </li><li>API call to update tags </li><li>Dummy plugin to test that updating tags works via WUI </li></ul> http://localhost/ticket/1767 http://localhost/ticket/1767 Report icmurray http://localhost/ticket/1768 http://localhost/ticket/1768 Report icmurray http://localhost/ticket/1769 http://localhost/ticket/1769 Report icmurray http://localhost/ticket/1770 http://localhost/ticket/1770 Report icmurray http://localhost/ticket/1771 http://localhost/ticket/1771 Report icmurray http://localhost/ticket/1772 http://localhost/ticket/1772 Report icmurray http://localhost/ticket/1773 http://localhost/ticket/1773 Report icmurray http://localhost/ticket/1774 http://localhost/ticket/1774 Report seanh <p> We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list. </p> <ul><li>Add tags_create() function that calls the existing tag_create() many times? </li><li>Also tags_delete()? </li><li>Or just let tag_create() and tag_delete() accept a list? </li><li>Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.) </li></ul> http://localhost/ticket/1775 http://localhost/ticket/1775 Report seanh <p> Add API calls for adding one or more tags to and removing one or more tags from a vocabulary, without affecting the other tags in that vocabulary and without having to pass the full list of the vocabulary's tags. </p> http://localhost/ticket/1776 http://localhost/ticket/1776 Report kindly <p> Basic description of multilingual features intended to be supported. </p> http://localhost/ticket/1777 http://localhost/ticket/1777 Report seanh <p> In many places we have @classmethods where simple module-level functions would do (and would be more idiomatic in Python), e.g. the <tt>get()</tt> classmethods in the model/ classes. </p> <p> Doing it with module functions lets us type module.function() instead of module.Class.method(). </p> <p> In many cases we're importing classes directly with <tt>from module import Class</tt> and then doing <tt>Class.method()</tt>, but <tt>from module import foo</tt> is bad style: <a class="ext-link" href="http://docs.python.org/dev/howto/doanddont.html"><span class="icon">​</span>http://docs.python.org/dev/howto/doanddont.html</a> </p> http://localhost/ticket/1778 http://localhost/ticket/1778 Report kindly <p> A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination. </p> http://localhost/ticket/1779 http://localhost/ticket/1779 Report kindly <p> Add api to translation to the term_translation table. </p> <ul><li>add to logic 2d </li><li>test 2d </li></ul> http://localhost/ticket/1780 http://localhost/ticket/1780 Report kindly <p> Add api to translation to the term_translation table. </p> <ul><li>add to logic 2d </li><li>test 1d </li></ul> http://localhost/ticket/1781 http://localhost/ticket/1781 Report rgrp <ul><li>Relationships in sidebar [WONTFIX yet without some link to documentation or datastructure, something for me to render?] </li><li><del>Icons for *types* of resources in resource list</del> </li><li>List of 'Owners' (administrators) at top right of sidebar (a la group page) <ul><li>Much prefer owner to administrator ... </li></ul></li></ul> http://localhost/ticket/1782 http://localhost/ticket/1782 Report dread <p> If the current page URL has unicode in it (e.g. <tt>u'/tag/biocombust\xedveis'</tt>) then I'm not sure how why it isn't more encoded, but we are seeing links such as <tt>http://beta.dados.gov.br/dados/locale?locale=hu&amp;return_to=/dados/tag/biocombust%C3%ADveis&amp;hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251</tt> which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash(). </p> http://localhost/ticket/1783 http://localhost/ticket/1783 Report seanh <p> The HTML templates for some activity stream activities link to objects (packages, resources, etc.) by their names, e.g. /dataset/foo, but if the object has since been renamed then the link will be broken. </p> <p> Possible solutions are to use the object ID to find its current name at rendering time and link to that, or link to the object by ID rather than by name. </p> <p> Linking by ID is both simpler to implement and better. Links by ID are ugly, but they are permanent and don't break when an object is renamed. We should be using links by ID everywhere on CKAN, so that when third-party sites copy-paste our URLs and link to us, their links don't break later. </p> <p> Fixing this means fixing the dataset_link(), resource_link(), group_link() etc. functions in ckan/lib/helpers.py which will change the links everywhere in CKAN where those functions are used to link to objects (which should be everywhere where we link to an object). </p> <p> If we want to have nice looking by-name URLs and what them to be permanent, then we need to change CKAN so that objects cannot be renamed. </p> http://localhost/ticket/1784 http://localhost/ticket/1784 Report seanh <p> Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead? </p> http://localhost/ticket/1785 http://localhost/ticket/1785 Report mirko.spasic@… <p> If I have a relationship between two packages, and then I edit any property of one of them (Author email, for example), the relationship will automatically be lost. </p> http://localhost/ticket/1786 http://localhost/ticket/1786 Report dread <ul><li>Lists of entities should be full URLs, rather than just the names </li><li>Discoverability - /api/v3/rest should list the entity types that can be listed </li></ul><p> This could be v3 of the RESTful interface. </p> http://localhost/ticket/1787 http://localhost/ticket/1787 Report zephod <p> From Antonia Acuno, via Ira: </p> <p> It appears that the current layout for the datahub does not work in IE7, which would mean a problem for us as most gov uses it and is one of those we have to support...any views? </p> <p> Tested in several gov departments computers, so a common enough issue. </p> <p> <a class="ext-link" href="http://imgur.com/ip4xa"><span class="icon">​</span>http://imgur.com/ip4xa</a> </p> <p> From Rufus: We haven't officially supported IE7 so far but can put engineering effort into this if absolutely needed. I note that IE 7 is now 3y out of date and is likely a security risk. Obviously this is something to ask Antonio about. </p> http://localhost/ticket/1788 http://localhost/ticket/1788 Report seanh <p> So users can rename a tag and/or move it between vocabularies using the API. </p> <p> Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all. </p> <p> What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary). </p> http://localhost/ticket/1789 http://localhost/ticket/1789 Report dread <p> From Pablo: </p> <p> Editing the tags field is clumsy when there are too many tags. Could show existing effectively as tags (like delicious), then allow clicks to delete. New tags added via text box. </p> http://localhost/ticket/1790 http://localhost/ticket/1790 Report dread <p> When you are on the /dataset/new page and you try and change locale then you get a 500 error. </p> <p> This is because it adds the '<span class="underline">cache' parameter, to ensure any proxy cache in the chain does not just send the cached page. </span></p> <p> e.g. <a class="ext-link" href="http://127.0.0.1:5000/dataset/new?__cache=37713707"><span class="icon">​</span>http://127.0.0.1:5000/dataset/new?__cache=37713707</a> </p> http://localhost/ticket/1791 http://localhost/ticket/1791 Report kindly <p> The api currently uses the default validation schemas regardless of the form you specified on the front end. There should be a way to post through the api using the same validation rules. </p> http://localhost/ticket/1792 http://localhost/ticket/1792 Report ross <p> Changes to publishers for customer </p> http://localhost/ticket/1793 http://localhost/ticket/1793 Report ross <p> Changes to publishers for customer that would otherwise clutter up the master branch. </p> http://localhost/ticket/1794 http://localhost/ticket/1794 Report ross <p> The Package model should have a approval_status as the Group model does. See migration 049. </p> http://localhost/ticket/1795 http://localhost/ticket/1795 Report dread <p> lxml is used in CKAN core in two places: </p> <ul><li>ckan/lib/helpers.py:239 Parsing Markdown and ensuring it is valid XML (i.e. tags close nicely so the HTML of the rest of the CKAN page is not disruptable). </li><li>reading the SOLR config XML to determine what version it is. </li></ul><p> Both of these could be done with xml.dom built into Python and would save us a sizeable and slightly problematic dependency. </p> <p> lxml is needed by these extensions: ckanext-inspire, ckanext-harvest, ckanext-csw &amp; ckanext-wordpresser, so we'd have to add this to their requirements. </p> http://localhost/ticket/1796 http://localhost/ticket/1796 Report rgrp <p> Main tasks: </p> <ul><li>Integrated seamlessly into CKAN (authorization etc) </li><li>Based off <a class="missing wiki">ElasticSearch?</a> </li></ul><p> Etherpad: <a class="ext-link" href="http://ckan.okfnpad.org/webstore"><span class="icon">​</span>http://ckan.okfnpad.org/webstore</a> </p> <h1 id="Tasks">Tasks</h1> <ul><li><del>New API point (config + controller) [0.75d]</del> <ul><li>/api/data/{resource-id} </li></ul></li><li><del>Authorization [0.5d]</del> </li><li><del>Documentation [0.25d]</del> </li><li><del>Upgrade data viewer to use ES [0.5d]</del> </li><li><del>Convert webstorer to new setup - <a class="ext-link" href="http://github.com/okfn/ckanext-webstorer"><span class="icon">​</span>http://github.com/okfn/ckanext-webstorer</a> [1d] - <a class="closed ticket" href="http://localhost/ticket/1753" title="enhancement: Last fixes of Webstorer + Deploy on the DataHub (closed: fixed)">#1753</a></del> <ul><li>See below </li></ul></li><li>"Create <a class="missing wiki">DataStore?</a> Resource" in Resource creation (??) <ul><li>Just creates a resource with blank url and datastore enabled (webstore_url non null) </li></ul></li><li><del>Tick box to enable <a class="missing wiki">DataStore?</a> in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)</del> <ul><li>Also need to use this info in <a class="missing wiki">DataStore?</a> extension? </li></ul></li><li><del>Create nice instructions on Resource page about using the <a class="missing wiki">DataStore?</a></del> </li><li><del>Rename webstore -&gt; datastore</del> </li><li><del>Deploy [0.5d]</del> </li></ul><h1 id="Details">Details</h1> <h2 id="LayoutonES">Layout on ES</h2> <pre class="wiki">elasticsearch/{index}/{type-id} index = ckan-{ckan-id} type-id = resource-id (1) </pre><h2 id="Requestpath">Request path</h2> <p> Req =&gt; Nginx =&gt; CKAN App (authz) =&gt; req to elasticsearch and returns results </p> <p> See as an example: <a class="ext-link" href="https://github.com/okfn/elastic-proxy"><span class="icon">​</span>https://github.com/okfn/elastic-proxy</a> </p> <h2 id="Questions">Questions</h2> <ol><li>How do we decide what to use in the viewer e.g. Webstore data versus <a class="missing wiki">DataProxy?</a> <ul><li>Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't. </li></ul></li><li>How do we decide when webstore is activated <ul><li>Ans: see previous. </li></ul></li><li>Do we (and how do we) allow for resources which are just for webstores <ul><li>Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ... </li></ul></li><li>How does webstorer work ... <ul><li>CSV import - NS: make sure we add e.g. rownum or <span class="underline">id</span> = rownum ... </li><li>Separate mini-service (webstorer.thedatahub.org ...) </li></ul></li><li>How do we selectively disable recline where not appropriate (e.g. query support makes no sense for <a class="missing wiki">DataProxy?</a>) <ul><li>Ans: do not bother at present. Focus on Webstore route. </li></ul></li><li>Where does geodata support come in (in Recline or separately) <ul><li>This is a separate issue </li></ul></li></ol><h2 id="UserStory1">User Story 1</h2> <p> User -&gt; CKAN instance -&gt; Add Dataset -&gt; Adds a Resource via uploading a CSV file -&gt; Save </p> <p> =&gt; Result = New Dataset + Resource + Data API to resource and nice Preview of the data </p> <p> Solution: </p> <ol><li>Allow normal upload process </li><li>Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database. </li><li>Update the resource in some way (set webstore_url) </li></ol><h1 id="UserStories">User Stories</h1> <h2 id="UserStory2">User Story 2</h2> <p> User -&gt; CKAN instance -&gt; Add Dataset -&gt; Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise) </p> <p> =&gt; Result = New Dataset + Resource + nice Preview of the data (wherever possible) </p> <h2 id="UserStory3">User Story 3</h2> <p> User -&gt; CKAN instance -&gt; Add Dataset -&gt; Adds a Resource which will be webstore database in CKAN -&gt; User saves data to this data store. </p> <p> =&gt; Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user </p> <p> Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...) </p> http://localhost/ticket/1797 http://localhost/ticket/1797 Report dread <p> You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q. </p> <p> For example: </p> <pre class="wiki">http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93 </pre><p> This "N-dash" (Unicode character 2013) causes this exception: </p> <pre class="wiki">Module ckan.controllers.api:460 in search &lt;&lt; if ver in u'12': # Otherwise, put all unrecognised ones into the q parameter params = convert_legacy_parameters_to_solr(params) query = query_for(model.Package) results = query.run(params) &gt;&gt; params = convert_legacy_parameters_to_solr(params) Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr &lt;&lt; for search_key in non_solr_params: value_obj = legacy_params[search_key] value = str(value_obj).replace('+', ' ') if search_key == 'all_fields': if value: &gt;&gt; value = str(value_obj).replace('+', ' ') UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128) </pre><p> This problem affects CKAN 1.5 and 1.5.1 only. </p> http://localhost/ticket/1798 http://localhost/ticket/1798 Report dread <p> When you are logged in, you cannot login as a different user before first logging out. (This is a restricting in repoze.who, which gives priority to an existing recognised identity over another one you supply.) </p> <p> Yet, the CKAN login form appears when you try to do something you're not allowed to, even if you are logged in already. You login with a different identity and nothing happens - you remain as the first identity - this is incorrect. </p> <p> (It is also a little confusing to be sent to the log-in page, although we do have a flash message explaining, and you might want to relogin as a superuser if you can.) </p> <p> Whilst in this area, it would be worth fixing the problem for developers where you are logged in, do "paster db clean" and then try to log in - <a class="missing wiki">AuthTkt?</a> recognises your old cookie but the User object doesn't exist, so you get an error logging in. In this instance we should tell people to log-out. </p> http://localhost/ticket/1799 http://localhost/ticket/1799 Report seanh <p> For consistency all the *_list() functions should list objects only and not accept an optional search query. There should be *_search() functions whenever search is needed. </p> <p> Currently it is inconsistent, e.g. package_list() and resource_list() do not accept a search query option and there are package_search() and resource_search(), but user_list() does accept a query and there is no user_search(). tag_list() also accepts a query, and there is also a tag_search() that apparently duplicates the search functionality. </p> http://localhost/ticket/1800 http://localhost/ticket/1800 Report dread <p> You can reset your password (<a class="closed ticket" href="http://localhost/ticket/1186" title="enhancement: Password reset facility (closed: fixed)">#1186</a>) but you have to know the URI (/user/reset) - there is no link to this page! </p> http://localhost/ticket/1801 http://localhost/ticket/1801 Report dread <p> See how possible it is to run CKAN on a Windows server. </p> <p> Plan: </p> <ul><li>Get CKAN code and dependencies installed on Windows <ul><li>ideally as similar to standard package install as possible </li><li>virtualenv/pip should work </li><li>try Cygwin to help </li><li>SOLR and Postgres could potentially run on other servers if necessary. </li></ul></li><li>Try to run on Apache </li><li>See if there are any issues </li></ul> http://localhost/ticket/1802 http://localhost/ticket/1802 Report toby http://localhost/ticket/1803 http://localhost/ticket/1803 Report dread <p> e.g. <a class="ext-link" href="http://189.9.137.65/dados/"><span class="icon">​</span>http://189.9.137.65/dados/</a> clicking on Deutsch link is <tt>http://189.9.137.65/dados/locale?locale=de&amp;return_to=%2Fdados%2F&amp;hash=1dc17c315c419df850da0dd3599eefa9da76fbeb</tt> and redirect goes to <tt>http://189.9.137.65/dados/dados/?__cache=97995106</tt> so /dados/dados/ when it should be /dados. </p> <p> Affects CKAN 1.6b only (not yet released). </p> http://localhost/ticket/1804 http://localhost/ticket/1804 Report toby <p> eg 404 page </p> <p> create a 404 error via a bad url - translation links go to /document/error not the actual bad url </p> http://localhost/ticket/1805 http://localhost/ticket/1805 Report toby <p> language chooser should look like the ec one </p> http://localhost/ticket/1806 http://localhost/ticket/1806 Report seanh <p> For EC Portal need a page (or HTML snippet embedded in the front page?) listing recently added or updated datasets. </p> <ul><li>Could be generated from the activity streams </li></ul><ul><li>Should it list datasets only, or other activities as well? </li></ul><ul><li>May be related to <a class="accepted ticket" href="http://localhost/ticket/1785" title="enhancement: Replace 'Revisions' page with site-wide activity stream (accepted)">#1785</a> (Replace 'Revisions' page with site-wide activity stream) </li></ul> http://localhost/ticket/1807 http://localhost/ticket/1807 Report seanh <p> All data is that is translatable should be translated when viewed. </p> <p> This includes tag names will be ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language. </p> <p> Related to the dataset view extension points for translation, recently added by kindly/=. </p> http://localhost/ticket/1808 http://localhost/ticket/1808 Report johnglover <p> Some request exceptions are currently not being caught (see the celery log on thedatahub for examples) </p> http://localhost/ticket/1809 http://localhost/ticket/1809 Report johnglover <ul><li>Tag list not be added to CKAN core, what is the best way to do this in an extension? </li><li>Does the list need to be updated via API/WUI? </li></ul> http://localhost/ticket/1810 http://localhost/ticket/1810 Report johnglover http://localhost/ticket/1811 http://localhost/ticket/1811 Report johnglover <ul><li>see how this currently works with DGU first </li></ul> http://localhost/ticket/1812 http://localhost/ticket/1812 Report amercader <p> Tasks include: </p> <p> Estimate 4d </p> <ul><li>Update requirements (git) </li><li>Search facets error </li><li>Error on dataset read (extras in context) </li><li>Url generator for rdf </li><li>Make templates work with latest theme </li><li>Use dicts in templates (isopen()) </li></ul> http://localhost/ticket/1813 http://localhost/ticket/1813 Report amercader <p> Estimate 2d </p> <p> Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras). </p> <p> Also check non-CKAN harvesters (specially scrappers) to see if they are still working. </p> <p> Make a list of current harvesters with status and potential ones. </p> http://localhost/ticket/1814 http://localhost/ticket/1814 Report amercader <p> Estimate: ? </p> <p> The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset. </p> http://localhost/ticket/1815 http://localhost/ticket/1815 Report amercader <p> Estimate [2-3d] </p> <p> If possible, it should work with both 1.5.1 and 1.6. </p> <p> Needed for PDEU, also at some point for IATI </p> http://localhost/ticket/1816 http://localhost/ticket/1816 Report johnglover <ul><li>write importer for publisher info (currently in JSON format), add to ckanext-ecportal </li><li>add group translations to translations table </li><li>run on test server </li></ul> http://localhost/ticket/1817 http://localhost/ticket/1817 Report amercader <p> Estimate 2.5d </p> <p> Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset. </p> <p> We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by <a class="missing wiki">OpenLayers?</a> <a class="missing changeset" title="No default repository defined">[1]</a>, <a class="missing changeset" title="No default repository defined">[2]</a>. </p> <p> Things to take into account: </p> <ul><li>Integration with the form sections. </li><li>Allow some degree of configuration (default extent, limit geometry types...) </li></ul><p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://openlayers.org/dev/examples/draw-feature.html"><span class="icon">​</span>http://openlayers.org/dev/examples/draw-feature.html</a> <a class="missing changeset" title="No default repository defined">[2]</a> <a class="ext-link" href="http://openlayers.org/dev/examples/modify-feature.html"><span class="icon">​</span>http://openlayers.org/dev/examples/modify-feature.html</a> </p> http://localhost/ticket/1818 http://localhost/ticket/1818 Report kindly <p> Search index code needs to use package_dictize so that it can get the new vocablurary information and is more consistant with the rest of the system. This is in preperation for their translation being put in the search indexer. </p> http://localhost/ticket/1819 http://localhost/ticket/1819 Report kindly <p> All translated fields and vocabularies need to be added to search index in the correct fields. </p> http://localhost/ticket/1820 http://localhost/ticket/1820 Report kindly <p> This extension point needs to combine. </p> <ul><li> Making sure the correct data get put in search index. </li><li> Make the search queries are modified to weight multilingual search correctly. </li><li> View pages, including search results are translated. </li><li> Bring together all search schema modifications. </li></ul> http://localhost/ticket/1821 http://localhost/ticket/1821 Report kindly <p> Dismax query across languages. Titles and selected languages, weighted more highly. </p> http://localhost/ticket/1822 http://localhost/ticket/1822 Report rgrp <p> Huge number of accumulated (and likely unnecessary) scripts in /bin directory. </p> http://localhost/ticket/1823 http://localhost/ticket/1823 Report seanh <p> For a free tag foo you can visit the page at /tag/foo and see a list of all the datasets that have the tag foo, and when the tag appears on dataset view pages etc. it's linked to this tag page. </p> <p> We should do the same thing for vocabulary tags. A tag bar in vocabulary baz should be hyperlinked to a page /tag/baz/bar, or perhaps /vocab/baz/bar. </p> http://localhost/ticket/1824 http://localhost/ticket/1824 Report seanh <p> Done on branch feature-1698-tag-taxonomies </p> http://localhost/ticket/1825 http://localhost/ticket/1825 Report kindly <p> Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for <a class="closed ticket" href="http://localhost/ticket/1822" title="enhancement: Weight mulilingual searches correctly (closed: fixed)">#1822</a> as the dismax option is needed for multilingual search. </p> http://localhost/ticket/1826 http://localhost/ticket/1826 Report dread <pre class="wiki">I have just deny visitors the create-user permission: sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message "Unauthorized to create a user" (when not logged in). But anyway that is an improvement. </pre> http://localhost/ticket/1827 http://localhost/ticket/1827 Report rgrp <p> Recline v0.3 has introduced some breaking changes in API and usage. Upgrade to use it. </p> http://localhost/ticket/1828 http://localhost/ticket/1828 Report dread <p> On the homepage click "francais" and then "English". The flash message reads "Le langage a été fixé à: français" when it should say "The language is now: English". </p> http://localhost/ticket/1829 http://localhost/ticket/1829 Report dread <p> Allow OpenID to be disabled as a login option. </p> <p> Currently (CKAN 1.6) there is a config option openid_enabled which stops the login option being displayed, but: </p> <ul><li>In the User edit form you can still add your OpenID </li><li>The repoze.who.openid middleware is still required - can it be avoided? </li></ul> http://localhost/ticket/1830 http://localhost/ticket/1830 Report ross <p> Users should be able to log in using either their username, or their email address, both of which are unique. </p> <p> Will require a change to <a class="missing wiki">UsernamePasswordAuthenticator?</a> in ckan.lib.authenticator.py and possible a useful User.by_email in the user model if it doesn't already exist. </p> http://localhost/ticket/1831 http://localhost/ticket/1831 Report dread <p> Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance. </p> <p> Without this, they need to resort to more difficult methods such as: </p> <ul><li>cleaning and reloading the database </li><li>setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names. </li></ul><p> Requested for NHSIC. </p> <h2 id="Implementation">Implementation</h2> <ul><li>This could slot into the Action API. </li><li>Of course we would need to ensure the user's had been given the specific right to purge. </li><li>I suggest we log the full details of the dataset being purged. </li></ul> http://localhost/ticket/1832 http://localhost/ticket/1832 Report zydio <p> If OFS is configured with Pairtree to use a local file storage, all POST requests to add/update metadata ( /api/storage/metadata/{label} ) will fail. </p> <p> This is due to the use of <a class="ext-link" href="https://github.com/okfn/ofs/blob/master/ofs/remote/botostore.py"><span class="icon">​</span>BotoOFS</a> specific <em>private</em> methods in <a class="ext-link" href="https://github.com/okfn/ckan/blob/master/ckan/controllers/storage.py"><span class="icon">​</span>StorageAPIController.set_metadata()</a>, eg: self.ofs.<strong>_require_bucket</strong>(bucket), self.ofs.<strong>_get_key</strong>(b, label), self.ofs.<strong>_update_key_metadata</strong>(k, metadata) ... those methods can't be found in <a class="ext-link" href="https://github.com/okfn/ofs/blob/master/ofs/local/pairtreestore.py"><span class="icon">​</span>POTFS</a> and this causes errors. The API should use only <a class="ext-link" href="https://github.com/okfn/ofs/blob/master/ofs/base.py"><span class="icon">​</span>OFSInterface</a> methods, or should conditionally make calls based off the actual type of <em>self.ofs</em>. </p> <p> <em>PS: I did set "ckan" as "Component" in the ticket because storage has been integrated back into the core in CKAN 1.6</em> </p> http://localhost/ticket/2197 http://localhost/ticket/2197 Report zydio <p> Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on <a class="ext-link" href="https://github.com/okfn/ckanext-storage/blob/master/README.rst"><span class="icon">​</span>Storate Metadata API</a>. </p> http://localhost/ticket/2198 http://localhost/ticket/2198 Report seanh http://localhost/ticket/2199 http://localhost/ticket/2199 Report seanh <p> Currently tag_show() only works with free tags, it's not possible to get a tag_show for a vocabulary tag. </p> http://localhost/ticket/2200 http://localhost/ticket/2200 Report rgrp <p> Suggest: </p> <p> %title. %author. Retrieved %date. %site_title. </p> <p> For resource: %title = %dataset_title. %resource_name. </p> http://localhost/ticket/2201 http://localhost/ticket/2201 Report rgrp <p> Just like we display download counts we should display view counts. </p> http://localhost/ticket/2202 http://localhost/ticket/2202 Report rgrp <ul><li>Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf </li><li>propose we use pdf.js - <a class="ext-link" href="https://github.com/mozilla/pdf.js"><span class="icon">​</span>https://github.com/mozilla/pdf.js</a> </li><li>We could also try iframing for browsers with native support but this seems more complex </li></ul> http://localhost/ticket/2203 http://localhost/ticket/2203 Report rgrp <p> This is a reworking of the existing apps extension. </p> <p> Initial proposal at <a class="ext-link" href="http://wiki.ckan.org/Proposals#Apps_in_CKAN"><span class="icon">​</span>http://wiki.ckan.org/Proposals#Apps_in_CKAN</a> and <a class="ext-link" href="http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN"><span class="icon">​</span>http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN</a> </p> <h2 id="Naming">Naming</h2> <p> What do we call this extension? </p> <ul><li>related <ul><li>RP best IMO but perhaps too close to separate relationships concept? </li></ul></li><li>relatedstuff </li><li>links </li><li>references </li></ul><h2 id="Proposal">Proposal</h2> <p> "Related Stuff": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension) </p> <ul><li>This case seems a general example of wanting to link some external thing to a dataset (e.g. this policy paper refs this dataset, this application uses this dataset, this visualization is built on this dataset cf <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html</a>) </li></ul><h2 id="Implementation">Implementation</h2> <p> New table named Related with following structure </p> <pre class="wiki">|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp) </pre><ul><li>type = Idea | App | API | Visualization | Post | Paper | News Article <ul><li>Suggest we make this a ckan.ini config option (comma separated ...?) <ul><li>Do we want the possibility of different templates for different types of Related objects? </li></ul></li></ul></li></ul><ul><li>image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See <a class="closed ticket" href="http://localhost/ticket/1692" title="enhancement: Add image attribute to Dataset and Group (closed: wontfix)">#1692</a> (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url. </li></ul><ul><li>owner_id = user_id or creating user (see authorization below) </li></ul><p> Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off. </p> <pre class="wiki">|| id || dataset_id || related_id || status </pre><p> status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related. </p> <h3 id="Url">Url</h3> <ul><li>/dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified} <ul><li>If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id} </li></ul></li><li>/dataset/{dataset-name}/related/add =&gt; Modal dialog on related tab so we can use API to create them. </li></ul><p> /dataset/{id}/related &lt;- list </p> <ul><li>use image_url for small icon etc, title description (shortened?) </li><li>Click through to full related item (optional) </li><li>dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} ) </li><li>related owner sees an edit button / icon (pops up modal) </li></ul><p> /dataset/{id}/related/{id} (optional) </p> <p> /dataset/{id} will have a Related tab (with bubble with count). </p> <ul><li>Drop down with Add Related -&gt; Pop-up modal and save via API </li></ul><blockquote> <blockquote> <p> (Not used: /related/add with dataset prefilled ... ) </p> </blockquote> </blockquote> <p> /related/{id}/edit </p> <h3 id="Authorization">Authorization</h3> <p> Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y). </p> <p> Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete. </p> <p> NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1. </p> <h3 id="Tasksandestimates7.5d">Tasks and estimates {7.5d}</h3> <p> [x] Model + Migration for Related table. {0.75d} </p> <p> [x] Controller for Related (or relevant sections in Package controller). {0.75d} </p> <p> [x] Routing setup. {0.25d} </p> <p> [x] Schema for related. {0.5d} </p> <p> [x] Logic layer actions. {1.0d} </p> <p> [x] Auth (default + publisher). {0.5d} </p> <p> [x] Templates + Dataset changes (new tab etc). {1.0d} </p> <p> [x] JS Application for adding Related objects in a modal. {1.0d} </p> <p> [x] Testing. {0.75d} </p> <p> [x] Dataset owner disabling of Related (via M2M table). {0.5d} </p> <ul><li>Updated to allow author of related to delete as well </li></ul><p> [x] Documentation. {0.25d} </p> http://localhost/ticket/2204 http://localhost/ticket/2204 Report johnglover http://localhost/ticket/2205 http://localhost/ticket/2205 Report johnglover http://localhost/ticket/2206 http://localhost/ticket/2206 Report seanh <p> Apparently it's working when developers run it locally but not on the demo site. </p> http://localhost/ticket/2207 http://localhost/ticket/2207 Report seanh <p> All strings from the ckanext-ecportal extension (and from CKAN core) need to be exported to a pot file to send to tenforce. </p> http://localhost/ticket/2208 http://localhost/ticket/2208 Report ross <p> Super is <a class="missing changeset" title="No default repository defined">[1649]</a> </p> <p> We need to provide customisable mappings between CKAN models and RDF. This needs to be easily configurable for different clients and should take into account objects related to the model being mapped. </p> <p> Current contenders for impl are: </p> <ol><li>ordf.org's Object Description Mapper (which uses OWL) </li></ol><ol start="2"><li>RDFAlchemy, although this looks like it may result in fixed mappings </li></ol><ol start="3"><li>A custom scheme with a simple json file </li></ol><ol start="4"><li>A template based approach using Genshi templates? </li></ol><p> Approach is to get a mapping complete of the attributes directly in the package class, and then provide mappings for related objects (tags/resources etc). </p> <p> We'll then need to be able to publish the RDF to a triplestore. </p> <p> See <a class="ext-link" href="http://ckan.okfnpad.org/rdfmapping"><span class="icon">​</span>http://ckan.okfnpad.org/rdfmapping</a> for further analysis. </p> http://localhost/ticket/2209 http://localhost/ticket/2209 Report ross <p> It is necessary to add a helper method to Group for adding members to that group so that </p> <ul><li>nobody needs to work with the Member model </li><li>It handles existing membership and existing membership with a different capacity </li></ul> http://localhost/ticket/2210 http://localhost/ticket/2210 Report ross <p> Most of the get_group/add_member style calls should be handled, or at least provided via logic layer. </p> <p> Don't need to break existing code, but do need to provide access via logic layer </p> http://localhost/ticket/2211 http://localhost/ticket/2211 Report johnglover http://localhost/ticket/2212 http://localhost/ticket/2212 Report johnglover <ul><li><del>create importer for geo tags</del> </li><li><del>add translations to database</del> </li><li><del>translate tags in WUI</del> </li><li><del>test</del> </li><li><del>deploy</del> </li><li><del>document how to add a geo coverage field in <a class="ext-link" href="http://docs.ckan.org/en/latest/forms.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/forms.html</a></del> </li></ul> http://localhost/ticket/2213 http://localhost/ticket/2213 Report seanh http://localhost/ticket/2214 http://localhost/ticket/2214 Report seanh http://localhost/ticket/2215 http://localhost/ticket/2215 Report seanh http://localhost/ticket/2216 http://localhost/ticket/2216 Report seanh http://localhost/ticket/2217 http://localhost/ticket/2217 Report seanh http://localhost/ticket/2218 http://localhost/ticket/2218 Report seanh <p> User names and descriptions are not translated. </p> http://localhost/ticket/2219 http://localhost/ticket/2219 Report toby <p> need to sort cookies etc </p> <p> should work with nginx and varnish </p> http://localhost/ticket/2220 http://localhost/ticket/2220 Report zephod <p> Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension. </p> <p> Test output here: <a class="ext-link" href="https://gist.github.com/2007985"><span class="icon">​</span>https://gist.github.com/2007985</a> </p> http://localhost/ticket/2221 http://localhost/ticket/2221 Report zephod <p> Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension. </p> <p> Test output here: <a class="ext-link" href="https://gist.github.com/2007985"><span class="icon">​</span>https://gist.github.com/2007985</a> </p> http://localhost/ticket/2222 http://localhost/ticket/2222 Report rgrp <p> Why? </p> <ul><li>We already use quite a bit of boostrap </li><li>We already plan to use bootstrap JS </li><li>Bootstrap is excellent and pretty light </li></ul> http://localhost/ticket/2223 http://localhost/ticket/2223 Report rgrp <ul><li>Get rid of jquery UI <ul><li>Requires switching to bootstrap lookahead plus a small bit of coding for ajax part (see <a class="ext-link" href="https://gist.github.com/2007241"><span class="icon">​</span>https://gist.github.com/2007241</a>) </li></ul></li><li>Remove modernizr (what do we use it for?) </li><li>Convert jquery.tmpl to jquery mustache (?) </li><li>Remove ckanjs moving core parts of it directly into our code. </li></ul><p> Compress everything for releases and add minified version to build (with switch to enable use of this?) </p> http://localhost/ticket/2224 http://localhost/ticket/2224 Report rgrp http://localhost/ticket/2225 http://localhost/ticket/2225 Report rgrp <p> Documentation is key to the success of projects! </p> <h1 id="Movematerialintomaindocsfromwiki">Move material into main docs from wiki</h1> <ul><li><a class="ext-link" href="http://wiki.ckan.org/Writing_asynchronous_tasks"><span class="icon">​</span>http://wiki.ckan.org/Writing_asynchronous_tasks</a> </li><li><a class="ext-link" href="http://wiki.ckan.org/Domain_Model"><span class="icon">​</span>http://wiki.ckan.org/Domain_Model</a> </li><li><a class="ext-link" href="http://wiki.ckan.org/Deployment"><span class="icon">​</span>http://wiki.ckan.org/Deployment</a> </li></ul><h1 id="CreateaUserGuide">Create a User Guide</h1> <p> Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. <a class="ext-link" href="http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/"><span class="icon">​</span>http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/</a>) </p> <ul><li>Publishing data </li><li>Authorization and workflows <ul><li>Publisher Profile (Workflow) </li></ul></li></ul><h1 id="Breakintosections">Break into sections</h1> <p> Suggest something like: </p> <pre class="wiki"># General * Intro * Conceptual Overview * What CKAN helps you do (http://ckan.org/) * FAQ ? (or point to the wiki) # For Administrators * Installation and Setup * Configuration options * Customization - Theming etc * Authorization and Workflows * Storage # For Users (Publishers, Data Wranglers, etc etc) * Walkthrough of publishing a dataset * Storage * API (see Developers section) # CKAN Developers * Domain Model * API * Extensions * i18n CKAN Developers * Buildbot </pre> http://localhost/ticket/2226 http://localhost/ticket/2226 Report toby <p> not all tests run and they take too long :( </p> http://localhost/ticket/2227 http://localhost/ticket/2227 Report ross <p> IGroupForm should allow overriding of non-edit templates so that it can specify the index +/or search +/or view. </p> http://localhost/ticket/2228 http://localhost/ticket/2228 Report kindly <p> The logic test did not have <span class="underline">init</span>. This caused lots of tests to fail because there were mock extensions that ran automatically in them. Fix plugin system so this can work. </p> http://localhost/ticket/2229 http://localhost/ticket/2229 Report seanh <p> Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point): </p> <p> The package_search() function is adding the search facets to the search results twice with two different data structures, with keys "facets" and "new_facets". It should be reduced to just the new facets (with the key changed to "facets"). </p> <p> Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets). </p> <p> The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure. </p> <p> In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets. </p> <p> In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead. </p> <p> Anywhere that "new_facets" appears it will have to be changed to "facets" (e.g. in the ckanext/multilingual/plugin.py. </p> <p> This is the merge commit that introduced the duplication: <a class="ext-link" href="https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d"><span class="icon">​</span>https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d</a> </p> <p> This is the earlier commit where the search facets were refactored: <a class="ext-link" href="https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c"><span class="icon">​</span>https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c</a> </p> http://localhost/ticket/2230 http://localhost/ticket/2230 Report amercader <p> On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) <a class="missing changeset" title="No default repository defined">[1]</a> </p> <p> Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins <a class="missing changeset" title="No default repository defined">[2]</a>. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected </p> <pre class="wiki">ValidationError: {'Id': u'The input field id was not expected.'} </pre><p> We need a way to either: </p> <ol><li>Be more flexible accepting schemas on the logic functions </li><li>Modify the harvester plugin to implement the necessary interface </li></ol><p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115"><span class="icon">​</span>https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115</a> </p> <p> <a class="missing changeset" title="No default repository defined">[2]</a> <a class="ext-link" href="https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38"><span class="icon">​</span>https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38</a> </p> http://localhost/ticket/2231 http://localhost/ticket/2231 Report amercader <p> In some cases this exception is fired when building the search index. As explained in <a class="closed ticket" href="http://localhost/ticket/1616" title="defect: Catch exceptions when rebuilding the search index (closed: fixed)">#1616</a> this makes the whole process stop. </p> <pre class="wiki"> File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper return self._update(content, query) File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update rsp = self._post(selector, request, self.xmlheaders) File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post return check_response_status(self.conn.getresponse()) File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status raise ex SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document. Traceback (most recent call last): File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in &lt;module&gt; load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')() File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run result = self.command() File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command rebuild() File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild {'id': pkg.id} File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict return self.update_dict(data) File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict self.index_package(pkg_dict) File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package raise SearchIndexError(e) ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document. </pre> http://localhost/ticket/2232 http://localhost/ticket/2232 Report kindly <p> There is a disrepency between what we put in and get out when using a custom form/api. Narrow this gap by making sure package_show shows the correct serilization. </p> http://localhost/ticket/2233 http://localhost/ticket/2233 Report seanh <p> You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page. </p> <p> This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful. </p> <p> Details </p> <ul><li>Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds </li></ul><ul><li>Feed items should probably be cached somewhere </li></ul><p> Suggested </p> <ul><li>The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry. </li></ul><ul><li>For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference </li></ul><ul><li>Caching is pretty crucial and should probably obey the ttl of the feed. </li></ul> http://localhost/ticket/2234 http://localhost/ticket/2234 Report rgrp http://localhost/ticket/2235 http://localhost/ticket/2235 Report rgrp <ul><li>(Recline) Better graphing (multiple graph types etc) </li><li>(Recline) Bootstrapify </li><li>Integrate </li></ul> http://localhost/ticket/2236 http://localhost/ticket/2236 Report kindly http://localhost/ticket/2237 http://localhost/ticket/2237 Report seanh http://localhost/ticket/2238 http://localhost/ticket/2238 Report ross <p> Implement the EC/ODP template for RDF to match the requested schemas. </p> <p> Using the previous work on allowing for templatised RDF representations we should implement the proposed schema for EC/ODP. </p> <p> We also agreed to allow upload of RDF as an extra (raised to top level in package schema) which will instead be used for the RDF template. On insertion extra CKAN triples will be added... initially this is just owl:sameAs although we're awaiting EC/ODP agreement on which triples we should add to show the CKAN link. </p> http://localhost/ticket/2239 http://localhost/ticket/2239 Report ross <p> Document the ideas raised on 14/03 for <a class="missing wiki">CityData?</a> and distribute to team for discussion </p> http://localhost/ticket/2240 http://localhost/ticket/2240 Report seanh <p> ...but only if it can be done reasonably quickly. </p> http://localhost/ticket/2241 http://localhost/ticket/2241 Report seanh http://localhost/ticket/2242 http://localhost/ticket/2242 Report seanh http://localhost/ticket/2243 http://localhost/ticket/2243 Report toby <ul><li>cache admins etc </li><li>aim to move check_access so independent of logic function </li><li>general spruce up </li></ul> http://localhost/ticket/2244 http://localhost/ticket/2244 Report johnglover http://localhost/ticket/2245 http://localhost/ticket/2245 Report johnglover <ul><li>probably needs a new converter, as needs to be usable via API as 'published_by'. </li></ul> http://localhost/ticket/2246 http://localhost/ticket/2246 Report zephod <p> From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. <a class="ext-link" href="http://i.imgur.com/KJaqz.png"><span class="icon">​</span>http://i.imgur.com/KJaqz.png</a> </p> http://localhost/ticket/2247 http://localhost/ticket/2247 Report johnglover <p> Should describe implementing IDatasetForm, subclassing controllers is deprecated </p> http://localhost/ticket/2248 http://localhost/ticket/2248 Report kindly <p> Need google spreadsheet app to work with the new datastore. This should simplify it a lot. </p> http://localhost/ticket/2249 http://localhost/ticket/2249 Report kindly http://localhost/ticket/2250 http://localhost/ticket/2250 Report kindly <p> Page views and Resources clicks need to be tracked. </p> <h2 id="UserStories">User Stories</h2> <p> US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded. </p> <p> US1a Next to a resource or a dataset see how often it has been downloaded / viewed </p> <p> US1b I want to see datasets or resources ranked by most downloaded or viewed </p> <p> US1c See a trend graph for a dataset (and resources) </p> <p> Adminstrative Dashboard (?) </p> <ul><li>I want to see the traffic breakdown by country to my site ... </li><li>Ditto for browser type, language, etc etc </li><li>I want to see it graphed over time ... </li></ul><h2 id="ImplementationDetails">Implementation Details</h2> <ol><li>How do we store this data in CKAN? </li><li>How do we track (and store)? </li><li>How do we display </li></ol><ul><li>Config option ckan.status.enabled = False (by default) </li></ul><h3 id="StoringData">Storing Data</h3> <p> How does ckanext-googleanalytics do this? Current table: </p> <p> package_id | count_recent | count_total </p> <p> Move to a new stats_summary table </p> <p> id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value </p> <p> Do we store this data into the search (solr) so we can search by it? </p> <h3 id="DisplayingData">Displaying Data</h3> <ul><li>Helper functions / dictize: <ul><li>Helper function: h.stats_get(object_type, id, stats_type) <ul><li>h.stats_top_ranked(object_type, number) -&gt; returns object_dicts or just labels or ... </li></ul></li><li>Change to dictize </li></ul></li><li>Location in the default theme (do we show for example in search results too!) </li><li>Support for ranking by most popular in search? </li></ul><h3 id="TrackingData">Tracking Data</h3> <ul><li>Our own solution (just write to site_tracking) </li><li>Google analytics (plus extension for retrieving data) &lt;- would need a refactor </li><li>Piwiki </li></ul><h4 id="OwnSolution">Own Solution</h4> <p> (For later: not as part of this ticket probably) </p> <p> site_tracking table </p> <p> id | url | timestamp | action (page_view, resource_download) | </p> <ul><li>Make javascript to make request to ckan to store clicks and page views. </li><li>Add middleware so these requests do not go through pylons and just store data quickly. </li></ul> http://localhost/ticket/2251 http://localhost/ticket/2251 Report amercader <p> See trace below. </p> <p> As in <a class="closed ticket" href="http://localhost/ticket/2231" title="defect: Base harvester broken on 1.6.1 (closed: fixed)">#2231</a>, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context. </p> <pre class="wiki"> File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage self.import_gemini_object(harvest_object.content) File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object package = self.write_package_from_gemini_string(unicode_gemini_string) File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin package = self._create_package_from_data(package_dict, package = package) File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat package_dict = action_function(context, package_dict) File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update data, errors = validate(data_dict, schema, context) File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate converted_data, errors = _validate(flattened, schema, context) File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate convert(converter, key, converted_data, errors, context) File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert converter(key, converted_data, errors, context) File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license if data[('license_id',)]== '__extra__': KeyError: ('access_constraints',) 2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',) </pre> http://localhost/ticket/2252 http://localhost/ticket/2252 Report toby <p> Somewhere for CMAP stuff not in other tickets </p> <p> need to create some general tickets </p> <ul><li>template changes </li><li>general demo server setup </li></ul> http://localhost/ticket/2253 http://localhost/ticket/2253 Report toby <p> TD look at issues and then email ckan-dev </p> <p> meeting notes from 21-3-2012 general ideas non specific </p> <p> Problem: </p> <ul><li>We want to remove the Transformer usage due to performance and its close tie to genshi </li><li>Want a standard way for core code and extensions to put code into the templates where *templates request it* (templates control insertion rather than extensions ...) </li><li>Stacking versus overwriting on entry points <ul><li>Relationship to h.requires('jquery', version) - <a class="ext-link" href="http://codex.wordpress.org/Function_Reference/wp_enqueue_script"><span class="icon">​</span>http://codex.wordpress.org/Function_Reference/wp_enqueue_script</a> </li></ul></li></ul><p> Options: </p> <ul><li>placeholders or helper methods </li><li>{{user _info}} {{sub_menu}} is this equivalent to &lt;span class="insert-user-info"&gt;&lt;/span&gt; <ul><li>[xyz arguments] -&gt; h.load_snippet('xyz', ...) </li></ul></li><li>${h.snipit()} </li></ul> http://localhost/ticket/2254 http://localhost/ticket/2254 Report ross <p> [x] The publisher_form extension and the 'publisher' component of the system will be renamed to organizations to both make its functionality clear, and to differentiate it from the publish auth profile. </p> <p> Ownership of datasets is important and this ticket will enforce membership for a dataset to a single individual organization. The workflow for adding a dataset to a publisher/organization will stay the same but there will be no mechanism for adding existing datasets to an organization from within the organization forms. </p> <blockquote> <p> [x] New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below) </p> </blockquote> <blockquote> <p> [x] Datasets created from direct access to /dataset/new will have a drop down showing all of the organizations available to the current user. </p> </blockquote> <blockquote> <p> [x] When editing a dataset the organization can only be changed if the current user is the admin of the current organization, and only then to one of the organizations available to the current user. </p> </blockquote> <p> </p> <blockquote> <blockquote> <p> Only Organization admin can see the edit Organization form. </p> </blockquote> </blockquote> <p> [x] Privacy will be enforced by changing the dataset form to have a public/private switch next to the organization drop-down which will be set as the capacity with which the dataset is added to an Organization. Private means visible only within the Organization, and should be the default. </p> <p> [x] Search will be affected as the public/private status of a dataset will need to be added to the schema </p> <p> [x] Only public datasets showing in site search, and private showing in organization dataset search (if user is within the organization). </p> <p> [x] Make sure resources are restricted in visibility if the parent dataset is private </p> <p> [x] Documentation should be updated to reflect these changes. </p> <p> [x] Verify, then add, then re-add to documentation. </p> <ul><li>A User joins or creates an Organization </li></ul><ul><li>If creator of the Organization User becomes administrator of the Organization otherwise they become a Member. </li></ul><ul><li><del>Creation of new Organization must be approved by a System Administrator</del> </li></ul><ul><li>New Members must be approved by the Organization Administrator </li></ul> http://localhost/ticket/2255 http://localhost/ticket/2255 Report toby <p> make core independent of TDH and make core as minimal app </p> http://localhost/ticket/2256 http://localhost/ticket/2256 Report toby <p> look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc </p> http://localhost/ticket/2257 http://localhost/ticket/2257 Report rgrp <ul><li>Customize text at bottom of forms </li><li>Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...) </li></ul> http://localhost/ticket/2258 http://localhost/ticket/2258 Report johnglover <p> Some recent changes caused things to break in ckanext-ecportal: </p> <ul><li>change from blueprint to bootstrap </li><li>other style / css changes (eg: tables, resource view notes field ID, etc. Forms are also broken, but that will be addressed in <a class="closed ticket" href="http://localhost/ticket/2262" title="enhancement: Update ecportal form and dataset style/ux to match CKAN master (closed: fixed)">#2262</a>) </li><li>change to going through db_to_form_schema on package_show </li></ul> http://localhost/ticket/2259 http://localhost/ticket/2259 Report johnglover http://localhost/ticket/2260 http://localhost/ticket/2260 Report kindly <p> For the ecportal case we need links to be like the following </p> <p> (%host)/open-data/en/data/(.*)<br /> (%host)/open-data/fr/data/(.*) </p> http://localhost/ticket/2261 http://localhost/ticket/2261 Report johnglover http://localhost/ticket/2262 http://localhost/ticket/2262 Report amercader <p> Don't provide a field in the form or spreadsheet import for this, just use dataset.metadata_modified. </p> http://localhost/ticket/2263 http://localhost/ticket/2263 Report seanh <p> Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated. </p> http://localhost/ticket/2264 http://localhost/ticket/2264 Report dread <p> When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar. </p> <p> SOLR has a feature for finding documents similar to a particular document, called 'More Like This'. </p> <p> We would like this for DGU. </p> http://localhost/ticket/2265 http://localhost/ticket/2265 Report dread <p> Nothing happens if you set resources=[] or resources=null. </p> http://localhost/ticket/2266 http://localhost/ticket/2266 Report rgrp <p> Add generator=ckan tag in head of templates to identify site as generated by CKAN </p> http://localhost/ticket/2267 http://localhost/ticket/2267 Report amercader <p> Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach. </p> http://localhost/ticket/2268 http://localhost/ticket/2268 Report johnglover <ul><li>code review with D.R. for ckanext-ecportal </li><li>merge and test all multilingual code </li><li>fix server issues: routing, Git access to CKAN repository </li></ul><p> Bugs to fix: </p> <ul><li><del>rename groups to publishers</del> </li><li><del>update home page layout (recent updates along side publishers)</del> </li><li><del>don't add 'en' to main menu links if english</del> </li><li><del>resource edit page, fix missing images</del> </li><li><del>change dataset 'settings' to 'edit'</del> </li><li><del>translate 'published by' drop-down list</del> </li><li><del>fix JS on 'add a dataset page'</del> </li><li><del>prepopulate 'published_by' when clicking 'add a dataset' from group</del> </li><li><del>chosen broken (dataset edit) if in different language</del> </li></ul> http://localhost/ticket/2269 http://localhost/ticket/2269 Report johnglover <ul><li>get up to speed with <a class="missing wiki">DataExplorer?</a> code (and dependencies like Backbone) </li></ul> http://localhost/ticket/2270 http://localhost/ticket/2270 Report amercader <p> When importing a CSV file for a certain publisher, records don't come up on the search index. After some investigation this has tunrn out to be caused by the records being deleted. We need to investigate if this was caused by the import or just made by the users. </p> http://localhost/ticket/2271 http://localhost/ticket/2271 Report seanh http://localhost/ticket/2272 http://localhost/ticket/2272 Report seanh http://localhost/ticket/2273 http://localhost/ticket/2273 Report seanh http://localhost/ticket/2274 http://localhost/ticket/2274 Report seanh <p> Groups should be able to specify a URL to an image file for a group logo, the image will be shown on the group page. </p> <p> <del>Actually uploading an image file (rather than giving a URL) and cropping and resizing the file will be done later, for now just give a URL.</del> (later ticket) </p> <p> <del>Further customisation of group pages (CSS, templates) also for later.</del> (not sure if we really want this, but in a later ticket if so) </p> http://localhost/ticket/2275 http://localhost/ticket/2275 Report seanh http://localhost/ticket/2276 http://localhost/ticket/2276 Report icmurray <p> Atom feeds have been implemented in core in <a class="closed ticket" href="http://localhost/ticket/1593" title="enhancement: [super] Create feeds from search results (closed: fixed)">#1593</a>. The IATI-specific implementation can now be removed when IATI move to CKAN &gt;= 1.6.1 </p> http://localhost/ticket/2277 http://localhost/ticket/2277 Report rgrp <p> Major UX gotcha reported by several people is that <a class="missing wiki">DataStore?</a> is disabled by default and must be enabled for <a class="missing wiki">DataStorer?</a> to run and import data. </p> <p> Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want. </p> http://localhost/ticket/2278 http://localhost/ticket/2278 Report rgrp <p> Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this. </p> <p> This is documentation / example only (no code!) </p> http://localhost/ticket/2279 http://localhost/ticket/2279 Report johnglover http://localhost/ticket/2280 http://localhost/ticket/2280 Report dread <p> Make the search results pagination styled by Bootstrap. </p> <p> Needs putting into a &lt;ul&gt; structure. Current page and '..' need become links that are styled disabled. </p> <p> Wanted by DGU. </p> http://localhost/ticket/2281 http://localhost/ticket/2281 Report toby <p> This was part of <a class="closed ticket" href="http://localhost/ticket/2251" title="enhancement: Internal analytics for ckan. (closed: fixed)">#2251</a> but is now it's own ticket. </p> <p> we need clarity on this. The use cases so far have been possible limiting api calls for users. </p> <p> so we want to monitor the user or ip address </p> <p> what about the api calls? </p> <p> do we want to record all calls or just some? do we want to record the type of call and to what level of detail? </p> <p> recording the url just seems to specific to me. </p> http://localhost/ticket/2282 http://localhost/ticket/2282 Report rgrp <ul><li>Basic geo map </li><li>Facets </li><li>Filtering </li><li>Pivot table support (?) </li><li>Export data (?) </li><li>Save data query and share </li><li>More and better graphing </li><li>Embedding! - /resource/../embed </li></ul><p> User stories and analysis: <a class="ext-link" href="http://ckan.okfnpad.org/feature-2283-data-viewer-v3"><span class="icon">​</span>http://ckan.okfnpad.org/feature-2283-data-viewer-v3</a> </p> http://localhost/ticket/2283 http://localhost/ticket/2283 Report ross <p> From <a class="closed ticket" href="http://localhost/ticket/2204" title="enhancement: [super] Related (Stuff) Extension (closed: fixed)">#2204</a> </p> <p> not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ... </p> <p> ### Comments </p> <p> In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ... </p> http://localhost/ticket/2284 http://localhost/ticket/2284 Report icmurray <p> Allow a user to embed the data viewer in their webpage. </p> <ul><li>[x] Embedding widget [1d] </li></ul><ul><li>[x] The embed page [1-2d] </li></ul><ul><li>[x] Documentation and Announce [0.5d] </li></ul><p> Depends upon: </p> <ul><li>[x] Recline improvements <a class="closed ticket" href="http://localhost/ticket/2285?d" title="enhancement: Data Previewer / Viewer v3 : Embedding! - /resource/../embed (closed: fixed)">?d</a> </li></ul><p> All the above, plus user stories are fleshed out in <a class="missing changeset" title="No default repository defined">[1]</a> </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://ckan.okfnpad.org/feature-2283-data-viewer-v3"><span class="icon">​</span>http://ckan.okfnpad.org/feature-2283-data-viewer-v3</a>? </p> http://localhost/ticket/2285 http://localhost/ticket/2285 Report zephod <p> I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from <a class="missing wiki">TheDataHub?</a> core, for example.) </p> <ul><li>Switch to the latest minifed CSS + JS </li><li>QA-sweep the site, especially dataset/edit </li></ul> http://localhost/ticket/2286 http://localhost/ticket/2286 Report seanh <p> Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support. </p> <p> For example, on the user index page right now there are time period strings like "less than 1 month" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly). </p> <p> ISO date format everywhere might be a good idea. </p> <p> Need to consider local timezone issues. </p> <p> Python standard library and Babel may have useful helpers for this. </p> <p> Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page? </p> http://localhost/ticket/2287 http://localhost/ticket/2287 Report seanh <p> There are a couple of strings on the authorization group add/edit form that are not being translated: </p> <p> "Name: Please enter a value" (error message when you try to add an auth group with no name) </p> <p> "User name" </p> <p> "Save" button </p> <p> The form seems to be built using <a class="missing wiki">FormBuilder?</a>. See ckan/forms/authorizationgroup.py, builder.set_displayed_fields() </p> http://localhost/ticket/2288 http://localhost/ticket/2288 Report seanh <p> There are a couple of flash messages in ckan/controllers/user.py that show up in English, even if another language is in use, and even though the strings are marked for translation with gettext and have translations in the po and mo files: </p> <p> "[USER NAME] is now logged in" </p> <p> "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)" </p> http://localhost/ticket/2289 http://localhost/ticket/2289 Report seanh <p> The strings Tags, Res_format and Groups on e.g. the package index page should be marked for translation. These strings have been refactored in the feature-1821-multilingual-extension branch, so wait until that branch has been merged into master. </p> http://localhost/ticket/2290 http://localhost/ticket/2290 Report rgrp <p> See <a class="ext-link" href="http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670"><span class="icon">​</span>http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670</a> </p> <p> This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do). </p> <p> Assigning to David for triage and review. </p> http://localhost/ticket/2291 http://localhost/ticket/2291 Report rgrp <p> Can only add one dataset at a time now (not sure when this broke but was working ~2-3m ago). </p> <p> Assigning to seanh for review and estimation. I'd rate this as quite annoying but not absolutely critical (hence assigned priority). </p> http://localhost/ticket/2292 http://localhost/ticket/2292 Report rgrp <p> Rename of group makes all the datasets associated with the group disappear on the group listing page. (But they are still there if you look at edit). </p> <p> Suspect this is because we are using group name (rather than id!!) in the search index ... </p> <p> As group rename should be rare I'm marking this as minor though actual effect is major. </p> <p> Assigning to rossjones for review and cost assessment. </p> http://localhost/ticket/2293 http://localhost/ticket/2293 Report rgrp <p> Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients). </p> <p> Fix this by ensuring that error messsages on API get passed through correctly. </p> <p> Minor because affect is annoying but not terminal </p> http://localhost/ticket/2294 http://localhost/ticket/2294 Report icmurray <ul><li>Create Jenkins task for packaging CKAN </li><li>... and publishing to s3://apt.ckan.org </li></ul> http://localhost/ticket/2295 http://localhost/ticket/2295 Report icmurray <p> What needs doing, and how long will it take? </p> http://localhost/ticket/2296 http://localhost/ticket/2296 Report icmurray http://localhost/ticket/2297 http://localhost/ticket/2297 Report icmurray <p> Sort-by functionality was exposed through the package controller in <a class="missing changeset" title="No default repository defined">[1]</a>. But no controls were added to the search-page. </p> <ul><li>What should the sortBy controls/widget look like? </li><li>Which fields should be exposed? </li></ul><p> The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c"><span class="icon">​</span>https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c</a> </p> http://localhost/ticket/2298 http://localhost/ticket/2298 Report seanh http://localhost/ticket/2299 http://localhost/ticket/2299 Report seanh http://localhost/ticket/2300 http://localhost/ticket/2300 Report seanh <p> Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words. </p> <p> Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.: </p> <p> 'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible') </p> <p> <a class="ext-link" href="http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting"><span class="icon">​</span>http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting</a> </p> <p> We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated. </p> http://localhost/ticket/2301 http://localhost/ticket/2301 Report ross <p> Needs re-testing after the ODS re-theme. ==== </p> <p> Simple image upload, text changes and css updates for a single instance. All done on a single configuration/admin page </p> <p> See <a class="ext-link" href="http://ckan.okfnpad.org/simple-themes"><span class="icon">​</span>http://ckan.okfnpad.org/simple-themes</a> for user stories </p> <p> [x] Initial version will only take an image_url rather than allowing an upload. </p> <p> [x] New page with form for title, tagline, image loader etc. </p> <p> [x] Allow the user to paste in values for the CSS, do we really want a colour swatch? </p> <p> [x] Generate local CSS override which should *somehow* override colors in the original CSS. </p> <p> [x] Allow user to provide text for an about page. </p> http://localhost/ticket/2302 http://localhost/ticket/2302 Report ross <p> Move ECPortal rdf dump paster command into core so that it can be run on thedatahub for virtuoso install. </p> http://localhost/ticket/2303 http://localhost/ticket/2303 Report seanh <p> User story <a class="closed ticket" href="http://localhost/ticket/37" title="enhancement: Purge a Revision (i.e. purge all changes associated with that revision) (closed: fixed)">#037</a> here: <a class="ext-link" href="http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories"><span class="icon">​</span>http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories</a> </p> <p> Technical analysis here: <a class="ext-link" href="http://ckan.okfnpad.org/27"><span class="icon">​</span>http://ckan.okfnpad.org/27</a> </p> http://localhost/ticket/2304 http://localhost/ticket/2304 Report seanh <p> User story <a class="closed ticket" href="http://localhost/ticket/35" title="defect: IP Address for non-logged in users is always 127.0.0.1 when deployed ... (closed: fixed)">#035</a> here: <a class="ext-link" href="http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories"><span class="icon">​</span>http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories</a> </p> <p> Technical analysis here: <a class="ext-link" href="http://ckan.okfnpad.org/27"><span class="icon">​</span>http://ckan.okfnpad.org/27</a> </p> http://localhost/ticket/2305 http://localhost/ticket/2305 Report johnglover <p> Finalised metadata model requires several changes to our existing model: </p> <ul><li>new extra: alternative title </li><li>new extra: identifier </li><li>new extra: interop level </li><li>new extra: version description </li><li>new extra: dataset status </li><li>dataset type: change to fixed vocab </li><li>resource types: prefix with documentation/distribution/etc for clarity, add new resource types </li><li>rename update frequency to accrual periodicity </li><li>add new fixed vocab field: language coverage </li><li>resources: missing issued date - add as new core field called 'created' </li></ul><p> Also add functional tests for full metadata model. </p> http://localhost/ticket/2306 http://localhost/ticket/2306 Report icmurray <p> There are a number of features developed against 1.6.1 that need to be merged into master. </p> http://localhost/ticket/2307 http://localhost/ticket/2307 Report icmurray <p> Seanh and icmurray both touched on the html snippets in facets.html. So there's a merge issue. icmurray to merge changes in the multi-lingual branch prior to merging into master. </p> http://localhost/ticket/2308 http://localhost/ticket/2308 Report ross <p> Review organizations code with Sean and explain the key parts. </p> http://localhost/ticket/2309 http://localhost/ticket/2309 Report icmurray <p> The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed). </p> <p> Refactor this to remove duplication. </p> http://localhost/ticket/2310 http://localhost/ticket/2310 Report toby <p> get working on TDH </p> <p> look at removing stream filters </p> http://localhost/ticket/2311 http://localhost/ticket/2311 Report ross <p> <strong> DUPLICATE OF <a class="assigned ticket" href="http://localhost/ticket/2548" title="enhancement: Object ownership for groups/package (assigned)">#2548</a> </strong></p> <p> Currently datasets can only be part of a group but that is quite heavyweight when a single user wants to upload a single dataset. To resolve this it would be great if a dataset could be attached to a user directly - find out how. </p> http://localhost/ticket/2312 http://localhost/ticket/2312 Report icmurray <p> In branch feature-1821-multilingual-extension, a new faceting datastructure was introduced. This makes the old one and related functions obsolete. These have been marked as deprecated for the 1.7 release, and should be removed for the 1.8 release. </p> <ul><li>helpers.py:facet_items() </li><li>facets.html:facet_sidebar() </li><li>uses of c.facets (rather than c.new_facets) </li></ul> http://localhost/ticket/2313 http://localhost/ticket/2313 Report seanh <p> These two tests: </p> <p> ckan.tests.lib.test_helpers.<a class="missing wiki">TestHelpers?</a>.test_parse_rfc_2822_gmt_case </p> <p> ckan.tests.lib.test_helpers.<a class="missing wiki">TestHelpers?</a>.test_parse_rfc_2822_with_offset </p> <p> have been failing in certain timezones. In the UK they pass, if you're in Germany they'll fail. </p> <p> The function being tested uses datetime.datetime.fromtimestamp() which uses the local time, but the tests compare the results against hardcoded strings, so the tests cannot work in different timezones. </p> http://localhost/ticket/2314 http://localhost/ticket/2314 Report dread <p> Change login cookie from a default expiry of 50 years to 2 years. You can also uncheck a 'remember me' checkbox on the login form for the cookie to just last the session. </p> <p> Background conversation on ckan-dev: </p> <p> DR: I wonder if anyone objects to the expiry of the login cookie to be changed from 50 years to 2 years? 50 years might be appropriate for thedatahub.org, but for government sites it seems (to me) to be too lax. </p> <p> Toby: is this the repoze.who cookie? If so that seems sensible to me. </p> <p> Rufus: Definitely agree. I would also like to see introduction of a standard "remember me" checkbox (set to true by default). At the moment a login lasts forever (until you logout) automatically. </p> http://localhost/ticket/2315 http://localhost/ticket/2315 Report kindly <p> add resource_status_show action function to see the state of celery tasks. </p> <p> Need to get information from both out task status table and the celery_taskmeta table. </p> http://localhost/ticket/2316 http://localhost/ticket/2316 Report rgrp <ul><li>Add home page url </li><li>Remove group option (seen it confuse several users) - perhaps replace with tags (?) </li><li>Check form looks ok in portugese (seems to be a problem with css) </li></ul> http://localhost/ticket/2317 http://localhost/ticket/2317 Report rgrp <p> If you switch to Portugese and try to add a resource on dataset edit it will fail. This is because there is a string translation in js_strings.html that has quotes in it. When this is inserted into the file this causes a js exception which then prevents any further js processing. </p> <p> Specifically: </p> <pre class="wiki">CKAN.Strings.youHaveUnsavedChanges = "Você tem alterações não salvas. Certifique-se de ter clicado "Salvar Alterações" abaixo antes de sair desta página."; Uncaught SyntaxError: Unexpected identifier </pre><p> To fix is simple i imagine: need to escape " correctly in js_strings.html (or translations used). </p> <p> I have temporarily patched this live in order to allow the hackday here to continue. </p> http://localhost/ticket/2318 http://localhost/ticket/2318 Report rgrp <p> Use jquery chosen? </p> http://localhost/ticket/2319 http://localhost/ticket/2319 Report rgrp <p> Some datasets become obsolete: e.g. <a class="ext-link" href="http://datahub.io/dataset/uncdb"><span class="icon">​</span>http://datahub.io/dataset/uncdb</a> Do not want to delete them but want them marked as archived or deprecated or something. </p> http://localhost/ticket/2320 http://localhost/ticket/2320 Report rgrp <p> Would make them much nicer but requires: </p> <ul><li>Ensure uniqueness (or deal with ambiguity) </li><li>(?) Slug/names versus titles <ul><li>Aside: what is currently called name should probably be title </li></ul></li></ul> http://localhost/ticket/2321 http://localhost/ticket/2321 Report rgrp <h2 id="Raw">Raw</h2> <p> /raw or /download (latter may be better) </p> <pre class="wiki">/dataset/{id}/resource/{resource-id}/raw =&gt; redirects to resource url (download) </pre><h2 id="API">API</h2> <pre class="wiki">/dataset/{id}/resource/{resource-id}/api[/....] =&gt; either redirects or directly serves /api/data/{resource-id} </pre><h2 id="ViewerExplorer">Viewer (Explorer?)</h2> <p> Do data viewer but without anything else showing (full-screen?). this would not be just the data explorer but other cases too (including iframe etc ...) </p> <pre class="wiki">/dataset/{id}/resource/{resource-id}/viewer Resource (pre)-viewer without anything else (perhaps include resource title) </pre><hr /> <p> Assigning for kindly for review. </p> <p> These would take about 20m to implement and would be *very* convenient from my experience of using the data API. </p> <hr /> <p> Implementation at enhancement-2322-convenience-urls </p> http://localhost/ticket/2322 http://localhost/ticket/2322 Report ross <p> Apparently auto-complete in Organizations may not work. Check and fix. </p> <p> See <a class="ext-link" href="https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5"><span class="icon">​</span>https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5</a> </p> http://localhost/ticket/2323 http://localhost/ticket/2323 Report amercader <p> The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version. </p> <p> Also, remove the 'API' link (to /api/1) as pointless and misleading </p> http://localhost/ticket/2324 http://localhost/ticket/2324 Report dread <p> Also need to fix bug where 'Other' is never highlighted. </p> http://localhost/ticket/2325 http://localhost/ticket/2325 Report toby <p> create permalinks </p> <p> eg \dataset\~a5-DQ\nice-name </p> <p> ~XXXXX defines the permalink the nice-name is ignored </p> <ul><li>packages table needs a new field for this data </li></ul><ul><li>existing datasets need updating </li></ul><ul><li>new datasets need the identifier creating </li></ul><ul><li>routing needs updating </li></ul><ul><li>needs to work for all urls like edit </li></ul><p> Once this is implemented we can decide when we will use permalinks vs the normal urls. </p> http://localhost/ticket/2326 http://localhost/ticket/2326 Report kindly <ul><li>Add Ascii folding filter to text fields. </li><li>Add capacity field for public, private access. </li><li>Add title_string so you can sort alphabetically on title. </li><li>Fields related to analytics, access and view counts. </li><li>Add data_dict field for the whole package_dict. </li></ul><p> Actual implementation of data_dict adding will be added later. </p> http://localhost/ticket/2327 http://localhost/ticket/2327 Report rgrp <p> At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page? </p> http://localhost/ticket/2328 http://localhost/ticket/2328 Report rgrp <p> E.g. on dataset page, on revision page, on user page and on search results. </p> http://localhost/ticket/2329 http://localhost/ticket/2329 Report icmurray <p> Whitelist any GETable api actions, and optionally construct the query from url params rather than body. </p> http://localhost/ticket/2330 http://localhost/ticket/2330 Report rgrp <p> Appears current default search in CKAN ORs terms rather than ANDing them (i.e. adding more terms increasing number of items found rather than reducing it). </p> <p> Not sure when this crept in or if it has been there for a long time. </p> http://localhost/ticket/2331 http://localhost/ticket/2331 Report rgrp <p> A place to list crucial fixes for v1.7 release: </p> <p> Related extension: </p> <p> All not complete now moved to <a class="closed ticket" href="http://localhost/ticket/2347" title="enhancement: Related stuff dashboard and changes (closed: fixed)">#2347</a> </p> <ul><li><del>Related Item has comments on index page (they should not show up here -- why is this happening!)</del> </li><li><del>Related item should be to left of history (history could in fact move to RHS)</del> <ul><li>What about losing icons since we duplicate for related item </li></ul></li><li>Related item plus sign is very poorly aligned within box <ul><li>Add related items should be in drop down like for resources (?) to make it easy to add items </li><li>Hover should show a title that gives some information about what related are (not at all clear) </li></ul></li><li>Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket <a class="closed ticket" href="http://localhost/ticket/2204" title="enhancement: [super] Related (Stuff) Extension (closed: fixed)">#2204</a> </li></ul><p> Search results: </p> <ul><li><a class="reopened ticket" href="http://localhost/ticket/2331" title="defect: Search should AND terms not OR terms (reopened)">#2331</a> - search should add not or </li><li><del>Search results should be similar across group pages and normal search results (suggest adopting group page style)</del> <ul><li>Search results resource link should link to resource page rather than url for that resource </li><li>Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?) </li></ul></li></ul><p> Data viewer: </p> <ul><li><del>Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) <a class="ext-link" href="https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07"><span class="icon">​</span>https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07</a></del> <ul><li>Ensure still using read-only (this seems to have been lost in upgrade) </li></ul></li><li>Do not show failed preview with red-bar (esp when just empty datastore ...) </li></ul><blockquote> <blockquote> <blockquote> <p> a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) <a class="missing wiki">DataStore?</a> enabled is off by default and only turned on either explicitly by user or by <a class="missing wiki">DataStorer?</a> on successful store ... </p> </blockquote> </blockquote> </blockquote> <ul><li>(? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state) <ul><li>Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...) </li><li>Do <a class="closed ticket" href="http://localhost/ticket/2322" title="enhancement: Convenience offset on end of resource urls: /raw, /api, /viewer (closed: fixed)">#2322</a> ... </li></ul></li><li>Decent powered by icon for the Embed for the <a class="missing wiki">DataHub?</a> (make this text customizable and allow html ...) </li><li>Embed link uses uuid rather than dataset name (this goes to perma-url question ...) </li><li>Data viewer embed url includes all the resource info </li></ul><p> Dataset page: </p> <ul><li>Put home page link somewhere more prominent (e.g. the top of the sidebar) </li><li>Put author more prominent (at top right) </li><li>(? - new work) List CKAN owners at top right in sidebar </li><li>Make resources stand out more </li><li>Rename resources tab and heading to Data &amp; Resources </li></ul><p> Dataset edit and create: </p> <ul><li>Corrections to dataset creation form for v1.7 - <a class="closed ticket" href="http://localhost/ticket/2317" title="enhancement: Corrections to dataset creation form for v1.7 (closed: wontfix)">#2317</a> </li><li>Better auto-complete for groups on dataset edit/new page - <a class="closed ticket" href="http://localhost/ticket/2319" title="enhancement: Better auto-complete for groups on dataset edit page (closed: wontfix)">#2319</a> <ul><li>Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?) </li></ul></li><li>Description / notes text area is not wide enough. <ul><li>Should also be larger on edit page </li></ul></li><li>Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d." </li><li>Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either <ul><li>A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar) </li><li>Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...) </li></ul></li></ul><p> Resource view: </p> <ul><li>Do not show no description in no description (just show dataset info) </li></ul><p> Theme: </p> <ul><li><del>Fix spacing above page title (i.e. dataset title etc)</del> </li><li><del>Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer</del> </li><li><del>Fix subnav width on resources pages (does not extend all the way to the right like it should)</del> </li><li><del>Fix spacing of top menu and search box (do not look right)</del> </li><li>Group pages: <ul><li>view page: image / logo is really poorly locate and sized, see e.g. <a class="ext-link" href="http://datahub.io/group/wikimedia"><span class="icon">​</span>http://datahub.io/group/wikimedia</a> </li><li><del>view page: admin heading on RHS sidebar is wrongly indented</del> </li></ul></li></ul><p> Miscellaneous: </p> <ul><li>Ensure that we out-of-the-box config for CKAN has (<a class="closed ticket" href="http://localhost/ticket/2388" title="enhancement: Update &#34;out of the box&#34; settings for CKAN (closed: fixed)">#2388</a>): <ul><li>Only logged in users able to create </li><li>Creation permissions have only creators able to update a dataset </li></ul></li></ul><p> Deployments (without deployment we cannot know these are working): </p> <ul><li><del>Social extension</del> </li><li><del>Analytics</del> </li></ul> http://localhost/ticket/2332 http://localhost/ticket/2332 Report rgrp <p> This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly: </p> <ul><li>New Option: <a class="missing wiki">DataStore?</a> Table in add a resource form </li><li>URL field is disabled (or points at /api/data or something rather pointless) </li><li>Download button is disabled <ul><li>Better would be to find a way to serialize an ES table or stream it ... </li></ul></li></ul> http://localhost/ticket/2333 http://localhost/ticket/2333 Report rgrp <p> Placeholder for thinking about this ... </p> <ul><li>Allow users to configure info for CSV import (e.g. separators, quotes etc) </li></ul> http://localhost/ticket/2334 http://localhost/ticket/2334 Report seanh <p> You get a <a class="missing wiki">UnicodeDecodeError?</a>, looks like it tries to decode the ini file as ascii. This is a problem because someone might try to set their site description to something with an umlaut in it, for example. </p> http://localhost/ticket/2335 http://localhost/ticket/2335 Report seanh http://localhost/ticket/2336 http://localhost/ticket/2336 Report seanh <p> For example if you follow the source install instructions then you have development.ini, then when you go onto Post-Installation Setup it talks about std.ini. People don't realise that these files are the same. </p> <p> Either fix the docs or fix CKAN to use same ini file names in package and source installs/ </p> http://localhost/ticket/2337 http://localhost/ticket/2337 Report johnglover <p> I won't be needed for this during the current sprint, closing. </p> http://localhost/ticket/2338 http://localhost/ticket/2338 Report seanh http://localhost/ticket/2339 http://localhost/ticket/2339 Report seanh http://localhost/ticket/2340 http://localhost/ticket/2340 Report seanh http://localhost/ticket/2341 http://localhost/ticket/2341 Report seanh http://localhost/ticket/2342 http://localhost/ticket/2342 Report johnglover <ul><li>convert non-free fields to vocab fields (eg: interoperability_level) </li><li>store values as JSON files in the ecportal repo </li><li>translate output on dataset view page </li><li>add tests </li></ul> http://localhost/ticket/2343 http://localhost/ticket/2343 Report seanh <p> After checking out a commit to test from CKAN's <a class="missing wiki">GitHub?</a>, Jenkins runs a script that creates a new virtual environment and installs CKAN and its dependencies into it, and does some other necessary tasks. Jenkins then runs the tests in this virtualenv. </p> <p> The install script may have to change from one commit to the next as CKAN's install instructions change, so it would be good if the script was shipped in CKAN core. That way Jenkins will run different versions of the script depending on which commit it's testing and if the tests fail because the script is wrong then that's actually a bug that needs to be fixed in CKAN core. Also the CKAN install instructions could be simplified a lot by just having users run this script instead of doing each step by hand. </p> http://localhost/ticket/2344 http://localhost/ticket/2344 Report seanh http://localhost/ticket/2345 http://localhost/ticket/2345 Report johnglover <p> Changes: </p> <ul><li>tidy up activity feed on home page or replace with search results (by modified date) </li><li><del>move ckan menu to a left sidebar (see Sam's template)</del> (not doing this now) </li><li>update templates with changes from master </li><li>use local image in language dropdown (ecportal.js) </li><li>update dataset read template </li></ul><p> To discuss: </p> <ul><li>should we rename 'organization' to 'publisher'? Requires updating all templates and the url mapping. </li></ul> http://localhost/ticket/2346 http://localhost/ticket/2346 Report ross <p> Some additions to the related stuff functionality to make it more useful post the 1.7 release. </p> <p> [x] Edit item (after checking auth) </p> <p> [x] My related things on my user page </p> <blockquote> <p> [x] Show user created related things on the users' page with links direct to the related page of that dataset. {0.5d} </p> </blockquote> <p> </p> <p> [x] Related stuff usage counts { 1d } </p> <blockquote> <p> Record the number of times the link for a related item is clicked. This is likely to mean that the link fires to call CKAN and then redirects to the actual URL. </p> </blockquote> <p> [x] Un-attached related items {0.25d} </p> <blockquote> <p> Allow related items to be created without being attached to datasets. This may mean that the dataset field is optional in the API. </p> </blockquote> <p> [ ] Dashboard for related </p> <blockquote> <p> A dashboard for related items, site-wide (not package specific) and orderable by date, views, {rating}, popularity of the dataset. Would be useful (possibly) for site-admin to mark one or more as 'featured' </p> </blockquote> <blockquote> <p> [x] Dashboard + Controller (Related) {1d} </p> </blockquote> <blockquote> <p> [x] Ordering at logic layer (date, views, rating, popularity of package, featured) {1d} </p> <blockquote> <p> [x] date [x] views [x] featured </p> </blockquote> </blockquote> <blockquote> <p> [x] 'Featured' related item with mechanism to fetch it. {0.5d} </p> </blockquote> <blockquote> <blockquote> <p> [ ] UI within dashboard for sysadmins to check/uncheck feature status {0.5d} </p> </blockquote> </blockquote> <p> [x] Changing auth for related. </p> <blockquote> <p> Currently auth is based on deletion being possible by dataset owner and the related owner but it may be that a lot of the auth for related items should be configured to have more restrictive auth. </p> </blockquote> <blockquote> <p> [x] Implement auth changes for publisher auth profile. { 0.5d } </p> </blockquote> <p> [x] General issues </p> <blockquote> <p> [x] Serverside validation of URLs </p> </blockquote> <blockquote> <p> [x] Activity updates </p> </blockquote> <blockquote> <p> [x] Make optional </p> </blockquote> <blockquote> <p> [x] Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket <a class="closed ticket" href="http://localhost/ticket/2204" title="enhancement: [super] Related (Stuff) Extension (closed: fixed)">#2204</a> </p> </blockquote> <p> [ ] UI Changes </p> <blockquote> <p> [ ] General tidy up, fix form etc (icon for menu bar) </p> </blockquote> <blockquote> <p> [ ] Highlight featured items py:if=&#34;related.featured" </p> </blockquote> <blockquote> <p> [ ] Show either view count or popularity ${ related.view_count } </p> </blockquote> <blockquote> <p> [ ] A user's related items should be on their home page </p> </blockquote> <blockquote> <p> [ ] Related item plus sign is very poorly aligned within box </p> </blockquote> <blockquote> <p> [ ] Add related items should be in drop down like for resources (?) to make it easy to add items </p> </blockquote> <blockquote> <p> [ ] Hover should show a title that gives some information about what related are (not at all clear) </p> <blockquote> <p> "Apps, Ideas, Visualization and other material related to this dataset" </p> </blockquote> </blockquote> <p> [x] Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. "Apps, Ideas, Visualization and other material related to this dataset") -- btw i thought this was in <a class="closed ticket" href="http://localhost/ticket/2332" title="enhancement: Fixes for v1.7 release (closed: fixed)">#2332</a> but realize it wasn't (apologies for that!) </p> <ul><li>Went with Apps &amp; Ideas (for now) </li></ul><p> [x] Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like <a class="ext-link" href="http://docs.ckan.org/en/ckan-1.7/commenting.html"><span class="icon">​</span>http://docs.ckan.org/en/ckan-1.7/commenting.html</a> (and probably coming right after that in the ToC) which says </p> <ul><li>What it does </li><li>How to enable (and perhaps includes a nice-screenshot!) </li><li>How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...) </li><li>Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...) </li></ul><p> [ ] Clearly mark the old extension as deprecated from v1.7 forward: <a class="ext-link" href="https://github.com/okfn/ckanext-apps"><span class="icon">​</span>https://github.com/okfn/ckanext-apps</a> (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7) </p> http://localhost/ticket/2347 http://localhost/ticket/2347 Report ross <p> Start a Java library for interacting with the CKAN Action API. </p> <p> Currently requires support for Groups, Tags, Resources, and Search. </p> <p> An example app would be very useful. </p> http://localhost/ticket/2348 http://localhost/ticket/2348 Report ross <p> [x] Redeploy Sparql endpoint </p> <blockquote> <p> Done at linkeddata.openlinksw.com/sparql </p> </blockquote> <p> [x] Make sure it is available at datahub.io/sparql or semantic.datahub.io/sparql </p> <p> [x] Generate daily dumps </p> <p> [x] Make dumps available via web </p> <p> [x] Notify Hugh once running </p> <p> [x] Announce to LD guys </p> <p> [x] Think about how we can apply this to publicdata.eu (can we do the same?) ... (being done elsewhere) </p> http://localhost/ticket/2349 http://localhost/ticket/2349 Report icmurray <p> When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to <a class="missing wiki">PackageController?</a> subclasses to do form validation/processing. </p> <p> Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override. </p> <p> NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&amp;Cs. </p> <ul><li>[ ] Add hooks </li><li>[ ] Use in ckanext-example </li></ul> http://localhost/ticket/2350 http://localhost/ticket/2350 Report icmurray <p> Install and document (step-by-step) CKAN (and dependencies, and extensions) onto VM on s031.okserver.org which has CentOS 6.2 installed. Paying particular attention to the fact that we won't have root access on the staging nor production servers, and so a) anything that requires root access must be documented carefully; and b) we want to be able to perform maintenance without being root. </p> <h2 id="Maintenance">Maintenance</h2> <p> We want to have <em>non-root</em> access to: </p> <ul><li>CKAN logs; (relevant) apache logs; qa process' logs; rdf cron logs </li><li>restarting CKAN </li><li>restarting qa processes </li><li>changing/disabling rdf cronjob </li><li>our database (through sql) </li><li>restarting solr </li></ul><h2 id="CKANandDependencies">CKAN and Dependencies</h2> <ul><li>CKAN installed from source </li><li>Required plugins: synchronous_search ecportal ecportal_form organizations multilingual_dataset multilingual_group multilingual_tag qa </li></ul><h2 id="Extraconfiguration">Extra configuration</h2> <ul><li>uploads </li><li>analytics (toby) </li><li>qa </li><li>rdf cron </li><li>caching </li><li>datastore </li></ul><h2 id="Thingstoremember">Things to remember</h2> <ul><li>Use ckanext/multilingual/solr/schema.xml and stopword files </li><li>Use nginx as proxy. </li></ul> http://localhost/ticket/2351 http://localhost/ticket/2351 Report seanh http://localhost/ticket/2352 http://localhost/ticket/2352 Report ross <p> Provide some provisional user stories for the CMS/Blogging feature. Should include some of the functionality from wordpresser but be more generic. </p> http://localhost/ticket/2353 http://localhost/ticket/2353 Report aron.carroll <p> Try and increase the page load times for the front end. </p> <h1 id="Now">Now</h1> <ul><li>Always include scripts at the bottom </li><li>Include CSS at the top </li><li>Load fonts with JS </li><li>Concatenate and minify <a class="missing wiki">JavaScript?</a> files </li><li>Create a build script to load dependancies seperately </li><li>Enable Gzip compression on resources </li><li>Enable future cache headers on resources </li></ul><h1 id="Next">Next</h1> <ul><li>Support serving assets from a CDN </li><li>Look into optimising the CSS - Remove large portions of unused CSS </li><li>Remove extreaneous libraries, can jQuery UI be replaced with bootstrap? </li></ul><h1 id="Tools">Tools</h1> <ul><li>Webkit Inspector Audits </li><li>YSlow Firebug Plugin - <a class="ext-link" href="http://yslow.org/"><span class="icon">​</span>http://yslow.org/</a> </li><li>Speed Tracer - <a class="ext-link" href="https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl"><span class="icon">​</span>https://chrome.google.com/webstore/detail/ognampngfcbddbfemdapefohjiobgbdl</a> </li></ul><h1 id="ContinuousIntegration">Continuous Integration</h1> <p> YSlow can be hooked up to Jenkins via Phantom JS </p> http://localhost/ticket/2354 http://localhost/ticket/2354 Report toby <ul><li>Do an analysis of why upgrades are problematic </li><li>What can we do better </li><li>Decide what to implement </li></ul> http://localhost/ticket/2355 http://localhost/ticket/2355 Report aron.carroll http://localhost/ticket/2356 http://localhost/ticket/2356 Report aron.carroll <p> Should minify and concatenate <a class="missing wiki">JavaScript?</a> and CSS as part of <a class="closed ticket" href="http://localhost/ticket/2354" title="enhancement: Front end performance improvements [super] (closed: wontfix)">#2354</a> </p> http://localhost/ticket/2357 http://localhost/ticket/2357 Report amercader <p> Various stuff related with release 1.7: </p> <ul><li>Review patches sent to the release branch </li><li>Regularly update TDH with latest changes </li><li>write changelog, announce email, etc </li><li>... </li></ul> http://localhost/ticket/2358 http://localhost/ticket/2358 Report amercader <p> Tom Kralidis kindly told us on IRC <a class="missing changeset" title="No default repository defined">[1]</a> that the latest trunk of pycsw includes support for wsgi and different backends, which sounds like could be really useful for CKAN integration. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://irclogs.okfn.org/ckan/2012-04-25.txt"><span class="icon">​</span>http://irclogs.okfn.org/ckan/2012-04-25.txt</a> 17:16 </p> http://localhost/ticket/2359 http://localhost/ticket/2359 Report amercader <p> I've been writing a doc with different issues regarding INSPIRE support in CKAN. It is mostly done, except for some sections on CSW support. Depends on <a class="closed ticket" href="http://localhost/ticket/2359" title="task: Investigate pycsw integration (closed: fixed)">#2359</a> </p> http://localhost/ticket/2360 http://localhost/ticket/2360 Report amercader <p> ditto </p> http://localhost/ticket/2361 http://localhost/ticket/2361 Report kindly <p> Improve documentation for plugins and add examples. Could use ckanext-example as a base. </p> http://localhost/ticket/2362 http://localhost/ticket/2362 Report kindly <p> Need better documentation on best practices in making page loads faster for non logged in users. </p> http://localhost/ticket/2363 http://localhost/ticket/2363 Report ross <p> Need to investigate alternative payment gateways </p> http://localhost/ticket/2364 http://localhost/ticket/2364 Report ross <p> What would as a multisite CKAN look like? </p> <p> This is really part of the work around turnkey/hosted CKAN </p> http://localhost/ticket/2365 http://localhost/ticket/2365 Report amercader <p> 1.7 will hide the link on the footer and put a deprecation notice on the related pages, but they should be completely removed on 1.8. </p> http://localhost/ticket/2366 http://localhost/ticket/2366 Report aron.carroll <p> Currently an eye and a number, needs to look a little nicer. </p> http://localhost/ticket/2367 http://localhost/ticket/2367 Report seanh <p> Solr install instructions install the wrong version </p> http://localhost/ticket/2368 http://localhost/ticket/2368 Report seanh http://localhost/ticket/2369 http://localhost/ticket/2369 Report seanh <p> The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there. </p> http://localhost/ticket/2370 http://localhost/ticket/2370 Report toby <p> aims </p> <ul><li>allow templates to specify needed css/js </li></ul><ul><li>allow split js files to be bundled (including plugin supplied if possible - stage 2) </li></ul><ul><li>serve minified in production/ non-minified in dev </li></ul><ul><li>possibly look at doing the minification via paster - maybe as different ticket </li></ul> http://localhost/ticket/2371 http://localhost/ticket/2371 Report aron.carroll <p> This should cover formatting and best practices for HTML, CSS and <a class="missing wiki">JavaScript?</a> </p> <p> Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch. </p> http://localhost/ticket/2372 http://localhost/ticket/2372 Report dread <p> e.g. 'Cabinet Office' returns 0 results, whereas 'Cabinet' returns 'Cabinet Office'. </p> <pre class="wiki">/api/2/util/authorizationgroup/autocomplete?q=Cabinet%20Office </pre><p> This feature only seems to be used by DGU at the moment (I think) </p> http://localhost/ticket/2373 http://localhost/ticket/2373 Report dread <p> Currently in the logic function tag_search you can specify limit and offset, but no count is returned. Therefore pagination is not possible for tag results. This is desired though. </p> http://localhost/ticket/2374 http://localhost/ticket/2374 Report aron.carroll <p> As designed by Sam Smith -&gt; <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/</a> Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same. </p> http://localhost/ticket/2375 http://localhost/ticket/2375 Report seanh <p> This needs to be fixed in the CKAN templates ("no-sidebar" should not be marked for translation) but for 1.7 just fixed it in the translation files. </p> http://localhost/ticket/2376 http://localhost/ticket/2376 Report seanh http://localhost/ticket/2377 http://localhost/ticket/2377 Report ross <p> Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( <a class="ext-link" href="http://en.wikipedia.org/wiki/Exchangeable_image_file_format"><span class="icon">​</span>http://en.wikipedia.org/wiki/Exchangeable_image_file_format</a> ) data. </p> <p> [ ] Obtain list of embedded metadata 'standards' </p> <p> [ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded. </p> <blockquote> <p> [ ] HTML? </p> </blockquote> <blockquote> <p> [ ] Image formats (exif/gif etc) </p> </blockquote> <blockquote> <p> [ ] PDF </p> </blockquote> <p> [ ] Eventually consider ... </p> <blockquote> <p> [ ] MS Office documents </p> </blockquote> <p> </p> http://localhost/ticket/2378 http://localhost/ticket/2378 Report ross <p> From: Pierre-Yves Vandenbussche </p> <p> In my use case I need to fetch every sparql endpoint associated to a dataset. With the previous version of your endpoint, I was doing this query: SELECT DISTINCT ?dataset ?endpoint ?title ?identifier WHERE { </p> <blockquote> <p> ?dataset &lt;<a class="ext-link" href="http://www.w3.org/ns/dcat#distribution"><span class="icon">​</span>http://www.w3.org/ns/dcat#distribution</a>&gt; ?distribution. ?distribution &lt;<a class="ext-link" href="http://purl.org/dc/terms/format"><span class="icon">​</span>http://purl.org/dc/terms/format</a>&gt; [ &lt;<a class="ext-link" href="http://moat-project.org/ns#taggedWithTag"><span class="icon">​</span>http://moat-project.org/ns#taggedWithTag</a>&gt; [ &lt;<a class="ext-link" href="http://moat-project.org/ns#name"><span class="icon">​</span>http://moat-project.org/ns#name</a>&gt; "api/sparql" ] ]. ?distribution &lt;<a class="ext-link" href="http://www.w3.org/ns/dcat#accessURL"><span class="icon">​</span>http://www.w3.org/ns/dcat#accessURL</a>&gt; ?endpoint. ?dataset &lt;<a class="ext-link" href="http://purl.org/dc/terms/title"><span class="icon">​</span>http://purl.org/dc/terms/title</a>&gt; ?title. ?dataset &lt;<a class="ext-link" href="http://purl.org/dc/terms/identifier"><span class="icon">​</span>http://purl.org/dc/terms/identifier</a>&gt; ?identifier. </p> </blockquote> <p> } ORDER BY ?title </p> <p> Using your new version,dcterms:title of a dataset is now a rdfs:label ... OK </p> <p> Unfortunately, I don't have the information of "api/sparql" anymore. So I can not differentiate between a dump file and a SPARQL endpoint... <span class="underline"></span> </p> <p> Add the required information to the RDF template. </p> http://localhost/ticket/2379 http://localhost/ticket/2379 Report ross <p> Provide a new test install of DataGM based on the ODS demo site being completed through June. </p> http://localhost/ticket/2380 http://localhost/ticket/2380 Report ross <p> Make sure the related item count is available on all dataset pages (inc. follow) when it is enabled. </p> http://localhost/ticket/2381 http://localhost/ticket/2381 Report amercader <p> A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk) </p> http://localhost/ticket/2382 http://localhost/ticket/2382 Report icmurray http://localhost/ticket/2383 http://localhost/ticket/2383 Report dread <p> The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by <a class="missing wiki">CreateTestData?</a> etc and will be used by DGU. </p> http://localhost/ticket/2384 http://localhost/ticket/2384 Report ross <p> Currently you can specify the capacity in the API search, this should be stripped and default to public for now. </p> http://localhost/ticket/2385 http://localhost/ticket/2385 Report seanh http://localhost/ticket/2386 http://localhost/ticket/2386 Report seanh http://localhost/ticket/2387 http://localhost/ticket/2387 Report amercader <ul><li>Only logged in users able to create </li><li>Creation permissions have only creators able to update a dataset </li><li>Synchronous search is enabled (<a class="closed ticket" href="http://localhost/ticket/1678" title="enhancement: Make synchronous search the default behaviour (closed: fixed)">#1678</a>) </li></ul> http://localhost/ticket/2388 http://localhost/ticket/2388 Report amercader <p> US-0a </p> <p> Do not show failed preview with red-bar (esp when just empty datastore ...) </p> <ul><li>Data Viewer checks if datastore is empty nad falls back to ther options if emtpy </li><li><a class="missing wiki">DataStore?</a> enabled is off by default and only turned on either explicitly by user or by <a class="missing wiki">DataStorer?</a> on successful store ... </li></ul><p> Loading/showing error reports </p> http://localhost/ticket/2389 http://localhost/ticket/2389 Report dread <p> It would be useful (well for DGU, but no doubt for core too) to have be able to create multiple users with full properties using one commit. Also to be able to create rights (<a class="missing wiki">UserObjectRoles?</a>). </p> http://localhost/ticket/2390 http://localhost/ticket/2390 Report seanh http://localhost/ticket/2391 http://localhost/ticket/2391 Report icmurray <p> An extra field on datasets called "metadata_language". The language that the meta-data is in. Drop-down list needs to be translatable. Default value is "English". </p> http://localhost/ticket/2392 http://localhost/ticket/2392 Report icmurray <p> Map licenses: given <a class="ext-link" href="http://creativecommons.org/licenses/by/2.5/{countrycode}/"><span class="icon">​</span>http://creativecommons.org/licenses/by/2.5/{countrycode}/</a> as a license value through the API, convert it to <a class="ext-link" href="http://www.opendefinition.org/licenses/cc-by"><span class="icon">​</span>http://www.opendefinition.org/licenses/cc-by</a> . Should also work if given <a class="ext-link" href="http://www.opendefinition.org/licenses/cc-by"><span class="icon">​</span>http://www.opendefinition.org/licenses/cc-by</a> . NOTE: the API may send through a list of licenses; in this situation, take the head of the list. </p> http://localhost/ticket/2393 http://localhost/ticket/2393 Report dread <p> We need to separate the CLI part of <a class="missing wiki">CreateTestData?</a> from the bit which creates test fixtures. This is because of the problem with imports and logging. Otherwise we have to do unpythonic things to get logging to work when deriving from <a class="missing wiki">CreateTestData?</a> from an extension such as ecportal / dgu. I've only introduced logging into <a class="missing wiki">CreateTestData?</a> but I think it is a good thing to have. This change doesn't cause breakages </p> http://localhost/ticket/2394 http://localhost/ticket/2394 Report dread <p> If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init. </p> <p> Can't you just modify the clean and init functions to run without extensions enabled? </p> <p> The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket. </p> http://localhost/ticket/2395 http://localhost/ticket/2395 Report seanh http://localhost/ticket/2396 http://localhost/ticket/2396 Report seanh http://localhost/ticket/2397 http://localhost/ticket/2397 Report seanh http://localhost/ticket/2398 http://localhost/ticket/2398 Report seanh <p> root@lucid32:/usr/local/feature-2375-demo-theme/pyenv/src/ckan# sudo -u www-data /usr/local/feature-2375-demo-theme/pyenv/bin/paster --plugin=ckanext-spatial spatial initdb 4326 --config=/usr/local/feature-2375-demo-theme/pyenv/src/ckan/development.ini Traceback (most recent call last): </p> <blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/bin/paster", line 9, in &lt;module&gt; </p> <blockquote> <p> load_entry_point('<a class="missing wiki">PasteScript?</a>==1.7.3', 'console_scripts', 'paster')() </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run </p> <blockquote> <p> invoke(command, command_name, options, args[1:]) </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke </p> <blockquote> <p> exit_code = runner.run(args) </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run </p> <blockquote> <p> result = self.command() </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/commands/spatial.py", line 38, in command </p> <blockquote> <p> self._load_config() </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/src/ckan/ckan/lib/cli.py", line 53, in _load_config </p> <blockquote> <p> load_environment(conf.global_conf, conf.local_conf) </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/src/ckan/ckan/config/environment.py", line 204, in load_environment </p> <blockquote> <p> plugin.configure(config) </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/plugin.py", line 60, in configure </p> <blockquote> <p> setup_model() </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/model.py", line 20, in setup </p> <blockquote> <p> define_spatial_tables(srid) </p> </blockquote> <p> File "/usr/local/feature-2375-demo-theme/pyenv/src/ckanext-spatial/ckanext/spatial/model.py", line 66, in define_spatial_tables </p> <blockquote> <p> package_extent_table = Table('package_extent', metadata, </p> </blockquote> </blockquote> <p> <a class="missing wiki">NameError?</a>: global name 'metadata' is not defined </p> http://localhost/ticket/2399 http://localhost/ticket/2399 Report amercader <p> One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed </p> <pre class="wiki">"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)" </pre> http://localhost/ticket/2400 http://localhost/ticket/2400 Report dread <p> To enable you to easily track down what is taking all the time when you make a request. </p> http://localhost/ticket/2401 http://localhost/ticket/2401 Report kindly <p> Search results are slow as we do many package_dictize. Store the package_dict in the search index so that we do not have to hit the database for each of them. </p> http://localhost/ticket/2402 http://localhost/ticket/2402 Report kindly <p> Home page loads *all* groups and dictizes them, it should only dictize the top 10 or so from the search facets. </p> http://localhost/ticket/2403 http://localhost/ticket/2403 Report rgrp <p> As a User I want an image generated for my related item automatically so that I do not have to take picture myself and upload it somewhere </p> <p> Implementation: (?) Use phantomjs or something like <a class="ext-link" href="http://snapito.com/"><span class="icon">​</span>http://snapito.com/</a> Investigate for 1.8, will be needed for TDH Oct. </p> http://localhost/ticket/2404 http://localhost/ticket/2404 Report rgrp <ul><li>Decent powered by icon for the Embed for the <a class="missing wiki">DataHub?</a>? (make this text customizable and allow html ...) <ul><li>Current icon and text looks really bad </li></ul></li><li>Embed link uses uuid rather than dataset name (this goes to perma-url question ...) </li><li>Data viewer embed url includes all the resource info (not needed -- just url and backend info ...) </li><li>Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org </li></ul> http://localhost/ticket/2405 http://localhost/ticket/2405 Report markw <p> There are a few errors or unclarities in the source install doc - see </p> <p> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html</a> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html</a> </p> http://localhost/ticket/2406 http://localhost/ticket/2406 Report markw <p> This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied': </p> <p> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html</a> </p> <p> The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer. </p> http://localhost/ticket/2407 http://localhost/ticket/2407 Report markw <p> The settings &gt; extras page for a dataset allows extra fields to be added, but only four can be added a time in a single atomic update. It would be helpful to have an 'add another' button on the form that can be used any number of times. </p> http://localhost/ticket/2408 http://localhost/ticket/2408 Report markw <p> CKAN returns HTML when an Accept header should make it return RDF/XML: </p> <p> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-May/002237.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-May/002237.html</a> </p> http://localhost/ticket/2409 http://localhost/ticket/2409 Report ross <p> Taken from <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-May/002218.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-May/002218.html</a> </p> <hr /> <p> A few months ago, the RDF provided when dereferencing a dataset's URI was very useful. It described the datasets using many of the "right" RDF vocabularies (dcterms, void, etc). </p> <p> The RDF that is returned now no longer has these nice descriptions. Instead, it seems to be a mindless transcription of the CKAN record. </p> <p> For example, the older descriptions gave: </p> <p> &lt;<a class="ext-link" href="http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states</a>&gt; </p> <blockquote> <p> void:subset [ </p> <blockquote> <p> void:target &lt;<a class="ext-link" href="http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states</a>&gt;, &lt;<a class="ext-link" href="http://thedatahub.org/dataset/geonames-semantic-web"><span class="icon">​</span>http://thedatahub.org/dataset/geonames-semantic-web</a>&gt; ; void:triples 50 ; a void:Linkset </p> </blockquote> <p> ], [ </p> <blockquote> <p> void:target &lt;<a class="ext-link" href="http://thedatahub.org/dataset/dbpedia"><span class="icon">​</span>http://thedatahub.org/dataset/dbpedia</a>&gt;, &lt;<a class="ext-link" href="http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states</a>&gt; ; void:triples 52 ; a void:Linkset </p> </blockquote> <p> ], [ </p> <blockquote> <p> void:target &lt;<a class="ext-link" href="http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states</a>&gt;, &lt;<a class="ext-link" href="http://thedatahub.org/dataset/govtrack"><span class="icon">​</span>http://thedatahub.org/dataset/govtrack</a>&gt; ; void:triples 52 ; a void:Linkset </p> </blockquote> <p> ] ; void:triples 130005 ; </p> </blockquote> <p> but now it's simply: </p> <p> &lt;<a class="ext-link" href="http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states</a>&gt; </p> <blockquote> <p> dct:relation [ </p> <blockquote> <p> rdf:value "130005" ; rdfs:label "triples" </p> </blockquote> <p> ], [ </p> <blockquote> <p> rdf:value "52" ; rdfs:label "links:dbpedia" </p> </blockquote> <p> ], [ </p> <blockquote> <p> rdf:value "50" ; rdfs:label "links:geonames-semantic-web" </p> </blockquote> <p> ], [ </p> <blockquote> <p> rdf:value "52" ; rdfs:label "links:govtrack" </p> </blockquote> <p> ] ; </p> </blockquote> <p> My old copies of the dataset descriptions mentioned a "GoLD CKAN 1.0" that was credited for the nicer modeling. </p> <p> Where did that go? Can we get it back? </p> <p> Thanks, Tim Lebo </p> <p> &lt;<a class="ext-link" href="http://semantic.ckan.net/record/91d2c0de-75a4-4bb6-b260-bc2946e1be8b"><span class="icon">​</span>http://semantic.ckan.net/record/91d2c0de-75a4-4bb6-b260-bc2946e1be8b</a>&gt; </p> <blockquote> <p> dc:source &lt;<a class="ext-link" href="http://ckan.net/"><span class="icon">​</span>http://ckan.net/</a>&gt; ; opmv:wasGeneratedBy [ </p> <blockquote> <p> opmv:used &lt;<a class="ext-link" href="http://ckan.net/api/rest/package/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://ckan.net/api/rest/package/farmers-markets-geographic-data-united-states</a>&gt;, &lt;<a class="ext-link" href="http://ckan.net/api/rest/revision/9c2545b8-ebf1-42bf-ac92-633ff5ceba3f"><span class="icon">​</span>http://ckan.net/api/rest/revision/9c2545b8-ebf1-42bf-ac92-633ff5ceba3f</a>&gt; ; opmv:wasControlledBy [ </p> <blockquote> <p> a foaf:Agent ; foaf:name "GoLD CKAN 1.0" </p> </blockquote> <p> ] ; opmv:wasPerformedAt [ </p> <blockquote> <p> a time:Instant ; time:inXSDDateTime "2012-01-31T05:34:32Z&#34;^^xsd:dateTime </p> </blockquote> <p> ] ; a opmv:Process </p> </blockquote> <p> ] ; a dcat:CatalogRecord ; dcat:dataset &lt;<a class="ext-link" href="http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states"><span class="icon">​</span>http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states</a>&gt; ; </p> </blockquote> <p> … </p> http://localhost/ticket/2410 http://localhost/ticket/2410 Report markw <p> The .ini file needs cleaning up in assorted ways which Toby will expand upon. </p> http://localhost/ticket/2411 http://localhost/ticket/2411 Report icmurray <p> When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state. </p> <p> Repro: </p> <ol><li>Go to /dataset/new </li><li>Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size... </li><li>Add another resource, doing the same thing: make one of the fields invalid. </li><li>Try to save the dataset. </li><li>The entered resource information will be lost, and a js error "Uncaught Error: Can't add the same model to a set twice,: backbone.js:586" will be thrown. </li></ol> http://localhost/ticket/2412 http://localhost/ticket/2412 Report ross <p> Take revelvant parts of autoneg ( <a class="ext-link" href="https://github.com/wwaites/autoneg.git"><span class="icon">​</span>https://github.com/wwaites/autoneg.git</a> ) so that we can remove the dependency (just for two functions) and thereby remove flup as well. </p> http://localhost/ticket/2413 http://localhost/ticket/2413 Report ross <p> Core CKAN currently has a dependency on LXML and it would be nice if we could remove it (although this *may* be indirectly through genshi). </p> http://localhost/ticket/2414 http://localhost/ticket/2414 Report markw <p> On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message: </p> <p> "The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_" </p> <p> What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...' </p> <p> But even better would be to make the login case-insensitive and allow capitals to be entered. </p> http://localhost/ticket/2415 http://localhost/ticket/2415 Report aron.carroll <p> Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json” </p> <p> See the following Basecamp thread for UI examples and discussion. </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056</a> </p> http://localhost/ticket/2416 http://localhost/ticket/2416 Report aron.carroll <p> Currently due to the data structure the search result filters are output incorrectly. Repeating the facet with each tag. </p> <p> See <a class="ext-link" href="http://s031.okserver.org:2375/dataset?tags=fibre&amp;tags=terrestrial&amp;q=Africa"><span class="icon">​</span>http://s031.okserver.org:2375/dataset?tags=fibre&amp;tags=terrestrial&amp;q=Africa</a> </p> <p> It outputs: </p> <p> Tags: fibre Tags: terrestrial </p> <p> It should be: </p> <p> Tags: fibre terrestrial </p> http://localhost/ticket/2417 http://localhost/ticket/2417 Report aron.carroll <p> Currently facets in the new demo site disappear from the sidebar once they are selected. </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset?tags=fibre&amp;tags=terrestrial&amp;q=Africa"><span class="icon">​</span>http://s031.okserver.org:2375/dataset?tags=fibre&amp;tags=terrestrial&amp;q=Africa</a> </p> <p> Instead they should be marked with an "active" class and clicking the link should remove the filter in the same way as clicking the cross in a token below the search field works. </p> http://localhost/ticket/2418 http://localhost/ticket/2418 Report aron.carroll <p> Currently for some reason I'm not getting the errors/data dict in the package/new.html template. </p> http://localhost/ticket/2419 http://localhost/ticket/2419 Report toby <p> create a depreciated decorator to wrap functions we intend to remove </p> <p> logs a warning maybe check that 'depreciated' is included in the docstring too </p> http://localhost/ticket/2420 http://localhost/ticket/2420 Report toby <p> if pills cross onto 2 lines they split which they shouldn't to see just add some tags/formats till it breaks </p> http://localhost/ticket/2421 http://localhost/ticket/2421 Report ross <p> Reported in IRC by @floapps </p> <p> In previous ckan versions i could use paster commands to remove reader rights from visitors on all packages and this would then throw the user to login screen when trying to see a dataset but, when i did this in ckan 1.7, i could still see all datasets as a visitor </p> <p> Using <a class="ext-link" href="http://docs.ckan.org/en/latest/authorization.html#permissions-publisher-mode"><span class="icon">​</span>http://docs.ckan.org/en/latest/authorization.html#permissions-publisher-mode</a> </p> <blockquote> <p> paster rights remove visitor reader package:all </p> </blockquote> <p> Also has an empty list of roles in config for default visitor roles </p> <p> I tested on my local install and could replicate with that command doing nothing. Neither affects existing or newly created datasets. </p> http://localhost/ticket/2422 http://localhost/ticket/2422 Report seanh http://localhost/ticket/2423 http://localhost/ticket/2423 Report seanh http://localhost/ticket/2424 http://localhost/ticket/2424 Report seanh http://localhost/ticket/2425 http://localhost/ticket/2425 Report seanh http://localhost/ticket/2426 http://localhost/ticket/2426 Report seanh http://localhost/ticket/2427 http://localhost/ticket/2427 Report seanh <p> This ticket depends on <a class="closed ticket" href="http://localhost/ticket/2423" title="enhancement: Get rid of CKAN's lxml dependency (closed: duplicate)">#2423</a>, <a class="closed ticket" href="http://localhost/ticket/2424" title="enhancement: Get rid of CKAN's autoneg dependency (closed: duplicate)">#2424</a>, <a class="closed ticket" href="http://localhost/ticket/2425" title="enhancement: Get rid of CKAN's flup dependency (closed: fixed)">#2425</a>, <a class="new ticket" href="http://localhost/ticket/2426" title="enhancement: Get rid of CKAN's formalchemy dependency (new)">#2426</a> and <a class="new ticket" href="http://localhost/ticket/2427" title="enhancement: Reduce cruft installed into CKAN's virtualenv due to Pylons (new)">#2427</a>. </p> <ul><li>Try to remove as many of CKAN's dependencies as possible </li></ul><ul><li>Try to update as many of the remaining ones as possible </li></ul><ul><li>Try to get rid of some dependencies that are fetched from random hg or git servers instead of pypi </li></ul><ul><li>Then try to get the order of deps in the requirements files right so that you end up with the right versions of everything </li></ul><ul><li>Each module in the dependencies file should specify the exact version, to protect us from changes due to them upgrading versions under us and breaking things or changing their own deps </li></ul><ul><li>Consider shipping dependencies with ckan instead of pip installing them? </li></ul><ul><li>This is related to moving CKAN to Ubuntu 12.04 </li></ul><ul><li>Add a test case that runs pip freeze and tests that all the right versions of deps are installed? </li></ul> http://localhost/ticket/2428 http://localhost/ticket/2428 Report seanh http://localhost/ticket/2429 http://localhost/ticket/2429 Report seanh http://localhost/ticket/2430 http://localhost/ticket/2430 Report seanh http://localhost/ticket/2431 http://localhost/ticket/2431 Report seanh <p> We want a new way of working with trac tickets and milestones etc: </p> <ul><li>We will have a single queue of tickets, ordered by priority, that we all work on </li></ul><ul><li>Tickets only get into the queue after triaging (which the devs do at regular intervals), so we and others can use new trac tickets as a dumping ground for any and all CKAN bugs and ideas without messing up the priority queue. Therefore the priority queue needs to be a special CKAN milestone or something. The devs need to regularly go through the inbox of untriaged tickets and triage them all, moving them all out of the inbox. </li></ul><ul><li>Will we also assign tickets from the priority queue to individual devs in sprint planning meetings like we currently do? </li></ul> http://localhost/ticket/2432 http://localhost/ticket/2432 Report toby <p> see <a class="ext-link" href="https://github.com/okfn/ckan/pull/5"><span class="icon">​</span>https://github.com/okfn/ckan/pull/5</a> </p> http://localhost/ticket/2433 http://localhost/ticket/2433 Report seanh http://localhost/ticket/2434 http://localhost/ticket/2434 Report seanh http://localhost/ticket/2435 http://localhost/ticket/2435 Report seanh http://localhost/ticket/2436 http://localhost/ticket/2436 Report icmurray <ul><li>[ ] Write up python coding standards as discussed at dev-meetup into CKAN docs. (Mostly written-up as part of dev-meetup notes.) </li></ul><ul><li>[ ] Collate frontend coding standards (<a class="ext-link" href="http://aron.github.com/ckan-style/styleguide/"><span class="icon">​</span>http://aron.github.com/ckan-style/styleguide/</a>) into same doc(s). </li></ul> http://localhost/ticket/2437 http://localhost/ticket/2437 Report icmurray <p> This action is not currently usable via a GET request as it relies upon a nested dict of fields. </p> http://localhost/ticket/2438 http://localhost/ticket/2438 Report icmurray <p> <tt>tag_search</tt> and <tt>tag_autocomplete</tt>: The <tt>fields</tt> argument is not available when accessing this action with a GET request. </p> http://localhost/ticket/2439 http://localhost/ticket/2439 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375"><span class="icon">​</span>http://s031.okserver.org:2375</a> </p> http://localhost/ticket/2440 http://localhost/ticket/2440 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/</a> </p> http://localhost/ticket/2441 http://localhost/ticket/2441 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending</a> </p> http://localhost/ticket/2442 http://localhost/ticket/2442 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62308209/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62308209/comments</a> </p> <p> Implementation: </p> <p> (to be added once test server is working) </p> http://localhost/ticket/2443 http://localhost/ticket/2443 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending/related"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending/related</a> </p> http://localhost/ticket/2444 http://localhost/ticket/2444 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending/related/new"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending/related/new</a> </p> <p> Needs a new endpoint at /dataset/DATASET/related/new this should have a form containing the following fields. </p> <ul><li>Title (required) </li><li>Type of item (Application|Visualisation) </li><li>Description </li><li>URL (required) </li><li>Image URL </li></ul><p> When submitted if an item is created it should redirect (303) back to /dataset/DATASET/related with a flash message saying "Related item was successfully created". </p> <p> If failed to create it should redirect (303) back to the form and populate the error messages, see add dataset for examples (<a class="ext-link" href="https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/package/new_package_form.html"><span class="icon">​</span>https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/package/new_package_form.html</a>) </p> http://localhost/ticket/2445 http://localhost/ticket/2445 Report aron.carroll <p> This is a low priority for the demo site. </p> <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/history/adur_district_spending"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/history/adur_district_spending</a> </p> http://localhost/ticket/2446 http://localhost/ticket/2446 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62234769/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62234769/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new</a> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_resource"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_resource</a> </p> http://localhost/ticket/2447 http://localhost/ticket/2447 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/group"><span class="icon">​</span>http://s031.okserver.org:2375/group</a> </p> http://localhost/ticket/2448 http://localhost/ticket/2448 Report aron.carroll <p> Discussion: </p> <blockquote> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments</a> </p> </blockquote> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/group/test-group"><span class="icon">​</span>http://s031.okserver.org:2375/group/test-group</a> </p> http://localhost/ticket/2449 http://localhost/ticket/2449 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/group/new"><span class="icon">​</span>http://s031.okserver.org:2375/group/new</a> </p> http://localhost/ticket/2450 http://localhost/ticket/2450 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/user"><span class="icon">​</span>http://s031.okserver.org:2375/user</a> </p> http://localhost/ticket/2451 http://localhost/ticket/2451 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/user/aron"><span class="icon">​</span>http://s031.okserver.org:2375/user/aron</a> </p> http://localhost/ticket/2452 http://localhost/ticket/2452 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/user/edit"><span class="icon">​</span>http://s031.okserver.org:2375/user/edit</a> </p> http://localhost/ticket/2453 http://localhost/ticket/2453 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/user/login"><span class="icon">​</span>http://s031.okserver.org:2375/en/user/login</a> </p> http://localhost/ticket/2454 http://localhost/ticket/2454 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998362/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998362/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/user/register"><span class="icon">​</span>http://s031.okserver.org:2375/en/user/register</a> </p> http://localhost/ticket/2455 http://localhost/ticket/2455 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62935295/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/user/reset"><span class="icon">​</span>http://s031.okserver.org:2375/en/user/reset</a> </p> http://localhost/ticket/2456 http://localhost/ticket/2456 Report aron.carroll <p> This includes the tag page as well for now. </p> <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/tag"><span class="icon">​</span>http://s031.okserver.org:2375/en/tag</a> </p> http://localhost/ticket/2457 http://localhost/ticket/2457 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998513/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998513/comments</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/stats"><span class="icon">​</span>http://s031.okserver.org:2375/en/stats</a> </p> http://localhost/ticket/2458 http://localhost/ticket/2458 Report johnglover <ul><li>the facet tags should only show free (non-vocab) tags </li><li>decide on what fields to include in search facet list </li></ul> http://localhost/ticket/2459 http://localhost/ticket/2459 Report johnglover http://localhost/ticket/2460 http://localhost/ticket/2460 Report aron.carroll <p> I believe this is to be merged in with site activity. </p> <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/revision"><span class="icon">​</span>http://s031.okserver.org:2375/en/revision</a> </p> http://localhost/ticket/2461 http://localhost/ticket/2461 Report johnglover http://localhost/ticket/2462 http://localhost/ticket/2462 Report johnglover http://localhost/ticket/2463 http://localhost/ticket/2463 Report johnglover <ul><li>Already done for basic publisher forms (index, read, edit), but needs to be done for additional forms (in particular, form paths like 'organization_apply' that are hardcoded into the organizations controller). </li><li>change organization mapping to /publisher </li></ul> http://localhost/ticket/2464 http://localhost/ticket/2464 Report johnglover http://localhost/ticket/2465 http://localhost/ticket/2465 Report seanh http://localhost/ticket/2466 http://localhost/ticket/2466 Report amercader <p> After moving to Bootstrap the tables on the frontend look broken. </p> http://localhost/ticket/2467 http://localhost/ticket/2467 Report amercader <p> Please see these <a class="missing wiki">GitHub?</a> issues: </p> <p> <a class="ext-link" href="https://github.com/okfn/recline/issues/130"><span class="icon">​</span>https://github.com/okfn/recline/issues/130</a> </p> <p> <a class="ext-link" href="https://github.com/okfn/recline/issues/131"><span class="icon">​</span>https://github.com/okfn/recline/issues/131</a> </p> http://localhost/ticket/2468 http://localhost/ticket/2468 Report seanh <p> We want a much quicker and easier way of deploying multiple CKAN instances to the same or multiple servers and managing them (e.g. configuration, themes, extensions, upgrades, and adding new instances). Currently we just deploy and manage each instance separately, which doesn't scale.. We'd also want to have (as far as possible) one single way of deploying CKAN that works across different situations, e.g. deployment server, demo server, jenkins, local deployment for development, these should share the same tools for installing CKAN and its dependencies. </p> <p> Components that this might include: </p> <p> Virtualenv bootstrap script <a class="ext-link" href="https://gist.github.com/2206132"><span class="icon">​</span>https://gist.github.com/2206132</a> </p> <p> Ansible <a class="ext-link" href="http://ansible.github.com/"><span class="icon">​</span>http://ansible.github.com/</a> </p> <p> CKAN deb package </p> http://localhost/ticket/2469 http://localhost/ticket/2469 Report johnglover <p> Datasets listed in search results have not gone through schema validation (but on the user page they do), so package_list snippet currently has to display both 'notes' and 'description'. </p> http://localhost/ticket/2470 http://localhost/ticket/2470 Report ross <p> If you go to <a href="http://ckan.org/features/geospatial/">http://ckan.org/features/geospatial/</a> there is a problem with the pop up tag shown (and zoomed in) - missing status and description. </p> http://localhost/ticket/2471 http://localhost/ticket/2471 Report kindly <p> Datastore_url enabled checks are useless. </p> <p> We should probably change this to datastore_url = 'active' and only update the flag to when a attempt at a post is made. </p> <p> This will need to be removed from the form at some point. </p> http://localhost/ticket/2472 http://localhost/ticket/2472 Report kindly <p> Recline views should have a default table order, being the same as the csv that was imported. </p> http://localhost/ticket/2473 http://localhost/ticket/2473 Report aron.carroll <p> Percieved benefits of switching to Jinja 2 </p> <ul><li>Block inheritance for base templates which is much simpler to understand. </li><li>Large number of developers (and designers) are familiar with it due to its similarities with the Django templates. </li><li>Should be much faster at parsing and rendering templates. </li><li>Has localisation support. </li><li>Good documentation both on it's website and on sites like Stack Overflow. </li></ul><p> We should convert a couple of existing templates to test the switchover and then get peoples feedback. </p> http://localhost/ticket/2474 http://localhost/ticket/2474 Report seanh <p> This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12. </p> <p> <a class="ext-link" href="http://trac.edgewall.org/wiki/ChangeLog"><span class="icon">​</span>http://trac.edgewall.org/wiki/ChangeLog</a> </p> http://localhost/ticket/2475 http://localhost/ticket/2475 Report johnglover <p> Check that lists of strings are being correctly translated. See <a class="ext-link" href="https://github.com/okfn/ckan/commit/f1d68c3d2d4d25a0c0f8a89a68940643fc19b156"><span class="icon">​</span>https://github.com/okfn/ckan/commit/f1d68c3d2d4d25a0c0f8a89a68940643fc19b156</a> </p> http://localhost/ticket/2476 http://localhost/ticket/2476 Report shevski <p> Need a central up to date place for currently supported extensions (closest we have is <a class="ext-link" href="http://wiki.ckan.org/List_of_Extensions"><span class="icon">​</span>http://wiki.ckan.org/List_of_Extensions</a>) &amp; their documentation. I suggest this gets added to a dedicated section of docs.ckan.org </p> http://localhost/ticket/2477 http://localhost/ticket/2477 Report seanh http://localhost/ticket/2478 http://localhost/ticket/2478 Report markw <p> CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank. </p> <p> The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest: </p> <p> (1) Rename the fields 'Owner' and 'Maintainer'. </p> <p> (2) Change the explanatory text for the relevant fields: </p> <p> Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.' </p> <p> (3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile. </p> http://localhost/ticket/2479 http://localhost/ticket/2479 Report markw <p> If a dataset has no resources the resources list currently says '(none)'. </p> <p> Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.' </p> http://localhost/ticket/2480 http://localhost/ticket/2480 Report seanh http://localhost/ticket/2481 http://localhost/ticket/2481 Report toby http://localhost/ticket/2482 http://localhost/ticket/2482 Report markw <p> At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally. </p> <p> This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table. </p> <p> Suggested fix: </p> <ul><li>Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource) </li><li>Disable the Download button unless the resource is stored locally. </li></ul> http://localhost/ticket/2483 http://localhost/ticket/2483 Report toby http://localhost/ticket/2484 http://localhost/ticket/2484 Report nils.toedtmann <p> CKAN has <a class="closed ticket" href="http://localhost/ticket/1345" title="enhancement: Investigate possible memory leak (closed: fixed)">memory leaks</a>. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced. </p> <p> One should ... </p> <ul><li>research ways to limit requests-per-process with the different web servers (see below for a start); </li><li>explain these safeguards in the CKAN documentation and encourage users to apply them; </li><li>consider the helper script <strong><a class="ext-link" href="https://github.com/okfn/ckan/blob/master/ckan_deb/usr/bin/ckan-create-instance"><span class="icon">​</span>ckan-create-instance</a></strong> to generate Apache configs with <strong>WSGIDaemonProcess ... maximum-requests=XY</strong> being active instead of <a class="ext-link" href="https://github.com/okfn/ckan/blob/master/ckan_deb/usr/lib/ckan/common.sh#L262"><span class="icon">​</span>commented out</a>. Or at least let it warn the user to use <strong>MaxRequestsPerChild</strong>; </li><li>Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them. </li></ul><hr /> <p> How to limit requests-per-process </p> <ul><li>Apache: <ul><li>Use <strong>WSGIDaemonProcess</strong> with <strong>maximum-requests=50</strong> or whatever limit is appropriate. We did this sucessfully on <a class="ext-link" href="http://trac.okfn.org/ticket/904"><span class="icon">​</span>datahub.io</a> and the <a class="ext-link" href="http://trac.okfn.org/ticket/1245"><span class="icon">​</span>ckan farm</a> </li><li>(This need to be verified) Without WSGIDaemonProcess, <strong>MaxRequestsPerChild 50</strong> should achieve the same. </li></ul></li><li>nginx/supervisord: to be researched. </li></ul> http://localhost/ticket/2485 http://localhost/ticket/2485 Report seanh http://localhost/ticket/2486 http://localhost/ticket/2486 Report johnglover <p> Quite a few of the current test dataset resources have their format incorrectly specified as text/html. Attempting to preview these results in the browser trying to automatically download the resource, so for now we will not preview this format. </p> http://localhost/ticket/2487 http://localhost/ticket/2487 Report johnglover <p> Reported on IRC by 'floapps'. </p> <ul><li>when posting a group update with {packages: [package-1, package-2, etc.]}, getting <a class="missing wiki">ValidationError?</a>: {'Packages': {'name': <a class="missing wiki">Missing value?</a>}}, although package objects did contain a name field </li><li>same API call worked with organization and publisher profile disabled </li><li>also worked if added a form_to_db_schema_options function which used default_update_group_schema for api updates instead of default_group_schema </li><li>tried on both 1.7 and 1.7.1b </li></ul> http://localhost/ticket/2488 http://localhost/ticket/2488 Report johnglover <p> When clicking 'add dataset' on a publisher/group page, the new dataset form should select the appropriate group. </p> http://localhost/ticket/2489 http://localhost/ticket/2489 Report seanh http://localhost/ticket/2490 http://localhost/ticket/2490 Report aron.carroll <p> The following pages are missing complete breadcrumbs. </p> <p> Add related page: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending/related/new"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending/related/new</a> </p> <p> Needs to look like. </p> <p> Dataset &gt; {dataset_title} &gt; Related &gt; Create Related </p> <p> Template here: <a class="ext-link" href="https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/pages/form_page.html#L6"><span class="icon">​</span>https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/pages/form_page.html#L6</a> </p> <p> Same deal with Edit related. </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending/related/edit/5f9e665b-3e3c-428f-88b1-8554189cf14a"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending/related/edit/5f9e665b-3e3c-428f-88b1-8554189cf14a</a> </p> <p> Needs to look like. </p> <p> Dataset &gt; {dataset_title} &gt; Related &gt; Edit Related </p> <p> <a class="ext-link" href="https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/edit.html#L5"><span class="icon">​</span>https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/edit.html#L5</a> </p> http://localhost/ticket/2491 http://localhost/ticket/2491 Report aron.carroll <p> See: <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending/related/new"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending/related/new</a> </p> <p> Submit the form to see broken formatting. </p> <p> eg. [u'Missing value'] </p> <p> <a style="padding:0; border:none" href="http://localhost/attachment/ticket/2492/ckan-broken-error.png"><img src="http://localhost/raw-attachment/ticket/2492/ckan-broken-error.png" alt="Example of broken image" title="Example of broken image" /></a> </p> http://localhost/ticket/2492 http://localhost/ticket/2492 Report toby <p> templates/error_document_template.html </p> <p> I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content </p> <p> just go to a non-existant url to see </p> <p> Discusion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/</a> </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/404"><span class="icon">​</span>http://s031.okserver.org:2375/404</a> </p> http://localhost/ticket/2493 http://localhost/ticket/2493 Report aron.carroll <p> Example </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/not-here"><span class="icon">​</span>http://s031.okserver.org:2375/not-here</a> </p> <p> Should show the 404 template. </p> http://localhost/ticket/2494 http://localhost/ticket/2494 Report rgrp <p> <a class="ext-link" href="http://datahub.io/stats"><span class="icon">​</span>http://datahub.io/stats</a> </p> http://localhost/ticket/2495 http://localhost/ticket/2495 Report toby <p> stacktrace </p> http://localhost/ticket/2496 http://localhost/ticket/2496 Report toby <p> if not logged in then go to add dataset url the language is lost </p> <p> check if this is a ckan 1.7 issue </p> http://localhost/ticket/2497 http://localhost/ticket/2497 Report toby http://localhost/ticket/2498 http://localhost/ticket/2498 Report icmurray <p> There's no docs for setting up the internal analytics, nor what it provides. </p> <h2 id="Setup">Setup</h2> <p> Add: <tt>ckan.tracking_enabled = true</tt> to .ini file </p> <p> Run: <tt>paster tracking -c dev.ini</tt> </p> <h2 id="Relatedlinks">Related links</h2> <p> <a class="ext-link" href="http://trac.ckan.org/ticket/2251"><span class="icon">​</span>http://trac.ckan.org/ticket/2251</a> </p> http://localhost/ticket/2499 http://localhost/ticket/2499 Report icmurray <h2 id="Originalbugreport:">Original bug report:</h2> <blockquote class="citation"> <p> Hi CKAN devs, </p> <p> Can I request that ckan.logic.get_action raises an exception if the action requested doesn't exist? </p> </blockquote> <blockquote class="citation"> <p> Two of us had a frustrating time tracking down an exception this morning. It's difficult when the exception is raised in python in a template because you don't get a line number. The problem was caused by a new logic function being added to another extension, and we hadn't updated the code there. So if get_action raised an proper exception rather than silently returning None then I think we'd have found the problem quicker. </p> </blockquote> <blockquote class="citation"> <p> I'd write the code for this, but as I understand that the new system would require a ticket, branch, pull request, code review, merge. It seems a lot to do for such a small thing. </p> </blockquote> <blockquote class="citation"> <p> Dave </p> </blockquote> http://localhost/ticket/2500 http://localhost/ticket/2500 Report aron.carroll <p> New form should allow three steps. </p> <ol><li>Add basic fields to create a dataset </li><li>Add a data resource to the dataset (this can be repeated multiple times using "Save and add another" </li><li>Add final metadata to the dataset. </li></ol><p> It should be possible to jump backward through the steps by clicking the arrows at the top. Errors in fields should be validated for each step when the next button is clicked. </p> <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new</a> </p> http://localhost/ticket/2501 http://localhost/ticket/2501 Report aron.carroll <p> Most important is {% snippet %} but {% url_for %} would be great too. </p> http://localhost/ticket/2502 http://localhost/ticket/2502 Report aron.carroll <p> For example: </p> <p> Creating a template in ckanext-demo/templates/base.html containing something like </p> <pre class="wiki">{% ckan_extends "base.html" %} </pre><p> Will extend ckan/templates/base.html </p> http://localhost/ticket/2503 http://localhost/ticket/2503 Report aron.carroll http://localhost/ticket/2504 http://localhost/ticket/2504 Report amercader <p> There are some areas where the documentation could use some improvement and it would be good to have it available for 1.7.1. </p> <ul><li>Add an introduction to harvesting support </li><li>Add an introduction to CSW support </li><li>Better upgrade docs: Some users suggested improvements to the upgrade docs on the list <a class="missing changeset" title="No default repository defined">[1]</a>. Also we should mention how the upgrade would work for point point releases </li><li>Source install: see <a class="new ticket" href="http://localhost/ticket/2406" title="defect: Errors in source install doc (new)">#2406</a> </li></ul><p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-May/002178.html</a> </p> http://localhost/ticket/2505 http://localhost/ticket/2505 Report seanh http://localhost/ticket/2506 http://localhost/ticket/2506 Report seanh <p> If you change the language to one where the word for 'Search' is longer than the English word (e.g. fr) then the search button wraps onto the next line, below the text field, which looks bad. </p> http://localhost/ticket/2507 http://localhost/ticket/2507 Report seanh <p> Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language. </p> <p> This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc. </p> <p> In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files. </p> http://localhost/ticket/2508 http://localhost/ticket/2508 Report johnglover <p> Looks like some of the style has been updated/changed on the group read page but the changes haven't been applied to the organizations read page. </p> http://localhost/ticket/2509 http://localhost/ticket/2509 Report aron.carroll <p> Instead of pointing to /tags/{tagname} all tag links should just point to a dataset search filtered by that tag. </p> <p> Examples are on the homepage and the dataset pages. </p> http://localhost/ticket/2510 http://localhost/ticket/2510 Report toby http://localhost/ticket/2511 http://localhost/ticket/2511 Report ross <p> On the group read page <a class="ext-link" href="http://s031.okserver.org:2375/group/nhs"><span class="icon">​</span>http://s031.okserver.org:2375/group/nhs</a> the add dataset button goes to the edit group page instead of the add dataset url. </p> http://localhost/ticket/2512 http://localhost/ticket/2512 Report ross <p> Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8. Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set. </p> <p> This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside. </p> http://localhost/ticket/2513 http://localhost/ticket/2513 Report seanh <p> It's in the tab bar (maybe not the right place for a button? but I wanted it to be next to the follower tab and count) and doesn't like very much like a button </p> http://localhost/ticket/2514 http://localhost/ticket/2514 Report seanh <p> The related tab on the dataset page doesn't change its appearance when selected like the other tabs do </p> http://localhost/ticket/2515 http://localhost/ticket/2515 Report seanh <p> there's a setting for this </p> http://localhost/ticket/2516 http://localhost/ticket/2516 Report johnglover <ul><li>created repo for final list of instructions: <a class="ext-link" href="http://github.com/okfn/ecportal-server-setup"><span class="icon">​</span>http://github.com/okfn/ecportal-server-setup</a> </li></ul> http://localhost/ticket/2517 http://localhost/ticket/2517 Report johnglover <ul><li>branch from 1.7.1 </li><li>apply CKAN patches that are current part of installation instructions </li><li>create i18n patch so custom i18n directories can be specified via a config option </li></ul> http://localhost/ticket/2518 http://localhost/ticket/2518 Report ross <p> Whilst the error messages returned from the dataproxy are informative, they aren't particularly human readable. It isn't clear to the user whether they should retry, or whether the format of their data is causing the problem etc... </p> <p> Would be lovely if the error messages returned contained a human readable version of the error. </p> http://localhost/ticket/2519 http://localhost/ticket/2519 Report seanh <p> There are 21 undocumented config options in CKAN, some of which are not mentioned in the config file template either: </p> <p> ckan.admin.name ckan.admin.email ckan.default.group_type ckan.page_cache_enabled ckan.cache_enabled ckan.cache_expires ckan.extra_resource_fields ckan.extra_resource_group_fields ckan.storage.key_prefix ckan.storage.max_content_length ckan.feeds.authority_name * ckan.feeds.date * ckan.feeds.author_name * ckan.feeds.author_link * ckan.mail_from ckan.gravatar_default * ckan.plugins ckan.api_url ckan.auth.profile ckan.datastore.enabled ckan.tracking_enabled </p> <p> There are also some options that are in the default deployment.ini even though they're deprecated: </p> <p> ckan.async_notifier carrot_messaging_library ckan.build_search_index_synchronously </p> <p> See email to ckan-dev from David Read: <a class="ext-link" href="http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html"><span class="icon">​</span>http://lists.okfn.org/pipermail/ckan-dev/2012-June/002447.html</a> </p> <p> It'd be best if the docs could be automatically pulled from the source into sphinx using autodoc, see <a class="assigned ticket" href="http://localhost/ticket/1358" title="enhancement: Generate configuration documentation automatically from the ... (assigned)">#1358</a> </p> http://localhost/ticket/2520 http://localhost/ticket/2520 Report seanh <p> It still contains the old build_search_index_synchronously which no longer works </p> http://localhost/ticket/2521 http://localhost/ticket/2521 Report toby <p> the user pic should be square on the user read page </p> http://localhost/ticket/2522 http://localhost/ticket/2522 Report toby <p> <a class="ext-link" href="http://localhost:5000/dataset/new_metadata/dataset"><span class="icon">​</span>http://localhost:5000/dataset/new_metadata/dataset</a> </p> <p> Controller : package Action : new_metadata </p> <p> template package/snippets/package_metadata_form.html </p> http://localhost/ticket/2523 http://localhost/ticket/2523 Report kindly <p> If a user selects a language there are no mo files for then an error is raised. Revert to default language instead. </p> http://localhost/ticket/2524 http://localhost/ticket/2524 Report aron.carroll http://localhost/ticket/2525 http://localhost/ticket/2525 Report aron.carroll <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_metadata"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_metadata</a> </p> http://localhost/ticket/2526 http://localhost/ticket/2526 Report aron.carroll <p> <a class="missing wiki">JavaScript?</a> needs to be able to insert html templates into the document. </p> <p> There are three common solutions to templating at the moment. </p> <ol><li>Insert the html into &lt;script&gt; tags at the bottom of the page and pull them in using the DOM and innerHTML. This is a very simple solution but not very portable. </li><li>Request the HTML snippets as plain text/html using XHR. This adds an extra HTTP request per template but is very clean. It would also be possible to inline the HTML into the JS at the minification stage for production. </li><li>Include the HTML in <a class="missing wiki">JavaScript?</a> strings (horrible to maintain and bloats the js) </li></ol><p> It makes sense to keep these with the other templates so that we can take advantage of localisation where possible. This indicates 1 or 2. </p> http://localhost/ticket/2527 http://localhost/ticket/2527 Report aron.carroll <p> Implementation: </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/edit/5b451cab-8671-4513-91c4-257784c1f484"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/edit/5b451cab-8671-4513-91c4-257784c1f484</a> </p> http://localhost/ticket/2528 http://localhost/ticket/2528 Report rgrp <p> Simple widgets in pure JS. For example: </p> <ul><li>Count of datasets in a group (could generalise to a query but not sure how useful that is ...) </li><li>Embeddable list of top X (5) datasets for a given query </li><li>Embeddable list of *my* datasets </li></ul><p> Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query). </p> <p> Or: we have this attached to areas of site where relevant. </p> <p> Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page). </p> <p> Cf. <a class="ext-link" href="http://okfnlabs.org/ckanjs/"><span class="icon">​</span>http://okfnlabs.org/ckanjs/</a> </p> http://localhost/ticket/2529 http://localhost/ticket/2529 Report rgrp <p> See <a class="ext-link" href="http://datahub.io/ckan-admin/trash"><span class="icon">​</span>http://datahub.io/ckan-admin/trash</a> and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like: </p> <p> Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (<a class="missing wiki">IntegrityError?</a>) update or delete on table "revision" violates foreign key constraint "group_extra_revision_revision_id_fkey" on table "group_extra_revision" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table "group_extra_revision". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'} </p> http://localhost/ticket/2530 http://localhost/ticket/2530 Report rgrp <p> Deleted means things will get purged at some point. </p> <p> Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated. </p> <p> @richard cyganiak ... </p> http://localhost/ticket/2531 http://localhost/ticket/2531 Report toby <p> We should allow the user activity stream to be accessible with no js by passing a param or something. Need to agree how we hide/show basically agree what classes we use. </p> <p> This should be a generalised solution that we can use in other places as needed. </p> <p> I think this just needs discussion </p> http://localhost/ticket/2532 http://localhost/ticket/2532 Report seanh <p> Looks like recent CKAN cleanups broke some imports: </p> <p> <a class="ext-link" href="https://gist.github.com/2935639"><span class="icon">​</span>https://gist.github.com/2935639</a> </p> http://localhost/ticket/2533 http://localhost/ticket/2533 Report seanh <p> Looks like recent cleanups in CKAN broke an import in the extension: </p> <p> <a class="ext-link" href="https://gist.github.com/2935688"><span class="icon">​</span>https://gist.github.com/2935688</a> </p> http://localhost/ticket/2534 http://localhost/ticket/2534 Report rgrp <p> <a class="missing wiki">DataHub?</a> is increasingly used and we should ensure it uses ssl as part of general security. </p> <p> See also <a class="closed ticket" href="http://localhost/ticket/1446" title="enhancement: Data Explorer v2 (closed: fixed)">#1446</a> (Need to support https login for multiple instances as part of the CKAN package install) </p> http://localhost/ticket/2535 http://localhost/ticket/2535 Report toby http://localhost/ticket/2536 http://localhost/ticket/2536 Report seanh <p> <a class="ext-link" href="https://github.com/okfn/ckanbuild"><span class="icon">​</span>https://github.com/okfn/ckanbuild</a> </p> <p> Verify that what's there so far still works, write a README explaining how it works </p> http://localhost/ticket/2537 http://localhost/ticket/2537 Report seanh <p> Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command. </p> <p> May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command. </p> http://localhost/ticket/2538 http://localhost/ticket/2538 Report seanh <p> Do we want to build on top of the existing debian packaging code? Or throw it away and start fresh? </p> http://localhost/ticket/2539 http://localhost/ticket/2539 Report seanh <p> When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once. </p> http://localhost/ticket/2540 http://localhost/ticket/2540 Report seanh <p> We want some extensions from outside of CKAN core to be included in ckanbuild. These would be pip installed into the virtualenv before packaging the debian package. Decide which extensions to include. </p> http://localhost/ticket/2541 http://localhost/ticket/2541 Report seanh <p> It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests. </p> http://localhost/ticket/2542 http://localhost/ticket/2542 Report icmurray <p> Not all solr facet parameters are available through the pcakage_search action. In particular, <tt>facet.sort</tt> has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too. </p> <p> See: <a class="ext-link" href="http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort"><span class="icon">​</span>http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort</a> </p> http://localhost/ticket/2543 http://localhost/ticket/2543 Report icmurray <p> Not all solr facet parameters are available through the pcakage_search action. In particular, <tt>facet.sort</tt> has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too. </p> <p> See: <a class="ext-link" href="http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort"><span class="icon">​</span>http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort</a> </p> http://localhost/ticket/2544 http://localhost/ticket/2544 Report ross <h2 id="Requirements">Requirements</h2> <p> We need to clone the demo ODS site for DataGM as soon as we have a mostly working demo so that the DataGM team can try it out and comment/feedback on functionality. </p> <h2 id="Interface">Interface</h2> <p> None </p> <h2 id="UserStories">User Stories</h2> <p> None </p> <h2 id="Tasks">Tasks</h2> <p> [ ] Deploy demo DataGM site. </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2545 http://localhost/ticket/2545 Report ross <h2 id="Requirements">Requirements</h2> <p> Require the ability for users to control some level of content that is visible on the home page of their ODS installation. This may be through RSS/Atom feeds (see <a class="assigned ticket" href="http://localhost/ticket/2234" title="enhancement: Write a CKAN extension for pulling items from RSS/Atom feeds into CKAN ... (assigned)">#2234</a>) or another mechanism but should result in admins being able to change blocks of text on their homepage. </p> <p> This should not be configuration, but accessible through WUI. </p> <h2 id="Interface">Interface</h2> <p> None </p> <h2 id="UserStories">User Stories</h2> <ul><li>As a system administrator I want to have control over content displayed on the front page beyond featured/popular items. </li></ul><p> </p> <ul><li>As a system administrator I don't want to manage content through having to write an extension. </li></ul><p> </p> <h2 id="Tasks">Tasks</h2> <p> [ ] Analysis </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2546 http://localhost/ticket/2546 Report ross <h2 id="Requirements">Requirements</h2> <p> The ODS demo site will need data adding, initially as fixtures but it would also be useful if we started evaluating datasets that we can ship with ODS installations (at least in the UK) from places such as DGU and ONS. </p> <p> May wish to create a ticket for making sure the datasets within the system are reset every X hours. Perhaps. </p> <h2 id="Interface">Interface</h2> <p> None </p> <h2 id="UserStories">User Stories</h2> <ul><li>As a new system administrator for an ODS instance, I don't want to have a site devoid of any data. Geographically relevant datasets would be welcomed. </li></ul><ul><li>As a bizdev person I would like to be able to demonstrate how ODS works with real datasets. </li></ul><h2 id="Tasks">Tasks</h2> <p> [ ] Identify relevant sources for datasets </p> <p> [ ] Pick datasets </p> <p> [ ] Set them up for import </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2547 http://localhost/ticket/2547 Report ross <h2 id="Requirements">Requirements</h2> <p> We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'. </p> <p> At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner. </p> <p> More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature. </p> <h2 id="Interface">Interface</h2> <p> None </p> <h2 id="UserStories">User Stories</h2> <p> None </p> <h2 id="Tasks">Tasks</h2> <p> [ ] <a class="missing wiki">Analysis/Clarification?</a> </p> <p> [ ] Tests </p> <p> [ ] Migration </p> <p> [ ] <a class="missing wiki">Code/Schema?</a> changes </p> <p> [ ] Documentation </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2548 http://localhost/ticket/2548 Report ross <h2 id="Requirements">Requirements</h2> <p> Merge the organizations plugin into the core groups functionality. Most of this is templating and a small number of changes to the group controller. </p> <p> Investigate the possibility of removing the Membership model and using FKs. </p> <h2 id="Interface">Interface</h2> <p> This is likely to consist of: </p> <ul><li>Updating the templates in templates/group/ </li></ul><ul><li>Configuring whether we want to constrain datasets to a single group (as per organizations) </li></ul><ul><li>Implementing the separate user management (add users to group). </li></ul><ul><li>Checking the publisher auth to not necessarily rely on group type = 'organization' </li></ul><ul><li>Making sure that the changes still work with publisher auth. </li></ul><h2 id="UserStories">User Stories</h2> <p> None </p> <h2 id="Tasks">Tasks</h2> <p> [ ] Code </p> <blockquote> <p> [ ] Configuration </p> </blockquote> <p> </p> <blockquote> <p> [ ] <a class="missing wiki">Model/Migration?</a> </p> </blockquote> <p> </p> <blockquote> <p> [ ] Fix templates </p> </blockquote> <p> [ ] Documentation </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2549 http://localhost/ticket/2549 Report ross <h2 id="Requirements">Requirements</h2> <p> In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub). </p> <h2 id="Requiredinterface">Required interface</h2> <p> Once changes have been made to the user schema, for a given user we want to be able to: </p> <ul><li>determine if they have a paid or a free account, and </li><li>get a string name of the type of paid account. </li></ul><p> Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin. </p> <h2 id="UserStories">User Stories</h2> <p> User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands. </p> <p> Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all. </p> <ul><li>As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely. </li></ul><ul><li>As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have. </li></ul><ul><li>As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else. </li></ul><ul><li>As a user, and only if I have one, I'd like to see my current payment level on my user profile page. </li></ul><p> </p> <h2 id="Tasks">Tasks</h2> <p> [x] Tests </p> <p> [x] Plugin based migration </p> <p> [x] Code </p> <blockquote> <p> [x] Model </p> </blockquote> <blockquote> <p> [x] API </p> </blockquote> <p> [x] Documentation </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2550 http://localhost/ticket/2550 Report ross <h2 id="Requirements">Requirements</h2> <p> Would be useful if we could extend the Google Analytics extension to push track events to GA so that we can determine how the system is being used, and where we can optimise the workflow. </p> <p> Assuming it isn't configured to be off, track events should be added to some key action buttons so that the information on what users are doing is logged for analysis. </p> <h2 id="Interface">Interface</h2> <p> This could possibly just be a list of JS calls to $('x').click(_gaq.push('track'...)) </p> <h2 id="UserStories">User Stories</h2> <ul><li>As a developer I would like to know which components of the system are used the most so I can streamline the processes. </li></ul><ul><li>As a support developer I would like to know which parts add complexity to the system for minimal gain and could possibly be removed, de-prioritised. </li></ul><p> For which metrics, as many of the following as possible that are achievable without changing the templates. I think for now it is enough to just record counts of these pages/actions. </p> <ul><li>Any group link on homepage </li></ul><p> </p> <ul><li>Clicking on username (to go to profile) </li></ul><p> </p> <ul><li>In user profile, clicking on Edit Profile </li></ul><p> </p> <ul><li>Clicking Save Changes on Edit Profile page </li></ul><p> </p> <ul><li>Clicking on any dataset link on User Profile page </li></ul><p> </p> <ul><li>Any of the group links on /group </li></ul><p> </p> <ul><li>Clicking any of the right hand sidebar tags on /group/X </li></ul><p> </p> <ul><li>Visiting /group/history/X </li></ul><p> </p> <ul><li>Compare Button on /group/history/X </li></ul><p> </p> <ul><li>Compare Button on /dataset/history/X </li></ul><p> </p> <ul><li>Tags on right hand sidebar of /dataset/X </li></ul><p> </p> <ul><li>Download button on any /dataset/X/resource/* page </li></ul><p> </p> <ul><li>Data API button on any /dataset/X/resource/* page </li></ul><h2 id="Tasks">Tasks</h2> <p> [ ] <a class="missing wiki">Analysis/Clarification?</a> </p> <p> [ ] <a class="missing wiki">Code/Config?</a> changes </p> <p> [ ] Documentation </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2551 http://localhost/ticket/2551 Report ross <h2 id="Requirements">Requirements</h2> <p> To provide a freemium service it is necessary to be able to provide differing <span class="underline">levels</span> of functionality based on the <strong>type</strong> of user (see <a class="assigned ticket" href="http://localhost/ticket/2550" title="enhancement: User types (assigned)">#2550</a>). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks. </p> <p> Initial implementation should focus on limiting access to datastore disk space. </p> <h2 id="Interface">Interface</h2> <p> These changes are currently only for the data hub and should be kept as much as possible within the data hub extension. </p> <h2 id="UserStories">User Stories</h2> <ul><li>As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited). </li></ul><ul><li>As a system administrator I don't expect to need to manage the levels of users <strong>or</strong> the features that this applies to. </li></ul><p> </p> <h2 id="Tasks">Tasks</h2> <p> [ ] Clarification of requirements/analysis </p> <p> [ ] Tests </p> <p> [ ] Code </p> <blockquote> <p> [ ] Model </p> </blockquote> <blockquote> <p> [ ] API </p> </blockquote> <p> </p> <blockquote> <p> [ ] UI </p> </blockquote> <p> [ ] Documentation </p> <h2 id="Estimates">Estimates</h2> http://localhost/ticket/2552 http://localhost/ticket/2552 Report toby <p> just passes template name </p> <p> ?template_name=... </p> http://localhost/ticket/2553 http://localhost/ticket/2553 Report ross <p> Look into writing a cartridge for importing CKAN data into a Virtuoso quadstore </p> <p> <a class="ext-link" href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger#How"><span class="icon">​</span>http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger#How</a> Does It Work? </p> http://localhost/ticket/2554 http://localhost/ticket/2554 Report aron.carroll <p> Something to make building breadcrumbs a bit nicer </p> http://localhost/ticket/2555 http://localhost/ticket/2555 Report aron.carroll <p> They've become inconsistent as the design has changed. </p> <ul><li>Remove white stripe </li><li>Make text sizes consistent </li><li>Make white space around text boxes consistent </li></ul> http://localhost/ticket/2556 http://localhost/ticket/2556 Report aron.carroll <ul><li>Crosses have disappeared </li><li>Formats should stretch across the page </li></ul> http://localhost/ticket/2557 http://localhost/ticket/2557 Report aron.carroll <ul><li>Needs tags </li><li>Needs groups </li><li>Needs slug </li><li>Licenses links to something </li><li>Needs tooltips (Ira to write some tooltips) </li></ul> http://localhost/ticket/2558 http://localhost/ticket/2558 Report aron.carroll <ul><li>Show resources in the sidebar </li><li>Allow people to go backward and forward </li><li>Have a back button on each form </li><li>Lose resource type </li><li>Add extras as optional metadata </li><li>Remove save </li><li>Add back/Previous </li><li>Add Another </li></ul> http://localhost/ticket/2559 http://localhost/ticket/2559 Report aron.carroll <ul><li>Hide remove button when adding new field </li><li>Finish takes you to the newly created dataset </li></ul> http://localhost/ticket/2560 http://localhost/ticket/2560 Report aron.carroll <ul><li>Add tags </li><li>Add groups </li><li><a class="missing wiki">JavaScript?</a> for custom fields </li></ul> http://localhost/ticket/2561 http://localhost/ticket/2561 Report aron.carroll <p> Tags should be implemented in two sections. </p> <ul><li>Custom predefined tags using vocabularies </li><li>Provide suggestions, auto completion then fallback to free text </li></ul> http://localhost/ticket/2562 http://localhost/ticket/2562 Report aron.carroll <p> Allowing them to be displayed in many sizes, colours and resolutions. </p> <p> <a class="ext-link" href="http://www.zurb.com/playground/foundation-icons"><span class="icon">​</span>http://www.zurb.com/playground/foundation-icons</a> </p> http://localhost/ticket/2563 http://localhost/ticket/2563 Report shevski <p> E.g. <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/adur_district_spending/resource/04127ad5-77e5-4a08-9f40-12d3c383e460"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/adur_district_spending/resource/04127ad5-77e5-4a08-9f40-12d3c383e460</a> </p> <p> When there are no maintainer/contact details or extra fields we should hide the additional info box completely. Otherwise hide the last updated and licence fields which are already displayed in sidebar. </p> <p> Update sidebar to provide context - so 'CSV' -&gt; 'CSV format' + 'unknown' -&gt; 'Last updated: unknown' </p> http://localhost/ticket/2564 http://localhost/ticket/2564 Report shevski <p> Share this (twitter/fb) links open in new tab (no JS) or nice lightbox of some description in demo </p> http://localhost/ticket/2565 http://localhost/ticket/2565 Report shevski http://localhost/ticket/2566 http://localhost/ticket/2566 Report shevski <p> No real need/use for this for demo/ODS yet </p> http://localhost/ticket/2567 http://localhost/ticket/2567 Report shevski <p> So that they don't interfere with buttons such as here: <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/afterfibre/resource/5d19f9fa-c5a4-426c-84de-c624d6e8c3b3"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/afterfibre/resource/5d19f9fa-c5a4-426c-84de-c624d6e8c3b3</a> </p> http://localhost/ticket/2568 http://localhost/ticket/2568 Report shevski <p> Allow user to navigate directly from one resource page to another via resource box in sidebar </p> http://localhost/ticket/2569 http://localhost/ticket/2569 Report shevski <p> Suggest making this consistent with <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending/related"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending/related</a> by adding same button in top right of resource pages </p> http://localhost/ticket/2570 http://localhost/ticket/2570 Report toby <p> This function is incomplete </p> <p> If you want I can do this but I need guidance over how to do this properly. ie would I go through lib.dictization.model_save.package_resource_list_save() which looks wrong to me as I'd need the package. resource_dict_save() seems better but to be honest the total lack of docstrings etc makes me feel ill. </p> http://localhost/ticket/2571 http://localhost/ticket/2571 Report toby <p> attempt to disengage the stats plugin from core as much as possible </p> http://localhost/ticket/2572 http://localhost/ticket/2572 Report icmurray <p> Solr allows its facet parameters to be specified on a per-field basis, eg. <tt>facet.limit</tt> applies to <em>all</em> facet fields, but solr allows it to be overriden for a specific field, eg. <tt>facet.tags.limit</tt>. </p> <p> We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See <tt>ckan.lib.search.query.VALID_SOLR_PARAMETERS</tt>. </p> http://localhost/ticket/2573 http://localhost/ticket/2573 Report toby <p> we want to be able to add new datasets but not have them indexed or seem by people </p> <p> is the vdm.state the place to do this </p> http://localhost/ticket/2574 http://localhost/ticket/2574 Report amercader <p> License titles at least for some of them need translating. This wants to be done in a more efficient and simple manner than currently exists. </p> <p> the original problem was: </p> <p> The license dropdown on the dataset form shows titles in Swedish by default. Changing the language seems to default to English, but after refreshing the page a couple of times it shows a wrong language again. </p> <p> Thanks to pudo for reporting. </p> http://localhost/ticket/2575 http://localhost/ticket/2575 Report toby <p> assuming you have datahub data </p> <p> <a class="ext-link" href="http://localhost:5000/en/dataset/2008_facility_ghg_emissions_by_province_territory/resource/28650645-8878-4826-9405-f20b9dfa4c12"><span class="icon">​</span>http://localhost:5000/en/dataset/2008_facility_ghg_emissions_by_province_territory/resource/28650645-8878-4826-9405-f20b9dfa4c12</a> </p> <p> also the breadcrumb becomes overly long too </p> http://localhost/ticket/2576 http://localhost/ticket/2576 Report ross <p> When given a url parameter for a file it cannot fetch, the app raises a 500 instead of something useful. </p> <p> Better error reporting in the JSON would be useful </p> http://localhost/ticket/2577 http://localhost/ticket/2577 Report toby <p> The demo branch fails huge numbers of tests. </p> <p> many are going to be template based - these need to be sorted - ideally we need to get them to work with legacy/new templates but see how we go on that one </p> <p> anyhow this is not a priority whilst we are branched but needs resolving before we merge into master </p> http://localhost/ticket/2578 http://localhost/ticket/2578 Report toby <p> make these more available but keep existing functionality so not to break any users </p> <p> remove_field() </p> <p> drill_down_url() </p> <p> etc </p> http://localhost/ticket/2579 http://localhost/ticket/2579 Report toby <p> david I've got the add resource working but it is happy to add empty resources which implies that it isn't doing validation correctly. </p> <p> I'm not sure what the validation rules are so hard to test this - I'm coming through as an api call </p> <p> let me know if i can help </p> http://localhost/ticket/2580 http://localhost/ticket/2580 Report toby <p> pass some stuff through the context </p> http://localhost/ticket/2581 http://localhost/ticket/2581 Report rgrp <p> Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?) </p> <p> Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket. </p> http://localhost/ticket/2582 http://localhost/ticket/2582 Report toby <p> in development many auth checks may have been lost we need to check they are still working etc </p> http://localhost/ticket/2583 http://localhost/ticket/2583 Report toby <p> <a class="ext-link" href="http://localhost:5000/user/edit/"><span class="icon">​</span>http://localhost:5000/user/edit/</a>&lt;my user name&gt; </p> http://localhost/ticket/2584 http://localhost/ticket/2584 Report seanh <p> Suggestion from David Read: </p> <p> We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds". </p> <p> So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine: </p> <p> <a class="ext-link" href="http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/"><span class="icon">​</span>http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/</a> </p> <p> Perhaps you would consider providing this in CKAN core in future? </p> <p> I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen? </p> http://localhost/ticket/2585 http://localhost/ticket/2585 Report toby <p> tags is list not string </p> http://localhost/ticket/2586 http://localhost/ticket/2586 Report toby <p> is this the expected behaviour seems crap to me, should we just see the author but just no link </p> http://localhost/ticket/2587 http://localhost/ticket/2587 Report toby <p> when package is added it does not seem to get indexed </p> http://localhost/ticket/2588 http://localhost/ticket/2588 Report toby <p> need to add some logic for this </p> http://localhost/ticket/2589 http://localhost/ticket/2589 Report ross <p> Need proper user stories but ... </p> <p> Publisher admins/editors may need a more useful group read page showing things like: </p> <ul><li>The current search </li></ul><ul><li>Recent activity </li></ul><ul><li>People within the group </li></ul><ul><li>Followers </li></ul><ul><li>Others? </li></ul> http://localhost/ticket/2590 http://localhost/ticket/2590 Report toby <p> sqlalchemy fails on postgres 9.1.4 when unicode is returned </p> <p> see <a class="ext-link" href="http://docs.sqlalchemy.org/en/rel_0_7/dialects/postgresql.html"><span class="icon">​</span>http://docs.sqlalchemy.org/en/rel_0_7/dialects/postgresql.html</a> </p> http://localhost/ticket/2591 http://localhost/ticket/2591 Report seanh <p> Make the same instructions work for Ubuntu Lucid and Precise and make them simpler </p> <p> Also see <a class="closed ticket" href="http://localhost/ticket/1471" title="enhancement: Be compatible with new ubuntu release in April (closed: fixed)">#1471</a> </p> http://localhost/ticket/2592 http://localhost/ticket/2592 Report johnglover <p> Add caching config to ecportal-server-setup repository. </p> http://localhost/ticket/2593 http://localhost/ticket/2593 Report johnglover <p> EU27 option seems to have been removed at some point, needs to be added back to geographical coverage vocabulary and tested. </p> http://localhost/ticket/2594 http://localhost/ticket/2594 Report aron.carroll <p> For example. </p> <ol><li>Start on stage one </li><li>Add dataset click next </li><li>Click previous </li></ol><p> Expect Stage 1 to be "active", stage two to be "complete" </p> <ol><li>Start on stage one </li><li>Add dataset click next </li><li>Click next </li><li>Click previous </li></ol><p> Expect Stage 2 to be "active", stage one and three to be "complete" </p> <ol><li>Start on stage one </li><li>Add dataset click next </li><li>Click next </li><li>Click previous </li><li>Click previous </li></ol><p> Expect Stage 1 to be "active", stage two and three to be "complete" </p> http://localhost/ticket/2595 http://localhost/ticket/2595 Report aron.carroll <p> Should have all supported extensions enabled, those that don't work should be disabled and ticketed. Also needs a customised homepage. </p> http://localhost/ticket/2596 http://localhost/ticket/2596 Report aron.carroll <ol><li>Go to new dataset </li><li>Add a slug </li><li>Click next </li></ol><p> Get error. </p> <pre class="wiki">URL: http://localhost:5000/dataset/new File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__ return request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__ response = request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__ res = WSGIController.__call__(self, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 422 in new return self._save_new(context) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 703 in _save_new return self.new(data_dict, errors, error_summary) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 430 in new data['tag_string'] = ', '.join(h.dict_list_reduce(c.pkg_dict['tags'], 'name')) TypeError: string indices must be integers, not str </pre> http://localhost/ticket/2597 http://localhost/ticket/2597 Report toby <p> I see X where none should be </p> http://localhost/ticket/2598 http://localhost/ticket/2598 Report aron.carroll <p> can we promote this to phase 3? </p> http://localhost/ticket/2599 http://localhost/ticket/2599 Report seanh <p> If you're not logged in and you click this button, it takes you to the add group page not the login page. Anyway, the default behaviour everywhere else in CKAN is to hide buttons that you don't have permission to use (e.g. add dataset, edit dataset, etc.) </p> http://localhost/ticket/2600 http://localhost/ticket/2600 Report seanh <p> Add the organizations and organizations_dataset plugins to ini file, run ckan, login, create an organization, create a dataset and put in organization, edit the dataset and set organization to none, try to edit the dataset again (you must be sysadmin to do this) and get a 500: </p> <pre class="wiki">Error - &lt;class 'genshi.template.eval.UndefinedError'&gt;: [] has no member named "get" URL: http://127.0.0.1:5000/dataset/edit/fooset File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in __call__ app_iter = self.application(environ, sr_checker) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__ return self.wrap_app(environ, session_start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 221 in __call__ res = WSGIController.__call__(self, environ, start_response) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File '/home/seanh/Projects/ckan/ckan/ckan/controllers/package.py', line 499 in edit extra_vars=vars) File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 148 in render return cached_template(template_name, render_template, loader_class=loader_class) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template return render_func() File '/home/seanh/Projects/ckan/ckan/ckan/lib/base.py', line 101 in render_template return literal(stream.render(method=method, encoding=None, strip_whitespace=True)) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 183 in render return encode(generator, method=method, encoding=encoding, out=out) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 57 in encode return _encode(''.join(list(iterator))) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 339 in __call__ for kind, data, pos in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 670 in __call__ for kind, data, pos in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 771 in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/output.py', line 586 in __call__ for ev in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/core.py', line 288 in _ensure for event in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 605 in _include for event in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/markup.py', line 327 in _match for event in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 545 in _flatten for kind, data, pos in stream: File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/directives.py', line 169 in _generate attrs = _eval_expr(self.expr, ctxt, vars) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/base.py', line 277 in _eval_expr retval = expr.evaluate(ctxt) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 178 in evaluate return eval(self.code, _globals, {'__data__': data}) File '/home/seanh/Projects/ckan/ckan/ckanext/organizations/templates/organization_package_form.html', line 110 in &lt;Expression u"{'selected':'selected'} if organization.get('id','') == group['id'] else {}"&gt; &lt;option value="${group['id']}" py:attrs="{'selected':'selected'} if organization.get('id','') == group['id'] else {}"&gt;${group['title']}&lt;/option&gt; File '/home/seanh/Projects/ckan/ckan/ckan/config/environment.py', line 248 in genshi_lookup_attr val = cls.undefined(key, owner=obj) File '/home/seanh/.virtualenvs/ckan/local/lib/python2.7/site-packages/genshi/template/eval.py', line 410 in undefined raise UndefinedError(key, owner=owner) UndefinedError: [] has no member named "get" </pre> http://localhost/ticket/2601 http://localhost/ticket/2601 Report seanh <p> With the organizations and organizations_dataset plugins enabled login and create a new dataset, adding it to a group that you are a member of. Edit the dataset and set the group to none. Now you can no longer edit your own dataset. (You can also specify None for the organization when first creating the dataset, same problem.) </p> http://localhost/ticket/2602 http://localhost/ticket/2602 Report icmurray <p> The fields parameter of resource_search was deprecated when fixing <a class="closed ticket" href="http://localhost/ticket/2438" title="enhancement: resource_search action not accessible via GET request (closed: fixed)">#2438</a>. It can be removed in release 1.9, and the action tidied up as a result. </p> http://localhost/ticket/2603 http://localhost/ticket/2603 Report amercader <p> Requests (<a class="ext-link" href="http://python-requests.org"><span class="icon">​</span>http://python-requests.org</a>) is really useful to deal with http requests and saves a lot of time. </p> <p> I propose we include it as a requirement for the tests for the time being, and even for core if necessary at a later stage. </p> http://localhost/ticket/2604 http://localhost/ticket/2604 Report ross <p> From DR at DGU </p> <p> Basically get_groups() appears to cache its results, but this falls down when subsequent calls are in a different session. We get this when saving a dataset. get_groups() first gets called in the controller when it is trying to work out what permissions the user has. It later gets called, and by this time c.userobj is detached, so get_groups() fails. </p> <p> I'm working around this by refreshing c.userobj (from c.user) before calling get_groups(), but I wonder if the get_groups caching could detect the detached session and bypass the cache in this case? </p> <p> diff at <a class="ext-link" href="https://gist.github.com/3003117"><span class="icon">​</span>https://gist.github.com/3003117</a> </p> http://localhost/ticket/2605 http://localhost/ticket/2605 Report toby <p> ckan/main not main </p> http://localhost/ticket/2606 http://localhost/ticket/2606 Report seanh <p> if the user tries to upload a file they will get "Failed to get credentials for storage upload. Upload cannot proceed" </p> <p> Maybe add a test for it this time, this bug has appeared and reappeared before </p> http://localhost/ticket/2607 http://localhost/ticket/2607 Report seanh <p> I'm not seeing any javascript or python error, the upload just hangs. </p> http://localhost/ticket/2608 http://localhost/ticket/2608 Report aron.carroll <ol><li>Complete step one of add dataset </li><li>Click next without adding any resource data. </li></ol><p> Page errors. </p> <pre class="wiki">URL: http://localhost:5000/dataset/new_resource/asdasd File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__ return request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__ resp = self.call_func(req, *args, **self.kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func return self.func(req, *args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/fanstatic/injector.py', line 52 in __call__ response = request.get_response(self.app) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response application, catch_exc_info=False) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application app_iter = application(self.environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/base.py', line 258 in __call__ res = WSGIController.__call__(self, environ, start_response) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__ response = self._dispatch_call() File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call response = self._inspect_call(func) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call result = self._perform_call(func, args) File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call return func(**args) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/package.py', line 495 in new_resource get_action('resource_create')(context, data) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/logic/action/create.py', line 240 in resource_create raise ValidationError(errors, _error_summary(errors)) ValidationError: {u'URL': u'Missing value'} </pre><p> <a class="ext-link" href="http://localhost:5000/dataset/new_resource/asdasd"><span class="icon">​</span>http://localhost:5000/dataset/new_resource/asdasd</a> </p> http://localhost/ticket/2609 http://localhost/ticket/2609 Report shevski <p> In the footer: Change 'Powered by <a class="missing wiki">CityData?</a>' to Powered by <a class="missing wiki">DataSuite?</a> (with link to ckan.org/datasuite) </p> <p> Links in left hand side of footer should change to a link to ckan.org, link to OKFN, link to ckan.org/datasuite/about and link to Pricing Page (on ckan.org .. will be /datasuite/solutions) </p> <p> In create dataset form, prefilled tag examples should include one example with a space, e.g. "mental health" </p> http://localhost/ticket/2610 http://localhost/ticket/2610 Report shevski <p> In the footer: Change 'Powered by <a class="missing wiki">CityData?</a>' to Powered by <a class="missing wiki">DataSuite?</a> (with link to ckan.org/datasuite) </p> <p> Links in left hand side of footer should change to a link to ckan.org, link to OKFN, link to ckan.org/datasuite/about and link to Pricing Page (on ckan.org .. will be /datasuite/solutions) </p> <p> In create dataset form, prefilled tag examples should include one example with a space, e.g. "mental health" </p> <p> Change 'Groups' to 'Publishers' everywhere </p> <p> Homepage copy/images (Ira to supply?) </p> http://localhost/ticket/2611 http://localhost/ticket/2611 Report shevski <p> For example, hide the 'edit' button on user profiles <a class="ext-link" href="http://s031.okserver.org:2375/user/ross2#activity"><span class="icon">​</span>http://s031.okserver.org:2375/user/ross2#activity</a> unless I'm logged in as Ross </p> http://localhost/ticket/2612 http://localhost/ticket/2612 Report shevski <ol><li>file uploads on dataset creation </li><li>create dataset validation </li><li>create dataset slugs </li><li>Add dataset to group process: pop-up allowing you to quickly search for a pick a dataset to add to group. Must check you have necc auth over dataset to be added validation of user name </li><li>Language box </li><li>toggling activity stream </li><li>Search sorting </li><li>autocomplete on tags </li><li>add and remove custom fields on forms </li><li>the rest: tooltip on popular datasets with number of views </li></ol><blockquote> <p> facets to update automatically </p> </blockquote> <blockquote> <p> creating a dataset without reloading page between steps </p> </blockquote> <blockquote> <p> hover on licences information </p> </blockquote> <blockquote> <p> autocomplete on search terms </p> </blockquote> <blockquote> <p> group filtering </p> </blockquote> <blockquote> <p> add comment to datasets </p> </blockquote> <blockquote> <p> social share buttons in lightboxes </p> </blockquote> <blockquote> <p> dataset counts on homepage? </p> </blockquote> http://localhost/ticket/2613 http://localhost/ticket/2613 Report toby <p> needs to move to a better home </p> <p> better ubuntu install instructions plus test they work </p> <p> <tt>apt-get install nodejs</tt> </p> <p> integrate with paster </p> http://localhost/ticket/2614 http://localhost/ticket/2614 Report seanh http://localhost/ticket/2615 http://localhost/ticket/2615 Report aron.carroll <p> Currently stages of the same colour blur together, these could do with a nice divider. </p> http://localhost/ticket/2616 http://localhost/ticket/2616 Report aron.carroll http://localhost/ticket/2617 http://localhost/ticket/2617 Report toby http://localhost/ticket/2618 http://localhost/ticket/2618 Report seanh <p> Activities about private datasets should not appear in public activity streams. </p> <p> I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams. </p> <p> I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out' </p> <p> I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?) </p> <p> The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets. </p> <p> An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private. </p> <p> Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered. </p> http://localhost/ticket/2619 http://localhost/ticket/2619 Report aron.carroll <p> See the related sidebar box </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/adur_district_spending"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/adur_district_spending</a> </p> <p> If a package has related items the first (or a random) related item should be passed into the snippet. </p> http://localhost/ticket/2620 http://localhost/ticket/2620 Report icmurray <p> This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict. See <a class="closed ticket" href="http://localhost/ticket/2439" title="defect: tag_search and tag_autocomplete not fully available via GET request (closed: fixed)">#2439</a> </p> <p> In a future release of CKAN (probably 1.9) it can be removed. </p> <p> Internal uses of it were removed in <a class="closed ticket" href="http://localhost/ticket/2439" title="defect: tag_search and tag_autocomplete not fully available via GET request (closed: fixed)">#2439</a>, but there are tests that still use it. </p> http://localhost/ticket/2621 http://localhost/ticket/2621 Report seanh <p> Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)" </p> http://localhost/ticket/2622 http://localhost/ticket/2622 Report seanh <p> Enable the organizations and organizations_dataset plugins. Start CKAN. Login and create a new organization. Logout. Login with a different account that is a sysadmin. Go to the organization you just created with the other account. There are Edit and Join buttons. Click on the Edit button, and the Join button changes to Authorization. </p> <p> It should probably be an Authorization button right from the start if you are a sysadmin. </p> http://localhost/ticket/2623 http://localhost/ticket/2623 Report seanh <p> Enable the organizations and organizations_dataset plugins, login to ckan, create a group, create a dataset and add it to the group. Edit the dataset, add some tags to it. The new tags are not saved when you submit. </p> <p> Disable the organizations and organizations_dataset plugins and tag editing works again. </p> http://localhost/ticket/2624 http://localhost/ticket/2624 Report seanh <p> Have to decide which non-core extensions are going to be supported first. </p> http://localhost/ticket/2625 http://localhost/ticket/2625 Report toby <p> <a class="ext-link" href="http://localhost:5000/dataset/edit/2010-nigeria-budget"><span class="icon">​</span>http://localhost:5000/dataset/edit/2010-nigeria-budget</a> </p> <p> the edit url button appears too high up </p> http://localhost/ticket/2626 http://localhost/ticket/2626 Report seanh <p> IDatasetForm's form_to_db_schema() method is not getting called by ckan </p> http://localhost/ticket/2627 http://localhost/ticket/2627 Report seanh http://localhost/ticket/2628 http://localhost/ticket/2628 Report seanh http://localhost/ticket/2629 http://localhost/ticket/2629 Report seanh http://localhost/ticket/2630 http://localhost/ticket/2630 Report aron.carroll http://localhost/ticket/2631 http://localhost/ticket/2631 Report toby <p> in lib helpers make sure the allowed function lists functions exist or throw an error. </p> http://localhost/ticket/2632 http://localhost/ticket/2632 Report toby <p> so we find newly created templates </p> http://localhost/ticket/2633 http://localhost/ticket/2633 Report toby <p> make sure documented </p> http://localhost/ticket/2634 http://localhost/ticket/2634 Report dread <p> You can't run the search-index reindex on a live server because it will give us bad results for 2 to 3 hours while it runs. Can there be an option that doesn't delete the entire index at the start? </p> <p> Instead it could just delete any items that don't exist any more, then delete them and regenerate them one by one. So the total number of datasets doesn't change much. </p> http://localhost/ticket/2635 http://localhost/ticket/2635 Report aron.carroll http://localhost/ticket/2636 http://localhost/ticket/2636 Report aron.carroll http://localhost/ticket/2637 http://localhost/ticket/2637 Report toby http://localhost/ticket/2638 http://localhost/ticket/2638 Report amercader <p> New recline version has significant improvements like the <a class="missing wiki">SlickGrid?</a> based grid view, it'd be good to have it on 1.8. </p> http://localhost/ticket/2639 http://localhost/ticket/2639 Report amercader <p> Depends on <a class="closed ticket" href="http://localhost/ticket/2639" title="enhancement: Update recline version for 1.8 release (closed: fixed)">#2639</a> </p> http://localhost/ticket/2640 http://localhost/ticket/2640 Report amercader <p> Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now </p> http://localhost/ticket/2641 http://localhost/ticket/2641 Report amercader http://localhost/ticket/2642 http://localhost/ticket/2642 Report aron.carroll <p> Discussion: </p> <p> <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments</a> </p> http://localhost/ticket/2643 http://localhost/ticket/2643 Report toby <p> we now have a user dashboard that needs theming not sure if we need sam to look at it </p> <p> <a class="ext-link" href="http://localhost:5000/user/dashboard"><span class="icon">​</span>http://localhost:5000/user/dashboard</a> </p> http://localhost/ticket/2644 http://localhost/ticket/2644 Report johnglover <p> Server setup scripts need to have two distinct steps: </p> <p> 1) Download all files that aren't in RHEL repositories, create RPMs where necessary (python virtualenv). </p> <p> 2) Install from local files. </p> http://localhost/ticket/2645 http://localhost/ticket/2645 Report toby <p> make it so </p> http://localhost/ticket/2646 http://localhost/ticket/2646 Report toby <p> make it as a snippet calling thing like aron would love </p> http://localhost/ticket/2647 http://localhost/ticket/2647 Report aron.carroll http://localhost/ticket/2648 http://localhost/ticket/2648 Report ross <p> When a resource doesn't have a description, it is rendered as markdown from the description of the dataset. </p> <p> If the resource does have a description it isn't rendered as markdown .. </p> <p> See <a class="ext-link" href="http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96"><span class="icon">​</span>http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96</a> </p> http://localhost/ticket/2649 http://localhost/ticket/2649 Report ross <p> When a resource doesn't have a description, it is rendered as markdown from the description of the dataset. </p> <p> If the resource does have a description it isn't rendered as markdown .. </p> <p> See <a class="ext-link" href="http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96"><span class="icon">​</span>http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96</a> </p> http://localhost/ticket/2650 http://localhost/ticket/2650 Report ross <p> TSV support doesn't seem to work very well, may be the mimetype ( text/tab-separated-values ) </p> <p> See <a class="ext-link" href="http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96"><span class="icon">​</span>http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96</a> </p> http://localhost/ticket/2651 http://localhost/ticket/2651 Report toby http://localhost/ticket/2652 http://localhost/ticket/2652 Report aron.carroll <p> Try: <a class="ext-link" href="http://localhost:5000/api/storage/auth/form/photo-of-a-duck"><span class="icon">​</span>http://localhost:5000/api/storage/auth/form/photo-of-a-duck</a> </p> <p> Gets: </p> <pre class="wiki">File '&lt;string&gt;', line 2 in auth_form File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/jsonp.py', line 26 in jsonpify data = func(*args, **kwargs) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 407 in auth_form authorize(method, bucket, label, c.userobj, self.ofs) File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 200 in ofs StorageAPIController._ofs_impl = get_ofs() File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 71 in get_ofs storage_backend = config['ofs.impl'] File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/paste/registry.py', line 146 in __getitem__ return self._current_obj()[key] KeyError: 'ofs.impl' </pre> http://localhost/ticket/2653 http://localhost/ticket/2653 Report ross <p> The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name </p> <p> Add support for this for datahub now, and discuss for new 1.9 UI </p> http://localhost/ticket/2654 http://localhost/ticket/2654 Report shevski <p> Editing and saving a resource wipes the data from the dataset. Happened with <a class="ext-link" href="http://demo.ckan.org/dataset/my-tes"><span class="icon">​</span>http://demo.ckan.org/dataset/my-tes</a> I clicked on edit, changed a tag, clicked update and the data was gone! </p> http://localhost/ticket/2655 http://localhost/ticket/2655 Report dread <p> This page <a class="ext-link" href="http://thedatahub.org/feeds/custom.atom?q=wombat"><span class="icon">​</span>http://thedatahub.org/feeds/custom.atom?q=wombat</a> has 0 results and contains a link to <a class="ext-link" href="http://thedatahub.org/feeds/custom.atom?q=wombat&amp;amp;page=0"><span class="icon">​</span>http://thedatahub.org/feeds/custom.atom?q=wombat&amp;amp;page=0</a> which the page=0 causes this exception: </p> <pre class="wiki">ckan.lib.search.common.SearchError'&gt;: SOLR returned an error running query Error: "'start' parameter cannot be negative" </pre> http://localhost/ticket/2656 http://localhost/ticket/2656 Report shevski <p> History button in top right of <a class="ext-link" href="http://demo.ckan.org/group/data-explorer"><span class="icon">​</span>http://demo.ckan.org/group/data-explorer</a> goes to <a class="ext-link" href="http://demo.ckan.org/group/history/data-explorer"><span class="icon">​</span>http://demo.ckan.org/group/history/data-explorer</a> and needs hiding </p> http://localhost/ticket/2657 http://localhost/ticket/2657 Report aron.carroll <p> Can we do this as a localisation? </p> http://localhost/ticket/2658 http://localhost/ticket/2658 Report shevski <p> See <a class="ext-link" href="http://demo.ckan.org/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b"><span class="icon">​</span>http://demo.ckan.org/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b</a> </p> <p> David previously fixed this on some version of demo, seems to have regressed. Just want the window to be longer </p> http://localhost/ticket/2659 http://localhost/ticket/2659 Report shevski <p> same as yesterday: <a class="ext-link" href="http://demo.ckan.org/dataset/edit/newcastle-city-council-payments-over-500"><span class="icon">​</span>http://demo.ckan.org/dataset/edit/newcastle-city-council-payments-over-500</a> clicking update/save returns error </p> http://localhost/ticket/2660 http://localhost/ticket/2660 Report toby <p> in 2375-demo-theme-development </p> <p> templates/package/confirm_delete.html needs to be made to look nice note some writing in black </p> <p> delete button needs nice home on dataset/edit </p> <p> you can add js support add ?confirm=yes to url on dataset button to force delete </p> <p> Please note delete is not yet implemented - but will be in the next few hours </p> http://localhost/ticket/2661 http://localhost/ticket/2661 Report shevski <p> changing a tag from "country-uk" to "country:uk" and updating gives server error </p> <p> otherwise updating metadata / adding tags seem fine </p> <p> either we should allow colons or form should return a message indicating what characters are allowed </p> http://localhost/ticket/2662 http://localhost/ticket/2662 Report toby <p> This function is shit and needs cleaning up and a doc string </p> <p> description is markdown and should be treated properly </p> <p> either we should truncate all or leave it to the templates but work universally </p> <p> url if no name / desc this is in demo-theme branch </p> http://localhost/ticket/2663 http://localhost/ticket/2663 Report toby http://localhost/ticket/2664 http://localhost/ticket/2664 Report toby <p> the stages snippet needs to be inside &lt;form&gt; so I can make some links buttons - eg stage 1 -&gt; stage 2 I want to save the dataset not just move to add resource </p> <p> if you can just move it within the form html leaving everything else the same - I can then add buttons and pass back for styling </p> http://localhost/ticket/2665 http://localhost/ticket/2665 Report toby <p> There is a new template for editing resources outside of the add dataset logic </p> <p> this is the template it just throws some crap onto the page at the moment - we need to create a form - the form will end up with the usual data/edit stuff but for now you can use the resource to populate the form and I can then change it to use data - I just want to know which fields we'll have </p> <p> templates/package/resource_edit.html </p> <p> when done reassign to me and I'll get it functioning </p> http://localhost/ticket/2666 http://localhost/ticket/2666 Report toby <p> I thought I'd made this ticket before but can't find it. When we have a fully created dataset we need to be able to edit the resources. </p> <p> template: package/resource_edit.html </p> <p> when you've made a form reassign to me and I'll get it working </p> http://localhost/ticket/2667 http://localhost/ticket/2667 Report dread <p> As a sysadmin I should be able to view deleted groups/publishers (publisher profile) but I can't. </p> <p> In addition, there don't seem to be tests for permissions of deleted publishers. </p> http://localhost/ticket/2668 http://localhost/ticket/2668 Report toby <p> I've added [Draft] to partial completed datasets in the package list and read commit 6a8e74d0 </p> <p> templates/snippets/package_list.html </p> <p> templates/package/read.html </p> <p> do you want to style them in some way? </p> http://localhost/ticket/2669 http://localhost/ticket/2669 Report dread <p> I believe it should only return active ones. On DGU we see deleted publishers popping up. </p> http://localhost/ticket/2670 http://localhost/ticket/2670 Report toby <p> need to be able to delete stuff explicitly </p> http://localhost/ticket/2671 http://localhost/ticket/2671 Report icmurray <p> According to sqlalchemy docs <a class="missing changeset" title="No default repository defined">[1]</a>, when calling Session.is_modified(), the <tt>passive</tt> argument needs to be passed in as True. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html?highlight=is_modified#sqlalchemy.orm.session.Session.is_modified"><span class="icon">​</span>http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html?highlight=is_modified#sqlalchemy.orm.session.Session.is_modified</a> </p> http://localhost/ticket/2672 http://localhost/ticket/2672 Report rgrp <p> Far too many resource options. Lets restrict back to data file and API. Visualizations etc can either get linked in description or in the Related items. </p> http://localhost/ticket/2673 http://localhost/ticket/2673 Report shevski <p> Not loading for all resources as far as I can tell; e.g. <a class="ext-link" href="http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360</a> </p> <p> Fine on demo.ckan.org </p> http://localhost/ticket/2674 http://localhost/ticket/2674 Report shevski <p> see <a class="ext-link" href="http://s031.okserver.org:2375/dataset/gold-prices/related/edit/46ba1174-3666-486e-b822-b5b45ffbb703"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/gold-prices/related/edit/46ba1174-3666-486e-b822-b5b45ffbb703</a> </p> <p> when editing a record, button at bottom right should change to save instead of create </p> http://localhost/ticket/2675 http://localhost/ticket/2675 Report shevski <p> Need to add JS (presumably?) to the related app box on dataset page: <a class="ext-link" href="http://s031.okserver.org:2375/dataset/gold-prices"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/gold-prices</a> to allow user to scroll through when multiple related items exist </p> http://localhost/ticket/2676 http://localhost/ticket/2676 Report shevski <p> Clicking on the blue buttons or images on related page <a class="ext-link" href="http://s031.okserver.org:2375/dataset/gold-prices/related"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/gold-prices/related</a> should take you to a new tab instead of navigating you away </p> http://localhost/ticket/2677 http://localhost/ticket/2677 Report shevski <p> Needs to be a way for admins (owners of dataset) to delete related items. I suggest adding a button to the edit page </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/gold-prices/related/edit/d3249863-040a-4f55-a5d8-c86697d0b7b8"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/gold-prices/related/edit/d3249863-040a-4f55-a5d8-c86697d0b7b8</a> </p> http://localhost/ticket/2678 http://localhost/ticket/2678 Report icmurray <p> The current behaviour of <a class="missing wiki">TemplateController?</a>.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file. </p> <p> Although this may be a feature that some instances want. In general, it leads to: </p> <ul><li>500s when attempting to access a normal template (eg - <a class="ext-link" href="http://datahub.io/importer/preview"><span class="icon">​</span>http://datahub.io/importer/preview</a>) </li><li>A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template). </li></ul><p> Solution: </p> <ul><li>Change the controller to 404 </li><li>Ensure there's a way for existing ckan instances to override that behaviour should they need it. </li></ul> http://localhost/ticket/2679 http://localhost/ticket/2679 Report shevski <p> Super admins &amp; owners/creators of groups need a way to delete them </p> <p> Is the auth model in place for all this stuff? May be worth sitting down and going through how this should work </p> <p> Currently can change state from active to deleted - this is not ery user friendly (want a button with a 2 step process maybe?) &amp; I'd like to know who can view this option? Everyone who can edit a group? Who can edit? </p> http://localhost/ticket/2680 http://localhost/ticket/2680 Report shevski <p> This is pretty key for standardization. </p> <p> Also would like us to change the grey text to be: e.g. CSV, JSON, [more examples... i have no idea what they key ones are tbh] </p> http://localhost/ticket/2681 http://localhost/ticket/2681 Report seanh <p> CKAN's internal tracking seems to count each page view twice, the problem appears to be with the SQL in the update_tracking() method in ckan/lib/cli.py. </p> <p> The internal tracking feature needs some tests, and some of the code could maybe do with some more explanatory comments, e.g. what is the intended difference between count and running_total? </p> http://localhost/ticket/2682 http://localhost/ticket/2682 Report seanh http://localhost/ticket/2683 http://localhost/ticket/2683 Report toby <p> I think it would be nice to have all form pages monitor their forms - excluding search ones - and if a field has changed then warn the user that they have not saved the form data but are leaving the page. </p> <p> This would help with things like changing resources on the add dataset resource form etc. </p> http://localhost/ticket/2684 http://localhost/ticket/2684 Report toby <p> stop bots/browsers etc following links and auto deleting stuff </p> http://localhost/ticket/2685 http://localhost/ticket/2685 Report shevski <p> First I noticed that the gold prices dataset preview was not displaying &amp; has data API enabled Secondly I tried enabling datastore for <a class="ext-link" href="http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f"><span class="icon">​</span>http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f</a> and after I saved the preview no longer worked. Unticking the datastore &amp; data api checkbox brought it back </p> <p> Is this a known issue? </p> http://localhost/ticket/2686 http://localhost/ticket/2686 Report shevski <p> Once you upload a file, all the buttons on the page stop working. You can't complete &amp; go to next step or save &amp; add another or go to previous </p> <p> Clicking on these just seems to re-load the page </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_resource/super"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_resource/super</a> </p> http://localhost/ticket/2687 http://localhost/ticket/2687 Report ross <p> Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed) </p> http://localhost/ticket/2688 http://localhost/ticket/2688 Report shevski <p> Clicking on 'save &amp; add another' button here <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_resource/yo"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_resource/yo</a> takes you to step 3 instead of leaving you on step 2 when you haven't added any data. </p> <p> Instead it should show an alert/ message that no data has been added &amp; that you need to link to a file or upload something </p> <p> Also, can upload be option 2 instead of 3? </p> http://localhost/ticket/2689 http://localhost/ticket/2689 Report shevski <p> See attachment. Just need to amke sure that there's enough spacing around the edit button and that it lines up so it doesn't look too wierd </p> http://localhost/ticket/2690 http://localhost/ticket/2690 Report shevski <p> 404 Not Found The resource could not be found. </p> http://localhost/ticket/2691 http://localhost/ticket/2691 Report shevski <p> the "add your own data" link in "Try another search term, browse the datasets below or add your own data." here <a class="ext-link" href="http://s031.okserver.org:2375/dataset?q=stuff"><span class="icon">​</span>http://s031.okserver.org:2375/dataset?q=stuff</a> doesn't take you to add a dataset, just back to the search/datasets page </p> http://localhost/ticket/2692 http://localhost/ticket/2692 Report toby <p> <a class="ext-link" href="https://github.com/okfn/ckan/blob/master/LICENSE.txt"><span class="icon">​</span>https://github.com/okfn/ckan/blob/master/LICENSE.txt</a> mentions requires/lucid_conflict.txt which is no longer the case for 1.8 and needs updating </p> <p> not sure where requirements-testing/docs fit in </p> <p> @sean I've assigned this to you as you changed the requirements and my plate is full feel free to pass on to someone else </p> http://localhost/ticket/2693 http://localhost/ticket/2693 Report toby <p> we need to have proper fix for this - not sure if there is some caching issue </p> <p> this around the group search </p> http://localhost/ticket/2694 http://localhost/ticket/2694 Report shevski <p> in dataset creation / edit form </p> http://localhost/ticket/2695 http://localhost/ticket/2695 Report shevski <p> in create/edit dataset forms </p> http://localhost/ticket/2696 http://localhost/ticket/2696 Report shevski <p> Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check) </p> http://localhost/ticket/2697 http://localhost/ticket/2697 Report shevski <p> for description / other fields with markdown support </p> http://localhost/ticket/2698 http://localhost/ticket/2698 Report shevski <p> needs review &amp; speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group </p> http://localhost/ticket/2699 http://localhost/ticket/2699 Report shevski <p> lang box improvements: styling, images, submit upon selection </p> http://localhost/ticket/2700 http://localhost/ticket/2700 Report shevski <p> disqus?? </p> http://localhost/ticket/2701 http://localhost/ticket/2701 Report shevski <p> tooltip on popular datasets with number of views facets to update automatically creating a dataset without reloading page between steps hover on licences information autocomplete on search terms group filtering social share buttons in lightboxes dataset counts on homepage </p> http://localhost/ticket/2702 http://localhost/ticket/2702 Report shevski <p> with add button on dataset page and in 'data &amp; resource' section when there's no data </p> http://localhost/ticket/2703 http://localhost/ticket/2703 Report shevski <p> when editing a dataset (e.g. here: <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/edit/afghanistan-election-data"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/edit/afghanistan-election-data</a>) add sidebar list of resources for user to click &amp; browse through taking you to the relevant resource edit pages </p> http://localhost/ticket/2704 http://localhost/ticket/2704 Report shevski <p> e.g. on <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/afghanistan-election-data/resource_edit/f6331f99-51f6-44d9-95b9-b20f3b74f360"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/afghanistan-election-data/resource_edit/f6331f99-51f6-44d9-95b9-b20f3b74f360</a> </p> <p> &amp; ensure consistant with sidebar of edit dataset page </p> http://localhost/ticket/2705 http://localhost/ticket/2705 Report shevski <p> i.e. mimetype, size, last updated etc </p> http://localhost/ticket/2706 http://localhost/ticket/2706 Report shevski <p> This currently looks terrible. Some simple things we should do to improve this. </p> <ul><li>Localise and captitalise known (non custom) keys into human titles. eg. "size", "created_at", "mime_type". </li><li>Apply units to values of known keys where possible eg. "size in appropiate units" </li><li>Convert dates into a more human friendly format. We can avoid localisation hell by using an ISO format like YYYY/MM/DD hh:mm:ss which is still better than ISO8061 </li><li>Move the non user fields to the top of the table. </li></ul> http://localhost/ticket/2707 http://localhost/ticket/2707 Report toby <blockquote> <p> context<a class="missing wiki">package_limits?</a> = { 'tags': 5, &lt;- get first 5 </p> <blockquote> <p> 'extras': 0, &lt;- get all </p> </blockquote> <p> } </p> </blockquote> <p> only get what you ask for have to be explicit </p> <blockquote> <p> context<a class="missing wiki">group_limits?</a> = {} only main item </p> </blockquote> <p> start with datasets/groups expand if we like it </p> http://localhost/ticket/2708 http://localhost/ticket/2708 Report markw <p> There doesn't seem to be any documentation yet for Atom feeds. </p> http://localhost/ticket/2709 http://localhost/ticket/2709 Report toby <p> you know the score </p> http://localhost/ticket/2710 http://localhost/ticket/2710 Report toby <p> it would be nice if this didn't fall over when the dir is removed maybe a linux only issue? </p> http://localhost/ticket/2711 http://localhost/ticket/2711 Report toby <p> we need a delete_resource logic action and enable for resources </p> http://localhost/ticket/2712 http://localhost/ticket/2712 Report toby <p> to reproduce run </p> <p> paster color fuchsia -c development.ini </p> <p> then go to <a class="ext-link" href="http://localhost:5000/dataset/new"><span class="icon">​</span>http://localhost:5000/dataset/new</a> </p> <p> Add button is blue gradient at the top this should be overridden in variables.less </p> <p> when added send this ticket to me and i'll update the color command to change it </p> http://localhost/ticket/2713 http://localhost/ticket/2713 Report shevski <p> see <a class="ext-link" href="http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873"><span class="icon">​</span>http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873</a> </p> <p> what XLS previews do we support? </p> http://localhost/ticket/2714 http://localhost/ticket/2714 Report shevski <p> hide trackbacks &amp; subscribe wboxes on dataset pages until we add disqus </p> http://localhost/ticket/2715 http://localhost/ticket/2715 Report shevski <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/hi-people/resource/4558c365-fd7a-4f0e-9f2a-90ad47db6b39"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/hi-people/resource/4558c365-fd7a-4f0e-9f2a-90ad47db6b39</a> </p> http://localhost/ticket/2716 http://localhost/ticket/2716 Report toby <p> when language is french </p> <p> a) the menu overlaps the logo </p> <p> b) much of the logo becomes unclickable </p> <p> @shevski </p> <p> a) seems like you b) is aron </p> http://localhost/ticket/2717 http://localhost/ticket/2717 Report shevski <p> trying to add a dataset to another group means it's no longer part of the first group </p> <p> <a class="ext-link" href="http://demo.ckan.org/dataset/edit/afterfibre"><span class="icon">​</span>http://demo.ckan.org/dataset/edit/afterfibre</a> </p> http://localhost/ticket/2718 http://localhost/ticket/2718 Report dread <p> Results in bad user experience and <a class="missing wiki">WebApp?</a> errors emailed out. Seen in 1.7.1 </p> http://localhost/ticket/2719 http://localhost/ticket/2719 Report shevski <p> <a class="ext-link" href="http://s031.okserver.org:2375/user"><span class="icon">​</span>http://s031.okserver.org:2375/user</a> </p> <p> needs some spacing between the image &amp; user name </p> <p> We could also display some stats here (e.g. number of dataset created, timestamp of last activity) </p> http://localhost/ticket/2720 http://localhost/ticket/2720 Report shevski <p> Groups previously deleted still show up in the add dataset process in step 3 'Additional info' </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_metadata/ff"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_metadata/ff</a> </p> http://localhost/ticket/2721 http://localhost/ticket/2721 Report shevski <ol><li>go to a group, e.g. <a class="ext-link" href="http://s031.okserver.org:2375/group/data-explorer"><span class="icon">​</span>http://s031.okserver.org:2375/group/data-explorer</a> </li><li>Order by anything, e.g. choose 'popular' and click go </li><li>"sorry no datasets found" message appears instead of ordering the group datasets in wanted order </li></ol><p> <a class="ext-link" href="http://s031.okserver.org:2375/group/data-explorer?q=&amp;sort=views_recent+desc"><span class="icon">​</span>http://s031.okserver.org:2375/group/data-explorer?q=&amp;sort=views_recent+desc</a> </p> http://localhost/ticket/2722 http://localhost/ticket/2722 Report shevski <p> Text goes over the order by text, with drop down floating on top meaning it's all impossible to read </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset?q=unemployment%2C+data&amp;sort=metadata_modified+desc"><span class="icon">​</span>http://s031.okserver.org:2375/dataset?q=unemployment%2C+data&amp;sort=metadata_modified+desc</a> </p> http://localhost/ticket/2723 http://localhost/ticket/2723 Report shevski <ol><li>One of the datasets here: <a class="ext-link" href="http://s031.okserver.org:2375/en/user/shevski"><span class="icon">​</span>http://s031.okserver.org:2375/en/user/shevski</a> has [DRAFT] after it. It's not clear why this has been added -&gt; user confusion </li><li>The draft dataset is not clickable in the dataset list </li><li>The ff dataset (same as the ff [DRAFT] dataset) in the Activity Steam section a) is clickable and b) does not have [DRAFT] after it </li></ol> http://localhost/ticket/2724 http://localhost/ticket/2724 Report shevski <p> My feeling is that 'country-US' and 'country-us' should be the same tag. However currently tags with caps are treated differently </p> <p> see <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/test-dataset"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/test-dataset</a> </p> <p> with TEST and test - there also get indexed twice in the search page </p> http://localhost/ticket/2725 http://localhost/ticket/2725 Report shevski <ol><li>If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this </li></ol><ol start="2"><li>If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28</a> </li></ol> http://localhost/ticket/2726 http://localhost/ticket/2726 Report shevski <p> see <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/test-dataset/resource_edit/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/test-dataset/resource_edit/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28</a> </p> <p> &amp; image attached </p> http://localhost/ticket/2727 http://localhost/ticket/2727 Report shevski <p> 'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see <a class="ext-link" href="http://s031.okserver.org:2375/dataset?groups=test-group"><span class="icon">​</span>http://s031.okserver.org:2375/dataset?groups=test-group</a> </p> http://localhost/ticket/2728 http://localhost/ticket/2728 Report shevski <p> which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work! </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&amp;sort=relevance+asc"><span class="icon">​</span>http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&amp;sort=relevance+asc</a> </p> http://localhost/ticket/2729 http://localhost/ticket/2729 Report shevski <p> Pricing &lt;ckan.org/datasuite/solutions&gt; needs to go to &lt;ckan.org/datasuite/services&gt; </p> <p> <a href="http://ckan.org/datasuite/services/">http://ckan.org/datasuite/services/</a> </p> http://localhost/ticket/2730 http://localhost/ticket/2730 Report markw <p> A large number of XXX.ckan.net sites give the following message: </p> <p> "This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins." </p> <p> The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed): </p> <p> <a class="ext-link" href="http://trac.okfn.org/ticket/933"><span class="icon">​</span>http://trac.okfn.org/ticket/933</a> </p> <p> However, the redirection only seems to have worked in one case, <a class="ext-link" href="http://si.ckan.net"><span class="icon">​</span>http://si.ckan.net</a>. </p> <p> The problem still affects the following sites - the first 4 of which have supposedly been merged: </p> <ul><li><a class="ext-link" href="http://lt.ckan.net"><span class="icon">​</span>http://lt.ckan.net</a> </li><li><a class="ext-link" href="http://hu.ckan.net"><span class="icon">​</span>http://hu.ckan.net</a> </li><li><a class="ext-link" href="http://be.ckan.net"><span class="icon">​</span>http://be.ckan.net</a> </li><li><a class="ext-link" href="http://gr.ckan.net"><span class="icon">​</span>http://gr.ckan.net</a> </li></ul><ul><li><a class="ext-link" href="http://nz.ckan.net"><span class="icon">​</span>http://nz.ckan.net</a> </li><li><a class="ext-link" href="http://bg.ckan.net"><span class="icon">​</span>http://bg.ckan.net</a> </li><li><a class="ext-link" href="http://fi.ckan.net"><span class="icon">​</span>http://fi.ckan.net</a> </li><li><a class="ext-link" href="http://katalogdanych.centrumcyfrowe.pl"><span class="icon">​</span>http://katalogdanych.centrumcyfrowe.pl</a> </li><li><a class="ext-link" href="http://pl.ckan.net"><span class="icon">​</span>http://pl.ckan.net</a> </li><li><a class="ext-link" href="http://southampton.ckan.net"><span class="icon">​</span>http://southampton.ckan.net</a> </li><li><a class="ext-link" href="http://sl.ckan.net"><span class="icon">​</span>http://sl.ckan.net</a> </li></ul><p> Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate. </p> http://localhost/ticket/2731 http://localhost/ticket/2731 Report ross <p> We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is <a class="ext-link" href="http://ckan.okfnpad.org/uploads"><span class="icon">​</span>http://ckan.okfnpad.org/uploads</a> </p> <p> <strong>Simplifying uploads</strong> </p> <p> Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces. </p> <p> We want to fix: </p> <ul><li>File uploads themselves </li><li>Storage of uploaded files </li><li>Notification of the upload to other components </li></ul><p> <strong>File uploads</strong> </p> <p> Things file upload should do: </p> <ul><li>Allow sysadmin to disable </li><li>Allow auth'ed users to upload </li><li>Store whatever they send on disk, and store DB entry linking the file to the person </li><li>When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files. </li></ul><pre class="wiki">Can we do the upload asynchronously and then associate the uploaded key with the resource before the save ? What happens if the user tries to submit before asymc upload finishes ? Should we delay them? </pre><p> The upload workflow should look like... </p> <ol><li>File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data. </li><li> <ol class="loweralpha"><li>When called via ajax then the ID of the newly created file should be returned, </li><li>When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param. </li></ol></li><li>The resource save should check whether it has a file id and in that case updates the file object to point to the resource. </li></ol><p> This should enable: </p> <ul><li>Separate file upload into a user's temporary store, either individually or as a batch. </li><li>Creating resources and simply choosing from previously uploaded, unassigned files </li><li>Adding files/data to a resource after the fact. </li></ul><p> </p> <p> <strong>File storage</strong> </p> <p> File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request. </p> <p> File storage should: </p> <ul><li>allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths). </li><li>provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up. </li><li>allow for collection of information (i.e. estimate of storate space used) </li><li>check whether there is enough space and handling the conequences cleanly </li><li>ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage. </li><li>Have a configurable maximum accepted blob size during upload. </li><li>Should store what meta-data was provided with the upload, such as mimetype. </li></ul><p> Somewhere in the DB we should store ... </p> <table class="wiki"> <tr><td><strong>Column</strong></td><td><strong>Notes</strong> </td></tr><tr><td>id</td><td>An identifier </td></tr><tr><td>owner</td><td>The owning user, who uploaded the file </td></tr><tr><td>path</td><td>The path (from the 'storage root') to the file </td></tr><tr><td>size</td><td>The size in bytes of the file on disk </td></tr><tr><td>mimetype</td><td>The mimetype of the file, as provided by the uploader </td></tr><tr><td>upload_date</td><td>When the data was uploaded </td></tr><tr><td>resource</td><td>The ID of the resource it belongs to. A unidirectional relationship. </td></tr><tr><td>archived_url</td><td>The URL where this file has been archived </td></tr></table> <p> Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files. </p> <p> <strong>Notifications</strong> </p> <p> We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store. </p> <p> We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file). </p> http://localhost/ticket/2732 http://localhost/ticket/2732 Report johnglover <h2 id="Wheredoesthedatago">Where does the data go?</h2> <p> In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url. </p> <p> The user should have rights to create tables. </p> <h2 id="Whatstheapilike">Whats the api like?</h2> <p> We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern. </p> <h2 id="Whataretheinitiallogicfunctions">What are the initial logic functions?</h2> <ul><li>datastore_create </li><li>datastore_delete </li><li>datastore_show </li></ul><h2 id="WhatistheJSONinputformatfordatastore_create">What is the JSON input format for datastore_create</h2> <p> To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id. </p> <pre class="wiki">{ resource_id: resource_id # the data is going to be stored against. fields: a list of dictionaries of fields/columns and their extra metadata. records: a list of dictionaries of the data eg [{"dob": "2005", "some_stuff": ['a', b']}, ..] } </pre><ul><li>The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions. </li><li>rows: can be empty so that you can just set the fields </li><li>fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering. </li></ul><p> eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though. </p> <ul><li>Any error results in total failure!! For now pass back the actual error. </li><li>Should be transactional </li></ul><h2 id="Whatjsondoesdatastore_deletetake">What json does datastore_delete take?</h2> <pre class="wiki">{ resource_id: resource_id # the data is going to be deleted. filters: dictionary of matching conditions to delete e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ". No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table. } </pre><h2 id="Whatjsondoesdatastore_searchtake">What json does datastore_search take?</h2> <pre class="wiki">{ resource_id: resource_id # the data is going to be selected. filters : dictionary of matching conditions to select e.g {'key1': 'a. 'key2': 'b'} this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' " q: full text query limit: limit the amount of rows to size default 100 offset: offset the amount of rows fields: list of fields return in that order, defaults (empty or not present) to all fields in fields order. sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc" } </pre><p> Some free code: <a class="ext-link" href="https://gist.github.com/3163864"><span class="icon">​</span>https://gist.github.com/3163864</a> </p> <h2 id="Whatjsondoesdatastore_searchreturn">What json does datastore_search return?</h2> <pre class="wiki">{ fields: same type as datastore_create accepts (i.e. with metadata) offset: The same offset that was supplied in datastore_show limit: The original limit filters: The filters that were applied in data_show total: # total matching records without size or offset records: [same as data_create] # list of matching results } </pre><p> On error will return: </p> <pre class="wiki">{ __error__: … sql error … } </pre><h2 id="Whattypesareallowed">What types are allowed?</h2> <p> Aim to support as many postgres/postgis types that have string representations. </p> <p> <a class="ext-link" href="http://www.postgresql.org/docs/9.1/static/datatype.html"><span class="icon">​</span>http://www.postgresql.org/docs/9.1/static/datatype.html</a> </p> <p> <a class="ext-link" href="http://www.postgresql.org/docs/9.1/static/sql-createdomain.html"><span class="icon">​</span>http://www.postgresql.org/docs/9.1/static/sql-createdomain.html</a> </p> <h2 id="IDs">IDs</h2> <p> Each row in a table will be given an _id column which has an id generated by us which you can use in queries. </p> <h2 id="OtherFeatures">Other Features</h2> <p> Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc. </p> http://localhost/ticket/2733 http://localhost/ticket/2733 Report shevski <p> Steps to reproduce: </p> <ol><li>go to edit a resource, e.g. <a class="ext-link" href="http://s031.okserver.org:2375/dataset/ted/resource_edit/a621a6ad-6f1c-46d1-b858-03cdb7a25c07"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/ted/resource_edit/a621a6ad-6f1c-46d1-b858-03cdb7a25c07</a> </li></ol><ol start="2"><li>Change the format field to something it doesn't get. e.g. add a comma "HTML," </li></ol><ol start="3"><li>Get message: "The form contains invalid entries: </li></ol><p> Last modified: Date format incorrect Size: Invalid integer" </p> <ol start="4"><li>Format field reverts, try to re-save/submit form - i.e. click on "update resource" </li></ol><ol start="5"><li>nothing happens (same if you even try changing the format field </li></ol> http://localhost/ticket/2734 http://localhost/ticket/2734 Report shevski <p> I think the datasets on user pages <a class="ext-link" href="http://s031.okserver.org:2375/user/me"><span class="icon">​</span>http://s031.okserver.org:2375/user/me</a> should be ordered by latest updated (with most recent at the top) instead of in alphabetical order. </p> <p> What do you think? </p> http://localhost/ticket/2735 http://localhost/ticket/2735 Report ross <p> We need to check the archiver to make sure it works as we would expect after the changes in <a class="assigned ticket" href="http://localhost/ticket/2732" title="enhancement: New file upload functionality (assigned)">#2732</a>: </p> <ul><li>Receive notification of a file that needs processing </li><li>Check the file's DB entry and upload the file to the configured remote service </li><li>Update the URL of the file (both in the file's DB entry and the resource) </li><li>Once we're sure the file is safely archived, we should mark the file as being archived </li></ul><blockquote> <blockquote> <p> so that we know it can safely be deleted at some point. </p> </blockquote> </blockquote> <p> Maybe this could/should be implemented as a service rather than a celery task? </p> http://localhost/ticket/2736 http://localhost/ticket/2736 Report shevski <p> Should be basic format (which is what the datastore wants &amp; what we display to users) e.g. "CSV, XML, JSON etc" </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_resource/"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_resource/</a> </p> http://localhost/ticket/2737 http://localhost/ticket/2737 Report shevski <p> Why is datastore not enabled for this resource <a class="ext-link" href="http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd</a> ? </p> <p> Is this going to be a site wide setting? How do new resources get added to datastore? Or are we just using the dataproxy for demo? </p> http://localhost/ticket/2738 http://localhost/ticket/2738 Report shevski <p> Can we please add some text next to the format field when user adding a new resource along the lines of: </p> <p> "This will be generated automatically, but you can edit if you wish" </p> <p> It always seems to work better when the user does not fill it themselves... </p> http://localhost/ticket/2739 http://localhost/ticket/2739 Report shevski <p> When adding a resource, the breadcrumb should say "add data" or "add resource" not "create dataset" as you see here: <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new_resource/frere"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new_resource/frere</a> </p> http://localhost/ticket/2740 http://localhost/ticket/2740 Report toby <p> <a class="ext-link" href="http://localhost:5000/ckan-admin"><span class="icon">​</span>http://localhost:5000/ckan-admin</a> </p> <p> you need to be an admin </p> <p> paster sysadmin add &lt;username&gt; -c dev.ini </p> <p> we need to get links to other admin pages etc - I think this will want to be flexible so extensions can add extra links etc - but that can come later </p> http://localhost/ticket/2741 http://localhost/ticket/2741 Report toby <p> <a class="ext-link" href="http://localhost:5000/ckan-admin/authz"><span class="icon">​</span>http://localhost:5000/ckan-admin/authz</a> </p> <p> you need to be an admin </p> <p> paster sysadmin add &lt;username&gt; -c dev.ini </p> <p> like <a class="closed ticket" href="http://localhost/ticket/2741" title="enhancement: admin index styling (closed: fixed)">#2741</a> we need other admin links </p> <p> you may want to make the inputs use your form macros - I've just done a basic port </p> <p> also we will need to get all the tables done but looks like we need to get auth groups working for that </p> http://localhost/ticket/2742 http://localhost/ticket/2742 Report shevski <p> Currently the 'add dataset' button on a group page <a class="ext-link" href="http://s031.okserver.org:2375/group/another-group-in-the-wall"><span class="icon">​</span>http://s031.okserver.org:2375/group/another-group-in-the-wall</a> goes to edit group. This is confusing since the interface for this is no way streamlined or helpful. </p> <p> Can we get this to go to the create dataset process instead (ideally with the group pre-chosen) for now? </p> <p> Further on will need to re-think &amp; plan the workflow for publishers which will be a large part of new work </p> http://localhost/ticket/2743 http://localhost/ticket/2743 Report toby <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/new"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/new</a> </p> <p> license dropdown - when open we need some line spacing at least in FF I think it'd be nice to fix this in phase 2 </p> http://localhost/ticket/2744 http://localhost/ticket/2744 Report amercader <p> Instead of showing a notice, the password reset page throws an exception if the key parameter is missing: </p> <pre class="wiki">Module ckan.controllers.user:329 in perform_reset c.reset_key = request.params.get('key') if not mailer.verify_reset_link(user_obj, c.reset_key): h.flash_error(_('Invalid reset key. Please try again.')) abort(403) if not mailer.verify_reset_link(user_obj, c.reset_key): Module ckan.lib.mailer:100 in verify_reset_link if not user.reset_key or len(user.reset_key) &lt; 5: return False return key.strip() == user.reset_key return key.strip() == user.reset_key AttributeError: 'NoneType' object has no attribute 'strip' </pre><p> Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param: </p> <p> <a class="ext-link" href="http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8"><span class="icon">​</span>http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8</a> </p> <p> instead of: </p> <p> <a class="ext-link" href="http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8"><span class="icon">​</span>http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8</a> </p> http://localhost/ticket/2745 http://localhost/ticket/2745 Report shevski <p> Hey, this is still missing. It's the one with the add dataset text </p> http://localhost/ticket/2746 http://localhost/ticket/2746 Report shevski <p> For example here: <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices</a> The breadcrumb shows "gold-prices" which is not the title of the dataset </p> http://localhost/ticket/2747 http://localhost/ticket/2747 Report shevski <p> When editing a resource you see the current and any other existing resources in sidebar </p> <p> (e.g. see <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True</a>) </p> <p> Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page </p> http://localhost/ticket/2748 http://localhost/ticket/2748 Report toby <p> title broken for </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/new_resource/gold-prices</a> </p> <p> we have inherited the title from the extended page not from the base </p> <p> should be 'add data - ckan' not 'add data - create dataset - ckan' </p> <p> any idea how we fix this? </p> http://localhost/ticket/2749 http://localhost/ticket/2749 Report toby <p> we need to ensure that translations in .js files are ripper for translation </p> <p> look at what is available as preference to creating our own </p> http://localhost/ticket/2750 http://localhost/ticket/2750 Report toby <p> need to check everything gets translated - sean did this before so will have info </p> http://localhost/ticket/2751 http://localhost/ticket/2751 Report ross <p> When navigating to <a class="ext-link" href="http://s031.okserver.org:2375/group"><span class="icon">​</span>http://s031.okserver.org:2375/group</a> on the demo server, the Group link in the navbar is not highlighted in the same was as the other links are when visiting those pages. </p> http://localhost/ticket/2752 http://localhost/ticket/2752 Report shevski <p> Currently lower case 'active' is the name of this page: <a class="ext-link" href="http://s031.okserver.org:2375/user"><span class="icon">​</span>http://s031.okserver.org:2375/user</a> </p> <p> Should be 'All Users' or 'Active Users' </p> http://localhost/ticket/2753 http://localhost/ticket/2753 Report shevski <p> Specifically the 'add resource' button and the 'add item' button in related section </p> http://localhost/ticket/2754 http://localhost/ticket/2754 Report shevski <p> navigating to a user page -one's own or <a class="ext-link" href="http://s031.okserver.org:2375/en/user/ira"><span class="icon">​</span>http://s031.okserver.org:2375/en/user/ira</a> or <a class="ext-link" href="http://s031.okserver.org:2375/en/user/shevski"><span class="icon">​</span>http://s031.okserver.org:2375/en/user/shevski</a>, even when signed out is noticeably slower than navigating between other parts of the site; over 2 seconds. Maybe even 3 </p> http://localhost/ticket/2755 http://localhost/ticket/2755 Report seanh <p> It doesn't pass extras_as_string </p> http://localhost/ticket/2756 http://localhost/ticket/2756 Report seanh http://localhost/ticket/2757 http://localhost/ticket/2757 Report toby <p> We get an error which should be improved the actual problem is this but not passed to user </p> <blockquote> <p> ⇝ <a class="missing wiki">KeyError?</a>: 'ofs.impl' </p> </blockquote> <p> View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> data = func(*args, <strong>kwargs) </strong></p> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.storage:407 in auth_form view </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> authorize(method, bucket, label, c.userobj, self.ofs) </p> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.storage:200 in ofs view </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> StorageAPIController._ofs_impl = get_ofs() </p> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.storage:71 in get_ofs view </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> storage_backend = config<a class="missing wiki">ofs.impl?</a> </p> </blockquote> </blockquote> </blockquote> <p> Module paste.registry:146 in <span class="underline">getitem</span> view </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self._current_obj()[key] </p> </blockquote> </blockquote> </blockquote> <p> <a class="missing wiki">KeyError?</a>: 'ofs.impl' </p> http://localhost/ticket/2758 http://localhost/ticket/2758 Report toby <p> On the add resource forms new or existing dataset </p> <p> when hovering over <tt>Upload a file</tt> css cursor should be pointer not text </p> http://localhost/ticket/2759 http://localhost/ticket/2759 Report shevski <p> see spacing between 'download' button and 'data api' button <a class="ext-link" href="http://s031.okserver.org:2375/dataset/afterfibre/resource/66f5b980-06db-4ccd-a005-c18de4d68a4b"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/afterfibre/resource/66f5b980-06db-4ccd-a005-c18de4d68a4b</a> </p> http://localhost/ticket/2760 http://localhost/ticket/2760 Report seanh <p> Add it to a 'Troubleshooting' section on the filestore page: <a class="ext-link" href="http://docs.ckan.org/en/ckan-1.7.1/filestore.html"><span class="icon">​</span>http://docs.ckan.org/en/ckan-1.7.1/filestore.html</a> </p> <p> For the error messages and their solutions, see various threads on ckan-dev </p> http://localhost/ticket/2761 http://localhost/ticket/2761 Report seanh <p> /home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py </p> <p> <a class="missing wiki">ImportError?</a> (cannot import name assert_regexp_matches) </p> http://localhost/ticket/2762 http://localhost/ticket/2762 Report seanh <p> test_multilingual_plugin.<a class="missing wiki">TestDatasetTermTranslation?</a>.test_dataset_index_translation, test_multilingual_plugin.<a class="missing wiki">TestDatasetTermTranslation?</a>.test_group_read_translation both failing for me on master </p> http://localhost/ticket/2763 http://localhost/ticket/2763 Report rgrp <p> Definitely do: </p> <ul><li>Local file filestore: Remove pairtree (and OFS) and do something very simple </li><li> </li></ul><p> Options: </p> <ol><li>Local file only. Allow for uploaders to GS / S3 in the background <ul><li>Advantages: simpler, if local upload can do progress bars etc etc </li><li>Disadvantages: strain for web-app (upload a 2GB file what happens) <ul><li>This is probably fixable ... </li></ul></li></ul></li><li>Stick with how we are </li></ol><h3 id="Newfilestorewithoutpairtree">New filestore without pairtree</h3> <p> Each uploaded file is located on disk at: </p> <p> {uuid}/{filename} </p> <p> Alternative: </p> <p> yyyy/mm/dd/{uuid}/{filename} </p> <p> Need the uuid to avoid collisions. </p> <p> Metadata: Store no metadata. </p> http://localhost/ticket/2764 http://localhost/ticket/2764 Report aron.carroll <p> I'd suggest we add a generic "failed to login, invalid fields" in the error_summary block. Then more specific "this field is required" when a required field is left empty. </p> <p> So in general, provide an error dict </p> http://localhost/ticket/2765 http://localhost/ticket/2765 Report shevski <p> The new ckan creates datasets as part of a 3 phase process. To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'. These datasets should not be seen as active by the activity stream. </p> <p> If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created. When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created </p> <p> We do not want these adding. Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it. </p> <p> finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream. </p> <p> The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there. </p> <p> Let me know if you need any help with this ticket or a better explanation of the problem. </p> http://localhost/ticket/2766 http://localhost/ticket/2766 Report aron.carroll <p> This keeps the intended order. </p> http://localhost/ticket/2767 http://localhost/ticket/2767 Report shevski <p> so that data proxy works </p> http://localhost/ticket/2768 http://localhost/ticket/2768 Report shevski <p> Can we add the related dashboard (<a class="ext-link" href="http://trac.ckan.org/ticket/2347"><span class="icon">​</span>http://trac.ckan.org/ticket/2347</a>) to demo - central place to show off all the related stuff added. </p> http://localhost/ticket/2769 http://localhost/ticket/2769 Report seanh <p> Luca De Santis reported these strings not marked in 1.7.1: </p> <blockquote class="citation"> <ul><li>/user/login.html ("Remember me" &amp; "Sign in") </li><li>/package/search.html and template/package/group/read.html (Tags, Resource Formats, Groups) </li><li>/package/resource_read.html (the "No description" message) </li><li>the "Add related item" dialog (some strings are i18n savvy, others not) </li><li>the stats page (labels on the graphs) </li><li>the history page (as far as the log messages are concerned). </li></ul></blockquote> <p> lets check that they are marked in 1.8 </p> http://localhost/ticket/2770 http://localhost/ticket/2770 Report seanh <p> Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests. </p> <p> The IDatasetForm example should use tag vocabularies (two birds with one stone) </p> <p> The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples. </p> <p> Tab Vocabularies docs should reference IDatasetForm example. </p> <p> When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe) </p> <p> There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date. </p> http://localhost/ticket/2771 http://localhost/ticket/2771 Report seanh <p> Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first </p> http://localhost/ticket/2772 http://localhost/ticket/2772 Report markw <p> The about page for the <a class="missing wiki">DataHub?</a> (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh. </p> http://localhost/ticket/2773 http://localhost/ticket/2773 Report aron.carroll <p> We need to be able to do the following: </p> <ul><li>Rip the localised strings from the <a class="missing wiki">JavaScript?</a> source code. These are strings that look like <tt>_('String')</tt>, <tt>translate('String') </tt> and <tt> isPlural('String') </tt>. </li><li>An endpoint that provides the translations in the correct locale for the user. </li><li>The endpoint should provide the locale as JSON (possibly using <a class="ext-link" href="http://jsgettext.berlios.de/doc/html/po2json.html"><span class="icon">​</span>http://jsgettext.berlios.de/doc/html/po2json.html</a> or similar tool) </li></ul><p> The <a class="missing wiki">JavaScript?</a> can then read this file on page load. </p> http://localhost/ticket/2774 http://localhost/ticket/2774 Report aron.carroll <p> Ideally the ./bin/less command would be run when the server is started. </p> <ul><li>Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }} </li><li>The command could also detect missing node binaries and redirect to the documentation. </li></ul> http://localhost/ticket/2775 http://localhost/ticket/2775 Report shevski <p> to use on demos instead of the ckan logo &amp; tag line. </p> <p> Design here: <a class="ext-link" href="https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381"><span class="icon">​</span>https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381</a> (First, text only one at the bottom) </p> http://localhost/ticket/2776 http://localhost/ticket/2776 Report icmurray http://localhost/ticket/2777 http://localhost/ticket/2777 Report toby <p> apps dashboard available at <a class="ext-link" href="http://localhost:5000/apps"><span class="icon">​</span>http://localhost:5000/apps</a> in main dev branch </p> <p> I have got this to work in the demo however </p> <p> a) it is a bit ugly </p> <p> b) the template could do with some sorting eg the dropdowns should be being created in a nicer way </p> http://localhost/ticket/2778 http://localhost/ticket/2778 Report shevski <p> Currently, as a logged in admin, you can view deleted datasets just as if they are live datasets. Nothing distinguishes them apart from normal datasets other than they don't show up on the dataset page </p> <p> This is potentially very confusing Can we add some kind of banner / label to deleted datasets to show they have been deleted? </p> <p> &amp; Do you guys need Sam to design this? </p> <p> Related ticket for undeleting datasets - <a class="ext-link" href="http://trac.ckan.org/ticket/2780"><span class="icon">​</span>http://trac.ckan.org/ticket/2780</a> </p> http://localhost/ticket/2779 http://localhost/ticket/2779 Report shevski <p> Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted &amp; not viewable by normal users - ticket: <a class="closed ticket" href="http://localhost/ticket/2779" title="enhancement: add 'deleted' to deleted datasets that admins can view (closed: fixed)">#2779</a> b) way to undelete such datasets - this ticket </p> <p> I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message? </p> http://localhost/ticket/2780 http://localhost/ticket/2780 Report shevski <p> demo.ckan.org has a small field for adding text on homepage current text: "Welcome to the CKAN demo Try out standard CKAN functionality in a sandbox environment. Search for datasets directly from the homepage or by navigating to the Datasets search page where you can facet by tags, groups and format." </p> <p> Want to make this editable for demos </p> http://localhost/ticket/2781 http://localhost/ticket/2781 Report shevski <p> use <a class="ext-link" href="http://leaflet.cloudmade.com/"><span class="icon">​</span>http://leaflet.cloudmade.com/</a> properly to generate map backgrounds for demo instances </p> http://localhost/ticket/2782 http://localhost/ticket/2782 Report shevski <p> Really we want this for CKAN Hosted &amp; for creating demos. </p> <p> Currently we're using <a class="ext-link" href="http://leaflet.cloudmade.com/"><span class="icon">​</span>http://leaflet.cloudmade.com/</a> - but by finding the area &amp; taking screenshots. Would be better to interate with the app so that: a) this happens automatically and b) correct attribution will be visible on maps </p> http://localhost/ticket/2783 http://localhost/ticket/2783 Report icmurray <p> The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not. </p> http://localhost/ticket/2784 http://localhost/ticket/2784 Report aron.carroll <p> Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource? </p> <p> Ira, what are your thoughts? </p> http://localhost/ticket/2785 http://localhost/ticket/2785 Report shevski <p> e.g. if I click on download here: <a class="ext-link" href="http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b</a> can it take me to a new tab instead of navigating away from ckan? </p> http://localhost/ticket/2786 http://localhost/ticket/2786 Report aron.carroll <p> Currently the stats plugin has a template and templates_legacy directory. We need to ensure that the templates in templates_legacy are loaded. </p> http://localhost/ticket/2787 http://localhost/ticket/2787 Report amercader <p> Specially needed when importing large numbers of datasets. </p> <p> Profiling the import command from the harvesting extension has shown some areas where improvements could be made. </p> http://localhost/ticket/2788 http://localhost/ticket/2788 Report toby <p> With the text logo I'd like the tag line to be under the site name so some css like </p> <p> <tt>float:left;clear:left;font-size:13px;line-height:0.1;</tt> </p> <p> Also the site name can be larger if no tagline </p> <p> commit daac54dbd856 I've added a new global g.header_css that sets the hgroup class to </p> <p> 'header-image' 'header-text-logo' 'header-text-logo-tagline' </p> <p> I think maybe we should also have 'header-image-css' for the equivalent of the class="logo" in the link to allow/stop the image replacement </p> http://localhost/ticket/2789 http://localhost/ticket/2789 Report toby <p> This is a very slow call it would benefit from the sort of speed-ups that package-search received </p> <p> for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization </p> <p> can we just grab json blobs from solr? </p> <p> also is it possible to specify a sort order/paging? </p> <p> I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now </p> http://localhost/ticket/2790 http://localhost/ticket/2790 Report toby <p> @aron </p> <p> We should be able to have longish tag lines that look ok currently they wrap and stop the logo text being clickable </p> <p> Can we have <tt>white-space:nowrap;</tt> for the tagline so long ones don't wrap and maybe look at aligning long ones </p> http://localhost/ticket/2791 http://localhost/ticket/2791 Report toby <p> <a class="ext-link" href="http://s031.okserver.org:2375/"><span class="icon">​</span>http://s031.okserver.org:2375/</a> language dropdown expands downwards it should open upwards </p> <p> select2 can do this see <a class="ext-link" href="http://ivaynberg.github.com/select2/#documentation"><span class="icon">​</span>http://ivaynberg.github.com/select2/#documentation</a> can you see if we can get this in ckan demo </p> http://localhost/ticket/2792 http://localhost/ticket/2792 Report ross <p> As per <a class="closed ticket" href="http://localhost/ticket/2650" title="enhancement: Description on resources not rendered as markdown (closed: fixed)">#2650</a> (on master/1.8 now) </p> <p> If the resource has a description it isn't rendered as markdown .. </p> http://localhost/ticket/2793 http://localhost/ticket/2793 Report toby <p> <a class="ext-link" href="http://s031.okserver.org:2375/dataset/resource-markdown/resource_edit/ba1b094d-697c-47c4-a72a-6c2557da5409"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/resource-markdown/resource_edit/ba1b094d-697c-47c4-a72a-6c2557da5409</a> </p> <p> should have <tt>back to resource</tt> as a minimum </p> http://localhost/ticket/2794 http://localhost/ticket/2794 Report toby <p> Ensure that we are being standards compliant </p> http://localhost/ticket/2795 http://localhost/ticket/2795 Report ross <p> A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data. </p> http://localhost/ticket/2796 http://localhost/ticket/2796 Report toby <p> <a class="ext-link" href="http://localhost:5000/ckan-admin/config"><span class="icon">​</span>http://localhost:5000/ckan-admin/config</a> uses a plain (non markup) textarea for custom css </p> <p> I've added form macro plain_textarea to provide one, however it renders without a bottom border due to markup textarea styling can you fix this - thanks </p> http://localhost/ticket/2797 http://localhost/ticket/2797 Report toby <p> this want to </p> <p> a) be a link on the left of the form like deletes </p> <p> b) needs a confirm like delete </p> <p> c) wants to work with the autoform </p> http://localhost/ticket/2798 http://localhost/ticket/2798 Report aron.carroll <p> Also they seem to be missing the odd <tt> _() </tt> block. Such in the new package form. </p> <p> <a class="ext-link" href="https://github.com/okfn/ckan/blob/2375-demo-theme-development/ckan/templates/package/snippets/package_form.html#L44"><span class="icon">​</span>https://github.com/okfn/ckan/blob/2375-demo-theme-development/ckan/templates/package/snippets/package_form.html#L44</a> </p> http://localhost/ticket/2799 http://localhost/ticket/2799 Report shevski <p> be good to do this in phase 3 </p> http://localhost/ticket/2800 http://localhost/ticket/2800 Report shevski <p> e.g. see here: <a class="ext-link" href="http://s031.okserver.org:2375/apps"><span class="icon">​</span>http://s031.okserver.org:2375/apps</a> same with related page on a dataset </p> http://localhost/ticket/2801 http://localhost/ticket/2801 Report aron.carroll <p> This is what we need the JSON to look like. Taken from the Jed documentation (<a class="ext-link" href="http://slexaxton.github.com/Jed/"><span class="icon">​</span>http://slexaxton.github.com/Jed/</a>). </p> <pre class="wiki">{ // The empty string key is used as the configuration // block for each domain "" : { // Domain name "domain" : "the_domain", // Language code "lang" : "en", // Plural form function for language "plural_forms" : "nplurals=2; plural=(n != 1);" }, // Other keys in a domain contain arrays as values // that map to the translations for that key. // Gettext suggests that you use english as your keys // in case the key isn't found, and it can just pass // the value directly through. // Note: by convention, the 0-index location of the translations // is never accessed. It's just a thing, I guess. "a key" : [ null, "the translation", "the plural translations", ... ], // The plural form string is converted into a function // and the value that's passed into the gettext call // is passed into the plural form function. It's result // (plus 1) is the index that the translation comes from. // We're using sprintf interpolation on our keys so we can // then sub in the _actual_ values into the result. "%d key" : [ null, "%d key", "%d keys" ], // Contexts are keys that are just prefixed with a context string // with a unicode \u0004 as the delimiter. // You can use it for anything. Usually it's just for being content aware // in some way (e.g. male vs. female, product vs. category) "context\u0004%d key": [ null, "context %d key", "context %d keys" ] } </pre><p> This looks like an appropriate python library to do the formatting. <a class="ext-link" href="http://pypi.python.org/pypi/pojson/"><span class="icon">​</span>http://pypi.python.org/pypi/pojson/</a> but I'll leave to your discretion. </p> <p> Finally an API endpoint that returns the appropriate request would save us making requests to the i18n directory. </p> <pre class="wiki">/api/i18n/{locale} </pre><p> For example if the current url is <a class="ext-link" href="http://localhost:5000/pl/dataset"><span class="icon">​</span>http://localhost:5000/pl/dataset</a> then the client would request: </p> <pre class="wiki">http://localhost:5000/api/i18n/pl </pre><p> This should return the Content-Type: application/json and a 200 with an appropriate JSON body (this can be the language requested or an appropriate fallback). I think that if the fallback is the default language (in most cases English) then we just return a 204 No Content to save data. </p> http://localhost/ticket/2802 http://localhost/ticket/2802 Report aron.carroll http://localhost/ticket/2803 http://localhost/ticket/2803 Report shevski <p> on resource pages (e.g. <a class="ext-link" href="http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360</a>), same as old ckan for continuity and to highlight! </p> http://localhost/ticket/2804 http://localhost/ticket/2804 Report toby <p> the .po file for zh_TW does not contain Plural-Forms which breaks the i18n <a class="missing wiki">JavaScript?</a> Extractions for ckan-demo. </p> <p> I've added the following which may or may not be correct - from web search "Plural-Forms: nplurals=1; plural=0\n" </p> <p> I suspect it will get overwritten when translations are pulled from transiflex so passing to you as you are the expert in that realm </p> http://localhost/ticket/2805 http://localhost/ticket/2805 Report shevski <p> e.g. currently no way to read full description for the 'data explorer examples' group here:http://s031.okserver.org:2375/group/data-explorer </p> <p> Can we add a 'read more' link that expands and contracts the sidebar box or similar? </p> http://localhost/ticket/2806 http://localhost/ticket/2806 Report toby <p> @aron, </p> <p> edit dataset has a js error </p> <p> <a class="ext-link" href="http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada"><span class="icon">​</span>http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada</a> </p> <p> <a class="missing wiki">TypeError?</a>: options is undefined [Break On This Error] </p> <p> this.lookup(options.term, options.callback); </p> <p> autocomplete.js (line 231) </p> http://localhost/ticket/2807 http://localhost/ticket/2807 Report toby <p> If you make your browser window short - say half screen height on a page with delete eg <a class="ext-link" href="http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada"><span class="icon">​</span>http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada</a> </p> <p> the dialog box is partially off the screen - it should try to stay on screen </p> http://localhost/ticket/2808 http://localhost/ticket/2808 Report toby <p> eg on confirm delete </p> <p> <a class="ext-link" href="http://localhost:5000/zh_TW/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada"><span class="icon">​</span>http://localhost:5000/zh_TW/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada</a> </p> <p> _(....) gives [object Object] not translation </p> http://localhost/ticket/2809 http://localhost/ticket/2809 Report kindly <p> Get ckan working on heroku </p> http://localhost/ticket/2810 http://localhost/ticket/2810 Report seanh <p> Edit a dataset and add an author, dataset read page still says 'author not given'. If you fill out both the author and author email fields then it works. </p> http://localhost/ticket/2811 http://localhost/ticket/2811 Report markw <p> The 'What are Groups?' text at &lt;<a class="ext-link" href="http://demo.ckan.org/group"><span class="icon">​</span>http://demo.ckan.org/group</a>&gt; is a bit naff. Here is a suggested replacement: </p> <p> What are groups? </p> <p> Groups allow you to group users and data together so that they are easier to manage. For example datasets in a group could be visible only to group members, or publicly visible but only editable by group members. </p> <p> Groups can be used to collect together datasets from a single publisher, or related to a particular project, for example. </p> http://localhost/ticket/2812 http://localhost/ticket/2812 Report markw <p> On a dataset page on demo.ckan.org, the left sidebar is confusing. </p> <ul><li>It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear. </li><li>The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll). </li></ul> http://localhost/ticket/2813 http://localhost/ticket/2813 Report markw <ol start="3"><li>Uploading a file behaves counter-intuitively (I would suggest wrongly). </li></ol><p> When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen: </p> <ul><li>the pathname of that file is filled in the box; </li><li>nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc). </li></ul><p> Instead of this, </p> <ul><li>the file is immediately and, seemingly, irrevocably uploaded; </li><li>the box is filled with a mysterious URL (<a class="ext-link" href="https://commondatastorage.googleapis.com/ckan-demo"><span class="icon">​</span>https://commondatastorage.googleapis.com/ckan-demo</a> ...) </li><li>the mysterious URL is so long I can't even see the final element, which is my file name. </li></ul> http://localhost/ticket/2814 http://localhost/ticket/2814 Report seanh <p> If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates. </p> http://localhost/ticket/2815 http://localhost/ticket/2815 Report toby <p> Over-enthusiastic removal of some model items </p> <p> this breaks qa extension </p> http://localhost/ticket/2816 http://localhost/ticket/2816 Report danieljohnlewis <p> Problem: When creating a related item, if you put in a invalid URL it loses the choice of Item Type (e.g. Visualisation, Application) and reverts to the first time (API). Expected: It remembers which one was chosen </p> http://localhost/ticket/2817 http://localhost/ticket/2817 Report danieljohnlewis <p> Problem: When creating a related item (e.g. a Visualisation), if you don't put in a URL it succeeds, but on the related items and apps pages it renders it as a link to the same page. Expected: Always require a URL and it should only submit if one is added </p> http://localhost/ticket/2818 http://localhost/ticket/2818 Report danieljohnlewis <p> Problem: When creating a related item it mentions that you can use markdown in the description, however markdown doesn't render in apps page nor in related items page. Expected: Make shortened descriptions (on apps page &amp; related items page) render markdown </p> http://localhost/ticket/2819 http://localhost/ticket/2819 Report danieljohnlewis <p> Problem: In the English version (which has a UK flag, indicating British English), the word "Visualization" is used. For an example see the "Filter by type" drop down on the /apps page. Expected: This should be "Visualisation" in British English. Any instances of "Visualize" should be changed to "Visualise" too. </p> http://localhost/ticket/2820 http://localhost/ticket/2820 Report danieljohnlewis <p> Problem: On /apps page in the Filter Results box there is a "Only show featured items" checkbox, on selection it comes up with 0 solutions. Expected: Presumably an admin can create "featured items" so that they can be randomly selected on front page (is this correct)? If there are no "featured items" in the whole database can this check box be hidden? Bug is: no UI or obvious way to create featured items. Also the checkbox looks un-styled </p> http://localhost/ticket/2821 http://localhost/ticket/2821 Report toby <p> the title for additional info should be translated </p> <p> capitalised etc </p> http://localhost/ticket/2822 http://localhost/ticket/2822 Report toby <p> Order the items so that none user fields are first from ticket <a class="closed ticket" href="http://localhost/ticket/2707" title="enhancement: Tidy up additional info boxes on dataset and resource pages (closed: fixed)">#2707</a> </p> http://localhost/ticket/2823 http://localhost/ticket/2823 Report danieljohnlewis <p> There is some bizarre language flag issues. 1) Log in to <a class="ext-link" href="http://s031.okserver.org:2375/"><span class="icon">​</span>http://s031.okserver.org:2375/</a> (I'm logged in as danieljohnlewis) 2) Visit the apps page: <a class="ext-link" href="http://s031.okserver.org:2375/apps"><span class="icon">​</span>http://s031.okserver.org:2375/apps</a> 3) Have a look at the languages drop down, flags look fine (although Deutsche does have an EU flag rather than the German flag.. but that's irrelevant) 4) Change filters to, say, type: Application and sort: Newest, and apply. Web address becomes: <a class="ext-link" href="http://s031.okserver.org:2375/apps?page=1&amp;type=application&amp;sort=created_desc"><span class="icon">​</span>http://s031.okserver.org:2375/apps?page=1&amp;type=application&amp;sort=created_desc</a> 5) Look at the languages drop down: Flags are largely incorrect, with more than one country having the same flag. </p> <p> Please see the attached screenshot. </p> <p> Note: this was originally tested with Google Chrome on Linux (Ubuntu). </p> http://localhost/ticket/2824 http://localhost/ticket/2824 Report danieljohnlewis <p> Problem: On /apps page if you start in one language, filter the data, and then change languages it loses the correct format for the HTTP Get Parameters (and therefore loses the filters). Example, start in English language, change filter settings to to Application and Newest, change language to German. Expected: Retain correct format for HTTP Get Parameters so that the filter settings are retained. Solution will involve not turning "=" into "%3D" in the URL, i.e. turn off URL encoding. </p> http://localhost/ticket/2825 http://localhost/ticket/2825 Report danieljohnlewis <p> Problem: Example group ( <a class="ext-link" href="http://s031.okserver.org:2375/en/group/example-group"><span class="icon">​</span>http://s031.okserver.org:2375/en/group/example-group</a> ) has a broken image, image not found. Link either incorrect, or no image set, if no image set then shouldn't it show a default image? Expected: There should be an image. </p> http://localhost/ticket/2826 http://localhost/ticket/2826 Report danieljohnlewis <p> Problem: Server Error (HTTP Code 500) when trying to see information related to an RSS feed... example: <a class="ext-link" href="http://s031.okserver.org:2375/en/dataset/testerisation/resource/974df29e-77bb-4b5e-982c-cb7a295c1923"><span class="icon">​</span>http://s031.okserver.org:2375/en/dataset/testerisation/resource/974df29e-77bb-4b5e-982c-cb7a295c1923</a> Expected: A page that looks a bit more like this: <a class="ext-link" href="http://s031.okserver.org:2375/dataset/epims-lod2/resource/97531bd8-e75c-4efa-bfa2-bb51d0ba90d0"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/epims-lod2/resource/97531bd8-e75c-4efa-bfa2-bb51d0ba90d0</a> </p> http://localhost/ticket/2827 http://localhost/ticket/2827 Report shevski <p> reported as editing datasets incorrect </p> <p> e.g. clicking on edit here <a class="ext-link" href="http://s031.okserver.org:2375/dataset/ff"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/ff</a> takes you to create dataset page <a class="ext-link" href="http://s031.okserver.org:2375/dataset/edit/ff"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/edit/ff</a> </p> <p> but this is the correct behaviour of a draft dataset </p> <p> We need to show draft datasets correctly </p> <p> proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc </p> http://localhost/ticket/2828 http://localhost/ticket/2828 Report toby <p> Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end </p> <pre class="wiki">$ paster archiver update -c ../ckan/development.ini 2012-08-09 11:01:37,636 INFO [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources) 2012-08-09 11:01:37,671 INFO [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13 2012-08-09 11:01:37,900 INFO [ckan.lib.base] /api/action/package_show render time 0.043 seconds Traceback (most recent call last): File "/home/toby/okfn/pyenv/bin/paster", line 8, in &lt;module&gt; load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')() File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run invoke(command, command_name, options, args[1:]) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke exit_code = runner.run(args) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run result = self.command() File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command response = app.post(api_url + '/package_show', data) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post expect_errors=expect_errors) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request return self.do_request(req, status=status) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request self._check_status(status, res) File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status res.body)) paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show) {"help": "Return the metadata of a dataset (package) and its resources.\n\n :param id: the id or name of the dataset\n :type id: string\n\n :rtype: dictionary\n\n ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}} </pre> http://localhost/ticket/2829 http://localhost/ticket/2829 Report toby <p> need controller action and front-end method </p> http://localhost/ticket/2830 http://localhost/ticket/2830 Report aron.carroll <p> Allowing people to use the full range of markdown results in extremely messy output across the site. I'd suggest limiting support to only a subset of common use cases. </p> <ul><li>Allow all inline elements, this allows bold, italic, code and links. </li><li>Allow lists. </li></ul><p> Disallow </p> <ul><li>Horizontal Rules </li><li>Headings </li><li>Block quote and code (this may turn out to be useful and so could be included) </li></ul><p> This way you get markdowns paragraph handling and a few inline styles without breaking the entire layout of the page. </p> <p> Here's the full syntax if anyone is interested <a class="ext-link" href="http://daringfireball.net/projects/markdown/syntax"><span class="icon">​</span>http://daringfireball.net/projects/markdown/syntax</a> </p> http://localhost/ticket/2831 http://localhost/ticket/2831 Report shevski <p> add to groups is a drop down menu where you can only choose one needs a new UI &amp; logic allowing user to add new groups &amp; potentially remove from other lists </p> http://localhost/ticket/2832 http://localhost/ticket/2832 Report aron.carroll <p> I think this would be a nice feature for remote loading templates if the options.template value ends in ".html". </p> <pre class="wiki">ckan.module('my-module', { options: { template: 'my-template.html' }, initialize: function () { this.template // This is the loaded template. } }); </pre> http://localhost/ticket/2833 http://localhost/ticket/2833 Report aron.carroll <p> Currently if a module uses sandbox.publish() in its initilaize() function then any modules initialised afterwards will miss the event. </p> <p> We should queue all calls to .publish() in the pubsub module in an array until all modules have loaded then iterate over and execute them. </p> http://localhost/ticket/2834 http://localhost/ticket/2834 Report aron.carroll <pre class="wiki">Client#getTemplate(name, params, success, error); </pre><p> Where params, success and error are optional arguments. test/index.html already has an implementation called loadFixture(). </p> http://localhost/ticket/2835 http://localhost/ticket/2835 Report toby <p> like we have on the form here </p> <p> <a class="ext-link" href="http://s031.okserver.org:2375/harvest/new"><span class="icon">​</span>http://s031.okserver.org:2375/harvest/new</a> </p> <p> you said it partly exists so maybe I just need some training </p> http://localhost/ticket/2836 http://localhost/ticket/2836 Report toby <p> Is this done I know checkboxes are a pain but it looks a bit out of place compared to the other items. </p> http://localhost/ticket/2837 http://localhost/ticket/2837 Report seanh <p> I was doing this: </p> <pre class="wiki">context = {'model': base.model, 'session': base.model.Session, 'user': toolkit.c.user or toolkit.c.author, 'extras_as_string': True} group_dict = logic.get_action('group_show')(context, {'id': group_id}) </pre><p> in an extension and one of the group_dicts fields, one that uses convert_to/from_extras, was coming out with the wrong value. It took me ages to realise that I had to pass <tt>'extras_as_string': True</tt> in the context. I don't think this or other context variables are documented anywhere. </p> http://localhost/ticket/2838 http://localhost/ticket/2838 Report seanh <p> Create a group, create some datasets and add them to the group, then view the group's page with a sort-order specified in the URL, e.g. <tt>/group/foo?q=&amp;sort=title_string+asc</tt>, no datasets are displayed even though there are datasets in the group that match the search query. Remove the sort order from the URL and the datasets will appear. </p> http://localhost/ticket/2839 http://localhost/ticket/2839 Report toby <p> @aron </p> <p> Is it possible to get the tagline text to float the other way </p> <p> at the moment we have </p> <p> ...........Title Tag line to here </p> <p> I'd like </p> <p> ...........Title ...........Tag line to here </p> <p> is this possible or too much pain </p> <p> based partly on this comment Markw wrote </p> <p> A minor problem visible on the home page &lt;<a class="ext-link" href="http://s031.okserver.org:2375/"><span class="icon">​</span>http://s031.okserver.org:2375/</a>&gt; and other pages, when viewing in both Chrome and Firefox: </p> <p> if the browser width is not very wide, the tag line 'Open source data portal' vanishes off the *left* hand side of the screen. For some reason the browser doesn't even recognise this with a horizontal scroll bar (as it does when stuff is off the *right* hand side) - it just chops off the text. </p> http://localhost/ticket/2840 http://localhost/ticket/2840 Report amercader <p> On the add dataset form, slug generation fails with JS error: </p> <pre class="wiki">Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning' basic-form.js:5 </pre> http://localhost/ticket/2841 http://localhost/ticket/2841 Report seanh http://localhost/ticket/2842 http://localhost/ticket/2842 Report seanh <p> On the organization read page there is only an 'Add Dataset' button, not a drop-down with add new dataset or add existing dataset as on the default group page. </p> <p> On the organization edit page you do have the drop-down, but if you click add existing dataset nothing happens except it appends #datasets to the URL </p> http://localhost/ticket/2843 http://localhost/ticket/2843 Report rgrp <ul><li><del>Search needs to run of local DB (again)</del> </li><li>paster db clean attemps to connect to SOLR (still works as does db first but then excepts which is not nice UX) </li></ul> http://localhost/ticket/2844 http://localhost/ticket/2844 Report seanh <p> Run CKAN with the publisher profile and organizations plugins enabled. Make an organization. Make a dataset and add it to the organization. Edit the organization and change e.g. its description. After saving the organization, all its datasets have been removed from the organization and now belong to no organization. </p> http://localhost/ticket/2845 http://localhost/ticket/2845 Report seanh <p> that doesn't make any sense </p> http://localhost/ticket/2846 http://localhost/ticket/2846 Report shevski <p> On harvest page: <a class="ext-link" href="http://s031.okserver.org:2375/harvest"><span class="icon">​</span>http://s031.okserver.org:2375/harvest</a> the checkbox appears above the text and not clear what it refers to/ </p> <p> Beter to relocate to the left (as on publicdata.eu) see screenshots attached. </p> http://localhost/ticket/2847 http://localhost/ticket/2847 Report shevski <p> <a class="ext-link" href="http://s031.okserver.org:2375/"><span class="icon">​</span>http://s031.okserver.org:2375/</a> </p> http://localhost/ticket/2848 http://localhost/ticket/2848 Report shevski <p> should be left alighned, not right aligned. Otherwise tag line disappears left off the page. </p> <p> See: header here: <a class="ext-link" href="http://s031.okserver.org:2375/ckan-admin/config"><span class="icon">​</span>http://s031.okserver.org:2375/ckan-admin/config</a> </p> http://localhost/ticket/2849 http://localhost/ticket/2849 Report aron.carroll <p> The <a class="missing wiki">JavaScript?</a> just isn't going to work without it. Look into either the newer JSON3: </p> <p> <a class="ext-link" href="http://bestiejs.github.com/json3/"><span class="icon">​</span>http://bestiejs.github.com/json3/</a> </p> <p> or the old faithful: </p> <p> <a class="ext-link" href="https://github.com/douglascrockford/JSON-js/"><span class="icon">​</span>https://github.com/douglascrockford/JSON-js/</a> </p> http://localhost/ticket/2850 http://localhost/ticket/2850 Report ross <p> e.g. <a class="ext-link" href="http://datahub.io/dataset/ccc-gistemp/resource/80ebdbd6-d91b-4fef-9db4-d3dfbd7e868e"><span class="icon">​</span>http://datahub.io/dataset/ccc-gistemp/resource/80ebdbd6-d91b-4fef-9db4-d3dfbd7e868e</a> </p> <p> CKAN attempts to render the PDF as a datastore tabular date even though the mimetype of the PDF is correctly specified. </p> http://localhost/ticket/2851 http://localhost/ticket/2851 Report shevski <p> Currently if you try to refresh (i.e. schedule the harvester to re-run) an inactive source (one who's state is 'withdrawn' - e.g. <a class="ext-link" href="http://s031.okserver.org:2375/harvest/3feea629-32d8-4431-9957-9f31d32864f1"><span class="icon">​</span>http://s031.okserver.org:2375/harvest/3feea629-32d8-4431-9957-9f31d32864f1</a>) </p> <p> you get the error message: "An error occurred: [Can not create jobs on inactive sources]" </p> <p> This could be better and not red, e.g. and info box (maybe yellow) saying that "Cannot re-run harvesting on inactive sources. Please first change the status to 'active'" </p> http://localhost/ticket/2852 http://localhost/ticket/2852 Report shevski <p> Tried to refresh the datagm harvest source to get error message: </p> <p> "An error occurred: [There already is an unrun job for this source]" </p> <p> This is unclear and not sure why it created an error. We should have something informative such as "A harvest has already been scheduled for this source" </p> <p> Do we have a date/time for when this has been scheduled that we can display? </p> http://localhost/ticket/2853 http://localhost/ticket/2853 Report shevski <p> e.g. from <a class="ext-link" href="http://s031.okserver.org:2375/apps?page=1"><span class="icon">​</span>http://s031.okserver.org:2375/apps?page=1</a> you cannot edit a related item as an admin or view what dataset the item is related to </p> http://localhost/ticket/2854 http://localhost/ticket/2854 Report shevski <p> e.g. tried changing the lang here: <a class="ext-link" href="http://s031.okserver.org:2375/"><span class="icon">​</span>http://s031.okserver.org:2375/</a> to no effect. Doesn't work on any other pages either. Flags have also dissappeared </p> http://localhost/ticket/2855 http://localhost/ticket/2855 Report shevski <ol><li>go to <a class="ext-link" href="http://s031.okserver.org:2375/user/okfn"><span class="icon">​</span>http://s031.okserver.org:2375/user/okfn</a> </li><li>Click on activity stream </li><li>Click on 'Deletion test' related item that's linked to in entries 1 &amp; 2 </li><li>Server error </li></ol> http://localhost/ticket/2856 http://localhost/ticket/2856 Report shevski <p> e.g. second option in left hand sidebar is follow another user' <a class="ext-link" href="http://s031.okserver.org:2375/user/dashboard"><span class="icon">​</span>http://s031.okserver.org:2375/user/dashboard</a> but if you click on that and then any user, there's no follow button </p> <p> to me the dashboard user stuff is highly unfinished and probably should not be included unless someone is very keen for it </p> http://localhost/ticket/2857 http://localhost/ticket/2857 Report toby <p> I've had a look at the new harvesting theme, it looks great, thanks. I did some minor changes and reworded the errors as per tickets <a class="closed ticket" href="http://localhost/ticket/2852" title="enhancement: better message for trying to refresh inactive sources in harvest (closed: fixed)">#2852</a> and <a class="closed ticket" href="http://localhost/ticket/2853" title="enhancement: unclear error message in refreshing harvest source (closed: fixed)">#2853</a>, but I kept the messages red, as the only other ones available ("flash_notice") are blue and are not very prominent. </p> <p> Toby, on the old form there was a list of descriptions for all available harvesters which could be nice to have back. I couldn't find an easy way with the new form macro, so I don't know how easy would be to implement </p> <p> <a class="ext-link" href="http://i.imgur.com/pzS8z.png"><span class="icon">​</span>http://i.imgur.com/pzS8z.png</a> </p> http://localhost/ticket/2858 http://localhost/ticket/2858 Report icmurray <p> The requires files have been removed. These were required by buildkit. </p> <p> We've lost the information regarding which packages were already in ubuntu repos, which conflicted and which were missing. But perhaps we can just package everything up. </p> http://localhost/ticket/2859 http://localhost/ticket/2859 Report shevski <p> can we get the link going to an explanation of the 5 star system on page <a class="ext-link" href="http://s031.okserver.org:2375/qa/dataset/five_stars"><span class="icon">​</span>http://s031.okserver.org:2375/qa/dataset/five_stars</a> to open in new tab? it's leaving demo all together so better to open in new tab than navigate away </p> http://localhost/ticket/2860 http://localhost/ticket/2860 Report shevski <p> can we add in the breadcrumb navigation to qa pages so that you can go back to <a class="ext-link" href="http://s031.okserver.org:2375/qa"><span class="icon">​</span>http://s031.okserver.org:2375/qa</a> from sub pages such as <a class="ext-link" href="http://s031.okserver.org:2375/qa/dataset/five_stars"><span class="icon">​</span>http://s031.okserver.org:2375/qa/dataset/five_stars</a> </p> http://localhost/ticket/2861 http://localhost/ticket/2861 Report markw <p> Revised text for 'What are groups?' box at demo.ckan.org/group (after discussion with IB re <a class="closed ticket" href="http://localhost/ticket/2812" title="enhancement: Groups description needs improving (closed: invalid)">#2812</a>): </p> <p> What are groups? </p> <p> Groups allow you to group users and data together so that they are easier to manage. Group owners can assign roles and authorisations, giving each project or department control of its own data publishing. </p> <p> Users can browse or facet by groups, which could be an organisation (for example, the Department of Health) or topic (e.g. Transport, Health), making it easier to find the data they are looking for. </p> http://localhost/ticket/2862 http://localhost/ticket/2862 Report shevski <p> edit button shows up for daniel lewis for <a class="ext-link" href="http://s031.okserver.org:2375/dataset/ff"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/ff</a> </p> http://localhost/ticket/2863 http://localhost/ticket/2863 Report shevski <p> e.g. I can see 'You haven't created any datasets. Create one now?. ' <a class="ext-link" href="http://s031.okserver.org:2375/user/kindly"><span class="icon">​</span>http://s031.okserver.org:2375/user/kindly</a> when logged in as shevski </p> http://localhost/ticket/2864 http://localhost/ticket/2864 Report ross <p> URL: <a class="ext-link" href="http://thedatahub.org/el/tag/jutatt%C3%A1sok"><span class="icon">​</span>http://thedatahub.org/el/tag/jutatt%C3%A1sok</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:155 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:131 in <span class="underline">call</span> &lt;&lt; r'\1', oldpath) </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:125 in <span class="underline">call</span> &lt;&lt; </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> if 'paste.testing_variables' in environ and hasattr(response, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.dispatch(controller, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: </p> <blockquote> <p> log.debug("Calling controller class with WSGI interface") </p> </blockquote> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> def load_test_env(self, environ): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:239 in <span class="underline">call</span> &lt;&lt; try: </p> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> <blockquote> <p> finally: </p> <blockquote> <p> model.Session.remove() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:221 in <span class="underline">call</span> &lt;&lt; return response(environ, self.start_response) </p> <blockquote> <p> response = self._dispatch_call() if not start_response_called: </p> <blockquote> <p> self.start_response = start_response </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._dispatch_call() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ<a class="missing wiki">pylons.action_method?</a> = func </p> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> <blockquote> <p> else: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.<span class="underline">name</span>, args) </p> <blockquote> <p> try: </p> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> <p> except HTTPException, httpe: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = 'before_and_this' return func(<strong>args) </strong></p> </blockquote> <p> </p> <blockquote> <p> def _inspect_call(self, func): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return func(<strong>args) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.template:30 in view &lt;&lt; """ </p> <blockquote> <p> try: </p> <blockquote> <p> return render(url) </p> </blockquote> <p> except <a class="missing wiki">TemplateNotFound?</a>: </p> <blockquote> <p> if url.endswith('.html'): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return render(url) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:153 in render &lt;&lt; try: </p> <blockquote> <p> return cached_template(template_name, render_template, </p> <blockquote> <p> loader_class=loader_class) </p> </blockquote> </blockquote> <blockquote> <p> except ckan.exceptions.<a class="missing wiki">CkanUrlException?</a>, e: </p> <blockquote> <p> raise </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> loader_class=loader_class) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.templating:249 in cached_template &lt;&lt; return content </p> <blockquote> <p> else: </p> <blockquote> <p> return render_func() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return render_func() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:95 in render_template &lt;&lt; template = globs<a class="missing wiki">app_globals?</a>.genshi_loader.load(template_name, </p> <blockquote> <p> cls=loader_class) </p> </blockquote> <blockquote> <p> stream = template.generate(<strong>globs) </strong></p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> cls=loader_class) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.loader:223 in load &lt;&lt; loadfunc = directory(loadfunc) </p> <blockquote> <p> try: </p> <blockquote> <p> filepath, filename, fileobj, uptodate = loadfunc(filename) </p> </blockquote> <p> except IOError: </p> <blockquote> <p> continue </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> filepath, filename, fileobj, uptodate = loadfunc(filename) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.loader:286 in _load_from_directory &lt;&lt; def _load_from_directory(filename): </p> <blockquote> <p> filepath = os.path.join(path, filename) fileobj = open(filepath, 'U') mtime = os.path.getmtime(filepath) def _uptodate(): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> fileobj = open(filepath, 'U') </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> <a class="missing wiki">UnicodeEncodeError?</a>: 'ascii' codec can't encode character u'\xe1' in position 73: ordinal not in range(128) CGI Variables CKAN_CURRENT_URL '/el/tag/jutatt%C3%A1sok' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +<a class="ext-link" href="http://www.google.com/bot.html"><span class="icon">​</span>http://www.google.com/bot.html</a>)' PATH_INFO '/el/tag/jutatt\xc3\xa1sok' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/el/tag/jutatt\xc3\xa1sok' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '49520' REQUEST_METHOD 'GET' REQUEST_URI '/el/tag/jutatt%c3%a1sok' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7f5312a51650&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7f5312a51790&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7f5312a516d0&gt;&gt; beaker.session {'_accessed_time': 1345113298.778472, '_creation_time': 1345113298.778472} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script <em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host </em> mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: &gt;, <em>) paste.registry &lt;paste.registry.Registry object at 0x7f5309add610&gt; paste.throw_errors True pylons.action_method &lt;bound method <a class="missing wiki">TemplateController?</a>.view of &lt;ckan.controllers.template.<a class="missing wiki">TemplateController?</a> object at 0x7f52b0ea2d10&gt;&gt; pylons.controller &lt;ckan.controllers.template.<a class="missing wiki">TemplateController?</a> object at 0x7f52b0ea2d10&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7f52b0ea2b90&gt; pylons.routes_dict {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'} repoze.who.logger &lt;logging.Logger object at 0x7f53127b2b50&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 139994769488336&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 139994769488528&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7f5312d893d0&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 139994771858704&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350&gt;} routes.route &lt;routes.route.Route object at 0x7f531296ea10&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f5309add810&gt; webob._parsed_query_vars (GET([]), </em>) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52d4004990&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f5309add810&gt;, {'url': u'el/tag/jutatt\xe1sok', 'action': u'view', 'controller': u'template'}) </p> http://localhost/ticket/2865 http://localhost/ticket/2865 Report ross <p> URL: <a class="ext-link" href="http://thedatahub.org/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8"><span class="icon">​</span>http://thedatahub.org/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:155 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:131 in <span class="underline">call</span> &lt;&lt; r'\1', oldpath) </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:125 in <span class="underline">call</span> &lt;&lt; </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> if 'paste.testing_variables' in environ and hasattr(response, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.dispatch(controller, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: </p> <blockquote> <p> log.debug("Calling controller class with WSGI interface") </p> </blockquote> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> def load_test_env(self, environ): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:239 in <span class="underline">call</span> &lt;&lt; try: </p> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> <blockquote> <p> finally: </p> <blockquote> <p> model.Session.remove() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:221 in <span class="underline">call</span> &lt;&lt; return response(environ, self.start_response) </p> <blockquote> <p> response = self._dispatch_call() if not start_response_called: </p> <blockquote> <p> self.start_response = start_response </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._dispatch_call() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ<a class="missing wiki">pylons.action_method?</a> = func </p> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> <blockquote> <p> else: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.<span class="underline">name</span>, args) </p> <blockquote> <p> try: </p> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> <p> except HTTPException, httpe: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = 'before_and_this' return func(<strong>args) </strong></p> </blockquote> <p> </p> <blockquote> <p> def _inspect_call(self, func): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return func(<strong>args) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.user:409 in perform_reset &lt;&lt; c.reset_key = request.params.get('key') </p> <blockquote> <p> if not mailer.verify_reset_link(user_obj, c.reset_key): </p> <blockquote> <p> h.flash_error(_('Invalid reset key. Please try again.')) abort(403) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> if not mailer.verify_reset_link(user_obj, c.reset_key): </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.mailer:100 in verify_reset_link &lt;&lt; if not user.reset_key or len(user.reset_key) &lt; 5: </p> <blockquote> <p> return False </p> </blockquote> <blockquote> <p> return key.strip() == user.reset_key </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return key.strip() == user.reset_key </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> <a class="missing wiki">AttributeError?</a>: '<a class="missing wiki">NoneType?</a>' object has no attribute 'strip' CGI Variables CKAN_CURRENT_URL '/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%253Db4c2d03fa8' CKAN_LANG 'it' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +<a class="ext-link" href="http://www.google.com/bot.html"><span class="icon">​</span>http://www.google.com/bot.html</a>)' PATH_INFO '/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d' QUERY_STRING 'key%3Db4c2d03fa8' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '54634' REQUEST_METHOD 'GET' REQUEST_URI '/it/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key%3Db4c2d03fa8' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7f5312a51650&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7f5312a51790&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7f5312a516d0&gt;&gt; beaker.session {'_accessed_time': 1345066159.520708, '_creation_time': 1345066159.520708} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script <em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host </em> mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: &gt;, <em>) paste.parsed_dict_querystring (<a class="missing wiki">MultiDict?</a>([('key=b4c2d03fa8', </em>)]), 'key%3Db4c2d03fa8') paste.parsed_querystring ([('key=b4c2d03fa8', <em>)], 'key%3Db4c2d03fa8') paste.registry &lt;paste.registry.Registry object at 0x7f52eba637d0&gt; paste.throw_errors True pylons.action_method &lt;bound method <a class="missing wiki">UserController?</a>.perform_reset of &lt;ckan.controllers.user.<a class="missing wiki">UserController?</a> object at 0x7f52e49f7a50&gt;&gt; pylons.controller &lt;ckan.controllers.user.<a class="missing wiki">UserController?</a> object at 0x7f52e49f7a50&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7f52f5649ad0&gt; pylons.routes_dict {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'} repoze.who.logger &lt;logging.Logger object at 0x7f53127b2b50&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 139994769488336&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 139994769488528&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7f5312d893d0&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 139994771858704&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350&gt;} routes.route &lt;routes.route.Route object at 0x7f5312961ad0&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f52de7f6390&gt; webob._parsed_query_vars (GET([('key=b4c2d03fa8', </em>)]), 'key%3Db4c2d03fa8') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52fc4f1a08&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f52de7f6390&gt;, {'action': u'perform_reset', 'controller': u'user', 'id': u'3086e91c-fe09-4a98-92e1-19de67a9ac9d'}) </p> http://localhost/ticket/2866 http://localhost/ticket/2866 Report ross <p> From: &lt;thedatahub.org@…&gt; Date: Thu, Aug 16, 2012 at 10:30 AM Subject: <a class="missing wiki">WebApp?</a> Error: &lt;class 'routes.util.<a class="missing wiki">GenerationException?</a>'&gt;: url_for can only return a string, got unicode instead: <a class="ext-link" href="https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&amp;Expires=1345109432&amp;AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&amp;x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6"><span class="icon">​</span>https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&amp;Expires=1345109432&amp;AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&amp;x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6</a> To: ckan-sysadmin@…, kindly@… </p> <p> URL: <a class="ext-link" href="http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca"><span class="icon">​</span>http://ckan.net/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:155 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:131 in <span class="underline">call</span> &lt;&lt; r'\1', oldpath) </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:125 in <span class="underline">call</span> &lt;&lt; </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> if 'paste.testing_variables' in environ and hasattr(response, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.dispatch(controller, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: </p> <blockquote> <p> log.debug("Calling controller class with WSGI interface") </p> </blockquote> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> def load_test_env(self, environ): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:239 in <span class="underline">call</span> &lt;&lt; try: </p> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> <blockquote> <p> finally: </p> <blockquote> <p> model.Session.remove() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:221 in <span class="underline">call</span> &lt;&lt; return response(environ, self.start_response) </p> <blockquote> <p> response = self._dispatch_call() if not start_response_called: </p> <blockquote> <p> self.start_response = start_response </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._dispatch_call() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ<a class="missing wiki">pylons.action_method?</a> = func </p> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> <blockquote> <p> else: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.<span class="underline">name</span>, args) </p> <blockquote> <p> try: </p> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> <p> except HTTPException, httpe: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = 'before_and_this' return func(<strong>args) </strong></p> </blockquote> <p> </p> <blockquote> <p> def _inspect_call(self, func): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return func(<strong>args) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.storage:190 in file &lt;&lt; return fapp(request.environ, self.start_response) </p> <blockquote> <p> else: </p> <blockquote> <p> h.redirect_to(file_url) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> h.redirect_to(file_url) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.helpers:57 in redirect_to &lt;&lt; if are_there_flash_messages(): </p> <blockquote> <p> kw<a class="missing wiki">__no_cache__?</a> = True </p> </blockquote> <blockquote> <p> return _redirect_to(url_for(*args, <strong>kw)) </strong></p> </blockquote> <p> </p> <blockquote> <p> def url(*args, <strong>kw): </strong></p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return _redirect_to(url_for(*args, <strong>kw)) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.helpers:79 in url_for &lt;&lt; # fix ver to include the slash </p> <blockquote> <p> kw<a class="missing wiki">ver?</a> = '/%s' % ver </p> </blockquote> <blockquote> <p> my_url = _routes_default_url_for(*args, <strong>kw) kw<a class="missing wiki">__ckan_no_root?</a> = no_root return _add_i18n_to_url(my_url, locale=locale, </strong>kw) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> my_url = _routes_default_url_for(*args, <strong>kw) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.util:265 in url_for &lt;&lt; if not isinstance(url, str) and url is not None: </p> <blockquote> <p> raise <a class="missing wiki">GenerationException?</a>("url_for can only return a string, got " </p> <blockquote> <p> "unicode instead: %s" % url) </p> </blockquote> </blockquote> <blockquote> <p> if url is None: </p> <blockquote> <p> raise <a class="missing wiki">GenerationException?</a>( </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> "unicode instead: %s" % url) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> <a class="missing wiki">GenerationException?</a>: url_for can only return a string, got unicode instead: <a class="ext-link" href="https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&amp;Expires=1345109432&amp;AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&amp;x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6"><span class="icon">​</span>https://ckannet-storage.commondatastorage.googleapis.com/file/fd3e00d9-eb0d-479b-8303-38909e5400ca?Signature=cmuU3CzRHQ86F3aZ0gljxv%2B3IqQ%3D&amp;Expires=1345109432&amp;AWSAccessKeyId=GOOGC6OU3AYPNY47B66M&amp;x-goog-meta-uploaded-by=d5c8ed25-70de-4035-b29d-ddbe363913c6</a> CGI Variables CKAN_CURRENT_URL '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_CHARSET 'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@…)' PATH_INFO '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '37236' REQUEST_METHOD 'GET' REQUEST_URI '/storage/f/file/fd3e00d9-eb0d-479b-8303-38909e5400ca' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7f5312a51650&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7f5312a51790&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7f5312a516d0&gt;&gt; beaker.session {'_accessed_time': 1345109433.273328, '_creation_time': 1345109433.273328} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script <em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host </em> mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: &gt;, <em>) paste.registry &lt;paste.registry.Registry object at 0x7f5303b05610&gt; paste.throw_errors True pylons.action_method &lt;bound method <a class="missing wiki">StorageController?</a>.file of &lt;ckan.controllers.storage.<a class="missing wiki">StorageController?</a> object at 0x7f530891fa10&gt;&gt; pylons.controller &lt;ckan.controllers.storage.<a class="missing wiki">StorageController?</a> object at 0x7f530891fa10&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7f52f864f850&gt; pylons.routes_dict {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'} repoze.who.logger &lt;logging.Logger object at 0x7f53127b2b50&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 139994769488336&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 139994769488528&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7f5312d893d0&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 139994771858704&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350&gt;} routes.route &lt;routes.route.Route object at 0x7f531296e690&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f52f82cd450&gt; webob._parsed_query_vars (GET([]), </em>) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52f5870cd8&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f52f82cd450&gt;, {'action': u'file', 'controller': u'ckan.controllers.storage:StorageController', 'label': u'file/fd3e00d9-eb0d-479b-8303-38909e5400ca'}) </p> http://localhost/ticket/2867 http://localhost/ticket/2867 Report ross <p> From: &lt;thedatahub.org@…&gt; Date: Wed, Aug 15, 2012 at 8:34 AM Subject: <a class="missing wiki">WebApp?</a> Error: &lt;class 'ckan.lib.search.common.<a class="missing wiki">SearchError?</a>'&gt;: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:&#34;welcome&#34; capacity:&#34;public&#34; +site_id:&#34;www.ckan.net&#34; +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: "'start' parameter cannot be negative" To: ckan-sysadmin@…, kindly@… </p> <p> URL: <a class="ext-link" href="http://thedatahub.org/feeds/group/welcome.atom?page=0"><span class="icon">​</span>http://thedatahub.org/feeds/group/welcome.atom?page=0</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:155 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:131 in <span class="underline">call</span> &lt;&lt; r'\1', oldpath) </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:125 in <span class="underline">call</span> &lt;&lt; </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> if 'paste.testing_variables' in environ and hasattr(response, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.dispatch(controller, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: </p> <blockquote> <p> log.debug("Calling controller class with WSGI interface") </p> </blockquote> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> def load_test_env(self, environ): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:239 in <span class="underline">call</span> &lt;&lt; try: </p> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> <blockquote> <p> finally: </p> <blockquote> <p> model.Session.remove() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:221 in <span class="underline">call</span> &lt;&lt; return response(environ, self.start_response) </p> <blockquote> <p> response = self._dispatch_call() if not start_response_called: </p> <blockquote> <p> self.start_response = start_response </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._dispatch_call() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ<a class="missing wiki">pylons.action_method?</a> = func </p> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> <blockquote> <p> else: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.<span class="underline">name</span>, args) </p> <blockquote> <p> try: </p> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> <p> except HTTPException, httpe: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = 'before_and_this' return func(<strong>args) </strong></p> </blockquote> <p> </p> <blockquote> <p> def _inspect_call(self, func): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return func(<strong>args) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.feed:180 in group &lt;&lt; data_dict<a class="missing wiki">fq?</a> = 'groups:&#34;%s&#34;' % id </p> <blockquote> <p> item_count, results = _package_search(data_dict) </p> </blockquote> <p> </p> <blockquote> <p> navigation_urls = self._navigation_urls(params, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> item_count, results = _package_search(data_dict) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.feed:57 in _package_search &lt;&lt; # package_search action modifies the data_dict, so keep our copy intact. </p> <blockquote> <p> query = get_action('package_search')(context, data_dict.copy()) </p> </blockquote> <p> </p> <blockquote> <p> return query<a class="missing wiki">count?</a>, query<a class="missing wiki">results?</a> </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> query = get_action('package_search')(context, data_dict.copy()) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.logic.action.get:1130 in package_search &lt;&lt; query = search.query_for(model.Package) </p> <blockquote> <p> query.run(data_dict) </p> </blockquote> <p> </p> <blockquote> <p> for package in query.results: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> query.run(data_dict) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.search.query:350 in run &lt;&lt; except <a class="missing wiki">SolrException?</a>, e: </p> <blockquote> <p> raise <a class="missing wiki">SearchError?</a>('SOLR returned an error running query: %r Error: %r' % </p> <blockquote> <p> (query, e.reason)) </p> </blockquote> </blockquote> <blockquote> <p> try: </p> <blockquote> <p> data = json.loads(solr_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> (query, e.reason)) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> <a class="missing wiki">SearchError?</a>: SOLR returned an error running query: {'sort': 'metadata_modified desc', 'fq': u'groups:&#34;welcome&#34; capacity:&#34;public&#34; +site_id:&#34;www.ckan.net&#34; +state:active', 'facet.mincount': 1, 'rows': 21, 'facet.limit': '50', 'facet': 'true', 'q': '*:*', 'start': -20, 'wt': 'json', 'fl': 'id data_dict'} Error: "'start' parameter cannot be negative" CGI Variables CKAN_CURRENT_URL '/feeds/group/welcome.atom?page%3D0' CKAN_LANG 'ja' CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT '*/*' HTTP_ACCEPT_ENCODING 'gzip,deflate' HTTP_CONNECTION 'close' HTTP_FROM 'googlebot(at)googlebot.com' HTTP_HOST 'thedatahub.org' HTTP_USER_AGENT 'Mozilla/5.0 (compatible; Googlebot/2.1; +<a class="ext-link" href="http://www.google.com/bot.html"><span class="icon">​</span>http://www.google.com/bot.html</a>)' PATH_INFO '/feeds/group/welcome.atom' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/ja/feeds/group/welcome.atom' QUERY_STRING 'page=0' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '33139' REQUEST_METHOD 'GET' REQUEST_URI '/ja/feeds/group/welcome.atom?page=0' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'thedatahub.org' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.22 (Ubuntu) Server at thedatahub.org Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7f5312a51650&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7f5312a51790&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7f5312a516d0&gt;&gt; beaker.session {'_accessed_time': 1345016040.884951, '_creation_time': 1345016040.884951} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script <em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host </em> mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: &gt;, <em>) paste.parsed_dict_querystring (<a class="missing wiki">MultiDict?</a>([('page', '0')]), 'page=0') paste.parsed_querystring ([('page', '0')], 'page=0') paste.registry &lt;paste.registry.Registry object at 0x7f5302eef7d0&gt; paste.throw_errors True pylons.action_method &lt;bound method <a class="missing wiki">FeedController?</a>.group of &lt;ckan.controllers.feed.<a class="missing wiki">FeedController?</a> object at 0x7f52f0341750&gt;&gt; pylons.controller &lt;ckan.controllers.feed.<a class="missing wiki">FeedController?</a> object at 0x7f52f0341750&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7f52f0341450&gt; pylons.routes_dict {'action': u'group', 'controller': u'feed', 'id': u'welcome'} repoze.who.logger &lt;logging.Logger object at 0x7f53127b2b50&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 139994769488336&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 139994769488528&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7f5312d893d0&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 139994771858704&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350&gt;} routes.route &lt;routes.route.Route object at 0x7f5312961e90&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f52f0341210&gt; webob._parsed_query_vars (GET([('page', '0')]), 'page=0') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f52ed4873f0&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f52f0341210&gt;, {'action': u'group', 'controller': u'feed', 'id': u'welcome'}) </em></p> http://localhost/ticket/2868 http://localhost/ticket/2868 Report shevski <p> Reported by Daniel: </p> <p> Problem: DISQUS comments don't appear with profile details (inc. username). How to reproduce: 1) Go to a dataset, e.g. I went to <a class="ext-link" href="http://s031.okserver.org:2375/dataset/ff"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/ff</a> 2) I logged into DISQUS using the Google Authentication, e.g. in my case that is okfn.daniel.lewis@… 3) I made a comment and submitted 4) It comes up with the comment, but no linked profile until the page gets refreshed </p> http://localhost/ticket/2869 http://localhost/ticket/2869 Report ross <p> From: &lt;thedatahub.org@…&gt; Date: Thu, Aug 16, 2012 at 3:20 AM Subject: <a class="missing wiki">WebApp?</a> Error: &lt;class 'genshi.template.eval.<a class="missing wiki">UndefinedError?</a>'&gt;: "tag_list" not defined To: ckan-sysadmin@…, kindly@… </p> <p> URL: <a class="ext-link" href="http://ckan.net/dataset/uk-postboxes"><span class="icon">​</span>http://ckan.net/dataset/uk-postboxes</a> Module weberror.errormiddleware:162 in <span class="underline">call</span> &lt;&lt; <span class="underline">traceback_supplement</span> = Supplement, self, environ </p> <blockquote> <p> sr_checker = <a class="missing wiki">ResponseStartChecker?</a>(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) </p> </blockquote> <blockquote> <p> except: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> app_iter = self.application(environ, sr_checker) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:73 in <span class="underline">call</span> &lt;&lt; self.cache_manager) </p> <blockquote> <p> environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module beaker.middleware:155 in <span class="underline">call</span> &lt;&lt; headers.append(('Set-cookie', cookie)) </p> <blockquote> <p> return start_response(status, headers, exc_info) </p> </blockquote> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> <p> </p> <blockquote> <p> def _get_session(self): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return self.wrap_app(environ, session_start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module routes.middleware:131 in <span class="underline">call</span> &lt;&lt; r'\1', oldpath) </p> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> # Wrapped in try as in rare cases the attribute will be gone already </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.app(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:125 in <span class="underline">call</span> &lt;&lt; </p> <blockquote> <p> controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> if 'paste.testing_variables' in environ and hasattr(response, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self.dispatch(controller, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.wsgiapp:324 in dispatch &lt;&lt; if log_debug: </p> <blockquote> <p> log.debug("Calling controller class with WSGI interface") </p> </blockquote> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> <p> </p> <blockquote> <p> def load_test_env(self, environ): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return controller(environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:239 in <span class="underline">call</span> &lt;&lt; try: </p> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> <blockquote> <p> finally: </p> <blockquote> <p> model.Session.remove() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> res = WSGIController.<span class="underline">call</span>(self, environ, start_response) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:221 in <span class="underline">call</span> &lt;&lt; return response(environ, self.start_response) </p> <blockquote> <p> response = self._dispatch_call() if not start_response_called: </p> <blockquote> <p> self.start_response = start_response </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._dispatch_call() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:172 in _dispatch_call &lt;&lt; req.environ<a class="missing wiki">pylons.action_method?</a> = func </p> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> <blockquote> <p> else: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> response = self._inspect_call(func) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:107 in _inspect_call &lt;&lt; func.<span class="underline">name</span>, args) </p> <blockquote> <p> try: </p> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> <p> except HTTPException, httpe: </p> <blockquote> <p> if log_debug: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = self._perform_call(func, args) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.controllers.core:60 in _perform_call &lt;&lt; """Hide the traceback for everything above this method""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = 'before_and_this' return func(<strong>args) </strong></p> </blockquote> <p> </p> <blockquote> <p> def _inspect_call(self, func): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return func(<strong>args) </strong></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.controllers.package:322 in read &lt;&lt; template = template[:template.index('.') + 1] + format </p> <blockquote> <p> return render(template, loader_class=loader) </p> </blockquote> <p> </p> <blockquote> <p> def comments(self, id): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return render(template, loader_class=loader) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:153 in render &lt;&lt; try: </p> <blockquote> <p> return cached_template(template_name, render_template, </p> <blockquote> <p> loader_class=loader_class) </p> </blockquote> </blockquote> <blockquote> <p> except ckan.exceptions.<a class="missing wiki">CkanUrlException?</a>, e: </p> <blockquote> <p> raise </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> loader_class=loader_class) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module pylons.templating:249 in cached_template &lt;&lt; return content </p> <blockquote> <p> else: </p> <blockquote> <p> return render_func() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return render_func() </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckan.lib.base:102 in render_template &lt;&lt; if loader_class == <a class="missing wiki">NewTextTemplate?</a>: </p> <blockquote> <p> return literal(stream.render(method="text", encoding=None)) </p> </blockquote> <p> </p> <blockquote> <p> return literal(stream.render(method=method, encoding=None, </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return literal(stream.render(method="text", encoding =None)) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:183 in render &lt;&lt; method = self.serializer or 'xml' </p> <blockquote> <p> generator = self.serialize(method=method, <strong>kwargs) return encode(generator, method=method, encoding=encoding, out=out) </strong></p> </blockquote> <p> </p> <blockquote> <p> def select(self, path, namespaces=None, variables=None): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return encode(generator, method=method, encoding=encoding, out =out) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.output:57 in encode &lt;&lt; _encode = lambda string: string </p> <blockquote> <p> if out is None: </p> <blockquote> <p> return _encode(<em>.join(list(iterator))) </em></p> </blockquote> <p> for chunk in iterator: </p> <blockquote> <p> out.write(_encode(chunk)) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return _encode(<em>.join(list(iterator))) </em></p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.output:569 in <span class="underline">call</span> &lt;&lt; def <span class="underline">call</span>(self, stream): </p> <blockquote> <p> strip_markup = self.strip_markup for event in stream: </p> <blockquote> <p> if event<a class="missing changeset" title="No default repository defined">[0]</a> is TEXT: </p> <blockquote> <p> data = event<a class="missing changeset" title="No default repository defined">[1]</a> </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): </p> <blockquote> <p> for mark, event in stream: </p> <blockquote> <p> kind = event<a class="missing changeset" title="No default repository defined">[0]</a> if not (kind is None or kind is ATTR or kind is BREAK): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:1145 in <span class="underline">call</span> &lt;&lt; :param stream: The marked event stream to filter </p> <blockquote> <p> """ for mark, event in stream: </p> <blockquote> <p> yield mark, event if mark is ENTER: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:714 in <span class="underline">call</span> &lt;&lt; stream = iter(stream) </p> <blockquote> <p> next = stream.next for mark, event in stream: </p> <blockquote> <p> if mark is None: </p> <blockquote> <p> yield mark, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> yield OUTSIDE, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): </p> <blockquote> <p> for mark, event in stream: </p> <blockquote> <p> kind = event<a class="missing changeset" title="No default repository defined">[0]</a> if not (kind is None or kind is ATTR or kind is BREAK): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:1145 in <span class="underline">call</span> &lt;&lt; :param stream: The marked event stream to filter </p> <blockquote> <p> """ for mark, event in stream: </p> <blockquote> <p> yield mark, event if mark is ENTER: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:714 in <span class="underline">call</span> &lt;&lt; stream = iter(stream) </p> <blockquote> <p> next = stream.next for mark, event in stream: </p> <blockquote> <p> if mark is None: </p> <blockquote> <p> yield mark, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> yield OUTSIDE, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): </p> <blockquote> <p> for mark, event in stream: </p> <blockquote> <p> kind = event<a class="missing changeset" title="No default repository defined">[0]</a> if not (kind is None or kind is ATTR or kind is BREAK): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:1145 in <span class="underline">call</span> &lt;&lt; :param stream: The marked event stream to filter </p> <blockquote> <p> """ for mark, event in stream: </p> <blockquote> <p> yield mark, event if mark is ENTER: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:714 in <span class="underline">call</span> &lt;&lt; stream = iter(stream) </p> <blockquote> <p> next = stream.next for mark, event in stream: </p> <blockquote> <p> if mark is None: </p> <blockquote> <p> yield mark, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> yield OUTSIDE, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): </p> <blockquote> <p> for mark, event in stream: </p> <blockquote> <p> kind = event<a class="missing changeset" title="No default repository defined">[0]</a> if not (kind is None or kind is ATTR or kind is BREAK): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module ckanext.googleanalytics.plugin:93 in download_adder &lt;&lt; [downloaded %s times]&lt;/span&gt;<strong> </strong></p> <blockquote> <p> count = None for mark, (kind, data, pos) in stream: </p> <blockquote> <p> if mark and kind == START: </p> <blockquote> <p> href = data<a class="missing changeset" title="No default repository defined">[1]</a>.get('href') </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, (kind, data, pos) in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:714 in <span class="underline">call</span> &lt;&lt; stream = iter(stream) </p> <blockquote> <p> next = stream.next for mark, event in stream: </p> <blockquote> <p> if mark is None: </p> <blockquote> <p> yield mark, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> yield OUTSIDE, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): </p> <blockquote> <p> for mark, event in stream: </p> <blockquote> <p> kind = event<a class="missing changeset" title="No default repository defined">[0]</a> if not (kind is None or kind is ATTR or kind is BREAK): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:1175 in <span class="underline">call</span> &lt;&lt; """ </p> <blockquote> <p> callable_value = hasattr(self.value, '<span class="underline">call</span>') for mark, (kind, data, pos) in stream: </p> <blockquote> <p> if mark is ENTER: </p> <blockquote> <p> if callable_value: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, (kind, data, pos) in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:714 in <span class="underline">call</span> &lt;&lt; stream = iter(stream) </p> <blockquote> <p> next = stream.next for mark, event in stream: </p> <blockquote> <p> if mark is None: </p> <blockquote> <p> yield mark, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> yield OUTSIDE, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.core:288 in _ensure &lt;&lt; # unchanged </p> <blockquote> <p> yield event for event in stream: </p> <blockquote> <p> yield event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:686 in _unmark &lt;&lt; def _unmark(self, stream): </p> <blockquote> <p> for mark, event in stream: </p> <blockquote> <p> kind = event<a class="missing changeset" title="No default repository defined">[0]</a> if not (kind is None or kind is ATTR or kind is BREAK): </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:1145 in <span class="underline">call</span> &lt;&lt; :param stream: The marked event stream to filter </p> <blockquote> <p> """ for mark, event in stream: </p> <blockquote> <p> yield mark, event if mark is ENTER: </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:714 in <span class="underline">call</span> &lt;&lt; stream = iter(stream) </p> <blockquote> <p> next = stream.next for mark, event in stream: </p> <blockquote> <p> if mark is None: </p> <blockquote> <p> yield mark, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for mark, event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.filters.transform:682 in _mark &lt;&lt; def _mark(self, stream): </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> yield OUTSIDE, event </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.base:605 in _include &lt;&lt; from genshi.template.loader import <a class="missing wiki">TemplateNotFound?</a> </p> <blockquote> <p> for event in stream: </p> <blockquote> <p> if event<a class="missing changeset" title="No default repository defined">[0]</a> is INCLUDE: </p> <blockquote> <p> href, cls, fallback = event<a class="missing changeset" title="No default repository defined">[1]</a> </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> for event in stream: </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.base:565 in _flatten &lt;&lt; elif kind is EXPR: </p> <blockquote> <p> result = _eval_expr(data, ctxt, vars) if result is not None: </p> <blockquote> <p> # First check for a string, otherwise the iterable test </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> result = _eval_expr(data, ctxt, vars) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.base:277 in _eval_expr &lt;&lt; if vars: </p> <blockquote> <p> ctxt.push(vars) </p> </blockquote> <blockquote> <p> retval = expr.evaluate(ctxt) if vars: </p> <blockquote> <p> ctxt.pop() </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> retval = expr.evaluate(ctxt) </p> </blockquote> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.eval:178 in evaluate &lt;&lt; <span class="underline">traceback_hide</span> = 'before_and_this' </p> <blockquote> <p> _globals = self._globals(data) return eval(self.code, _globals, {'<span class="underline">data</span>': data}) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> return eval(self.code, _globals, {'<span class="underline">data</span>': data}) </p> </blockquote> </blockquote> </blockquote> </em></blockquote> <p> Module ?:51 in &lt;Expression u"tag_list(c.pkg_dict.get('tags', <em>))"&gt; &lt;&lt; &lt;li py:if=&#34;c.pkg_dict.get('tags')" class="sidebar-section"&gt; </p> <blockquote> <p> &lt;h3&gt;Tags&lt;/h3&gt; ${tag_list(c.pkg_dict.get('tags', <em>))} </em></p> </blockquote> <blockquote> <p> &lt;/li&gt; </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> ${tag_list(c.pkg_dict.get('tags', <em>))} </em></p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.eval:309 in lookup_name &lt;&lt; val = BUILTINS.get(name, val) </p> <blockquote> <p> if val is UNDEFINED: </p> <blockquote> <p> val = cls.undefined(name) </p> </blockquote> </blockquote> <blockquote> <p> return val </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> val = cls.undefined(name) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> Module genshi.template.eval:410 in undefined &lt;&lt; """Raise an <tt></tt><a class="missing wiki">UndefinedError?</a><tt></tt> immediately.""" </p> <blockquote> <p> <span class="underline">traceback_hide</span> = True raise <a class="missing wiki">UndefinedError?</a>(key, owner=owner) </p> <blockquote class="citation"> <blockquote class="citation"> <blockquote> <p> raise <a class="missing wiki">UndefinedError?</a>(key, owner=owner) </p> </blockquote> </blockquote> </blockquote> </blockquote> <p> <a class="missing wiki">UndefinedError?</a>: "tag_list" not defined CGI Variables CKAN_CURRENT_URL '/dataset/uk-postboxes' CKAN_LANG 'en' CKAN_LANG_IS_DEFAULT True CKAN_PAGE_CACHABLE True CONTENT_TYPE '; charset=utf-8' DOCUMENT_ROOT '/etc/apache2/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'text/html,text/plain,text/xml' HTTP_ACCEPT_CHARSET 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING 'gzip' HTTP_ACCEPT_LANGUAGE 'vi,en-us;q=0.7,en;q=0.3' HTTP_CONNECTION 'close' HTTP_HOST 'ckan.net' HTTP_USER_AGENT 'coccoc/1.0 ()' PATH_INFO '/dataset/uk-postboxes' PATH_TRANSLATED '/etc/ckan/datahub/apache.wsgi/dataset/uk-postboxes' REMOTE_ADDR '127.0.0.1' REMOTE_PORT '47065' REQUEST_METHOD 'GET' REQUEST_URI '/dataset/uk-postboxes' SCRIPT_FILENAME '/etc/ckan/datahub/apache.wsgi' SERVER_ADDR '127.0.0.1' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.0' SERVER_SIGNATURE '&lt;address&gt;Apache/2.2.22 (Ubuntu) Server at ckan.net Port 80&lt;/address&gt;\n' SERVER_SOFTWARE 'Apache/2.2.22 (Ubuntu)' WSGI Variables application &lt;beaker.middleware.<a class="missing wiki">CacheMiddleware?</a> object at 0x7f5312a51650&gt; beaker.cache &lt;beaker.cache.<a class="missing wiki">CacheManager?</a> object at 0x7f5312a51790&gt; beaker.get_session &lt;bound method <a class="missing wiki">SessionMiddleware?</a>._get_session of &lt;beaker.middleware.<a class="missing wiki">SessionMiddleware?</a> object at 0x7f5312a516d0&gt;&gt; beaker.session {'_accessed_time': 1345083636.079552, '_creation_time': 1345083636.079552} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.handler_script <em> mod_wsgi.input_chunked '0' mod_wsgi.listener_host </em> mod_wsgi.listener_port '8080' mod_wsgi.process_group 'datahub' mod_wsgi.request_handler 'wsgi-script' mod_wsgi.script_reloading '1' mod_wsgi.version (3, 3) paste.cookies (&lt;SimpleCookie: &gt;, <em>) paste.registry &lt;paste.registry.Registry object at 0x7f52e47477d0&gt; paste.throw_errors True pylons.action_method &lt;bound method <a class="missing wiki">PackageController?</a>.read of &lt;ckan.controllers.package.<a class="missing wiki">PackageController?</a> object at 0x7f52e477b810&gt;&gt; pylons.controller &lt;ckan.controllers.package.<a class="missing wiki">PackageController?</a> object at 0x7f52e477b810&gt; pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons &lt;pylons.util.<a class="missing wiki">PylonsContext?</a> object at 0x7f52e477be90&gt; pylons.routes_dict {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'} repoze.who.logger &lt;logging.Logger object at 0x7f53127b2b50&gt; repoze.who.plugins {'openid': &lt;<a class="missing wiki">OpenIdIdentificationPlugin?</a> 139994769488336&gt;, 'friendlyform': &lt;<a class="missing wiki">FriendlyFormPlugin?</a> 139994769488528&gt;, 'ckan.lib.authenticator:UsernamePasswordAuthenticator': &lt;ckan.lib.authenticator.<a class="missing wiki">UsernamePasswordAuthenticator?</a> object at 0x7f5312d893d0&gt;, 'auth_tkt': &lt;<a class="missing wiki">AuthTktCookiePlugin?</a> 139994771858704&gt;, 'ckan.lib.authenticator:OpenIDAuthenticator': &lt;ckan.lib.authenticator.OpenIDAuthenticator object at 0x7f5312d89350&gt;} routes.route &lt;routes.route.Route object at 0x7f5312955a50&gt; routes.url &lt;routes.util.URLGenerator object at 0x7f52eb8c9090&gt; webob._parsed_query_vars (GET([]), </em>) webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper &lt;built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f5308983a80&gt; wsgi.version (1, 1) wsgiorg.routing_args (&lt;routes.util.URLGenerator object at 0x7f52eb8c9090&gt;, {'action': u'read', 'controller': u'package', 'id': u'uk-postboxes'}) </p> http://localhost/ticket/2870 http://localhost/ticket/2870 Report shevski <p> tested with my <a class="ext-link" href="http://s031.okserver.org:2375/user/shevski"><span class="icon">​</span>http://s031.okserver.org:2375/user/shevski</a> and daniel's account. I have added about text and yet see ' Irina Bolychevsky </p> <p> You have not provided a biography.' </p> <p> The about text is saved (and viewable on the edit form itself) but does not show on profile </p> http://localhost/ticket/2871 http://localhost/ticket/2871 Report seanh <p> Set a dataset's license to 'Other (Not Open)', the dataset will have an Open Data icon next to it </p> http://localhost/ticket/2872 http://localhost/ticket/2872 Report danieljohnlewis <p> For example see: <a class="ext-link" href="http://s031.okserver.org:2375/qa/dataset/five_stars"><span class="icon">​</span>http://s031.okserver.org:2375/qa/dataset/five_stars</a> </p> <p> Which has: Example dataset as a score of: 4 After examination of the dataset ( <a class="ext-link" href="http://s031.okserver.org:2375/dataset/example-dataset"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/example-dataset</a> ) it is made up of a CSV and an HTML file. </p> http://localhost/ticket/2873 http://localhost/ticket/2873 Report rgrp <p> Full of obsolete material </p> http://localhost/ticket/2874 http://localhost/ticket/2874 Report toby <p> Fanstatic does not support </p> <p> @import ... </p> <p> relative imports for images etc </p> <p> need to get a fix for these asap </p> http://localhost/ticket/2875 http://localhost/ticket/2875 Report toby <p> need to make sure these update everywhere when done </p> http://localhost/ticket/2876 http://localhost/ticket/2876 Report rgrp <p> In progress </p> <ol><li><del>[major] q does not seem to work reliably. e.g. using the setup from this gist <a class="ext-link" href="https://gist.github.com/1930806"><span class="icon">​</span>https://gist.github.com/1930806</a> and doing a ?q=DE yields no results (does not work with "q=de" either)</del> <ul><li><del>q=second does work ...</del> </li></ul></li><li>[major] q does not work with 2 values (see below) </li><li>Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT <ul><li>Also weird fact that limit is returned but as as as string - should it not be an integer </li></ul></li><li>Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...) </li></ol><h3 id="Multiplequeryvalues">Multiple query values</h3> <p> Try a query such as: "second UK" and you will get 500 error: </p> <p> <a class="ext-link" href="http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&amp;q=second%20UK"><span class="icon">​</span>http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&amp;q=second%20UK</a> </p> <pre class="wiki">ProgrammingError: (ProgrammingError) syntax error in tsquery: "second UK" 'select "_id", "id", "date", "x", "y", "z", "country", "title", "lat", "lon", count(*) over() as "_full_count"\n from "4f1299ab-a100-4e5f-ba81-e6d234a2f3bd" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',) </pre><h3 id="Suggestions">Suggestions</h3> <p> Filter support: should think in more detail about this (may want to follow recline style) </p> <p> Simple filters in query parameters would be nice too ... </p> http://localhost/ticket/2877 http://localhost/ticket/2877 Report ross <p> As part of merging Organisations into core, it is necessary that we clarify the capacity field with which the users/datasets are added as members to the group 'subclass'. </p> <p> Rather than the capacity being an opaque string that implies auth but doesn't clearly specify it, we will use role names where roles are defined in the database - with a clearly defined set of standard roles. The Role table is expected to have simply a string name/representation and acts as a container for permissions. </p> <p> Each permission is a string of the form object.action (such as package.add, group.delete) of which several are expected to be associated with a role. This means the permission table will contain a string and a reference to the role. </p> <p> This work will require UI changes to the screens allowing users to be added to a group/organisation so that the list of available roles is available to add those users. </p> <p> [x] Model for Role and Permission </p> <p> [ ] Logic layer changes for managing roles/permissions etc. </p> <p> [ ] Determine default roles, perhaps just admin/editor/viewer </p> <p> [ ] Fix the auth layer to use the permissions/roles - may be better implemented as another ticket. </p> http://localhost/ticket/2878 http://localhost/ticket/2878 Report ross <p> The Package model should have a boolean field added to it describing its visibility beyond that supplied via auth. </p> <p> The related permission for viewing packages where public is set to False would be package.view (show, whatever), and *never* visible for unauthenticated users. </p> http://localhost/ticket/2879 http://localhost/ticket/2879 Report ross <p> Each dataset should have an explicit owner that is an organisation. This is the organisation that owns the dataset* and will be used for specifying who can move a dataset out of the default organisation. </p> <p> This should probably be a foreign key link from dataset to organisation (or rather group). </p> <ul><li>Currently this won't necessarily be the organisation that the dataset is a member of, but it is likely that this will be the case initially (at least until someone moves it to another organisation). </li></ul> http://localhost/ticket/2880 http://localhost/ticket/2880 Report seanh <p> <a class="ext-link" href="http://docs.ckan.org/en/ckan-1.8/domain-model.html"><span class="icon">​</span>http://docs.ckan.org/en/ckan-1.8/domain-model.html</a> </p> <p> Currently not very useful. Also these pages: </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/latest/domain-model-dataset.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/domain-model-dataset.html</a> </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/latest/domain-model-resource.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/domain-model-resource.html</a> </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/latest/domain-model-task-status.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/domain-model-task-status.html</a> </p> <p> This just seems very incomplete to me (surely there is more to the domain model the dataset, resource and task status?) and very likely to not get updated as changes are mode to the code. Would anyone seriously refer to this rather than just looking at the source code? </p> <p> If we want to have something along these lines in the docs, I suggest adding docstrings to packages, modules, classes and methods and then using autodoc to pull them into sphinx. </p> http://localhost/ticket/2881 http://localhost/ticket/2881 Report seanh <p> Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation? </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/latest/user-stories.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/user-stories.html</a> </p> http://localhost/ticket/2882 http://localhost/ticket/2882 Report seanh <p> Here's a draft: <a class="ext-link" href="https://gist.github.com/3414107"><span class="icon">​</span>https://gist.github.com/3414107</a> </p> http://localhost/ticket/2883 http://localhost/ticket/2883 Report seanh <p> Set the license of a dataset to blank (the first option) or to 'License Not Specified' or 'Other (Non-Commercial)' and a lock icon appears next to the license on the dataset read page. Shouldn't this be a question mark or something? </p> http://localhost/ticket/2884 http://localhost/ticket/2884 Report seanh http://localhost/ticket/2885 http://localhost/ticket/2885 Report ross <p> Related items (Apps &amp; Ideas) have a collection of types, which are currently fixed. Some of these types are ambiguous, such as ideas in that they may not have a link - and if they do it is likely to be to a blog post - another existing type. </p> <p> Whilst the URL is required, and we think this should stay required, we should also allow users to change the types found in Apps &amp; Ideas to a shortened list that suits their requirements. </p> http://localhost/ticket/2886 http://localhost/ticket/2886 Report seanh <p> ckan/templates/home/index.html contains: </p> <blockquote> <p> &lt;h1 class="page_heading"&gt;Welcome to ${g.site_title}!&lt;/h1&gt; </p> </blockquote> <p> which means that just "Welcome to" ends up in the ckan.po files for translation. This makes the phrase untranslatable in languages that have a different word ordering than English where the site title needs to be somewhere other than at the end of the sentence. </p> http://localhost/ticket/2887 http://localhost/ticket/2887 Report kindly <p> Try and put datapreview in Iframe. </p> http://localhost/ticket/2888 http://localhost/ticket/2888 Report seanh <p> For example, edit the warandpeace dataset and add an extra with key "comments" and value "[{u'date': u'Tue Aug 21 13:01:48 GMT 2012', u'comment': u'testing ckan', u'userId': u'11702'}, {u'date': u'Wed Aug 22 13:01:48 GMT 2012', u'comment': u'testing ckan again', u'userId': u'11702'}]", now visit API URLs such as: </p> <p> /api/search/dataset?sort=metadata_modified+desc&amp;all_fields=1&amp;limit=10 </p> <p> /api/rest/dataset/warandpeace </p> <p> /api/action/package_show?id=warandpeace </p> <p> The value of the extra comes out as a string in the dataset JSON, rather than having its contents unpacked into a list of JSON objects. </p> http://localhost/ticket/2889 http://localhost/ticket/2889 Report seanh <p> Currently there is this page: </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/latest/data-viewer.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/data-viewer.html</a> </p> <p> which covers Recline Data Explorer and other kinds of data preview in CKAN. It is under the Publishing Datasets section in the documentation. I had to to a search for 'recline' to find it. </p> <p> Separately there is this page: <a class="ext-link" href="http://docs.ckan.org/en/ckan-1.7.1/datastore.html"><span class="icon">​</span>http://docs.ckan.org/en/ckan-1.7.1/datastore.html</a> which covers datastore, datastorer, and the data api. </p> <p> I suggest collecting this together in one chapter called 'Data Previews'. If I understand it right the general gist would be: </p> <p> CKAN has builtin previews of data resources on resource pages, enabled by default. </p> <p> Images, Google Documents, and web page resources will be loaded into embedded iframes for preview. </p> <p> Text-like files will be displayed raw. </p> <p> CSV or Excel files uploaded to CKAN will be previewed using Recline Data Explorer. </p> <p> Additionally, you can enable CKAN's <a class="missing wiki">DataStore?</a>, requires you to install <a class="missing wiki">ElasticSearch?</a> and nginx and put datastore.enabled=1 in your ini file. Lets you use the Data API to query data. </p> <p> Does having <a class="missing wiki">DataStore?</a> enabled mean you get preview of more types of resources? Any resource that's available via the Data API will be previewed using Recline, </p> <p> You can install ckanext-datastorer, and then CSV and Excel files _linked to_ as CKAN resources will be previewed using Recline also. Requires celeryd. </p> http://localhost/ticket/2890 http://localhost/ticket/2890 Report seanh <p> Currently this page: </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/ckan-1.8/datastore.html"><span class="icon">​</span>http://docs.ckan.org/en/ckan-1.8/datastore.html</a> </p> <p> has an embedded Google docs presentation about data store which contains some outdated information, e.g. links to datastoreclient which was merged into ckanclient. </p> <p> Suggest moving the google presentation into the documentation proper, so it works with grep etc. </p> http://localhost/ticket/2891 http://localhost/ticket/2891 Report seanh <p> It looks like when an IDatasetForm plugin with a form_to_db_schema() method is active, then the posted data dict is validated against the schema even if the user is posting to an API endpoint like /api/action/package_update etc. This breaks the API, for example 'name' and 'title' are optional keys for package_update (you can supply only the 'id'), but the schema validation will fail if either are not present in the posted dict. </p> <p> It may be we need different schema for package create and package update, currently IDatasetForm has just the one form_to_db_schema() method. </p> http://localhost/ticket/2892 http://localhost/ticket/2892 Report seanh <p> With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this: </p> <p> [ckan.controllers.api] Validation error: "{'<span class="underline">type': 'Validation Error', '</span>junk': [u'The input field <span class="underline">junk was not expected.']}" </span></p> <p> They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active). </p> <p> In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema. </p> http://localhost/ticket/2893 http://localhost/ticket/2893 Report seanh <p> If there's an IDatasetForm plugin active with a form_to_db_schema(), then packages with e.g. no extras get the 'extras' key entirely stripped by the validation in package_show, which causes some tests to crash, e.g. in ckan/tests/functional/api/test_activity.py (if the tests are run with a suitable IDatasetForm plugin active). </p> <p> If a package has no extras the dict returned by package_show should still have an 'extras' key with an empty list as the value. </p> <p> Suspect this also effects packages with no tags, no groups, etc. </p> http://localhost/ticket/2894 http://localhost/ticket/2894 Report rgrp <p> And as a result timeout on e.g. the datahub ... </p> http://localhost/ticket/2895 http://localhost/ticket/2895 Report seanh <p> ckan/logic/action/create.py:resource_create(): </p> <blockquote> <p> try: </p> <blockquote> <p> pkg_dict = _get_action('package_update')(context, pkg_dict) </p> </blockquote> <p> except <a class="missing wiki">ValidationError?</a>, e: </p> <blockquote> <p> errors = e.error_dict<a class="missing wiki">resources?</a>[-1] raise <a class="missing wiki">ValidationError?</a>(errors) </p> </blockquote> </blockquote> <p> This assumes that package_update always returns an error dict with 'resources' in it. This assumption doesn't hold. </p> http://localhost/ticket/2896 http://localhost/ticket/2896 Report seanh <p> If there's an IDatasetForm plugin active with a db_to_form_schema() method based on db_to_form_package_schema(), then viewing historical versions of datasets (e.g. <a class="ext-link" href="http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191"><span class="icon">​</span>http://127.0.0.1:5000/dataset/dasd%402012-08-30T12%3A26%3A18.524191</a>) crashes. </p> <p> Reason is that revison_id and revision_timestamp need to be added to db_to_form_package_schema(), something like: </p> <blockquote> <p> schema.update({'revision_id': [validators.ignore_missing, unicode]}) schema.update({'revision_timestamp': </p> <blockquote> <p> [validators.ignore_missing, unicode]}) </p> </blockquote> </blockquote> http://localhost/ticket/2897 http://localhost/ticket/2897 Report johnmartin <p> To reproduce: </p> <ol><li>Enter details for new dataset without being logged in </li></ol><ol start="2"><li>Hit login page for just before step 2 </li></ol><ol start="3"><li>Once logged in you end up back at the empty add dataset form (instead of expected 2nd step of dataset edit) </li></ol> http://localhost/ticket/2898 http://localhost/ticket/2898 Report johnmartin <p> Goto: </p> <ol><li>Add dataset </li></ol><ol start="2"><li>Get to step 2 </li></ol><ol start="3"><li>Click "Save &amp; add another" without adding any data </li></ol><ol start="4"><li>Ends on step 3 </li></ol><p> Should: fail to validate and remain on step 2 </p> http://localhost/ticket/2899 http://localhost/ticket/2899 Report seanh <p> Sometimes a dataset dict comes through package_show() with 'notes': None and then the 'notes' key gets stripped from the dict when validation happens (validation only happens if there's an IDatasetForm plugin with a db_to_form_schema() method) and then package_list.html template crashes, e.g. on user profile pages. </p> <p> Fix in the template? Or in the schema? </p> <p> The template also crashes on packages with no tags (empty list gets stripped from dict) and probably other empty fields as well </p> http://localhost/ticket/2900 http://localhost/ticket/2900 Report johnmartin <p> The language dropdown in Internet Explorer 7 behaves oddly. Essentially I think the problem is with the way that the dropdown decides to go into "drop-up mode" (e.g there aint enough screen space below the initial state so it has to drop up). </p> http://localhost/ticket/2901 http://localhost/ticket/2901 Report ross <p> The Group controller uses the genshi module where it may be better for it to use Markup() in the template. </p> <p> See <a class="ext-link" href="https://github.com/okfn/ckan/pull/117/files#r1524863"><span class="icon">​</span>https://github.com/okfn/ckan/pull/117/files#r1524863</a> </p> http://localhost/ticket/2902 http://localhost/ticket/2902 Report johnmartin <p> I think the architecture on the search page is a bit odd. Not sure I like that there is a doubling up of the search box (e.g. one in the header and one in the body of the search results page) </p> <p> Needs some thought </p> http://localhost/ticket/2903 http://localhost/ticket/2903 Report johnmartin <p> On the search results page when there are no more formats, groups or tags to display it should not display the show more links. </p> http://localhost/ticket/2904 http://localhost/ticket/2904 Report johnmartin <p> On step 1 for adding a dataset can we add a ajax ping that checks if the URL that is generated by the JS is taken? Behaviour should be as follows: </p> <ol><li>User types title in box (as current) </li></ol><ol start="2"><li>URL key is regexed by JS (as current) </li></ol><ol start="3"><li>The JS does a ajax call to check if the generated URL key is valid. The ajax callback returns a JSON dump which contains what the URL key should be (e.g. if 'test' was taken it returns 'test-1') </li></ol><ol start="4"><li>If the URL key returned in the ajax is different than the one generated by the JS then: </li></ol><p> 4a. If the user hasn't clicked the edit button then: it automatically updates the URL key field with the URL key supplied in the ajax </p> <p> 4b. If the user has clicked the edit button then: a form error next to the input appears saying that the URL key is taken and within that error is suggests the URL key returned by the ajax as alternative (the user can click said URL key within the error to automatically choose that one) </p> <p> The URL key isn't reserved by the ajax and upon form submission the usual validation should still apply. </p> http://localhost/ticket/2905 http://localhost/ticket/2905 Report dominik <p> The flags in the dropdown on details pages are not right. </p> <p> You can see the error on <a class="ext-link" href="http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0"><span class="icon">​</span>http://s031.okserver.org:2375/dataset/malawi-aid-projects/resource/b717c20e-2006-4ad4-82d2-59b57ebc1ab0</a>. </p> http://localhost/ticket/2906 http://localhost/ticket/2906 Report johnmartin <p> The user login / avatar with in the header really should be within another topbar, as it goes against convention to have the user information within the normal header. </p> http://localhost/ticket/2907 http://localhost/ticket/2907 Report johnmartin <p> At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio. </p> http://localhost/ticket/2908 http://localhost/ticket/2908 Report johnmartin <p> They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain. </p> http://localhost/ticket/2909 http://localhost/ticket/2909 Report johnmartin <p> They are currently are just a small image / name. They should be: Larger image, Full name (username), description, edits/dataset </p> http://localhost/ticket/2910 http://localhost/ticket/2910 Report kindly <p> A summary/user story doc of how organizations and groups are expected to work. </p> http://localhost/ticket/2911 http://localhost/ticket/2911 Report seanh <p> When adding a dataset with the organizations extension on, neither the Public or the Private radio button is initially selected. Public should be selected. </p> http://localhost/ticket/2912 http://localhost/ticket/2912 Report seanh <p> Should be group_form() I think </p> http://localhost/ticket/2913 http://localhost/ticket/2913 Report seanh <p> This optional method is supported when creating or updating groups and packages and is used in <a class="missing wiki">DefaultGroupForm?</a> and <a class="missing wiki">DefaultPackageForm?</a>, but seems to be missing from the interfaces. Should be added with docstring. </p> http://localhost/ticket/2914 http://localhost/ticket/2914 Report seanh <p> Having two methods form_to_db_schema(self) and form_to_db_schema_options(self, options) seems unnecessary, why not just have form_to_db_schema(self, options=None)? </p> <p> Fixing this might break existing extensions although I don't think any are using form_to_db_schema_options() so it should be okay. </p> <p> The same thing goes for db_to_form_schema_options(). </p> <p> Also why are we passing a dict 'options' that always contains the same three keys 'api', 'context' and 'type'? Why not three params api=None, contenxt=None, type=None? </p> http://localhost/ticket/2915 http://localhost/ticket/2915 Report ross <p> The user management in organizations ( <a class="ext-link" href="http://localhost:5000/organization/users/NAME"><span class="icon">​</span>http://localhost:5000/organization/users/NAME</a> ) should autocomplete names in the dropdown box so that only valid usernames are added to the table. </p> http://localhost/ticket/2916 http://localhost/ticket/2916 Report ross <p> Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end. </p> http://localhost/ticket/2917 http://localhost/ticket/2917 Report ross <p> When you remove someone, without adding them, the text box at the bottom (which should probably autocomplete) is empty, and this causes problems on the server. </p> <p> Ideally when you add a user (select from the autocomplete) it would add another row to the table, defaulting the user to editor and setting the names to user<span class="underline">{{X}}</span>name and user<span class="underline">{{X}}</span>capacity where X is $('tr').size() </p> http://localhost/ticket/2918 http://localhost/ticket/2918 Report seanh <p> Still in docs but seems gone from code </p> http://localhost/ticket/2919 http://localhost/ticket/2919 Report seanh <p> In setup.py, do we still need: </p> <blockquote> <p> ('templates/<strong>.txt', 'genshi', { </strong></p> <blockquote> <p> 'template_class': 'genshi.template:TextTemplate' </p> </blockquote> <p> }), </p> </blockquote> <p> in message_extractors? Doesn't like there are any txt files in there anymore. </p> http://localhost/ticket/2920 http://localhost/ticket/2920 Report seanh <p> I think it couldn't hurt for this module to have a docstring at the top of the file explaining what the module is for. I know from setup.py that it's there to provide the extract_ckan() string extractor function for Babel, but I think looking at the file on its own it's not so obvious. </p> <p> Also a couple of other small fixes: </p> <p> jinja2_cleaner looks like a helper function only meant to be used by extract_ckan(), might make things clearer if it was called _jinja2_cleaner(), just so it doesn't look like a public function the module wants to export. </p> <p> jinja_extensions seems to be a module-level variable that is only used in one function, could be moved into the function, unless you think there might be more functions that want it in future. </p> http://localhost/ticket/2921 http://localhost/ticket/2921 Report seanh <p> I'm unsure about what's going on here. As I understand it, when we run <tt>python setup.py extract_messages</tt> it's going to use the extract_ckan() function from ckan/lib/extract.py to process the HTML files. For the HTML files that are jinja2 templates, extract_ckan() will call jinja2_cleaner() which will call regularise_html() on the strings. So the strings are regularised when they are extracted from the source files. </p> <p> But then we have the parse() method of <a class="missing wiki">CkanInternationalizationExtension?</a> also calling regularise_html(). I don't get what's happening here. Why do the strings need to be regularised twice? </p> <p> My guess is that <a class="missing wiki">CkanInternationalizationExtension?</a> is used when the strings are extracted from the templates at runtime, and they need to be regularised at this time in order to match them against the regularised strings in the mo files to find the translations to output? </p> <p> Maybe <a class="missing wiki">CkanInternationalizationExtension?</a> needs a better docstring saying what it does? </p> http://localhost/ticket/2922 http://localhost/ticket/2922 Report seanh <p> The function is called regularise_html(), can't remember what file it's in. </p> http://localhost/ticket/2923 http://localhost/ticket/2923 Report seanh <p> Add a better docstring to for trans js command explaining what it does and why, and how to use it. </p> <p> Also add this command to the CKAN Release Process as it's needed there </p> http://localhost/ticket/2924 http://localhost/ticket/2924 Report seanh <ul><li>Is <tt>trans mangle</tt> really necessary? If you upload a pot file to Transifex, it can generate a po file for you with 100% strings translated into a fictional pseudo language where everything is really long strings of unicode characters. I found this worked well for coverage testing, and also tests handling of unicode and long strings all over the place. </li></ul> http://localhost/ticket/2925 http://localhost/ticket/2925 Report seanh <p> Finish the tests for the per-package-type custom template support in IPackageForm, IGroupForm and IOrganizationForm (i.e. the new_template(), read_template() etc. methods). </p> <p> The new way that template inheritance works in jinja looks like it should allow this to be tested nicely without duplicating entire templates in the example extensions. </p> http://localhost/ticket/2926 http://localhost/ticket/2926 Report seanh <p> IPackageForm, IGroupForm and IOrganizationForm allow you to define e.g. new package types with new top-level URLs, but the example extensions don't have tests for this yet. </p> http://localhost/ticket/2927 http://localhost/ticket/2927 Report seanh <p> Before releasing CKAN 2.0 we need to run all the CKAN tests with a modified test-core.ini with the example_idatasetform, example_igroupform and example_iorganizationform plugins enabled. If any tests fail, fix the bugs. This needs to be done for each release so add it to the release process. </p> http://localhost/ticket/2928 http://localhost/ticket/2928 Report seanh <p> If a plugin wants to take over the default package type for example, it can return <a class="missing wiki">dataset?</a> from its package_types() method. I don't see how there can be a package type that is not the default 'package' and is not returned by the package_types() method of a plugin, because the URLs for such a package type would not be mapped. Am I missing something or is is_fallback() not needed? </p> <p> Same for IGroupForm and IOrganizationForm </p> http://localhost/ticket/2929 http://localhost/ticket/2929 Report seanh <p> Use an IGroupForm plugin to add a custom metadata field to groups using convert_to_extras() and convert_from_extras(), when calling group show the value comes back quoted, e.g. '"my_value"' </p> <p> Should add tests to example_igroupform and others that setting and getting the custom fields works through the action API. </p> http://localhost/ticket/2930 http://localhost/ticket/2930 Report seanh <p> <strong> The application's Globals object </strong> is not very informative. </p> http://localhost/ticket/2931 http://localhost/ticket/2931 Report seanh <p> What is the system_info table for? It looks like a way to override config file settings in the database, perhaps so that admins can use a web interface to change site title etc. Could do with a docstring explaining the intended purpose of the table. </p> http://localhost/ticket/2932 http://localhost/ticket/2932 Report seanh <p> There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs. </p> <p> I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents. </p> http://localhost/ticket/2933 http://localhost/ticket/2933 Report Dominik <p> The tests in CKAN only test single pages (functional tests) or single methods (unit tests). In order to make sure that the whole system still works as expected, we need tests that cover whole processes. </p> <p> An example user story to be tested would be: "As a user with create permissions I want to be able to log in, create a new resource, save it and then make sure that it is in the system." or: "As a user I want to use the search functionality and open a resource from the search page." </p> <p> These user stories require a different testing system with interaction and processes. </p> http://localhost/ticket/2934 http://localhost/ticket/2934 Report kindly http://localhost/ticket/2935 http://localhost/ticket/2935 Report seanh <p> If the group dict that you post to the API does not have any 'packages' key then all the group's packages get removed. I think it would be better if you could just update e.g. the group's description without having to also post the list of packages, and apparently this is how other update API actions work. </p> <p> Might be worth checking all the update API actions for this behaviour, and making sure they're all consistent </p> http://localhost/ticket/2936 http://localhost/ticket/2936 Report seanh <p> <a class="missing wiki">GroupController?</a>'s history() method doesn't pass 'extras_as_string': True in the context when it calls group_show. This means that if you have an IGroupForm plugin that is adding a custom metadata field and using convert_to/from_extras() then a field value of 'foo' will be returned as '"foo"' </p> <p> Other <a class="missing wiki">GroupController?</a> and <a class="missing wiki">PackageController?</a> methods do pass 'extras_as_string' </p> http://localhost/ticket/2937 http://localhost/ticket/2937 Report dominik <p> Since it makes sense to bundle resources into one js file that are commonly used together, there should be some explanations on how separate js scripts are merged into one file for production. </p> <p> Explanations could include how groups are handled or different resource files. </p> http://localhost/ticket/2938 http://localhost/ticket/2938 Report toby <p> holder for ticket number </p> http://localhost/ticket/2939 http://localhost/ticket/2939 Report johnmartin <p> We need a nice way to be able to get to editing the different resources that are associated to a dataset. </p> http://localhost/ticket/2940 http://localhost/ticket/2940 Report johnmartin <p> Add the views and functionality of following users and groups into 2.0 </p> http://localhost/ticket/2941 http://localhost/ticket/2941 Report dread <p> When POSTing to the API, if specified, the 'Content-Type' header must be blank or 'application/x-www-form-urlencoded'. Otherwise we get an error like: "Bad request - JSON Error: Could not extract request body data: Bad content type: \'; charset=utf-8\'"" </p> <p> The problem is that this is a very reasonable header to send. Indeed requests 0.14 sends this particular header. </p> <p> This affects all versions of CKAN. </p> <p> This is due to webob/requests.py:1248 being pretty basic. </p> http://localhost/ticket/2942 http://localhost/ticket/2942 Report dominik <p> Chrome does not resize iframe after a full refresh/ on first load </p> http://localhost/ticket/2943 http://localhost/ticket/2943 Report dominik <p> Opera cannot show a recline preview. I shows 0 records. </p> http://localhost/ticket/2944 http://localhost/ticket/2944 Report dominik <p> The pdf preview does not load in IE 9. </p> http://localhost/ticket/2945 http://localhost/ticket/2945 Report dominik <p> The pdf preview does not load in IE 9. </p> http://localhost/ticket/2946 http://localhost/ticket/2946 Report dominik <p> Use story: As a user of the datasore I want to be able to put the resource id without the dataset name in a url and see the resource page. </p> <p> Proposed redirect: </p> <p> /resource/{res-id} -&gt; dataset/foo-bar/resource/{res-id} </p> http://localhost/ticket/2947 http://localhost/ticket/2947 Report dominik <p> Recline allows negative ranges. The backend returns an error 500 for that which breaks the recline preview. </p> http://localhost/ticket/2948 http://localhost/ticket/2948 Report amercader <p> The checks to show or not the button need to be updated for the latest datastore version </p> http://localhost/ticket/2949 http://localhost/ticket/2949 Report icmurray <p> With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is: </p> <ul><li>the fanstatic_resources defined in <tt>ckan/lib/fanstatic_resources.py</tt> (bottom of module) should be prepared, and minified. </li></ul><p> This ticket is to: </p> <ul><li>provide a paster command which when run will generate the minified javascript as css files. </li><li>stop the auto-minification of files when CKAN starts up. </li><li>remove reference to the minified files in the <tt>.gitignore</tt> file. </li><li>add instruction to the release process to run this command, and check-in the minified files to the repo. </li></ul><p> ## Background </p> <p> Currently, minification works seamlessly without the need for any preparation when CKAN is started in a development setup. But on a production site, the webserver will (almost certainly) not have write-access to the directories that will contain the minified files. And so the minification will fail, and the site will end up serving the un-minified media, or even *old* minified media. </p> <p> One way around this would be to allow webserver write access to the directory its serving out of. But this is not generally considered good practice. </p> <p> Another method would be to distribute the minified files with CKAN. This ticket describes how to do this without causing a lot of noise in the commit history of the repo. </p> <p> The auto-minifcation occurs when importing <tt>ckan/lib/fanstatic_resources.py</tt>. </p> <p> ## Changing the process slightly </p> <p> The reason for moving away from auto-minifying files at start-up, to minifying files when running a paster command is: </p> <ul><li>we should be distributing the minified files when we make a release (as we do with translation files). This makes it easier to install CKAN from source. </li></ul><ul><li>if we distribute the minified files, then they need to be checked-in to the repo. </li></ul><ul><li>if they are auto-minified, whenever small changes are made, then this will create a lot of noise in the repo. It will be clearer to have the minified-files generated manually as part of the release process. (Or whever we make a deployment internally). </li></ul> http://localhost/ticket/2950 http://localhost/ticket/2950 Report icmurray <p> With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is: </p> <ul><li>the less files should be compiled to css (main.css, not just main.debug.css) </li></ul><p> This ticket is to provide a paster command which will compile the .less into the main.css file. The idea of the paster command is that it will be run by developers, and they will check in the resulting .css files. It will also provide a convenient mechanism for production scenarios with node.js installed on the production server, as they will be able to compile the .less there as well. </p> <p> ## Background </p> <p> The .css files that need to be served are built using <tt>less</tt>. The css files that are generated *are* checked-in to the repo; but they are a build artifact. So the general workflow is: </p> <ul><li>commit changes to the less files </li><li>build the main.css file and commit </li></ul><p> This paster command slots into the above workflow for convenience. </p> <p> ## Notes </p> <ul><li>doc/frontend-development.rst </li><li>bin/less </li></ul><p> to run this paster command, <tt>Node</tt> will be required. So that dependency should be checked. </p> <p> ## The paster command </p> <p> This is the proposed behaviour of the paster command: </p> <ul><li>ensure <tt>custom.less</tt> does not exist </li><li>for each colour in{fuchsia,green,maroon,red}: <ul><li>generate a <tt>custom.less</tt> file for $colour. There's a paster command that does this already: `paster color &lt;color name&gt;. </li><li>generate the <tt>css</tt> from the <tt>less</tt> files. ie - the equivelant of running <tt>bin/less --production</tt>. <ul><li>this will generate a <tt>main.css</tt> file, which should be renamed to <tt>$color.css</tt>. </li></ul></li></ul></li></ul><ul><li>ensure <tt>custom.less</tt> does not exist </li><li>generate the <tt>css</tt> from the <tt>less</tt> files. This will create a <tt>main.css</tt> with the default colour scheme. </li></ul><ul><li>call the paster command that minifies css and js files. (This command does not exist yet, see <a class="closed ticket" href="http://localhost/ticket/2950" title="requirement: paster command to minify javascript and css (closed: fixed)">#2950</a>) </li></ul> http://localhost/ticket/2951 http://localhost/ticket/2951 Report icmurray <p> There are now javascript translations. There's a paster command, <tt>trans js</tt>. This is a ticket to ensure that using <tt>trans js</tt> is documented in the normal ckan translation workflow (which is followed after a feature freeze, prior to release). And to see if there's any need for further incorporation into the current translation process. </p> http://localhost/ticket/2952 http://localhost/ticket/2952 Report dominik <p> If you go to: /{any template dir} for example /home or /related, a Server errror occurs. </p> <p> IOError: [Errno 21] Is a directory: u'/../venv/src/ckan/ckan/templates/home' </p> http://localhost/ticket/2953 http://localhost/ticket/2953 Report johnmartin <p> From the google doc: When I move between grid, graph and map view mode (or click on filters) the whole data explorer flickers (jumps up &amp; down slightly) for a few moments before settling. This is happening on all resources for me and when you initially open launch the page: <a class="ext-link" href="http://demo.ckan.org/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360"><span class="icon">​</span>http://demo.ckan.org/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360</a>. Only in Chrome, working fine in Firefox. </p> http://localhost/ticket/2954 http://localhost/ticket/2954 Report dominik http://localhost/ticket/2955 http://localhost/ticket/2955 Report johnmartin <p> On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually. </p> http://localhost/ticket/2956 http://localhost/ticket/2956 Report dominik <p> The <tt></tt>GRANT SELECT ON ALL<tt></tt> and <tt></tt>ALTER DEFAULT PRIVILEGES<tt></tt> are not available in Postgres prior to version 9.0. </p> http://localhost/ticket/2957 http://localhost/ticket/2957 Report seanh <p> e.g. 2012_08(주요국가).xls </p> <p> I tested in CKAN 2.0 and it seemed to work fine there. </p> http://localhost/ticket/2958 http://localhost/ticket/2958 Report icmurray <p> Repro: </p> <ul><li>Create a new Group, named "test-group". </li><li>Add a dataset to it. </li><li>Verify the dataset belongs to the group by visiting the Group's read-page </li><li>Update the Group through the action api (group_update), using the uid in the "id" field, and a new name in the "name" field. </li><li>Visit the group's read-page. The list of datasets will be empty. </li></ul><p> This was an issue when editing a Group through the web interface, which was fixed in <a class="missing changeset" title="No default repository defined">[1]</a>. However it only fixes the issue in the group controller. </p> <p> <a class="missing changeset" title="No default repository defined">[1]</a> <a class="ext-link" href="https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335"><span class="icon">​</span>https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335</a> </p> http://localhost/ticket/2959 http://localhost/ticket/2959 Report dominik <p> At the moment everyone can read all resources. There is no support for hiding resources that are in private datasets. </p> http://localhost/ticket/2960 http://localhost/ticket/2960 Report dominik <p> Users could write ckan extensions that offer previews for a certain datatype. </p> <p> Advantages: </p> <ul><li>We can say that you can add your own previews </li><li>Every preview would have to define it's own dependencies </li><li>External development (community creates new extensions?) </li></ul><p> Questions to answer: </p> <ul><li>Specification of the interface </li><li>Precedence of previews for the same data types </li><li>Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible. </li></ul><p> As far as I can see, this should be fairly easy to implement by following these instructions: <a class="ext-link" href="http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface"><span class="icon">​</span>http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface</a> </p> http://localhost/ticket/2961 http://localhost/ticket/2961 Report dominik <p> Could be done by: </p> <ul><li>using the solr distributed search <ul><li>difficult set up </li></ul></li><li>merging result sets from apis <ul><li>make sure that results can be merged properly (score, facets, ...) </li></ul></li></ul> http://localhost/ticket/2962 http://localhost/ticket/2962 Report amercader <p> e.g. <a class="ext-link" href="http://thedatahub.org/tag/lod"><span class="icon">​</span>http://thedatahub.org/tag/lod</a> </p> <p> Tags with less datasets work fine (e.g. <a class="ext-link" href="http://thedatahub.org/tag/railways"><span class="icon">​</span>http://thedatahub.org/tag/railways</a>) </p> http://localhost/ticket/2963 http://localhost/ticket/2963 Report seanh <p> If you are the only admin of an organization you can edit the organization's members and demote yourself, then the org has no admins and no one (except sysadmins maybe) can edit it. </p> <p> Last admin should not be able to remove or demote herself. </p> <p> Also applies to groups. </p> http://localhost/ticket/2964 http://localhost/ticket/2964 Report amercader <ul><li>Graphs not showing (looks like a flot related file is missing) </li><li>Wrong groups counts (e.g. Data Explorer Examples show 1800 datasets when it onlu has 8) </li></ul> http://localhost/ticket/2965 http://localhost/ticket/2965 Report seanh <p> e.g. if I just changed a member's capacity I am not adding anything 'save' is better maybe </p> http://localhost/ticket/2966 http://localhost/ticket/2966 Report seanh <p> Edit an organizations members page and demote yourself from organization admin, after saving the members edit page reloads even though you are no longer an admin and should no longer be able to access this page. </p> http://localhost/ticket/2967 http://localhost/ticket/2967 Report seanh <p> The button will not show if you are not authorized but browse to /organization/members/foo and you can edit the members, it does stop you when you try to save your changes, but you shouldn't be able to get to the page at all </p> http://localhost/ticket/2968 http://localhost/ticket/2968 Report seanh <p> The group members page (e.g. /group/members/roger) seems broken, IDs instead of user names are shown for the members, and clicking on a member 500s </p> http://localhost/ticket/2969 http://localhost/ticket/2969 Report seanh <p> e.g. it the 'Members' button links to /organization/members/0a44... instead of /organization/members/foobar </p> http://localhost/ticket/2970 http://localhost/ticket/2970 Report seanh <p> when deleting members from groups and orgs </p> http://localhost/ticket/2971 http://localhost/ticket/2971 Report seanh <p> it is no longer used </p> http://localhost/ticket/2972 http://localhost/ticket/2972 Report seanh <p> Makes sense to keep all auth stuff under logic/auth. </p> <p> Also decide which functions from new_authz.pu should be marked private with _. </p> <p> Also decide which functions from new_authz.py are really helpers for the auth functions in get.py, update.py and delete.py, and should maybe moved to another file, e.g. logic/auth/helpers.py </p> http://localhost/ticket/2973 http://localhost/ticket/2973 Report seanh <p> From Toby: </p> <p> A general cleanup of all auth functions (in logic/auth) to check their fitness, error messages, etc. </p> http://localhost/ticket/2974 http://localhost/ticket/2974 Report seanh <p> From Toby: tests around the auth functions and new actions - probably needs an improved testing mechanism and things like test data creation on via actions </p> http://localhost/ticket/2975 http://localhost/ticket/2975 Report seanh <p> From Toby: A little polish to the member pages and probably a little hardening of the controller and logic actions </p> http://localhost/ticket/2976 http://localhost/ticket/2976 Report seanh <p> From Toby: @johnmartin the user autocomplete on member add needs fixing it gets data but does not understand what to do with it </p> http://localhost/ticket/2977 http://localhost/ticket/2977 Report seanh <p> From Toby: checking the logic around the who can do what from the user stories - via some tests may be a good approach </p> http://localhost/ticket/2978 http://localhost/ticket/2978 Report seanh <p> From Toby: The requesting membership user stories and implementation remain needed </p> http://localhost/ticket/2979 http://localhost/ticket/2979 Report seanh <p> From Toby: add the functionality to legacy templates </p> http://localhost/ticket/2980 http://localhost/ticket/2980 Report seanh <p> From Toby: remove the config from jinja2 templates get all such stuff via g. (before 2.0) </p> http://localhost/ticket/2981 http://localhost/ticket/2981 Report seanh <p> From Toby: strip more functionality from controllers and add via template helpers eg activity streams </p> http://localhost/ticket/2982 http://localhost/ticket/2982 Report seanh <p> From Toby: c) refactor lib/base.py to remove the circular import issues (render functions to lib/render.py - would fix much of that) </p> http://localhost/ticket/2983 http://localhost/ticket/2983 Report seanh <p> From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action() </p> http://localhost/ticket/2984 http://localhost/ticket/2984 Report seanh <p> From Toby: make lib/helpers more template specific and less generally shared with none template uses (some functions are shared but many shouldn't be) - again help avoid circular imports </p> http://localhost/ticket/2985 http://localhost/ticket/2985 Report seanh <p> From Toby: make lib/base.py be more like it was originally intended eg defines stuff that is shared eg _ but maybe try to include stuff like render/get_action if the circular import stuff can be solved - which should be doable if hard and likely needs an extra file or two for the externals _ etc </p> http://localhost/ticket/2986 http://localhost/ticket/2986 Report seanh <p> From Toby </p> http://localhost/ticket/2987 http://localhost/ticket/2987 Report seanh <p> Afaik the new CKAN 2.0 frontend has no functional tests </p> http://localhost/ticket/2988 http://localhost/ticket/2988 Report seanh <p> 'Add dataset to organization' button, when you get through to the third stage of the new dataset form the organization you came from is not selected. </p> <p> I don't see any option to choose the group when adding or updating a dataset, but if I add a dataset via the "Add dataset to group" button on a group's page, then the dataset seems to get added to that group. (And I can also add/remove existing datasets by editing the group.) I wonder if organizations should work the same way, instead of having an Organization drop-down when creating or updating a dataset. The add dataset page needs to somehow indicate that you're adding a dataset to a certain group or organization though, doesn't currently. </p> http://localhost/ticket/2989 http://localhost/ticket/2989 Report seanh http://localhost/ticket/2990 http://localhost/ticket/2990 Report seanh <p> Currently if you want to customise the header.html template you have to copy the whole thing, it isn't broken up into blocks like other templates are. </p> http://localhost/ticket/2991 http://localhost/ticket/2991 Report johnmartin http://localhost/ticket/2992 http://localhost/ticket/2992 Report seanh http://localhost/ticket/2993 http://localhost/ticket/2993 Report seanh <p> If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button. </p> <p> These popovers should appear wherever datasets or users are listed, throughout the site. </p> http://localhost/ticket/2994 http://localhost/ticket/2994 Report seanh <p> If you hover the mouse over a dataset or user in an activity stream, you now get a popover with some info about that dataset or user and a follow/unfollow button. </p> <p> It would be nice to add similar popovers for other types of object in CKAN, e.g. resources, groups, organizations, tags... </p> http://localhost/ticket/2995 http://localhost/ticket/2995 Report seanh <p> The user popovers that appear when you hover the mouse over a user in an activity stream include a View Profile button that takes you to that user's profile page. If you are already on that user's profile page then it simply reloads the same page, but it sends you to the Datasets tab if you were on another tab. </p> <p> Possible solutions: </p> <p> Don't show button if already on user's page? </p> <p> Button just makes popover disappear, if already on user's page? </p> <p> Button does reload the page, but reloads the same tab (datasets, followers, activity stream) that you were on. </p> <p> Marking this low priority </p> http://localhost/ticket/2996 http://localhost/ticket/2996 Report seanh <p> Add an activity stream tab to dataset pages, like we have on user profile pages. Dataset activity streams are already implemented in the backend. </p> http://localhost/ticket/2997 http://localhost/ticket/2997 Report seanh <p> Before we do this we should implement smarter group activity streams (i.e. what activities should appear in a group's activity stream?) and following of groups. </p> http://localhost/ticket/2998 http://localhost/ticket/2998 Report seanh <p> Before we do this we need to merge the organizations branch into master, we need to implement smarter activity streams for organizations, and we need to implement following of organizations. </p> http://localhost/ticket/2999 http://localhost/ticket/2999 Report seanh <p> User profile pages now show some interesting stats about the user: num. datasets, num. edits and num. followers. Would be nice to add some similar info to dataset pages. At least num. followers can be done (already implemented in backend). </p> http://localhost/ticket/3000 http://localhost/ticket/3000 Report seanh <p> Enable the multilingual plugins: </p> <p> ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag </p> <p> and set your default language to one not supported by the multilingual plugin, e.g. </p> <p> ckan.locale_default = cs_CZ </p> <p> now run CKAN and try to add a dataset: </p> <p> File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index </p> <blockquote> <p> text_field_items<a class="missing wiki">+ default_lang?</a>.extend(all_terms) </p> </blockquote> <p> <a class="missing wiki">KeyError?</a>: 'text_cs_CZ' </p> <p> It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not. </p> <p> A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes. </p> <p> I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0 </p> http://localhost/ticket/3001 http://localhost/ticket/3001 Report amercader <p> Just to track @tauberer patch on Github. Would be nice to write a test for it. Probably going to 1.8.1 </p> http://localhost/ticket/3002 http://localhost/ticket/3002 Report amercader <p> Just to track @tauberer patch on Github. Probably going to 1.8.1 </p> http://localhost/ticket/3003 http://localhost/ticket/3003 Report seanh <p> This is happening whenever people try to run paster commands. polib should only be needed for the check-po-files command don't import otherwise. </p> http://localhost/ticket/3004 http://localhost/ticket/3004 Report seanh http://localhost/ticket/3005 http://localhost/ticket/3005 Report seanh <p> There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too. </p> http://localhost/ticket/3006 http://localhost/ticket/3006 Report seanh <p> Adding a new dataset creates multiple activity stream activities, e.g. seanh created the dataset foo, seanh add the resource bar to the dataset foo, and seanh updated the dataset foo are all created when I add a new dataset. I wonder if these can be collapsed into a single activity. </p> http://localhost/ticket/3007 http://localhost/ticket/3007 Report seanh <p> We need to test that only logged in users can see their own dashboard activity streams, and they can only see their own dashboard activity streams not those of other users. </p> http://localhost/ticket/3008 http://localhost/ticket/3008 Report seanh <p> We want a "bubble" with a number in it to appear in the top-right of the site next to your user name when you're logged in, telling you how many new activity stream items you have in your dashboard activity stream. </p> <p> There's a github issue for this here: </p> <p> <a class="ext-link" href="https://github.com/okfn/ckanext-pdeu/issues/11"><span class="icon">​</span>https://github.com/okfn/ckanext-pdeu/issues/11</a> </p> http://localhost/ticket/3009 http://localhost/ticket/3009 Report seanh <p> When minified files are in use (set debug = false in ini file) the pin images for points don't show up in the data explorer map view when viewing a resource with geolocation data. </p> <p> set debug = true will fix the problem by using unminified files instead but also turns on a bunch of debug stuff you wouldn't want on a production site. </p> <p> middleware.py, around line 73, is where the decision to turn minified files on or off based on the debug setting is made. This can be hacked on a production site to run with unminified files but otherwise run on production mode thereby getting around the issue with the pins. But that doesn't fix the underlying bug. </p> http://localhost/ticket/3010 http://localhost/ticket/3010 Report johnglover <p> Fixes and updates to Recline / Datapreview for CKAN 2.0. </p> http://localhost/ticket/3011 http://localhost/ticket/3011 Report toby http://localhost/ticket/3012 http://localhost/ticket/3012 Report dominik <p> Since the update of the doc theme, the page became unreadable. </p> <p> <a class="ext-link" href="http://docs.ckan.org/en/latest/common-error-messages.html"><span class="icon">​</span>http://docs.ckan.org/en/latest/common-error-messages.html</a> </p> http://localhost/ticket/3013 http://localhost/ticket/3013 Report seanh <p> From Knud Möller: </p> <p> when I try to delete a non-empty tag vocabulary via the API (through HTTP), I get an internal server error. Checking the logs, this turns out to be a consistency error raised by sqlalchemy: </p> <p> Error - &lt;class 'sqlalchemy.exc.<a class="missing wiki">IntegrityError?</a>'&gt;: (<a class="missing wiki">IntegrityError?</a>) update or delete on table "vocabulary" violates foreign key constraint "tag_vocabulary_id_fkey" on table "tag" DETAIL: Key (id)=(21421955-7560-467c-af30-9f790b73e6ae) is still referenced from table "tag". </p> <blockquote> <p> 'DELETE FROM vocabulary WHERE vocabulary.id = %(id)s' {'id': u'21421955-7560-467c-af30-9f790b73e6ae'} </p> </blockquote> <p> URL: <a class="ext-link" href="http://33.33.33.10:5000/api/action/vocabulary_delete"><span class="icon">​</span>http://33.33.33.10:5000/api/action/vocabulary_delete</a> </p> <p> The error makes sense, but I'm wondering if it would be useful to extend the API to also allow the deletion of non-empty vocabularies, possibly via a parameter (not sure what best practice in API design is). At the very least, it would be cool if the error message coming back in the response had more information in it. </p> http://localhost/ticket/3014 http://localhost/ticket/3014 Report johnmartin <p> Essentially the default bootstrap dropdowns don't work properly without <a class="missing wiki">JavaScript?</a> which is a little bit of an issue with JS not working. </p> http://localhost/ticket/3015 http://localhost/ticket/3015 Report johnmartin <p> Just a ticket to keep track of a few suggested template changes. </p> http://localhost/ticket/3016 http://localhost/ticket/3016 Report johnglover http://localhost/ticket/3017 http://localhost/ticket/3017 Report johnmartin <p> Activity streams should be able to load more than 15 items within them. Suggest the default amount of loading is around 30 and then click to load more. </p> http://localhost/ticket/3018 http://localhost/ticket/3018 Report seanh <p> Deleting extras in the web interface is broken </p> http://localhost/ticket/3019 http://localhost/ticket/3019 Report seanh http://localhost/ticket/3020 http://localhost/ticket/3020 Report johnmartin <p> Essentially, the functionality should be as follows: </p> <ul><li>Add logout link that has <tt>.js-hide</tt> attached to it within the header that isn't hidden within a dropdown </li></ul><p> See <a class="ext-link" href="http://plus.google.com/"><span class="icon">​</span>http://plus.google.com/</a> (when logged in) with and without JS to see an example of the actual sign-out working without JS </p> http://localhost/ticket/3021 http://localhost/ticket/3021 Report amercader <p> On the package controller the package_type is not passed to the lookup function, so the setup_template_variables defined on the extensions is never called </p> http://localhost/ticket/3022 http://localhost/ticket/3022 Report amercader <p> Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it </p> http://localhost/ticket/3023 http://localhost/ticket/3023 Report seanh http://localhost/ticket/3024 http://localhost/ticket/3024 Report amercader <p> Because yes please </p> http://localhost/ticket/3025 http://localhost/ticket/3025 Report amercader <p> nav_named_link won't include the nice icon even when passing the icon keyword param. </p> http://localhost/ticket/3026 http://localhost/ticket/3026 Report kindly <p> change mm support solr 3 and 4 add *_date field </p> http://localhost/ticket/3027 http://localhost/ticket/3027 Report seanh <p> <a class="ext-link" href="https://github.com/okfn/ckanext-pdeu/issues/13"><span class="icon">​</span>https://github.com/okfn/ckanext-pdeu/issues/13</a> </p> http://localhost/ticket/3028 http://localhost/ticket/3028 Report dread <p> <tt>http://datahub.io/api/2/search/package?jsonp=jsonpcallback&amp;q=canada</tt> returns </p> <pre class="wiki">{"count": 0, "results": []} </pre><p> I believe this worked in CKAN 1.4 or 1.5, but it is broken on 1.7.1, 1.8 and whatever demo.ckan.org is running. I suspect the jsonpcallback parameter is getting sent to SOLR. </p> <p> This bug prevents using javascript on another site to search CKAN (although hopefully the action API would work). </p> http://localhost/ticket/3029 http://localhost/ticket/3029 Report toby http://localhost/ticket/3030 http://localhost/ticket/3030 Report