From: ${c.revision_from.id} - ${c.revision_from.timestamp}
>> From: ${c.revision_from.id} - ${c.revision_from.timestamp} Module genshi.template.eval:324 in lookup_attr << val = obj[key] except (KeyError, TypeError): val = cls.undefined(key, owner=obj) return val >> val = cls.undefined(key, owner=obj) Module genshi.template.eval:410 in undefined << """"""Raise an ``UndefinedError`` immediately."""""" __traceback_hide__ = True raise UndefinedError(key, owner=owner) >> raise UndefinedError(key, owner=owner) UndefinedError: None has no member named ""id"" CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-silverlight, */*' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'en-us' HTTP_CONNECTION 'TE, close' HTTP_HOST 'ckan.net' HTTP_TE 'deflate,gzip;q=0.3' HTTP_UA_CPU 'x86' HTTP_USER_AGENT 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; MS-RTC LM 8)' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1),' REMOTE_ADDR '89.145.108.201' REMOTE_PORT '39586' REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1),' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application tag
## This is an tag
###### This is an tag
Text styles
*This text will be italic*
_This will also be italic_
**This text will be bold**
__This will also be bold__
*You **can** combine them*
Lists
Unordered
* Item 1
* Item 2
* Item 2a
* Item 2b
Ordered
1. Item 1
2. Item 2
3. Item 3
* Item 3a
* Item 3b
Miscellaneous
Images
Format: 
Links
http://github.com - automatic!
[GitHub](http://github.com)
Blockquotes
As Kanye West said:
> We're living the future so
> the present is our past.
}}}"
738 1294408864000000 thejimmyg This needs more discussion. It is a separate problem from deleting a harvest source from the admin interface which is dealt with in #884.
738 1294409432000000 thejimmyg "Note from John Bywater on duplicate ticket #783 says: ""We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer."""
738 1297685069000000 thejimmyg "It may be handy to have a history of harvested documents and also be able to delete documents but still get their revisions.
The package comparison code will need to look at these revisions."
738 1299761436000000 thejimmyg This is now complete and on UAT.
739 1294247758000000 thejimmyg This needs to be implemented via an API call that Drupal can use
739 1296592842000000 thejimmyg Implemented today. May want a better implementation later.
740 1294408931000000 thejimmyg This will have to be done via the API though for DGU/UKLP. Should probably have it on the package interface in main CKAN too though.
740 1296592889000000 thejimmyg Duplicate of #739.
741 1291988087000000 rgrp Done in cset:49d5bd0a6a99 and cset:68522feabfeb among others. Documentation of progress on http://ckan.okfnpad.org/plugins
742 1288038147000000 johnbywater Moved from sprint 1.3.0
743 1294412807000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
744 1294412813000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
745 1294412820000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
746 1294412827000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
747 1294412834000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
748 1294412916000000 thejimmyg Merging with #676.
748 1294412976000000 thejimmyg Sorry, that should have been #691.
749 1294412986000000 thejimmyg Merge with #691
750 1289483616000000 johnbywater Referencing ticket #728 has changed sprint.
750 1294408970000000 thejimmyg How would this be done? Is it part of the CSW spec?
750 1296592940000000 thejimmyg Duplicate of #728
752 1294408472000000 thejimmyg Duplicate of #712. See #711 now.
753 1289209646000000 johnbywater Moved from sprint 1.3.2
753 1289815657000000 johnbywater Moved from sprint 1.3.3
753 1294247922000000 thejimmyg Not an explicit requirement yet so closing ticket. Will re-open if needed.
754 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
754 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
754 1294408726000000 thejimmyg "We may not need to do any filtering in practice but if we did, the implementation would have these stages:
* Add filter attribute to harvest source entity
* Add filter field to harvest source form
* Construct and send filtered CSW GetRecords request
Thus these tickets can now be marked as duplicated: #755 #756 #759."
754 1294409145000000 thejimmyg "Also:
* Change API documentation to indicate harvest source entity has filter attribute
* Change ""CSW Get Records"" request to accept and use given CSW filter
Now #758 and #760 can be closed too."
755 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
755 1288264918000000 johnbywater Referencing ticket #754 has changed sprint.
755 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
755 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
755 1294408632000000 thejimmyg Duplicate of #754
756 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
756 1288264917000000 johnbywater Referencing ticket #754 has changed sprint.
756 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
756 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
756 1294408642000000 thejimmyg Duplicate of #754
757 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
757 1288264918000000 johnbywater Referencing ticket #754 has changed sprint.
757 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
757 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
757 1294233016000000 thejimmyg Don't fully understand this ticket. Will return to it.
757 1296593448000000 thejimmyg Duplicate of #754. If we added filtering we'd need to write a migration script anyway.
758 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
758 1288264917000000 johnbywater Referencing ticket #754 has changed sprint.
758 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
758 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
758 1294409053000000 thejimmyg Also a duplicate of #754.
759 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
759 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
759 1294408652000000 thejimmyg Duplicate of #754
760 1288041238000000 johnbywater Referencing ticket #759 has changed sprint.
760 1288265044000000 johnbywater Referencing ticket #759 has changed sprint.
760 1289209646000000 johnbywater Referencing ticket #759 has changed sprint.
760 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
760 1294409111000000 thejimmyg Duplicate of #754
763 1288606918000000 dread Referencing ticket #441 has changed sprint.
763 1292775248000000 thejimmyg "Notes from my discussion with David a while ago:
Following irc discussion, it looks like read-only mode is simply
achieved by Apache config:
* 503 for PUT/POST operations - stops writes
* 503 for GETs to URIs containing: /edit, /create, /new, /authz -
stops providing forms that lead to a write
* setenv CKAN_READONLY=""Undergoing maintenance 12.00 UTC for one hour""
which can be picked up by CKAN to be displayed as we see fit in the
future. Friedrich's current IATI sprint may link into this and grey
out edit links etc."
764 1289209648000000 johnbywater Moved from sprint 1.3.2
764 1289815657000000 johnbywater Moved from sprint 1.3.3
764 1310133334000000 shevski will be done as part of #1094
765 1288606918000000 dread Referencing ticket #441 has changed sprint.
766 1289815658000000 johnbywater Moved from sprint 1.3.3
766 1294248066000000 thejimmyg I haven't heard this mentioned yet, but yes, let's try to implement it if possible. Appears to expose a CSW interface? http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#ext_csw_clnts.htm
766 1296593519000000 thejimmyg This is a duplicate of #767
767 1289209649000000 johnbywater Moved from sprint 1.3.2
767 1289815658000000 johnbywater Referencing ticket #766 has changed sprint.
767 1294409336000000 thejimmyg "Our CSW code should be able to cope with ArcGIS too though surely? Worth testing though.
To do this we'd need to:
* Obtain example ArcGIS CSW server
* Write a test
Now we can close #768 and #769."
767 1296593504000000 thejimmyg "I haven't heard this mentioned yet, but yes, let's try to implement it if possible. Appears to expose a CSW interface? http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#ext_csw_clnts.htm
"
767 1304936413000000 thejimmyg We are now live and have used a variety of servers in testing.
768 1288190630000000 johnbywater Referencing ticket #767 has changed sprint.
768 1289209649000000 johnbywater Referencing ticket #767 has changed sprint.
768 1289815658000000 johnbywater Referencing ticket #766 has changed sprint.
768 1294409308000000 thejimmyg Duplicate of #767
769 1288190630000000 johnbywater Referencing ticket #767 has changed sprint.
769 1289209649000000 johnbywater Referencing ticket #767 has changed sprint.
769 1289815658000000 johnbywater Referencing ticket #766 has changed sprint.
769 1294409318000000 thejimmyg Duplicate of #767
770 1289209650000000 johnbywater Moved from sprint 1.3.2
770 1289815658000000 johnbywater Moved from sprint 1.3.3
770 1294413390000000 thejimmyg "Running CLI harvester command without arguments or with --help would return help for the harvester (currently crashes).
We can therefore close #772, #773 and #774"
770 1296593925000000 thejimmyg I don't care about this. It is too trivial. If I get around to fixing it they great but I don't need it logged as a ticket.
771 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
771 1288190682000000 johnbywater Referencing ticket #770 has changed sprint.
771 1288190711000000 johnbywater Referencing ticket #770 has changed sprint.
771 1289209650000000 johnbywater Referencing ticket #770 has changed sprint.
771 1289815658000000 johnbywater Referencing ticket #770 has changed sprint.
771 1294413256000000 thejimmyg "This works for me.
{{{
paster help
}}}
and
{{{
paster --help
}}}
both show help text for me.
"
772 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
772 1288190682000000 johnbywater Referencing ticket #770 has changed sprint.
772 1288190711000000 johnbywater Referencing ticket #770 has changed sprint.
772 1289209650000000 johnbywater Referencing ticket #770 has changed sprint.
772 1289815658000000 johnbywater Referencing ticket #770 has changed sprint.
772 1294413313000000 thejimmyg Merging with #770
773 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
773 1288190682000000 johnbywater Referencing ticket #770 has changed sprint.
773 1288190711000000 johnbywater Referencing ticket #770 has changed sprint.
773 1289209650000000 johnbywater Referencing ticket #770 has changed sprint.
773 1289815658000000 johnbywater Referencing ticket #770 has changed sprint.
773 1294413400000000 thejimmyg Merging with #770
774 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
774 1288190682000000 johnbywater Referencing ticket #772 has changed sprint.
774 1288190711000000 johnbywater Referencing ticket #772 has changed sprint.
774 1289209650000000 johnbywater Referencing ticket #772 has changed sprint.
774 1289815658000000 johnbywater Referencing ticket #772 has changed sprint.
774 1294413412000000 thejimmyg Merging with #770
775 1295260144000000 pudo fixed in cset:98c9997e80e8
776 1297066840000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/3/avoid-generating-vdm-warnings
777 1289209649000000 johnbywater Referencing ticket #764 has changed sprint.
777 1289815658000000 johnbywater Referencing ticket #764 has changed sprint.
777 1292586843000000 dread James did this a while ago.
778 1289209652000000 johnbywater Moved from sprint 1.3.2
778 1289815658000000 johnbywater Moved from sprint 1.3.3
778 1294416567000000 thejimmyg This is now done.
779 1289209652000000 johnbywater Moved from sprint 1.3.2
779 1289815658000000 johnbywater Moved from sprint 1.3.3
779 1294416581000000 thejimmyg This is now done.
780 1289209653000000 johnbywater Moved from sprint 1.3.2
780 1289815658000000 johnbywater Moved from sprint 1.3.3
780 1294416608000000 thejimmyg This is now done.
781 1289209653000000 johnbywater Moved from sprint 1.3.2
781 1289815658000000 johnbywater Moved from sprint 1.3.3
781 1294415081000000 thejimmyg This is a duplicate of #812 so closing.
782 1289218497000000 pudo "This has existed for quite a while, but misses documentation. Has been documented in cset:db18a16a5b4d
"
783 1294409399000000 thejimmyg Exact duplicate of #738.
784 1289815658000000 johnbywater Moved from sprint 1.3.3
784 1294410089000000 thejimmyg "In order to have end to end testing for UKLII we need to:
* Review document from PP with the different things we need to test
* Set up a GeoNetwork instance with sample documents to be tested
We may also want to test ArcGIS, see #767"
784 1304936251000000 thejimmyg The UKLP test manager is looking at a new test proposal, we have had approval to go live without this, we can re-open later if necessary.
786 1289815658000000 johnbywater Moved from sprint 1.3.3
786 1294410004000000 thejimmyg Duplicate of #884.
787 1289295207000000 pudo "Alternative:
* The Drupal system will expose a resource at data.gov.uk/profile which wil contain the reqesting user's nickname, fullname, email etc. in JSON (or XML) form.
* The Drupal system will also run an OAuth server (and provide /request_token, /access_token, /authorize).
* CKAN's login form will be rewired to initiate a client OAuth request on the /profile resource.
* Recognizing CKAN's callback URL at catalogue.data.gov.uk, Drupal will automatically grant the request if a user is signed in.
* Upon return, a user is created, optionally using the OAuth access token as the users API key, thus making it known both to Drupal and CKAN.
* The authorizer will be extended to check access to the /profile resource for OAuth accounts (slow but safe).
Advantages:
* Well-known protocol, does not depend on cookies (which are strange and never behave as defined, or even the same in multiple browsers)
* Python code is available: http://oauth.googlecode.com/svn/code/python/oauth/example
* Drupal seems to have very complete support: http://drupal.org/node/296205
* Can be fully implemented as a plugin, using an OAuthClientController for callback and adding hooks to the Authorizer (perhaps need to reconfigure who.ini)
"
787 1289815658000000 johnbywater Moved from sprint 1.3.3
787 1303118054000000 thejimmyg The AuthAPI now exists as an IMiddleware plugin, we really need the permission system moved into CKAN before it is useful though and this depends on a refactor of the Auth system. See #1094
787 1315821118000000 thejimmyg "The joint authentication was implemented a long time ago and is deployed on catalogue.data.gov.uk. We'll build the authorisation layer in ticket #1326 so marking this as fixed.
"
788 1289815658000000 johnbywater Referencing ticket #786 has changed sprint.
788 1294410007000000 thejimmyg Duplicate of #884.
789 1294248289000000 thejimmyg Will, this means that we just get given a URL to harvest from and we need to determine whether it is a CSW or a WAF. Can you look at this please? The code may already do it, I'm not sure.
789 1296593257000000 thejimmyg Covered by #736 now.
790 1289815658000000 johnbywater Moved from sprint 1.3.3
790 1294409709000000 thejimmyg Duplicate of #736.
791 1294409723000000 thejimmyg Duplicate of #736
793 1289815658000000 johnbywater Moved from sprint 1.3.3
793 1294409836000000 thejimmyg "Anyone else know what this means?
I can't find changeset aa9aa32e00a9 in the repository so marking as invalid."
794 1289815658000000 johnbywater Moved from sprint 1.3.3
794 1294248216000000 thejimmyg This work is now underway in the underlying DGU implementation.
794 1297686491000000 thejimmyg Actually, for the timebeing we will match but not do anything with that matched information, until there is a clear use case. Publisher is simply the publisher for which the source was registered. Closing this ticket.
794 1300877517000000 thejimmyg I think we need to look at this again in relation to scotland.
795 1296593361000000 thejimmyg This is a duplicate of #794 because if we needed to do this it would be because we need to support harvest sources registered on behalf of publishers by agents,
796 1291640040000000 rgrp We no longer have an alpha page due to refactor of search and usage of search for list view (see ticket:847 and subtickets).
797 1289402982000000 rgrp Fixed yesterday night in cset:5b522c762e71
798 1289815658000000 johnbywater Moved from sprint 1.3.3
798 1294916973000000 dread This was done a while ago.
799 1294232675000000 thejimmyg "How can we tell a WAF document has changed, we simply need to re-harvest it to see surely?
Moving the issue to ticket #728 to be dealt with together."
800 1289483682000000 johnbywater Referencing ticket #799 has changed sprint.
800 1294245610000000 thejimmyg Discussion for this ticket is now at #728
801 1289483616000000 johnbywater Referencing ticket #728 has changed sprint.
801 1289483682000000 johnbywater Referencing ticket #799 has changed sprint.
801 1294233386000000 thejimmyg Perhaps this should be implemented differently so that each harvest attempt creates its own database entry with a timestamp and a status attached. We'll need to move to this when we move to a queue based system.
801 1296593735000000 thejimmyg I don't see why we'd really need to do this. Under the current system a job would be created just before the harvesting would be run so the creation date of the job is (give or take a few seconds) the date of the harvesting. If we moved to a queue system this would be done better anyway.
801 1297686706000000 thejimmyg We do have a requirement for this now. The job model has changed so that it is hidden from the user. We therefore want to know the timestamp the job started and the timestamp it finished. We'll therefore need migrations adding too.
801 1300196714000000 thejimmyg This will now be solved as part of the larger harvesting refactor. See #921.
802 1294233294000000 thejimmyg Merging with #801
803 1289489024000000 dread "I'm not sure a template is needed for the test script - surely repeated stuff can just be factored out?
Creating the model dump file:
* 'hg up' to ckan as it was at the last model migration
* 'paster db clean', 'paster db init', 'paster create-test-data'
* 'pgdump'
* 'hg up default'
* Check in the dumped file
Do we really want a script to mess with our repo like this? Isn't it just safer to do manually with good instructions?"
803 1314031451000000 dread We now use nose to test migrations instead.
804 1294233156000000 thejimmyg This is the document I drafted after discussions with each party which was approved by JF.
805 1298379084000000 dread "Migration tests added to buildbot using kindly's new nose option #965.
Also removed legacy system of migration testing in: ckan/migration/tests
and updated docs. cset:643673c7db3e"
806 1290071159000000 dread This is done now I think? Please list changesets, time taken vs estimate and close ticket.
806 1291128886000000 rgrp Duplicate of ticket:827
806 1291134978000000 dread This is not a duplicate of #827
806 1294916284000000 dread This looks finished
806 1324034356000000 dread This was introduced in CKAN 1.3
807 1297075372000000 rgrp wontfix as with traffic we have this is not a massive issue and these are very minor items (we can reopen later if this comes up again).
808 1297783658000000 pudo implemented in cset:8200247e74e9
809 1297075561000000 thejimmyg This is something to think about more in the future.
810 1297679091000000 pudo At the moment, this crashes the groups field for some mysterious reason. Since this is going to be redundant with the new forms and the ticket has a low priority, I'm bumping this back 2 weeks.
810 1300093797000000 rgrp Moving back to backlog for v1.4 as should be dependent on forms overhaul and seems to be problematic (and not that urgent).
810 1310128477000000 thejimmyg This ticket is over 6 months old so closing.
811 1294415158000000 thejimmyg See also related extras field tickets #811 and #893
812 1294658293000000 dread Just added stuff from duplicate #781
813 1294411047000000 thejimmyg "This is now done. The text reads ""Add a Package"" in the main navigation. Thanks!"
814 1294411109000000 thejimmyg "This is now done. It's the ""About"" link in the main application. Thanks!"
815 1294410951000000 thejimmyg This is a duplicate of #234. Please have a look at existing tickets or re-open closed ones rather than adding new ones so that we can keep the old discussions. I've added your point to #234. Thanks!
816 1311171466000000 thejimmyg "John, could you have a look at this one please. We'd need an entry in the logic layer for it and then some autocomplete JavaScript on the format field.
Can we standardise on jQueryUI going forward.
http://jqueryui.com/demos/autocomplete/"
816 1312192499000000 rgrp "What about idea of 'model' refactor for resource, specifically to have:
* format: human created format string with possible nesting e.g. zip:csv
* kind: file | api | example | service
* mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
* mimetype-inner: mimetype of innermost object (so for example would be text/csv)
See http://lists.okfn.org/pipermail/ckan-discuss/2011-April/001139.html"
816 1312212782000000 dread "I like rgrp's idea to change the model. But since it's complicated it would be good to have a UI.
e.g. You might make format field simply a drop-down with only some common options: 'CSV, Excel, Zip, Other'. If you go for Other, then a more substantial UI appears, to cover more filetypes, APIs, services and a text box for Other. And if you select 'Zip' then another set of options appears to say what is in the zip. etc.etc.
Another approach would be to have a background process downloading the data file and filling this in for you."
816 1312295371000000 johnglover "I have implemented the resource autocomplete so am closing this for now.
I agree that more information on the resource would be a good addition. It seems a bit of a different issue to just adding autocomplete though, with potentially far more changes to the codebase, so maybe it would be better discussed in a separate ticket/CREP. However, any new fields would presumably want to either be constrained or have an autocomplete and so can reuse the work done on this feature.
Cheers,
John"
816 1319804698000000 dread John did this in cset:1697dfa2552c for release 1.5
816 1319812324000000 dread And branch feature-816-format-autocomplete
817 1289997628000000 cygri "I'm not allowed to create attachments for some reason. Here's the screenshot:
[[Image(http://richard.cyganiak.de/2010/ckan/ckan-resources-table-proposal.png)]]
http://richard.cyganiak.de/2010/ckan/ckan-resources-table-proposal.png"
817 1297073724000000 thejimmyg This appears to be fixed now.
818 1305559442000000 dread 1.4 complete
819 1323171435000000 thejimmyg This issue is fixed. Further improvements to the UX of the autocomplete will be dealt with in other tickets.
820 1311325226000000 dread This has been fixed
821 1298486642000000 dread "Investigating several of these packages, it works for me (and David Raznick).
For example ni_013_migrants_english_language_skills_and_knowledge, one resource is seen created in the diffs, is displayed in CKAN, in the API and in the dumps. Yet looking at the dump from 17/11/10 when this ticket was created, the resource didn't have a URI, which by the current model is a requirement, so it suggests the data was fine underneath, but it had problems displaying this field, and is now fixed."
822 1290506354000000 dread "Fixed in cset:045cab10aa1d
Thanks for the patch!"
823 1290506116000000 anonymous Fixed in cset:3845a501ed5f
824 1290595559000000 dread Done on metastable for 1.2 release.
825 1290624449000000 dread Added plenty of info in cset:627249a1c102
826 1297414412000000 dread "It's great that the extras are added in-line with other Resource properties (as opposed to package extras which are a dict not off 'package' but off 'package.extras').
However, the resource extra field keys are defined in config option ""ckan.extra_resource_fields"". This config option should be removed - extras need to be entirely flexible for our purposes. (In the next ticket we should make it possible to add/remove both keys and values from the Web UI or API.)
It would also be good to tidy things in this direction: http://wiki.okfn.org/Coding_Standards
I've merged from default to the branch enhancment_826_resource_extra_fields ready for you."
826 1297415095000000 pudo "I want to strongly support david in his call for fully flexible extras: one of my use cases for them is to store the various bits of fallout from an archiving process, such as:
* last-status
* last-crawled
* last-etag
* last-expires
* last-md5
* failure-count
* fall-back-url
These are things needed to really archive the data well, but they have nothing to do with CKAN core ops. Essentially, the archiver is a seperate concern and it need not appear in the CKAN config."
826 1297416879000000 kindly "There is nothing to stop anyone from putting any extra attributes in the extra_info field dict. So any have the flexibility you need.
The config option is to add some fields that act in exactly the same way as python attributes, having the same semantics as them. i.e if you have an extra field called alturl, you can do obj.alturl = 'fdsffs'.
This is the best of both worlds as far as I can tell."
826 1297417900000000 kindly "I forgot to mention that he main advantage of the fixed fields, is that we can make them properly searchable i.e the values searchable. This currently does not work for package extra values as they are jsons.
I have added this searchability for the sql backend."
826 1297421022000000 dread Ah - I understand. I'd like it if you could make all the extra fields work as attributes and be searchable - is that possible?
826 1297423342000000 kindly "This would be too much of a hack. You do not want users overwriting any attributes on the object. If they called the attribute ""__init__"" it would write over the actual __init__.
"
826 1297429910000000 dread "Merged into default in cset:613d7bd5fc96.
Future tickets in this area include:
* #978 Full edit in Web UI
* #979 Edit Resource extras in the API"
826 1306766057000000 dread This went into ckan release 1.3.2.
827 1290766239000000 rgrp Fixed in cset:63a0c0223da4
828 1290696586000000 rgrp Closed in cset:66a3efe16ed4
829 1290697386000000 dread Some work towards this in cset:066b1b553595. No exceptions and test skip in place. Looks like formalchemy changed its interface.
829 1303118864000000 thejimmyg Is this still relevant?
829 1303122056000000 dread Rufus originally specified it. Reassigning to him to decide whether we still want it or not.
829 1303838115000000 rgrp This is a wontfix for the present.
830 1311180263000000 thejimmyg We can now implement themes via extensions or change them via config options. See https://bitbucket.org/okfn/ckanext-exampletheme/ for an example.
831 1290760864000000 rgrp Done over the last week (finally fixed weird trac no __dict__ bug today and enabled user accounts on wednesday).
832 1296334980000000 rgrp Done (~4w ago). See https://bitbucket.org/okfn/ckanext-stats and remove from core cset:311313e4afdb.
833 1295263447000000 rgrp Have repo https://bitbucket.org/okfn/ckanext-admin
833 1298889104000000 rgrp In progress now (sysadmin view and update nearly done).
833 1302276855000000 rgrp Authz subsystem complete.
833 1303236364000000 rgrp Two main tickets done so closing.
834 1291633657000000 rgrp "Alexander: Documentation is here: http://packages.python.org/ckan/api/version2.html#search-api
Those queries should work but I do suggest you use the stable version of CKAN which is v1.2 and we suggest you use that. If you still have problems please reopen. "
835 1291644820000000 pudo done in cset:d41b77099954
836 1291851041000000 rgrp Fixed in cset:2f6d54341b47 and branch feature-286-siteurl
836 1302882808000000 dread This has been a (minor) issue since release v1.3
838 1291299716000000 memespring Package page redesign: http://ckan.org/ticket/839
838 1291636351000000 memespring Merge css files http://ckan.org/ticket/846
838 1291721955000000 memespring Show welcome message: http://ckan.org/ticket/850
838 1291729819000000 memespring Prompt users to enter missing info - http://ckan.org/ticket/863
838 1291736461000000 memespring Search results changes: http://ckan.org/ticket/864
838 1291812318000000 memespring Add download formats to search results (http://ckan.org/ticket/866)
838 1295259773000000 rgrp I'm going to close this ticket as almost everything done and remaining ticket can be done on its own.
839 1291736541000000 memespring Done with the exception of the discuss/comments page. Plugin won't install on my setup. Fredrich looking into it.
840 1291325251000000 dread "dread:
] That's the proxy_cache decorator which sets a Beaker expiry time only.
] Whether that is on or off, you still end up run the
] ""etag_cache(cache_key)"" command in the package controller won't you?
] And won't that will either insert the Etags header or abort 304 for a
] repeat? I must admit I've not tried it, asking purely with the code in
] front of me and welcome you pointing out where I've gone wrong.
ww:
Right. So those etag calls predate the cache decorators and should
probably be either moved up into the decorator (e.g. use @ckan_cache
instead of @proxy_cache) or wrapped in a check for the config
parameter. (And @ckan_cache could be changed to use pylons' etag_cache
function rather than just setting the header...)"
840 1302694123000000 dread Basic on/off switch added, tested & documented in cset:0da189c9630e on default.
841 1300364333000000 thejimmyg See #995
842 1296468313000000 rgrp Change to awaitingtriage as definitely not critical.
842 1303474131000000 thejimmyg "As a user I come to a package:
Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
At the bottom is a section of the package display titled ""ToDo"" where I see a list of all toDos for the package most recent at the top
If I am logged in
see a form for ""Add to do"" at the top of the todo section and can add one straight away
I see a ""now resolved"" button next to each which goes green when you hover.
When clicked the todo fades away.
Otherwise I see a button that says ""login to add todo""
expands out the form
The form
One of the fields is category -> autocomplete the category (not constrained)
Add a description
Submit, the todo gets added via AJAX to the list at the top as the most recent todo
Model:
todo
id
package_id
todo_category_id (required)
description (required)
date=NOW()
resolved=False
todo_category
id
name
Prepopulate with: broken-resource-link, no-author, bad-format
"
842 1303474228000000 thejimmyg "
> Otherwise I see a button that says ""login to add todo""
>
> expands out the form
>
Actually rather than expanding the form, you will go away to the login page and come back to see the expanded form (question: how does this redirect you back to the bottom?)"
843 1291652696000000 pudo The real name is also requested via AX/SReg extensions in OpenID so for new users who are not using google, this should usually be filled in automatically.
843 1319710380000000 dread " * ""we should show the openid as well to distinguish between users with the same name."" - when ""Full name"" is not distinguishable, maybe best to display the unique 'name' field as a hover-over.
* ""on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name."" - yes you always got take to the personal details page. We should use a flash message at this point if they have not filled in the ""Full name"" field to suggest they click edit and do this.
* ""List is to long"" - this has been addressed - see http://thedatahub.org/user"
843 1319721601000000 dread "Ok I misunderstood this ticket. This is referring to adding a user in e.g. http://thedatahub.org/group/authz/energy-data
This UI seems to be updated. You start typing the name, full name or open id of the person and it has a dropdown that autocompletes. This seems to be sufficient for Will's points 1 and 3.
Would be still good to have a flash message on account creation to encourage people to add personal their Full name. This is similar to #1413 so I'll close this ticket and add it there."
844 1296340486000000 rgrp Looking at DNS this apparently has been fixed.
845 1291723492000000 dread This was completed on the feature-845-required-fields branch and merged into default in cset:3b5635ffaa7d
846 1291715212000000 rgrp Reopening and fix used illegal css ;) -- it is illegal to have an @import statement (as we do with extras.css) after normal css statements. We therefore need to move the core css back out into a ckan.css (or similar file).
846 1291719074000000 memespring It's been done on a branch and pushed to bitbucket: http://bitbucket.org/memespring/ckan/src/9e74c40ff073/ckan/public/css/style.css
847 1291639903000000 rgrp Done my best to pull all the solr related tickets and work together in one place.
847 1292844627000000 rgrp Still have ticket:669 and ticket:874 to do (431 probably won't be done for a while).
847 1295259902000000 pudo This is now functional, minus the index testing tool.
849 1291715179000000 pudo == #846
851 1292860957000000 wwaites " * link checker above uses the queue. queue not running generally
* quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest"
851 1292892515000000 wwaites "Ready to put into cron job. cf:
* http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions)
* http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions)
* http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi 1.2)
* https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update)"
851 1292894661000000 wwaites currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow...
851 1292941303000000 wwaites "urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp.
change the httpReq action to use http://curl.haxx.se/libcurl/python/"
851 1293025112000000 wwaites "Implemented curlReq that does a curl request and returns statements
similar to or analogous to httpReq. Require curate<=0.8
This in a cron job is sufficient to go through all the packages and update
them with a broken link tag now:
{{{
curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY
}}}"
852 1291723207000000 wwaites plugin documentation: http://packages.python.org/ckan/plugins.html
852 1295259620000000 rgrp Removing from sprint and moving to release as a master ticket.
852 1315821628000000 rgrp Moved dataset archiving into new ticket #1327 and this is therefore now done.
853 1291723143000000 wwaites done in http://bitbucket.org/ww/ofs need to merge back into main ofs repo
853 1294047550000000 pudo "Looking at the changeset this cannot be functional yet: where is the implementation of the policy document exchange. It seems to me like this is currently adding the actual credentials to the request (self.ofs.conn.add_aws_auth_header(headers, 'PUT', path)).
c.f.:
* http://doc.s3.amazonaws.com/proposals/post.html#Form_Fields
* http://code.google.com/apis/storage/docs/reference-methods.html#postobject
"
853 1294594581000000 wwaites "we don't need a policy document exchange. it's simpler than that. the ""server"" instance has already permissions to upload. it just calculates the headers and such that are needed (based on the ""client""'s initial headers) and gives them to the ""client"" the client then uploads without knowing the ""server""'s credentials. The ""client"" never needs any of its own goostor credentials at all.
the only separate step is to make the widget readable by the world.
ticket #879 is to expose this as a small set of API calls."
854 1304351843000000 johnlawrenceaspden "Coverage now up to 84% and 81%. Remaining untested code is error conditions, which we decided weren't worth the effort of locking down.
fixed on feature-854-tests-for-authz-groups, now merged into default code."
855 1311176988000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. The auth code is in the process of a major refactor anyway.
856 1311177066000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
857 1311177075000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
858 1291734297000000 dread On branch feature-858-diff removed ckan.lib.diff as it was not being used. Merged into default in cset:f9ba1ae63ddd
859 1311177461000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
860 1291726067000000 wwaites http://bitbucket.org/ww/ckanrdf/changeset/67df6dc33ec4
861 1311168845000000 thejimmyg This ticket has been open for than 6 months so I'm closing it.
862 1291726261000000 wwaites not to mention sparql endpoint...
863 1295259827000000 rgrp Removing milestone as not certain when we'll do this.
863 1338206455000000 ross UI has changed rather a lot in last 18 months, so I killing this bug.
864 1291741028000000 memespring Havent implemented all text changes because worried about impact on other themes.
865 1294414639000000 thejimmyg Isn't this in there now? Can we close? Thanks
865 1295259773000000 pudo This has been fixed on the respective branch and merged into default..
867 1291916278000000 dread Done in ckanclient cset:d40fb101aba9
867 1292410515000000 dread Also ckan cset:c5b018bfe9bb
867 1299866685000000 rgrp This was a breaking change for loaders code. Obviously we don't have tests for that so would not have been noticed ... Fixed in cset:af81e54bd590/ckanclient
868 1292890251000000 kindly@gmail.com "Below is a patch to make the tests run at least 2.5 times faster (about 15 mins on my old laptop). Instead of dropping the tables each time, it just deletes everything in them, using a low level connection. All the tests pass this way. It's a surprisingly clean patch. Here are a few points concerning it.
* I tested truncating the tables but it's slower. If there are any big tables in the tests this way is the fastest (faster than drop).
* The sequences (id columns) will start from where they left off.
* I also investigated making postgres template database and cloning it, but the complication was not worth it.
* sqlalchemy iterates the tables in reverse dependency order, which make this possible.
* I targeted rebuild_db as that what most of the tests I saw where using, however I have not checked all tests to see if they all are.
* There is a slight hack on the repo object to make sure it knows that ""clean_db"" is coming from the tests.
* I refactored init_db for code reuse.
* I have not done a version check. sqlalchemy >= 0.5 do this in a different way as outlined in the comments.
{{{
diff -r 7f2239b0f743 ckan/model/__init__.py
--- a/ckan/model/__init__.py Fri Dec 17 10:34:47 2010 +0000
+++ b/ckan/model/__init__.py Mon Dec 20 23:25:04 2010 +0000
@@ -41,6 +41,9 @@
def init_db(self):
super(Repository, self).init_db()
+ self.add_initial_data()
+
+ def add_initial_data(self):
# assume if this exists everything else does too
if not User.by_name(PSEUDO_USER__VISITOR):
visitor = User(name=PSEUDO_USER__VISITOR)
@@ -69,6 +72,26 @@
import migrate.versioning.api as mig
version = mig.version(self.migrate_repository)
return version
+
+ def clean_db(self):
+ # delete only added for tests
+ if hasattr(self, ""delete_only"") and self.delete_only:
+ self.delete_all()
+ else:
+ super(Repository, self).clean_db()
+
+ def delete_all(self):
+
+ self.session.remove()
+ ## use raw connection for performance
+ connection = self.session.connection()
+ ## sqla sorts in reverse dependancy order.
+ ## in >= 0.5 use reversed(metadata.sorted_tables()) instead of table_iterator
+ for table in self.metadata.table_iterator():
+ connection.execute('delete from ""%s""' % table.name)
+ self.session.commit()
+
+ self.add_initial_data()
def setup_migration_version_control(self, version=None):
import migrate.versioning.exceptions
diff -r 7f2239b0f743 ckan/tests/__init__.py
--- a/ckan/tests/__init__.py Fri Dec 17 10:34:47 2010 +0000
+++ b/ckan/tests/__init__.py Mon Dec 20 23:25:04 2010 +0000
@@ -55,6 +55,7 @@
import ckan.model as model
model.repo.rebuild_db()
+model.repo.delete_only = True
class BaseCase(object):
}}}
"
868 1292921428000000 sebbacon "See also #867
Thanks for the patch."
868 1292921456000000 sebbacon I mean #876, of course.
868 1293401808000000 anonymous Attached are the timings I have for the tests after I upgraded to 0.57 and after a few simple test tweaks. They do not include setup and teardown time at the class level as they are not assignable to individual tests.
868 1294753596000000 dread "I've merged in David Raznick's patches:
* no_autoflush_deletes.diff cset:2b9591172182
* postgres_speed.diff cset:fa1b7e3a4e0f
* vdm_purge_no_autoflush.diff vdm cset 8accdd0b9b7f
I've also merged in Seb's fork: cset:68d63fda4814 which closes this ticket, achieving test speeds of under 3 minutes!"
869 1292059662000000 rgrp Closed by cset:83734b5e251c which implements an IConfigurer interface.
870 1294862485000000 kindly@gmail.com "A patch is available here.
https://bitbucket.org/kindly/ckan/changeset/9a1d6f55587b"
870 1294914243000000 anonymous Merged into default in cset:54ae110094be
871 1292322361000000 wwaites "Regarding rkhunter -- yes, eu1 appears to be clean
Regarding the upgrade -- upgradede to 4.72 from backports which, looking more closely, appears to still have the privilege escalation bug but not the remote root exploit.
Regarding exim on other hosts, there is no reason for them to be running a full mta, something like ssmtp should suffice.
Also very worth the thought of moving to postfix. It's much easier to configure and I haven't known it to have any comparable bugs in the decade or so I've been running it. In fact I've never seen anyone actually use exim before..."
871 1292323656000000 nils.toedtmann "Re postfix: I second ww. I like to run some super simple local MTA (e.g. ""nullmailer"") on all but one server, using a central postfix (or a send-only GMail account) as smarthost. Am happy with postfix for >10years, it's straightforward and rock solid.
"
871 1296340558000000 rgrp This is not a ckan issue. should have been on http://knowledgeforge.net/okfn/tasks
872 1292957110000000 dread Done in ckanext cset:8a7e931ef37c
873 1292939274000000 dread Fixed in dgu cset:8bfb867e247a and ckanext cset:fabd31544a73
875 1297085261000000 pudo So far opting for route 1) (not implementing facets), therefore this can be closed!
876 1292891133000000 kindly@gmail.com "I have looked into this already so I can give you a head start. I am working on a project that uses many backends so I have some experience. So here is what I have found so far.
== nested transactions ==
VDM does not support sqlite, as it uses nested transactions. I do not think vdm needs nested transactions. It can use a flush instead. Here is the patch that works. All vdm tests pass.
{{{
--- a/vdm/sqlalchemy/base.py Sat Sep 11 23:06:26 2010 +0000
+++ b/vdm/sqlalchemy/base.py Mon Dec 20 16:16:34 2010 +0000
@@ -40,9 +40,8 @@
self.setattr(session, 'HEAD', True)
self.setattr(session, 'revision', revision)
if revision.id is None:
- session.begin_nested()
session.add(revision)
- session.commit()
+ session.flush()
}}}
== indexes ==
The index file 021_postgres_upgrade.sql in the migrate repository will not run as it uses syntax particular to postgres. Another will need to be made thats similar. sqlite does not support complex indexes like upper(text), so a work around will need to be found.
== unicode ==
The harvesting returns utf8 encoded strings and pysqlite dbapi only supports python unicode objects (as far as I can tell). There will need to be a process in converting all strings that get into the database with string.decode(""utf8"")
== dates ==
Have not looked into this one too much. However, as sqlite stores everything as strings the timestamps appear to be failing on conversion back into python.
I have solved the above two issues before by adding attribute extensions to sqlalchemy mappers to do the conversions without effecting too much code.
== in memory sqlite ==
Some tests need to change in order to make sure the database is created first because the database gets lost each time. In the tests that I have made pass, they run in about a seventh of the time as they do on postgres.
== Other things to keep in mind. ==
* Need a new flag in test.ini to remove full text indexing completely, or always use it with solr.
* There are enough incompatibilities between the databases that you would also want to test against postgres as well, at least before a release.
* I would probably upgrade sqlalchemy first, so you will not have to the changes twice. The new versions are significantly faster too.
* I have submitted a patch to #868 that makes the tests run about 2.5 times as fast and I think there are more low hanging fruit if the aim is test speed.
"
876 1292924064000000 sebbacon "Thanks for the info!
Re. nested transactions. I am getting repeated non-deterministic test failures against sqlite (and indeed postgres, but these failures appear more frequent against sqlite). One of them I seemed to be able to get rid of by eliminating the savepoint as per your first point. However, it appears that sqlite [http://www.sqlite.org/lang_savepoint.html does] support savepoints; to demonstrate it, the following test code appears to work in the latest sqlalchemy:
{{{
#!python
from sqlalchemy import *
from sqlalchemy.orm import *
db = create_engine('sqlite:///')
metadata = MetaData(db)
users = Table('users', metadata,
Column('user_id', Integer, primary_key=True),
Column('name', String(40)),)
users.create()
class User(object):
pass
usermapper = mapper(User, users)
Session = sessionmaker()
session = Session()
fred = User()
fred.name = ""Fred""
sue = User()
sue.name = ""Sue""
amy = User()
amy.name = ""Amy""
session.add(fred)
session.add(amy)
session.begin_nested()
session.add(sue)
session.rollback()
session.commit()
assert session.query(User).count() == 2
print ""OK""
}}}
So, while I agree they're not needed, I'm not sure they're a problem. What do you think? Also, have you seen non-deterministic test errors like this?"
876 1292939711000000 kindly@gmail.com "I have read quite a lot of people having problems with savepoints with sqlite and thought they were not supported on sqlalchemy. They are at least not consistant with postgres ones. I may well be out of date on this.
Here is an [http://groups.google.com/group/sqlalchemy/browse_thread/thread/dc9d1b61044bf730/65a62a33ec313842?lnk=gst&q=no+such+savepoint#65a62a33ec313842 example] even though its a bit old.
I did get some non deterninistic errors, the above seemed to fix them. A failed subtransaction is not handled well by sqlalchemy and I think this causes knockon effects due to the unresolved transaction. I would stay well clear of them entirely if possible.
What are the errors you are getting??
My 2 cents. ignore me at will...
I would think about using a different backend for testing than production. [http://stackoverflow.com/questions/2716847/sqlalchemy-sqlite-for-testing-and-postgresql-for-development-how-to-port look here].
If you want to support both then you should test on both.
There are simple ways to scrape a few more minutes off the tests. If you want real speed, then a multiprocess solution (with a database per core) would be sensible if a bit tricky.
"
876 1293188088000000 anonymous "Thanks for your feedback, very useful.
I don't really agree with the people in the linked discussion who say it's pointless testing against a different database from production.
The goal here is to make it easy enough for people to run as many tests as possible that they actually do so. Even 15 minutes is too long in that case. With sqlite we can get it in at under 5 minutes. I would also like to identify the longest running tests (which I would characterise as ""functional"" or ""integration"" tests and make them run as a separate suite, and then encourage a culture of writing true unit tests before functional tests, so that running unit tests can happen in 1 minute and be part of the regular development cycle.
That's no replacement for also running *all* tests periodically, and also running tests under postgres, which we can continue to do on the continuous integration server.
Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests."
876 1293218714000000 anonymous "I agree with all your points about testing apart form using sqlite, especially splitting out the functional tests and continuous integration.
> Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests.
Well there are two options
1. refactor the tests
2. refactor the code to use sqlite and postress
It is a value judgment as to which is more complicated. I personally think 2 is more complicated but may be wrong on that. The real danger with 2 is that you are needlessly adding complication to production code, with 1 you are only changing the tests.
Upgrading to sqlalchemy 0.5+ should happen first regardless. You will need upto date documentation.
There is another option too. Put the postgres data directory on tempfs/ramfs and turn off durability [http://www.postgresql.org/docs/9.0/static/non-durability.html here]. We would need a way to db init before the tests where run (or) at boot). This may be the best of both worlds.
Anyway Happy xmas!!
"
876 1294753889000000 dread Seb and David have completed this I believe. I've merged the changes into core CKAN in cset:68d63fda4814.
877 1297680579000000 rgrp Basic pass on an implementation (no permissions yet etc): https://bitbucket.org/okfn/ckanext-upload/changeset/9ae543f0645f
877 1298624165000000 rgrp "Various tidying in https://bitbucket.org/okfn/ckanext-upload/changeset/0fad7aa7aa97 (success messages, permissions on uploaded file - public-read) and completed permissions in https://bitbucket.org/okfn/ckanext-upload/changeset/a83ce00a1266.
Still need to integrate into general workflow (e.g. create a Resource on successful upload) but that is a separate item so this ticket is now done."
878 1314404621000000 rgrp A working version of this in ckanjs as of 2 weeks ago. Need to integrate into main CKAN.
878 1315820838000000 rgrp This is now done in feature-1294-ux-improvements-dataset. see e.g. cset:c6f7f5018b4f
879 1294604066000000 wwaites first cut: https://bitbucket.org/ww/ckanext-storage/src
880 1292945137000000 dread Fixed in ckanclient cset:e7c0af586367 and ckanext cset:82d974ab6860 with test in dgu cset:c0b2c5fd95ea
881 1294410574000000 thejimmyg "Hello fccoelho,
Could you please post the output you get from running pip?
Thanks,
James
"
881 1296335072000000 rgrp Closing due to lack of clarification.
883 1300196622000000 thejimmyg Now complete.
884 1293319984000000 wwaites "Implemented in feature-884-rmsource branch at https://bitbucket.org/ww/ckan/changeset/50cbfc5a6bee
"
884 1294247654000000 thejimmyg Related to ticket #665, reopening so that I can close it once I've got it working with Drupal too.
884 1294253563000000 wwaites see feature-884-rmsource in the main ckan repo deleting my own repo
884 1296592140000000 thejimmyg OK, I've implemented an API call for this too now and all is merged into default.
885 1293278564000000 wwaites Friedrich had some mixed experiences with owslib and some German CSW endpoints: http://pudo.okfnpad.org/geodaten
885 1293314219000000 wwaites "See: https://bitbucket.org/ww/ckanext-ows
impossible as of yet to test against live server from the OS "
885 1294253497000000 wwaites done with feature-885-owslib
886 1294916538000000 dread Being done as part of DGU ticket: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/757
887 1300196600000000 thejimmyg "This is done. See https://bitbucket.org/okfn/ckanext-harvest
At the moment we are not adding migrations to remove the harvesting tables. We are designing a harvesting refactor and will write migrations once the refactor is complete so that instances that use harvesting get upgraded, and those that don't get their harvesting tables removed.
Also, see this #1030 for the moving harvesting out of the REST API.
"
888 1294830297000000 Stiivi "Chages to Data Proxy:
* tests added with configurable list of known URLs
* use brewery for transformations (included reference to brewery framework in a new vendor directory)
* side effect: code to make google find external packages in vendor directory (from now on, all external packages should go there and be referenced from .hgsub if they are mercurial repositories)
* changed response contents: moved from 'headers' to root, renamed 'response' to 'data', added field list as 'fields'
* changed way of registering transformers (now class object is used instead of name)
* added 'encoding' and 'dialect' parameters for CSV
* added optional data audit (parameter 'audit')
Changes: https://bitbucket.org/Stiivi/dataproxy/changeset/fccbdd275be5
Data information: http://databrewery.org/doc/data_quality.html#brewery.dq.FieldStatistics
"
888 1307352537000000 thejimmyg I don't think any progress has been made on this for a bit so I'm assigning it to me.
888 1311773103000000 johnglover "Dataproxy / Dataapi now deprecated in favour of the combination of new QA archive / process commands and the webstore.
Changes in relation to Dataproxy / Dataapi:
* Currently only supports CSV files, but plans to add support for excel and google docs spreadsheets soon.
* Uses David Raznick's CSV parser instead of Brewery for parsing, handles messy CSV data better.
Changes in relation to old QA functionality:
* decoupled archiving (downloading) and QA process
* added a new 'process' command which parses downloaded files and adds them to a local webstore
Closing for now, any improvements/feature requests should be in tickets relating to either the QA functionality or the webstore."
889 1293715109000000 rgrp Done in cset:cdbb2e6128b3 (had added item to template earlier pulling from 'g' object but had not tied that up to config as yet). Also added test and also '''removed existing google analytics code''' -- so need to explicitly set this in config from now on.
890 1318529648000000 rgrp May want to close as invalid as obsoleted by more recent queue work.
890 1318599247000000 kindly Invalid due to #1397, We will be using celery instead.
891 1318602128000000 rgrp May only do link-checker and not do full storage in this sprint.
891 1320143424000000 johnglover "Almost finished (see http://github.com/okfn/ckanext-archiver).
Still to address:
- check headers to see if hash / cache / max-age / expires indicates that the resource does not need to be downloaded.
- add cache url to resource"
891 1320149841000000 johnglover "Added cache_url and cache_last_updated to resources after archiving.
Not checking for hash value in headers. This process will generally only run when a new resource is added or someone updates a URL, so we don't expect to be regularly downloading the same resource.
We will need something along these lines if this is running as a regular cron job, but in that case the logic will be added to the cron job itself (probably a paster command)."
892 1324314636000000 rgrp Moved to v1.6.
892 1324402480000000 johnglover "https://github.com/okfn/ckan/compare/bff538b%5E...d700680
Merged with master and deployed on test.ckan.org"
893 1298293527000000 thejimmyg We don't understand the use case for this requirement. Closing for now until a use case can be demonstrated.
894 1300196388000000 thejimmyg All test CSWs have been successfully harvested from. CSW harvesting is disabled on that source so we can't harvest from it. I don't think we need a ticket for this now do we?
896 1300217375000000 thejimmyg "Actually, I think this should be implemented instead by considering a different CKAN instance as a potential harvest source and then harvesting from it. By thinking of it this way in the first instance, we effectively get a read-only copy of other data in CKAN but one which will be kept up to date.
Marking as duplicate. Discussion can now carry on via #987 Common harvesting framework."
897 1294914333000000 dread Same as #870
898 1294659537000000 dread From David Raznick
898 1294662408000000 dread Merged in cset:89bf917f9b46
899 1294662417000000 dread Merged in cset:89bf917f9b46
900 1294662424000000 dread Merged in cset:89bf917f9b46
901 1294662466000000 dread Merged in cset:b8a649f51cab from Seb's changes.
902 1296638257000000 rgrp "Also confirmed in FF. David (R) and I have both had a go and fixing this and it seems to be a bit tricky (very odd css behaviour!).
Rather than directly fix this I'm thinking of doing a bit of rework of that set of menu tabs as they could be much nicer anyway and hence re-titling of this ticket."
902 1297078885000000 rgrp Fixed in cset:42102bf7f922 and cset:42102bf7f922 by switching to a new 'tabs' look that has none of these issues. (See branch feature-902-submenu).
903 1295265645000000 pudo This is now partially solved by filtering for deleted packages. As two further measures, we should also add a package deletion method to the Solr backend and skip deleted packages when indexing.
903 1296588070000000 pudo fixed in https://bitbucket.org/okfn/ckanext-solr/changeset/be4cffa9b987
904 1299840539000000 rgrp We're already now into improving the docs and ticket:927 is now reasonably detailed.
905 1328638536000000 dread "This works with the current SOLR search.
e.g. http://thedatahub.org/dataset?q=gr%C3%B6%C3%9Ften"
906 1324299793000000 rgrp Do we need to change in core code or just configure solr?
906 1328639465000000 dread "This was done in #1701 but only in a custom SOLR schema in the ecportal extension: https://github.com/okfn/ckanext-ecportal/commit/6682926d8895f146cdf1e52ab4fbead9b065af77
Can the ASCIIFoldingFilterFactory be added to core CKAN's SOLR schema for all CKAN users to benefit from?"
906 1328782087000000 amercader Yes, I don't think there will be any problem, and we won't need to create a new version of the schema as the change is backwards compatible.
907 1296335337000000 rgrp "Basic implementation done and deployed. However plenty to improve, e.g.
* Support more formats (use external systems for preview?)
* json (!)
* html (trivial!)
* sparql
* ...
* Do not display preview if no preview
"
907 1297072303000000 rgrp Closing this now and improvements can go in a new ticket.
908 1296403695000000 wwaites "see also from http://knowledgeforge.net/okfn/tasks/ticket/485
{{{
(pyenv)okfn@eu7:~/var/srvc/ckan.net$ sudo uwsgi -C -iH /home/okfn/var/srvc/ckan.net/pyenv --paste config:ckan.net.ini --uid www-data -s 10.48.162.201:9001
*** Starting uWSGI 0.9.6.6 (32bit) on [Sun Jan 30 16:00:13 2011] ***
compiled with version: 4.4.3
Python version: 2.6.5 (r265:79063, Apr 16 2010, 13:28:26)
[GCC 4.4.3]
uWSGI running as root, you can use --uid/--gid/--chroot options
setuid() to 33
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
allocated 412 bytes (0 KB) for 1 request's buffer.
Setting PythonHome to /home/okfn/var/srvc/ckan.net/pyenv...
binding on TCP port: 9001
your server socket listen backlog is limited to 64 connections
initializing hooks...done.
Loading paste environment: config:ckan.net.ini
Traceback (most recent call last):
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/__init__.py"", line 3, in
from loadwsgi import *
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py"", line 8, in
import pkg_resources
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2691, in
add_activation_listener(lambda dist: dist.activate())
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 668, in subscribe
callback(dist)
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2691, in
add_activation_listener(lambda dist: dist.activate())
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2195, in activate
map(declare_namespace, self._get_metadata('namespace_packages.txt'))
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 1790, in declare_namespace
_handle_ns(packageName, path_item)
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 1761, in _handle_ns
loader.load_module(packageName); module.__path__ = path
File ""/usr/lib/python2.6/pkgutil.py"", line 238, in load_module
mod = imp.load_module(fullname, self.file, self.filename, self.etc)
File ""/home/okfn/var/srvc/ckan.net/pyenv/src/ckanext-dataapi/ckanext/dataapi/__init__.py"", line 36, in from pylons import config
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/__init__.py"", line 4, in
from pylons.config import config
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/config.py"", line 2, in
from pylons.configuration import *
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/configuration.py"", line 25, in
import pylons.legacy
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/legacy.py"", line 11, in
from pylons.util import deprecated, func_move
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/util.py"", line 18, in
from paste.script.appinstall import Installer
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/script/appinstall.py"", line 23, in
from paste.deploy import appconfig
ImportError: cannot import name appconfig
}}}"
908 1296659056000000 pudo fixed in https://bitbucket.org/okfn/ckan/changeset/08c0e1a819e4 and https://bitbucket.org/okfn/ckanext-stats/changeset/6e86c071db99
908 1296660052000000 wwaites "Not fixed. Try running with:
{{{
uwsgi -C -iH /path/to/your/pyenv --paste config:ckan.net.ini -s 127.0.0.1:9000
}}}
Still get giant traceback relating to a circular import centered on an arbitrary plugin"
908 1296730578000000 pudo I've moved the uswgi/nginx part to #952, closing this issue since the problem it describes has been fixed.
909 1346669602000000 ross Backlog until funded.
910 1310128544000000 thejimmyg Not enough information and over 6 months old so closing.
911 1296467375000000 pudo Done.
912 1306774876000000 pudo moving to ckan
913 1299841413000000 rgrp If at all possible this should uris from the OKF licenses registry at http://licenses.opendefinition.org/
913 1306774962000000 pudo moving to CKAN, still need a resolver
915 1296467518000000 pudo Live and running :-)
916 1297066902000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/4/port-new-vdm-to-mongodb
917 1295280232000000 dread Fixed in cset:f2865f43d8ee
918 1315911507000000 dread Preview feature is now deprecated
919 1303202627000000 dread Simple to fix this in the process of fixing #108. Fix went in cset:304d30d85816 for release 1.3.4.
920 1295355318000000 hellmann@informatik.uni-leipzig.de my first email address was wrong...
920 1300217199000000 thejimmyg @David Should we implement this as a change to the way tags work or via a tidy up cron job?
920 1300319140000000 kindly "The only issue here is that we are listing tags that relate to 'inactive' packages. We are already not listing tags that relate to NO packages.
I have fixed this.
cset:cd0347eed69f
The tag in the example is related to a deleted package so should not be deleted. With this patch it no longer gets displayed."
921 1300197629000000 thejimmyg Seb started this, I completed it and the code is now live. Further refactoring will be done first in #1037 and then in #987.
922 1310128789000000 thejimmyg "See also #358 which I'm closing because it gets merged into this ticket.
The reason we were blockd on #358 was to do with issues around authorization."
922 1320664187000000 rgrp Duplicate of #1032 (I think)
925 1323168588000000 thejimmyg This is fixed in CKAN 1.5
926 1296637927000000 rgrp "Comments from RP - http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html
Libraries I have used: FormEncode, FormAlchemy (what we are currently
using, before that formencode).
Neither seemed perfect but I think the form issue is a 'hard' problem
(perhaps with no perfect answer) [1]. FormAlchemy, in retrospect, was
probably a mistake as it merges too much model/validation/form
generation into one thing.
At least 3 functions involved:
1. Generating (or just filling) a form template with 'form data' (and errors)
2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
3. Converting form data to model data (and validating) (inverse of
previous step)
I think one can and should separate 1 from 2+3 (and one of problems
with formalchemy is it doesn't -- the attraction being you don't
repeat yourself as forms get generated from model but I think this is
actually a false economy in medium-term).
I'm not specifically recommending the following as I haven't used them
but I've looked through docs, they are active and reasonably mature:
1. Flatland: http://discorporate.us/projects/flatland/docs/tip/
* Only does 2+3 which is a good thing IMO
2. WTForms: http://wtforms.simplecodes.com/
* Used in standard flask docs: /path/to/file/you/want"
982 1298482394000000 dread Need to do this for older branches which isn't subject to #963.
982 1298887980000000 dread Buildbot scripts now fixed.
983 1297773407000000 dread "Error was tracked down to cset:214a8f9fc1c2 (26-9-2010):
upgrade_db called validate_authorization_setup() which calls setup_default_user_roles(System())
Fixed in cset:9f51a1c8ac83 for 1.3 branch and merged to default."
984 1297628554000000 kindly "These are the errors listed in severity order.
* group revision tables joins wrong in many ways.
* changemask table not added.
* licence_id wrong type.
* package_revision.download_url and changset.status not dropped.
* package.name and tag.name unique constraint not added.
* update cascaded defined wrongly.
Attached are the fixes that will need to be run in ckan_migration_fixes.sql"
984 1297682116000000 kindly fixed see https://bitbucket.org/okfn/ckan/changeset/d56ea86d4303
985 1298571248000000 pudo digitialiser.dk has been assigned to Stefan Marsirske to get him into this Framework, everything else is delayed.
985 1303118298000000 thejimmyg We'd like CKAN to CKAN harvesting this week if possible.
985 1306407617000000 amercader Moving this to the PDEU (publicdata.eu) project
985 1306408134000000 amercader Duplicate of #1155 and #1156
986 1297812401000000 wwitzel3 https://bitbucket.org/okfn/ckanext-qa/src/be57e20c60ef/
987 1311177705000000 thejimmyg This has largely been implemented now for publicdata.eu. There is a CREP #1134 outstanding to take the harvesting to the next level so marking this one as duplicate for now.
989 1297700363000000 kindly "It would be nice to know some use cases. I think that plugins should control their own storage, or share a storage that is designed to be flexible (mongo, redis ...).
We do not seem to be able to keep our current migrate repository in sync let alone add plugins to the mix."
989 1297700818000000 pudo "Kindly, I agree - it would be much preferable to have independent storage for plugins and this would be easy to do if we were using another type of storage already. As it stands, however, our storage mechanism is SQL. I think we should use it for what it is as much as possible and do the weird, vertical stuff (k,v tables, swapping to redis) only if we really need it. For everything else: lets use SQL as it was intended.
Examples:
* We want to develop an apps catalogue as a CKAN plugin. While we could certainly put this in Redis, there is no reason why we can't have the following table: application (id, name, title, description, author, project_url, site_url, code_url, image).
* A watchlist plugin could essentially work on UUIDs alone. What you'd end up with is something like this: watch (id, user, scope_id).
Re migrations you're right, but my first intention would be to handle that seperatly for each plugin (i.e. they need to have their own migration repositories that they keep track of, e.g. via an apps_migrate_version table)"
989 1297706620000000 kindly "I do not think we need to 'extend the model' if you intend to make the migrations separate. If the schema is decoupled, then there are no problems. So each plugin can have its own model and use sqlalchemy independently i.e have their own metadata, classes and mappers. They do not have to even use sqlalchemy.
What I mean is that there is no need to do anything apart from.
* Agree on a naming convention of the plugin tables (including their own migrate table each)
* Agree to the rule that no plugin can add a column to an existing table.
* Agree that no table can have a (database level) foreign key constraint between the core tables and itself in either direction. They *can* have implied sqlalchemy level joins.
* Maybe have a hook that on db upgrade all plugins are upgraded.
Each plugin will have to redefine the tables, classes and mappers they need to join onto the core tables themselves. reusing/extending the core model will not be worth the trouble.
This seems to cover your use cases and this way everything is nicely decoupled. Best of all there is very little work to do... "
990 1311180850000000 thejimmyg This now works.
991 1298037717000000 dread Fixed in cset:56cccbbb9d1a in time for ckan 1.3 release. This did not affect previous releases.
992 1298060474000000 rgrp Fixed in cset:08548ef8f0e9
993 1298373114000000 dread Fixed on 1.3 cset:7708c8b521ed and merged to default. Deployed to ckan.net.
994 1298912830000000 kindly see cset:93188d42fc12
995 1300364316000000 thejimmyg "Also, update the docs:
Documentation article on caching / improving performance. (To complement configuration docs.)
* Different sorts of cache - beaker style, etags, package_dict in search results(?)
* How each one affects performance
* How to turn them on/off and configure them
* Is it possible to bypass each of them in the browser or with wget/curl?
Therefore closing #841."
995 1311175353000000 thejimmyg The caching still isn't brilliant but there is less urgent need to refactor it. In the future our caching methodology should be to cache at the logic layer so we don't need to refactor the existing caching at this stage.
995 1311179009000000 thejimmyg "See also previous tickets:
* #537 Caching and Performance improvement
* #543 Investigate partial page caching and edge-side includes"
996 1300364398000000 thejimmyg This is now fixed, the outcome will be used to inform #995 for the caching.
998 1298369862000000 dread 'paster db create' (or init) should do exactly what we ask. Surely we should simply tell people to use 'paster db upgrade' instead?
998 1298371191000000 anonymous I am happy to get rid of paster db create altogether as a compromise? Or add a depreciation warning to it?
998 1298372171000000 dread Yes I agree - either of those sounds good. I think I've always used 'db init' in preference anyway.
999 1300707328000000 rgrp Done. See https://bitbucket.org/okfn/ckan-ckan.net and updated ckan.net.ini.
1000 1298912726000000 kindly fixed cset:630513f550d5
1001 1300367555000000 thejimmyg Yes, I think the two should be consolidated, but as was noted elsewhere (I think) we don't want the API to ever be authenticated via a cookie, otherwise we open ourselves to CSRF attacks. Instead we could support the API key as a query parameter as an alternative to via the HTTP header if it is hard to set an HTTP header in JS libraries?
1001 1300625888000000 pudo "My vote on this would be to enable ""httpbasic"" auth via repoze.who and to add an API key challenger to ckan/lib/authenticator.py that accepts API keys, from within repoze.who. This way we won't have to work around the framework at all and can retain compatibility to the existing mechanisms."
1001 1300703779000000 rgrp First work in cset:6b8cbe465b61
1001 1300704249000000 pudo Wrote a WDMMG version of this yesterday: https://bitbucket.org/okfn/wdmmg/changeset/398ce0eb3901#chg-wdmmg/lib/authenticator.py
1001 1301229082000000 rgrp Completed and merged into default in cset:cb200f339dbb. At the moment have done the very simple option but pudo's approach seems better and we could refactor towards that going forward.
1001 1301307814000000 dread You mentioned writing tests? Also the CSRF question from James hasn't been addressed.
1001 1301310351000000 rgrp This ticket did not include CSRF matters - please raise in a separate ticket if you wish. I looked in some detail at the testing options and nothing seemed simple to do that didn't duplicate code we already have (since no way to access the base controller halfway through a request). This can be tested as necessary as part of the development of the new js work.
1002 1311173185000000 thejimmyg I've done this in my branch and it will get merged in.
1002 1311675754000000 dread "Can you also delete the ""paster changes"" in ckan/lib/cli with this removal?"
1002 1311676199000000 dread Reopening this, since it has not been pushed or merged in afaict.
1002 1322515684000000 dread I've redone these changes in cset:91258a6296db71d78a on master, aiming for 1.5.2.
1003 1298490126000000 rgrp Work so far in http://bitbucket.org/rgrp/ckanjs
1003 1298889293000000 rgrp Have now started refactor to use backbone and have basic inline editing working and started on Add dataset view.
1003 1300100411000000 rgrp Converted completely to backbone and now have fully-operational add dataset functionality. I'm closing this ticket now - further improvements can be in there own tickets.
1004 1311173560000000 thejimmyg "At the moment it says ""Create a new group"" and clicking it takes you to the login page. We want it to do the same thing but if you aren't logged in the label should say ""Log in to create a group""."
1004 1323171375000000 thejimmyg The yellow box on the right is now back and you don't get taken to the login page. We'll write more detailed instructions once the group refactor is done.
1004 1323172489000000 dread "There is a description of groups, but that is not the subject of this ticket. In fact if you're not logged in you do get taken to the login page (which is quite correct). I agree with your earlier comment that the text of the link needs changing.
Sorry to be reopening this, but I think the previous comment misses the point. Doing this ticket would be a valuable quick fix. Putting on backlog."
1004 1323174597000000 thejimmyg "I don't understand. We just had a team meeting about this, all discussed it and agreed to close it. Yes, you get taken to the login page, but that is the correct behaviour.
The problem in the past was that the link was in the yellow box and there was no explanation as to why you had to login. This time you click from the header bar and there is a clear message saying ""Unauthorized to create a group"" - exactly as a user would expect. Even if the exact text of the ticket description isn't fully implemented in the current release, the UX isn't broken anymore. Yes, it might be even nicer to have a message warning them in advanced but these improvements will be taken forward in the UX work - maybe there is an even better solution than a message?
Since you are unhappy about closing it I'm marking it as ""Duplicate"" of #1521. As agreed earlier with the entire team, we'll take this forward as part of the groups refactor.
See #1521 for more information.
"
1004 1323177994000000 dread "My suggested fix:
{{{
diff --git a/ckan/templates/group/index.html b/ckan/templates/group/index.html
index 8502df9..064e066 100644
--- a/ckan/templates/group/index.html
+++ b/ckan/templates/group/index.html
@@ -9,7 +9,15 @@
What Are Groups?
- Whilst tags are great at collecting datasets together,
+ Whilst tags are great at collecting datasets togethe
+
+
+ Creat
+
+
+ To create a new group, please first
${h.subnav_link(c, h.icon('group') + _('List Groups'), controller='grou
-
+
}}}
BUT this fix doesn't completely work. When you log-in and create a group, at this point the nav bar changes from the (correct) ""Add a group"" to (incorrect) ""Log-in to add a group""."
1004 1323187352000000 dread "Here is the unclipped version:
{{{
Create a new group
To create a new group, please first log-in.
}}}
{{{
${h.subnav_link(c, h.icon('group_add') + _('Add a Group'), controller='group', action='new')}
${h.subnav_link(c, h.icon('group_add') + _('Log-in to add a Group'), controller='group', action='new')}
}}}"
1004 1323194016000000 zephod "Dread, that fix wouldn't work because the class attribute was being defined in Genshi and again in HTML, with slighly undefined behaviour (in this case Genshi overwrites HTML).
I have performed a very very simple fix which modifies the navbar text when you're not logged in.
https://github.com/okfn/ckan/commit/a89a48731ba548170045a60ac2930e0019c299c7
I don't think the link should be restored in the sidebar, it was explicitly removed as part of a site-wide sweep to make the sidebar a passive, helpful element rather than an active element with action links.
Site looks great to me, I'm closing this ticket."
1004 1323195485000000 dread "Great stuff zeph! Diff looks really good and cheers for the explanation.
On master, cset:a89a48731ba548170045a60ac2930e0019c299c7 and I've cherry picked this for release 1.5.1. too."
1005 1300100085000000 dread Moved to dgu trac ticket 869
1006 1298631145000000 dread "This command is slightly different to your branch policy as of two weeks ago:
{{{
stable: stable code
metastable: (will soon be deprecated) for code preparing to be stable
default: development HEAD
}}}
which I prefer.
My ideal would be to get rid of the confusing name 'metastable' and unneeded 'stable' and start a new branch called 'released', which will act the same as 'master' in this diagram but with a more intuitive name:
http://nvie.com/posts/a-successful-git-branching-model
Then for each ckan instance we can either use the most recent release (from 'released') or choose a specific one (e.g. 'ckan-1.3' or even 'default' or 'enh-865' for getting latest features). This gives a good degree of flexibility, is more understandable to newbies and probably a more widely understood branching model."
1006 1300098217000000 dread Reassigning to rgrp for response
1006 1300105927000000 rgrp "Not sure when i described that other process but I'm definitely of the opinion that we should:
* Deprecate and remove stable branch
* Deprecate and remove metastable branch (going forward we can use release branches for what we did with metastable in the past)
May want to update BranchingPolicy to reflect this (also should probably have some statement about closing release branches and tagging)
Reassigning to David Raznick as he is our release guru now."
1006 1300372286000000 thejimmyg I'm updating the branching policy now. David has closed stable and closed metastable. Marking as fixed.
1008 1298821826000000 rgrp I've removed the eval in cset:1b8fedeb7ab0 - the more general question about caching should go in a separate ticket.
1009 1298638447000000 pudo "Some more ideas:
* /user should list users, sorted by number of packages contributed/editied
* /user/{name}/packages shows a list of packages to which users have contributed
"
1010 1298733856000000 rgrp Complete see branch feature-1010-list-users and closing changeset cset:feature-1010-list-users.
1010 1298740889000000 rgrp Meant this cset:d2651db566ef
1011 1298820235000000 sebbacon On reflection, may as well make a Plugin interface called IAuthorizer, which allows customisation of get_authorization_groups, get_roles, and is_authorized....
1011 1298821699000000 rgrp I agree that IAuthorizer is useful but not sure how it addresses the requirement of the ticket. AuthorizationGroups are already editable via the web interface at /authorizationgroup
1011 1298824649000000 sebbacon "The ""external source"" is an Oauth service. We need to lookup user groups from that service."
1011 1298825600000000 sebbacon Proposed implementation at https://bitbucket.org/okfn/ckan/changeset/187e65afb35f
1011 1299245206000000 sebbacon Merged to default https://bitbucket.org/okfn/ckan/changeset/e8217c317a8e
1012 1300217601000000 thejimmyg "See also #103 which says:
As a user I want to view a package at a given revision:
* When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
* package history page should provide links to these pages
Could you please investigate how doable this would be given the current package read.html page. I expect it could be very easy once the revision API is in place because we could build a c.pkg object from the revision data instead of the model data.
Perhaps another case where we could try introducing the dictization?"
1012 1300352334000000 kindly "This is entirely not trivial at the moment. Hopefully after the dictization it will become more so.
Simply putting in the package revision object in place of the package does not work.
It will obviously work for changes to the package object itself. However, there are no mappers on that object for getting out the related package_tags, resources and extras at that revision. You will have to construct a fake pkg object with some messy and painful queries using dates."
1012 1300362584000000 kindly "cset:5649d6e761fc
The basic revision history is merged. I will keep this ticket open if it is not sufficient.
All it does is give a list with the most recent first of revision_ids, authors and timestamps.
i.e
{{{
[{""timestamp"": ""2011-03-16T15:55:19.941961"",
""author"": ""southampton-ac-uk"",
""revision"": ""202e9eb8-afaa-4bc9-b8a1-a317561547ea""},
{""timestamp"": ""2011-03-15T17:59:16.430804"",
""author"": ""southampton-ac-uk"",
""revision"": ""8235bd0a-d39a-49e0-887a-b0f231be8a92""}]
}}}
"
1012 1300364620000000 thejimmyg The API part is now done. Bumping the view history part to 1.5
1012 1300368449000000 dread As I understand it, the whole point to vdm is to make it easy to get old versions of not only the package, but its related objects (like tags, extras). David take a look at this function: Package.get_as_of(revision)
1012 1301909937000000 thejimmyg No-one really seems to have requested this part.
1012 1301943096000000 dread Changing this to 'Fixed' and #103 to 'Wont fix' to ensure this feature is noted.
1013 1299245157000000 sebbacon "This is now resolved, but depends on core CKAN behaviour (specifically pluggable middleware and unicode-aware error pages) to function:
https://bitbucket.org/okfn/ckan/changeset/c846794c1799
"
1014 1299245293000000 sebbacon Run out of time for decoupling, but tests and README.txt written (including pointers about how to customise for anyone who needs to decouple in the future)
1015 1298902753000000 kindly The migration fixes should sort this out, but I will keep the ticket open to check.
1015 1299788821000000 rgrp Today kindly applied the sql fixes and I can confirm this is now fixed. Well done kindly for all the great work here.
1017 1299668555000000 pudo fixed in cset:86d49a775fd3
1018 1299073340000000 dread Done in cset:e4167f8b3f80 on default
1019 1299166930000000 pudo fixed in https://bitbucket.org/okfn/ckanext-webhooks/changeset/034647931921
1020 1300196215000000 thejimmyg This is now on DGU live.
1021 1299518828000000 pudo fixed in cset:9f1b38add19f
1022 1299512991000000 pudo "We're now using fileConfig to configure the logger API from the worker config file and this enables us to use SMTPHandler to send out error messages on queue processing failures.
Marking as fixed. "
1023 1299514847000000 pudo "Tried implementing this with AMQPs msg.requeue() and channel.basic_recover() but RabbitMQ yield a NOT_IMPLEMENTED error.
Bit clueless on how to proceed. "
1024 1299668648000000 pudo re-created this (#1027)
1025 1299598708000000 dread Done in enh-1025-config-default-authz and goes into release 1.3.2.
1025 1299750097000000 rgrp (Link to changeset). Could you briefly clarify why this was needed in config -- we already have a process for putting things into a more restricted mode (see ticket:833) and have been working on creating a WUI to be able to do this automatically (see that ticket).
1025 1299751045000000 dread "Thanks for pointing this out. Although changing the definition of 'editor' to not allow edit (as you admit) is a bit hacky, and I think prone to confusion. James and I weren't aware there was a precedent for doing it this way, but if we had, we may well have followed it.
I mentioned the branch for this cset, in preference to the csets."
1026 1299605128000000 dread Done in cset:2dde3bd563fd for branch 1.3.2
1027 1299666326000000 pudo "
1. home controller -> __before__ (check ""site-read"" on model.System)
2. user -> each individually (repoze-who pseudo action must not be blocked)
* user-read (index/read/update pages for users)
* user-create (register)
3. revision -> __before__ (check ""site-read"" on model.System)
4. tag -> site-read (__before__)
functional/test_authz.py
* denies site-read ...
* checks for visitor / logged in user ..
* checks you can still visit login
"
1027 1299682082000000 pudo fixed in cset:532c3ad2743b
1028 1299760360000000 dread Fixed in cset 4d860a53fbad on 1.3.2 and merged to default.
1029 1301311643000000 thejimmyg This was fixed by kindly as part of the 1.3.2 release.
1030 1300103984000000 amercader "This includes:
* Take out all references to harvestsource, harvestingjob and harvesteddocument in the rest API
* Move the harvesting bits of ckan/lib/cli.py into ckanext-harvest.
* Move ckan/controller/harvesting.py and cknan/model/harvesting.py to ckanext-harvest as well
* Update ckanext-csw to be able to find the code it needs in the new place."
1030 1300209975000000 thejimmyg "Also:
* Delete harvest sources
* Implement a Genshi template filter so that if there is an INSPIRE package_extra set to True, you can link to the XML and HTML generation API calls in ckanext-csw
* Write tests in ckanext-dgu for all the API calls that Drupal is making
"
1030 1303117978000000 amercader All the points described in this ticket have been completed. Harvesting work is being done mainly under #1037
1031 1299863222000000 rgrp Done, see cset:78d96b520679
1032 1300380606000000 rgrp This is urgent for me as without upload does not really function.
1032 1303117292000000 thejimmyg Rufus, I think this is potentially quite a big change and needs to be done as part of the entity refactor rather than as a quick hack. I'd like David Raznick to work on timeseries this week so I'd like to understand a bit more about this ticket. Could you flesh it out please or let's discuss in the catch up?
1032 1303118226000000 thejimmyg See related ticket #922
1032 1304094162000000 rgrp Agreed to defer this. Resource editing, to the extent it happens can be kept very simple and part of package editing.
1032 1330348463000000 zephod Streamlining supertickets. #1506 is the new parent of #978; all tasks remaining in this ticket are complete.
1033 1308310446000000 dread Fixed in 5c7caca30737 for v1.4.1
1034 1320174353000000 dread Same problem as #1321 I believe
1035 1323172908000000 thejimmyg Re-assigning to me temporarily to investigate now that Seb has left. It may be that this should be deleted given that it is over 6 month's old.
1035 1338206251000000 ross Removing until we can come up with a case for implementing
1037 1303117000000000 thejimmyg "We spent last week integrating the new harvesting architecture and testing the code but there are still some areas that need looking at
* The source type and label should be part of the plugin, not named in DGU.
* Need warnings if a document changes but its date doesn't -> do we have these?
* I noticed there are some tests in DGU, should these perhaps be in ckanext-harvest?
* If active is False, the job should not be put on the queue
* Log if the wrong type of URL is entered as an error the user can see
* Deny if the source is already registered
* Overwrite all extras, not just merge new ones.
* During the import stage use iswms.py to add an extra during import if it is a WMS so that we can add a link to the WMS later https://gist.github.com/900878
* Can errors/warnings be logged in the import stage? Do all fetched documents get passed to import in one go?
"
1037 1304937601000000 thejimmyg Closing this now, any outstanding small issues will be logged in new tickets.
1038 1300212841000000 dread Done in cset:0373737f0866 for release 1.3.2
1039 1300212856000000 dread Done in cset:97d563bbf01d for release 1.3.2
1040 1300213922000000 dread Fixed by James in 904f1f36a672 for 1.3.2
1040 1328805778000000 dread "The problem arises when you do two things together:
1. Have code like this in a CKAN extension's plugin.py:
{{{
config['extra_public_paths'] = ','.join([our_public_dir,
config.get('extra_public_paths', '')])
}}}
which will have a trailing blank value (unless you set extra_public_paths in the config file)
{{{
extra_public_paths=['/some/path/', '']
}}}
2. Run CKAN with this code:
{{{
extra_public_paths = config.get('extra_public_paths')
if extra_public_paths:
static_parsers = [StaticURLParser(public_path) \
for public_path in \
extra_public_paths.split(',')] + static_parsers
}}}
Introduced in #340 https://github.com/okfn/ckan/commit/0593c3e5050882a9dbf54fb6c263f4a2f1de2097#ckan/config/middleware.py in CKAN 1.0.2
This was made safer in https://github.com/okfn/ckan/commit/e0d5f02168116a18a7aeb7b058e4160bbd402a02#ckan/config/middleware.py for CKAN 1.3.2 (it ignores blank paths)
"
1040 1328806824000000 dread "Here are all the CKAN instances potentially involved in this issue:
== ckan.net / DGU ==
These suffered this problem briefly and were fixed 15/3/11
== opengov_es http://opengov.es/ 1.2 ==
This does not have a theme extension. Is not serving files from root.
== nosdonnees_fr http://www.nosdonnees.fr/ 1.3.1a ==
This does have a theme extension but does not seem to have a trailing blank public path since it is not serving files from root.
"
1041 1300284809000000 thejimmyg Also, this means moving documentation out of the current CKAN dev wiki at trac.ckan.org and into the community wiki. Core developer docs should remain in the Sphinx-based .rst format in CKAN. Much-needed new tutorials go on the community wiki.
1041 1310135309000000 thejimmyg Child ticket of #1142
1041 1312372367000000 rgrp James says this is not priority and he does not plan to do docs for debs until he has got that system functioning.
1042 1300969265000000 dread Done in ckanext cset:7610512277bd and ckanext-importlib cset:7737716e0d7b.
1042 1300969865000000 dread "Remaining parts of ckanext are:
* blackbox (rgrp, pudo, dread)
* hammer (ww)
* link_checker (ww)
* solr configuration (pudo) & indexer (pudo)
* amqp listener command (ww)
* geo form field (rgrp)
* GData Loaders:
* aidprojdata (johnbywater)
* norwaygovdata (johnbywater)
* ordsurv (johnbywater)
* Gemini2 (iso19139) (johnbywater, ww)"
1043 1300321033000000 kindly "cset:c894f92c5b9a
Sesion.remove() needed to be run before configure as we want the original session made not to be used over the newly configured ones."
1044 1300359562000000 rgrp Think this would be resolved by refactoring implied in ticket:1001 (specifically checks for api key and remote user should lead to availability of same set of identification information).
1044 1300382367000000 dread "I agree with leaving #1001 solving this issue. Centralising this is good. This ticket can instead focus on:
* returning the correct 403 error (rather than the 302 currently)
* creating some tests
* documenting
* consider renaming the SITE_READ variable and rethinking purpose"
1044 1300449905000000 dread "I've added in tests for SITE_READ and corrected the 403 response - see branch bug-1044-site-read.
The tests have uncovered some inconsistencies in the use of SITE_READ between reading/editing a package in the Web interface vs the REST API. Friedrich, do you want to take a look at ckan/tests/functional/test_authz.py:TestSiteRead.test_pkggroupadmin_read for details and have a think about how this should best work. How about making SITE_READ more widespread in the WUI, to protect read/editing/searching things, in addition to the READ and EDIT RoleActions.
Once that is decided, it should be pretty easy to refer to the tests and document SITE_READ."
1044 1300627007000000 pudo "David, thanks for writing those tests - perhaps we should combine them with the ones below (""TestLockedDownUsage"") which attempt to basically do the same.
As for the inconsistency introduced by the global check in the REST API you're right: Of course it is strange that WUI access checks are more granular than the API checks. The alternative is that we either move authz checks into all relevant REST places (a major refactoring I would be suspicious of) or that we introduce duplicate checks on the WUI actions (I actually have performance concerns about that, authz is incredibly slow - and it introduces another level of special authz that I think we really should not have). Given the choice I'd opt for the REST refactor - there is no good reason to make SITE_READ a duplicate check where authz already applies.
On the other hand, this is a function we really don't want to enable or even have and that was only added to satisfy some very specific user demands. Given that these are fulfilled, I'm actually OK keeping the inconsistency for now - nobody will see it in normal operations and in a locked-down environment, users will need to have API keys anyway.
Regarding the naming, I'm pretty opionion-less: SITE_READ was proposed by rgrp and I think its pretty fitting, while OTHER_READ would just confuse me. PUBLIC_READ might work, though. "
1044 1300712223000000 dread "I'm happy to leave the inconsistencies in SITE_READ for now, since this will be resolved when authz is centralised between WUI and API in the dictization refactor.
I'm also happy to leave both TestSiteRead and TestLockedDownUsage as they complement each other nicely actually.
So I think all that remains for this ticket is to add a paragraph to the authz documentation about SITE_READ. Pudo would you mind doing this? It's well worth noting where it applies and where it doesn't."
1044 1301304871000000 dread I think all that remains for this ticket is to add a paragraph to the authz documentation about SITE_READ. Pudo would you mind doing this? It's well worth noting where it applies and where it doesn't.
1044 1302096155000000 dread I've added in the docs for this in cset:013da53052d1 ready for release 1.3.3.
1045 1300457708000000 dread Fixed in cset:98eb8b8d063e on default, scheduled for CKAN version 1.4.
1045 1300787161000000 dread Want to identify groups by ID also in the Web interface.
1045 1300793261000000 dread Fixed in cset:71621df50983 on default ready for ckan 1.4
1046 1302777668000000 kindly cset:35ba6ad033ae
1048 1300702752000000 dread Done in changeset 9d7bfa124757 on default and expect it to go into 1.4 release.
1049 1323169424000000 thejimmyg This is over 6 months old. In accordance with wiki policy, closing.
1050 1300452200000000 johnlawrenceaspden there's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests.
1050 1300458454000000 dread "I may misunderstand how you intend to build the WUI Admin interface, but I think most of the stuff in authztool is just marshalling command line parameters anyway. The bits which do anything are factored out.
For example, to list rights you just loop over obj_classes to call {{{model.Session.query(obj_class).all()}}} and then display the values: {{{type(subj).__name__, subj.name, role, type(obj).__name__, obj.name}}}.
To change a right you simply call {{{model.add_user_to_role(subj, role, obj)}}} or {{{model.add_authorization_group_to_role(subj, role, obj)}}}.
But of course if there is useful stuff to factor out then be my guest!"
1050 1303118188000000 thejimmyg This should also feed into #1075 which will be being worked on this week.
1050 1315394117000000 dread There's been no objections to my comment 6 months ago saying this is invalid, so marking it invalid. Feel free to reopen if there is indeed a case here.
1051 1301305079000000 sebbacon This has been completed in https://bitbucket.org/okfn/ckan/changeset/64a949990e0b
1052 1300895410000000 dread Fixed in branch feature-DGU#889-authorization-in-lockdown and merged to default at cset:e6643cf1324c. Only remaining issue is listing in Web interface of package relationships, which shows links to packages even to ones you can't read.
1053 1310125444000000 thejimmyg This is now fixed as part of the API refactor and revisioning updates David Raznick has done.
1053 1310126546000000 dread "Looks to me like DELETE always changed the state to Deleted, so an invalid ticket.
We can deal with purging in another ticket if needed."
1054 1300989764000000 dread I've added some tests in the branch: defect-1054-resource-order
1054 1300992195000000 dread I've put in a potential fix for this into the branch - do take a look and see what you think.
1054 1301002995000000 kindly "Looks great. Had a look and changed a minor thing because I was not confident with the handling of the null values. I made a fake resource and did an asdict on that to make the identity.
Part of my patch I reverted as I mucked up my version of the vdm, so it was not needed.
"
1054 1301305615000000 dread This branch is now merged.
1055 1300987241000000 dread This amounted to 14 tests. Fixed in cset:5bbd0005c57e ready for ckan release 1.4.
1056 1302626373000000 dread Fixed in cset:a025b5b3293c on default.
1056 1302882616000000 dread This was broken in cset:b681bbedfa62a68b71260ef48a0da6063109734 which was released in 1.3.2
1057 1329150236000000 toby fixed in commit 3d7cbf0
1058 1301303221000000 dread Fixed in cset:8e1817ab8d1c on default, ready for 1.4 release.
1059 1301312516000000 dread Fixed in ckanext-importlib cset:f6d19129ac43.
1060 1301312487000000 dread Fixed in ckan-importlib cset:bc52bba86d71 and ckan cset:2751f76fb17a in time for ckan 1.4
1061 1301397271000000 dread Fixed in cset:fc3bc103db8c ready for release 1.3.3
1061 1301922325000000 dread Actually there was a link at the bottom of the package edit form that my grep didn't pick up: h.url_for(str('/license')). So it is not orphaned at all.
1061 1301922350000000 dread Backed out original change fc3bc103db8c here: 7ae9aff8bc68
1062 1311178145000000 thejimmyg "Hi John,
Can you please check that the new webstore+preview works correctly with this one please?
Cheers,
James"
1062 1311773731000000 johnglover The file is in Turtle format (serialisation format for RDF, which we currently don't archive and have no preview code for. The new preview code will not even create a preview button for this file type. Moving to the backlog for now, support for RDF and related formats should be added along with support for excel docs and google spreadsheets.
1063 1301522483000000 rgrp "This sounds absolutely sensible and suggest we even have a helper method such as lib.helpers.group_link(group) (as we do for user_link).
(@Seb: apologies for misunderstanding your query on the list -- I was talking about the group listing table at /group)"
1063 1302514033000000 sebbacon Fixed in https://bitbucket.org/okfn/ckan/changeset/145eaae3168a
1064 1323169787000000 thejimmyg Marking as duplicate because #1464 will take it on.
1065 1301914004000000 sebbacon "To reflect further conversations:
* We are parking this ticket until dictization is complete
* We would prefer to see roles as asserted globally by default, with packages just storing exceptions. E.g. If I am a ""reader"" globally, then I have ""read-package"" permission on all packages new and existing, unless otherwise asserted locally on a package.
* Also w.r.t. algorithm above, consider repoze.what's model; ""possible"" is analagous to ""has_permission"". See http://what.repoze.org/docs/1.0/Manual/Predicates/Builtin.html"
1065 1302033244000000 dread "Wow, lots of solutions here, and not enough evidence of the problems encountered / difficult use cases.
I have tried here to extract what John and Seb might be suggesting is difficult with our current http://en.wikipedia.org/wiki/Role-based_access_control system.
1. (Seb) It seems crazy to change the User-Object-Roles for every single package to 'lockdown' a CKAN instance. Instead of using Roles 'reader' & 'editor' we start using Roles 'logged_in_user' and 'anon_user'. Then, with only changing a couple of lines of the Role-Action table for these Roles, you can give or remove permissions to edit / create packages, groups etc.
2. (John) Actions are being added all the time by migrations and extensions. I'm not sure what the problem is here - can someone explain? The separation of Users and the new Actions by adding the Role 'join' means the migration/extension can add existing Users with the new Actions. For example when we added the 'Group' protection object, logged in users could previously create groups, so it is natural to add this action to the 'editor' role. If you get rid of Roles, as John suggests, then you end up having to add as many Rights objects as users. And you might end up trying to infer new Rights on the basis of old Rights, rather than a role, which would be more natural. (Is the only reason to denormalise the UOR and RA tables the sake of ease of understanding? I would think we can explain the concept better now...!)
3. (John) This Group Hierarchy description sounds a bit like earlier incarnations of the DGU requirements. ""or any of his groups group-children (but not user-children)"" I'm not sure I understand the user-children bit, and I'm wary of any difficult-to-grasp concept. This modelling seems to take the UserGroup/User hierarchy model and use that as a hard-coded interpretation of the Authorized permission. In contrast, in DGU Evan split the Organisation Hierarchy model from the Authorization model. Each User has explicit permissions for each Organisation he is allowed to add/edit packages for. This helps for these two use cases we have to contend with: when a user is an admin for unrelated branches of the Organisation hierarchy, and when an Organisation is actually in two branches of the Organisation hierarchy. It also seems more generally flexible.
"
1065 1302075994000000 sebbacon "Re the point against my name, yes -- I think the fundamental principles of the current system are fine, but the implementation means asserting things for every single object, whereas we should be able to optimise it for the general cases -- something along the lines you suggest.
I would be very happy to write up a full, detailed proposal for the system before we implement anything, if only as a proper straw man to have debates around."
1065 1302081747000000 dread Great stuff Seb, cheers for that.
1065 1303118756000000 thejimmyg This is now more urgent and I think we have enough consensus to go ahead. See #1094 too.
1065 1311174208000000 thejimmyg Merging with #1094 so see the discussion there too.
1065 1318528721000000 rgrp Reassigning to zephod as he is looking at this. I think most of this is wontfix or will become obsolete with other stuff but worth looking at. Also moving to v1.6 at the same time.
1065 1324550041000000 dread "All subtickets are done now, apart from #1198 which I've moved into superticket #1477 Group refactor
Change name of AuthzGroup? to UserGroup? to reflect what it is for
I think this is completely covered off in #1477 Group refactor
Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
This is not clear - please raise a ticket for this if still promising.
Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)
These three points I've moved into a new ticket #1596 as it was getting lost rather in this one.
Change UserGroups? so that they can have a hierarchical structure,
I think this is completely covered off in #1477 Group refactor
Conclusion: closing this ticket as all points are covered or moved into tickets elsewhere"
1066 1301904246000000 dread Need a new role 'ANON_EDITOR' which is the default role for Visitor, which can create packages, but not groups.
1066 1301930311000000 dread "Migrations:
* Default (open) CKAN instances will have visitor as a reader on the system, and will have to upgrade them to anon_editor.
* DGU & DataGM instances have visitor as a reader on system and must stay like that.
* Pudo's specially locked instance, where editor has been changed to only read can now use the 'reader' role (assuming he's happy for them to create users)
Migration script (37) is designed to cope with the default (open) setup, and the other special cases must be dealt with using CLI at the time of upgrade:
DGU & DataGM must run:
{{{
paster rights remove visitor anon_editor system:
paster rights make visitor reader system:
}}}"
1066 1301932136000000 dread Done on branch defect-1066-reader-too-permissive and merged into release-1.3.3
1067 1302186503000000 dread Done in cset:d75ab5fc3311 for release 1.3.3
1068 1302109505000000 dread "The problem was the use of mktime which is localised. Also spotted a problem with the use of localised time stored in the datestamp created in migration script 29, when initialising the repo.
Fixed in a2094932e5e4 in release 1.3.3 branch."
1069 1301909423000000 thejimmyg "This is what the new ""Ideas"" section in the ckanext-community extension is for. Marking as duplicate of #941, can we have discussion there please."
1069 1302510609000000 rgrp Mistakenly assigned to this sprint.
1069 1310133794000000 thejimmyg Yes, and this will be more important in thedatahub.org
1069 1340632215000000 icmurray Assigned to @tobes as he's looking at publication workflows.
1070 1310117129000000 rgrp Added link to http://wiki.ckan.net/Domain_Model and closing as work mostly done (and nothing clear remaining to be done -- agreement perhaps!)
1071 1301943180000000 dread Done in cset:db9e2c4f65bb for release-v1.3.3
1072 1302106474000000 dread Done in cset:2a97d2d9ba4a for release 1.3.3.
1073 1302185825000000 dread Done in cset:9dfc60db90ed on default branch.
1074 1314278296000000 rgrp johnlawrenceaspden did quite a bit on this in feature-1074-refactor-authz-wui branch (including implementing for package and group). However, did not implement for authzgroup and also repeated code a lot. Reassigning to me for completion.
1074 1314283492000000 rgrp "Also work on branch feature-1074-authz-wui (that is primary one in fact)
"
1074 1314303581000000 rgrp Closed in cset:18a5b48bbee6. Note the templating isn't completely reused but rather uses common methods from _util.html. Likely that a bit more refactoring could be done to share common code but not worth it at present.
1075 1303227982000000 johnlawrenceaspden "closing ticket
see http://wiki.ckan.net/Authorization
and https://bitbucket.org/okfn/ckanext-admin"
1076 1302515674000000 sebbacon See also http://trac.ckan.org/ticket/948
1076 1302731255000000 rgrp All main work done as of cset:556efcb91f19 in branch feature-1076-improve-purge-system (+ associated work in ckanext-admin trash bin management). All that remains is to add some tests.
1076 1303236302000000 rgrp Test of trash functionality in ckanext-admin in https://bitbucket.org/okfn/ckanext-admin/changeset/e86c102a3c60 so now closing (could do with some functional tests in the main interface but can leave for later -- were none before so situation no worse ;0 ).
1077 1319797284000000 dread I guess this is not a priority now.
1078 1305828965000000 kindly complete: cset:843b78bae016
1079 1302777496000000 kindly Complete see cset:35ba6ad033ae
1081 1303332388000000 johnlawrenceaspden Actually it seems to be worse than that. If I add a user to an authz group then I get added too, and then there's no way to remove me.
1081 1303489474000000 johnlawrenceaspden "Fixed as part of feature-854-tests-for-authz-groups branch.
Original behaviour was that creating/modifying user is explicitly added to the users in the authz group.
"
1082 1315317825000000 dread "The flash message 'Language set to: English' was always in the previous language, not the new one. Pudo fixed this in cset:eaf342823caf and I have transplanted this into release-1.4.3.
I've also added/corrected translations for all languages for 'Language set to: English' so that it makes sense!
""Set language back to English causes server error:""
I can't reproduce this. The exception looks like a problem with the forwarding, which improvements were made in #1126."
1082 1315917217000000 dread Language switch problem has been around since it was introduced in the 1.3.1 release.
1083 1304975681000000 johnlawrenceaspden "Current behaviour:
from the web interface, russianfan can be added as an admin on warandpeace twice.
and you can then delete the two roles separately.
desired behaviour is that you can only add him once.
from the command line, you can only add him once, but if he's in there twice, you can't remove him because the command line bombs
desired behaviour is that he can be removed
from the shell, you can add him twice but can't remove him if he's been added twice.
if you try, it bombs.
The desired behaviour in all cases is that adding and removing are effective and idempotent in all three cases."
1083 1305051061000000 johnlawrenceaspden It appears to be possible to get into a state where you can't add or remove roles from the command line. Investigating.
1083 1305058621000000 johnlawrenceaspden "Fixed the problem where you can't delete a role if it's been added twice, but had to leave the add function as it was since putting a commit in it breaks many tests.
It appears that the WUI doesn't call these functions anyway. Left that as it is since we're planning to replace it.
The cli rights command doesn't appear to add the roles that it says it adds. Possibly because it's not doing the required commit? Reported this as a different bug."
1084 1303894555000000 dread ww you did this didn't you? Perhaps you could update us with the branch/changeset it was in and close this ticket please.
1085 1302621951000000 rgrp "I don't think this is very useful output :) Most of this is test/demo/rdf and therefore not relevant.
Also have you checked your ckan.site_url in your development.ini -- which configures where what is used as based for all external urls in the templates?
IMO there is nothing to fix here."
1085 1302624602000000 dread As discussed on IRC, the default ckan.site_url should be blank, rather than ckan.net.
1085 1302625314000000 dread Fixed in cset:01b3de5d4e31 on default.
1086 1303381200000000 johnlawrenceaspden There is no (obvious) way to delete an authzgroup.
1086 1323346552000000 dread Authorization groups are deprecated with the incoming Groups Refactor #1477
1087 1310134714000000 thejimmyg Child ticket of #954
1087 1320866159000000 dread "Implemented in cset:0b7e2b0bed44 as status_show logic action, heading for 1.5.1 release.
Took 0.5h"
1088 1302630624000000 wwaites "you use the negotiate function in autoneg.py with a cfg dictionary like this,
{{{
cfg = {
""mime_types"": [
(""application/xhtml+xml"", [""html""]),
(""text/html"", [""html""],
(""application/rdf+xml"", [""rdf"", ""owl""]),
(""application/turtle"", [""ttl""]),
(""text/plain"", [""nt""]),
(""text/x-graphviz"", [""dot""]),
]
for content_type, exts in negotiate(cfg, accept_header):
if ""html"" in exts:
render page normally
else:
redirect to semantic.ckan.net
}
}}}"
1088 1303035487000000 wwaites "fixed in 5c091fc08474 (bug-1088-autoneg-is-wonky) tip
"
1089 1302631733000000 dread Done in cset:6d41ecc431e3 on default.
1090 1302635699000000 dread Fixed in cset:e0af88edf0f0 on default.
1091 1303746408000000 rgrp "This is a well known issue (see some discussoin in #142). There is nothing directly we can do about it (this is what google gives us) but we could make usernames editable (this has been discussed but not ticketed I believe). Given our general ""downer"" on openids I'm not sure this is a big priority."
1091 1323102767000000 rgrp wontfix plus we now requires usernames see #1386
1092 1305570822000000 kindly completed cset: ca1ac4112ea2
1093 1302865470000000 dread Fixed by this API call use the existing license cache. Done in cset:a95888ea39be on release-v1.3.4
1094 1303129670000000 rgrp "See also comments on the mailing list.
Item 1 seems fine (what is difference from current extension mechanism?)
Item 2: concerns here. What about list views? What about editing 'permissions'? I also think getting rid of System object isn't really a benefit (if anything may be a cost).
Item 3: feel this may be better as part of big domain model change (also gives us time to really dig into this -- this is an important requirement/conceptual issue).
Item 4: No objections but seems very minor gain for fairly significant migration work.
"
1094 1304354036000000 johnlawrenceaspden "UserGroup/PackageGroup might also be confusing.
A PackageGroup is *just* a group of packages.
A UserGroup is both a group of users, and a thing affecting authorizations.
Perhaps PackageGroup and UserAuthorizationGroup? Or PackageGroup and AuthorizedUserGroup?
I was quite confused by all this at first.
I think I understand how the whole thing works pretty well now, and I still can't think of good names for the two concepts, although I can already feel the normal English meanings of the words changing to what I now know they are for.
We should be a little wary of this. Things that are even slightly difficult to understand end up being understood by very few people. Would any of us be prepared to sit an exam on exactly how UNIX file permissions work, even though we all use them?
"
1094 1311173649000000 thejimmyg Merging with #1065 and closing.
1096 1307444626000000 nils.toedtmann See also http://ckan.okfnpad.org/multisite
1096 1310125173000000 thejimmyg "Renaming to CKAN Hosted because whether we introduce a site domain object for subdomain tenants or not is an implementation detail.
As far as progress on CKAN hosted goes, we now have:
* A build tool to package ckanext-dgu, ckanext-std etc into .deb files
* The ability to apt-get install CKAN
* A script that can set up multiple non-conflicting CKAN instances on the same machine
What we need next is the ability to generate a ckanext-xxx repository for each new instance we want to host. At that point most of the underlying techniques for hosting multiple CKAN instances concurrently will be in place. Once we have that working and know what clients want, we can look at going further and speccing up a multisite solution off a single DB, rather than the multi-instance solution proposed here.
"
1096 1311182262000000 thejimmyg Milestone ckan-v-future deleted
1097 1303293476000000 dread Done in cset:5dec7eb609ab on branch release-v1.3.4.
1098 1303387896000000 dread Fixed in b2b37e39de4d in release-1.3.4. Was only broken on this branch a few days before.
1098 1303406017000000 dread Also found problem with some tests using delete_all. Fixed in cset:f9764dd43a0a release_1.3.4.
1099 1324057106000000 dread Authorization groups deprecated with the Group refactor #1477
1100 1303406103000000 dread Done in release_1.3.4 cset:580e2b11b4ef and release 1.3.3 cset:cac55b8a0bd9
1101 1330082908000000 dread Stats already in site nav - adjusting title to just mention google analytics.
1102 1303744552000000 rgrp (Same ticket created 3 times by accident I think -- see #1104).
1103 1303744575000000 rgrp (Same ticket created 3 times by accident I think -- see #1104).
1104 1303744486000000 rgrp "You haven't done paster search-index rebuild I think.
If you haven't the search index will not have been built (Create Test Data code creates data without indexing it). This could be considered a bug of Create Test Data but I'm not sure -- if you think so re-open this ticket and change description."
1104 1303920492000000 dread I agree this should be fixed.
1104 1303920791000000 dread Fixed on default cset:52a3fb230074 - create-test-data now indexes the package it creates.
1106 1303834069000000 rgrp fixed in branch:defect-1106-bugs-with-routing.
1107 1310134646000000 thejimmyg Child ticket of #954
1107 1320321170000000 dread "Notes to complete this:
* ckan/logic/action/get.py:'def package_autocomplete' exists and is the one that should be used
* need to ensure all places in the templates and public/scripts use this action directly, not dataset/autocomplete (or its alias package/autocomplete)
* ckan/controllers/package.py:'def autocomplete' should be removed.
* ckan/config/routes.py dataset/autocomplete should be removed."
1107 1340632612000000 icmurray autocomplete action has been deprecated in favour of /api/2/util/dataset/autocomplete : https://github.com/okfn/ckan/blob/master/ckan/controllers/package.py#L655
1108 1310128678000000 thejimmyg Now that we are using ckan.net as thedatahub.org we should go full steam ahead and create a brand new UI to reflect thedatahub.org's new role. See http://ckan.org/2011/07/08/ckan-vision-update/
1108 1311180204000000 thejimmyg See also comment on #1200 about using the pdeu theme.
1108 1314878772000000 zephod "First, here is a repo with the new DataHub theme:
https://bitbucket.org/okfn/ckanext-datahub-theme
We plan to integrate this into ckan-core, thereby replacing the current theme. "
1108 1315140879000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/9be1ae232ec3 cset:9be1ae232ec3
1109 1303862352000000 kindly It is fixed now in 1.3.4.1.
1109 1303916487000000 dread Close ticket?
1109 1304698621000000 dread Did we decide that this facility for storing non-strings via the API is a new feature, rather than a bug?
1109 1305123852000000 dread What's the status?
1109 1305124697000000 kindly I am happy this is fixed in cset:445fc04333dd.
1111 1304935139000000 lucychambers "CKAN FAQ now complete (Sections 1,2,5,6 - needs uploading onto site): http://ckan.okfnpad.org/FAQ
At CKAN Community Meetup it was decided that Sections 3 and 4 would form a separate contributor's manual. "
1112 1304358643000000 dread This is complete in the branch feature-1112-allow-search-all for release 1.3.5
1113 1304024611000000 kindly cset: 52a3fb230074
1114 1304085484000000 dread Done in cset:d7bd4b0f89de on default.
1115 1324054704000000 dread Authorization groups deprecated with the Group refactor #1477
1116 1304358664000000 dread This is complete in branch defect-1116-boolean-search-options for release-1.3.5
1117 1304277240000000 nils.toedtmann "Oh, it *does* depend against python-pastescript. Ignore.
For some reason it was not (completely/correctly) installed with ckanext-datanl on us4, but is a different issue then. Might be due to missing locale ""en_GB.utf8"" and dpkg-configrue failing."
1118 1311174062000000 thejimmyg "We haven't ever come across this. Perhaps if you could provide an example we can re-open it.
Cheers,
James
"
1119 1304104689000000 rgrp " * Merged ckanext-upload into ckanext-storage
* Got existing API systemin in ckanext-storage fully working (e.g. api section was broken -- maybe due to changed boto ...)
* API now supports GET requests and JSONP where appropriate
* Added auth form to API (could be useful for ajax form stuff going forward)
* Improved UI of upload including having ckan based file links (using redirects)
* Set uploaded-by field on uploaded files
* Checked on AWS And GS
Also did:
* Improved ofs in various ways including change to standard metadata handling and improved testing (see e.g. https://bitbucket.org/okfn/ofs/changeset/36fac29b7775)
* Tracked down [https://github.com/okfn/boto/commit/0d9635c8a9785c9b20b44ee93a0679c002961592 bug with boto and metadata updating on GS] and provided patch."
1121 1307358426000000 dread We don't need to support JSON extras at the moment.
1122 1304368075000000 dread @pudo I notice you tackling solr indexing of lists/tuples in ckanext-solr cset:ecc2d69df991 so have updated to use this on ckan.net and reindexed, but #1122 still seems an issue - any chance you can take a look?
1122 1305718009000000 dread This user is now storing his list as a string so is happy. But I'll leave this ticket open, for pudo to decide whether there is an issue with SOLR indexing tuples/lists, as per the previous comment.
1122 1306662099000000 pudo I don't think there's really a nice way to do this with the current solr indexing library, but I've commented out the list handling code which should really make handling this cleaner.
1122 1306747706000000 dread "Changing this to ""won't fix"" just to be clear"
1123 1307374691000000 dread "I'll change the architecture to ""all"" as per this article:
http://linuxreviews.org/man/deb-control/"
1123 1307790563000000 thejimmyg "This isn't necessarily a trivial thing. Let's get the build system for the packages stable before we start changing it all to support alternative architectures. Once the packaging is working well it would be trivial to switch those servers to amd64.
I'm sorry, but this isn't worth the investment in manpower at the moment."
1123 1307795508000000 rgrp You can't get small amazon instances as 64 bit and many people's own machines are 32 bit. How costly is packaging no-arch exactly (it must be more than a simple 'switch' in the build if you are saying this is costly :-) )
1123 1311863798000000 nils.toedtmann "Looks like the packages on apt.okfn.org are now architecture ""all"" - great.
To avoid future confusion, i change this ticket's solution to ""fixed""."
1124 1304601771000000 nils.toedtmann I installed the python-ckanext-solr package rom http://apt-alpha.ckan.org/datanl-dev onto us10/DataGM too, and it seems to work just fine, see http://datagm.test.ckan.net/package?q=manchester
1124 1323168132000000 thejimmyg "Solr is now properly supported in the ckan-1.5 repository.
"
1124 1323168156000000 thejimmyg Re-open this if it still affects DataGM.
1126 1304587078000000 dread Fixed in cset:61bb142a6b7c for ckan v1.3.5.
1127 1304602084000000 wwaites "The proposal is not comparable to PEP and DEP because the projects have vastly different requirements. The API stability needs of a programming language or an operating system (e.g. fundamental building blocks that you don't expect to change often or radically) are very different from a web application. The plone one is comparable.
The idea itself is a double-edged sword. It will promote stability which is good but can also tend towards rigidity and stagnation which is bad. Each added bit of bureaucracy and process means fewer people will be willing to collaborate or participate in improving the software.
Overall, -1.
"
1127 1304606094000000 dread Fixed links
1127 1305128000000000 thejimmyg This is great idea and it has taken on well. +1 from me. We can update/create new CREP policies as needed.
1127 1305458020000000 rgrp "I'm obviously a strong +1 but feel we should stick with the CEP name (we have PEP and DEP and CREP really does sound weird compared to CEP).
Also I think we may need to do a bit more to clarify when you just do a good ticket and when one does a CEP (looking at what has occurred already). I think this may be one reason to keep CEPs separate in drafting from trac as will clearly distinguish CEPs from standard tickets. "
1127 1305458489000000 rgrp one other thing: i think the plan is for CEPs to be in restructured text (for easy transfer to docs). This should be mentioned and the template in the wiki should be in ReST.
1128 1310567803000000 dread "Loader in ckanext/bin/loadscotland.py.
Done upload to ckan.net:
{{{
(pyenv-ckanext)dread@dread-laptop:~/hgroot/ckanext$ scotland_loader -k xyz -H http://ckan.net/api /home/dread/ckan-local/data/scotland-2010-10-18.csv
/home/dread/hgroot/pyenv-ckanext/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
Engine = entry_point.load()
No handlers could be found for logger ""vdm""
2011-07-13 15:29:17,983 INFO [ckanext.importlib.loader] Loading scottish_neighbourhood_statistics
2011-07-13 15:29:20,109 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:22,089 INFO [ckanext.importlib.loader] Loading traveline_scotland
2011-07-13 15:29:24,579 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:27,045 INFO [ckanext.importlib.loader] Loading sqa_qualifications
2011-07-13 15:29:30,869 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:32,561 INFO [ckanext.importlib.loader] Loading scotxed
2011-07-13 15:29:34,017 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:35,286 INFO [ckanext.importlib.loader] Loading iscjis_vocabularies
2011-07-13 15:29:36,592 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:37,782 INFO [ckanext.importlib.loader] Loading ecare_multi-agency_store_data_model_and_vocabulary
2011-07-13 15:29:39,043 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:40,267 INFO [ckanext.importlib.loader] Loading digital_archive_scottish_archive_network
2011-07-13 15:29:41,690 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:43,727 INFO [ckanext.importlib.loader] Loading the_scottish_register_of_tartans
2011-07-13 15:29:45,159 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:46,816 INFO [ckanext.importlib.loader] Loading scotlands_images
2011-07-13 15:29:48,563 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:49,983 INFO [ckanext.importlib.loader] Loading national_library_of_scotland
2011-07-13 15:29:51,709 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:53,122 INFO [ckanext.importlib.loader] Loading scottish_government_statistics
2011-07-13 15:29:55,857 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:57,118 INFO [ckanext.importlib.loader] Loading scotbis
2011-07-13 15:29:58,883 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:00,264 INFO [ckanext.importlib.loader] Loading historic_scotland
2011-07-13 15:30:02,086 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:03,609 INFO [ckanext.importlib.loader] Loading scheduled_monuments_in_scotland
2011-07-13 15:30:05,057 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:06,987 INFO [ckanext.importlib.loader] Loading scotlands_house_prices_service_consumer_information
2011-07-13 15:30:08,687 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:09,932 INFO [ckanext.importlib.loader] Loading air_quality_in_scotland
2011-07-13 15:30:11,278 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:12,637 INFO [ckanext.importlib.loader] Loading river_and_coastal_flooding_updates_for_scotland
2011-07-13 15:30:13,940 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:15,637 INFO [ckanext.importlib.loader] Loading public_contracts_scotland
2011-07-13 15:30:17,280 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:18,666 INFO [ckanext.importlib.loader] Loading scottish_government_consultations
2011-07-13 15:30:20,673 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:22,234 INFO [ckanext.importlib.loader] Loading scottish_public_health_observatory
2011-07-13 15:30:24,025 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:26,297 INFO [ckanext.importlib.loader] Loading scotlands_information_landscape
2011-07-13 15:30:28,308 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:29,758 INFO [ckanext.importlib.loader] Loading isd_scotland
2011-07-13 15:30:31,847 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:33,074 INFO [ckanext.importlib.loader] Loading scotlandspeople
2011-07-13 15:30:34,541 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:35,754 INFO [ckanext.importlib.loader] Loading general_register_office_for_scotland_statistics
2011-07-13 15:30:37,273 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:39,330 INFO [ckanext.importlib.loader] Loading baseline_scotland_-_groundwater_chemistry_data
2011-07-13 15:30:40,716 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:41,919 INFO [ckanext.importlib.loader] Loading the_centre_for_the_study_of_retailing_in_scotland_data_sets
2011-07-13 15:30:43,275 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:44,515 INFO [ckanext.importlib.loader] Loading growing_up_in_scotland
2011-07-13 15:30:46,215 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:47,582 INFO [ckanext.importlib.loader] Loading scottish_parliament_msp_allowances
2011-07-13 15:30:49,580 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:51,272 INFO [ckanext.importlib.loader] Loading scottish_government_ministerial_hospitality
2011-07-13 15:30:53,764 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:55,642 INFO [ckanext.importlib.loader] Loading scottish_government_expenditure_over_25000
2011-07-13 15:30:58,012 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:59,674 INFO [ckanext.importlib.loader] Loading scottish_local_government_financial_statistics
2011-07-13 15:31:01,869 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:03,382 INFO [ckanext.importlib.loader] Loading scottish_government_disclosure_of_high_earners
2011-07-13 15:31:07,086 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:11,493 INFO [ckanext.importlib.loader] Loading scotlands_places
2011-07-13 15:31:14,452 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:31,743 INFO [ckanext.importlib.loader] Loading scottish_national_heritage_facts
2011-07-13 15:31:36,223 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:37,830 INFO [ckanext.importlib.loader] Loading scottish_natural_spaces
2011-07-13 15:31:40,245 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:41,764 INFO [ckanext.importlib.loader] Loading scotland_national_address_gazetteer
2011-07-13 15:31:43,810 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:45,069 INFO [ckanext.importlib.loader] Loading scottish_environment_protection_agency
2011-07-13 15:31:47,975 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:49,525 INFO [ckanext.importlib.loader] Loading traffic_scotland
2011-07-13 15:31:52,396 INFO [ckanext.importlib.loader] ...creating package
}}}"
1128 1310568028000000 dread Note, I added a title column and made one or two minor corrections that I spotted - see attached CSV for the result.
1129 1305116419000000 thejimmyg "Great proposal, here's my thinking:
* The current VDM model (where we have things like a single package table which logs changes to a package_revision table) is very well tested and works well and we should only consider a radical change with good reason
* Although we are making a move from *model objects* being the central components to the *logic layer functions* being the central components, there is no need yet to have the database structures we store (for revisioning or otherwise) start to look like the dictized representation. The logic layer itself is designed to do that mapping.
* Since we do have this logic layer though, there is less need for ""magic"" SQLAlchemy objects to support dicts and lists (eg the stateful objects used to make a list of tags on a package behave like a list), a few simple (and easily debuggable) queries in the logic layer would work better.
In the longer term we may want to look at serialising more dictized-looking objects and that may lend itself to a more dictized changeset model or even a more dictized storage system (eg no-SQL) but for the time being we are not at that point.
I recommend the following:
* We continue to use the current default branch of VDM (not the changeset branch)
* We continue to treat the package table (and other non-revision tables) as the most recent revision (even though the *active* revision displayed in CKAN might well be in the revision tables because more recent changes haven't been moderated yet)
* We slowly stop using stateful lists and dicts in CKAN because we have move control with explicit queries in the logic layer
Sound good?
"
1129 1305139980000000 rgrp "It seems like this ticket is getting a bit confused with ticket:1076 (changes re vdm) :-)
Some general conclusions relevant to both:
* General feeling is not to change to changeset model
* I'm personally +0 on changeset model
* I would note main benefit is simplicity and orthogonality of changesets to core system. I also think cost of change is small.
* That said we can simplify current model as well.
* There seems some misunderstanding: change to have logic layer has almost nothing to do with being able to remove main stateful stuff in vdm. To be able to remove most of stateful stuff in vdm requires us to make some other changes (re foreign keys from revision objects to continuity)
* There are other simplifications we should make to vdm before embarking on this (e.g. move to SessionExtension from MapperExtension). This is easy as that work has been done in changeset branch and can be backported.
What I don't see in this CEP as yet is any discussion of the complexities around pending stuff (e.g. do we allow multiple pending, do new edits get shown current or latest pending). I'll try and comment more on this but a lot of this was in original vdm discussion last summer."
1129 1305209601000000 rgrp "Replying to [comment:2 rgrp]:
> It seems like this ticket is getting a bit confused with ticket:1076 (changes re vdm) :-)
I of course ;0 meant ticket:1077"
1129 1305212467000000 kindly "> * There seems some misunderstanding: change to have logic layer has almost nothing to do with being able to remove main stateful stuff in vdm. To be able to remove most of stateful stuff in vdm requires us to make some other changes (re foreign keys from revision objects to continuity)
The logic layer does not automatically help out, however it makes our life easier if we want to handle state ourselves. For example take package tags, if we remove the stateful_m2m properties and just use normal sqlalchemy relations. We will still want statefullness (i.e active, pending, deleted) on the package_tags table. We should update those on the table ourselves in the logic layer.
> * There are other simplifications we should make to vdm before embarking on this (e.g. move to SessionExtension from MapperExtension). This is easy as that work has been done in changeset branch and can be backported.
I agree but event thought the MapperExtension way is not great, it is very well field tested."
1129 1325268100000000 rgrp More UI work remains in #1141
1130 1340633514000000 icmurray Unassign for triage.
1131 1305537897000000 dread Fixed on branch release-v1.3.5 cset:3461ed6bab0c
1132 1307352675000000 dread "Yes, these are apparently search-related and so are skipped when testing under sqlite. See README.rst for more info.
Compare with this:
{{{
$ nosetests --ckan --with-pylons=test-core.ini ckan/tests/functional/test_authz.py
............................................
----------------------------------------------------------------------
Ran 44 tests in 69.287s
OK
}}}"
1133 1324057072000000 dread "Works for me - change occurs successfully.
{{{
(pyenv-ckan)dread@dread-laptop:~/gitroot/ckan$ paster rights add russianfan admin warandpeace
/home/dread/gitroot/pyenv-ckan/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
Engine = entry_point.load()
User russianfan -> is admin on -> Package warandpeace
(pyenv-ckan)dread@dread-laptop:~/gitroot/ckan$ paster rights
/home/dread/gitroot/pyenv-ckan/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
Engine = entry_point.load()
22 results
User visitor -> is editor on -> Package annakarenina
User logged_in -> is editor on -> Package annakarenina
User testsysadmin -> is admin on -> System system
User logged_in -> is editor on -> System system
User visitor -> is reader on -> Group david
User annafan -> is admin on -> Package annakarenina
AuthorizationGroup anauthzgroup -> is reader on -> Package warandpeace
User visitor -> is editor on -> Package warandpeace
AuthorizationGroup anauthzgroup -> is editor on -> Group david
User logged_in -> is editor on -> Package warandpeace
User logged_in -> is reader on -> Group david
User visitor -> is reader on -> Group roger
User logged_in -> is reader on -> Group roger
User russianfan -> is admin on -> Package warandpeace
User visitor -> is admin on -> Package warandpeace
AuthorizationGroup anotherauthzgroup -> is editor on -> AuthorizationGroup anauthzgroup
User russianfan -> is admin on -> Group roger
User russianfan -> is admin on -> Group david
AuthorizationGroup anauthzgroup -> is admin on -> AuthorizationGroup anotherauthzgroup
User visitor -> is anon_editor on -> System system
User visitor -> is admin on -> Group roger
AuthorizationGroup anauthzgroup -> is editor on -> System system
}}}
Tests would be nice, but it currently works, is only for admins and is therefore not as high priority as other other tests, such as front-end js."
1134 1305125228000000 thejimmyg "Great stuff.
Yes, I agree with all this, but as initial thoughts I suggest that for phase 1, the info() method returns one more key called ""form_config_interface"" which takes a string representing a the type of interface for the config field on the form. If the key is missing it is treated as ""None"".
The two possible values are:
None - the field is not present and must always be stored as NULL
Text - a single text field will be provided on the interface
Whatever the interface, the value built will always be stored in the config column of the table.
We then also provide a ""get_schema()"" method that returns a schema cablable of parsing the data submitted by the form and storing it as a single key named ""config"". ckanext-inspire will then add other functions to the schema for URL etc so that regardless of what the harvest plugin does, the key fields ckanext-harvest are always there.
This then allows:
* Customiseable config interfaces
* Customisable valiation
* A consistent place to store config in the database
Sound good?"
1134 1305623531000000 sebbacon "Agreed with James that we should consider *from the start* how to provide for a nice UI -- my worry is if we start out with people editing raw JSON we'll end up keeping that for years :)
"
1135 1305210134000000 rgrp "Work on this is largely complete in changeset branch: https://bitbucket.org/okfn/vdm/src/changeset/
Overview docs: https://bitbucket.org/okfn/vdm/src/changeset/doc/changeset.rst"
1135 1340632267000000 icmurray Re-assigned to kindly
1136 1305210925000000 rgrp "Most of this change has been implemented in the changeset branch [1] but could be backported reasonably easily to main.
[1]: https://bitbucket.org/okfn/vdm/src/changeset/"
1136 1305216218000000 kindly This would be nice but it is not necessary. The current mapper implementation may be nice but not is battle tested..
1136 1305217553000000 kindly "Replying to [comment:2 kindly]:
> This would be nice but it is not necessary. The current mapper implementation may be nice but not is battle tested..
I meant the mapper extension ""IS"" battle tested. "
1136 1340632980000000 icmurray Re-assigned to @kindly.
1137 1305211672000000 rgrp "Most of this change has been implemented in the changeset branch [1] but could be backported reasonably easily to main.
[1]: https://bitbucket.org/okfn/vdm/src/changeset/"
1137 1305217449000000 kindly "I am not sure this needs to be done. I think we should keep the continuity object always in the table, even if it is deleted. The querying should be done through the logic layer so the deleted state should not be an issue.
The clients should be entirely state aware.
The only thing that needs to be done is to remove all statefullness of relations. These are the only things that are complicated.
This would make vdm just a simple copy on write mechanism, with the client controlling the state."
1138 1316965357000000 rgrp Cannot see this issue any more on default (as deployed on e.g. test).
1140 1312491320000000 kindly "fixed cset:987da68ea4f6
The package group table needed to trigger a reindex of the package.
"
1140 1312537257000000 dread Broken since CKAN v1.1. I have added this fix to v1.4.2 and v1.4.3 releases.
1141 1308828928000000 dread "I've merged this into default in preparation for release 1.4.2, but there may be a couple of useful additions that could be done on this or another branch/ticket.
Getting a package at a specified revision is really useful. This can be used:
* links in the package history (#103) (while in there, the 'moderated' flag should be shown in the package history too if not already)
* should also be added to the package API (as a replacement for the AJAX call?)
The history_ajax call looks like an almost duplicate of the Package Revision API. Can these be merged and put just in the API?
"
1141 1318862183000000 dread What's the status of this now? The ckanext-moderatededits README still says you need a branch of ckan. Can you say which version of CKAN is required instead? I say we close this if ajax call is now in a better place in the API. (I expect it is now, with the logic layer).
1141 1318862773000000 johnglover Well, it runs on datacatalogs.org with the current default so CKAN 1.5, but it needs quite a bit of updating to work with all of the new 1.5 UI changes, and as it has been a low priority for quite a while I haven't scheduled time to work on it. Really I would need to spend a few days on it to tidy it up for standard 1.5.
1141 1325352507000000 dread Closing, with remaining tidy up work for newer CKAN versions split off into new ticket: #1604
1142 1305880901000000 sebbacon "The guys at Wikimedia also produced this spreadsheet including notable
users and features of other software. Could be useful.:
https://spreadsheets2.google.com/ccc?authkey=CLWu1cgP&hl=en&key=tYiwc9kUVLLlALOQEzFqujw&hl=en&authkey=CLWu1cgP#gid=0
"
1142 1305881244000000 lucychambers "Replying to [ticket:1142 rgrp]:
> Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927
>
> * CKAN 1-page overview (for enterprise and for data hackers)
> * Administrator's Guide (including install)
> * Extensions Guide
> * Separate CKAN.net info from Software Documentation (?)
>
> == Minor Items ==
>
* Include guide on how to configure local filesystem storage, instead of Google storage. (http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001235.html)
"
1142 1308326118000000 annapowellsmith "List of new docs tasks agreed by APS and RGRP:
* Tutorial on the wiki (long-term): Integration with Wordpress and Drupal and Javascript
* Overview of domain model both for average users and for devs
* 2-page glossy brochure
* Vision for CKAN http://notebook.okfn.org/2011/04/27/data-hubs-data-management-systems-and-ckan/
* Comparison : use http://wiki.ckan.net/Related_Software as starting point - build on wiki - Socrata and OGDI
* Renaming: separation between ckan.org and ckan.net
* More documentation in the code ...
* Config options generated from code or move this to WUI ...
See also proposed overhaul at http://wiki.ckan.net/Documentation_Plans"
1142 1313775665000000 rgrp This was completed 4 weeks ago and can now be marked as complete. See http://docs.ckan.org/ and http://wiki.ckan.net/ and http://ckan.org/
1143 1307024681000000 dread "All done in ckanext-stats cset:bc54790cacd4 (apart from ""fix x axis to start at first revision"" which was not a problem after all)"
1148 1305969925000000 kindly complete cset:96a43c9d8bd7
1149 1305971672000000 kindly cset:b1634d405066
1149 1306090663000000 kindly "Failed dgu tests due to as_dict not working on deleted objects.
fixed by cset:c9b9cf513e44"
1150 1311178163000000 thejimmyg "Hi John,
Can you please check that the new webstore+preview works correctly with this one too please?
Cheers,
James"
1150 1311774141000000 johnglover Unicode data should be fine in the new datapreview code, for example: http://test.ckan.net/package/afghanistan-election-data
1151 1330908188000000 rgrp Not sure we will be able to fix this easily as we need a dataproxy for general case. Think it better to focus on the case where data in DataStore and we use Recline. As such closing as wontfix for the present.
1152 1307352165000000 amercader Bumping to ckan-v1.5-sprint-3 and updating the CC email addresses so people actually get any updates.
1154 1314120497000000 johnglover Fixed in core (branch feature-1275-solr-search).
1154 1314177477000000 dread "Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?)
I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request. e.g.
{{{
Module ckan.controllers.home:29 in index
<< query = query_for(model.Package)
query.run(query='*:*', facet_by=g.facets,
limit=0, offset=0, username=c.user)
c.facets = query.facets
c.fields = []
>> limit=0, offset=0, username=c.user)
Module ckan.lib.search.common:115 in run
<< self.query = QueryParser(query, terms, fields)
self.query.validate()
self._run()
self._format_results()
return {'results': self.results, 'count': self.count}
>> self._run()
Module ckanext.solr.backend:63 in _run
<< # this wrapping will be caught further up in the WUI.
log.exception(e)
raise SearchError(e)
finally:
conn.close()
>> raise SearchError(e)
SearchError: [Errno 111] Connection refused
}}} (recently from nl.ckan.net)"
1154 1314178325000000 johnglover "> Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any
> packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a
> useful thing to do? Without search, perhaps CKAN should simply not start?)
When discussing the ticket we agreed that we should allow ckan to start without search. It is possible that people simply want to list/view information without searching for it and we decided not to prevent them from doing so.
> I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this
> time pudo and I get exception emails saying connection failed for request.
Ok, other options then:
* if an error occurs during search, return 0 results
* if an error occurs during search, redirect to a generic 'search is unavailable' page
* if an error occurs during search, retry the query a certain number of times before giving up and doing one of the above.
What do you suggest?"
1154 1314179290000000 dread "Search requests - option one seems like a bad user experience, and option three seems too complicated. I should do option 2, explain it may well be a temporary problem, and let the user press reload, as per any normal web request.
A more difficult problem though is what to do about a failed search indexing. Because of the try/except catch we've had until the patch yesterday, we get no exception emails for this, but I'm worried that this has caused packages to not be indexed, and hence are essentially lost in CKAN (unless someone realises there is a problem and does a reindex at some point).
BTW I added paster command 'search-index check' for looking at this problem, but I don't think it was ever added to solr. Is it easy to add get_all_entity_ids for the purpose of checking for this problem?"
1154 1314191496000000 johnglover "Ok I'll implement option 2 for search query issues.
Yes it is easy to get a list of IDs from Solr so I'll have a look at that command and make sure that it works. I think a case could be made for not allowing a package to be added if it cannot be indexed however, and making the user simply retry the request."
1154 1314192324000000 dread Yes, forcing a retry may be possible, although I guess on these occasions you'd be looking at rolling back the creation of the package object too, which is not really in the spirit of the notifications system at present. This is why we had the queue for solr for occasions like this - is it here no longer?
1154 1314192753000000 johnglover Not in core, there is a queue feature in the extension still which I could add to core. I think though it was a case of using either the queue or synchronous search, I don't think it would automatically fall back to the queue if there was an indexing problem, but we could look at that. Was there not talk of changing the queue system recently though? There are also no tests for the Solr queue so that would take a bit of work as well.
1154 1314193633000000 johnglover RE search queries failing: Just having a look through the controller code and the package search controller and the search api both perform option 2 already, so I'll just update the home controller.
1154 1314195684000000 dread "Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception, this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this tally with what you mentioned in the last comment or is what's there better?
Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo know if it is?). In this case we should be clear that it is not what it says it is: ""This extension adds support for package search using Apache Solr to CKAN"" and maybe simply mark it as deprecated. Pudo what do you think?"
1154 1314196683000000 johnglover "> Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception,
> this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this
> tally with what you mentioned in the last comment or is what's there better?
This sounds good to me, definitely better than what is currently there.
> Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo
> know if it is?).
I'm not actually sure if the queue works or not, but if it does I'll have to write some tests for it before we could add it to core. A working queue system would be useful generally however, as there are other things that it would be useful for (such as QA), but yes, going forward the Solr extension should really be deprecated."
1154 1314287519000000 johnglover "All attempts to index packages should now throw a SearchIndexError if Solr is unavailable and the create/update will be rolled back. These errors are caught in the main package controller (redirecting to an error page) and the API controller. I have added tests for both controllers.
Also updated the paster 'search-index check' command to work with solr."
1155 1306773174000000 pudo "Now implemented based on CSV export available on the catalogue page:
* http://data.london.gov.uk/catalogue"
1156 1306855111000000 pudo Both are now implemented but may need further work to adhere to developing conventions wrt to location etc.
1159 1307615133000000 pudo Rudimentary RDFa has been added with seeAlso to API.
1161 1306408026000000 amercader Duplicate of #1157
1164 1308647224000000 amercader Done. See it in action here: http://publicdata.eu/map
1165 1306413953000000 nils.toedtmann this is similar, but not equal to #1096
1165 1307444733000000 nils.toedtmann See also http://ckan.okfnpad.org/multisite
1166 1306847326000000 thejimmyg Also, rather than INSPIRE=true we should probably have information about the harvesting mechanism and the harvest object type? This also relates to our use of the kind field on the resource.
1166 1317381932000000 amercader Fixed on d5bee3de9957
1167 1306856992000000 sebbacon "Yes to all the above.
Further question: which extensions should be installed as standard? I would argue:
* ckanext-stats
* ckanext-googleanalytics
* ckanext-disqus
* ckanext-solr (implies installing a running solr, too...)
* ckanext-follower
* ckanext-admin
Note that googleanalytics and disqus both require user accounts set up at the corresponding service. Not sure how we would handle this in the instance setup.
"
1167 1306857142000000 sebbacon Another thing to consider: how do we ensure we keep a single, up-to-date version available? (I'm thinking wrt developer / sysadmin workflow)
1167 1306857750000000 thejimmyg "Creating the AMI is really just the icing on the cake. The hard bit is getting the packaging right so that's where we need to concentrate first. Once the packaging works we just make sure apt-get upgrade is run regularly.
+1 to the list above.
I also agree we should install solr by default. Who has actually done a CKAN solr install? I haven't but together with whoever has I can try to package it up.
Wrt to usernames and passwords, I'll look into how dpkg manages those blue pop-up screens for entering configuration options."
1167 1307358318000000 dread "debconf seems the preferred way to question the user post-install and run setup scripts. It seems a bit of a faff.
The person that is able to spin up an EC2 instance is no doubt technically able enough to follow a couple of instructions in README.txt saying edit /etc/ckan//.ini and add your Google Analytics credentials. So I don't think this ticket is the driver for a debconf setup."
1167 1311178516000000 thejimmyg "I've now written a 20 page guide on using CKAN on Amazon EC2. I haven't explicitly created an AMI, but I don't think we should do that anyway for a few reasons:
* it is fiddly
* there is no advantage to the user as CKAN installation is so easy and so well documented
* we don't want to have to maintain the image going forward
* we might not want to promote a non-open approach
The guide will form part of Anna PS's work on the overall documentation but you can also see the original version here:
https://bitbucket.org/okfn/ckan-debs-public/src/489a5ecf369f/ec2/"
1168 1310134901000000 thejimmyg This is now working on dgu-buildbot.okfn.org. The code that manages it is partly the repo.py script from http://hg.3aims.com/public/BuildKit/ and partly the fab code in https://bitbucket.org/okfn/buildbot-scripts
1168 1310396585000000 dread This might run manually on the us7, but the buildbot is not running these automatically or any sort of testing, which is what this ticket is about.
1168 1325866142000000 thejimmyg "Quick update... we now have:
* An automated way to build the deb packages
* An automated way to install them into a fresh virtual machine
We don't currently run them via buildbot or have smoke tests.
"
1169 1306754833000000 dread Done on default. Will appear on ckan.net next time it is branched and updated.
1170 1306864422000000 dread Done in cset:4d2d0857281f on default
1171 1347358705000000 ross "I have implemented this on the datahub, but this change likely needs to be made to CKAN core. Probably as part of 2.0.
The format on tdh is currently:
${TITLE}. ${AUTHOR}.
Retrieved ${UTC time}.
${DATASET LINK}
Best discuss and agree on what format should be supported. Mark, I've assigned this to you as it is tagged academia ;)
"
1172 1312991332000000 dread I've started this on branch defect-1172-exceptions
1172 1314118281000000 dread "Is in cset:47225f03e2af and merged to default for 1.4.4.
It now barfs instead of failing silently:
* if the ckan config file has any errors
* search indexing error
There are some other ""except Exception"" statements "
1172 1314118883000000 dread "I took out ""raise"" by mistake in ckan/lib/search/worker.py:26 but John has added it in in the corresponding place in ckan/lib/search/__init__.py on branch feature-1275-solr-search anyway."
1173 1307000249000000 pudo "I propose we do a bit of frameworkiness here: set up a two-part content type registry and refactor the API controller to use it.
The idea would be to do the following things:
1. Create a registry mapping format extensions to mime types: RepresentationRegistry.add_format('json', 'application/json')
2. Create a registry of (mime types, entity types) -> converter functions that yield an appropriate representation of the entity.
3. Hook this into the _finish method of the API controller based on Accept: handling
4. Add support for /api/rest/ENTITY/NAME.{format} and /ENTITY/NAME.{format} to routing, use registry from (1) to rewrite accept headers.
5. Register converters in load_environment or via IConfigurable plugins
6. Document
What do you think?
"
1173 1307615271000000 pudo Now available in ckanext-rdf and linked to from WUI.
1174 1307615200000000 pudo "wontfix as thejimmyg wants to turn REST API into RPC API and this doesn't fit in.
Not an excellent plan in my idea but this is to be discussed elsewhere. "
1175 1307350866000000 dread Flavio, can you post here the exception that you get?
1175 1307372833000000 fccoelho Thats the thing, I don't get any error message, no even in the logs. But I can't even load a page.
1175 1314030683000000 dread Hi Flavio, is this still a concern? I'd like to close this ticket unless you can provide details of the error from your logs or exception email.
1175 1325355170000000 dread This error (which will be in the apache or ckan log) must be unrelated. Stats extension runs fine elsewhere.
1176 1310557852000000 dread This is occurring several times a day now - James can you suggest what's happening?
1176 1314273012000000 dread "This occurs when you browse {{{/error/document}}} directly, rather than being directed there on error. In this case there is no 'original_response'.
Fixed in cset:c67d5934fb0b for ckan 1.4.3 and merged to default."
1177 1307360643000000 rgrp What revision of the CKAN code are you using (do hg id to get revision if installing from source). Have you modified the package template in any way? Have you restarted apache after upgrading?
1177 1307361157000000 fccoelho "to upgrade I used the method describeb on bitbucket, after upgrading pip-requirements:
pip -E pyenv install -r pip-requirements.txt
after that I did a
paster --plugin ckan db upgrade --config ckan.emap.fgv.br.ini
the revision id on my pyenv/src/ckan is:
5730c79db461+ tip
yes, I have restarted apached after upgrading"
1177 1307365496000000 rgrp "The issue here is that the line should never be reached because g.has_commenting should be False: https://bitbucket.org/okfn/ckan/src/5730c79db461/ckan/lib/app_globals.py
@Flavio: have you modded your code in any way? is self.has_commenting False in your app_globals.py
A quick fix would be to remove the template lines calling to remove the lines in the template:
{{{
${h.subnav_link(c, h.icon('comments') + _('Comments & Questions'), controller='package', action='comments', id=c.pkg.name)}
}}}"
1177 1307366621000000 fccoelho "I have not modded the code in any way...
I'll look for those lines in the template.which template is it? read.html does not have a line such as this."
1177 1307367354000000 fccoelho OK I did an fgrep on all package template and found the line on layout.html, I'll comment the block and test.
1177 1307367923000000 fccoelho "Ok it fixed it!
thanks a lot!!
Is there any side effects to the removal of this line, I should know of?"
1177 1307374897000000 dread "I think rgrp was suggesting you have mistakenly set has_commenting to ""True"" in app_globals or elsewhere. But either way, this should have no side-effects."
1178 1308045351000000 rgrp Fixed. This was due to incorrect config of bucket option.
1180 1307544223000000 dread "Both issues solved using a whitelist on anchor tags.
Second issue was a link with unicode expression of the quote. e.g. somelink"
1181 1307532816000000 dread Fixed in ckan default in cset:72f7d48d7f31. Have left Package url and Resource url links though, as these are the key links we want google to see. So we need to check these carefully for spam.
1183 1311181719000000 thejimmyg "Hi John,
Can you have a look at this one too please?
Cheers,
James
"
1183 1311771069000000 johnglover Fixed in new datapreview code (currently running on the test.ckan.net), which now gets the preview data from a webstore instead of the dataproxy API. It also only now shows a preview button for resources that were successfully archived and saved in the webstore.
1186 1310556519000000 dread This was done by pudo in branch feature-1198-mailer and went into default, lined up for release 1.4.2.
1187 1308143605000000 dread "Also stopped creating revision for user edits.
Done in cset:795ccd6405ba for 1.4.1 release"
1189 1323173227000000 thejimmyg Closing this ticket in line with ticketing policy since it is over 6 months old. If someone would like to develop an extension that uses spideroak as a back end we can look at it again.
1190 1320143592000000 rgrp A very substantial portion of this got done as part of 0.5 including the webstore #1208 and the majority of automated conversion #1398. #1432 is currently deferred so we're going to close this ticket on basis that item (3) is now out of scope.
1191 1308650930000000 dread Done in cset:c3822feffb38 for 1.4.1 release
1192 1308337652000000 dread "I really think forms are a key asset of CKAN, and keeping the docs about customising and integrating them is important - i.e. forms.rst and forms-integration.rst. There are ideas being suggested for replacing bits of front-end, but forms hard to do, and David's new system is a significant offering on top of the CKAN back-end, so it seems crazy to me to abandon it.
forms.rst has been updated by David Raznick in the release_v1.4.1 branch.
i18n.rst - include here the creation of the pot file, deploying a translation, transifex admin
load-testing.rst - I think performance should be mentioned in the glossy guide (some sort of basic metrics of speed profile, to return web pages, run searches and API operations, for 100 or 1000 packages, on a given spec machine) and have it based on a report showing how we measured it. The report would look a bit like this file, and it might do as a stub, but it seems pretty lacking.
distributed.rst - agreed, remove.
design.rst - agreed, delete.
loaders.rst / loader_scripts.rst - yes move to wiki
feeds.rst - this should be split into a tutorial for the wiki and the reference page left in sphinx
importer.rst - this should be moved to ckanext-importer/doc
admin.html - this has already gone in 1.4.1
authorization.rst - This is a bit jumbled. We're talking about changes to the code, so maybe not worth working hard to make this good, so I suggest leaving it, but delete the boring section ""Requirements and Use Cases"".
deb.rst - this is an administrator task, so should stay why move this to the wiki?
buildbot.rst - ok to stay here?
vm.rst - I assume there's no problem with this"
1192 1310135468000000 thejimmyg This is now a child ticket of #1142
1192 1312191462000000 rgrp Marking this as closed as Anna has completed and this is now deployed at http://docs.ckan.org/.
1193 1309768960000000 kindly fixed cset:87d6140e06ad
1194 1317043212000000 dread "Suggest changing this to ""Welcome, Rob"" rather than ""Welcome back, Rob"", avoiding the problem."
1194 1317053674000000 dread Done in cset:49e2c05e69ea
1194 1317053688000000 dread Will go into ckan 1.5.
1195 1311332049000000 dread Also this one: http://ckan.net/storage/f/file/
1195 1311841763000000 rgrp Fixed in https://bitbucket.org/okfn/ckanext-storage/changeset/49734285f1ef
1196 1308743857000000 dread "Reason here:
http://www.freewisdom.org/projects/python-markdown/Tickets/000018
Using the suggested workaround. Fixed on default for 1.4.2 in cset:715212cd220c"
1197 1340010256000000 toby "personally I'm against these for 4 reasons
a) The site should be self explanatory and if it is not this is a failure of design
b) a cookie would be needed to prevent these happening all the time - cookies are bad as they interfere with caching proxies.
c) I find stuff like this patronising/annoying for the user
d) They are a maintenance burden for ckan as we do not have a single instance
Therefore I'd suggest closing this as a won't fix - I agree this is just a personal view point and therefore am not going to actually close the ticket but just record my opinion."
1197 1340018903000000 seanh "Hi Toby, yes I think I agree with you, it reminds me of when I install new Android apps on my phone, some of them start off by bothering me with some sort of guided tour or long instructional text, I hate this and always skip through without reading it, expecting to be able to figure out the interface for myself.
The one thing that something like this could be useful is, rather than showing someone how to use the site, showing them what the site is capable of so they can assess whether it meets their needs/is something they're interested in. However, we already have a good guided tour of this type on ckan.org/features, those pages are really pretty good.
Close this as wontfix?"
1197 1340019675000000 toby "Thinking horrible thoughts, there was a rumour of a ckan tour, for something like that this would be useful.
Not sure where that plan is at? if there is a ticket for it maybe link to this one and close?"
1197 1340020357000000 seanh "AFAIK there isn't a ticket for that, it was something originally requested as part of the demo site, but since the devs think it's a horrible idea I don't think it's going to happen, easiest way to make a CKAN tour would be a series of static pages like the one we already have on ckan.org, I think that combined with a real functioning demo site probably meets the requirement.
Anyway, closing this as wontfix"
1198 1308821489000000 pudo "This ticket is essential I think. I would propose we should be a bit more radical here, though:
* Introduce an ""Agent"" SQLAlchemy object and derive both publishers and users via single table inheritance.
* Make agent the central object of authorization, e.g. by introducing c.agencies including the user account, visitor, logged_in, etc.
* Abolish AuthorizationGroups in favor of publishers. Yes, they are different but the use case for what AuthzGroups do and Publishers don't is just nerdery.
* Restructure URLs to be // rather than /package/ and have / be a useful list of managed resources rather than a revision list (users current home page).
* Make Agents a primary facet for all kinds of navigation.
Of course this would be done incrementally but I think we should agree on some such scheme as a marching direction. "
1198 1308824870000000 dread "I think it is a good idea to allow both users and publishers to be the object of authorization. The user_object_role table and all logic surrounding it would be simpler.
I deliberately didn't suggest any detail on the authz front because I know other related changes are afoot so it would be useful for David Raznick to contribute here.
I really like the url ideas - more like github. / having a list of packages and a search box, plus a link to the latest revisions would be great. /package/ would usefully redirect to //. I do think this should be split off into a separate ticket though. What do you think?"
1199 1308822557000000 rgrp Not sure we need exposure this via the API (at least at this point) as things like auto mass emailing could be in done in separate component (that talks to CKAN).
1199 1308922389000000 pudo merged to default with password reset forms
1199 1310555589000000 dread "This was done in branch feature-1198-mailer. No tests - ticketed here: #1222.
Additional feature in this branch 'password reset' was ticket #1186."
1200 1311180218000000 thejimmyg This is a duplicate of #1108. Let's have the discussion there please.
1202 1314030352000000 dread "The CKAN package page links to datapkg at: http://blog.okfn.org/2010/02/23/introducing-datapkg/ (it has always been this link).
I think Rolf is referring to the three links to knowledgeforge.net in the blog post itself. These need to be edited by Rufus. Note datapkg has moved again to github."
1202 1315821804000000 rgrp This now irrelevant and relevant content has gone from thedatahub theme.
1204 1311179980000000 thejimmyg "Sorry, is this a CKAN issue or a datacatalogs.org one? If it is datacatalogs.org do you mean a catalog rather than a package? If so, can you give an example of the sort of change you've made?
Marking as invalid until we get more information."
1205 1309768720000000 kindly "fixed in default cset:5e2070688e54
The upgrade now works locally, so the upgrade should work now."
1205 1310568631000000 dread This worked - cheers David.
1206 1309450216000000 dread "Fixed with cset:149be76faabc on default branch, aiming for ckan 1.4.2.
Thanks for help from Eric Lemoine."
1207 1311325343000000 dread This has been fixed.
1208 1310124599000000 thejimmyg Rufus and Friedrich are working on this at the moment so putting into the current sprint. I'm considering this as preliminary investigation to inform the wider project that David Raznick is leading on.
1208 1311525306000000 rgrp Propose this ticket can be closed as webstore is now functional and being used. Can open new tickets for specific improvements / adaptations as the need arises.
1208 1312191646000000 pudo "This should be closed:
* https://github.com/okfn/webstore - See README for details
* Python client lib: https://github.com/okfn/webstore-client
* http://webstore.thedatahub.org/ - Demo install
* http://wiki.ckan.net/Webstore - Overview
"
1209 1310571715000000 dread Was fixed in cset:d959a70a19ea. Bug was introduced and fixed before a release took place.
1210 1309974781000000 dread Fix in ckan for AssertionError in cset:8f6ba8ef63f3 lined up for CKAN release 1.4.2.
1211 1315910650000000 dread I've listed what was checked into this branch 6 weeks ago and is going into release 1.4.3. Is this ticket ready to close now or are there future plans?
1211 1315948703000000 rgrp Let's close and re-open (or open a new ticket) for future work.
1214 1309982828000000 dread "I'm working through these on branch defect-1214-api-improvements.
""Server returns text/html for errors"" - now fixed in cset:0a533a8da3df"
1214 1310041083000000 dread "Aron,
I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: ""Should be 405 Method Not Allowed?"" DELETE should be allowed. Or is that a response you're getting?
David"
1214 1310041605000000 dread """Tag returned as a JSON object when updating but as a string when requesting.""
I'm not sure what you mean.
http://test.ckan.net/api/2/rest/package/reference-example
gives tags: {{{""tags"": [""country-uk"", ""example-package""],}}}
Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix.
http://test.ckan.net/api/2/rest/tag/country-uk"
1214 1310042268000000 dread "Fixed tag entity returning package names for apiv2 - it now gives IDs.
The problem mentioned about ""Extras"" content - I'm not sure what's meant here. Package extras are just strings. You might choose to store a package name in an extra, but there is no magic to convert it to a package id should you retrieve the package via apiv2. Is this just a misunderstanding or have I missed your point?"
1214 1310044531000000 dread "Deleting an extra using 'null' works for me:
{{{
$ curl -d '{""name"":""dtest2"", ""extras"":{""1"":""1"", ""2"":""2"", ""3"":""3""}}' http://test.ckan.net/api/rest/package -H ""Authorization: tester""
...
$ curl -d '{""name"":""dtest2"", ""extras"":{""1"":""1"", ""2"":""2"", ""3"":null}}' http://test.ckan.net/api/rest/package -H ""Authorization: tester""
{""maintainer"": null, ""name"": ""dtest2"", ""relationships_as_subject"": [], ""author"": null, ""url"": null, ""relationships_as_object"": [], ""notes"": null, ""title"": ""dtest2"", ""maintainer_email"": null, ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""author_email"": null, ""state"": ""active"", ""version"": null, ""groups"": [], ""license_id"": null, ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""tags"": [], ""id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""resources"": [], ""extras"": [{""state"": ""active"", ""value"": ""\""1\"""", ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""package_id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""key"": ""1"", ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""id"": ""d1937073-7bfc-48c5-b6ff-b00d90b451ae""}, {""state"": ""active"", ""value"": ""\""2\"""", ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""package_id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""key"": ""2"", ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""id"": ""8147886f-9769-440c-8c35-b7d6a2f46de7""}]}
}}}
"
1214 1310045026000000 dread "Ok I see the problem with extras. When you create a package you get back 'simple' extras:
{{{
$ curl -d '{""name"":""dtest3"", ""extras"":{""1"":""1""}, ""tags"":[""australia"", ""barbeque""]}' http://test.ckan.net/api/rest/package -H ""Authorization: tester"" --trace log
{""maintainer"": null, ""maintainer_email"": null, ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""metadata_created"": ""2007-04-10T21:19:38"", ""relationships"": [], ""metadata_modified"": ""2011-07-07T13:06:06.228718"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [""australia"", ""barbeque""], ""groups"": [], ""name"": ""dtest3"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""ckan_url"": ""http://test.ckan.net/package/dtest3"", ""notes"": null, ""title"": ""dtest3"", ""ratings_average"": null, ""extras"": {""1"": ""1""}, ""ratings_count"": 0, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d
}}}
i.e. ""extras"": {""1"": ""1""}
and that's what you get if you GET the package:
{{{
$ curl http://test.ckan.net/api/rest/package/dtest3
{""maintainer"": null, ""maintainer_email"": null, ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""metadata_created"": ""2007-04-10T21:19:38"", ""relationships"": [], ""metadata_modified"": ""2011-07-07T13:06:06.228718"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [""australia"", ""barbeque""], ""groups"": [], ""name"": ""dtest3"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""ckan_url"": ""http://test.ckan.net/package/dtest3"", ""notes"": null, ""title"": ""dtest3"", ""ratings_average"": null, ""extras"": {""1"": ""1""}, ""ratings_count"": 0, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d11""}
}}}
but if you then edit it you get them expressed much more verbosely:
{{{
$ curl -d '{""name"":""dtest3"", ""extras"":{""1"":""2""}, ""tags"":[""australia"", ""barbeque""]}' http://test.ckan.net/api/rest/package/dtest3 -H ""Authorization: tester"" --trace log
{""maintainer"": null, ""name"": ""dtest3"", ""relationships_as_subject"": [], ""author"": null, ""url"": null, ""relationships_as_object"": [], ""notes"": null, ""title"": ""dtest3"", ""maintainer_email"": null, ""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""author_email"": null, ""state"": ""active"", ""version"": null, ""groups"": [], ""license_id"": null, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d11"", ""tags"": [{""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""state"": ""active"", ""id"": ""4aa0e776-ac2a-4a8b-82ba-d80237d35596"", ""name"": ""australia""}, {""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""state"": ""active"", ""id"": ""e890354d-e170-44c1-94b1-f0d8b38a49fc"", ""name"": ""barbeque""}], ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""resources"": [], ""extras"": [{""state"": ""active"", ""value"": ""\""2\"""", ""revision_timestamp"": ""2011-07-07T13:21:02.220368"", ""package_id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""key"": ""1"", ""revision_id"": ""61703821-a2f1-41f8-8f39-71130c5e6c1b"", ""id"": ""87d7d4fc-7eb7-4760-b199-e460ce505632""}]}
}}}
I'm going to ask David Raznick about what's best here. It would be good to make them uniform."
1214 1310053225000000 aron "Replying to [comment:2 dread]:
> Aron,
> I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: ""Should be 405 Method Not Allowed?"" DELETE should be allowed. Or is that a response you're getting?
> David
If I send a HEAD request to the server eg.
{{{
curl -i -XHEAD http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf
}}}
I get the following in the Access-Control-Allow-Methods header
{{{
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS
}}}
I've just realised that the ""Access-Control"" headers are for cross origin browser requests so I think you can ignore my comments about the correct response being 405 Method Not Allowed but perhaps DELETE should be added to the Access-Control-Allow-Methods list?
"
1214 1310054305000000 aron "Replying to [comment:3 dread]:
> ""Tag returned as a JSON object when updating but as a string when requesting.""
> I'm not sure what you mean.
>
> http://test.ckan.net/api/2/rest/package/reference-example
> gives tags: {{{""tags"": [""country-uk"", ""example-package""],}}}
I'm getting back objects for each tag in the response body from a PUT request. For example ""country-uk"" would be something like.
{{{
{
""id"" : ""600dc72e-6127-4704-b801-bee00474ec0c"",
""name"" : ""country-uk"",
""revision_timestamp"" : ""2011-07-06T09:09:21.578034"",
""state"" : ""active""
}
}}}
Hopefully this gist will illustrate the issue.
> Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix.
> http://test.ckan.net/api/2/rest/tag/country-uk
That was the fifth point on the list.
"
1214 1310054863000000 aron "Replying to [comment:5 dread]:
> Deleting an extra using 'null' works for me:
Hmm, it works for me too when there are other keys remaining in the ""extras"" object. However I can't seem to delete the last one. Take the following example.
{{{
curl -i -XPUT -d'{""extras"": {""Tester"": null}}' -H""X-CKAN-Type: application/json"" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf
}}}
Gives:
{{{
{ ""author"" : """",
""author_email"" : """",
""extras"" : [ { ""id"" : ""88cc50ca-9e29-4499-a542-09d364f5f64f"",
""key"" : ""Tester"",
""package_id"" : ""ec9cb930-d15f-441f-a1e1-36f4d5df19bf"",
""revision_id"" : ""e5c3ca9c-1dae-4a86-837f-b8c19ac31964"",
""revision_timestamp"" : ""2011-07-07T16:01:05.696025"",
""state"" : ""active"",
""value"" : ""\""Test Value\""""
} ],
""groups"" : [ ],
""id"" : ""ec9cb930-d15f-441f-a1e1-36f4d5df19bf"",
""license_id"" : """",
""maintainer"" : ""aron"",
""maintainer_email"" : """",
""name"" : ""my-test-package"",
""notes"" : ""Heading\r\n===\r\nThis _is_ some text"",
""relationships_as_object"" : [ ],
""relationships_as_subject"" : [ ],
""resources"" : [ ],
""revision_id"" : ""688d33fb-5629-4ab3-9f59-a649fc7caa00"",
""revision_timestamp"" : ""2011-07-06T10:37:50.182894"",
""state"" : ""active"",
""tags"" : [ { ""id"" : ""600dc72e-6127-4704-b801-bee00474ec0c"",
""name"" : ""test-tag"",
""revision_timestamp"" : ""2011-07-06T09:09:21.578034"",
""state"" : ""active""
} ],
""title"" : ""My Test Package"",
""url"" : """",
""version"" : """"
}
}}}
"
1214 1310055169000000 aron "Another edge case that popped up while verifying the above issues. A PUT request without a body throws a 500 Internal Sever Error
{{{
curl -i -XPUT -H""X-CKAN-API-KEY: tester"" -H""Content-Type: application/json"" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf
}}}
"
1214 1310057942000000 aron "One more for today. Please do let me know if I should be filing these as separate tickets?
For some reason performing a search query for packages with an underscore ""_ "" as a query string key fails to return any results.
{{{
curl ""http://test.ckan.net/api/2/search/package?q=osm""
}}}
Gives me 4 results.
{{{
curl ""http://test.ckan.net/api/2/search/package?q=osm&_=1310056826904""
}}}
Gives me none.
The underscore is generally used by JavaScript libraries as a way of bypassing the browser cache when making JSONP calls. It's easily worked around but is odd none the less.
"
1214 1310062570000000 dread "Fixes have gone into the branch for ""Tag returned as a JSON object when updating but as a string when requesting. Same with ""extras"" content."" and ""Doesn't return rendered_text property on package update."".
I've added DELETE to the Access-Control-Allow-Methods on the server, so you can try that now.
I've added a failing test for the problem of deleting the last extra. Am discussing with David Raznick how to fix that best.
I think the only other ones left to address are:
* Relationships use ""object"" key rather than id or package_id.
* underscore in search parameters"
1214 1310062621000000 dread "I believe ""A PUT request without a body throws a 500 Internal Sever Error"" was fixed yesterday in the code. (Not deployed to test.ckan.net yet though)"
1214 1310067327000000 dread " Relationships use ""object"" key rather than id or package_id.
I'm not totally clear on this one. The URLs for relationship editing can be package ids or names. When you GET a relationship packages are referred to in IDs for v2 API. When you create or update a relationship it returns package IDs for v2 API. Are you referring to the relationships listed in the response when you PUT a package? (If so that would be fixed by my previous fix.)"
1214 1310067831000000 dread " underscore in search parameters
I think its trying to filter search results by values of an extra field called ""_"", even though this probably doesn't occur. If it is not too ugly, I suggest using one of your other ideas for solving the cache issue. I suggest you bring it up with Rufus or the ckan-dev list to get opinions."
1214 1310072808000000 dread I've put the fixes onto test.ckan.net - repoen the ticket if there are any issues.
1214 1310396399000000 aron "The groups property in the package resource is always empty (or is for all packages I have viewed).
The following group lists one package.
{{{
http://test.ckan.net/api/2/rest/group/0ac963e7-ba29-49bc-83c8-98f8c1991649
}}}
But when viewing the package the groups array is empty.
{{{
http://test.ckan.net/api/2/rest/package/758c26d4-5949-4347-8b4d-023374146d94
}}}
"
1214 1314029628000000 dread This appears to have been fixed by version 1.4.3.
1215 1310335541000000 kindly "fixed cset: 8a317eadbb36
If you delete the last row then it just clears it instead of deleting the row."
1215 1310385485000000 dread This is now deployed on ckan.net (requires Ctrl-R to reload js).
1217 1320663277000000 dread I haven't seen this for a while - could have been due to database corruption which has been fixed.
1218 1310385419000000 dread This has been done in cset:bd0f83a2e287 for release 1.4.2. Deployed to ckan.net.
1219 1310471415000000 dread This is a known issue and of low importance.
1219 1310670324000000 erilem "The issue is due to IE7 (and IE6) not supporting {{{inline-block}}}. [attachment:patch-1219-A0.diff] works around the issue.
I'm reopening this ticket in case you want to consider my patch.
But note that I won't be offended if you close it as wontfix again. I agree this is pretty minor, and that working around old browsers' limitations isn't always a good idea."
1219 1310740534000000 dread Patch applied in cset:8c05d27db224 for 1.4.2 release.
1221 1310475683000000 dread "Exception details:
{{{
Module ckan.controllers.user:194 in request_reset
<< h.flash_error(_('No such user: %s') % id)
try:
mailer.send_reset_link(user)
h.flash_success(_('Please check your inbox for a reset code.'))
redirect('/')
>> mailer.send_reset_link(user)
Module ckan.lib.mailer:69 in send_reset_link
<< def send_reset_link(user):
user.reset_key = make_key()
model.Session.add(user)
model.Session.commit()
>> user.reset_key = make_key()
AttributeError: 'NoneType' object has no attribute 'reset_key'
}}}"
1221 1310556607000000 dread "Other edge cases and error conditions do not work properly either.
Fixed in cset:4737c86d1d81 for release 1.4.2."
1222 1310556618000000 dread Fixed in cset:4737c86d1d81 for release 1.4.2.
1223 1310573893000000 dread pudo did this in cset:1ab655b7b76a and cset:7a9c6b1060cf ready for release 1.4.2.
1225 1310642407000000 dread Done in ckanext repo cset:70fb690a0769
1226 1314029434000000 dread This appears to have gone away. No sign of it for two weeks on any OKF sites.
1228 1311086612000000 dread Done in cset:88e47b68e42d for 1.4.3
1229 1312191530000000 amercader Done, except for Revision related stuff, which probably overlaps with work being done by David Raznick.
1229 1319639472000000 dread I found another direct use of the d.b. in the home controller.
1229 1319645778000000 dread Replace code in home in cset:f19f9c5bec94 on default for release 1.5.1.
1230 1311154142000000 kindly "The standard way to add tables in a plugin has converged upon putting the tables in the iconfigurable plugin.
This runs at the correct point for when the application runs normally. For testing however there are issues due to the tables potentially being dropped, especially for the sqlite case. The fix is to make sure the iconfigurables are run at the start of each set of tests, hence adding it to the ckan_nose_plugin. This is not pretty, but good enough.
cset:8531b9fc1ee2
"
1231 1315948921000000 rgrp @kindly: Does this fit in with current plans? If not suggest we close as invalid/wontfix or put into the backlog (though given lack of content not sure that is very useful ...)
1231 1325474087000000 rgrp Closing as wontfix (or duplicate). We now have stats in footer as of #1576. googleanalytics (e.g. http://thedatahub.org/analytics/dataset/top) would be nice to have but should be part of stats
1231 1325474447000000 rgrp See #1101 for other duplicate ...
1232 1315948536000000 rgrp Removing from ckan-v1.5 as do not think any of this is top priority (correct me if I'm wrong!)
1233 1315948668000000 rgrp Removing from v1.5 as not a priority atm.
1234 1311183328000000 dread Fixed in cset:1ed3fde56a61 for 1.4.3.
1236 1311351476000000 dread Working on it on branch enh-1236-view-package-revision
1236 1311699843000000 dread This is merged to default (for rel 1.2.5) but isn't brilliant yet because of bug #1238.
1236 1311759816000000 dread The bug is in fact in the way I work out what revision I'm displaying - it just looks at the PackageRevision.
1236 1323090508000000 dread This feature is now released, but there is a remaining bug which I've put in this ticket: #1509
1236 1330019788000000 dread Went into CKAN 1.4.3
1238 1311583876000000 kindly "I cannot reproduce this, to me it looks like it does get the correct revision. If you for example look at the package a millisecond before i.e
http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627829
the tags are no longer there that were added in that revision."
1238 1311759784000000 dread Apologies, I have a problem instead with the way I identify the revision in #1236.
1239 1311442163000000 rgrp Fixed in cset:adf254b7c507 (see https://bitbucket.org/okfn/ckan/changeset/adf254b7c507)
1239 1312453156000000 dread Fixed in 1.4.2 release. Broken in UI in 1.4.1 and API in 1.4 & 1.4.1.
1240 1314180488000000 dread "I basically agree, but just thought I'd add a couple of comments into the mix.
> GET at root returns whole package set (a *bad* idea)
Sure it has a cost, but I don't see what the problem with this is if the list is cached. Forcing developers to paging through the list of packages is putting unnecessary obligations. I'd say that the top two RESTful operations are listing objects and getting a particular one, and you'd be taking away one of those.
> Unify on /api/v{version num}/... structure
Where is it not currently unified?"
1240 1316022145000000 rgrp add autocomplete
1240 1324319119000000 rgrp assigning to kindly for review
1240 1324320107000000 dread "Although I'm generally in favour of Action API over RESTful, Action API does need some sorting out wrt listing methods, advertising what params are allowed and checking those that are passed. Using the Action API is so hard without reading the code!
The RESTful API is handy when you want to do something quickly in a browser. I'd not want to lose these two, for example:
* to look at a package /api/rest/dataset/xyz
* API version etc. at /api/util/status (coming with ckan 1.5.2)
It's also really handy in demos and to send as links to people on the list."
1240 1325473312000000 rgrp IMO this is non-urgent and can move out of v1.6 as we have enough more important stuff to do.
1242 1323173292000000 thejimmyg This text is no longer present on the login page.
1243 1328000405000000 rgrp invalid as things have changed and this has been fixed or obsoleted.
1244 1323089995000000 dread No progress yet
1245 1311864530000000 annapowellsmith "Would be nice to list some extensions somewhere on this site (features page perhaps).
Along the lines of: http://docs.ckan.org/extensions.html#finding-extensions
It wasn't until I wrote this list that I realised just how many extensions there were, and what it said about the CKAN community that there were so many...
So I think it would be a good sales point. "
1245 1317422640000000 rgrp Closing as now mostly done and not really a ticket item.
1246 1311863583000000 dread "We have an unresolved issue with the display of the 'open data' icon that showed up when we upgraded ckan.net to the latest beta last week, but I didn't realise the search page http://ckan.net/package was showing all packages as open (the revision page and package pages seem fine).
Last week when I upgraded from 1.4.1 to 1.4.2beta, there was an exception thrown when loading the home page. The exception was in doing the isopen call. It was solved by making some code changes directly on the ckan.net server to make it an attribute query instead of function call. It was strange because this behaviour didn't occur on my machine and all the tests passed. Here are the manual patches on ckan.net (the application.js change is unrelated, by someone else):
{{{
diff -r adf254b7c507 ckan/public/scripts/application.js
--- a/ckan/public/scripts/application.js Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/public/scripts/application.js Thu Jul 28 14:10:23 2011 +0000
@@ -2,6 +2,7 @@
$(document).ready(function () {
CKAN.Utils.setupUserAutocomplete($('input.autocomplete-user'));
CKAN.Utils.setupAuthzGroupAutocomplete($('input.autocomplete-authzgroup'));
+ // $('.resource-url').parent(
});
}(jQuery));
diff -r adf254b7c507 ckan/templates/_util.html
--- a/ckan/templates/_util.html Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/templates/_util.html Thu Jul 28 14:10:23 2011 +0000
@@ -43,7 +43,7 @@
standard package listing -->
-
+ class=""${'fullyopen' if (package.isopen and package.resources) else None}"">
${h.link_to(package.title or package.name, h.url_for(controller='package', action='read', id=package.name))}
@@ -60,7 +60,7 @@
-
+
-
![]()
@@ -72,7 +72,7 @@
-
+
-
${h.icon('lock')} Not Openly Licensed
diff -r adf254b7c507 ckan/templates/home/index.html
--- a/ckan/templates/home/index.html Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/templates/home/index.html Thu Jul 28 14:10:23 2011 +0000
@@ -64,6 +64,7 @@
Recently changed packages
+
${package_list_from_dict(c.latest_packages)}
- ${h.nav_link(c, _('Add a package'), controller='package', action='new', id=None)}
+ - Upload Data
- ${h.nav_link(c, _('Tags'), controller='tag', action='index', id=None)}
- ${h.nav_link(c, _('Groups'), controller='group', action='index', id=None, highlight_actions = 'new index')}
- ${h.nav_link(c, _('About'), controller='home', action='about', id=None)}
diff -r adf254b7c507 ckan/templates/package/read.html
--- a/ckan/templates/package/read.html Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/templates/package/read.html Thu Jul 28 14:10:23 2011 +0000
@@ -56,7 +56,7 @@
-
-
+
This Package is Open
@@ -80,7 +80,7 @@
-
+
This package is Not Open
Either because it is not openly licensed or is missing
}}}"
1246 1311885360000000 pudo "I'm afraid to say I don't understand any of this: the faulty display of openness is triggered as removing the brackets turns the isopen() check into a check for the existence of the method. Reverting the changes on the server seems to work without generating exceptions. What exactly is the issue that was supposed to be fixed originally?
"
1247 1311955176000000 rgrp Done. See http://ckan.readthedocs.org/ and http://docs.ckan.org/ will propagate soon.
1248 1319797763000000 dread "Fixed in cset:188d27a56c89c35f97f3a79fdd0a6460223f86cf on branch release-v1.4.3.
This was only seen in CKAN 1.4.3beta."
1249 1311959356000000 pudo fixed in cset:0398243111d4
1250 1312370201000000 pudo "The issue is caused after search returns from solr and tries to get objects from the database. These are by default ordered by name and we need to restore the original order after fetching them from the database.
Fixed in module."
1251 1314406332000000 rgrp Really non-urgent and can be done once #1252 is done (which is dependent on v1.5)
1251 1317315159000000 dread rgrp did this last weekend.
1252 1314406271000000 rgrp Move to v1.5 since this will coincide with release.
1252 1316876747000000 rgrp Announced. thedatahub.org now running for 2+ months. Today did wiki.ckan.net redirect and hope to do thedatahub.org redirect.
1252 1317315135000000 dread This is done now?
1252 1318160334000000 rgrp This was done ~2 weeks ago.
1256 1312409296000000 kindly This has also caused problems in authorization for deleted packages.
1256 1312813529000000 kindly fixed cset:357cf9377b25
1256 1319812556000000 dread It sounds like this became an issue with CKAN 1.4.3 and the fix goes into CKAN 1.5.
1258 1312813614000000 kindly cset:e22d4e385fc8
1258 1319812452000000 dread I believe this was broken with the introduction of moderated edits in CKAN 1.4.3 and this fix went into 1.5.
1262 1314029150000000 dread "Pudo - I'm not sure what's meant here. Are you talking about moderation for new users? Or some other sort of enforcement? Is it spam you want to target?
The links you're concerned about - are they the spam ones in the User's 'about' field? If so, I think we *should* allow users to have links, although continue with applying them with 'nofollow' and attack spammers back in a concerted strategy across all editable fields - see ticket #1257."
1263 1314029279000000 dread This was fixed in cset:8d293393a9f2 for release 1.4.3.
1264 1314406294000000 rgrp Nothing urgent about this so moving to backlog.
1264 1318244619000000 zephod Created #1377 to handle bugs revealed while doing this work.
1264 1318245716000000 zephod Complete; cset:b216952644aa
1265 1312566052000000 dread Fixed in cset:a1b8dd169f4e on 1.4.3 release and merged to default.
1266 1312797477000000 dread Fixed in 987dc4fe078e on 1.4.3 branch and default.
1267 1312889130000000 dread John thinks it is a problem with WebOb 1.0.6
1267 1312890962000000 dread This error has been occurring on at.ckan.net which has webob 0.9.7.1-1
1267 1312894071000000 dread Running tests with WebOb 0.9.7.1, we see the error! Am now trying to fix.
1267 1312897724000000 dread "Fixed in cset:db5bf5b45b4f for ckan 1.4.3.
Later versions of webob add the charset header for unicode errors (webob/exc.py:generate_response):
{{{
extra_kw = {}
if isinstance(body, unicode):
extra_kw.update(charset='utf-8')
resp = Response(body,
status=self.status,
headerlist=headerlist,
content_type=content_type,
**extra_kw
)
}}}
This was fixed in webob cset:25fc14d49623 which went into WebOb 1.0.
This problem affects versions from CKAN 1.4.2 back to 1.3.1. We had been avoiding this in CKAN previously by only doing abort() with a plain string. "
1268 1317212499000000 kindly fixed cset:eebbe6071741
1270 1312912188000000 dread Fixed in cset:ca5295c9c8cf and released as ckanclient version 0.9
1271 1313753663000000 rgrp Closed in cset:749cb3a087e3
1272 1313745867000000 rgrp "Assume these are the 3 spatial fields listed on http://wiki.ckan.net/Geospatial_Capabilities. I'm a big +1 here. Starting point would be a bit more documentation on that wiki page. Questions:
* Is this work going in ckanext-spatial?
* What is being proposed here, I assume, is to automatically index those fields in our geospatial search.
* Should we develop a simple display widget using google maps or similar (this is orthogonal to the WMS preview i think) which *just* uses the core fields. This is probably a separate ticket and possibly separate from ckanext-spatial extension."
1275 1314110016000000 johnglover "Completed in branch feature-1275-solr-search. Up to date with current default but not yet merged into default.
John"
1275 1319812967000000 dread Going into release 1.5.
1276 1314886010000000 johnglover "Added to branch feature-1275-solr-search
Don't seem to be able to apply weighting to fields at index time using solrpy (see open issue: http://code.google.com/p/solrpy/issues/detail?id=6). So, switched to using the newer Solr edismax query parser and have applied weights to fields at search time.
Current weights are as follows:
name: 4
title: 4
group: 2
tag: 2
text (everything else): 1"
1276 1315948394000000 rgrp Closing since done (though not yet in main!)
1277 1314868204000000 johnglover Done in branch feature-1275-solr-search for all package searches (which use Solr). Resource and Tag searches still use the old ckan query parser.
1277 1315297186000000 rgrp This just requires merging into main but has been deferred as once solr branch is merged solr is *required* to run CKAN at all (we are thinking of having basic support for sqlite/dbs out of the box ...)
1277 1315948417000000 rgrp Closing since done (though still need to merge into default)
1279 1314303732000000 rgrp Wow, that's a pretty big omission. authz groups are important and are used. I think we should discuss this in next planning meeting.
1279 1314304326000000 rgrp In addition it appears that authz group was left out of the form refactoring so new and edit page use old formalchemy stuff. Should fix this.
1279 1338212028000000 ross Duplicate of #2366
1280 1328786670000000 dread David tells me that this was fixed in CKAN when we moved to SQLAlchemy 0.7 #1433 which went into CKAN 1.5.1.
1281 1314021919000000 dread "Done by pudo in cset:eaf342823caf on default branch, so headed for release 1.4.4.
""Work around the fact that locale changes do not affect
the running request and therefore led to an incorrect flashing
message. Now manually changing language."""
1282 1314404540000000 rgrp "Done in cset:4b6838a820bb.
Have not done either of the optional items. JS testing for what we have is relatively hard and of dubious value.
Re CDN stuff requires work and of doubtful of immediate value since can just configure cache or main site to serve static stuff directly. (Likely more value from compressing js and css ...)."
1283 1326720891000000 dread Since we switched completely to SOLR, this has been true. SOLR doesn't index deleted packages so you can't search them, but they can be read/edited by admins.
1283 1330083933000000 dread This went into CKAN 1.6
1284 1315212258000000 dread "Fixed by rgrp in cset:1952445d2802 on release-v1.4.3 and default. Comment:
do not create revision for changes to users as they are not revisioned.
* This had already been removed in cset:8d6fde0e2196 (then in user controller) but Adria seems to have reinstated this when refactoring things into the logic layer. This 'bug' explains the huge number of empty revisions we have been seeing on ckan.net."
1284 1315212603000000 dread I've added tests for this (to prevent yet another regression!) in cset:1b77ce8ce560. Closing.
1284 1315214358000000 rgrp I'm not sure this is the sort of thing we need tests for (we can have too many tests!). Something worse discussing ...
1284 1315215626000000 dread Really? How many times do you want to fix this bug again? For the sake of three lines of test...
1285 1314269414000000 dread Another instance is when the database tables are not created and you want to both display a sensible error and have an email alert. #1290
1285 1323173125000000 thejimmyg "I could write some middleware to handle this. Can we come up with a list of exceptions not to catch or should I create a base exception called ""NoEmailReportTriggeredException()"" which exceptions for this purpose have to be derived from or which code has to raise? "
1285 1323178678000000 dread Isn't the point is we don't to raise an exception at all - we want to call a method to send the email and then return from the controller a normal html page. Don't we just need a method to get the error email address from the config and send an email.
1286 1342436420000000 dread This can happen now, if it hasn't already.
1289 1317315211000000 dread Discussions have not resolved this either way. Decided to leave it like it is for now.
1290 1314270894000000 dread "Improved on option 2. For requests to Home controller, the first thing it does is check the user. I catch SQL errors that are to do with missing tables and that now produces a template-less error saying the site is off-line and then mentions database initialisation.
Done in cset:484e50c4f989 on default for ckan 1.4.4."
1291 1315948475000000 rgrp @kindly: is this not done? (I know there is more work to be done here but though table creation was done ...). If not need to move to next milestone so we can close this one.
1292 1315999202000000 dread i'm def +1 for removing API stuff entirely from i18n for the present
1292 1320145677000000 dread " Review process of releases and updating strings (pot) and translations (po) files
c.f. http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html
http://wiki.ckan.net/Release_process updated with basic process
Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
Welcome message is translated in all languages. Most language have upped their percentage translated significantly and Swedish and Polish added as well.
Review long list of strings for improvement
Consolidate any?
Get rid of i18n for strings in api
About 50% of the strings are templates, 30% are in the logic layer and 20% are in the controllers. Removed a handful from ckan/lib/base.py that were only displayed in the api, but the logic layer ones are generally errors that can be displayed in the Web UI as well as the API, so not ideal to remove.
Try and remove javascript
Tom did this.
Add in extensions
This could be done. Have raised ticket #1292 for this, next release.
#1374 Fix switching to en from other default language
Done
#1417 Fix browser language detection
Done"
1292 1320145732000000 dread These fixes all went into release 1.5.
1293 1314701675000000 johnglover "== Changes ==
* All templates
* Routing: /package/ -> /dataset/ (including API)
* Tests: h.url_for(controller='dataset') rather than package ...
* One test to check redirect works?
* Docs
== Leave ==
* Stay with Package in code e.g. stay with 'PackageController'
* also leave stuff like pkg = .... (gradual migration)
* Leave extensions
== Notes ==
* Package -> Dataset
* Data Package -> Dataset
* respect capitalization"
1293 1314905799000000 johnglover "Branch: feature-1293-rename-package-to-dataset
Routing and test editing should be finished (all tests passings). Have not added test for redirect yet."
1293 1315214527000000 rgrp Reassign for completion (templates and docs) as john is away.
1293 1315246628000000 zephod "cset:https://bitbucket.org/okfn/ckan/changeset/4a98589af6c1
Fixed"
1294 1315948112000000 rgrp Removing #1311 as that is not directly related to dataset edit / create.
1294 1315948257000000 rgrp Done with merge into default in cset:8bb0720a2150 and deployment to http://thedatahub.org/
1295 1315820564000000 rgrp This is mostly done but some final tweaks and name disabling to be done. See work in feature-1294-ux-improvements-dataset
1295 1315948009000000 rgrp Completed with merge into default in cset:8bb0720a2150
1296 1315403275000000 zephod No longer doing inline resource editing on dataset/view; instead use the edit form.
1296 1315821293000000 rgrp Have now done resource add pretty effectively and most of edit done (should complete today).
1296 1315948027000000 rgrp Completed with merge into default in cset:8bb0720a2150
1297 1315403878000000 zephod No longer doing inline editing on the view page.
1297 1315403974000000 zephod Largely done in cset:8901e722b158
1297 1315404322000000 zephod all done: cset:71096cd7b27d
1298 1321199769000000 seanh I've been working on this in this branch: https://github.com/seanh/ckan/commits/feature-1298-activities-table
1298 1323106307000000 seanh See the new super ticket and its branch: #1515
1298 1323710593000000 dread Sean says: This is done on the activity stream branch. Just would like David Raznick to review it.
1298 1324384191000000 seanh "David,
Ignore my link above to a branch on my ckan fork. Now that I have permissions I'm pushing my branches to the okfn ckan repo on github. My super branch for the activity streams feature is '''feature-1515-activity-streams''' on the okfn ckan repo on github. This page comparing my branch to master is particularly useful:
https://github.com/okfn/ckan/compare/master...feature-1515-activity-streams
(click on the Files Changed tab)
For reviewing this ticket, the relevant changes to review are:
* ckan/lib/activity.py, all of it
* ckan/model/activity.py, all of it
* My changes to ckan/model/meta.py
* My changes to ckan/model/package.py
* My changes to ckan/model/resource.py
* ckan/tests/models/test_activity.py, all of it
The other changes on my branch are for other activity streams tickets that follow this one. The super ticket #1515 has an overview of it all.
"
1299 1325866115000000 rgrp No estimate as yet? (Or is it in super ticket? (Link?)). Also isn't this a duplicate of #1510 (or vice-versa)?
1300 1315906283000000 rgrp page_heading and optional_footer done in #1108
1300 1315908961000000 rgrp Move of js to optional_footer done in cset:dcb21f0d976f. As part of #1294 have done a lot of css adding. Not sure how much more to do here.
1300 1319794520000000 rgrp Side bar menu is switchable (though not sure how -- zephod has done this!) CSS tidying and labelling of divs done in #1380. So closing!
1303 1314978842000000 dread Adria fixed this in cset:7637d8694388 on default. I transplanted it to release-1.4.3 and have updated thedatahub.org.
1305 1314971164000000 nils.toedtmann "Always use ""localhost"" as mail relay. On all servers, the local MTA should be configured correctly (e.g. in this case to relay via mail.fry-it.com), also to be able to send rootmail.
I fixed it on IATI (s049.okserver.org = ex eu23) and thedatahub.org (s055.okserver.org). Please verify that it's working now."
1305 1315317033000000 nils.toedtmann [15:22:02] Adrià Mercader: Yes it looks like it works on both thedatahub.org and iatiregistry.org
1306 1319797204000000 dread These are expressed as functional API tests - see tests/functional/api/test_action.py. There's not a lot of difference (apart from Paste coming in there) so no real need to duplicate API and logic layer tests, or to switch them to talk directly to the logic layer.
1307 1315246706000000 zephod cset:2c31fe24e1c9
1307 1315247298000000 zephod "Replying to [comment:2 zephod]:
Oops:
cset:cbd6677ec41e
"
1309 1315586335000000 dread Duplicate of #1318
1310 1315820419000000 dread This was fixed cset:e49781cb74fd for 1.4.3.
1312 1315327743000000 dread "The default Content-Type is ""application/x-www-form-urlencoded"" (that is what curl sends, and what CKAN defaults to). Therefore these characters need url encoding (""=&;"" -> ""%3D%26%3B""). Here is an example of it working:
{{{
(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test7"", ""title"": ""Test2 %3D%26%3B""}' -H ""Authorization:tester""
HTTP/1.0 201 Created
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 16:36:32 GMT
Pragma: no-cache
Cache-Control: no-cache
Location: http://localhost:5000/api/rest/package/5866a0f5-905a-44ff-91c3-c1489374a73e
Content-Type: application/json;charset=utf-8
Content-Length: 571
{""maintainer"": null, ""maintainer_email"": null, ""id"": ""5866a0f5-905a-44ff-91c3-c1489374a73e"", ""metadata_created"": ""2011-09-06T14:06:40.735488"", ""relationships"": [], ""metadata_modified"": ""2011-09-06T16:36:32.543461"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [], ""groups"": [], ""name"": ""test7"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""notes"": null, ""title"": ""Test2 =&;"", ""ratings_average"": null, ""extras"": {}, ""ratings_count"": 0, ""revision_id"": ""92ee0b2e-c5e4-47b9-b00c-b8e29662b950""}
}}}
Alternatively you can set the content type to ""application/json"" and you don't need to URL encode. e.g.:
{{{
curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test5"", ""title"": ""Test &""}' -H ""Authorization:tester"" -H ""Content-Type: application/json;charset=utf-8""
}}}
To help work out these problems with encoding in the future, I've improved the error message to show the decoded JSON string. cset:55273629a552 which is headed for CKAN 1.5 release."
1313 1317076432000000 rgrp "Error was due to maintenance. Need both config options for recaptcha to work so either one can be used to check config. Re docs see #1358 (auto generation of docs on config from the config source).
Given that only docs remain and that is part of #1358 am closing as duplicate."
1315 1315414218000000 dread Pretty good start added in cset:7a0e7a486df3 on default, aiming for release 1.5. Just needs some loose ends tidying up.
1315 1320173768000000 dread Tidied up in cset:39daf1afd64a for release 1.5.
1316 1338193724000000 kindly now get_or_bust function handles this.
1317 1315416649000000 dread "Ok, I think this example may be when searching for ""ira"". ""irina"" works me, but ""ira"" gives:
{{{""ira"" matched several users}}}"
1317 1315822670000000 rgrp This is very minor IMO. I would defer this as part of a more general user system ux refactor.
1317 1325473762000000 rgrp Moving out v1.6 as minor.
1318 1315569362000000 dread "Problem tracked down to Adria's cset:4f1ca1eb2630 on branch feature-1229-db-out-of-controllers that missed out repoze.who in the redirects after login, when refactoring the user registration. This code went into default on 2011-07-28 for release 1.4.3.
Fixed in cset:93731cd3b22e for release 1.4.3 and doesn't affect previous releases. Tests for user registration added too."
1319 1315476057000000 shevski Ideally we should also verify people's email addresses. Do we think that's overkill?
1319 1315904592000000 dread "No responses received about this on ckan-discuss, so I'm taking that as no objections to requiring email address on registration.
I think that verifying email address is not worth it in this case, because the value of supplying an email address is only for resetting a password. The chance of needing a password reset AND having inputted an incorrect email address is very low, so I don't think it justifies everyone having to do this extra step. If we used email address more then that may change."
1319 1318426807000000 rgrp @dread: hope it is OK to re-assign to me but this is very similar to #1386 (and is a su bticket of #1343)
1319 1318527613000000 rgrp Fixed in https://bitbucket.org/okfn/ckan/changeset/130cad62163c
1319 1324036947000000 dread Goes into release 1.5.1
1320 1317391018000000 zephod Extend misc.py to regex find http and https, surrounding it with <>. ALSO note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
1320 1317730999000000 zephod "Slightly painful when links appear inside naughty HTML tags or in markdown tags. Restriction: Links are auto-linked if they appear at the start of a line or after whitespace.
cset:f9dfb0506594"
1320 1318164603000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/f9dfb0506594
1321 1319558004000000 dread "We should not cache if there are flash messages to show (otherwise they get queued up for the next non-cached page view).
Fixed in cset:945e2a6ce74f for release 1.5"
1323 1315594333000000 dread "thedatahub has a couple of auth groups with no name. The form allows it!
Fixed in cset:bebd0a46ebc6
* form requires the name field
* existing auth groups without name can be viewed/edited if necessary (referring by ID)
This has probably been a problem since auth groups were added to CKAN."
1324 1315948336000000 rgrp Done and merged into main at cset:8bb0720a2150 (and deployed!)
1325 1315587113000000 dread Fixed in cset:0dd4d4457c97 for release 1.4.3. Previous Python 2.5 incompatible versions include 1.4.2 - 1.3.4.1 and maybe further.
1327 1320662446000000 rgrp Duplicate of #1397
1328 1340191065000000 ross Let's see if this is still an issue in 1.8
1329 1316624026000000 zephod "I've tried to do this wherever I could as I looked through ckan.pot for broken pieces of string. Generally it triggers errors unless things are very neatly structured (no nested links or ${a.variable} accesses, it seems) but things are quite a lot tidier.
cset:d5bf4c7ba293
Does that count as 'fixed', or is this a permenant issue?"
1329 1317118323000000 dread I'll take a look to see if I can do any more and then will close it.
1329 1317404566000000 dread "POT looks far better. I couldn't see any outstanding problems.
This has gone into ckan 1.5."
1330 1327060201000000 dread "This is simply incorrect. test_authz.py (I assume you mean the one in ckan/tests/functional?) tests that authz works all across the Web UI and API in enough detail.
The 'total site lockdown' test - perhaps you mean the TestLockedDownViaRoles class which makes up a small part of test_authz? It was a test added due to an untested use case implemented in IATI, DataGM etc. These are currently active so still need testing.
Regarding refactoring the tests to be ""proper"" then please open a new ticket, providing references to why it is not ""proper"" to test by searching for text in a web response."
1331 1325355631000000 dread This appears to be fixed in 1.5, 1.5.1 and 1.5.2a.
1332 1315999104000000 dread "Fixed in cset:860b357a2aec for release 1.4.3 and updated no.ckan.net.
This problem was due to {{{[1:authorization group]}}} being translated as {{{[autoriseringsgruppe]}}} and I corrected it to {{{[1:autoriseringsgruppe]}}}. This word has a bold tag, hence the brackets and the number is used to identify which bracketed word is which (when you have more than one).
It is a shame that Transifex doesn't draw attention to this problem and Babel doesn't spot the error on compilation."
1334 1316086680000000 dread cset:b01de680bf86 copes with the problem. Now to find out why revision.packages has some None values - very odd!
1334 1319798961000000 dread I think this has been a problem since 1.3.3. Fixed in v1.5.
1335 1315910117000000 dread "Done to a basic extent in branch feature-1211-drupal for release 1.4.3.
Further tickets:
* #1322: Action API improvements
* #1315: Document /api/action"
1338 1316951545000000 amercader Reopening, because another change is needed to support custom schemas
1338 1317117738000000 amercader Fixed in eca1edce3a0f
1339 1316010607000000 kindly "I have fixed the isodata and made a slightly modified int_converter for this case. In the correct place and raises invalids on not being an int. cset:a4af115116bb
The thinking was that the input of these fields would be through the api so the empty string case did not arise. These should clearly be converted to None (Null).
What issues in general?
Having done this lots of times before you always end up needing to write your own little validators as the standard ones never do what you want. Thats the point of them. Look in ckan/lib/validators if you need examples.
So what you did was correct..."
1340 1315991534000000 dread Fixed in default in cset:33856d3ce84e aimed at release 1.5
1341 1315996368000000 kindly "This was run to delete the users and their mistaken revisions that where created.
{{{
BEGIN;
delete from revision r where r.id in (
select r.id from ""user"" join revision r on r.author = ""user"".name
left join resource_revision rr on rr.revision_id = r.id
left join package_revision pr on pr.revision_id = r.id
left join group_revision gr on gr.revision_id = r.id
where ""user"".created between '2011-08-15' and '2011-09-06'
and gr.id is null and rr.id is null and pr.id is null
and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%')
and ""user"".name not like 'http%'
);
delete from ""user"" u where u.id in (
select ""user"".id from ""user""
left join revision r on r.author = ""user"".name
where r.id is null and ""user"".created between '2011-08-15' and '2011-09-06'
and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%')
and ""user"".name not like 'http%'
);
COMMIT;
}}}
"
1341 1319649573000000 dread The majority of our 6000 users are still spammers. Can we simply elimenate those who've not made any revisions? Even if we take a few genuine users with it, these are pretty dead accounts anyway.
1341 1319709997000000 dread Reopening - this is still a serious problem. We can't link to the user page because it is so embarrassing: http://thedatahub.org/user
1343 1316078440000000 dread I don't think email confirmation is necessary - see discussion at #1319
1343 1318528138000000 rgrp Remove from ckan-v1.5 since remaining non-complete items (user page improvements and modal login won't get done in v1.5.
1344 1316078757000000 kindly fixed cset:f08215845dab
1345 1337961353000000 nils.toedtmann This becomes an actual problem on the CKAN farm s057, see http://trac.okfn.org/ticket/1245
1345 1338551926000000 nils.toedtmann See #2485 for workarounds containing the leak(s).
1345 1340190737000000 ross Moved out of milestone so that it gets looked at again soon
1345 1343215489000000 amercader See https://github.com/okfn/ckan/pull/73
1346 1316086314000000 dread "Fixed in cset:df299e69d456 on default for release 1.5.
Problem exists on releases 1.4.3 - 1.3.1 and probably earlier too."
1346 1316094632000000 dread Also Transposed to 1.4.3.1.
1347 1316094608000000 dread Fixed on cset:7c3cc0fd6ae4 on branch release-v1.4.3.1.
1349 1316277490000000 zephod Done. cset:e074aee19923
1350 1316624288000000 zephod "Was going to reimplement an old workaround described by Friedrich, where a language.js file is served through the home controller (with appropriate settings in routing.py) and so ${_('string literals')} will be translated. See:
* https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/controllers/home.py#cl-48
* https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/templates/home/language.js
However the string-extraction mechanism seems geared towards Genshi templates and hacks are required to make it read a javascript file neatly (ie. without putting the *entire file* into a string for translation). A more elegant solution seems to be using an xi:include to pull in a genshi template which contains an html
}}}
where the jquery.html5.preie9fix.js is simply the [gist](https://gist.github.com/887560) published by Akkuma in a [jquery-tmpl issue (#36)](https://github.com/jquery/jquery-tmpl/issues/36#issuecomment-918495).
This script patches JQuery on the fly on IE < 9 to rework in memory html 5 blocks handled by JQuery, so that it doesn't break.
And this fixes the ""Upload a File"" button on IE.
I didn't commit/pull this specific fix for the following reasons:
* the script embeds innerShiv, which is a previous incarnation of html5shiv (now included in CKAN). It doesn't sound fine to include 2 pieces of code with similar effect, but I couldn't reproduce the fix with html5shiv!
* in my environment I have included this fix sitewide even if I didn't spot other problematic features..I don't know if it would be better to include it only on the dataset edit page in the official CKAN code.
Summary: I leave to you guys the choice on how to deal with this problem."
1788 1330624155000000 zydio "Erh...sorry for some formatting errors in my comment.
I used markdown formatting for links, and used the #NUMBER format to cite issues, which obviously is interpreted by Trac as references to his own issues...my bad."
1788 1330644056000000 rgrp Re the Upload file button not sure what's best. We do have plans to replace jquery-tmpl with mustache in the near future (but not clear exactly *how* near). We'll think about this and get back to you. Thanks for the detailed debug report.
1788 1333133365000000 zephod "I've been performing a full sweep of the site to rebuild the markup & stylesheets using Bootstrap, and in doing so have managed to close off this ticket. Bootstrap allows us to delegate lots of clever functionality and layouts to Twitter's tried-and-tested code, making it easier to be compatible with IE7.
It has brought up plenty of issues: Lots of Javascript was broken; plenty of pages generate broken doctypes causing the browser to go into quirks mode (which in turn causes the page to explode all over the screen); most dynamic content would fail to generate properly in one way or another.... (see the #1788 branch for a log of issues encountered).
I've been refactoring our JS and have managed to strip ckanjs, too, because the only significant element we'd used was the File Upload view (which also now works in IE7).
test.ckan.net has been redeployed on master; I'm browsing the site in IE7 and everything is looking great :-)"
1791 1329138315000000 dread Fixed in [release-v1.6 beaeaed]
1792 1329483249000000 toby package create update use correct schema in get_action
1792 1332755148000000 toby closing as done all that I was asked to do as part of this ticket
1792 1332872069000000 dread "What is the changeset for this? There is evidence of a branch ""enhancement-1792-api-logic"" but perhaps this name got fast-forwarded out through a bad merge? Is this merged into 1.6.1 release?"
1792 1332928847000000 toby This is in the 1.6.1 as far as I'm aware. The branch was deleted after the merge into master
1792 1333037710000000 dread "I have tracked down the changesets to:
* branched off master at dff9f7a8
* merged master in for the last time (and became the 'new master'?) at 1a9227d9ff73c
"
1796 1339586499000000 ross LXML removed, update requirements on the extensions
1797 1330550304000000 rgrp New API endpoint, Authorization and documentation done and merged to master. Time so far. 1.5d. https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24
1797 1330863639000000 rgrp Data Viewer support for new DataStore in https://github.com/okfn/ckan/commit/9ab8b0283bb086eb4cd663ff73c27066bdd3c79a
1797 1331412469000000 rgrp "All done!!
Help for data api done in https://github.com/okfn/ckan/commit/1d8e464f8542d4c33286bb93f4de50060665799f
Checkbox for datastore enabled in https://github.com/okfn/ckan/commit/3f1320cd92ae0e775fde1b5eada156260c55e0a6"
1797 1331412644000000 rgrp "Merge fix https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547
"
1798 1329395560000000 dread "Was fixed by Ian in https://github.com/okfn/ckan/commit/5a9054459e3833443bed3e118bbbb6c442e55b0b on branch feature-1453-flexible-tag-names.
This has gone into CKAN 1.6."
1799 1330000389000000 dread Fixed in [master ea2d824] for both logging in and registering whilst logged in. Cherry picked to release-v1.6
1801 1343144718000000 amercader Closing as both old and new theme now show password reset links
1802 1329409718000000 dread Current progress - I have got most of the dependencies installed - many by pip, but some I had to revert to Windows msi installers. Pad with details here: http://ckan.okfnpad.org/windows. So far done: 1 day.
1802 1329490081000000 dread "I've achieved CKAN running with paster and without SOLR (using simple_search).
Running locally on an old XP machine, performance seems fine - no noticeable issues.
I installed most things into a pyenv using pip. pyutilib, psycopg2 and lxml were slightly problematic, but I have workarounds.
I had problems pip and svn and the Cygwin window, so reverted to DOS prompt. I still had the Cygwin tools installed though, so next step is to try it without these at all - they are a big overhead."
1802 1329734716000000 dread Spent so far 1.75d
1802 1329830097000000 dread "* Refined install without using Cygwin - this seems a little easier.
* Added example Apache & modwsgi deployment.
* Documented extensively at: http://wiki.ckan.org/CKAN_install_on_Windows
* Have not tried SOLR on Windows - I'm not convinced this is high priority, and normal install should be fine as it is so separate from CKAN.
Spent another 0.5d. Total spent 2.25d"
1804 1329491156000000 toby This should be fixed in branch feature-1653-i18n_url_rewriting as we use different logic. However it may affect other return_to calls used elsewhere like adding a new package.
1804 1330000705000000 dread I'll fix this on release-v1.6 and then reassign to Toby to fix on feature-1653-i18n_url_rewriting too if the same issue crops up there.
1804 1330037202000000 dread "Fixed this on release-v1.6 in cset 557e72330db7. Spent 0.5h.
Reassigning for Toby to look at branch feature-1653-i18n_url_rewriting "
1804 1330347185000000 toby This has been fixed in master for a while and was only a 1.6 issue
1805 1329760795000000 toby this is fixed in branch feature-1653-i18n_url_rewriting
1806 1330942356000000 toby version one completed
1806 1331295310000000 toby closing as initial work has been done if we need to amend then create a new ticket
1807 1330809641000000 seanh "Implemented the recently-changed-datasets activity stream on branch feature-1807-recently-changed-datasets in CKAN and merged it into master (but the default templates don't make use of it).
Added the recently-changed-datasets activity stream to the front page template on branch master in ckanext-ecportal."
1809 1330528828000000 johnglover Caught exceptions in QA instead, error message stored as openness_score_reason
1810 1329918479000000 johnglover Deferred for now, moving to backlog.
1810 1335874745000000 johnglover Not using eurovoc tags.
1811 1329918517000000 johnglover Deferred for now, moving to backlog
1811 1335874765000000 johnglover Not using eurovoc tags.
1812 1335874864000000 johnglover Ross has fixed this via group authentication.
1816 1330863079000000 rgrp "As per detailed comments in http://wiki.ckan.org/Proposals#Apps_in_CKAN I think Apps is a confusing name. What we are talking about here is the ""Related Stuff"" item mentioned in that ticket.
Rather than update ckanext-apps I suggest we do a rewrite. I've created a detailed super ticket at #2204 and assigned to you Adria for review and thoughts. (While I understand #2204 may involve more work which we maybe cannot do before relaunch I'd still recommend starting on that rather than spending time upgrading this extension -- if we have to disable it for relaunch I don't think that is a big deal ...)"
1816 1331302835000000 amercader "+1 to a new, more flexible extension.
But in the meantime I've just spent a couple of hours making it work with latest CKAN, which was easier than expected, so we can deploy it with the new version of PDEU."
1819 1332163324000000 kindly Currently using package_show_rest. Should be moved to just use package_show but that is another ticket.
1828 1330525412000000 rgrp Fixed in https://github.com/okfn/ckan/commit/e6da48d9337575a83bc6a30896e68d1081b43847
1829 1330001828000000 dread "This was due to https://github.com/okfn/ckan/commit/f07bfdb9#diff-1 - string ""Language has been set to: English"" shouldn't have been translated. I've put in a patch to make the code intentions clearer and added a test: [release-v1.6 07d3a02]. This only affected the 1.6 beta, and is fixed for the release."
1830 1330089912000000 dread "Now if you set openid_enabled to false, you are not given the option in user settings to associate your account with openid. Cset: [master b295bde]
I looked at removing the openid middleware, but we use it for logging out users, whether they logged in or not - it looks like friendly form plugin is not as good as the openid one in this respect! We could hack the logout function into the friendly form to sort this out, but not sure the effort is worth it at the moment."
1831 1330956631000000 ross Already have a branch for this.
1831 1331549284000000 ross Email is not currently 'unique' in the user model. Find out if this is intentional or not.
1831 1331655458000000 ross Decided to delay this until later but code is in feature-1831-login-by-email
2198 1339771453000000 kindly Already in the docs.
2201 1330765400000000 rgrp Duplicate of #1171.
2202 1338204729000000 toby these are available through the pkg_dict
2202 1338204801000000 toby "re-reading this it's slightly different
re-opening"
2203 1348568961000000 dominik There is work on this in the dataviewer-iframe branch.
2204 1332341044000000 toby needs discussion - planned for 23-3-2012
2204 1332341080000000 toby http://ckan.okfnpad.org/feature-2204-related-stuff
2204 1332843232000000 toby still awaiting discussion
2204 1334077936000000 rgrp "Important comment: not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...
Also (@icmurray): does this interact with our desire to do embedding?"
2204 1334218735000000 ross "I'd be happier adding this to another ticket rather than moving the target for this ticket half way through so have created #2284
Should only be a short ticket in the next sprint."
2204 1335287636000000 aron.carroll "Making some notes of front-end changes that I think need to be implemented sooooon.
{{{
Implement better truncation (the expanded text should pop out on hover rather than expand the entire row)
Show the type of related item (an icon would be good here)
Test in IE
Make the form work without JavaScript by accepting x-www-form-urlencoded.
Add classes to first item in a row to clear the row (currently uses :nth-child CSS)
Touch gestures and re-ordering (David's request)
}}}"
2204 1335795665000000 ross Giving this ticket to Aron to fix the front end
2204 1337582843000000 rgrp "Surely this ticket was closed 3w ago?
@Ross: can you close ..."
2204 1337594723000000 ross Was waiting for documentation, can close now.
2206 1331046486000000 johnglover "Close enough for now. Still have to restyle language drop-down to match template, but will fix when updating the list of languages.
Also still to decide on menu styles/locations (for CKAN login and main menu, and ODP main menu)."
2208 1332251880000000 seanh "I've attached a ckan.pot file. To make it, I edited setup.py in CKAN core like this:
https://gist.github.com/2135849
and then ran python setup.py extract_messages. (And I had ckanext-ecportal installed at ~/pyenv/src/ckanext-ecportal.)
I guess I may need to run this again when the ckanext-ecportal source is finalised, so leaving this ticket open for now."
2209 1331547639000000 ross Option 4 is the most appealing at this point, although you probably should look at #2209
2209 1331547674000000 ross "Previous comment should read:
Option 4 is the most appealing at this point, although you probably should look at #1649 "
2209 1331551170000000 ross "Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template.
Pros:
Easy to implement
Easy to use
Not hard-coded as currently
Fast execution
Cons:
Requires knowledge of required RDF output if default is not useful
RDF and not any of the other formats yet.
Only works with package/resource/tags unless more work is done
"
2209 1331551393000000 ross "Can't edit comments .. so
Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template.
Pros:
Easy to implement
Easy to use
Not hard-coded as currently
Fast execution
Cons:
Requires knowledge of required RDF output if default is not useful
RDF and not any of the other formats yet.
Only works with package/resource/tags unless more work is done
"
2210 1331549350000000 ross This is now done through the logic layer member_create/member_list/member_delete
2212 1331719868000000 johnglover Added as a new paster command in ckanext-ecportal (run 'paster ecportal' for a list of commands and arguments).
2212 1331720191000000 johnglover Deployed on test server, where it imported 4087 datasets. A small number of datasets were not created as they failed CKAN validation - most of which had strange values such as 9999 and 0 for date fields (some also didn't have unique names).
2220 1331291241000000 toby "issues:
1) want to cache static files long term - need to add a ?ver=xxx to allow changes - this will need a change to all templates etc - could be seen as a good thing
2) we can only purge specific pages not whole cache so we need to know all the urls which want to be invalidated - including language variants
3) urls should be consistent so we should use id not name for datasets etc
4) for now we can just use short expiry times as this prevents some of these problems"
2220 1332950550000000 toby "Implementation of this has been carried out at the page level.
ckan.page_cache_enabled is config option to enable it
requires redis
need to document it somewhere
"
2221 1331561095000000 ross Fixed in https://github.com/okfn/ckan/compare/1eeda11...ad2358a
2223 1331558791000000 rgrp "Have introduced boostrap js and css on merge of #1797 https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547
However need to remove blueprint and make sure we keep our original theme."
2223 1332246030000000 zephod "I have been working overtime to try and get this change straightened out as it's suddenly on the critical path for something. Have a look here: https://github.com/okfn/ckan/tree/feature-2223-bootstrap
It's 80% of the way there; some form CSS needs overriding because Bootstrap tends to make things very large so they spill over. But on the whole the markup is in a steady state, which is what we need."
2223 1332842692000000 zephod Merged into master here: https://github.com/okfn/ckan/commit/0bf4b33955df3806785f9b42a55acaeb45596e25
2224 1332475944000000 rgrp Autocomplete based on bootstrap looks pretty non-trivial ...
2224 1338205358000000 ross Being done by Aron
2225 1343220502000000 aron.carroll There is a jshint.json config file that all JS in the new base theme conforms to.
2226 1332159640000000 seanh These deployment docs seem worth moving into sphinx as well: http://wiki.ckan.org/Deployment
2226 1332475468000000 rgrp All done (as of Tuesday in fact!). No doubt further work to do but that's a new ticket!
2227 1332843126000000 toby being done in other tickets
2232 1332331660000000 amercader "Fixed in 6b4c6c9
This was due to some characters not being supported by XML. We now strip them before indexing."
2234 1337237687000000 rgrp This looks interesting but is this a priority :-) (also no estimate as yet :-) )
2234 1337239588000000 ross Yes, definitely need a replacement for wordpresser and sooner rather than later. Doing some experimental noodling to work out an estimate.
2236 1332153688000000 rgrp Done. See https://github.com/okfn/ckan/commit/0d970d3a0e523a1695830e8c8c28c0ba10a8d8b9
2238 1343392681000000 seanh Ticket #2771 encompasses this and related changes. Vocabs probably won't be deployed to test.ckan.net but a working example extension using vocabs will be added to core instead. Closing this ticket as a dup
2239 1334581838000000 ross Changes to schema/rdf output
2240 1332408769000000 ross Distributed to ckan-coord
2242 1333375436000000 seanh Conclusion: jenkins should work, may need some jenkins plugin magic to get it to show what branch each build was run on.
2250 1332680690000000 rgrp Is this now how we are doing it? I thought we were integrating with RDF store (so it was more about sorting out the data viewer / recline stuff)?
2251 1332331650000000 rgrp Big +1: everyone wants to know page views. Would like detail of how this goes into interface. Downloads already being tracked. Also isn't this just an extension to ckanext-googleanalytics.
2251 1332332390000000 toby "not really part of ckanext-ga as we need to sit first in the middleware for speed reasons. I'm looking at recording the details and will do a summary page like ga extension.
As far as other uses like showing popular resources etc then we need to decide where the data will live - on the resource or separate.
Downloads tracked? where is this done in the code/data stored"
2251 1332340874000000 toby "notes from meeting 21-3-2012
US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.
US1a Next to a resource or a dataset see how often it has been downloaded / viewed
US1b I want to see datasets or resources ranked by most downloaded or viewed
US1c See a trend graph for a dataset (and resources)
Adminstrative Dashboard (?)
* I want to see the traffic breakdown by country to my site ...
* Ditto for browser type, language, etc etc
* I want to see it graphed over time ...
== Implementation Details ==
1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display
* Config option ckan.status.enabled = False (by default)
=== Storing Data ===
How does ckanext-googleanalytics do this? Current table:
package_id | count_recent | count_total
Move to a new stats_summary table
id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value
Do we store this data into the search (solr) so we can search by it?
=== Displaying Data ===
* Helper functions / dictize:
* Helper function: h.stats_get(object_type, id, stats_type)
* h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
* Change to dictize
* Location in the default theme (do we show for example in search results too!)
* Support for ranking by most popular in search?
=== Tracking Data ===
* Our own solution (just write to site_tracking)
* Google analytics (plus extension for retrieving data) <- would need a refactor
* Piwiki
==== Own Solution ====
site_tracking table
id | url | timestamp | action (page_view, resource_download) |
* Make javascript to make request to ckan to store clicks and page views.
* Add middleware so these requests do not go through pylons and just store data quickly.
"
2251 1332340947000000 toby "TODO
look at
1. How do we store this data in CKAN?
3. How do we display * Config option ckan.status.enabled = False (by default)
"
2251 1332524123000000 rgrp Update description in great detail.
2251 1332776926000000 toby "we want
resource show downloaded/viewed on resources
show on the dataset total/recent"
2251 1333535572000000 toby "we have various options on tracking unique users seems the sane approach.
* unique daily view - needed for nice graphing
* unique total views
* total unique daily views - higher numbers and feels more right - easier to calculate if original data is lost/or archived
for now will collect all and then decide which to use for display as gives us flexibility if needed
display
-------
why is group listing of datasets inconsistent with main search (TDH at least)?
should we make these consistent?
group listing allows downloading of resource without tracking (this should be fixed)
if we make listings uniform do we want the download links or not?
I don't like the term download when it is just a link (maybe that's just me - what would be better?)"
2251 1333644538000000 toby "most work completed: outstanding issues
* add a prune tracking_raw data to cli
* add indexes to tracking_raw
* add summary page for new data format
* add api counter
1) do we record the url or action etc (action seems more sensible)
2) add in __call__ or in action?
"
2251 1334074772000000 toby api counter moved to ticket #2282 as needs better specification
2251 1334144999000000 toby "For meeting today with Adria
* How to add tracking to solr index. Can the data live in tracking_summary? Do we need any extra indexes on table to help indexing?
* How will we add order by popularity to search?
* Can we add resources too?
* Will search be best way to find most popular packages/resources or should we get that pages data from tracking_summary?"
2252 1333108787000000 amercader "There is a new ticket for the same schema as forms issue: #2268
The exception was fixed on [https://github.com/okfn/ckanext-inspire/commit/0dff1f6 0dff1f6]"
2252 1333374836000000 amercader This needs more work on ckanext-dgu and testing.
2252 1334567495000000 amercader Fixed on 0dff1f6c2e
2253 1340038490000000 toby closing as nothing has ever happened with this ticket
2254 1332343198000000 toby "What do we want to be able to do?
what do we use the stream stuff for at the moment?
la la la"
2254 1332950678000000 toby "plan of attack
a) get removal of TDH from core - this will give a better idea of the issues involved
b) have a think"
2254 1338210316000000 toby "closing.
we have fanstatic integration in ckan.demo work
tdh is it's own extension
h.snippet is in 1.7
ITemplateHelpers in 1.7
This allows most of the issues resolved. Full removal of the IGenshiStream filter still neads some form of in template hooks but this is being looked at as a possiblity in ckan demo and can be considered out of scope for this ticket
closing"
2255 1332752936000000 rgrp "I would say Public is default for Organization with Private being an option (or do we make this a config option?).
What about ownership by users?
> New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below)
Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page."
2255 1332772340000000 ross "> What about ownership by users?
Good question. WIll add to description.
> > New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below)
> Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page.
The former, although currently we will have a slightly different form in the organizations extension, until such a time as we're ready to replace the existing one. Tried merging the two forms but it ended up with such a mess of logic that it was unmanageable.
"
2255 1332934041000000 ross Changed description based on docs of stuff to verify.
2255 1334582784000000 ross Review + restrict resource access (when private)
2256 1333620795000000 toby "I'd like to remove all 'custom' icons and just use bootstrap ones - is this reasonable? they are added in different ways but h.icon() could be easily adapted for this.
minimal .sj .css files"
2256 1333720237000000 rgrp Not sure what ticket means (remove TDH from core?). Also agree that we should glyphicons from bootstrap where possible going forward - though that would seem to fit more naturally with #2224 (Simplify javascript and css dependencies and add minified version) and have little to do with TDH versus core :-)
2256 1334051526000000 toby "@Rufus,
The idea of this ticket is to separate TDH into an extension rather than it being incorporated into core. This should help TDH by making it possible to push fixes without needing to do a new release of core. It should also help make core more stable and as slim line as possible. New features can be tested in TDH and then if they have a wider appeal they can be merged into core.
There is a strong desire for this within the dev team."
2256 1334051733000000 rgrp @toby: indeed and I'm strongly in favour of the basic idea too (and largely a dev question so up to you guys though feature / functionality implications would need to be talked through). My point was that custom icon stuff was #2224 :-)
2256 1334052087000000 toby "@Rufus,
Thanks, I think there is overlap and I'm happy to leave that bit of work to you (or future ticket owner) I was just adding some brief notes to this ticket on some of the outcomes I'd like to see and hence why I'd be happy to simplify the icons."
2256 1336046098000000 toby I'm closing this ticket now ckanext-datahub exists and is live - the other work is included in other tickets
2257 1332950416000000 toby pushed first round of changes into master
2257 1336731023000000 toby ckan.restrict_template_vars defaults to true for 1.8 release
2260 1333373978000000 johnglover Still some work to do on this, template needs to be finalised.
2260 1333554069000000 johnglover Finished for this iteration, waiting for design work to be completed before making further changes.
2261 1332838253000000 toby "new config option
ckan.root_path = /path/from/root/with/{{LANG}}/substitution
"
2261 1332840056000000 toby completed plus documentation updated
2261 1332840112000000 toby please test logging in/out as the redirect was not testable but should be correct
2264 1333635637000000 seanh "Many strings have been fixed (and committed on master). Still to go:
When uploading a file:
- The ""Browse..."" button. This comes from the user's browser?
- Hangs when I try to upload a file, so the rest of this UI is unchecked.
The licenses on the dataset add, edit and view pages, ""License Not Specified"" etc.
Dataset state on dataset view page, e.g. ""active"".
When adding or editing resources:
- The resource types (Data File, etc.) The strings seem to come from resource.resource_type.
- ""You can use Markdown formatting here"", ""Should a DataStore table and Data API be enabled for this resource?"", ""Dates are in ISO Format — eg. 2012-12-25 or 2010-05-31T14:30"". I can't get strings with hyperlinks in them to translate here.
Dataset view page:
- License not specified.
Dataset search page:
- ""Tags"", ""Res_format"" and ""Groups"" in the sidebar. Leaving because this code has been refactored on feature-1821-multilingual-extension. Fix after that branch has been merged.
Dataset view page:
- The resource types, e.g. ""plain text"".
- The field names under additional information: cache_last_updated etc.
Dataset history view: timestamps not localised.
Login page:
- ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"" This string is marked for translation but I can't seem to get translations to work for flash messages. Also ""Sean Hammond is now logged in"" flash message.
Register page:
- ""Missing value"" in error message when you try to register and don't give an email. The string seems to come from formencode. stdtrans() not working?
User view page:
- Activity streams strings. Activity streams genshi function needs a little refactoring to be i18n-friendly.
Users page:
- ""less than 1 month"". This string comes from Pylons' time_ago_in_words() function which doesn't appear to support i18n.
New auth group (and edit auth group) page:
- ""Unique identifier for group. 2+ chars, lowercase, using only 'a-z0-9' and '-_'"". This string is marked for translation but for some reason it isn't working.
- ""Name: Please enter a value"" (in an error msg)
- ""User name"". The string comes from class FormBuilder?
Also, need to check that all tooltips are translated!"
2264 1334348070000000 seanh "A few strings are still left to be marked for translation:
- http://trac.ckan.org/ticket/2289
- http://trac.ckan.org/ticket/2288
- http://trac.ckan.org/ticket/2287
- http://trac.ckan.org/ticket/2290"
2265 1332865837000000 dread "I started work on this and need to park it somewhere - here will do for now.
I was using the 'MLT Component' http://wiki.apache.org/solr/MoreLikeThis :
https://gist.github.com/2217684
But I think it would make more sense to use the 'MLT Handler' http://wiki.apache.org/solr/MoreLikeThisHandler which needs enabling by adding this to the config {{{/usr/share/solr/core1/conf/solrconfig.xml}}}:
{{{
}}}"
2266 1332932634000000 dread Fixed in [release-v1.6.1 ed2772b] with test.
2267 1343135541000000 toby this is not important as far as phase 2 is concerned - but you can do if you want
2267 1343229146000000 aron.carroll Fixed in eade4e3
2271 1334567582000000 amercader This was caused by an human error.
2275 1334159668000000 johnglover Done in branch feature-2275-group-logo, still need to decide on how to style.
2278 1333441177000000 rgrp https://github.com/okfn/ckan/commit/b7b32820a9132811028e7ccc243b2d2339492d4d
2281 1333967862000000 dread Done in [release-v1.6.1 9ecbfa5][release-v1.6.1 9ecbfa5]
2283 1340623843000000 kindly No super tickets anymore.
2284 1337583675000000 rgrp This needs connecting to the user stories being generated at http://ckan.okfnpad.org/dataexplorer-user-stories
2285 1334580266000000 rgrp "Recline improvements are done.
https://github.com/okfn/recline/issues/88"
2288 1338204815000000 ross Auth groups are being deprecated
2289 1340887101000000 toby needs docstrings
2292 1335021860000000 rgrp "This annoyed me so much I just fixed it: https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5
Issue was that our auto-completion depended on structure of the HTML which changes with bootstrapification.
BTW: have deployed this via copy and paste on datahub so i can use it. Will want to revert this when you do a deploy.
@Ross: I note that the same bug probably affects organization auto-complete."
2293 1339062853000000 ross "Intercepting the group name change and forcing a reindex of all of the datasets within that group seems like the obvious approach (rather than more lookups by ID after processing search results).
Should take less than a day."
2296 1334567141000000 icmurray "* ensure the read_ajax and history_ajax actions work in the routing.
* ensure the moderatededits.js works with the new templating
* ensure that the css works with the new templating
* ensure that the stream filters work with the new templating.
~ 6 days work"
2297 1334569632000000 seanh http://ckan.okfnpad.org/27
2298 1340285049000000 toby related to http://trac.ckan.org/ticket/2579 move sort functions to helpers
2302 1335516537000000 ross Will eventually want to allow customisation of different CSS but designs are currently under review/change.
2302 1342095074000000 toby "@ross
do you want to give this to aron or me?"
2302 1342097600000000 ross Have assigned it to Toby. Not sure how high priority this is, but only provides a very minimal level of configurability.
2302 1343315555000000 shevski "My ideas for this could include one day:
One page where the portal admin can choose and/or change the:
1. Portal name & tag line
2. Logo (to be displayed in header)
3. About page text
4. Featured dataset & text on homepage ?
5. Featured groups for homepage ?
6. Footer links
7. Colour scheme
8. Homepage background (?)"
2302 1343596669000000 shevski I reckon getting this live, esp with a easy to fill styled text header (see Sam's design) and a UI for changing About page text, featured dataset on homepage and welcome text is key
2302 1343636153000000 toby "@shevski,
please do not try to change the meaning of tickets/add features or they will just get closed as won't fix.
featured datasets is nothing whatsoever to do with this ticket so adding it to the ticket just causes me to get upset / demotivated etc. please do not do this. It ruins my day.
Also comments like see sams design is of no value without a url in fact it has negative value.
:(
I have changed the priority of this ticket to account for this but next time i'll just close it."
2302 1343820855000000 ross Just a quick note, if Rufus asks about editing about page, he's talking about this ticket.
2303 1334654727000000 ross feature-2303-rdf-export-command
2304 1335878372000000 seanh I'm setting the priority of this high just because it's a low-hanging fruit for this sprint, as it was almost done last sprint.
2304 1335885463000000 seanh Setting minor again after cleaning tickets out of this sprint
2304 1337073870000000 seanh Currently implemented in this branch https://github.com/okfn/ckan/compare/master...feature-2304-follow waiting for code review and merge into master
2304 1337158641000000 seanh Finished and merged into master
2305 1335879165000000 seanh Setting the priority to major as I think it's worth getting this done since the follow support (which is almost finished) is a bit pointless without this.
2305 1335885478000000 seanh Setting minor again after cleaning tickets out of this sprint
2305 1340879812000000 seanh Implemented by Sven
2311 1334837867000000 toby "most work has been completed
comments can be added to templates via ${h.disqus_comments()}
recent comments via ${h.disqus_recent()}
basic i18n works for languages except norwegian which reverts to english
needs ckan 1.7
awaiting clarification re urls changing from under us"
2312 1340188741000000 ross "We have a ticket for object ownership (group and package) #2548 which will make it much easier to implement this functionality of showing user owned datasets in a 'virtual group'.
Will close this in preference to the other ticket which is needed for other features too."
2313 1334756921000000 seanh "Note that c.new_facets has now been renamed c.search_facets.
Before this ticket is closed the old c.facets should also be removed from the template context. It's put there by ckan/logic/action/get.py:package_search(), which also puts the new c.search_facets there."
2315 1334919522000000 dread Done, with tests and bootstrapification of the form: [release-v1.6.1 b23b357]
2317 1340033433000000 kindly Getting replace as part of new theme.
2319 1340624083000000 ross Resolved with rework of UI
2322 1337339587000000 ross "+1 on these, although ../download instead of ../raw and ../view instead of ../viewer
"
2322 1337864855000000 ross "Viewer almost exists with the /embed link and so have just used that with a better default size to provide a prettier url. Can still use the width=&height= that are used in the /embed url.
"
2322 1339749795000000 rgrp When will this go live? Will this be in v1.7.1?
2322 1339750135000000 ross No, new new features in point releases, we only put bug features in unless there is an extremely pressing need.
2322 1339750208000000 ross "Going to try that last comment again.
No, no new features in point releases, we only put bug fixes in unless there is an extremely pressing need."
2322 1344086640000000 rgrp "Can I confirm this is in v1.8 ...
"
2323 1335516967000000 ross "Doesn't make sense in Organizations mode. Adding existing dataset would mean that dataset would have to already be in another organization.
Leaving this open though to fix in case we decide that datasets *can* be in more than one organization at a time."
2323 1340188400000000 ross Can create a new ticket if the requirements change.
2325 1335366188000000 dread Done in [release-v1.6.1 6343a9c]
2329 1346664271000000 ross Moved to 2.0 branch for new UI.
2330 1336149571000000 icmurray "Implemented, providing two methods of passing an action's parameters. Either, each parameter is specified as a url parameter, eg
{{{
curl http://127.0.0.1:5000/api/3/action/package_search?q=police
}}}
or, for cases where a more complex data_dict is required, the data_dict can be parsed as a json-encoded dict in a single url-parameter called ``data_dict``, ie:
{{{
curl http://127.0.0.1:5000/api/3/action/package_search?data_dict={'q':'police'}
}}}
As part of this ticket:
- I noticed that all module-public members of the action modules were being loaded as actions (including functions imported from other modules). So I've cleared up the action modules to prevent this.
- I've cleaned up the action modules to use ``get_or_bust()`` when accessing a passed in data_dict outside before it's been validated by a schema. This provides better error messages for clients.
- provided a decorator, ``@side_effect_free``, for users implementing their own actions through the IActions interface to be able to expose their custom/overridden actions as a GET request through the action API."
2330 1336387328000000 icmurray "Replying to [comment:3 icmurray]:
>
> or, for cases where a more complex data_dict is required, the data_dict can be parsed as a json-encoded dict in a single url-parameter called ``data_dict``, ie:
>
> {{{
> curl http://127.0.0.1:5000/api/3/action/package_search?data_dict={'q':'police'}
> }}}
>
Another option is to have the data_dict json-encoded in the GET request's body. This is possible, but
1) Although permissable, it's not, as I understand it, conventional to have a GET request's body form the options of the request.
2) Caching proxies won't inspect the body of the request, only the URL.
"
2331 1337782689000000 kindly "This is a wontfix. I think terms should be ored by default. All modern search engines work like this.
If there is an issue due to relevancy (i.e if you type mulitple words and your result not coming near the top) then we should use this examples so we can tweak the results."
2331 1338390995000000 rgrp "@kindly I really don't get this but I may understand poorly. Google does not work by or-ing does it (as i add things to the search query I get *less* results not more ...).
It also makes it really hard to find stuff: as I add query terms to my query I get more stuff not less which makes for terrible finding of stuff - I appreciate that there is a scoring aspect but that seems secondary."
2331 1338455981000000 kindly "Scoring is primary in my opinion. Who cares if you have 1000 results if the top few are yours.
If things are hard to find we need to change our relevancy first. So if you have examples of where you think the scoring is wrong then please make a ticket for that.
Google, I imagine, just has a cutoff of anything under a certain score not being shown. We could do that as well but it would take some working out what we wanted that score to be.
Full ""And"" queries also limit any accidental discovery, especially of rare terms. If you do not get your search exactly correct then you get nothing, which is bad.
Obviously you can still AND things or +things.
The correct solution to this is adding a minimum match parameter which is a middle ground. eg you can say that you want to match over half of the terms. ""thing1 thing2 thing3 thing3"" means you have to match at least 2. There are many options described here. http://wiki.apache.org/solr/DisMaxQParserPlugin. You can change this in an extension if you want it just requries adding an mm field to the before_search in the Ipackage controller. I do not personally think we should change it as default.
I am closing it as wont fix as its trivial to change and is a philosophical difference not a technical one."
2331 1343814758000000 rgrp I'm re-opening this. This is a reasonably significant UX problem -- not just a philosophical difference. It is just *weird* to get *more* results as I add search terms rather than less. Please can we have this fixed ...
2331 1343862230000000 kindly As I said I do not not agree with it being a UX problem or *wierd*. *wierd* is not acceptable in response to a thought out comment. So I am closing it again.
2331 1345191494000000 rgrp "Re-opening yet again. This is something brought up by entire client team on several occasions :-)
We really do need to fix this as it is really poor UX. To give, yet another, example:
http://datahub.io/dataset?q=thatcher+wages
This returns 15 datasets yet only one of them has any mention of thatchers (the first one!). (The others are there because they match wages). This isn't what I expect. I expect *only* to see datasets that match *both* terms ...
"
2331 1345192722000000 ross "This is the dullest game of ticket tennis ever :D
Google don't OR the results, they have an explicit operator for ORing query terms and default to ensuring that all terms are present (in one form or another) in the documents. I understand the argument about cut-offs and relevancy scores, but for a large search engine they have the benefit of scale both in resource and documents. I spent 6 months of my life tweaking weightings for n-dimensional vector space search, I don't think it is likely to be a number we can, or will find by accident.
Given that Marcus also got confused by the current behaviour I think we should either:
a) Make it configurable
b) Just change it to what has been asked for
Don't care which.
"
2331 1356474344000000 rgrp Now transferred to github at https://github.com/okfn/ckan/issues/258
2332 1337582891000000 rgrp @rossjones / @amercader: can you do a full review and generate sub-tickets as necessary (obviously may not all be this sprint ...)
2345 1335878420000000 seanh Setting priority high as we want this done by Friday for the 1.7 release.
2345 1336492725000000 seanh Work on this has started in branch feature-2345-action-api-autodocs
2345 1337962454000000 seanh Done but now I need to fix readthedocs again
2346 1338205553000000 johnglover Will create a separate ticket for 'organization' to 'publisher' renaming, the rest are done.
2347 1335962835000000 ross [ ] Check that dataset privacy can be taken into account.
2347 1336039458000000 ross Keeping track of UI changes required, and copied Aron
2347 1337238738000000 rgrp "This looks great Ross. Couple of things not included (which I have taken liberty of adding to the description as well - you will probably want to tidy up):
* Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. ""Apps, Ideas, Visualization and other material related to this dataset"") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!)
* Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says
* What it does
* How to enable (and perhaps includes a nice-screenshot!)
* How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...)
* Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...)
* Clearly mark this old extension as deprecated from v1.7 forward: https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7)
"
2347 1339602763000000 shevski "Name-wise: what do we think of 'Related Media' - that suggests news articles, apps & visualisations to me
I'n not sure about relating 'ideas' to specific datasets. Looking at the way ideas work in DGU & pd.eu - it's more general and portal-wide so should flesh out user stories for this before adding ideas to related extension"
2347 1340613938000000 ross Needs some UI work now, have changed status to blocker which appears to be the only way to denote something as blocked (awaiting resource).
2347 1343126819000000 icmurray "Small fix to create action, outlined in [1] to be done. Said I'd do it, so moved ticket over to myself.
[1] https://github.com/okfn/ckan/pull/62"
2348 1338193014000000 ross "A fairly good start has been made on this at https://github.com/okfn/CKANClient-J
Lowering priority for completion for now"
2349 1336037341000000 ross Openlink hoping to get RDF dump into Virtuoso today.
2349 1336555121000000 ross Chased Openlink today for update
2349 1338193148000000 ross Initial DNS change doesn't seem to have taken, chasing Nils to see if he can take a look.
2349 1339075361000000 ross "Created symlink from /var/www/nginx-default/dump to the rdf output folder, accessible via http://s055.okserver.org/dump/ with daily dump available at http://s055.okserver.org/dump/rdfexport.tgz
"
2349 1339428913000000 ross "Notified Hugh that http://datahub.io/dump/rdfexport.tgz will contain all of the RDF files every 24 hours if they want to fetch and import them.
Have asked to make sure that this is satisfactory."
2351 1340615641000000 icmurray Scripts and docs in https://github.com/okfn/ecportal-server-setup
2353 1336037264000000 ross Started the user stories at https://docs.google.com/document/d/185sWPEO9cn1SbquH83UFcLme7xKdgfhCNqOo7pdsRKw/edit
2355 1336731240000000 toby "ckan.plugins.toolkit introduced to help control access to ckan for extensions. This should reduce breakage for extensions using this.
ITemplateHelpers allows extensions to add helper functions so templates can be less reliant on Genshi filter"
2355 1338212946000000 toby "Need to do documentation which is #2362 so has own ticket
closing"
2358 1336641577000000 amercader Took longer than expected due to last minute changes and problems with the packaged version.
2361 1337016768000000 amercader I actually spent some more time of this because I upgraded PDEU to CKAN 1.7, which was needed to fix the search index and a good testing for the release as well.
2362 1341314244000000 toby "first part complete
part2 is the auto generation merge that is for 1.9"
2362 1342085420000000 toby need to get the auto-documentation done now
2364 1335890204000000 ross Review the FRY app and investigate alternative payment gateways.
2364 1336483769000000 ross Start of doc listing payment gateways is at https://docs.google.com/document/d/1E4lX6RnRLDO0ioieKRpVs9a-My9nh6lBjvVahWDm3SY/edit
2365 1336492342000000 ross See also #1165 and #1096
2365 1340266964000000 ross After discussion with DR, it was decided that multisite wasn't something we would approach now, instead going for individual (shared codebase) installs.
2366 1336395238000000 dread For my site I need the Authorization Groups sections gone from the authz pages, for v1.7.1, so I've made these check for the presence of any Authorization Groups in commit: [master 8947302]. So it's another useful stepping stone to getting rid of them cleanly.
2366 1336555441000000 ross I think the UI aspects have been removed in 1.7 with hopefully the rest of it going in 1.8 - assuming we have a clean migration script.
2366 1336560445000000 dread "Thanks Ross. It's the dataset authz and admin authz pages which still need Authorization Groups removing. e.g. http://thedatahub.org/dataset/authz/conservative-party-uk-donors-by-sector
and
http://thedatahub.org/en/ckan-admin/authz"
2366 1338215334000000 ross Need to make sure there is a clear migration between authgroups and organizations if desired.
2367 1336035031000000 aron.carroll Fixed in ckanext-datahub@4aa76740e3def64a80171df4417c195210f0eb25
2371 1338481174000000 toby implementation trial in ckan demo branch #2375
2371 1340038431000000 toby awaiting release and demo theme merge
2371 1345209675000000 toby closing as part of 2375 branches
2372 1343220160000000 aron.carroll These are now in https://github.com/okfn/ckan/blob/b31109b628a21072ec0c2dec2a5d5a1c2cc0ed9c/doc/coding-standards.rst#javascript-coding-standards
2373 1336142514000000 dread Fixed in https://github.com/okfn/ckan/commit/8b7141cd2ff619a0e35aaa31bbc1d17156647498 on master (and also on release-dgu1).
2374 1340287976000000 ross "count is returned at the top level of the dict (at the same level as results)
Was added by dread on 2012-05-04, must have been in a recent patch."
2375 1338292898000000 toby stealing this ticket
2375 1340710303000000 toby moving milestone
2379 1340033026000000 ross Implemented in master
2380 1337868169000000 ross "datagm is VERY broken with 1.7
Waiting for datasuite UI changes."
2380 1340627558000000 ross Dupe of #2545
2383 1337872916000000 icmurray "This helper function won't parse UTC offset if passed a UTC offset as ""+01:00""; but incorrectly parses the UTC offset ""+0100"" as 100 microseconds.
Resolution:
- I've decided to keep the behaviour of not parsing UTC offsets because we currently don't store timezone information in the database; and I think that either dropping the tzinfo, or adjusting for it would appear to be ""magic behaviour"" from the user's perspective. Also, as this is a helper function, I think it makes sense to keep the behaviour as is, even if it's slightly non-standard.
This fix [1] maintains all the old non-ISO behaviour:
- not allowing UTC offset to be specified.
- arbitrary delimiters between date and time values.
- microsecond precision.
But won't parse ""+0100"" as microsecond precision.
I tried using a couple of python modules for parsing date strings; but none maintained the same behaviour.
[1] https://github.com/okfn/ckan/commit/d6c9e48f8df5e926abd3fb1a1607f4f5a30075b4"
2384 1337100810000000 dread Done in 0bc06c6495c749d6b481164d59cceeca873acefb
2386 1337967006000000 seanh Done. ckanext-spatial and ckanext-qa are not activated because they don't work with the current version of CKAN. datastorer is not installed yet.
2389 1337237502000000 rgrp Just to note these were 2 *different* options. '''Either''' you do the fallback approach '''or''' we turn off DataStore by default and only use if enabled (which happens due to explicit enabling of DataStorer turning it on ...)
2390 1337100788000000 dread Done in d1c2ed08c90c2d94e1b2ced86e8758208a61ee96
2391 1338397982000000 seanh I've told them how to do it themselves and offered to help if they get stuck
2394 1337161917000000 dread Done in [master b486244]
2398 1337963342000000 seanh I'm closing this because we've got a whole bunch of tickets from the dev meetup about fixing the dependencies and requirements file that should deal with this problem
2399 1340706702000000 amercader Fixed in 493589aaf
2400 1338205947000000 toby In testing this is not replicable due to lack of info so closing
2401 1337273042000000 dread Done in master 112e405d0116bdcf196062520e3474e231337996
2402 1337302474000000 kindly cset: 12da5e9effeeb1aca0df321c355d8438647ef426
2403 1337302700000000 kindly cset: 05144f8621ee719c345373934e70719f46e87cf6
2405 1337583749000000 rgrp Assigning to ross for review and assignment.
2405 1338212357000000 ross Suggest Sam might be able to provide you the icon
2405 1338213564000000 icmurray "Questions for client team:
1. Did you have anything in mind for improving the ""Powered by CKAN"" text and icon? I'll sort out the vertical alignment of the text and icon; and see if Sam has a better icon.
2. The branded section itself is already customizable by a snippet (https://github.com/okfn/ckan/blob/master/ckan/templates/snippets/data-viewer-embed-branded-link.html). Is that enough? Or are you asking for something else?
3. Personally I think the URI should use the resource uuid, and not name. And from recent discussions on ckan-dev, I think the consensus is the same. How important is this for you? Perhaps it's something that could be overriden in the datahub extension."
2405 1340097846000000 icmurray Bump: Questions for client team above...
2406 1337766385000000 toby "I installed CKAN yesterday afternoon from source, using the instructions at
http://docs.ckan.org/en/ckan-1.7/install-from-source.html
Most things worked pretty smoothly and, luckily, when they didn't I
was lucky enough to have David Raznick on hand to help sort them out.
As a result I've made the following notes on where the doc could be
improved.
For some ghastly reason I am running Linux Mint, which is some variant
of Ubuntu. It's not 10.04, so no package, hence the source install.
Sec 2: Not wanting a mysterious file called 'pyenv' in my home
directory, I created a directory ~/ckan and typed 'virtualenv pyenv'
from there. Everything worked fine. But if I had needed the 'tip' it
would have been a little confusing. It would also be worth reminding
the user that the dubious package is one they've supposedly just
installed, so if that went OK, they won't need to worry about this.
The tip also isn't clear about under what circumstances I'd need to do
the 'easy_install pip' - I didn't need to (is that also only when
python-virtualenv is missing?)
Sec 4: This worked, but the example command for installing a
particular version is confusing. It says it is for 1.5.1, but
replacing '1.5.1' with '1.7' fails because there is a rogue 'c' in the
command.
Probably it would be better to use the latest version (1.7) as the
example anyway. Or is there a way of asking for the latest released
version?
I understand this checks out the whole Git repository with all history
- I can't imagine why I'd need to do this to build a particular
version. But David assures me it's very sensible.
Sec 6: This worked, but it wasn't very clear what the purpose of the
'List existing databases' step was.
Sec 8: So now I had to try to get Solr working using the instructions at
http://docs.ckan.org/en/ckan-1.7/solr-setup.html
This didn't work at all, in ways I can't now remember, to do with
Java. In the end David suggested using Tomcat rather than Jetty (don't
ask me what these do, but they have something to do with getting Solr
to work ...) and this turned out to be much more straightforward to
install and get working, with two caveats (below). David suggested
maybe Tomcat should be the default suggested in this doc.
- One problem with Jetty was to do with using the wrong version of
Java, which seemed to be fixed using ""sudo update-alternatives
--config java"". It didn't fix Jetty - not sure if it would also have
been needed with Tomcat.
- Soft linking the schema file didn't work - it seems Tomcat doesn't
like soft links. We made a copy.
After this everything worked, and I now have my very own CKAN. Hurrah!
Mark
"
2406 1337766413000000 toby "The 'post-installation setup' instructions at
http://docs.ckan.org/en/ckan-1.7/post-installation.html
claim to be for either package or source installations, but only work
as stated if you have done a package installation. E.g. after a source
install, ""paster --plugin=ckan user add admin
--config=/etc/ckan/std/std.ini"" doesn't work (the ini file is
somewhere completely different), but just typing ""paster user add
admin"" from the ckan directory is fine.
The instructions should also mention that the virtual environment
should be activated.
Mark
"
2407 1340633440000000 amercader This will require some thought. Bumping to 1.9.
2408 1341497601000000 aron.carroll This will be done as part of adding javascript in phase 2
2408 1343657014000000 aron.carroll Fixed in df502fd9269859c73458f4ad8de75939220e0315
2409 1337688965000000 ross This is in the Package controller and may be related to upcoming ticket on removal of autoneg.
2410 1338193633000000 ross Tim Lebo and Keith Alexander are both now looking at how they can implement this themselves, so for now this is wontfix. We can reopen this as part of any future work with VoID
2411 1338210872000000 toby This is a long term goal
2414 1337869578000000 ross "LXML is currently used for markdown_extract and check_solr_schema_version
Changing check_solr_schema_version should be possible,
Used in markdown_extract to just strip HTML tags."
2414 1338471374000000 ross Merged by kindly
2416 1338464207000000 toby commit 37622d5 does this but probably in the wrong place but works for now. Need to revisit at some point.
2416 1340710252000000 toby moving to phase 2 as not causing problems but still needs proper fix
2417 1337864155000000 toby fixed in demo-theme branch
2418 1337875172000000 toby "all facets now shown and selected have -
@aron styling of active facets with long names breaks. we could make the names shorter in snippet/facet_list.html via truncate or make the selected taggy thing wider"
2419 1337863024000000 toby "@aron
the reason for this is that controller/package.py renders the form `package/new_package_form.html` as part of it's work and then makes this available a `c.form` to the template. I'm not sure I like this as an approach but it is how it is currently done. If you split your form into a new template `package/new_package_form.html` then you get all the errors etc.
I think for now we should do this and then look at if doing it all in one template pass might be more sensible.
reassign to me if you want me to do this else I'll leave it with you"
2420 1337950422000000 toby merged into master
2422 1337945204000000 seanh Closing because I'm not able to reproduce and it turned out floapps had publisher auth on which was overriding his rights auth
2423 1338193199000000 ross Duplicate of #2414
2424 1338193269000000 ross Duplicate of #2413
2428 1341234933000000 seanh This awaiting merge, pull request: https://github.com/okfn/ckan/pull/46
2428 1341314475000000 seanh Merged. The only thing I didn't do was add a test case that runs pip freeze and tests that all the right versions of deps are installed.
2430 1338397815000000 seanh http://trac-hacks.org/wiki/MasterTicketsPlugin
2431 1338397906000000 seanh https://github.com/davglass/github-trac
2432 1338205526000000 aron.carroll "It would be great to use something like http://trac-hacks.org/wiki/SimpleTicketPlugin to remove most of the options when creating a ticket.
Suggestions to remove theme, repository, component and type unless they're actually useful."
2432 1338325120000000 shevski "Where do new tickets go to await triaging?
Is priority set by whoever creates the ticket? Does this get revised? Will we have a field for dev time estimate or where will this go?"
2433 1337960203000000 toby "the affected actions
===============
package_list
package_relationship_create
revision_show
group_list
package_update_validate
It would be nice to get this fixed and I'm loathed to suggest adding a version 4 hack.
If we make this change it will break things for existing users. This behaviour has existed since the action api was added as far as I'm aware
So should we
a) leave it forever and document
b) break things and fix it - when 1.7.1 or 1.8?"
2433 1338397268000000 toby after discussion it has been agreed to keep the existing behaviour and improve documentation as it is likely to break existing applications etc
2434 1341234522000000 seanh "We've moved to AWS hosting with docs generated by Jenkins instead, see:
http://trac.okfn.org/ticket/1254
There are still two issues to resolve:
1. Automatically updating the symlink so that docs.ckan.org always redirects to the latest stable version
2. Finding somewhere to put links to all the other versions"
2434 1350296272000000 amercader Seems to be working again, closing.
2437 1338484676000000 seanh "Please add to the file started to doc/coding-standards.rst
Also, for the Python coding standards see http://wiki.okfn.org/Coding_Standards as a starting point"
2437 1338487089000000 seanh "I've also added the commit message guidelines as they reference a trac plugin I'm going to get installed.
I'll leave the rest to you.
I remembered that http://wiki.ckan.org/Main_Page has a bunch of pages under Developer Resources that should probably be updated and merged into these coding standards and deleted from the wiki"
2438 1338212234000000 icmurray "Proposed solution:
Add a new q field which can be specified multiple times. Each q parameter is a string: `{field}:term`. `:` characters are escaped in the `{field}` by `\:`.
Mark the `fields` parameters as deprecated."
2438 1340736112000000 icmurray "Ready to be merged [1]
Brief synopsis of changes:
- altered return type of the action: from Resource domain objects to dictized resources. This is required if this action is to be made accessible through the action api as the domain object is not json-serializable. The 'old' return type is still available by passing in a flag in the context. In fact, this is required as the search layer uses this action, and deals with the domain objects directly.
- decided that escaping the ':' (as described in above comments) wasn't necessary as resource fields shouldn't have ':' characters in them anyway.
- improved action's docstring
- 'fields' parameter has been deprecated ( see #2603 ). Use of it by the search layer has been removed in favour of the new 'query' parameter. But it still works, and correctly maintains backward compatibility.
[1] https://github.com/okfn/ckan/pull/37"
2439 1338214158000000 icmurray "The `fields` argument doesn't provide any extra functionality that the `query` argument can't provide.
1. Figure out what the reason for this argument is, just to make sure.
2. Deprecate it, and change any internal uses of it to use `query` field instead."
2440 1339421746000000 aron.carroll "Has some styling as of b0153ba but needs content to be applied.
Toby, can you pull in the top 10 tags, as well as n of the most ""popular"" datasets and n of the most recent datasets. Actually just providing the functionality as helpers would be great.
Do we actually have ""featured"" datasets implemented at the moment?
Not sure what we're going to use for the box on the top left when on a new base CKAN instance."
2440 1339423966000000 toby "@aron,
tags what functionality do we want for this? I can just reuse the facet_list snippet but then we have all sorts of interactivity. Do we want this more a static list of tags (linked to somewhere but no add remove stuff)? seems more sensible I think
featured datasets I've no idea I'll try to discover"
2440 1339430956000000 toby "you may want to look at your css as the dataset is generated by the package_list snippet
I've made all packages popular for now"
2440 1339497700000000 aron.carroll Fixed this up, last thing I think we should do is ensure that the datasets passed to the homepage have descriptions.
2440 1339498655000000 toby "dataset descriptions is a data issue but I'll keep this open for now. We can probably do an exclude if needed.
"
2440 1339499074000000 aron.carroll Sure, but this is more about displaying nice complete examples on the homepage.
2440 1339581622000000 toby I am changing this to assigned not accepted as this makes trac easier to use
2440 1340373461000000 toby i'm closing we can make tickets for specific issues
2441 1338287243000000 aron.carroll Order by needs hooking up to form. Currently it's just a @
tag
Text styles
*This text will be italic*
_This will also be italic_
**This text will be bold**
__This will also be bold__
*You **can** combine them*
Lists
Unordered
* Item 1
* Item 2
* Item 2a
* Item 2b
Ordered
1. Item 1
2. Item 2
3. Item 3
* Item 3a
* Item 3b
Miscellaneous
Images
Format: 
Links
http://github.com - automatic!
[GitHub](http://github.com)
Blockquotes
As Kanye West said:
> We're living the future so
> the present is our past.
}}}"
738 1294408864000000 thejimmyg This needs more discussion. It is a separate problem from deleting a harvest source from the admin interface which is dealt with in #884.
738 1294409432000000 thejimmyg "Note from John Bywater on duplicate ticket #783 says: ""We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer."""
738 1297685069000000 thejimmyg "It may be handy to have a history of harvested documents and also be able to delete documents but still get their revisions.
The package comparison code will need to look at these revisions."
738 1299761436000000 thejimmyg This is now complete and on UAT.
739 1294247758000000 thejimmyg This needs to be implemented via an API call that Drupal can use
739 1296592842000000 thejimmyg Implemented today. May want a better implementation later.
740 1294408931000000 thejimmyg This will have to be done via the API though for DGU/UKLP. Should probably have it on the package interface in main CKAN too though.
740 1296592889000000 thejimmyg Duplicate of #739.
741 1291988087000000 rgrp Done in cset:49d5bd0a6a99 and cset:68522feabfeb among others. Documentation of progress on http://ckan.okfnpad.org/plugins
742 1288038147000000 johnbywater Moved from sprint 1.3.0
743 1294412807000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
744 1294412813000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
745 1294412820000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
746 1294412827000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
747 1294412834000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing.
748 1294412916000000 thejimmyg Merging with #676.
748 1294412976000000 thejimmyg Sorry, that should have been #691.
749 1294412986000000 thejimmyg Merge with #691
750 1289483616000000 johnbywater Referencing ticket #728 has changed sprint.
750 1294408970000000 thejimmyg How would this be done? Is it part of the CSW spec?
750 1296592940000000 thejimmyg Duplicate of #728
752 1294408472000000 thejimmyg Duplicate of #712. See #711 now.
753 1289209646000000 johnbywater Moved from sprint 1.3.2
753 1289815657000000 johnbywater Moved from sprint 1.3.3
753 1294247922000000 thejimmyg Not an explicit requirement yet so closing ticket. Will re-open if needed.
754 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
754 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
754 1294408726000000 thejimmyg "We may not need to do any filtering in practice but if we did, the implementation would have these stages:
* Add filter attribute to harvest source entity
* Add filter field to harvest source form
* Construct and send filtered CSW GetRecords request
Thus these tickets can now be marked as duplicated: #755 #756 #759."
754 1294409145000000 thejimmyg "Also:
* Change API documentation to indicate harvest source entity has filter attribute
* Change ""CSW Get Records"" request to accept and use given CSW filter
Now #758 and #760 can be closed too."
755 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
755 1288264918000000 johnbywater Referencing ticket #754 has changed sprint.
755 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
755 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
755 1294408632000000 thejimmyg Duplicate of #754
756 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
756 1288264917000000 johnbywater Referencing ticket #754 has changed sprint.
756 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
756 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
756 1294408642000000 thejimmyg Duplicate of #754
757 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
757 1288264918000000 johnbywater Referencing ticket #754 has changed sprint.
757 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
757 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
757 1294233016000000 thejimmyg Don't fully understand this ticket. Will return to it.
757 1296593448000000 thejimmyg Duplicate of #754. If we added filtering we'd need to write a migration script anyway.
758 1288041215000000 johnbywater Referencing ticket #754 has changed sprint.
758 1288264917000000 johnbywater Referencing ticket #754 has changed sprint.
758 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
758 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
758 1294409053000000 thejimmyg Also a duplicate of #754.
759 1289209646000000 johnbywater Referencing ticket #753 has changed sprint.
759 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
759 1294408652000000 thejimmyg Duplicate of #754
760 1288041238000000 johnbywater Referencing ticket #759 has changed sprint.
760 1288265044000000 johnbywater Referencing ticket #759 has changed sprint.
760 1289209646000000 johnbywater Referencing ticket #759 has changed sprint.
760 1289815657000000 johnbywater Referencing ticket #753 has changed sprint.
760 1294409111000000 thejimmyg Duplicate of #754
763 1288606918000000 dread Referencing ticket #441 has changed sprint.
763 1292775248000000 thejimmyg "Notes from my discussion with David a while ago:
Following irc discussion, it looks like read-only mode is simply
achieved by Apache config:
* 503 for PUT/POST operations - stops writes
* 503 for GETs to URIs containing: /edit, /create, /new, /authz -
stops providing forms that lead to a write
* setenv CKAN_READONLY=""Undergoing maintenance 12.00 UTC for one hour""
which can be picked up by CKAN to be displayed as we see fit in the
future. Friedrich's current IATI sprint may link into this and grey
out edit links etc."
764 1289209648000000 johnbywater Moved from sprint 1.3.2
764 1289815657000000 johnbywater Moved from sprint 1.3.3
764 1310133334000000 shevski will be done as part of #1094
765 1288606918000000 dread Referencing ticket #441 has changed sprint.
766 1289815658000000 johnbywater Moved from sprint 1.3.3
766 1294248066000000 thejimmyg I haven't heard this mentioned yet, but yes, let's try to implement it if possible. Appears to expose a CSW interface? http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#ext_csw_clnts.htm
766 1296593519000000 thejimmyg This is a duplicate of #767
767 1289209649000000 johnbywater Moved from sprint 1.3.2
767 1289815658000000 johnbywater Referencing ticket #766 has changed sprint.
767 1294409336000000 thejimmyg "Our CSW code should be able to cope with ArcGIS too though surely? Worth testing though.
To do this we'd need to:
* Obtain example ArcGIS CSW server
* Write a test
Now we can close #768 and #769."
767 1296593504000000 thejimmyg "I haven't heard this mentioned yet, but yes, let's try to implement it if possible. Appears to expose a CSW interface? http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#ext_csw_clnts.htm
"
767 1304936413000000 thejimmyg We are now live and have used a variety of servers in testing.
768 1288190630000000 johnbywater Referencing ticket #767 has changed sprint.
768 1289209649000000 johnbywater Referencing ticket #767 has changed sprint.
768 1289815658000000 johnbywater Referencing ticket #766 has changed sprint.
768 1294409308000000 thejimmyg Duplicate of #767
769 1288190630000000 johnbywater Referencing ticket #767 has changed sprint.
769 1289209649000000 johnbywater Referencing ticket #767 has changed sprint.
769 1289815658000000 johnbywater Referencing ticket #766 has changed sprint.
769 1294409318000000 thejimmyg Duplicate of #767
770 1289209650000000 johnbywater Moved from sprint 1.3.2
770 1289815658000000 johnbywater Moved from sprint 1.3.3
770 1294413390000000 thejimmyg "Running CLI harvester command without arguments or with --help would return help for the harvester (currently crashes).
We can therefore close #772, #773 and #774"
770 1296593925000000 thejimmyg I don't care about this. It is too trivial. If I get around to fixing it they great but I don't need it logged as a ticket.
771 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
771 1288190682000000 johnbywater Referencing ticket #770 has changed sprint.
771 1288190711000000 johnbywater Referencing ticket #770 has changed sprint.
771 1289209650000000 johnbywater Referencing ticket #770 has changed sprint.
771 1289815658000000 johnbywater Referencing ticket #770 has changed sprint.
771 1294413256000000 thejimmyg "This works for me.
{{{
paster help
}}}
and
{{{
paster --help
}}}
both show help text for me.
"
772 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
772 1288190682000000 johnbywater Referencing ticket #770 has changed sprint.
772 1288190711000000 johnbywater Referencing ticket #770 has changed sprint.
772 1289209650000000 johnbywater Referencing ticket #770 has changed sprint.
772 1289815658000000 johnbywater Referencing ticket #770 has changed sprint.
772 1294413313000000 thejimmyg Merging with #770
773 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
773 1288190682000000 johnbywater Referencing ticket #770 has changed sprint.
773 1288190711000000 johnbywater Referencing ticket #770 has changed sprint.
773 1289209650000000 johnbywater Referencing ticket #770 has changed sprint.
773 1289815658000000 johnbywater Referencing ticket #770 has changed sprint.
773 1294413400000000 thejimmyg Merging with #770
774 1288190596000000 johnbywater Referencing ticket #770 has changed sprint.
774 1288190682000000 johnbywater Referencing ticket #772 has changed sprint.
774 1288190711000000 johnbywater Referencing ticket #772 has changed sprint.
774 1289209650000000 johnbywater Referencing ticket #772 has changed sprint.
774 1289815658000000 johnbywater Referencing ticket #772 has changed sprint.
774 1294413412000000 thejimmyg Merging with #770
775 1295260144000000 pudo fixed in cset:98c9997e80e8
776 1297066840000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/3/avoid-generating-vdm-warnings
777 1289209649000000 johnbywater Referencing ticket #764 has changed sprint.
777 1289815658000000 johnbywater Referencing ticket #764 has changed sprint.
777 1292586843000000 dread James did this a while ago.
778 1289209652000000 johnbywater Moved from sprint 1.3.2
778 1289815658000000 johnbywater Moved from sprint 1.3.3
778 1294416567000000 thejimmyg This is now done.
779 1289209652000000 johnbywater Moved from sprint 1.3.2
779 1289815658000000 johnbywater Moved from sprint 1.3.3
779 1294416581000000 thejimmyg This is now done.
780 1289209653000000 johnbywater Moved from sprint 1.3.2
780 1289815658000000 johnbywater Moved from sprint 1.3.3
780 1294416608000000 thejimmyg This is now done.
781 1289209653000000 johnbywater Moved from sprint 1.3.2
781 1289815658000000 johnbywater Moved from sprint 1.3.3
781 1294415081000000 thejimmyg This is a duplicate of #812 so closing.
782 1289218497000000 pudo "This has existed for quite a while, but misses documentation. Has been documented in cset:db18a16a5b4d
"
783 1294409399000000 thejimmyg Exact duplicate of #738.
784 1289815658000000 johnbywater Moved from sprint 1.3.3
784 1294410089000000 thejimmyg "In order to have end to end testing for UKLII we need to:
* Review document from PP with the different things we need to test
* Set up a GeoNetwork instance with sample documents to be tested
We may also want to test ArcGIS, see #767"
784 1304936251000000 thejimmyg The UKLP test manager is looking at a new test proposal, we have had approval to go live without this, we can re-open later if necessary.
786 1289815658000000 johnbywater Moved from sprint 1.3.3
786 1294410004000000 thejimmyg Duplicate of #884.
787 1289295207000000 pudo "Alternative:
* The Drupal system will expose a resource at data.gov.uk/profile which wil contain the reqesting user's nickname, fullname, email etc. in JSON (or XML) form.
* The Drupal system will also run an OAuth server (and provide /request_token, /access_token, /authorize).
* CKAN's login form will be rewired to initiate a client OAuth request on the /profile resource.
* Recognizing CKAN's callback URL at catalogue.data.gov.uk, Drupal will automatically grant the request if a user is signed in.
* Upon return, a user is created, optionally using the OAuth access token as the users API key, thus making it known both to Drupal and CKAN.
* The authorizer will be extended to check access to the /profile resource for OAuth accounts (slow but safe).
Advantages:
* Well-known protocol, does not depend on cookies (which are strange and never behave as defined, or even the same in multiple browsers)
* Python code is available: http://oauth.googlecode.com/svn/code/python/oauth/example
* Drupal seems to have very complete support: http://drupal.org/node/296205
* Can be fully implemented as a plugin, using an OAuthClientController for callback and adding hooks to the Authorizer (perhaps need to reconfigure who.ini)
"
787 1289815658000000 johnbywater Moved from sprint 1.3.3
787 1303118054000000 thejimmyg The AuthAPI now exists as an IMiddleware plugin, we really need the permission system moved into CKAN before it is useful though and this depends on a refactor of the Auth system. See #1094
787 1315821118000000 thejimmyg "The joint authentication was implemented a long time ago and is deployed on catalogue.data.gov.uk. We'll build the authorisation layer in ticket #1326 so marking this as fixed.
"
788 1289815658000000 johnbywater Referencing ticket #786 has changed sprint.
788 1294410007000000 thejimmyg Duplicate of #884.
789 1294248289000000 thejimmyg Will, this means that we just get given a URL to harvest from and we need to determine whether it is a CSW or a WAF. Can you look at this please? The code may already do it, I'm not sure.
789 1296593257000000 thejimmyg Covered by #736 now.
790 1289815658000000 johnbywater Moved from sprint 1.3.3
790 1294409709000000 thejimmyg Duplicate of #736.
791 1294409723000000 thejimmyg Duplicate of #736
793 1289815658000000 johnbywater Moved from sprint 1.3.3
793 1294409836000000 thejimmyg "Anyone else know what this means?
I can't find changeset aa9aa32e00a9 in the repository so marking as invalid."
794 1289815658000000 johnbywater Moved from sprint 1.3.3
794 1294248216000000 thejimmyg This work is now underway in the underlying DGU implementation.
794 1297686491000000 thejimmyg Actually, for the timebeing we will match but not do anything with that matched information, until there is a clear use case. Publisher is simply the publisher for which the source was registered. Closing this ticket.
794 1300877517000000 thejimmyg I think we need to look at this again in relation to scotland.
795 1296593361000000 thejimmyg This is a duplicate of #794 because if we needed to do this it would be because we need to support harvest sources registered on behalf of publishers by agents,
796 1291640040000000 rgrp We no longer have an alpha page due to refactor of search and usage of search for list view (see ticket:847 and subtickets).
797 1289402982000000 rgrp Fixed yesterday night in cset:5b522c762e71
798 1289815658000000 johnbywater Moved from sprint 1.3.3
798 1294916973000000 dread This was done a while ago.
799 1294232675000000 thejimmyg "How can we tell a WAF document has changed, we simply need to re-harvest it to see surely?
Moving the issue to ticket #728 to be dealt with together."
800 1289483682000000 johnbywater Referencing ticket #799 has changed sprint.
800 1294245610000000 thejimmyg Discussion for this ticket is now at #728
801 1289483616000000 johnbywater Referencing ticket #728 has changed sprint.
801 1289483682000000 johnbywater Referencing ticket #799 has changed sprint.
801 1294233386000000 thejimmyg Perhaps this should be implemented differently so that each harvest attempt creates its own database entry with a timestamp and a status attached. We'll need to move to this when we move to a queue based system.
801 1296593735000000 thejimmyg I don't see why we'd really need to do this. Under the current system a job would be created just before the harvesting would be run so the creation date of the job is (give or take a few seconds) the date of the harvesting. If we moved to a queue system this would be done better anyway.
801 1297686706000000 thejimmyg We do have a requirement for this now. The job model has changed so that it is hidden from the user. We therefore want to know the timestamp the job started and the timestamp it finished. We'll therefore need migrations adding too.
801 1300196714000000 thejimmyg This will now be solved as part of the larger harvesting refactor. See #921.
802 1294233294000000 thejimmyg Merging with #801
803 1289489024000000 dread "I'm not sure a template is needed for the test script - surely repeated stuff can just be factored out?
Creating the model dump file:
* 'hg up' to ckan as it was at the last model migration
* 'paster db clean', 'paster db init', 'paster create-test-data'
* 'pgdump'
* 'hg up default'
* Check in the dumped file
Do we really want a script to mess with our repo like this? Isn't it just safer to do manually with good instructions?"
803 1314031451000000 dread We now use nose to test migrations instead.
804 1294233156000000 thejimmyg This is the document I drafted after discussions with each party which was approved by JF.
805 1298379084000000 dread "Migration tests added to buildbot using kindly's new nose option #965.
Also removed legacy system of migration testing in: ckan/migration/tests
and updated docs. cset:643673c7db3e"
806 1290071159000000 dread This is done now I think? Please list changesets, time taken vs estimate and close ticket.
806 1291128886000000 rgrp Duplicate of ticket:827
806 1291134978000000 dread This is not a duplicate of #827
806 1294916284000000 dread This looks finished
806 1324034356000000 dread This was introduced in CKAN 1.3
807 1297075372000000 rgrp wontfix as with traffic we have this is not a massive issue and these are very minor items (we can reopen later if this comes up again).
808 1297783658000000 pudo implemented in cset:8200247e74e9
809 1297075561000000 thejimmyg This is something to think about more in the future.
810 1297679091000000 pudo At the moment, this crashes the groups field for some mysterious reason. Since this is going to be redundant with the new forms and the ticket has a low priority, I'm bumping this back 2 weeks.
810 1300093797000000 rgrp Moving back to backlog for v1.4 as should be dependent on forms overhaul and seems to be problematic (and not that urgent).
810 1310128477000000 thejimmyg This ticket is over 6 months old so closing.
811 1294415158000000 thejimmyg See also related extras field tickets #811 and #893
812 1294658293000000 dread Just added stuff from duplicate #781
813 1294411047000000 thejimmyg "This is now done. The text reads ""Add a Package"" in the main navigation. Thanks!"
814 1294411109000000 thejimmyg "This is now done. It's the ""About"" link in the main application. Thanks!"
815 1294410951000000 thejimmyg This is a duplicate of #234. Please have a look at existing tickets or re-open closed ones rather than adding new ones so that we can keep the old discussions. I've added your point to #234. Thanks!
816 1311171466000000 thejimmyg "John, could you have a look at this one please. We'd need an entry in the logic layer for it and then some autocomplete JavaScript on the format field.
Can we standardise on jQueryUI going forward.
http://jqueryui.com/demos/autocomplete/"
816 1312192499000000 rgrp "What about idea of 'model' refactor for resource, specifically to have:
* format: human created format string with possible nesting e.g. zip:csv
* kind: file | api | example | service
* mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
* mimetype-inner: mimetype of innermost object (so for example would be text/csv)
See http://lists.okfn.org/pipermail/ckan-discuss/2011-April/001139.html"
816 1312212782000000 dread "I like rgrp's idea to change the model. But since it's complicated it would be good to have a UI.
e.g. You might make format field simply a drop-down with only some common options: 'CSV, Excel, Zip, Other'. If you go for Other, then a more substantial UI appears, to cover more filetypes, APIs, services and a text box for Other. And if you select 'Zip' then another set of options appears to say what is in the zip. etc.etc.
Another approach would be to have a background process downloading the data file and filling this in for you."
816 1312295371000000 johnglover "I have implemented the resource autocomplete so am closing this for now.
I agree that more information on the resource would be a good addition. It seems a bit of a different issue to just adding autocomplete though, with potentially far more changes to the codebase, so maybe it would be better discussed in a separate ticket/CREP. However, any new fields would presumably want to either be constrained or have an autocomplete and so can reuse the work done on this feature.
Cheers,
John"
816 1319804698000000 dread John did this in cset:1697dfa2552c for release 1.5
816 1319812324000000 dread And branch feature-816-format-autocomplete
817 1289997628000000 cygri "I'm not allowed to create attachments for some reason. Here's the screenshot:
[[Image(http://richard.cyganiak.de/2010/ckan/ckan-resources-table-proposal.png)]]
http://richard.cyganiak.de/2010/ckan/ckan-resources-table-proposal.png"
817 1297073724000000 thejimmyg This appears to be fixed now.
818 1305559442000000 dread 1.4 complete
819 1323171435000000 thejimmyg This issue is fixed. Further improvements to the UX of the autocomplete will be dealt with in other tickets.
820 1311325226000000 dread This has been fixed
821 1298486642000000 dread "Investigating several of these packages, it works for me (and David Raznick).
For example ni_013_migrants_english_language_skills_and_knowledge, one resource is seen created in the diffs, is displayed in CKAN, in the API and in the dumps. Yet looking at the dump from 17/11/10 when this ticket was created, the resource didn't have a URI, which by the current model is a requirement, so it suggests the data was fine underneath, but it had problems displaying this field, and is now fixed."
822 1290506354000000 dread "Fixed in cset:045cab10aa1d
Thanks for the patch!"
823 1290506116000000 anonymous Fixed in cset:3845a501ed5f
824 1290595559000000 dread Done on metastable for 1.2 release.
825 1290624449000000 dread Added plenty of info in cset:627249a1c102
826 1297414412000000 dread "It's great that the extras are added in-line with other Resource properties (as opposed to package extras which are a dict not off 'package' but off 'package.extras').
However, the resource extra field keys are defined in config option ""ckan.extra_resource_fields"". This config option should be removed - extras need to be entirely flexible for our purposes. (In the next ticket we should make it possible to add/remove both keys and values from the Web UI or API.)
It would also be good to tidy things in this direction: http://wiki.okfn.org/Coding_Standards
I've merged from default to the branch enhancment_826_resource_extra_fields ready for you."
826 1297415095000000 pudo "I want to strongly support david in his call for fully flexible extras: one of my use cases for them is to store the various bits of fallout from an archiving process, such as:
* last-status
* last-crawled
* last-etag
* last-expires
* last-md5
* failure-count
* fall-back-url
These are things needed to really archive the data well, but they have nothing to do with CKAN core ops. Essentially, the archiver is a seperate concern and it need not appear in the CKAN config."
826 1297416879000000 kindly "There is nothing to stop anyone from putting any extra attributes in the extra_info field dict. So any have the flexibility you need.
The config option is to add some fields that act in exactly the same way as python attributes, having the same semantics as them. i.e if you have an extra field called alturl, you can do obj.alturl = 'fdsffs'.
This is the best of both worlds as far as I can tell."
826 1297417900000000 kindly "I forgot to mention that he main advantage of the fixed fields, is that we can make them properly searchable i.e the values searchable. This currently does not work for package extra values as they are jsons.
I have added this searchability for the sql backend."
826 1297421022000000 dread Ah - I understand. I'd like it if you could make all the extra fields work as attributes and be searchable - is that possible?
826 1297423342000000 kindly "This would be too much of a hack. You do not want users overwriting any attributes on the object. If they called the attribute ""__init__"" it would write over the actual __init__.
"
826 1297429910000000 dread "Merged into default in cset:613d7bd5fc96.
Future tickets in this area include:
* #978 Full edit in Web UI
* #979 Edit Resource extras in the API"
826 1306766057000000 dread This went into ckan release 1.3.2.
827 1290766239000000 rgrp Fixed in cset:63a0c0223da4
828 1290696586000000 rgrp Closed in cset:66a3efe16ed4
829 1290697386000000 dread Some work towards this in cset:066b1b553595. No exceptions and test skip in place. Looks like formalchemy changed its interface.
829 1303118864000000 thejimmyg Is this still relevant?
829 1303122056000000 dread Rufus originally specified it. Reassigning to him to decide whether we still want it or not.
829 1303838115000000 rgrp This is a wontfix for the present.
830 1311180263000000 thejimmyg We can now implement themes via extensions or change them via config options. See https://bitbucket.org/okfn/ckanext-exampletheme/ for an example.
831 1290760864000000 rgrp Done over the last week (finally fixed weird trac no __dict__ bug today and enabled user accounts on wednesday).
832 1296334980000000 rgrp Done (~4w ago). See https://bitbucket.org/okfn/ckanext-stats and remove from core cset:311313e4afdb.
833 1295263447000000 rgrp Have repo https://bitbucket.org/okfn/ckanext-admin
833 1298889104000000 rgrp In progress now (sysadmin view and update nearly done).
833 1302276855000000 rgrp Authz subsystem complete.
833 1303236364000000 rgrp Two main tickets done so closing.
834 1291633657000000 rgrp "Alexander: Documentation is here: http://packages.python.org/ckan/api/version2.html#search-api
Those queries should work but I do suggest you use the stable version of CKAN which is v1.2 and we suggest you use that. If you still have problems please reopen. "
835 1291644820000000 pudo done in cset:d41b77099954
836 1291851041000000 rgrp Fixed in cset:2f6d54341b47 and branch feature-286-siteurl
836 1302882808000000 dread This has been a (minor) issue since release v1.3
838 1291299716000000 memespring Package page redesign: http://ckan.org/ticket/839
838 1291636351000000 memespring Merge css files http://ckan.org/ticket/846
838 1291721955000000 memespring Show welcome message: http://ckan.org/ticket/850
838 1291729819000000 memespring Prompt users to enter missing info - http://ckan.org/ticket/863
838 1291736461000000 memespring Search results changes: http://ckan.org/ticket/864
838 1291812318000000 memespring Add download formats to search results (http://ckan.org/ticket/866)
838 1295259773000000 rgrp I'm going to close this ticket as almost everything done and remaining ticket can be done on its own.
839 1291736541000000 memespring Done with the exception of the discuss/comments page. Plugin won't install on my setup. Fredrich looking into it.
840 1291325251000000 dread "dread:
] That's the proxy_cache decorator which sets a Beaker expiry time only.
] Whether that is on or off, you still end up run the
] ""etag_cache(cache_key)"" command in the package controller won't you?
] And won't that will either insert the Etags header or abort 304 for a
] repeat? I must admit I've not tried it, asking purely with the code in
] front of me and welcome you pointing out where I've gone wrong.
ww:
Right. So those etag calls predate the cache decorators and should
probably be either moved up into the decorator (e.g. use @ckan_cache
instead of @proxy_cache) or wrapped in a check for the config
parameter. (And @ckan_cache could be changed to use pylons' etag_cache
function rather than just setting the header...)"
840 1302694123000000 dread Basic on/off switch added, tested & documented in cset:0da189c9630e on default.
841 1300364333000000 thejimmyg See #995
842 1296468313000000 rgrp Change to awaitingtriage as definitely not critical.
842 1303474131000000 thejimmyg "As a user I come to a package:
Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
At the bottom is a section of the package display titled ""ToDo"" where I see a list of all toDos for the package most recent at the top
If I am logged in
see a form for ""Add to do"" at the top of the todo section and can add one straight away
I see a ""now resolved"" button next to each which goes green when you hover.
When clicked the todo fades away.
Otherwise I see a button that says ""login to add todo""
expands out the form
The form
One of the fields is category -> autocomplete the category (not constrained)
Add a description
Submit, the todo gets added via AJAX to the list at the top as the most recent todo
Model:
todo
id
package_id
todo_category_id (required)
description (required)
date=NOW()
resolved=False
todo_category
id
name
Prepopulate with: broken-resource-link, no-author, bad-format
"
842 1303474228000000 thejimmyg "
> Otherwise I see a button that says ""login to add todo""
>
> expands out the form
>
Actually rather than expanding the form, you will go away to the login page and come back to see the expanded form (question: how does this redirect you back to the bottom?)"
843 1291652696000000 pudo The real name is also requested via AX/SReg extensions in OpenID so for new users who are not using google, this should usually be filled in automatically.
843 1319710380000000 dread " * ""we should show the openid as well to distinguish between users with the same name."" - when ""Full name"" is not distinguishable, maybe best to display the unique 'name' field as a hover-over.
* ""on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name."" - yes you always got take to the personal details page. We should use a flash message at this point if they have not filled in the ""Full name"" field to suggest they click edit and do this.
* ""List is to long"" - this has been addressed - see http://thedatahub.org/user"
843 1319721601000000 dread "Ok I misunderstood this ticket. This is referring to adding a user in e.g. http://thedatahub.org/group/authz/energy-data
This UI seems to be updated. You start typing the name, full name or open id of the person and it has a dropdown that autocompletes. This seems to be sufficient for Will's points 1 and 3.
Would be still good to have a flash message on account creation to encourage people to add personal their Full name. This is similar to #1413 so I'll close this ticket and add it there."
844 1296340486000000 rgrp Looking at DNS this apparently has been fixed.
845 1291723492000000 dread This was completed on the feature-845-required-fields branch and merged into default in cset:3b5635ffaa7d
846 1291715212000000 rgrp Reopening and fix used illegal css ;) -- it is illegal to have an @import statement (as we do with extras.css) after normal css statements. We therefore need to move the core css back out into a ckan.css (or similar file).
846 1291719074000000 memespring It's been done on a branch and pushed to bitbucket: http://bitbucket.org/memespring/ckan/src/9e74c40ff073/ckan/public/css/style.css
847 1291639903000000 rgrp Done my best to pull all the solr related tickets and work together in one place.
847 1292844627000000 rgrp Still have ticket:669 and ticket:874 to do (431 probably won't be done for a while).
847 1295259902000000 pudo This is now functional, minus the index testing tool.
849 1291715179000000 pudo == #846
851 1292860957000000 wwaites " * link checker above uses the queue. queue not running generally
* quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest"
851 1292892515000000 wwaites "Ready to put into cron job. cf:
* http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions)
* http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions)
* http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi 1.2)
* https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update)"
851 1292894661000000 wwaites currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow...
851 1292941303000000 wwaites "urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp.
change the httpReq action to use http://curl.haxx.se/libcurl/python/"
851 1293025112000000 wwaites "Implemented curlReq that does a curl request and returns statements
similar to or analogous to httpReq. Require curate<=0.8
This in a cron job is sufficient to go through all the packages and update
them with a broken link tag now:
{{{
curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY
}}}"
852 1291723207000000 wwaites plugin documentation: http://packages.python.org/ckan/plugins.html
852 1295259620000000 rgrp Removing from sprint and moving to release as a master ticket.
852 1315821628000000 rgrp Moved dataset archiving into new ticket #1327 and this is therefore now done.
853 1291723143000000 wwaites done in http://bitbucket.org/ww/ofs need to merge back into main ofs repo
853 1294047550000000 pudo "Looking at the changeset this cannot be functional yet: where is the implementation of the policy document exchange. It seems to me like this is currently adding the actual credentials to the request (self.ofs.conn.add_aws_auth_header(headers, 'PUT', path)).
c.f.:
* http://doc.s3.amazonaws.com/proposals/post.html#Form_Fields
* http://code.google.com/apis/storage/docs/reference-methods.html#postobject
"
853 1294594581000000 wwaites "we don't need a policy document exchange. it's simpler than that. the ""server"" instance has already permissions to upload. it just calculates the headers and such that are needed (based on the ""client""'s initial headers) and gives them to the ""client"" the client then uploads without knowing the ""server""'s credentials. The ""client"" never needs any of its own goostor credentials at all.
the only separate step is to make the widget readable by the world.
ticket #879 is to expose this as a small set of API calls."
854 1304351843000000 johnlawrenceaspden "Coverage now up to 84% and 81%. Remaining untested code is error conditions, which we decided weren't worth the effort of locking down.
fixed on feature-854-tests-for-authz-groups, now merged into default code."
855 1311176988000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. The auth code is in the process of a major refactor anyway.
856 1311177066000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
857 1311177075000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
858 1291734297000000 dread On branch feature-858-diff removed ckan.lib.diff as it was not being used. Merged into default in cset:f9ba1ae63ddd
859 1311177461000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
860 1291726067000000 wwaites http://bitbucket.org/ww/ckanrdf/changeset/67df6dc33ec4
861 1311168845000000 thejimmyg This ticket has been open for than 6 months so I'm closing it.
862 1291726261000000 wwaites not to mention sparql endpoint...
863 1295259827000000 rgrp Removing milestone as not certain when we'll do this.
863 1338206455000000 ross UI has changed rather a lot in last 18 months, so I killing this bug.
864 1291741028000000 memespring Havent implemented all text changes because worried about impact on other themes.
865 1294414639000000 thejimmyg Isn't this in there now? Can we close? Thanks
865 1295259773000000 pudo This has been fixed on the respective branch and merged into default..
867 1291916278000000 dread Done in ckanclient cset:d40fb101aba9
867 1292410515000000 dread Also ckan cset:c5b018bfe9bb
867 1299866685000000 rgrp This was a breaking change for loaders code. Obviously we don't have tests for that so would not have been noticed ... Fixed in cset:af81e54bd590/ckanclient
868 1292890251000000 kindly@gmail.com "Below is a patch to make the tests run at least 2.5 times faster (about 15 mins on my old laptop). Instead of dropping the tables each time, it just deletes everything in them, using a low level connection. All the tests pass this way. It's a surprisingly clean patch. Here are a few points concerning it.
* I tested truncating the tables but it's slower. If there are any big tables in the tests this way is the fastest (faster than drop).
* The sequences (id columns) will start from where they left off.
* I also investigated making postgres template database and cloning it, but the complication was not worth it.
* sqlalchemy iterates the tables in reverse dependency order, which make this possible.
* I targeted rebuild_db as that what most of the tests I saw where using, however I have not checked all tests to see if they all are.
* There is a slight hack on the repo object to make sure it knows that ""clean_db"" is coming from the tests.
* I refactored init_db for code reuse.
* I have not done a version check. sqlalchemy >= 0.5 do this in a different way as outlined in the comments.
{{{
diff -r 7f2239b0f743 ckan/model/__init__.py
--- a/ckan/model/__init__.py Fri Dec 17 10:34:47 2010 +0000
+++ b/ckan/model/__init__.py Mon Dec 20 23:25:04 2010 +0000
@@ -41,6 +41,9 @@
def init_db(self):
super(Repository, self).init_db()
+ self.add_initial_data()
+
+ def add_initial_data(self):
# assume if this exists everything else does too
if not User.by_name(PSEUDO_USER__VISITOR):
visitor = User(name=PSEUDO_USER__VISITOR)
@@ -69,6 +72,26 @@
import migrate.versioning.api as mig
version = mig.version(self.migrate_repository)
return version
+
+ def clean_db(self):
+ # delete only added for tests
+ if hasattr(self, ""delete_only"") and self.delete_only:
+ self.delete_all()
+ else:
+ super(Repository, self).clean_db()
+
+ def delete_all(self):
+
+ self.session.remove()
+ ## use raw connection for performance
+ connection = self.session.connection()
+ ## sqla sorts in reverse dependancy order.
+ ## in >= 0.5 use reversed(metadata.sorted_tables()) instead of table_iterator
+ for table in self.metadata.table_iterator():
+ connection.execute('delete from ""%s""' % table.name)
+ self.session.commit()
+
+ self.add_initial_data()
def setup_migration_version_control(self, version=None):
import migrate.versioning.exceptions
diff -r 7f2239b0f743 ckan/tests/__init__.py
--- a/ckan/tests/__init__.py Fri Dec 17 10:34:47 2010 +0000
+++ b/ckan/tests/__init__.py Mon Dec 20 23:25:04 2010 +0000
@@ -55,6 +55,7 @@
import ckan.model as model
model.repo.rebuild_db()
+model.repo.delete_only = True
class BaseCase(object):
}}}
"
868 1292921428000000 sebbacon "See also #867
Thanks for the patch."
868 1292921456000000 sebbacon I mean #876, of course.
868 1293401808000000 anonymous Attached are the timings I have for the tests after I upgraded to 0.57 and after a few simple test tweaks. They do not include setup and teardown time at the class level as they are not assignable to individual tests.
868 1294753596000000 dread "I've merged in David Raznick's patches:
* no_autoflush_deletes.diff cset:2b9591172182
* postgres_speed.diff cset:fa1b7e3a4e0f
* vdm_purge_no_autoflush.diff vdm cset 8accdd0b9b7f
I've also merged in Seb's fork: cset:68d63fda4814 which closes this ticket, achieving test speeds of under 3 minutes!"
869 1292059662000000 rgrp Closed by cset:83734b5e251c which implements an IConfigurer interface.
870 1294862485000000 kindly@gmail.com "A patch is available here.
https://bitbucket.org/kindly/ckan/changeset/9a1d6f55587b"
870 1294914243000000 anonymous Merged into default in cset:54ae110094be
871 1292322361000000 wwaites "Regarding rkhunter -- yes, eu1 appears to be clean
Regarding the upgrade -- upgradede to 4.72 from backports which, looking more closely, appears to still have the privilege escalation bug but not the remote root exploit.
Regarding exim on other hosts, there is no reason for them to be running a full mta, something like ssmtp should suffice.
Also very worth the thought of moving to postfix. It's much easier to configure and I haven't known it to have any comparable bugs in the decade or so I've been running it. In fact I've never seen anyone actually use exim before..."
871 1292323656000000 nils.toedtmann "Re postfix: I second ww. I like to run some super simple local MTA (e.g. ""nullmailer"") on all but one server, using a central postfix (or a send-only GMail account) as smarthost. Am happy with postfix for >10years, it's straightforward and rock solid.
"
871 1296340558000000 rgrp This is not a ckan issue. should have been on http://knowledgeforge.net/okfn/tasks
872 1292957110000000 dread Done in ckanext cset:8a7e931ef37c
873 1292939274000000 dread Fixed in dgu cset:8bfb867e247a and ckanext cset:fabd31544a73
875 1297085261000000 pudo So far opting for route 1) (not implementing facets), therefore this can be closed!
876 1292891133000000 kindly@gmail.com "I have looked into this already so I can give you a head start. I am working on a project that uses many backends so I have some experience. So here is what I have found so far.
== nested transactions ==
VDM does not support sqlite, as it uses nested transactions. I do not think vdm needs nested transactions. It can use a flush instead. Here is the patch that works. All vdm tests pass.
{{{
--- a/vdm/sqlalchemy/base.py Sat Sep 11 23:06:26 2010 +0000
+++ b/vdm/sqlalchemy/base.py Mon Dec 20 16:16:34 2010 +0000
@@ -40,9 +40,8 @@
self.setattr(session, 'HEAD', True)
self.setattr(session, 'revision', revision)
if revision.id is None:
- session.begin_nested()
session.add(revision)
- session.commit()
+ session.flush()
}}}
== indexes ==
The index file 021_postgres_upgrade.sql in the migrate repository will not run as it uses syntax particular to postgres. Another will need to be made thats similar. sqlite does not support complex indexes like upper(text), so a work around will need to be found.
== unicode ==
The harvesting returns utf8 encoded strings and pysqlite dbapi only supports python unicode objects (as far as I can tell). There will need to be a process in converting all strings that get into the database with string.decode(""utf8"")
== dates ==
Have not looked into this one too much. However, as sqlite stores everything as strings the timestamps appear to be failing on conversion back into python.
I have solved the above two issues before by adding attribute extensions to sqlalchemy mappers to do the conversions without effecting too much code.
== in memory sqlite ==
Some tests need to change in order to make sure the database is created first because the database gets lost each time. In the tests that I have made pass, they run in about a seventh of the time as they do on postgres.
== Other things to keep in mind. ==
* Need a new flag in test.ini to remove full text indexing completely, or always use it with solr.
* There are enough incompatibilities between the databases that you would also want to test against postgres as well, at least before a release.
* I would probably upgrade sqlalchemy first, so you will not have to the changes twice. The new versions are significantly faster too.
* I have submitted a patch to #868 that makes the tests run about 2.5 times as fast and I think there are more low hanging fruit if the aim is test speed.
"
876 1292924064000000 sebbacon "Thanks for the info!
Re. nested transactions. I am getting repeated non-deterministic test failures against sqlite (and indeed postgres, but these failures appear more frequent against sqlite). One of them I seemed to be able to get rid of by eliminating the savepoint as per your first point. However, it appears that sqlite [http://www.sqlite.org/lang_savepoint.html does] support savepoints; to demonstrate it, the following test code appears to work in the latest sqlalchemy:
{{{
#!python
from sqlalchemy import *
from sqlalchemy.orm import *
db = create_engine('sqlite:///')
metadata = MetaData(db)
users = Table('users', metadata,
Column('user_id', Integer, primary_key=True),
Column('name', String(40)),)
users.create()
class User(object):
pass
usermapper = mapper(User, users)
Session = sessionmaker()
session = Session()
fred = User()
fred.name = ""Fred""
sue = User()
sue.name = ""Sue""
amy = User()
amy.name = ""Amy""
session.add(fred)
session.add(amy)
session.begin_nested()
session.add(sue)
session.rollback()
session.commit()
assert session.query(User).count() == 2
print ""OK""
}}}
So, while I agree they're not needed, I'm not sure they're a problem. What do you think? Also, have you seen non-deterministic test errors like this?"
876 1292939711000000 kindly@gmail.com "I have read quite a lot of people having problems with savepoints with sqlite and thought they were not supported on sqlalchemy. They are at least not consistant with postgres ones. I may well be out of date on this.
Here is an [http://groups.google.com/group/sqlalchemy/browse_thread/thread/dc9d1b61044bf730/65a62a33ec313842?lnk=gst&q=no+such+savepoint#65a62a33ec313842 example] even though its a bit old.
I did get some non deterninistic errors, the above seemed to fix them. A failed subtransaction is not handled well by sqlalchemy and I think this causes knockon effects due to the unresolved transaction. I would stay well clear of them entirely if possible.
What are the errors you are getting??
My 2 cents. ignore me at will...
I would think about using a different backend for testing than production. [http://stackoverflow.com/questions/2716847/sqlalchemy-sqlite-for-testing-and-postgresql-for-development-how-to-port look here].
If you want to support both then you should test on both.
There are simple ways to scrape a few more minutes off the tests. If you want real speed, then a multiprocess solution (with a database per core) would be sensible if a bit tricky.
"
876 1293188088000000 anonymous "Thanks for your feedback, very useful.
I don't really agree with the people in the linked discussion who say it's pointless testing against a different database from production.
The goal here is to make it easy enough for people to run as many tests as possible that they actually do so. Even 15 minutes is too long in that case. With sqlite we can get it in at under 5 minutes. I would also like to identify the longest running tests (which I would characterise as ""functional"" or ""integration"" tests and make them run as a separate suite, and then encourage a culture of writing true unit tests before functional tests, so that running unit tests can happen in 1 minute and be part of the regular development cycle.
That's no replacement for also running *all* tests periodically, and also running tests under postgres, which we can continue to do on the continuous integration server.
Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests."
876 1293218714000000 anonymous "I agree with all your points about testing apart form using sqlite, especially splitting out the functional tests and continuous integration.
> Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests.
Well there are two options
1. refactor the tests
2. refactor the code to use sqlite and postress
It is a value judgment as to which is more complicated. I personally think 2 is more complicated but may be wrong on that. The real danger with 2 is that you are needlessly adding complication to production code, with 1 you are only changing the tests.
Upgrading to sqlalchemy 0.5+ should happen first regardless. You will need upto date documentation.
There is another option too. Put the postgres data directory on tempfs/ramfs and turn off durability [http://www.postgresql.org/docs/9.0/static/non-durability.html here]. We would need a way to db init before the tests where run (or) at boot). This may be the best of both worlds.
Anyway Happy xmas!!
"
876 1294753889000000 dread Seb and David have completed this I believe. I've merged the changes into core CKAN in cset:68d63fda4814.
877 1297680579000000 rgrp Basic pass on an implementation (no permissions yet etc): https://bitbucket.org/okfn/ckanext-upload/changeset/9ae543f0645f
877 1298624165000000 rgrp "Various tidying in https://bitbucket.org/okfn/ckanext-upload/changeset/0fad7aa7aa97 (success messages, permissions on uploaded file - public-read) and completed permissions in https://bitbucket.org/okfn/ckanext-upload/changeset/a83ce00a1266.
Still need to integrate into general workflow (e.g. create a Resource on successful upload) but that is a separate item so this ticket is now done."
878 1314404621000000 rgrp A working version of this in ckanjs as of 2 weeks ago. Need to integrate into main CKAN.
878 1315820838000000 rgrp This is now done in feature-1294-ux-improvements-dataset. see e.g. cset:c6f7f5018b4f
879 1294604066000000 wwaites first cut: https://bitbucket.org/ww/ckanext-storage/src
880 1292945137000000 dread Fixed in ckanclient cset:e7c0af586367 and ckanext cset:82d974ab6860 with test in dgu cset:c0b2c5fd95ea
881 1294410574000000 thejimmyg "Hello fccoelho,
Could you please post the output you get from running pip?
Thanks,
James
"
881 1296335072000000 rgrp Closing due to lack of clarification.
883 1300196622000000 thejimmyg Now complete.
884 1293319984000000 wwaites "Implemented in feature-884-rmsource branch at https://bitbucket.org/ww/ckan/changeset/50cbfc5a6bee
"
884 1294247654000000 thejimmyg Related to ticket #665, reopening so that I can close it once I've got it working with Drupal too.
884 1294253563000000 wwaites see feature-884-rmsource in the main ckan repo deleting my own repo
884 1296592140000000 thejimmyg OK, I've implemented an API call for this too now and all is merged into default.
885 1293278564000000 wwaites Friedrich had some mixed experiences with owslib and some German CSW endpoints: http://pudo.okfnpad.org/geodaten
885 1293314219000000 wwaites "See: https://bitbucket.org/ww/ckanext-ows
impossible as of yet to test against live server from the OS "
885 1294253497000000 wwaites done with feature-885-owslib
886 1294916538000000 dread Being done as part of DGU ticket: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/757
887 1300196600000000 thejimmyg "This is done. See https://bitbucket.org/okfn/ckanext-harvest
At the moment we are not adding migrations to remove the harvesting tables. We are designing a harvesting refactor and will write migrations once the refactor is complete so that instances that use harvesting get upgraded, and those that don't get their harvesting tables removed.
Also, see this #1030 for the moving harvesting out of the REST API.
"
888 1294830297000000 Stiivi "Chages to Data Proxy:
* tests added with configurable list of known URLs
* use brewery for transformations (included reference to brewery framework in a new vendor directory)
* side effect: code to make google find external packages in vendor directory (from now on, all external packages should go there and be referenced from .hgsub if they are mercurial repositories)
* changed response contents: moved from 'headers' to root, renamed 'response' to 'data', added field list as 'fields'
* changed way of registering transformers (now class object is used instead of name)
* added 'encoding' and 'dialect' parameters for CSV
* added optional data audit (parameter 'audit')
Changes: https://bitbucket.org/Stiivi/dataproxy/changeset/fccbdd275be5
Data information: http://databrewery.org/doc/data_quality.html#brewery.dq.FieldStatistics
"
888 1307352537000000 thejimmyg I don't think any progress has been made on this for a bit so I'm assigning it to me.
888 1311773103000000 johnglover "Dataproxy / Dataapi now deprecated in favour of the combination of new QA archive / process commands and the webstore.
Changes in relation to Dataproxy / Dataapi:
* Currently only supports CSV files, but plans to add support for excel and google docs spreadsheets soon.
* Uses David Raznick's CSV parser instead of Brewery for parsing, handles messy CSV data better.
Changes in relation to old QA functionality:
* decoupled archiving (downloading) and QA process
* added a new 'process' command which parses downloaded files and adds them to a local webstore
Closing for now, any improvements/feature requests should be in tickets relating to either the QA functionality or the webstore."
889 1293715109000000 rgrp Done in cset:cdbb2e6128b3 (had added item to template earlier pulling from 'g' object but had not tied that up to config as yet). Also added test and also '''removed existing google analytics code''' -- so need to explicitly set this in config from now on.
890 1318529648000000 rgrp May want to close as invalid as obsoleted by more recent queue work.
890 1318599247000000 kindly Invalid due to #1397, We will be using celery instead.
891 1318602128000000 rgrp May only do link-checker and not do full storage in this sprint.
891 1320143424000000 johnglover "Almost finished (see http://github.com/okfn/ckanext-archiver).
Still to address:
- check headers to see if hash / cache / max-age / expires indicates that the resource does not need to be downloaded.
- add cache url to resource"
891 1320149841000000 johnglover "Added cache_url and cache_last_updated to resources after archiving.
Not checking for hash value in headers. This process will generally only run when a new resource is added or someone updates a URL, so we don't expect to be regularly downloading the same resource.
We will need something along these lines if this is running as a regular cron job, but in that case the logic will be added to the cron job itself (probably a paster command)."
892 1324314636000000 rgrp Moved to v1.6.
892 1324402480000000 johnglover "https://github.com/okfn/ckan/compare/bff538b%5E...d700680
Merged with master and deployed on test.ckan.org"
893 1298293527000000 thejimmyg We don't understand the use case for this requirement. Closing for now until a use case can be demonstrated.
894 1300196388000000 thejimmyg All test CSWs have been successfully harvested from. CSW harvesting is disabled on that source so we can't harvest from it. I don't think we need a ticket for this now do we?
896 1300217375000000 thejimmyg "Actually, I think this should be implemented instead by considering a different CKAN instance as a potential harvest source and then harvesting from it. By thinking of it this way in the first instance, we effectively get a read-only copy of other data in CKAN but one which will be kept up to date.
Marking as duplicate. Discussion can now carry on via #987 Common harvesting framework."
897 1294914333000000 dread Same as #870
898 1294659537000000 dread From David Raznick
898 1294662408000000 dread Merged in cset:89bf917f9b46
899 1294662417000000 dread Merged in cset:89bf917f9b46
900 1294662424000000 dread Merged in cset:89bf917f9b46
901 1294662466000000 dread Merged in cset:b8a649f51cab from Seb's changes.
902 1296638257000000 rgrp "Also confirmed in FF. David (R) and I have both had a go and fixing this and it seems to be a bit tricky (very odd css behaviour!).
Rather than directly fix this I'm thinking of doing a bit of rework of that set of menu tabs as they could be much nicer anyway and hence re-titling of this ticket."
902 1297078885000000 rgrp Fixed in cset:42102bf7f922 and cset:42102bf7f922 by switching to a new 'tabs' look that has none of these issues. (See branch feature-902-submenu).
903 1295265645000000 pudo This is now partially solved by filtering for deleted packages. As two further measures, we should also add a package deletion method to the Solr backend and skip deleted packages when indexing.
903 1296588070000000 pudo fixed in https://bitbucket.org/okfn/ckanext-solr/changeset/be4cffa9b987
904 1299840539000000 rgrp We're already now into improving the docs and ticket:927 is now reasonably detailed.
905 1328638536000000 dread "This works with the current SOLR search.
e.g. http://thedatahub.org/dataset?q=gr%C3%B6%C3%9Ften"
906 1324299793000000 rgrp Do we need to change in core code or just configure solr?
906 1328639465000000 dread "This was done in #1701 but only in a custom SOLR schema in the ecportal extension: https://github.com/okfn/ckanext-ecportal/commit/6682926d8895f146cdf1e52ab4fbead9b065af77
Can the ASCIIFoldingFilterFactory be added to core CKAN's SOLR schema for all CKAN users to benefit from?"
906 1328782087000000 amercader Yes, I don't think there will be any problem, and we won't need to create a new version of the schema as the change is backwards compatible.
907 1296335337000000 rgrp "Basic implementation done and deployed. However plenty to improve, e.g.
* Support more formats (use external systems for preview?)
* json (!)
* html (trivial!)
* sparql
* ...
* Do not display preview if no preview
"
907 1297072303000000 rgrp Closing this now and improvements can go in a new ticket.
908 1296403695000000 wwaites "see also from http://knowledgeforge.net/okfn/tasks/ticket/485
{{{
(pyenv)okfn@eu7:~/var/srvc/ckan.net$ sudo uwsgi -C -iH /home/okfn/var/srvc/ckan.net/pyenv --paste config:ckan.net.ini --uid www-data -s 10.48.162.201:9001
*** Starting uWSGI 0.9.6.6 (32bit) on [Sun Jan 30 16:00:13 2011] ***
compiled with version: 4.4.3
Python version: 2.6.5 (r265:79063, Apr 16 2010, 13:28:26)
[GCC 4.4.3]
uWSGI running as root, you can use --uid/--gid/--chroot options
setuid() to 33
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
allocated 412 bytes (0 KB) for 1 request's buffer.
Setting PythonHome to /home/okfn/var/srvc/ckan.net/pyenv...
binding on TCP port: 9001
your server socket listen backlog is limited to 64 connections
initializing hooks...done.
Loading paste environment: config:ckan.net.ini
Traceback (most recent call last):
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/__init__.py"", line 3, in
from loadwsgi import *
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py"", line 8, in
import pkg_resources
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2691, in
add_activation_listener(lambda dist: dist.activate())
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 668, in subscribe
callback(dist)
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2691, in
add_activation_listener(lambda dist: dist.activate())
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2195, in activate
map(declare_namespace, self._get_metadata('namespace_packages.txt'))
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 1790, in declare_namespace
_handle_ns(packageName, path_item)
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 1761, in _handle_ns
loader.load_module(packageName); module.__path__ = path
File ""/usr/lib/python2.6/pkgutil.py"", line 238, in load_module
mod = imp.load_module(fullname, self.file, self.filename, self.etc)
File ""/home/okfn/var/srvc/ckan.net/pyenv/src/ckanext-dataapi/ckanext/dataapi/__init__.py"", line 36, in from pylons import config
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/__init__.py"", line 4, in
from pylons.config import config
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/config.py"", line 2, in
from pylons.configuration import *
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/configuration.py"", line 25, in
import pylons.legacy
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/legacy.py"", line 11, in
from pylons.util import deprecated, func_move
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/util.py"", line 18, in
from paste.script.appinstall import Installer
File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/script/appinstall.py"", line 23, in
from paste.deploy import appconfig
ImportError: cannot import name appconfig
}}}"
908 1296659056000000 pudo fixed in https://bitbucket.org/okfn/ckan/changeset/08c0e1a819e4 and https://bitbucket.org/okfn/ckanext-stats/changeset/6e86c071db99
908 1296660052000000 wwaites "Not fixed. Try running with:
{{{
uwsgi -C -iH /path/to/your/pyenv --paste config:ckan.net.ini -s 127.0.0.1:9000
}}}
Still get giant traceback relating to a circular import centered on an arbitrary plugin"
908 1296730578000000 pudo I've moved the uswgi/nginx part to #952, closing this issue since the problem it describes has been fixed.
909 1346669602000000 ross Backlog until funded.
910 1310128544000000 thejimmyg Not enough information and over 6 months old so closing.
911 1296467375000000 pudo Done.
912 1306774876000000 pudo moving to ckan
913 1299841413000000 rgrp If at all possible this should uris from the OKF licenses registry at http://licenses.opendefinition.org/
913 1306774962000000 pudo moving to CKAN, still need a resolver
915 1296467518000000 pudo Live and running :-)
916 1297066902000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/4/port-new-vdm-to-mongodb
917 1295280232000000 dread Fixed in cset:f2865f43d8ee
918 1315911507000000 dread Preview feature is now deprecated
919 1303202627000000 dread Simple to fix this in the process of fixing #108. Fix went in cset:304d30d85816 for release 1.3.4.
920 1295355318000000 hellmann@informatik.uni-leipzig.de my first email address was wrong...
920 1300217199000000 thejimmyg @David Should we implement this as a change to the way tags work or via a tidy up cron job?
920 1300319140000000 kindly "The only issue here is that we are listing tags that relate to 'inactive' packages. We are already not listing tags that relate to NO packages.
I have fixed this.
cset:cd0347eed69f
The tag in the example is related to a deleted package so should not be deleted. With this patch it no longer gets displayed."
921 1300197629000000 thejimmyg Seb started this, I completed it and the code is now live. Further refactoring will be done first in #1037 and then in #987.
922 1310128789000000 thejimmyg "See also #358 which I'm closing because it gets merged into this ticket.
The reason we were blockd on #358 was to do with issues around authorization."
922 1320664187000000 rgrp Duplicate of #1032 (I think)
925 1323168588000000 thejimmyg This is fixed in CKAN 1.5
926 1296637927000000 rgrp "Comments from RP - http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html
Libraries I have used: FormEncode, FormAlchemy (what we are currently
using, before that formencode).
Neither seemed perfect but I think the form issue is a 'hard' problem
(perhaps with no perfect answer) [1]. FormAlchemy, in retrospect, was
probably a mistake as it merges too much model/validation/form
generation into one thing.
At least 3 functions involved:
1. Generating (or just filling) a form template with 'form data' (and errors)
2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
3. Converting form data to model data (and validating) (inverse of
previous step)
I think one can and should separate 1 from 2+3 (and one of problems
with formalchemy is it doesn't -- the attraction being you don't
repeat yourself as forms get generated from model but I think this is
actually a false economy in medium-term).
I'm not specifically recommending the following as I haven't used them
but I've looked through docs, they are active and reasonably mature:
1. Flatland: http://discorporate.us/projects/flatland/docs/tip/
* Only does 2+3 which is a good thing IMO
2. WTForms: http://wtforms.simplecodes.com/
* Used in standard flask docs: /path/to/file/you/want"
982 1298482394000000 dread Need to do this for older branches which isn't subject to #963.
982 1298887980000000 dread Buildbot scripts now fixed.
983 1297773407000000 dread "Error was tracked down to cset:214a8f9fc1c2 (26-9-2010):
upgrade_db called validate_authorization_setup() which calls setup_default_user_roles(System())
Fixed in cset:9f51a1c8ac83 for 1.3 branch and merged to default."
984 1297628554000000 kindly "These are the errors listed in severity order.
* group revision tables joins wrong in many ways.
* changemask table not added.
* licence_id wrong type.
* package_revision.download_url and changset.status not dropped.
* package.name and tag.name unique constraint not added.
* update cascaded defined wrongly.
Attached are the fixes that will need to be run in ckan_migration_fixes.sql"
984 1297682116000000 kindly fixed see https://bitbucket.org/okfn/ckan/changeset/d56ea86d4303
985 1298571248000000 pudo digitialiser.dk has been assigned to Stefan Marsirske to get him into this Framework, everything else is delayed.
985 1303118298000000 thejimmyg We'd like CKAN to CKAN harvesting this week if possible.
985 1306407617000000 amercader Moving this to the PDEU (publicdata.eu) project
985 1306408134000000 amercader Duplicate of #1155 and #1156
986 1297812401000000 wwitzel3 https://bitbucket.org/okfn/ckanext-qa/src/be57e20c60ef/
987 1311177705000000 thejimmyg This has largely been implemented now for publicdata.eu. There is a CREP #1134 outstanding to take the harvesting to the next level so marking this one as duplicate for now.
989 1297700363000000 kindly "It would be nice to know some use cases. I think that plugins should control their own storage, or share a storage that is designed to be flexible (mongo, redis ...).
We do not seem to be able to keep our current migrate repository in sync let alone add plugins to the mix."
989 1297700818000000 pudo "Kindly, I agree - it would be much preferable to have independent storage for plugins and this would be easy to do if we were using another type of storage already. As it stands, however, our storage mechanism is SQL. I think we should use it for what it is as much as possible and do the weird, vertical stuff (k,v tables, swapping to redis) only if we really need it. For everything else: lets use SQL as it was intended.
Examples:
* We want to develop an apps catalogue as a CKAN plugin. While we could certainly put this in Redis, there is no reason why we can't have the following table: application (id, name, title, description, author, project_url, site_url, code_url, image).
* A watchlist plugin could essentially work on UUIDs alone. What you'd end up with is something like this: watch (id, user, scope_id).
Re migrations you're right, but my first intention would be to handle that seperatly for each plugin (i.e. they need to have their own migration repositories that they keep track of, e.g. via an apps_migrate_version table)"
989 1297706620000000 kindly "I do not think we need to 'extend the model' if you intend to make the migrations separate. If the schema is decoupled, then there are no problems. So each plugin can have its own model and use sqlalchemy independently i.e have their own metadata, classes and mappers. They do not have to even use sqlalchemy.
What I mean is that there is no need to do anything apart from.
* Agree on a naming convention of the plugin tables (including their own migrate table each)
* Agree to the rule that no plugin can add a column to an existing table.
* Agree that no table can have a (database level) foreign key constraint between the core tables and itself in either direction. They *can* have implied sqlalchemy level joins.
* Maybe have a hook that on db upgrade all plugins are upgraded.
Each plugin will have to redefine the tables, classes and mappers they need to join onto the core tables themselves. reusing/extending the core model will not be worth the trouble.
This seems to cover your use cases and this way everything is nicely decoupled. Best of all there is very little work to do... "
990 1311180850000000 thejimmyg This now works.
991 1298037717000000 dread Fixed in cset:56cccbbb9d1a in time for ckan 1.3 release. This did not affect previous releases.
992 1298060474000000 rgrp Fixed in cset:08548ef8f0e9
993 1298373114000000 dread Fixed on 1.3 cset:7708c8b521ed and merged to default. Deployed to ckan.net.
994 1298912830000000 kindly see cset:93188d42fc12
995 1300364316000000 thejimmyg "Also, update the docs:
Documentation article on caching / improving performance. (To complement configuration docs.)
* Different sorts of cache - beaker style, etags, package_dict in search results(?)
* How each one affects performance
* How to turn them on/off and configure them
* Is it possible to bypass each of them in the browser or with wget/curl?
Therefore closing #841."
995 1311175353000000 thejimmyg The caching still isn't brilliant but there is less urgent need to refactor it. In the future our caching methodology should be to cache at the logic layer so we don't need to refactor the existing caching at this stage.
995 1311179009000000 thejimmyg "See also previous tickets:
* #537 Caching and Performance improvement
* #543 Investigate partial page caching and edge-side includes"
996 1300364398000000 thejimmyg This is now fixed, the outcome will be used to inform #995 for the caching.
998 1298369862000000 dread 'paster db create' (or init) should do exactly what we ask. Surely we should simply tell people to use 'paster db upgrade' instead?
998 1298371191000000 anonymous I am happy to get rid of paster db create altogether as a compromise? Or add a depreciation warning to it?
998 1298372171000000 dread Yes I agree - either of those sounds good. I think I've always used 'db init' in preference anyway.
999 1300707328000000 rgrp Done. See https://bitbucket.org/okfn/ckan-ckan.net and updated ckan.net.ini.
1000 1298912726000000 kindly fixed cset:630513f550d5
1001 1300367555000000 thejimmyg Yes, I think the two should be consolidated, but as was noted elsewhere (I think) we don't want the API to ever be authenticated via a cookie, otherwise we open ourselves to CSRF attacks. Instead we could support the API key as a query parameter as an alternative to via the HTTP header if it is hard to set an HTTP header in JS libraries?
1001 1300625888000000 pudo "My vote on this would be to enable ""httpbasic"" auth via repoze.who and to add an API key challenger to ckan/lib/authenticator.py that accepts API keys, from within repoze.who. This way we won't have to work around the framework at all and can retain compatibility to the existing mechanisms."
1001 1300703779000000 rgrp First work in cset:6b8cbe465b61
1001 1300704249000000 pudo Wrote a WDMMG version of this yesterday: https://bitbucket.org/okfn/wdmmg/changeset/398ce0eb3901#chg-wdmmg/lib/authenticator.py
1001 1301229082000000 rgrp Completed and merged into default in cset:cb200f339dbb. At the moment have done the very simple option but pudo's approach seems better and we could refactor towards that going forward.
1001 1301307814000000 dread You mentioned writing tests? Also the CSRF question from James hasn't been addressed.
1001 1301310351000000 rgrp This ticket did not include CSRF matters - please raise in a separate ticket if you wish. I looked in some detail at the testing options and nothing seemed simple to do that didn't duplicate code we already have (since no way to access the base controller halfway through a request). This can be tested as necessary as part of the development of the new js work.
1002 1311173185000000 thejimmyg I've done this in my branch and it will get merged in.
1002 1311675754000000 dread "Can you also delete the ""paster changes"" in ckan/lib/cli with this removal?"
1002 1311676199000000 dread Reopening this, since it has not been pushed or merged in afaict.
1002 1322515684000000 dread I've redone these changes in cset:91258a6296db71d78a on master, aiming for 1.5.2.
1003 1298490126000000 rgrp Work so far in http://bitbucket.org/rgrp/ckanjs
1003 1298889293000000 rgrp Have now started refactor to use backbone and have basic inline editing working and started on Add dataset view.
1003 1300100411000000 rgrp Converted completely to backbone and now have fully-operational add dataset functionality. I'm closing this ticket now - further improvements can be in there own tickets.
1004 1311173560000000 thejimmyg "At the moment it says ""Create a new group"" and clicking it takes you to the login page. We want it to do the same thing but if you aren't logged in the label should say ""Log in to create a group""."
1004 1323171375000000 thejimmyg The yellow box on the right is now back and you don't get taken to the login page. We'll write more detailed instructions once the group refactor is done.
1004 1323172489000000 dread "There is a description of groups, but that is not the subject of this ticket. In fact if you're not logged in you do get taken to the login page (which is quite correct). I agree with your earlier comment that the text of the link needs changing.
Sorry to be reopening this, but I think the previous comment misses the point. Doing this ticket would be a valuable quick fix. Putting on backlog."
1004 1323174597000000 thejimmyg "I don't understand. We just had a team meeting about this, all discussed it and agreed to close it. Yes, you get taken to the login page, but that is the correct behaviour.
The problem in the past was that the link was in the yellow box and there was no explanation as to why you had to login. This time you click from the header bar and there is a clear message saying ""Unauthorized to create a group"" - exactly as a user would expect. Even if the exact text of the ticket description isn't fully implemented in the current release, the UX isn't broken anymore. Yes, it might be even nicer to have a message warning them in advanced but these improvements will be taken forward in the UX work - maybe there is an even better solution than a message?
Since you are unhappy about closing it I'm marking it as ""Duplicate"" of #1521. As agreed earlier with the entire team, we'll take this forward as part of the groups refactor.
See #1521 for more information.
"
1004 1323177994000000 dread "My suggested fix:
{{{
diff --git a/ckan/templates/group/index.html b/ckan/templates/group/index.html
index 8502df9..064e066 100644
--- a/ckan/templates/group/index.html
+++ b/ckan/templates/group/index.html
@@ -9,7 +9,15 @@
What Are Groups?
- Whilst tags are great at collecting datasets together,
+ Whilst tags are great at collecting datasets togethe
+
+
+ Creat
+
+
+ To create a new group, please first
${h.subnav_link(c, h.icon('group') + _('List Groups'), controller='grou
-
+
}}}
BUT this fix doesn't completely work. When you log-in and create a group, at this point the nav bar changes from the (correct) ""Add a group"" to (incorrect) ""Log-in to add a group""."
1004 1323187352000000 dread "Here is the unclipped version:
{{{
Create a new group
To create a new group, please first log-in.
}}}
{{{
${h.subnav_link(c, h.icon('group_add') + _('Add a Group'), controller='group', action='new')}
${h.subnav_link(c, h.icon('group_add') + _('Log-in to add a Group'), controller='group', action='new')}
}}}"
1004 1323194016000000 zephod "Dread, that fix wouldn't work because the class attribute was being defined in Genshi and again in HTML, with slighly undefined behaviour (in this case Genshi overwrites HTML).
I have performed a very very simple fix which modifies the navbar text when you're not logged in.
https://github.com/okfn/ckan/commit/a89a48731ba548170045a60ac2930e0019c299c7
I don't think the link should be restored in the sidebar, it was explicitly removed as part of a site-wide sweep to make the sidebar a passive, helpful element rather than an active element with action links.
Site looks great to me, I'm closing this ticket."
1004 1323195485000000 dread "Great stuff zeph! Diff looks really good and cheers for the explanation.
On master, cset:a89a48731ba548170045a60ac2930e0019c299c7 and I've cherry picked this for release 1.5.1. too."
1005 1300100085000000 dread Moved to dgu trac ticket 869
1006 1298631145000000 dread "This command is slightly different to your branch policy as of two weeks ago:
{{{
stable: stable code
metastable: (will soon be deprecated) for code preparing to be stable
default: development HEAD
}}}
which I prefer.
My ideal would be to get rid of the confusing name 'metastable' and unneeded 'stable' and start a new branch called 'released', which will act the same as 'master' in this diagram but with a more intuitive name:
http://nvie.com/posts/a-successful-git-branching-model
Then for each ckan instance we can either use the most recent release (from 'released') or choose a specific one (e.g. 'ckan-1.3' or even 'default' or 'enh-865' for getting latest features). This gives a good degree of flexibility, is more understandable to newbies and probably a more widely understood branching model."
1006 1300098217000000 dread Reassigning to rgrp for response
1006 1300105927000000 rgrp "Not sure when i described that other process but I'm definitely of the opinion that we should:
* Deprecate and remove stable branch
* Deprecate and remove metastable branch (going forward we can use release branches for what we did with metastable in the past)
May want to update BranchingPolicy to reflect this (also should probably have some statement about closing release branches and tagging)
Reassigning to David Raznick as he is our release guru now."
1006 1300372286000000 thejimmyg I'm updating the branching policy now. David has closed stable and closed metastable. Marking as fixed.
1008 1298821826000000 rgrp I've removed the eval in cset:1b8fedeb7ab0 - the more general question about caching should go in a separate ticket.
1009 1298638447000000 pudo "Some more ideas:
* /user should list users, sorted by number of packages contributed/editied
* /user/{name}/packages shows a list of packages to which users have contributed
"
1010 1298733856000000 rgrp Complete see branch feature-1010-list-users and closing changeset cset:feature-1010-list-users.
1010 1298740889000000 rgrp Meant this cset:d2651db566ef
1011 1298820235000000 sebbacon On reflection, may as well make a Plugin interface called IAuthorizer, which allows customisation of get_authorization_groups, get_roles, and is_authorized....
1011 1298821699000000 rgrp I agree that IAuthorizer is useful but not sure how it addresses the requirement of the ticket. AuthorizationGroups are already editable via the web interface at /authorizationgroup
1011 1298824649000000 sebbacon "The ""external source"" is an Oauth service. We need to lookup user groups from that service."
1011 1298825600000000 sebbacon Proposed implementation at https://bitbucket.org/okfn/ckan/changeset/187e65afb35f
1011 1299245206000000 sebbacon Merged to default https://bitbucket.org/okfn/ckan/changeset/e8217c317a8e
1012 1300217601000000 thejimmyg "See also #103 which says:
As a user I want to view a package at a given revision:
* When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy
* package history page should provide links to these pages
Could you please investigate how doable this would be given the current package read.html page. I expect it could be very easy once the revision API is in place because we could build a c.pkg object from the revision data instead of the model data.
Perhaps another case where we could try introducing the dictization?"
1012 1300352334000000 kindly "This is entirely not trivial at the moment. Hopefully after the dictization it will become more so.
Simply putting in the package revision object in place of the package does not work.
It will obviously work for changes to the package object itself. However, there are no mappers on that object for getting out the related package_tags, resources and extras at that revision. You will have to construct a fake pkg object with some messy and painful queries using dates."
1012 1300362584000000 kindly "cset:5649d6e761fc
The basic revision history is merged. I will keep this ticket open if it is not sufficient.
All it does is give a list with the most recent first of revision_ids, authors and timestamps.
i.e
{{{
[{""timestamp"": ""2011-03-16T15:55:19.941961"",
""author"": ""southampton-ac-uk"",
""revision"": ""202e9eb8-afaa-4bc9-b8a1-a317561547ea""},
{""timestamp"": ""2011-03-15T17:59:16.430804"",
""author"": ""southampton-ac-uk"",
""revision"": ""8235bd0a-d39a-49e0-887a-b0f231be8a92""}]
}}}
"
1012 1300364620000000 thejimmyg The API part is now done. Bumping the view history part to 1.5
1012 1300368449000000 dread As I understand it, the whole point to vdm is to make it easy to get old versions of not only the package, but its related objects (like tags, extras). David take a look at this function: Package.get_as_of(revision)
1012 1301909937000000 thejimmyg No-one really seems to have requested this part.
1012 1301943096000000 dread Changing this to 'Fixed' and #103 to 'Wont fix' to ensure this feature is noted.
1013 1299245157000000 sebbacon "This is now resolved, but depends on core CKAN behaviour (specifically pluggable middleware and unicode-aware error pages) to function:
https://bitbucket.org/okfn/ckan/changeset/c846794c1799
"
1014 1299245293000000 sebbacon Run out of time for decoupling, but tests and README.txt written (including pointers about how to customise for anyone who needs to decouple in the future)
1015 1298902753000000 kindly The migration fixes should sort this out, but I will keep the ticket open to check.
1015 1299788821000000 rgrp Today kindly applied the sql fixes and I can confirm this is now fixed. Well done kindly for all the great work here.
1017 1299668555000000 pudo fixed in cset:86d49a775fd3
1018 1299073340000000 dread Done in cset:e4167f8b3f80 on default
1019 1299166930000000 pudo fixed in https://bitbucket.org/okfn/ckanext-webhooks/changeset/034647931921
1020 1300196215000000 thejimmyg This is now on DGU live.
1021 1299518828000000 pudo fixed in cset:9f1b38add19f
1022 1299512991000000 pudo "We're now using fileConfig to configure the logger API from the worker config file and this enables us to use SMTPHandler to send out error messages on queue processing failures.
Marking as fixed. "
1023 1299514847000000 pudo "Tried implementing this with AMQPs msg.requeue() and channel.basic_recover() but RabbitMQ yield a NOT_IMPLEMENTED error.
Bit clueless on how to proceed. "
1024 1299668648000000 pudo re-created this (#1027)
1025 1299598708000000 dread Done in enh-1025-config-default-authz and goes into release 1.3.2.
1025 1299750097000000 rgrp (Link to changeset). Could you briefly clarify why this was needed in config -- we already have a process for putting things into a more restricted mode (see ticket:833) and have been working on creating a WUI to be able to do this automatically (see that ticket).
1025 1299751045000000 dread "Thanks for pointing this out. Although changing the definition of 'editor' to not allow edit (as you admit) is a bit hacky, and I think prone to confusion. James and I weren't aware there was a precedent for doing it this way, but if we had, we may well have followed it.
I mentioned the branch for this cset, in preference to the csets."
1026 1299605128000000 dread Done in cset:2dde3bd563fd for branch 1.3.2
1027 1299666326000000 pudo "
1. home controller -> __before__ (check ""site-read"" on model.System)
2. user -> each individually (repoze-who pseudo action must not be blocked)
* user-read (index/read/update pages for users)
* user-create (register)
3. revision -> __before__ (check ""site-read"" on model.System)
4. tag -> site-read (__before__)
functional/test_authz.py
* denies site-read ...
* checks for visitor / logged in user ..
* checks you can still visit login
"
1027 1299682082000000 pudo fixed in cset:532c3ad2743b
1028 1299760360000000 dread Fixed in cset 4d860a53fbad on 1.3.2 and merged to default.
1029 1301311643000000 thejimmyg This was fixed by kindly as part of the 1.3.2 release.
1030 1300103984000000 amercader "This includes:
* Take out all references to harvestsource, harvestingjob and harvesteddocument in the rest API
* Move the harvesting bits of ckan/lib/cli.py into ckanext-harvest.
* Move ckan/controller/harvesting.py and cknan/model/harvesting.py to ckanext-harvest as well
* Update ckanext-csw to be able to find the code it needs in the new place."
1030 1300209975000000 thejimmyg "Also:
* Delete harvest sources
* Implement a Genshi template filter so that if there is an INSPIRE package_extra set to True, you can link to the XML and HTML generation API calls in ckanext-csw
* Write tests in ckanext-dgu for all the API calls that Drupal is making
"
1030 1303117978000000 amercader All the points described in this ticket have been completed. Harvesting work is being done mainly under #1037
1031 1299863222000000 rgrp Done, see cset:78d96b520679
1032 1300380606000000 rgrp This is urgent for me as without upload does not really function.
1032 1303117292000000 thejimmyg Rufus, I think this is potentially quite a big change and needs to be done as part of the entity refactor rather than as a quick hack. I'd like David Raznick to work on timeseries this week so I'd like to understand a bit more about this ticket. Could you flesh it out please or let's discuss in the catch up?
1032 1303118226000000 thejimmyg See related ticket #922
1032 1304094162000000 rgrp Agreed to defer this. Resource editing, to the extent it happens can be kept very simple and part of package editing.
1032 1330348463000000 zephod Streamlining supertickets. #1506 is the new parent of #978; all tasks remaining in this ticket are complete.
1033 1308310446000000 dread Fixed in 5c7caca30737 for v1.4.1
1034 1320174353000000 dread Same problem as #1321 I believe
1035 1323172908000000 thejimmyg Re-assigning to me temporarily to investigate now that Seb has left. It may be that this should be deleted given that it is over 6 month's old.
1035 1338206251000000 ross Removing until we can come up with a case for implementing
1037 1303117000000000 thejimmyg "We spent last week integrating the new harvesting architecture and testing the code but there are still some areas that need looking at
* The source type and label should be part of the plugin, not named in DGU.
* Need warnings if a document changes but its date doesn't -> do we have these?
* I noticed there are some tests in DGU, should these perhaps be in ckanext-harvest?
* If active is False, the job should not be put on the queue
* Log if the wrong type of URL is entered as an error the user can see
* Deny if the source is already registered
* Overwrite all extras, not just merge new ones.
* During the import stage use iswms.py to add an extra during import if it is a WMS so that we can add a link to the WMS later https://gist.github.com/900878
* Can errors/warnings be logged in the import stage? Do all fetched documents get passed to import in one go?
"
1037 1304937601000000 thejimmyg Closing this now, any outstanding small issues will be logged in new tickets.
1038 1300212841000000 dread Done in cset:0373737f0866 for release 1.3.2
1039 1300212856000000 dread Done in cset:97d563bbf01d for release 1.3.2
1040 1300213922000000 dread Fixed by James in 904f1f36a672 for 1.3.2
1040 1328805778000000 dread "The problem arises when you do two things together:
1. Have code like this in a CKAN extension's plugin.py:
{{{
config['extra_public_paths'] = ','.join([our_public_dir,
config.get('extra_public_paths', '')])
}}}
which will have a trailing blank value (unless you set extra_public_paths in the config file)
{{{
extra_public_paths=['/some/path/', '']
}}}
2. Run CKAN with this code:
{{{
extra_public_paths = config.get('extra_public_paths')
if extra_public_paths:
static_parsers = [StaticURLParser(public_path) \
for public_path in \
extra_public_paths.split(',')] + static_parsers
}}}
Introduced in #340 https://github.com/okfn/ckan/commit/0593c3e5050882a9dbf54fb6c263f4a2f1de2097#ckan/config/middleware.py in CKAN 1.0.2
This was made safer in https://github.com/okfn/ckan/commit/e0d5f02168116a18a7aeb7b058e4160bbd402a02#ckan/config/middleware.py for CKAN 1.3.2 (it ignores blank paths)
"
1040 1328806824000000 dread "Here are all the CKAN instances potentially involved in this issue:
== ckan.net / DGU ==
These suffered this problem briefly and were fixed 15/3/11
== opengov_es http://opengov.es/ 1.2 ==
This does not have a theme extension. Is not serving files from root.
== nosdonnees_fr http://www.nosdonnees.fr/ 1.3.1a ==
This does have a theme extension but does not seem to have a trailing blank public path since it is not serving files from root.
"
1041 1300284809000000 thejimmyg Also, this means moving documentation out of the current CKAN dev wiki at trac.ckan.org and into the community wiki. Core developer docs should remain in the Sphinx-based .rst format in CKAN. Much-needed new tutorials go on the community wiki.
1041 1310135309000000 thejimmyg Child ticket of #1142
1041 1312372367000000 rgrp James says this is not priority and he does not plan to do docs for debs until he has got that system functioning.
1042 1300969265000000 dread Done in ckanext cset:7610512277bd and ckanext-importlib cset:7737716e0d7b.
1042 1300969865000000 dread "Remaining parts of ckanext are:
* blackbox (rgrp, pudo, dread)
* hammer (ww)
* link_checker (ww)
* solr configuration (pudo) & indexer (pudo)
* amqp listener command (ww)
* geo form field (rgrp)
* GData Loaders:
* aidprojdata (johnbywater)
* norwaygovdata (johnbywater)
* ordsurv (johnbywater)
* Gemini2 (iso19139) (johnbywater, ww)"
1043 1300321033000000 kindly "cset:c894f92c5b9a
Sesion.remove() needed to be run before configure as we want the original session made not to be used over the newly configured ones."
1044 1300359562000000 rgrp Think this would be resolved by refactoring implied in ticket:1001 (specifically checks for api key and remote user should lead to availability of same set of identification information).
1044 1300382367000000 dread "I agree with leaving #1001 solving this issue. Centralising this is good. This ticket can instead focus on:
* returning the correct 403 error (rather than the 302 currently)
* creating some tests
* documenting
* consider renaming the SITE_READ variable and rethinking purpose"
1044 1300449905000000 dread "I've added in tests for SITE_READ and corrected the 403 response - see branch bug-1044-site-read.
The tests have uncovered some inconsistencies in the use of SITE_READ between reading/editing a package in the Web interface vs the REST API. Friedrich, do you want to take a look at ckan/tests/functional/test_authz.py:TestSiteRead.test_pkggroupadmin_read for details and have a think about how this should best work. How about making SITE_READ more widespread in the WUI, to protect read/editing/searching things, in addition to the READ and EDIT RoleActions.
Once that is decided, it should be pretty easy to refer to the tests and document SITE_READ."
1044 1300627007000000 pudo "David, thanks for writing those tests - perhaps we should combine them with the ones below (""TestLockedDownUsage"") which attempt to basically do the same.
As for the inconsistency introduced by the global check in the REST API you're right: Of course it is strange that WUI access checks are more granular than the API checks. The alternative is that we either move authz checks into all relevant REST places (a major refactoring I would be suspicious of) or that we introduce duplicate checks on the WUI actions (I actually have performance concerns about that, authz is incredibly slow - and it introduces another level of special authz that I think we really should not have). Given the choice I'd opt for the REST refactor - there is no good reason to make SITE_READ a duplicate check where authz already applies.
On the other hand, this is a function we really don't want to enable or even have and that was only added to satisfy some very specific user demands. Given that these are fulfilled, I'm actually OK keeping the inconsistency for now - nobody will see it in normal operations and in a locked-down environment, users will need to have API keys anyway.
Regarding the naming, I'm pretty opionion-less: SITE_READ was proposed by rgrp and I think its pretty fitting, while OTHER_READ would just confuse me. PUBLIC_READ might work, though. "
1044 1300712223000000 dread "I'm happy to leave the inconsistencies in SITE_READ for now, since this will be resolved when authz is centralised between WUI and API in the dictization refactor.
I'm also happy to leave both TestSiteRead and TestLockedDownUsage as they complement each other nicely actually.
So I think all that remains for this ticket is to add a paragraph to the authz documentation about SITE_READ. Pudo would you mind doing this? It's well worth noting where it applies and where it doesn't."
1044 1301304871000000 dread I think all that remains for this ticket is to add a paragraph to the authz documentation about SITE_READ. Pudo would you mind doing this? It's well worth noting where it applies and where it doesn't.
1044 1302096155000000 dread I've added in the docs for this in cset:013da53052d1 ready for release 1.3.3.
1045 1300457708000000 dread Fixed in cset:98eb8b8d063e on default, scheduled for CKAN version 1.4.
1045 1300787161000000 dread Want to identify groups by ID also in the Web interface.
1045 1300793261000000 dread Fixed in cset:71621df50983 on default ready for ckan 1.4
1046 1302777668000000 kindly cset:35ba6ad033ae
1048 1300702752000000 dread Done in changeset 9d7bfa124757 on default and expect it to go into 1.4 release.
1049 1323169424000000 thejimmyg This is over 6 months old. In accordance with wiki policy, closing.
1050 1300452200000000 johnlawrenceaspden there's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests.
1050 1300458454000000 dread "I may misunderstand how you intend to build the WUI Admin interface, but I think most of the stuff in authztool is just marshalling command line parameters anyway. The bits which do anything are factored out.
For example, to list rights you just loop over obj_classes to call {{{model.Session.query(obj_class).all()}}} and then display the values: {{{type(subj).__name__, subj.name, role, type(obj).__name__, obj.name}}}.
To change a right you simply call {{{model.add_user_to_role(subj, role, obj)}}} or {{{model.add_authorization_group_to_role(subj, role, obj)}}}.
But of course if there is useful stuff to factor out then be my guest!"
1050 1303118188000000 thejimmyg This should also feed into #1075 which will be being worked on this week.
1050 1315394117000000 dread There's been no objections to my comment 6 months ago saying this is invalid, so marking it invalid. Feel free to reopen if there is indeed a case here.
1051 1301305079000000 sebbacon This has been completed in https://bitbucket.org/okfn/ckan/changeset/64a949990e0b
1052 1300895410000000 dread Fixed in branch feature-DGU#889-authorization-in-lockdown and merged to default at cset:e6643cf1324c. Only remaining issue is listing in Web interface of package relationships, which shows links to packages even to ones you can't read.
1053 1310125444000000 thejimmyg This is now fixed as part of the API refactor and revisioning updates David Raznick has done.
1053 1310126546000000 dread "Looks to me like DELETE always changed the state to Deleted, so an invalid ticket.
We can deal with purging in another ticket if needed."
1054 1300989764000000 dread I've added some tests in the branch: defect-1054-resource-order
1054 1300992195000000 dread I've put in a potential fix for this into the branch - do take a look and see what you think.
1054 1301002995000000 kindly "Looks great. Had a look and changed a minor thing because I was not confident with the handling of the null values. I made a fake resource and did an asdict on that to make the identity.
Part of my patch I reverted as I mucked up my version of the vdm, so it was not needed.
"
1054 1301305615000000 dread This branch is now merged.
1055 1300987241000000 dread This amounted to 14 tests. Fixed in cset:5bbd0005c57e ready for ckan release 1.4.
1056 1302626373000000 dread Fixed in cset:a025b5b3293c on default.
1056 1302882616000000 dread This was broken in cset:b681bbedfa62a68b71260ef48a0da6063109734 which was released in 1.3.2
1057 1329150236000000 toby fixed in commit 3d7cbf0
1058 1301303221000000 dread Fixed in cset:8e1817ab8d1c on default, ready for 1.4 release.
1059 1301312516000000 dread Fixed in ckanext-importlib cset:f6d19129ac43.
1060 1301312487000000 dread Fixed in ckan-importlib cset:bc52bba86d71 and ckan cset:2751f76fb17a in time for ckan 1.4
1061 1301397271000000 dread Fixed in cset:fc3bc103db8c ready for release 1.3.3
1061 1301922325000000 dread Actually there was a link at the bottom of the package edit form that my grep didn't pick up: h.url_for(str('/license')). So it is not orphaned at all.
1061 1301922350000000 dread Backed out original change fc3bc103db8c here: 7ae9aff8bc68
1062 1311178145000000 thejimmyg "Hi John,
Can you please check that the new webstore+preview works correctly with this one please?
Cheers,
James"
1062 1311773731000000 johnglover The file is in Turtle format (serialisation format for RDF, which we currently don't archive and have no preview code for. The new preview code will not even create a preview button for this file type. Moving to the backlog for now, support for RDF and related formats should be added along with support for excel docs and google spreadsheets.
1063 1301522483000000 rgrp "This sounds absolutely sensible and suggest we even have a helper method such as lib.helpers.group_link(group) (as we do for user_link).
(@Seb: apologies for misunderstanding your query on the list -- I was talking about the group listing table at /group)"
1063 1302514033000000 sebbacon Fixed in https://bitbucket.org/okfn/ckan/changeset/145eaae3168a
1064 1323169787000000 thejimmyg Marking as duplicate because #1464 will take it on.
1065 1301914004000000 sebbacon "To reflect further conversations:
* We are parking this ticket until dictization is complete
* We would prefer to see roles as asserted globally by default, with packages just storing exceptions. E.g. If I am a ""reader"" globally, then I have ""read-package"" permission on all packages new and existing, unless otherwise asserted locally on a package.
* Also w.r.t. algorithm above, consider repoze.what's model; ""possible"" is analagous to ""has_permission"". See http://what.repoze.org/docs/1.0/Manual/Predicates/Builtin.html"
1065 1302033244000000 dread "Wow, lots of solutions here, and not enough evidence of the problems encountered / difficult use cases.
I have tried here to extract what John and Seb might be suggesting is difficult with our current http://en.wikipedia.org/wiki/Role-based_access_control system.
1. (Seb) It seems crazy to change the User-Object-Roles for every single package to 'lockdown' a CKAN instance. Instead of using Roles 'reader' & 'editor' we start using Roles 'logged_in_user' and 'anon_user'. Then, with only changing a couple of lines of the Role-Action table for these Roles, you can give or remove permissions to edit / create packages, groups etc.
2. (John) Actions are being added all the time by migrations and extensions. I'm not sure what the problem is here - can someone explain? The separation of Users and the new Actions by adding the Role 'join' means the migration/extension can add existing Users with the new Actions. For example when we added the 'Group' protection object, logged in users could previously create groups, so it is natural to add this action to the 'editor' role. If you get rid of Roles, as John suggests, then you end up having to add as many Rights objects as users. And you might end up trying to infer new Rights on the basis of old Rights, rather than a role, which would be more natural. (Is the only reason to denormalise the UOR and RA tables the sake of ease of understanding? I would think we can explain the concept better now...!)
3. (John) This Group Hierarchy description sounds a bit like earlier incarnations of the DGU requirements. ""or any of his groups group-children (but not user-children)"" I'm not sure I understand the user-children bit, and I'm wary of any difficult-to-grasp concept. This modelling seems to take the UserGroup/User hierarchy model and use that as a hard-coded interpretation of the Authorized permission. In contrast, in DGU Evan split the Organisation Hierarchy model from the Authorization model. Each User has explicit permissions for each Organisation he is allowed to add/edit packages for. This helps for these two use cases we have to contend with: when a user is an admin for unrelated branches of the Organisation hierarchy, and when an Organisation is actually in two branches of the Organisation hierarchy. It also seems more generally flexible.
"
1065 1302075994000000 sebbacon "Re the point against my name, yes -- I think the fundamental principles of the current system are fine, but the implementation means asserting things for every single object, whereas we should be able to optimise it for the general cases -- something along the lines you suggest.
I would be very happy to write up a full, detailed proposal for the system before we implement anything, if only as a proper straw man to have debates around."
1065 1302081747000000 dread Great stuff Seb, cheers for that.
1065 1303118756000000 thejimmyg This is now more urgent and I think we have enough consensus to go ahead. See #1094 too.
1065 1311174208000000 thejimmyg Merging with #1094 so see the discussion there too.
1065 1318528721000000 rgrp Reassigning to zephod as he is looking at this. I think most of this is wontfix or will become obsolete with other stuff but worth looking at. Also moving to v1.6 at the same time.
1065 1324550041000000 dread "All subtickets are done now, apart from #1198 which I've moved into superticket #1477 Group refactor
Change name of AuthzGroup? to UserGroup? to reflect what it is for
I think this is completely covered off in #1477 Group refactor
Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones.
Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system.
This is not clear - please raise a ticket for this if still promising.
Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages)
These three points I've moved into a new ticket #1596 as it was getting lost rather in this one.
Change UserGroups? so that they can have a hierarchical structure,
I think this is completely covered off in #1477 Group refactor
Conclusion: closing this ticket as all points are covered or moved into tickets elsewhere"
1066 1301904246000000 dread Need a new role 'ANON_EDITOR' which is the default role for Visitor, which can create packages, but not groups.
1066 1301930311000000 dread "Migrations:
* Default (open) CKAN instances will have visitor as a reader on the system, and will have to upgrade them to anon_editor.
* DGU & DataGM instances have visitor as a reader on system and must stay like that.
* Pudo's specially locked instance, where editor has been changed to only read can now use the 'reader' role (assuming he's happy for them to create users)
Migration script (37) is designed to cope with the default (open) setup, and the other special cases must be dealt with using CLI at the time of upgrade:
DGU & DataGM must run:
{{{
paster rights remove visitor anon_editor system:
paster rights make visitor reader system:
}}}"
1066 1301932136000000 dread Done on branch defect-1066-reader-too-permissive and merged into release-1.3.3
1067 1302186503000000 dread Done in cset:d75ab5fc3311 for release 1.3.3
1068 1302109505000000 dread "The problem was the use of mktime which is localised. Also spotted a problem with the use of localised time stored in the datestamp created in migration script 29, when initialising the repo.
Fixed in a2094932e5e4 in release 1.3.3 branch."
1069 1301909423000000 thejimmyg "This is what the new ""Ideas"" section in the ckanext-community extension is for. Marking as duplicate of #941, can we have discussion there please."
1069 1302510609000000 rgrp Mistakenly assigned to this sprint.
1069 1310133794000000 thejimmyg Yes, and this will be more important in thedatahub.org
1069 1340632215000000 icmurray Assigned to @tobes as he's looking at publication workflows.
1070 1310117129000000 rgrp Added link to http://wiki.ckan.net/Domain_Model and closing as work mostly done (and nothing clear remaining to be done -- agreement perhaps!)
1071 1301943180000000 dread Done in cset:db9e2c4f65bb for release-v1.3.3
1072 1302106474000000 dread Done in cset:2a97d2d9ba4a for release 1.3.3.
1073 1302185825000000 dread Done in cset:9dfc60db90ed on default branch.
1074 1314278296000000 rgrp johnlawrenceaspden did quite a bit on this in feature-1074-refactor-authz-wui branch (including implementing for package and group). However, did not implement for authzgroup and also repeated code a lot. Reassigning to me for completion.
1074 1314283492000000 rgrp "Also work on branch feature-1074-authz-wui (that is primary one in fact)
"
1074 1314303581000000 rgrp Closed in cset:18a5b48bbee6. Note the templating isn't completely reused but rather uses common methods from _util.html. Likely that a bit more refactoring could be done to share common code but not worth it at present.
1075 1303227982000000 johnlawrenceaspden "closing ticket
see http://wiki.ckan.net/Authorization
and https://bitbucket.org/okfn/ckanext-admin"
1076 1302515674000000 sebbacon See also http://trac.ckan.org/ticket/948
1076 1302731255000000 rgrp All main work done as of cset:556efcb91f19 in branch feature-1076-improve-purge-system (+ associated work in ckanext-admin trash bin management). All that remains is to add some tests.
1076 1303236302000000 rgrp Test of trash functionality in ckanext-admin in https://bitbucket.org/okfn/ckanext-admin/changeset/e86c102a3c60 so now closing (could do with some functional tests in the main interface but can leave for later -- were none before so situation no worse ;0 ).
1077 1319797284000000 dread I guess this is not a priority now.
1078 1305828965000000 kindly complete: cset:843b78bae016
1079 1302777496000000 kindly Complete see cset:35ba6ad033ae
1081 1303332388000000 johnlawrenceaspden Actually it seems to be worse than that. If I add a user to an authz group then I get added too, and then there's no way to remove me.
1081 1303489474000000 johnlawrenceaspden "Fixed as part of feature-854-tests-for-authz-groups branch.
Original behaviour was that creating/modifying user is explicitly added to the users in the authz group.
"
1082 1315317825000000 dread "The flash message 'Language set to: English' was always in the previous language, not the new one. Pudo fixed this in cset:eaf342823caf and I have transplanted this into release-1.4.3.
I've also added/corrected translations for all languages for 'Language set to: English' so that it makes sense!
""Set language back to English causes server error:""
I can't reproduce this. The exception looks like a problem with the forwarding, which improvements were made in #1126."
1082 1315917217000000 dread Language switch problem has been around since it was introduced in the 1.3.1 release.
1083 1304975681000000 johnlawrenceaspden "Current behaviour:
from the web interface, russianfan can be added as an admin on warandpeace twice.
and you can then delete the two roles separately.
desired behaviour is that you can only add him once.
from the command line, you can only add him once, but if he's in there twice, you can't remove him because the command line bombs
desired behaviour is that he can be removed
from the shell, you can add him twice but can't remove him if he's been added twice.
if you try, it bombs.
The desired behaviour in all cases is that adding and removing are effective and idempotent in all three cases."
1083 1305051061000000 johnlawrenceaspden It appears to be possible to get into a state where you can't add or remove roles from the command line. Investigating.
1083 1305058621000000 johnlawrenceaspden "Fixed the problem where you can't delete a role if it's been added twice, but had to leave the add function as it was since putting a commit in it breaks many tests.
It appears that the WUI doesn't call these functions anyway. Left that as it is since we're planning to replace it.
The cli rights command doesn't appear to add the roles that it says it adds. Possibly because it's not doing the required commit? Reported this as a different bug."
1084 1303894555000000 dread ww you did this didn't you? Perhaps you could update us with the branch/changeset it was in and close this ticket please.
1085 1302621951000000 rgrp "I don't think this is very useful output :) Most of this is test/demo/rdf and therefore not relevant.
Also have you checked your ckan.site_url in your development.ini -- which configures where what is used as based for all external urls in the templates?
IMO there is nothing to fix here."
1085 1302624602000000 dread As discussed on IRC, the default ckan.site_url should be blank, rather than ckan.net.
1085 1302625314000000 dread Fixed in cset:01b3de5d4e31 on default.
1086 1303381200000000 johnlawrenceaspden There is no (obvious) way to delete an authzgroup.
1086 1323346552000000 dread Authorization groups are deprecated with the incoming Groups Refactor #1477
1087 1310134714000000 thejimmyg Child ticket of #954
1087 1320866159000000 dread "Implemented in cset:0b7e2b0bed44 as status_show logic action, heading for 1.5.1 release.
Took 0.5h"
1088 1302630624000000 wwaites "you use the negotiate function in autoneg.py with a cfg dictionary like this,
{{{
cfg = {
""mime_types"": [
(""application/xhtml+xml"", [""html""]),
(""text/html"", [""html""],
(""application/rdf+xml"", [""rdf"", ""owl""]),
(""application/turtle"", [""ttl""]),
(""text/plain"", [""nt""]),
(""text/x-graphviz"", [""dot""]),
]
for content_type, exts in negotiate(cfg, accept_header):
if ""html"" in exts:
render page normally
else:
redirect to semantic.ckan.net
}
}}}"
1088 1303035487000000 wwaites "fixed in 5c091fc08474 (bug-1088-autoneg-is-wonky) tip
"
1089 1302631733000000 dread Done in cset:6d41ecc431e3 on default.
1090 1302635699000000 dread Fixed in cset:e0af88edf0f0 on default.
1091 1303746408000000 rgrp "This is a well known issue (see some discussoin in #142). There is nothing directly we can do about it (this is what google gives us) but we could make usernames editable (this has been discussed but not ticketed I believe). Given our general ""downer"" on openids I'm not sure this is a big priority."
1091 1323102767000000 rgrp wontfix plus we now requires usernames see #1386
1092 1305570822000000 kindly completed cset: ca1ac4112ea2
1093 1302865470000000 dread Fixed by this API call use the existing license cache. Done in cset:a95888ea39be on release-v1.3.4
1094 1303129670000000 rgrp "See also comments on the mailing list.
Item 1 seems fine (what is difference from current extension mechanism?)
Item 2: concerns here. What about list views? What about editing 'permissions'? I also think getting rid of System object isn't really a benefit (if anything may be a cost).
Item 3: feel this may be better as part of big domain model change (also gives us time to really dig into this -- this is an important requirement/conceptual issue).
Item 4: No objections but seems very minor gain for fairly significant migration work.
"
1094 1304354036000000 johnlawrenceaspden "UserGroup/PackageGroup might also be confusing.
A PackageGroup is *just* a group of packages.
A UserGroup is both a group of users, and a thing affecting authorizations.
Perhaps PackageGroup and UserAuthorizationGroup? Or PackageGroup and AuthorizedUserGroup?
I was quite confused by all this at first.
I think I understand how the whole thing works pretty well now, and I still can't think of good names for the two concepts, although I can already feel the normal English meanings of the words changing to what I now know they are for.
We should be a little wary of this. Things that are even slightly difficult to understand end up being understood by very few people. Would any of us be prepared to sit an exam on exactly how UNIX file permissions work, even though we all use them?
"
1094 1311173649000000 thejimmyg Merging with #1065 and closing.
1096 1307444626000000 nils.toedtmann See also http://ckan.okfnpad.org/multisite
1096 1310125173000000 thejimmyg "Renaming to CKAN Hosted because whether we introduce a site domain object for subdomain tenants or not is an implementation detail.
As far as progress on CKAN hosted goes, we now have:
* A build tool to package ckanext-dgu, ckanext-std etc into .deb files
* The ability to apt-get install CKAN
* A script that can set up multiple non-conflicting CKAN instances on the same machine
What we need next is the ability to generate a ckanext-xxx repository for each new instance we want to host. At that point most of the underlying techniques for hosting multiple CKAN instances concurrently will be in place. Once we have that working and know what clients want, we can look at going further and speccing up a multisite solution off a single DB, rather than the multi-instance solution proposed here.
"
1096 1311182262000000 thejimmyg Milestone ckan-v-future deleted
1097 1303293476000000 dread Done in cset:5dec7eb609ab on branch release-v1.3.4.
1098 1303387896000000 dread Fixed in b2b37e39de4d in release-1.3.4. Was only broken on this branch a few days before.
1098 1303406017000000 dread Also found problem with some tests using delete_all. Fixed in cset:f9764dd43a0a release_1.3.4.
1099 1324057106000000 dread Authorization groups deprecated with the Group refactor #1477
1100 1303406103000000 dread Done in release_1.3.4 cset:580e2b11b4ef and release 1.3.3 cset:cac55b8a0bd9
1101 1330082908000000 dread Stats already in site nav - adjusting title to just mention google analytics.
1102 1303744552000000 rgrp (Same ticket created 3 times by accident I think -- see #1104).
1103 1303744575000000 rgrp (Same ticket created 3 times by accident I think -- see #1104).
1104 1303744486000000 rgrp "You haven't done paster search-index rebuild I think.
If you haven't the search index will not have been built (Create Test Data code creates data without indexing it). This could be considered a bug of Create Test Data but I'm not sure -- if you think so re-open this ticket and change description."
1104 1303920492000000 dread I agree this should be fixed.
1104 1303920791000000 dread Fixed on default cset:52a3fb230074 - create-test-data now indexes the package it creates.
1106 1303834069000000 rgrp fixed in branch:defect-1106-bugs-with-routing.
1107 1310134646000000 thejimmyg Child ticket of #954
1107 1320321170000000 dread "Notes to complete this:
* ckan/logic/action/get.py:'def package_autocomplete' exists and is the one that should be used
* need to ensure all places in the templates and public/scripts use this action directly, not dataset/autocomplete (or its alias package/autocomplete)
* ckan/controllers/package.py:'def autocomplete' should be removed.
* ckan/config/routes.py dataset/autocomplete should be removed."
1107 1340632612000000 icmurray autocomplete action has been deprecated in favour of /api/2/util/dataset/autocomplete : https://github.com/okfn/ckan/blob/master/ckan/controllers/package.py#L655
1108 1310128678000000 thejimmyg Now that we are using ckan.net as thedatahub.org we should go full steam ahead and create a brand new UI to reflect thedatahub.org's new role. See http://ckan.org/2011/07/08/ckan-vision-update/
1108 1311180204000000 thejimmyg See also comment on #1200 about using the pdeu theme.
1108 1314878772000000 zephod "First, here is a repo with the new DataHub theme:
https://bitbucket.org/okfn/ckanext-datahub-theme
We plan to integrate this into ckan-core, thereby replacing the current theme. "
1108 1315140879000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/9be1ae232ec3 cset:9be1ae232ec3
1109 1303862352000000 kindly It is fixed now in 1.3.4.1.
1109 1303916487000000 dread Close ticket?
1109 1304698621000000 dread Did we decide that this facility for storing non-strings via the API is a new feature, rather than a bug?
1109 1305123852000000 dread What's the status?
1109 1305124697000000 kindly I am happy this is fixed in cset:445fc04333dd.
1111 1304935139000000 lucychambers "CKAN FAQ now complete (Sections 1,2,5,6 - needs uploading onto site): http://ckan.okfnpad.org/FAQ
At CKAN Community Meetup it was decided that Sections 3 and 4 would form a separate contributor's manual. "
1112 1304358643000000 dread This is complete in the branch feature-1112-allow-search-all for release 1.3.5
1113 1304024611000000 kindly cset: 52a3fb230074
1114 1304085484000000 dread Done in cset:d7bd4b0f89de on default.
1115 1324054704000000 dread Authorization groups deprecated with the Group refactor #1477
1116 1304358664000000 dread This is complete in branch defect-1116-boolean-search-options for release-1.3.5
1117 1304277240000000 nils.toedtmann "Oh, it *does* depend against python-pastescript. Ignore.
For some reason it was not (completely/correctly) installed with ckanext-datanl on us4, but is a different issue then. Might be due to missing locale ""en_GB.utf8"" and dpkg-configrue failing."
1118 1311174062000000 thejimmyg "We haven't ever come across this. Perhaps if you could provide an example we can re-open it.
Cheers,
James
"
1119 1304104689000000 rgrp " * Merged ckanext-upload into ckanext-storage
* Got existing API systemin in ckanext-storage fully working (e.g. api section was broken -- maybe due to changed boto ...)
* API now supports GET requests and JSONP where appropriate
* Added auth form to API (could be useful for ajax form stuff going forward)
* Improved UI of upload including having ckan based file links (using redirects)
* Set uploaded-by field on uploaded files
* Checked on AWS And GS
Also did:
* Improved ofs in various ways including change to standard metadata handling and improved testing (see e.g. https://bitbucket.org/okfn/ofs/changeset/36fac29b7775)
* Tracked down [https://github.com/okfn/boto/commit/0d9635c8a9785c9b20b44ee93a0679c002961592 bug with boto and metadata updating on GS] and provided patch."
1121 1307358426000000 dread We don't need to support JSON extras at the moment.
1122 1304368075000000 dread @pudo I notice you tackling solr indexing of lists/tuples in ckanext-solr cset:ecc2d69df991 so have updated to use this on ckan.net and reindexed, but #1122 still seems an issue - any chance you can take a look?
1122 1305718009000000 dread This user is now storing his list as a string so is happy. But I'll leave this ticket open, for pudo to decide whether there is an issue with SOLR indexing tuples/lists, as per the previous comment.
1122 1306662099000000 pudo I don't think there's really a nice way to do this with the current solr indexing library, but I've commented out the list handling code which should really make handling this cleaner.
1122 1306747706000000 dread "Changing this to ""won't fix"" just to be clear"
1123 1307374691000000 dread "I'll change the architecture to ""all"" as per this article:
http://linuxreviews.org/man/deb-control/"
1123 1307790563000000 thejimmyg "This isn't necessarily a trivial thing. Let's get the build system for the packages stable before we start changing it all to support alternative architectures. Once the packaging is working well it would be trivial to switch those servers to amd64.
I'm sorry, but this isn't worth the investment in manpower at the moment."
1123 1307795508000000 rgrp You can't get small amazon instances as 64 bit and many people's own machines are 32 bit. How costly is packaging no-arch exactly (it must be more than a simple 'switch' in the build if you are saying this is costly :-) )
1123 1311863798000000 nils.toedtmann "Looks like the packages on apt.okfn.org are now architecture ""all"" - great.
To avoid future confusion, i change this ticket's solution to ""fixed""."
1124 1304601771000000 nils.toedtmann I installed the python-ckanext-solr package rom http://apt-alpha.ckan.org/datanl-dev onto us10/DataGM too, and it seems to work just fine, see http://datagm.test.ckan.net/package?q=manchester
1124 1323168132000000 thejimmyg "Solr is now properly supported in the ckan-1.5 repository.
"
1124 1323168156000000 thejimmyg Re-open this if it still affects DataGM.
1126 1304587078000000 dread Fixed in cset:61bb142a6b7c for ckan v1.3.5.
1127 1304602084000000 wwaites "The proposal is not comparable to PEP and DEP because the projects have vastly different requirements. The API stability needs of a programming language or an operating system (e.g. fundamental building blocks that you don't expect to change often or radically) are very different from a web application. The plone one is comparable.
The idea itself is a double-edged sword. It will promote stability which is good but can also tend towards rigidity and stagnation which is bad. Each added bit of bureaucracy and process means fewer people will be willing to collaborate or participate in improving the software.
Overall, -1.
"
1127 1304606094000000 dread Fixed links
1127 1305128000000000 thejimmyg This is great idea and it has taken on well. +1 from me. We can update/create new CREP policies as needed.
1127 1305458020000000 rgrp "I'm obviously a strong +1 but feel we should stick with the CEP name (we have PEP and DEP and CREP really does sound weird compared to CEP).
Also I think we may need to do a bit more to clarify when you just do a good ticket and when one does a CEP (looking at what has occurred already). I think this may be one reason to keep CEPs separate in drafting from trac as will clearly distinguish CEPs from standard tickets. "
1127 1305458489000000 rgrp one other thing: i think the plan is for CEPs to be in restructured text (for easy transfer to docs). This should be mentioned and the template in the wiki should be in ReST.
1128 1310567803000000 dread "Loader in ckanext/bin/loadscotland.py.
Done upload to ckan.net:
{{{
(pyenv-ckanext)dread@dread-laptop:~/hgroot/ckanext$ scotland_loader -k xyz -H http://ckan.net/api /home/dread/ckan-local/data/scotland-2010-10-18.csv
/home/dread/hgroot/pyenv-ckanext/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
Engine = entry_point.load()
No handlers could be found for logger ""vdm""
2011-07-13 15:29:17,983 INFO [ckanext.importlib.loader] Loading scottish_neighbourhood_statistics
2011-07-13 15:29:20,109 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:22,089 INFO [ckanext.importlib.loader] Loading traveline_scotland
2011-07-13 15:29:24,579 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:27,045 INFO [ckanext.importlib.loader] Loading sqa_qualifications
2011-07-13 15:29:30,869 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:32,561 INFO [ckanext.importlib.loader] Loading scotxed
2011-07-13 15:29:34,017 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:35,286 INFO [ckanext.importlib.loader] Loading iscjis_vocabularies
2011-07-13 15:29:36,592 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:37,782 INFO [ckanext.importlib.loader] Loading ecare_multi-agency_store_data_model_and_vocabulary
2011-07-13 15:29:39,043 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:40,267 INFO [ckanext.importlib.loader] Loading digital_archive_scottish_archive_network
2011-07-13 15:29:41,690 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:43,727 INFO [ckanext.importlib.loader] Loading the_scottish_register_of_tartans
2011-07-13 15:29:45,159 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:46,816 INFO [ckanext.importlib.loader] Loading scotlands_images
2011-07-13 15:29:48,563 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:49,983 INFO [ckanext.importlib.loader] Loading national_library_of_scotland
2011-07-13 15:29:51,709 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:53,122 INFO [ckanext.importlib.loader] Loading scottish_government_statistics
2011-07-13 15:29:55,857 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:29:57,118 INFO [ckanext.importlib.loader] Loading scotbis
2011-07-13 15:29:58,883 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:00,264 INFO [ckanext.importlib.loader] Loading historic_scotland
2011-07-13 15:30:02,086 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:03,609 INFO [ckanext.importlib.loader] Loading scheduled_monuments_in_scotland
2011-07-13 15:30:05,057 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:06,987 INFO [ckanext.importlib.loader] Loading scotlands_house_prices_service_consumer_information
2011-07-13 15:30:08,687 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:09,932 INFO [ckanext.importlib.loader] Loading air_quality_in_scotland
2011-07-13 15:30:11,278 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:12,637 INFO [ckanext.importlib.loader] Loading river_and_coastal_flooding_updates_for_scotland
2011-07-13 15:30:13,940 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:15,637 INFO [ckanext.importlib.loader] Loading public_contracts_scotland
2011-07-13 15:30:17,280 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:18,666 INFO [ckanext.importlib.loader] Loading scottish_government_consultations
2011-07-13 15:30:20,673 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:22,234 INFO [ckanext.importlib.loader] Loading scottish_public_health_observatory
2011-07-13 15:30:24,025 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:26,297 INFO [ckanext.importlib.loader] Loading scotlands_information_landscape
2011-07-13 15:30:28,308 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:29,758 INFO [ckanext.importlib.loader] Loading isd_scotland
2011-07-13 15:30:31,847 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:33,074 INFO [ckanext.importlib.loader] Loading scotlandspeople
2011-07-13 15:30:34,541 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:35,754 INFO [ckanext.importlib.loader] Loading general_register_office_for_scotland_statistics
2011-07-13 15:30:37,273 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:39,330 INFO [ckanext.importlib.loader] Loading baseline_scotland_-_groundwater_chemistry_data
2011-07-13 15:30:40,716 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:41,919 INFO [ckanext.importlib.loader] Loading the_centre_for_the_study_of_retailing_in_scotland_data_sets
2011-07-13 15:30:43,275 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:44,515 INFO [ckanext.importlib.loader] Loading growing_up_in_scotland
2011-07-13 15:30:46,215 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:47,582 INFO [ckanext.importlib.loader] Loading scottish_parliament_msp_allowances
2011-07-13 15:30:49,580 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:51,272 INFO [ckanext.importlib.loader] Loading scottish_government_ministerial_hospitality
2011-07-13 15:30:53,764 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:55,642 INFO [ckanext.importlib.loader] Loading scottish_government_expenditure_over_25000
2011-07-13 15:30:58,012 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:30:59,674 INFO [ckanext.importlib.loader] Loading scottish_local_government_financial_statistics
2011-07-13 15:31:01,869 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:03,382 INFO [ckanext.importlib.loader] Loading scottish_government_disclosure_of_high_earners
2011-07-13 15:31:07,086 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:11,493 INFO [ckanext.importlib.loader] Loading scotlands_places
2011-07-13 15:31:14,452 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:31,743 INFO [ckanext.importlib.loader] Loading scottish_national_heritage_facts
2011-07-13 15:31:36,223 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:37,830 INFO [ckanext.importlib.loader] Loading scottish_natural_spaces
2011-07-13 15:31:40,245 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:41,764 INFO [ckanext.importlib.loader] Loading scotland_national_address_gazetteer
2011-07-13 15:31:43,810 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:45,069 INFO [ckanext.importlib.loader] Loading scottish_environment_protection_agency
2011-07-13 15:31:47,975 INFO [ckanext.importlib.loader] ...creating package
2011-07-13 15:31:49,525 INFO [ckanext.importlib.loader] Loading traffic_scotland
2011-07-13 15:31:52,396 INFO [ckanext.importlib.loader] ...creating package
}}}"
1128 1310568028000000 dread Note, I added a title column and made one or two minor corrections that I spotted - see attached CSV for the result.
1129 1305116419000000 thejimmyg "Great proposal, here's my thinking:
* The current VDM model (where we have things like a single package table which logs changes to a package_revision table) is very well tested and works well and we should only consider a radical change with good reason
* Although we are making a move from *model objects* being the central components to the *logic layer functions* being the central components, there is no need yet to have the database structures we store (for revisioning or otherwise) start to look like the dictized representation. The logic layer itself is designed to do that mapping.
* Since we do have this logic layer though, there is less need for ""magic"" SQLAlchemy objects to support dicts and lists (eg the stateful objects used to make a list of tags on a package behave like a list), a few simple (and easily debuggable) queries in the logic layer would work better.
In the longer term we may want to look at serialising more dictized-looking objects and that may lend itself to a more dictized changeset model or even a more dictized storage system (eg no-SQL) but for the time being we are not at that point.
I recommend the following:
* We continue to use the current default branch of VDM (not the changeset branch)
* We continue to treat the package table (and other non-revision tables) as the most recent revision (even though the *active* revision displayed in CKAN might well be in the revision tables because more recent changes haven't been moderated yet)
* We slowly stop using stateful lists and dicts in CKAN because we have move control with explicit queries in the logic layer
Sound good?
"
1129 1305139980000000 rgrp "It seems like this ticket is getting a bit confused with ticket:1076 (changes re vdm) :-)
Some general conclusions relevant to both:
* General feeling is not to change to changeset model
* I'm personally +0 on changeset model
* I would note main benefit is simplicity and orthogonality of changesets to core system. I also think cost of change is small.
* That said we can simplify current model as well.
* There seems some misunderstanding: change to have logic layer has almost nothing to do with being able to remove main stateful stuff in vdm. To be able to remove most of stateful stuff in vdm requires us to make some other changes (re foreign keys from revision objects to continuity)
* There are other simplifications we should make to vdm before embarking on this (e.g. move to SessionExtension from MapperExtension). This is easy as that work has been done in changeset branch and can be backported.
What I don't see in this CEP as yet is any discussion of the complexities around pending stuff (e.g. do we allow multiple pending, do new edits get shown current or latest pending). I'll try and comment more on this but a lot of this was in original vdm discussion last summer."
1129 1305209601000000 rgrp "Replying to [comment:2 rgrp]:
> It seems like this ticket is getting a bit confused with ticket:1076 (changes re vdm) :-)
I of course ;0 meant ticket:1077"
1129 1305212467000000 kindly "> * There seems some misunderstanding: change to have logic layer has almost nothing to do with being able to remove main stateful stuff in vdm. To be able to remove most of stateful stuff in vdm requires us to make some other changes (re foreign keys from revision objects to continuity)
The logic layer does not automatically help out, however it makes our life easier if we want to handle state ourselves. For example take package tags, if we remove the stateful_m2m properties and just use normal sqlalchemy relations. We will still want statefullness (i.e active, pending, deleted) on the package_tags table. We should update those on the table ourselves in the logic layer.
> * There are other simplifications we should make to vdm before embarking on this (e.g. move to SessionExtension from MapperExtension). This is easy as that work has been done in changeset branch and can be backported.
I agree but event thought the MapperExtension way is not great, it is very well field tested."
1129 1325268100000000 rgrp More UI work remains in #1141
1130 1340633514000000 icmurray Unassign for triage.
1131 1305537897000000 dread Fixed on branch release-v1.3.5 cset:3461ed6bab0c
1132 1307352675000000 dread "Yes, these are apparently search-related and so are skipped when testing under sqlite. See README.rst for more info.
Compare with this:
{{{
$ nosetests --ckan --with-pylons=test-core.ini ckan/tests/functional/test_authz.py
............................................
----------------------------------------------------------------------
Ran 44 tests in 69.287s
OK
}}}"
1133 1324057072000000 dread "Works for me - change occurs successfully.
{{{
(pyenv-ckan)dread@dread-laptop:~/gitroot/ckan$ paster rights add russianfan admin warandpeace
/home/dread/gitroot/pyenv-ckan/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
Engine = entry_point.load()
User russianfan -> is admin on -> Package warandpeace
(pyenv-ckan)dread@dread-laptop:~/gitroot/ckan$ paster rights
/home/dread/gitroot/pyenv-ckan/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages)
Engine = entry_point.load()
22 results
User visitor -> is editor on -> Package annakarenina
User logged_in -> is editor on -> Package annakarenina
User testsysadmin -> is admin on -> System system
User logged_in -> is editor on -> System system
User visitor -> is reader on -> Group david
User annafan -> is admin on -> Package annakarenina
AuthorizationGroup anauthzgroup -> is reader on -> Package warandpeace
User visitor -> is editor on -> Package warandpeace
AuthorizationGroup anauthzgroup -> is editor on -> Group david
User logged_in -> is editor on -> Package warandpeace
User logged_in -> is reader on -> Group david
User visitor -> is reader on -> Group roger
User logged_in -> is reader on -> Group roger
User russianfan -> is admin on -> Package warandpeace
User visitor -> is admin on -> Package warandpeace
AuthorizationGroup anotherauthzgroup -> is editor on -> AuthorizationGroup anauthzgroup
User russianfan -> is admin on -> Group roger
User russianfan -> is admin on -> Group david
AuthorizationGroup anauthzgroup -> is admin on -> AuthorizationGroup anotherauthzgroup
User visitor -> is anon_editor on -> System system
User visitor -> is admin on -> Group roger
AuthorizationGroup anauthzgroup -> is editor on -> System system
}}}
Tests would be nice, but it currently works, is only for admins and is therefore not as high priority as other other tests, such as front-end js."
1134 1305125228000000 thejimmyg "Great stuff.
Yes, I agree with all this, but as initial thoughts I suggest that for phase 1, the info() method returns one more key called ""form_config_interface"" which takes a string representing a the type of interface for the config field on the form. If the key is missing it is treated as ""None"".
The two possible values are:
None - the field is not present and must always be stored as NULL
Text - a single text field will be provided on the interface
Whatever the interface, the value built will always be stored in the config column of the table.
We then also provide a ""get_schema()"" method that returns a schema cablable of parsing the data submitted by the form and storing it as a single key named ""config"". ckanext-inspire will then add other functions to the schema for URL etc so that regardless of what the harvest plugin does, the key fields ckanext-harvest are always there.
This then allows:
* Customiseable config interfaces
* Customisable valiation
* A consistent place to store config in the database
Sound good?"
1134 1305623531000000 sebbacon "Agreed with James that we should consider *from the start* how to provide for a nice UI -- my worry is if we start out with people editing raw JSON we'll end up keeping that for years :)
"
1135 1305210134000000 rgrp "Work on this is largely complete in changeset branch: https://bitbucket.org/okfn/vdm/src/changeset/
Overview docs: https://bitbucket.org/okfn/vdm/src/changeset/doc/changeset.rst"
1135 1340632267000000 icmurray Re-assigned to kindly
1136 1305210925000000 rgrp "Most of this change has been implemented in the changeset branch [1] but could be backported reasonably easily to main.
[1]: https://bitbucket.org/okfn/vdm/src/changeset/"
1136 1305216218000000 kindly This would be nice but it is not necessary. The current mapper implementation may be nice but not is battle tested..
1136 1305217553000000 kindly "Replying to [comment:2 kindly]:
> This would be nice but it is not necessary. The current mapper implementation may be nice but not is battle tested..
I meant the mapper extension ""IS"" battle tested. "
1136 1340632980000000 icmurray Re-assigned to @kindly.
1137 1305211672000000 rgrp "Most of this change has been implemented in the changeset branch [1] but could be backported reasonably easily to main.
[1]: https://bitbucket.org/okfn/vdm/src/changeset/"
1137 1305217449000000 kindly "I am not sure this needs to be done. I think we should keep the continuity object always in the table, even if it is deleted. The querying should be done through the logic layer so the deleted state should not be an issue.
The clients should be entirely state aware.
The only thing that needs to be done is to remove all statefullness of relations. These are the only things that are complicated.
This would make vdm just a simple copy on write mechanism, with the client controlling the state."
1138 1316965357000000 rgrp Cannot see this issue any more on default (as deployed on e.g. test).
1140 1312491320000000 kindly "fixed cset:987da68ea4f6
The package group table needed to trigger a reindex of the package.
"
1140 1312537257000000 dread Broken since CKAN v1.1. I have added this fix to v1.4.2 and v1.4.3 releases.
1141 1308828928000000 dread "I've merged this into default in preparation for release 1.4.2, but there may be a couple of useful additions that could be done on this or another branch/ticket.
Getting a package at a specified revision is really useful. This can be used:
* links in the package history (#103) (while in there, the 'moderated' flag should be shown in the package history too if not already)
* should also be added to the package API (as a replacement for the AJAX call?)
The history_ajax call looks like an almost duplicate of the Package Revision API. Can these be merged and put just in the API?
"
1141 1318862183000000 dread What's the status of this now? The ckanext-moderatededits README still says you need a branch of ckan. Can you say which version of CKAN is required instead? I say we close this if ajax call is now in a better place in the API. (I expect it is now, with the logic layer).
1141 1318862773000000 johnglover Well, it runs on datacatalogs.org with the current default so CKAN 1.5, but it needs quite a bit of updating to work with all of the new 1.5 UI changes, and as it has been a low priority for quite a while I haven't scheduled time to work on it. Really I would need to spend a few days on it to tidy it up for standard 1.5.
1141 1325352507000000 dread Closing, with remaining tidy up work for newer CKAN versions split off into new ticket: #1604
1142 1305880901000000 sebbacon "The guys at Wikimedia also produced this spreadsheet including notable
users and features of other software. Could be useful.:
https://spreadsheets2.google.com/ccc?authkey=CLWu1cgP&hl=en&key=tYiwc9kUVLLlALOQEzFqujw&hl=en&authkey=CLWu1cgP#gid=0
"
1142 1305881244000000 lucychambers "Replying to [ticket:1142 rgrp]:
> Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927
>
> * CKAN 1-page overview (for enterprise and for data hackers)
> * Administrator's Guide (including install)
> * Extensions Guide
> * Separate CKAN.net info from Software Documentation (?)
>
> == Minor Items ==
>
* Include guide on how to configure local filesystem storage, instead of Google storage. (http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001235.html)
"
1142 1308326118000000 annapowellsmith "List of new docs tasks agreed by APS and RGRP:
* Tutorial on the wiki (long-term): Integration with Wordpress and Drupal and Javascript
* Overview of domain model both for average users and for devs
* 2-page glossy brochure
* Vision for CKAN http://notebook.okfn.org/2011/04/27/data-hubs-data-management-systems-and-ckan/
* Comparison : use http://wiki.ckan.net/Related_Software as starting point - build on wiki - Socrata and OGDI
* Renaming: separation between ckan.org and ckan.net
* More documentation in the code ...
* Config options generated from code or move this to WUI ...
See also proposed overhaul at http://wiki.ckan.net/Documentation_Plans"
1142 1313775665000000 rgrp This was completed 4 weeks ago and can now be marked as complete. See http://docs.ckan.org/ and http://wiki.ckan.net/ and http://ckan.org/
1143 1307024681000000 dread "All done in ckanext-stats cset:bc54790cacd4 (apart from ""fix x axis to start at first revision"" which was not a problem after all)"
1148 1305969925000000 kindly complete cset:96a43c9d8bd7
1149 1305971672000000 kindly cset:b1634d405066
1149 1306090663000000 kindly "Failed dgu tests due to as_dict not working on deleted objects.
fixed by cset:c9b9cf513e44"
1150 1311178163000000 thejimmyg "Hi John,
Can you please check that the new webstore+preview works correctly with this one too please?
Cheers,
James"
1150 1311774141000000 johnglover Unicode data should be fine in the new datapreview code, for example: http://test.ckan.net/package/afghanistan-election-data
1151 1330908188000000 rgrp Not sure we will be able to fix this easily as we need a dataproxy for general case. Think it better to focus on the case where data in DataStore and we use Recline. As such closing as wontfix for the present.
1152 1307352165000000 amercader Bumping to ckan-v1.5-sprint-3 and updating the CC email addresses so people actually get any updates.
1154 1314120497000000 johnglover Fixed in core (branch feature-1275-solr-search).
1154 1314177477000000 dread "Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?)
I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request. e.g.
{{{
Module ckan.controllers.home:29 in index
<< query = query_for(model.Package)
query.run(query='*:*', facet_by=g.facets,
limit=0, offset=0, username=c.user)
c.facets = query.facets
c.fields = []
>> limit=0, offset=0, username=c.user)
Module ckan.lib.search.common:115 in run
<< self.query = QueryParser(query, terms, fields)
self.query.validate()
self._run()
self._format_results()
return {'results': self.results, 'count': self.count}
>> self._run()
Module ckanext.solr.backend:63 in _run
<< # this wrapping will be caught further up in the WUI.
log.exception(e)
raise SearchError(e)
finally:
conn.close()
>> raise SearchError(e)
SearchError: [Errno 111] Connection refused
}}} (recently from nl.ckan.net)"
1154 1314178325000000 johnglover "> Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any
> packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a
> useful thing to do? Without search, perhaps CKAN should simply not start?)
When discussing the ticket we agreed that we should allow ckan to start without search. It is possible that people simply want to list/view information without searching for it and we decided not to prevent them from doing so.
> I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this
> time pudo and I get exception emails saying connection failed for request.
Ok, other options then:
* if an error occurs during search, return 0 results
* if an error occurs during search, redirect to a generic 'search is unavailable' page
* if an error occurs during search, retry the query a certain number of times before giving up and doing one of the above.
What do you suggest?"
1154 1314179290000000 dread "Search requests - option one seems like a bad user experience, and option three seems too complicated. I should do option 2, explain it may well be a temporary problem, and let the user press reload, as per any normal web request.
A more difficult problem though is what to do about a failed search indexing. Because of the try/except catch we've had until the patch yesterday, we get no exception emails for this, but I'm worried that this has caused packages to not be indexed, and hence are essentially lost in CKAN (unless someone realises there is a problem and does a reindex at some point).
BTW I added paster command 'search-index check' for looking at this problem, but I don't think it was ever added to solr. Is it easy to add get_all_entity_ids for the purpose of checking for this problem?"
1154 1314191496000000 johnglover "Ok I'll implement option 2 for search query issues.
Yes it is easy to get a list of IDs from Solr so I'll have a look at that command and make sure that it works. I think a case could be made for not allowing a package to be added if it cannot be indexed however, and making the user simply retry the request."
1154 1314192324000000 dread Yes, forcing a retry may be possible, although I guess on these occasions you'd be looking at rolling back the creation of the package object too, which is not really in the spirit of the notifications system at present. This is why we had the queue for solr for occasions like this - is it here no longer?
1154 1314192753000000 johnglover Not in core, there is a queue feature in the extension still which I could add to core. I think though it was a case of using either the queue or synchronous search, I don't think it would automatically fall back to the queue if there was an indexing problem, but we could look at that. Was there not talk of changing the queue system recently though? There are also no tests for the Solr queue so that would take a bit of work as well.
1154 1314193633000000 johnglover RE search queries failing: Just having a look through the controller code and the package search controller and the search api both perform option 2 already, so I'll just update the home controller.
1154 1314195684000000 dread "Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception, this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this tally with what you mentioned in the last comment or is what's there better?
Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo know if it is?). In this case we should be clear that it is not what it says it is: ""This extension adds support for package search using Apache Solr to CKAN"" and maybe simply mark it as deprecated. Pudo what do you think?"
1154 1314196683000000 johnglover "> Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception,
> this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this
> tally with what you mentioned in the last comment or is what's there better?
This sounds good to me, definitely better than what is currently there.
> Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo
> know if it is?).
I'm not actually sure if the queue works or not, but if it does I'll have to write some tests for it before we could add it to core. A working queue system would be useful generally however, as there are other things that it would be useful for (such as QA), but yes, going forward the Solr extension should really be deprecated."
1154 1314287519000000 johnglover "All attempts to index packages should now throw a SearchIndexError if Solr is unavailable and the create/update will be rolled back. These errors are caught in the main package controller (redirecting to an error page) and the API controller. I have added tests for both controllers.
Also updated the paster 'search-index check' command to work with solr."
1155 1306773174000000 pudo "Now implemented based on CSV export available on the catalogue page:
* http://data.london.gov.uk/catalogue"
1156 1306855111000000 pudo Both are now implemented but may need further work to adhere to developing conventions wrt to location etc.
1159 1307615133000000 pudo Rudimentary RDFa has been added with seeAlso to API.
1161 1306408026000000 amercader Duplicate of #1157
1164 1308647224000000 amercader Done. See it in action here: http://publicdata.eu/map
1165 1306413953000000 nils.toedtmann this is similar, but not equal to #1096
1165 1307444733000000 nils.toedtmann See also http://ckan.okfnpad.org/multisite
1166 1306847326000000 thejimmyg Also, rather than INSPIRE=true we should probably have information about the harvesting mechanism and the harvest object type? This also relates to our use of the kind field on the resource.
1166 1317381932000000 amercader Fixed on d5bee3de9957
1167 1306856992000000 sebbacon "Yes to all the above.
Further question: which extensions should be installed as standard? I would argue:
* ckanext-stats
* ckanext-googleanalytics
* ckanext-disqus
* ckanext-solr (implies installing a running solr, too...)
* ckanext-follower
* ckanext-admin
Note that googleanalytics and disqus both require user accounts set up at the corresponding service. Not sure how we would handle this in the instance setup.
"
1167 1306857142000000 sebbacon Another thing to consider: how do we ensure we keep a single, up-to-date version available? (I'm thinking wrt developer / sysadmin workflow)
1167 1306857750000000 thejimmyg "Creating the AMI is really just the icing on the cake. The hard bit is getting the packaging right so that's where we need to concentrate first. Once the packaging works we just make sure apt-get upgrade is run regularly.
+1 to the list above.
I also agree we should install solr by default. Who has actually done a CKAN solr install? I haven't but together with whoever has I can try to package it up.
Wrt to usernames and passwords, I'll look into how dpkg manages those blue pop-up screens for entering configuration options."
1167 1307358318000000 dread "debconf seems the preferred way to question the user post-install and run setup scripts. It seems a bit of a faff.
The person that is able to spin up an EC2 instance is no doubt technically able enough to follow a couple of instructions in README.txt saying edit /etc/ckan//.ini and add your Google Analytics credentials. So I don't think this ticket is the driver for a debconf setup."
1167 1311178516000000 thejimmyg "I've now written a 20 page guide on using CKAN on Amazon EC2. I haven't explicitly created an AMI, but I don't think we should do that anyway for a few reasons:
* it is fiddly
* there is no advantage to the user as CKAN installation is so easy and so well documented
* we don't want to have to maintain the image going forward
* we might not want to promote a non-open approach
The guide will form part of Anna PS's work on the overall documentation but you can also see the original version here:
https://bitbucket.org/okfn/ckan-debs-public/src/489a5ecf369f/ec2/"
1168 1310134901000000 thejimmyg This is now working on dgu-buildbot.okfn.org. The code that manages it is partly the repo.py script from http://hg.3aims.com/public/BuildKit/ and partly the fab code in https://bitbucket.org/okfn/buildbot-scripts
1168 1310396585000000 dread This might run manually on the us7, but the buildbot is not running these automatically or any sort of testing, which is what this ticket is about.
1168 1325866142000000 thejimmyg "Quick update... we now have:
* An automated way to build the deb packages
* An automated way to install them into a fresh virtual machine
We don't currently run them via buildbot or have smoke tests.
"
1169 1306754833000000 dread Done on default. Will appear on ckan.net next time it is branched and updated.
1170 1306864422000000 dread Done in cset:4d2d0857281f on default
1171 1347358705000000 ross "I have implemented this on the datahub, but this change likely needs to be made to CKAN core. Probably as part of 2.0.
The format on tdh is currently:
${TITLE}. ${AUTHOR}.
Retrieved ${UTC time}.
${DATASET LINK}
Best discuss and agree on what format should be supported. Mark, I've assigned this to you as it is tagged academia ;)
"
1172 1312991332000000 dread I've started this on branch defect-1172-exceptions
1172 1314118281000000 dread "Is in cset:47225f03e2af and merged to default for 1.4.4.
It now barfs instead of failing silently:
* if the ckan config file has any errors
* search indexing error
There are some other ""except Exception"" statements "
1172 1314118883000000 dread "I took out ""raise"" by mistake in ckan/lib/search/worker.py:26 but John has added it in in the corresponding place in ckan/lib/search/__init__.py on branch feature-1275-solr-search anyway."
1173 1307000249000000 pudo "I propose we do a bit of frameworkiness here: set up a two-part content type registry and refactor the API controller to use it.
The idea would be to do the following things:
1. Create a registry mapping format extensions to mime types: RepresentationRegistry.add_format('json', 'application/json')
2. Create a registry of (mime types, entity types) -> converter functions that yield an appropriate representation of the entity.
3. Hook this into the _finish method of the API controller based on Accept: handling
4. Add support for /api/rest/ENTITY/NAME.{format} and /ENTITY/NAME.{format} to routing, use registry from (1) to rewrite accept headers.
5. Register converters in load_environment or via IConfigurable plugins
6. Document
What do you think?
"
1173 1307615271000000 pudo Now available in ckanext-rdf and linked to from WUI.
1174 1307615200000000 pudo "wontfix as thejimmyg wants to turn REST API into RPC API and this doesn't fit in.
Not an excellent plan in my idea but this is to be discussed elsewhere. "
1175 1307350866000000 dread Flavio, can you post here the exception that you get?
1175 1307372833000000 fccoelho Thats the thing, I don't get any error message, no even in the logs. But I can't even load a page.
1175 1314030683000000 dread Hi Flavio, is this still a concern? I'd like to close this ticket unless you can provide details of the error from your logs or exception email.
1175 1325355170000000 dread This error (which will be in the apache or ckan log) must be unrelated. Stats extension runs fine elsewhere.
1176 1310557852000000 dread This is occurring several times a day now - James can you suggest what's happening?
1176 1314273012000000 dread "This occurs when you browse {{{/error/document}}} directly, rather than being directed there on error. In this case there is no 'original_response'.
Fixed in cset:c67d5934fb0b for ckan 1.4.3 and merged to default."
1177 1307360643000000 rgrp What revision of the CKAN code are you using (do hg id to get revision if installing from source). Have you modified the package template in any way? Have you restarted apache after upgrading?
1177 1307361157000000 fccoelho "to upgrade I used the method describeb on bitbucket, after upgrading pip-requirements:
pip -E pyenv install -r pip-requirements.txt
after that I did a
paster --plugin ckan db upgrade --config ckan.emap.fgv.br.ini
the revision id on my pyenv/src/ckan is:
5730c79db461+ tip
yes, I have restarted apached after upgrading"
1177 1307365496000000 rgrp "The issue here is that the line should never be reached because g.has_commenting should be False: https://bitbucket.org/okfn/ckan/src/5730c79db461/ckan/lib/app_globals.py
@Flavio: have you modded your code in any way? is self.has_commenting False in your app_globals.py
A quick fix would be to remove the template lines calling to remove the lines in the template:
{{{
${h.subnav_link(c, h.icon('comments') + _('Comments & Questions'), controller='package', action='comments', id=c.pkg.name)}
}}}"
1177 1307366621000000 fccoelho "I have not modded the code in any way...
I'll look for those lines in the template.which template is it? read.html does not have a line such as this."
1177 1307367354000000 fccoelho OK I did an fgrep on all package template and found the line on layout.html, I'll comment the block and test.
1177 1307367923000000 fccoelho "Ok it fixed it!
thanks a lot!!
Is there any side effects to the removal of this line, I should know of?"
1177 1307374897000000 dread "I think rgrp was suggesting you have mistakenly set has_commenting to ""True"" in app_globals or elsewhere. But either way, this should have no side-effects."
1178 1308045351000000 rgrp Fixed. This was due to incorrect config of bucket option.
1180 1307544223000000 dread "Both issues solved using a whitelist on anchor tags.
Second issue was a link with unicode expression of the quote. e.g. somelink"
1181 1307532816000000 dread Fixed in ckan default in cset:72f7d48d7f31. Have left Package url and Resource url links though, as these are the key links we want google to see. So we need to check these carefully for spam.
1183 1311181719000000 thejimmyg "Hi John,
Can you have a look at this one too please?
Cheers,
James
"
1183 1311771069000000 johnglover Fixed in new datapreview code (currently running on the test.ckan.net), which now gets the preview data from a webstore instead of the dataproxy API. It also only now shows a preview button for resources that were successfully archived and saved in the webstore.
1186 1310556519000000 dread This was done by pudo in branch feature-1198-mailer and went into default, lined up for release 1.4.2.
1187 1308143605000000 dread "Also stopped creating revision for user edits.
Done in cset:795ccd6405ba for 1.4.1 release"
1189 1323173227000000 thejimmyg Closing this ticket in line with ticketing policy since it is over 6 months old. If someone would like to develop an extension that uses spideroak as a back end we can look at it again.
1190 1320143592000000 rgrp A very substantial portion of this got done as part of 0.5 including the webstore #1208 and the majority of automated conversion #1398. #1432 is currently deferred so we're going to close this ticket on basis that item (3) is now out of scope.
1191 1308650930000000 dread Done in cset:c3822feffb38 for 1.4.1 release
1192 1308337652000000 dread "I really think forms are a key asset of CKAN, and keeping the docs about customising and integrating them is important - i.e. forms.rst and forms-integration.rst. There are ideas being suggested for replacing bits of front-end, but forms hard to do, and David's new system is a significant offering on top of the CKAN back-end, so it seems crazy to me to abandon it.
forms.rst has been updated by David Raznick in the release_v1.4.1 branch.
i18n.rst - include here the creation of the pot file, deploying a translation, transifex admin
load-testing.rst - I think performance should be mentioned in the glossy guide (some sort of basic metrics of speed profile, to return web pages, run searches and API operations, for 100 or 1000 packages, on a given spec machine) and have it based on a report showing how we measured it. The report would look a bit like this file, and it might do as a stub, but it seems pretty lacking.
distributed.rst - agreed, remove.
design.rst - agreed, delete.
loaders.rst / loader_scripts.rst - yes move to wiki
feeds.rst - this should be split into a tutorial for the wiki and the reference page left in sphinx
importer.rst - this should be moved to ckanext-importer/doc
admin.html - this has already gone in 1.4.1
authorization.rst - This is a bit jumbled. We're talking about changes to the code, so maybe not worth working hard to make this good, so I suggest leaving it, but delete the boring section ""Requirements and Use Cases"".
deb.rst - this is an administrator task, so should stay why move this to the wiki?
buildbot.rst - ok to stay here?
vm.rst - I assume there's no problem with this"
1192 1310135468000000 thejimmyg This is now a child ticket of #1142
1192 1312191462000000 rgrp Marking this as closed as Anna has completed and this is now deployed at http://docs.ckan.org/.
1193 1309768960000000 kindly fixed cset:87d6140e06ad
1194 1317043212000000 dread "Suggest changing this to ""Welcome, Rob"" rather than ""Welcome back, Rob"", avoiding the problem."
1194 1317053674000000 dread Done in cset:49e2c05e69ea
1194 1317053688000000 dread Will go into ckan 1.5.
1195 1311332049000000 dread Also this one: http://ckan.net/storage/f/file/
1195 1311841763000000 rgrp Fixed in https://bitbucket.org/okfn/ckanext-storage/changeset/49734285f1ef
1196 1308743857000000 dread "Reason here:
http://www.freewisdom.org/projects/python-markdown/Tickets/000018
Using the suggested workaround. Fixed on default for 1.4.2 in cset:715212cd220c"
1197 1340010256000000 toby "personally I'm against these for 4 reasons
a) The site should be self explanatory and if it is not this is a failure of design
b) a cookie would be needed to prevent these happening all the time - cookies are bad as they interfere with caching proxies.
c) I find stuff like this patronising/annoying for the user
d) They are a maintenance burden for ckan as we do not have a single instance
Therefore I'd suggest closing this as a won't fix - I agree this is just a personal view point and therefore am not going to actually close the ticket but just record my opinion."
1197 1340018903000000 seanh "Hi Toby, yes I think I agree with you, it reminds me of when I install new Android apps on my phone, some of them start off by bothering me with some sort of guided tour or long instructional text, I hate this and always skip through without reading it, expecting to be able to figure out the interface for myself.
The one thing that something like this could be useful is, rather than showing someone how to use the site, showing them what the site is capable of so they can assess whether it meets their needs/is something they're interested in. However, we already have a good guided tour of this type on ckan.org/features, those pages are really pretty good.
Close this as wontfix?"
1197 1340019675000000 toby "Thinking horrible thoughts, there was a rumour of a ckan tour, for something like that this would be useful.
Not sure where that plan is at? if there is a ticket for it maybe link to this one and close?"
1197 1340020357000000 seanh "AFAIK there isn't a ticket for that, it was something originally requested as part of the demo site, but since the devs think it's a horrible idea I don't think it's going to happen, easiest way to make a CKAN tour would be a series of static pages like the one we already have on ckan.org, I think that combined with a real functioning demo site probably meets the requirement.
Anyway, closing this as wontfix"
1198 1308821489000000 pudo "This ticket is essential I think. I would propose we should be a bit more radical here, though:
* Introduce an ""Agent"" SQLAlchemy object and derive both publishers and users via single table inheritance.
* Make agent the central object of authorization, e.g. by introducing c.agencies including the user account, visitor, logged_in, etc.
* Abolish AuthorizationGroups in favor of publishers. Yes, they are different but the use case for what AuthzGroups do and Publishers don't is just nerdery.
* Restructure URLs to be // rather than /package/ and have / be a useful list of managed resources rather than a revision list (users current home page).
* Make Agents a primary facet for all kinds of navigation.
Of course this would be done incrementally but I think we should agree on some such scheme as a marching direction. "
1198 1308824870000000 dread "I think it is a good idea to allow both users and publishers to be the object of authorization. The user_object_role table and all logic surrounding it would be simpler.
I deliberately didn't suggest any detail on the authz front because I know other related changes are afoot so it would be useful for David Raznick to contribute here.
I really like the url ideas - more like github. / having a list of packages and a search box, plus a link to the latest revisions would be great. /package/ would usefully redirect to //. I do think this should be split off into a separate ticket though. What do you think?"
1199 1308822557000000 rgrp Not sure we need exposure this via the API (at least at this point) as things like auto mass emailing could be in done in separate component (that talks to CKAN).
1199 1308922389000000 pudo merged to default with password reset forms
1199 1310555589000000 dread "This was done in branch feature-1198-mailer. No tests - ticketed here: #1222.
Additional feature in this branch 'password reset' was ticket #1186."
1200 1311180218000000 thejimmyg This is a duplicate of #1108. Let's have the discussion there please.
1202 1314030352000000 dread "The CKAN package page links to datapkg at: http://blog.okfn.org/2010/02/23/introducing-datapkg/ (it has always been this link).
I think Rolf is referring to the three links to knowledgeforge.net in the blog post itself. These need to be edited by Rufus. Note datapkg has moved again to github."
1202 1315821804000000 rgrp This now irrelevant and relevant content has gone from thedatahub theme.
1204 1311179980000000 thejimmyg "Sorry, is this a CKAN issue or a datacatalogs.org one? If it is datacatalogs.org do you mean a catalog rather than a package? If so, can you give an example of the sort of change you've made?
Marking as invalid until we get more information."
1205 1309768720000000 kindly "fixed in default cset:5e2070688e54
The upgrade now works locally, so the upgrade should work now."
1205 1310568631000000 dread This worked - cheers David.
1206 1309450216000000 dread "Fixed with cset:149be76faabc on default branch, aiming for ckan 1.4.2.
Thanks for help from Eric Lemoine."
1207 1311325343000000 dread This has been fixed.
1208 1310124599000000 thejimmyg Rufus and Friedrich are working on this at the moment so putting into the current sprint. I'm considering this as preliminary investigation to inform the wider project that David Raznick is leading on.
1208 1311525306000000 rgrp Propose this ticket can be closed as webstore is now functional and being used. Can open new tickets for specific improvements / adaptations as the need arises.
1208 1312191646000000 pudo "This should be closed:
* https://github.com/okfn/webstore - See README for details
* Python client lib: https://github.com/okfn/webstore-client
* http://webstore.thedatahub.org/ - Demo install
* http://wiki.ckan.net/Webstore - Overview
"
1209 1310571715000000 dread Was fixed in cset:d959a70a19ea. Bug was introduced and fixed before a release took place.
1210 1309974781000000 dread Fix in ckan for AssertionError in cset:8f6ba8ef63f3 lined up for CKAN release 1.4.2.
1211 1315910650000000 dread I've listed what was checked into this branch 6 weeks ago and is going into release 1.4.3. Is this ticket ready to close now or are there future plans?
1211 1315948703000000 rgrp Let's close and re-open (or open a new ticket) for future work.
1214 1309982828000000 dread "I'm working through these on branch defect-1214-api-improvements.
""Server returns text/html for errors"" - now fixed in cset:0a533a8da3df"
1214 1310041083000000 dread "Aron,
I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: ""Should be 405 Method Not Allowed?"" DELETE should be allowed. Or is that a response you're getting?
David"
1214 1310041605000000 dread """Tag returned as a JSON object when updating but as a string when requesting.""
I'm not sure what you mean.
http://test.ckan.net/api/2/rest/package/reference-example
gives tags: {{{""tags"": [""country-uk"", ""example-package""],}}}
Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix.
http://test.ckan.net/api/2/rest/tag/country-uk"
1214 1310042268000000 dread "Fixed tag entity returning package names for apiv2 - it now gives IDs.
The problem mentioned about ""Extras"" content - I'm not sure what's meant here. Package extras are just strings. You might choose to store a package name in an extra, but there is no magic to convert it to a package id should you retrieve the package via apiv2. Is this just a misunderstanding or have I missed your point?"
1214 1310044531000000 dread "Deleting an extra using 'null' works for me:
{{{
$ curl -d '{""name"":""dtest2"", ""extras"":{""1"":""1"", ""2"":""2"", ""3"":""3""}}' http://test.ckan.net/api/rest/package -H ""Authorization: tester""
...
$ curl -d '{""name"":""dtest2"", ""extras"":{""1"":""1"", ""2"":""2"", ""3"":null}}' http://test.ckan.net/api/rest/package -H ""Authorization: tester""
{""maintainer"": null, ""name"": ""dtest2"", ""relationships_as_subject"": [], ""author"": null, ""url"": null, ""relationships_as_object"": [], ""notes"": null, ""title"": ""dtest2"", ""maintainer_email"": null, ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""author_email"": null, ""state"": ""active"", ""version"": null, ""groups"": [], ""license_id"": null, ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""tags"": [], ""id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""resources"": [], ""extras"": [{""state"": ""active"", ""value"": ""\""1\"""", ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""package_id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""key"": ""1"", ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""id"": ""d1937073-7bfc-48c5-b6ff-b00d90b451ae""}, {""state"": ""active"", ""value"": ""\""2\"""", ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""package_id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""key"": ""2"", ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""id"": ""8147886f-9769-440c-8c35-b7d6a2f46de7""}]}
}}}
"
1214 1310045026000000 dread "Ok I see the problem with extras. When you create a package you get back 'simple' extras:
{{{
$ curl -d '{""name"":""dtest3"", ""extras"":{""1"":""1""}, ""tags"":[""australia"", ""barbeque""]}' http://test.ckan.net/api/rest/package -H ""Authorization: tester"" --trace log
{""maintainer"": null, ""maintainer_email"": null, ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""metadata_created"": ""2007-04-10T21:19:38"", ""relationships"": [], ""metadata_modified"": ""2011-07-07T13:06:06.228718"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [""australia"", ""barbeque""], ""groups"": [], ""name"": ""dtest3"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""ckan_url"": ""http://test.ckan.net/package/dtest3"", ""notes"": null, ""title"": ""dtest3"", ""ratings_average"": null, ""extras"": {""1"": ""1""}, ""ratings_count"": 0, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d
}}}
i.e. ""extras"": {""1"": ""1""}
and that's what you get if you GET the package:
{{{
$ curl http://test.ckan.net/api/rest/package/dtest3
{""maintainer"": null, ""maintainer_email"": null, ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""metadata_created"": ""2007-04-10T21:19:38"", ""relationships"": [], ""metadata_modified"": ""2011-07-07T13:06:06.228718"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [""australia"", ""barbeque""], ""groups"": [], ""name"": ""dtest3"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""ckan_url"": ""http://test.ckan.net/package/dtest3"", ""notes"": null, ""title"": ""dtest3"", ""ratings_average"": null, ""extras"": {""1"": ""1""}, ""ratings_count"": 0, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d11""}
}}}
but if you then edit it you get them expressed much more verbosely:
{{{
$ curl -d '{""name"":""dtest3"", ""extras"":{""1"":""2""}, ""tags"":[""australia"", ""barbeque""]}' http://test.ckan.net/api/rest/package/dtest3 -H ""Authorization: tester"" --trace log
{""maintainer"": null, ""name"": ""dtest3"", ""relationships_as_subject"": [], ""author"": null, ""url"": null, ""relationships_as_object"": [], ""notes"": null, ""title"": ""dtest3"", ""maintainer_email"": null, ""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""author_email"": null, ""state"": ""active"", ""version"": null, ""groups"": [], ""license_id"": null, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d11"", ""tags"": [{""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""state"": ""active"", ""id"": ""4aa0e776-ac2a-4a8b-82ba-d80237d35596"", ""name"": ""australia""}, {""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""state"": ""active"", ""id"": ""e890354d-e170-44c1-94b1-f0d8b38a49fc"", ""name"": ""barbeque""}], ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""resources"": [], ""extras"": [{""state"": ""active"", ""value"": ""\""2\"""", ""revision_timestamp"": ""2011-07-07T13:21:02.220368"", ""package_id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""key"": ""1"", ""revision_id"": ""61703821-a2f1-41f8-8f39-71130c5e6c1b"", ""id"": ""87d7d4fc-7eb7-4760-b199-e460ce505632""}]}
}}}
I'm going to ask David Raznick about what's best here. It would be good to make them uniform."
1214 1310053225000000 aron "Replying to [comment:2 dread]:
> Aron,
> I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: ""Should be 405 Method Not Allowed?"" DELETE should be allowed. Or is that a response you're getting?
> David
If I send a HEAD request to the server eg.
{{{
curl -i -XHEAD http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf
}}}
I get the following in the Access-Control-Allow-Methods header
{{{
Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS
}}}
I've just realised that the ""Access-Control"" headers are for cross origin browser requests so I think you can ignore my comments about the correct response being 405 Method Not Allowed but perhaps DELETE should be added to the Access-Control-Allow-Methods list?
"
1214 1310054305000000 aron "Replying to [comment:3 dread]:
> ""Tag returned as a JSON object when updating but as a string when requesting.""
> I'm not sure what you mean.
>
> http://test.ckan.net/api/2/rest/package/reference-example
> gives tags: {{{""tags"": [""country-uk"", ""example-package""],}}}
I'm getting back objects for each tag in the response body from a PUT request. For example ""country-uk"" would be something like.
{{{
{
""id"" : ""600dc72e-6127-4704-b801-bee00474ec0c"",
""name"" : ""country-uk"",
""revision_timestamp"" : ""2011-07-06T09:09:21.578034"",
""state"" : ""active""
}
}}}
Hopefully this gist will illustrate the issue.
> Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix.
> http://test.ckan.net/api/2/rest/tag/country-uk
That was the fifth point on the list.
"
1214 1310054863000000 aron "Replying to [comment:5 dread]:
> Deleting an extra using 'null' works for me:
Hmm, it works for me too when there are other keys remaining in the ""extras"" object. However I can't seem to delete the last one. Take the following example.
{{{
curl -i -XPUT -d'{""extras"": {""Tester"": null}}' -H""X-CKAN-Type: application/json"" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf
}}}
Gives:
{{{
{ ""author"" : """",
""author_email"" : """",
""extras"" : [ { ""id"" : ""88cc50ca-9e29-4499-a542-09d364f5f64f"",
""key"" : ""Tester"",
""package_id"" : ""ec9cb930-d15f-441f-a1e1-36f4d5df19bf"",
""revision_id"" : ""e5c3ca9c-1dae-4a86-837f-b8c19ac31964"",
""revision_timestamp"" : ""2011-07-07T16:01:05.696025"",
""state"" : ""active"",
""value"" : ""\""Test Value\""""
} ],
""groups"" : [ ],
""id"" : ""ec9cb930-d15f-441f-a1e1-36f4d5df19bf"",
""license_id"" : """",
""maintainer"" : ""aron"",
""maintainer_email"" : """",
""name"" : ""my-test-package"",
""notes"" : ""Heading\r\n===\r\nThis _is_ some text"",
""relationships_as_object"" : [ ],
""relationships_as_subject"" : [ ],
""resources"" : [ ],
""revision_id"" : ""688d33fb-5629-4ab3-9f59-a649fc7caa00"",
""revision_timestamp"" : ""2011-07-06T10:37:50.182894"",
""state"" : ""active"",
""tags"" : [ { ""id"" : ""600dc72e-6127-4704-b801-bee00474ec0c"",
""name"" : ""test-tag"",
""revision_timestamp"" : ""2011-07-06T09:09:21.578034"",
""state"" : ""active""
} ],
""title"" : ""My Test Package"",
""url"" : """",
""version"" : """"
}
}}}
"
1214 1310055169000000 aron "Another edge case that popped up while verifying the above issues. A PUT request without a body throws a 500 Internal Sever Error
{{{
curl -i -XPUT -H""X-CKAN-API-KEY: tester"" -H""Content-Type: application/json"" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf
}}}
"
1214 1310057942000000 aron "One more for today. Please do let me know if I should be filing these as separate tickets?
For some reason performing a search query for packages with an underscore ""_ "" as a query string key fails to return any results.
{{{
curl ""http://test.ckan.net/api/2/search/package?q=osm""
}}}
Gives me 4 results.
{{{
curl ""http://test.ckan.net/api/2/search/package?q=osm&_=1310056826904""
}}}
Gives me none.
The underscore is generally used by JavaScript libraries as a way of bypassing the browser cache when making JSONP calls. It's easily worked around but is odd none the less.
"
1214 1310062570000000 dread "Fixes have gone into the branch for ""Tag returned as a JSON object when updating but as a string when requesting. Same with ""extras"" content."" and ""Doesn't return rendered_text property on package update."".
I've added DELETE to the Access-Control-Allow-Methods on the server, so you can try that now.
I've added a failing test for the problem of deleting the last extra. Am discussing with David Raznick how to fix that best.
I think the only other ones left to address are:
* Relationships use ""object"" key rather than id or package_id.
* underscore in search parameters"
1214 1310062621000000 dread "I believe ""A PUT request without a body throws a 500 Internal Sever Error"" was fixed yesterday in the code. (Not deployed to test.ckan.net yet though)"
1214 1310067327000000 dread " Relationships use ""object"" key rather than id or package_id.
I'm not totally clear on this one. The URLs for relationship editing can be package ids or names. When you GET a relationship packages are referred to in IDs for v2 API. When you create or update a relationship it returns package IDs for v2 API. Are you referring to the relationships listed in the response when you PUT a package? (If so that would be fixed by my previous fix.)"
1214 1310067831000000 dread " underscore in search parameters
I think its trying to filter search results by values of an extra field called ""_"", even though this probably doesn't occur. If it is not too ugly, I suggest using one of your other ideas for solving the cache issue. I suggest you bring it up with Rufus or the ckan-dev list to get opinions."
1214 1310072808000000 dread I've put the fixes onto test.ckan.net - repoen the ticket if there are any issues.
1214 1310396399000000 aron "The groups property in the package resource is always empty (or is for all packages I have viewed).
The following group lists one package.
{{{
http://test.ckan.net/api/2/rest/group/0ac963e7-ba29-49bc-83c8-98f8c1991649
}}}
But when viewing the package the groups array is empty.
{{{
http://test.ckan.net/api/2/rest/package/758c26d4-5949-4347-8b4d-023374146d94
}}}
"
1214 1314029628000000 dread This appears to have been fixed by version 1.4.3.
1215 1310335541000000 kindly "fixed cset: 8a317eadbb36
If you delete the last row then it just clears it instead of deleting the row."
1215 1310385485000000 dread This is now deployed on ckan.net (requires Ctrl-R to reload js).
1217 1320663277000000 dread I haven't seen this for a while - could have been due to database corruption which has been fixed.
1218 1310385419000000 dread This has been done in cset:bd0f83a2e287 for release 1.4.2. Deployed to ckan.net.
1219 1310471415000000 dread This is a known issue and of low importance.
1219 1310670324000000 erilem "The issue is due to IE7 (and IE6) not supporting {{{inline-block}}}. [attachment:patch-1219-A0.diff] works around the issue.
I'm reopening this ticket in case you want to consider my patch.
But note that I won't be offended if you close it as wontfix again. I agree this is pretty minor, and that working around old browsers' limitations isn't always a good idea."
1219 1310740534000000 dread Patch applied in cset:8c05d27db224 for 1.4.2 release.
1221 1310475683000000 dread "Exception details:
{{{
Module ckan.controllers.user:194 in request_reset
<< h.flash_error(_('No such user: %s') % id)
try:
mailer.send_reset_link(user)
h.flash_success(_('Please check your inbox for a reset code.'))
redirect('/')
>> mailer.send_reset_link(user)
Module ckan.lib.mailer:69 in send_reset_link
<< def send_reset_link(user):
user.reset_key = make_key()
model.Session.add(user)
model.Session.commit()
>> user.reset_key = make_key()
AttributeError: 'NoneType' object has no attribute 'reset_key'
}}}"
1221 1310556607000000 dread "Other edge cases and error conditions do not work properly either.
Fixed in cset:4737c86d1d81 for release 1.4.2."
1222 1310556618000000 dread Fixed in cset:4737c86d1d81 for release 1.4.2.
1223 1310573893000000 dread pudo did this in cset:1ab655b7b76a and cset:7a9c6b1060cf ready for release 1.4.2.
1225 1310642407000000 dread Done in ckanext repo cset:70fb690a0769
1226 1314029434000000 dread This appears to have gone away. No sign of it for two weeks on any OKF sites.
1228 1311086612000000 dread Done in cset:88e47b68e42d for 1.4.3
1229 1312191530000000 amercader Done, except for Revision related stuff, which probably overlaps with work being done by David Raznick.
1229 1319639472000000 dread I found another direct use of the d.b. in the home controller.
1229 1319645778000000 dread Replace code in home in cset:f19f9c5bec94 on default for release 1.5.1.
1230 1311154142000000 kindly "The standard way to add tables in a plugin has converged upon putting the tables in the iconfigurable plugin.
This runs at the correct point for when the application runs normally. For testing however there are issues due to the tables potentially being dropped, especially for the sqlite case. The fix is to make sure the iconfigurables are run at the start of each set of tests, hence adding it to the ckan_nose_plugin. This is not pretty, but good enough.
cset:8531b9fc1ee2
"
1231 1315948921000000 rgrp @kindly: Does this fit in with current plans? If not suggest we close as invalid/wontfix or put into the backlog (though given lack of content not sure that is very useful ...)
1231 1325474087000000 rgrp Closing as wontfix (or duplicate). We now have stats in footer as of #1576. googleanalytics (e.g. http://thedatahub.org/analytics/dataset/top) would be nice to have but should be part of stats
1231 1325474447000000 rgrp See #1101 for other duplicate ...
1232 1315948536000000 rgrp Removing from ckan-v1.5 as do not think any of this is top priority (correct me if I'm wrong!)
1233 1315948668000000 rgrp Removing from v1.5 as not a priority atm.
1234 1311183328000000 dread Fixed in cset:1ed3fde56a61 for 1.4.3.
1236 1311351476000000 dread Working on it on branch enh-1236-view-package-revision
1236 1311699843000000 dread This is merged to default (for rel 1.2.5) but isn't brilliant yet because of bug #1238.
1236 1311759816000000 dread The bug is in fact in the way I work out what revision I'm displaying - it just looks at the PackageRevision.
1236 1323090508000000 dread This feature is now released, but there is a remaining bug which I've put in this ticket: #1509
1236 1330019788000000 dread Went into CKAN 1.4.3
1238 1311583876000000 kindly "I cannot reproduce this, to me it looks like it does get the correct revision. If you for example look at the package a millisecond before i.e
http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627829
the tags are no longer there that were added in that revision."
1238 1311759784000000 dread Apologies, I have a problem instead with the way I identify the revision in #1236.
1239 1311442163000000 rgrp Fixed in cset:adf254b7c507 (see https://bitbucket.org/okfn/ckan/changeset/adf254b7c507)
1239 1312453156000000 dread Fixed in 1.4.2 release. Broken in UI in 1.4.1 and API in 1.4 & 1.4.1.
1240 1314180488000000 dread "I basically agree, but just thought I'd add a couple of comments into the mix.
> GET at root returns whole package set (a *bad* idea)
Sure it has a cost, but I don't see what the problem with this is if the list is cached. Forcing developers to paging through the list of packages is putting unnecessary obligations. I'd say that the top two RESTful operations are listing objects and getting a particular one, and you'd be taking away one of those.
> Unify on /api/v{version num}/... structure
Where is it not currently unified?"
1240 1316022145000000 rgrp add autocomplete
1240 1324319119000000 rgrp assigning to kindly for review
1240 1324320107000000 dread "Although I'm generally in favour of Action API over RESTful, Action API does need some sorting out wrt listing methods, advertising what params are allowed and checking those that are passed. Using the Action API is so hard without reading the code!
The RESTful API is handy when you want to do something quickly in a browser. I'd not want to lose these two, for example:
* to look at a package /api/rest/dataset/xyz
* API version etc. at /api/util/status (coming with ckan 1.5.2)
It's also really handy in demos and to send as links to people on the list."
1240 1325473312000000 rgrp IMO this is non-urgent and can move out of v1.6 as we have enough more important stuff to do.
1242 1323173292000000 thejimmyg This text is no longer present on the login page.
1243 1328000405000000 rgrp invalid as things have changed and this has been fixed or obsoleted.
1244 1323089995000000 dread No progress yet
1245 1311864530000000 annapowellsmith "Would be nice to list some extensions somewhere on this site (features page perhaps).
Along the lines of: http://docs.ckan.org/extensions.html#finding-extensions
It wasn't until I wrote this list that I realised just how many extensions there were, and what it said about the CKAN community that there were so many...
So I think it would be a good sales point. "
1245 1317422640000000 rgrp Closing as now mostly done and not really a ticket item.
1246 1311863583000000 dread "We have an unresolved issue with the display of the 'open data' icon that showed up when we upgraded ckan.net to the latest beta last week, but I didn't realise the search page http://ckan.net/package was showing all packages as open (the revision page and package pages seem fine).
Last week when I upgraded from 1.4.1 to 1.4.2beta, there was an exception thrown when loading the home page. The exception was in doing the isopen call. It was solved by making some code changes directly on the ckan.net server to make it an attribute query instead of function call. It was strange because this behaviour didn't occur on my machine and all the tests passed. Here are the manual patches on ckan.net (the application.js change is unrelated, by someone else):
{{{
diff -r adf254b7c507 ckan/public/scripts/application.js
--- a/ckan/public/scripts/application.js Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/public/scripts/application.js Thu Jul 28 14:10:23 2011 +0000
@@ -2,6 +2,7 @@
$(document).ready(function () {
CKAN.Utils.setupUserAutocomplete($('input.autocomplete-user'));
CKAN.Utils.setupAuthzGroupAutocomplete($('input.autocomplete-authzgroup'));
+ // $('.resource-url').parent(
});
}(jQuery));
diff -r adf254b7c507 ckan/templates/_util.html
--- a/ckan/templates/_util.html Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/templates/_util.html Thu Jul 28 14:10:23 2011 +0000
@@ -43,7 +43,7 @@
standard package listing -->
-
+ class=""${'fullyopen' if (package.isopen and package.resources) else None}"">
${h.link_to(package.title or package.name, h.url_for(controller='package', action='read', id=package.name))}
@@ -60,7 +60,7 @@
-
+
-
![]()
@@ -72,7 +72,7 @@
-
+
-
${h.icon('lock')} Not Openly Licensed
diff -r adf254b7c507 ckan/templates/home/index.html
--- a/ckan/templates/home/index.html Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/templates/home/index.html Thu Jul 28 14:10:23 2011 +0000
@@ -64,6 +64,7 @@
Recently changed packages
+
${package_list_from_dict(c.latest_packages)}
- ${h.nav_link(c, _('Add a package'), controller='package', action='new', id=None)}
+ - Upload Data
- ${h.nav_link(c, _('Tags'), controller='tag', action='index', id=None)}
- ${h.nav_link(c, _('Groups'), controller='group', action='index', id=None, highlight_actions = 'new index')}
- ${h.nav_link(c, _('About'), controller='home', action='about', id=None)}
diff -r adf254b7c507 ckan/templates/package/read.html
--- a/ckan/templates/package/read.html Sat Jul 23 18:16:15 2011 +0100
+++ b/ckan/templates/package/read.html Thu Jul 28 14:10:23 2011 +0000
@@ -56,7 +56,7 @@
-
-
+
This Package is Open
@@ -80,7 +80,7 @@
-
+
This package is Not Open
Either because it is not openly licensed or is missing
}}}"
1246 1311885360000000 pudo "I'm afraid to say I don't understand any of this: the faulty display of openness is triggered as removing the brackets turns the isopen() check into a check for the existence of the method. Reverting the changes on the server seems to work without generating exceptions. What exactly is the issue that was supposed to be fixed originally?
"
1247 1311955176000000 rgrp Done. See http://ckan.readthedocs.org/ and http://docs.ckan.org/ will propagate soon.
1248 1319797763000000 dread "Fixed in cset:188d27a56c89c35f97f3a79fdd0a6460223f86cf on branch release-v1.4.3.
This was only seen in CKAN 1.4.3beta."
1249 1311959356000000 pudo fixed in cset:0398243111d4
1250 1312370201000000 pudo "The issue is caused after search returns from solr and tries to get objects from the database. These are by default ordered by name and we need to restore the original order after fetching them from the database.
Fixed in module."
1251 1314406332000000 rgrp Really non-urgent and can be done once #1252 is done (which is dependent on v1.5)
1251 1317315159000000 dread rgrp did this last weekend.
1252 1314406271000000 rgrp Move to v1.5 since this will coincide with release.
1252 1316876747000000 rgrp Announced. thedatahub.org now running for 2+ months. Today did wiki.ckan.net redirect and hope to do thedatahub.org redirect.
1252 1317315135000000 dread This is done now?
1252 1318160334000000 rgrp This was done ~2 weeks ago.
1256 1312409296000000 kindly This has also caused problems in authorization for deleted packages.
1256 1312813529000000 kindly fixed cset:357cf9377b25
1256 1319812556000000 dread It sounds like this became an issue with CKAN 1.4.3 and the fix goes into CKAN 1.5.
1258 1312813614000000 kindly cset:e22d4e385fc8
1258 1319812452000000 dread I believe this was broken with the introduction of moderated edits in CKAN 1.4.3 and this fix went into 1.5.
1262 1314029150000000 dread "Pudo - I'm not sure what's meant here. Are you talking about moderation for new users? Or some other sort of enforcement? Is it spam you want to target?
The links you're concerned about - are they the spam ones in the User's 'about' field? If so, I think we *should* allow users to have links, although continue with applying them with 'nofollow' and attack spammers back in a concerted strategy across all editable fields - see ticket #1257."
1263 1314029279000000 dread This was fixed in cset:8d293393a9f2 for release 1.4.3.
1264 1314406294000000 rgrp Nothing urgent about this so moving to backlog.
1264 1318244619000000 zephod Created #1377 to handle bugs revealed while doing this work.
1264 1318245716000000 zephod Complete; cset:b216952644aa
1265 1312566052000000 dread Fixed in cset:a1b8dd169f4e on 1.4.3 release and merged to default.
1266 1312797477000000 dread Fixed in 987dc4fe078e on 1.4.3 branch and default.
1267 1312889130000000 dread John thinks it is a problem with WebOb 1.0.6
1267 1312890962000000 dread This error has been occurring on at.ckan.net which has webob 0.9.7.1-1
1267 1312894071000000 dread Running tests with WebOb 0.9.7.1, we see the error! Am now trying to fix.
1267 1312897724000000 dread "Fixed in cset:db5bf5b45b4f for ckan 1.4.3.
Later versions of webob add the charset header for unicode errors (webob/exc.py:generate_response):
{{{
extra_kw = {}
if isinstance(body, unicode):
extra_kw.update(charset='utf-8')
resp = Response(body,
status=self.status,
headerlist=headerlist,
content_type=content_type,
**extra_kw
)
}}}
This was fixed in webob cset:25fc14d49623 which went into WebOb 1.0.
This problem affects versions from CKAN 1.4.2 back to 1.3.1. We had been avoiding this in CKAN previously by only doing abort() with a plain string. "
1268 1317212499000000 kindly fixed cset:eebbe6071741
1270 1312912188000000 dread Fixed in cset:ca5295c9c8cf and released as ckanclient version 0.9
1271 1313753663000000 rgrp Closed in cset:749cb3a087e3
1272 1313745867000000 rgrp "Assume these are the 3 spatial fields listed on http://wiki.ckan.net/Geospatial_Capabilities. I'm a big +1 here. Starting point would be a bit more documentation on that wiki page. Questions:
* Is this work going in ckanext-spatial?
* What is being proposed here, I assume, is to automatically index those fields in our geospatial search.
* Should we develop a simple display widget using google maps or similar (this is orthogonal to the WMS preview i think) which *just* uses the core fields. This is probably a separate ticket and possibly separate from ckanext-spatial extension."
1275 1314110016000000 johnglover "Completed in branch feature-1275-solr-search. Up to date with current default but not yet merged into default.
John"
1275 1319812967000000 dread Going into release 1.5.
1276 1314886010000000 johnglover "Added to branch feature-1275-solr-search
Don't seem to be able to apply weighting to fields at index time using solrpy (see open issue: http://code.google.com/p/solrpy/issues/detail?id=6). So, switched to using the newer Solr edismax query parser and have applied weights to fields at search time.
Current weights are as follows:
name: 4
title: 4
group: 2
tag: 2
text (everything else): 1"
1276 1315948394000000 rgrp Closing since done (though not yet in main!)
1277 1314868204000000 johnglover Done in branch feature-1275-solr-search for all package searches (which use Solr). Resource and Tag searches still use the old ckan query parser.
1277 1315297186000000 rgrp This just requires merging into main but has been deferred as once solr branch is merged solr is *required* to run CKAN at all (we are thinking of having basic support for sqlite/dbs out of the box ...)
1277 1315948417000000 rgrp Closing since done (though still need to merge into default)
1279 1314303732000000 rgrp Wow, that's a pretty big omission. authz groups are important and are used. I think we should discuss this in next planning meeting.
1279 1314304326000000 rgrp In addition it appears that authz group was left out of the form refactoring so new and edit page use old formalchemy stuff. Should fix this.
1279 1338212028000000 ross Duplicate of #2366
1280 1328786670000000 dread David tells me that this was fixed in CKAN when we moved to SQLAlchemy 0.7 #1433 which went into CKAN 1.5.1.
1281 1314021919000000 dread "Done by pudo in cset:eaf342823caf on default branch, so headed for release 1.4.4.
""Work around the fact that locale changes do not affect
the running request and therefore led to an incorrect flashing
message. Now manually changing language."""
1282 1314404540000000 rgrp "Done in cset:4b6838a820bb.
Have not done either of the optional items. JS testing for what we have is relatively hard and of dubious value.
Re CDN stuff requires work and of doubtful of immediate value since can just configure cache or main site to serve static stuff directly. (Likely more value from compressing js and css ...)."
1283 1326720891000000 dread Since we switched completely to SOLR, this has been true. SOLR doesn't index deleted packages so you can't search them, but they can be read/edited by admins.
1283 1330083933000000 dread This went into CKAN 1.6
1284 1315212258000000 dread "Fixed by rgrp in cset:1952445d2802 on release-v1.4.3 and default. Comment:
do not create revision for changes to users as they are not revisioned.
* This had already been removed in cset:8d6fde0e2196 (then in user controller) but Adria seems to have reinstated this when refactoring things into the logic layer. This 'bug' explains the huge number of empty revisions we have been seeing on ckan.net."
1284 1315212603000000 dread I've added tests for this (to prevent yet another regression!) in cset:1b77ce8ce560. Closing.
1284 1315214358000000 rgrp I'm not sure this is the sort of thing we need tests for (we can have too many tests!). Something worse discussing ...
1284 1315215626000000 dread Really? How many times do you want to fix this bug again? For the sake of three lines of test...
1285 1314269414000000 dread Another instance is when the database tables are not created and you want to both display a sensible error and have an email alert. #1290
1285 1323173125000000 thejimmyg "I could write some middleware to handle this. Can we come up with a list of exceptions not to catch or should I create a base exception called ""NoEmailReportTriggeredException()"" which exceptions for this purpose have to be derived from or which code has to raise? "
1285 1323178678000000 dread Isn't the point is we don't to raise an exception at all - we want to call a method to send the email and then return from the controller a normal html page. Don't we just need a method to get the error email address from the config and send an email.
1286 1342436420000000 dread This can happen now, if it hasn't already.
1289 1317315211000000 dread Discussions have not resolved this either way. Decided to leave it like it is for now.
1290 1314270894000000 dread "Improved on option 2. For requests to Home controller, the first thing it does is check the user. I catch SQL errors that are to do with missing tables and that now produces a template-less error saying the site is off-line and then mentions database initialisation.
Done in cset:484e50c4f989 on default for ckan 1.4.4."
1291 1315948475000000 rgrp @kindly: is this not done? (I know there is more work to be done here but though table creation was done ...). If not need to move to next milestone so we can close this one.
1292 1315999202000000 dread i'm def +1 for removing API stuff entirely from i18n for the present
1292 1320145677000000 dread " Review process of releases and updating strings (pot) and translations (po) files
c.f. http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html
http://wiki.ckan.net/Release_process updated with basic process
Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?)
Welcome message is translated in all languages. Most language have upped their percentage translated significantly and Swedish and Polish added as well.
Review long list of strings for improvement
Consolidate any?
Get rid of i18n for strings in api
About 50% of the strings are templates, 30% are in the logic layer and 20% are in the controllers. Removed a handful from ckan/lib/base.py that were only displayed in the api, but the logic layer ones are generally errors that can be displayed in the Web UI as well as the API, so not ideal to remove.
Try and remove javascript
Tom did this.
Add in extensions
This could be done. Have raised ticket #1292 for this, next release.
#1374 Fix switching to en from other default language
Done
#1417 Fix browser language detection
Done"
1292 1320145732000000 dread These fixes all went into release 1.5.
1293 1314701675000000 johnglover "== Changes ==
* All templates
* Routing: /package/ -> /dataset/ (including API)
* Tests: h.url_for(controller='dataset') rather than package ...
* One test to check redirect works?
* Docs
== Leave ==
* Stay with Package in code e.g. stay with 'PackageController'
* also leave stuff like pkg = .... (gradual migration)
* Leave extensions
== Notes ==
* Package -> Dataset
* Data Package -> Dataset
* respect capitalization"
1293 1314905799000000 johnglover "Branch: feature-1293-rename-package-to-dataset
Routing and test editing should be finished (all tests passings). Have not added test for redirect yet."
1293 1315214527000000 rgrp Reassign for completion (templates and docs) as john is away.
1293 1315246628000000 zephod "cset:https://bitbucket.org/okfn/ckan/changeset/4a98589af6c1
Fixed"
1294 1315948112000000 rgrp Removing #1311 as that is not directly related to dataset edit / create.
1294 1315948257000000 rgrp Done with merge into default in cset:8bb0720a2150 and deployment to http://thedatahub.org/
1295 1315820564000000 rgrp This is mostly done but some final tweaks and name disabling to be done. See work in feature-1294-ux-improvements-dataset
1295 1315948009000000 rgrp Completed with merge into default in cset:8bb0720a2150
1296 1315403275000000 zephod No longer doing inline resource editing on dataset/view; instead use the edit form.
1296 1315821293000000 rgrp Have now done resource add pretty effectively and most of edit done (should complete today).
1296 1315948027000000 rgrp Completed with merge into default in cset:8bb0720a2150
1297 1315403878000000 zephod No longer doing inline editing on the view page.
1297 1315403974000000 zephod Largely done in cset:8901e722b158
1297 1315404322000000 zephod all done: cset:71096cd7b27d
1298 1321199769000000 seanh I've been working on this in this branch: https://github.com/seanh/ckan/commits/feature-1298-activities-table
1298 1323106307000000 seanh See the new super ticket and its branch: #1515
1298 1323710593000000 dread Sean says: This is done on the activity stream branch. Just would like David Raznick to review it.
1298 1324384191000000 seanh "David,
Ignore my link above to a branch on my ckan fork. Now that I have permissions I'm pushing my branches to the okfn ckan repo on github. My super branch for the activity streams feature is '''feature-1515-activity-streams''' on the okfn ckan repo on github. This page comparing my branch to master is particularly useful:
https://github.com/okfn/ckan/compare/master...feature-1515-activity-streams
(click on the Files Changed tab)
For reviewing this ticket, the relevant changes to review are:
* ckan/lib/activity.py, all of it
* ckan/model/activity.py, all of it
* My changes to ckan/model/meta.py
* My changes to ckan/model/package.py
* My changes to ckan/model/resource.py
* ckan/tests/models/test_activity.py, all of it
The other changes on my branch are for other activity streams tickets that follow this one. The super ticket #1515 has an overview of it all.
"
1299 1325866115000000 rgrp No estimate as yet? (Or is it in super ticket? (Link?)). Also isn't this a duplicate of #1510 (or vice-versa)?
1300 1315906283000000 rgrp page_heading and optional_footer done in #1108
1300 1315908961000000 rgrp Move of js to optional_footer done in cset:dcb21f0d976f. As part of #1294 have done a lot of css adding. Not sure how much more to do here.
1300 1319794520000000 rgrp Side bar menu is switchable (though not sure how -- zephod has done this!) CSS tidying and labelling of divs done in #1380. So closing!
1303 1314978842000000 dread Adria fixed this in cset:7637d8694388 on default. I transplanted it to release-1.4.3 and have updated thedatahub.org.
1305 1314971164000000 nils.toedtmann "Always use ""localhost"" as mail relay. On all servers, the local MTA should be configured correctly (e.g. in this case to relay via mail.fry-it.com), also to be able to send rootmail.
I fixed it on IATI (s049.okserver.org = ex eu23) and thedatahub.org (s055.okserver.org). Please verify that it's working now."
1305 1315317033000000 nils.toedtmann [15:22:02] Adrià Mercader: Yes it looks like it works on both thedatahub.org and iatiregistry.org
1306 1319797204000000 dread These are expressed as functional API tests - see tests/functional/api/test_action.py. There's not a lot of difference (apart from Paste coming in there) so no real need to duplicate API and logic layer tests, or to switch them to talk directly to the logic layer.
1307 1315246706000000 zephod cset:2c31fe24e1c9
1307 1315247298000000 zephod "Replying to [comment:2 zephod]:
Oops:
cset:cbd6677ec41e
"
1309 1315586335000000 dread Duplicate of #1318
1310 1315820419000000 dread This was fixed cset:e49781cb74fd for 1.4.3.
1312 1315327743000000 dread "The default Content-Type is ""application/x-www-form-urlencoded"" (that is what curl sends, and what CKAN defaults to). Therefore these characters need url encoding (""=&;"" -> ""%3D%26%3B""). Here is an example of it working:
{{{
(pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test7"", ""title"": ""Test2 %3D%26%3B""}' -H ""Authorization:tester""
HTTP/1.0 201 Created
Server: PasteWSGIServer/0.5 Python/2.6.5
Date: Tue, 06 Sep 2011 16:36:32 GMT
Pragma: no-cache
Cache-Control: no-cache
Location: http://localhost:5000/api/rest/package/5866a0f5-905a-44ff-91c3-c1489374a73e
Content-Type: application/json;charset=utf-8
Content-Length: 571
{""maintainer"": null, ""maintainer_email"": null, ""id"": ""5866a0f5-905a-44ff-91c3-c1489374a73e"", ""metadata_created"": ""2011-09-06T14:06:40.735488"", ""relationships"": [], ""metadata_modified"": ""2011-09-06T16:36:32.543461"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [], ""groups"": [], ""name"": ""test7"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""notes"": null, ""title"": ""Test2 =&;"", ""ratings_average"": null, ""extras"": {}, ""ratings_count"": 0, ""revision_id"": ""92ee0b2e-c5e4-47b9-b00c-b8e29662b950""}
}}}
Alternatively you can set the content type to ""application/json"" and you don't need to URL encode. e.g.:
{{{
curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test5"", ""title"": ""Test &""}' -H ""Authorization:tester"" -H ""Content-Type: application/json;charset=utf-8""
}}}
To help work out these problems with encoding in the future, I've improved the error message to show the decoded JSON string. cset:55273629a552 which is headed for CKAN 1.5 release."
1313 1317076432000000 rgrp "Error was due to maintenance. Need both config options for recaptcha to work so either one can be used to check config. Re docs see #1358 (auto generation of docs on config from the config source).
Given that only docs remain and that is part of #1358 am closing as duplicate."
1315 1315414218000000 dread Pretty good start added in cset:7a0e7a486df3 on default, aiming for release 1.5. Just needs some loose ends tidying up.
1315 1320173768000000 dread Tidied up in cset:39daf1afd64a for release 1.5.
1316 1338193724000000 kindly now get_or_bust function handles this.
1317 1315416649000000 dread "Ok, I think this example may be when searching for ""ira"". ""irina"" works me, but ""ira"" gives:
{{{""ira"" matched several users}}}"
1317 1315822670000000 rgrp This is very minor IMO. I would defer this as part of a more general user system ux refactor.
1317 1325473762000000 rgrp Moving out v1.6 as minor.
1318 1315569362000000 dread "Problem tracked down to Adria's cset:4f1ca1eb2630 on branch feature-1229-db-out-of-controllers that missed out repoze.who in the redirects after login, when refactoring the user registration. This code went into default on 2011-07-28 for release 1.4.3.
Fixed in cset:93731cd3b22e for release 1.4.3 and doesn't affect previous releases. Tests for user registration added too."
1319 1315476057000000 shevski Ideally we should also verify people's email addresses. Do we think that's overkill?
1319 1315904592000000 dread "No responses received about this on ckan-discuss, so I'm taking that as no objections to requiring email address on registration.
I think that verifying email address is not worth it in this case, because the value of supplying an email address is only for resetting a password. The chance of needing a password reset AND having inputted an incorrect email address is very low, so I don't think it justifies everyone having to do this extra step. If we used email address more then that may change."
1319 1318426807000000 rgrp @dread: hope it is OK to re-assign to me but this is very similar to #1386 (and is a su bticket of #1343)
1319 1318527613000000 rgrp Fixed in https://bitbucket.org/okfn/ckan/changeset/130cad62163c
1319 1324036947000000 dread Goes into release 1.5.1
1320 1317391018000000 zephod Extend misc.py to regex find http and https, surrounding it with <>. ALSO note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package).
1320 1317730999000000 zephod "Slightly painful when links appear inside naughty HTML tags or in markdown tags. Restriction: Links are auto-linked if they appear at the start of a line or after whitespace.
cset:f9dfb0506594"
1320 1318164603000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/f9dfb0506594
1321 1319558004000000 dread "We should not cache if there are flash messages to show (otherwise they get queued up for the next non-cached page view).
Fixed in cset:945e2a6ce74f for release 1.5"
1323 1315594333000000 dread "thedatahub has a couple of auth groups with no name. The form allows it!
Fixed in cset:bebd0a46ebc6
* form requires the name field
* existing auth groups without name can be viewed/edited if necessary (referring by ID)
This has probably been a problem since auth groups were added to CKAN."
1324 1315948336000000 rgrp Done and merged into main at cset:8bb0720a2150 (and deployed!)
1325 1315587113000000 dread Fixed in cset:0dd4d4457c97 for release 1.4.3. Previous Python 2.5 incompatible versions include 1.4.2 - 1.3.4.1 and maybe further.
1327 1320662446000000 rgrp Duplicate of #1397
1328 1340191065000000 ross Let's see if this is still an issue in 1.8
1329 1316624026000000 zephod "I've tried to do this wherever I could as I looked through ckan.pot for broken pieces of string. Generally it triggers errors unless things are very neatly structured (no nested links or ${a.variable} accesses, it seems) but things are quite a lot tidier.
cset:d5bf4c7ba293
Does that count as 'fixed', or is this a permenant issue?"
1329 1317118323000000 dread I'll take a look to see if I can do any more and then will close it.
1329 1317404566000000 dread "POT looks far better. I couldn't see any outstanding problems.
This has gone into ckan 1.5."
1330 1327060201000000 dread "This is simply incorrect. test_authz.py (I assume you mean the one in ckan/tests/functional?) tests that authz works all across the Web UI and API in enough detail.
The 'total site lockdown' test - perhaps you mean the TestLockedDownViaRoles class which makes up a small part of test_authz? It was a test added due to an untested use case implemented in IATI, DataGM etc. These are currently active so still need testing.
Regarding refactoring the tests to be ""proper"" then please open a new ticket, providing references to why it is not ""proper"" to test by searching for text in a web response."
1331 1325355631000000 dread This appears to be fixed in 1.5, 1.5.1 and 1.5.2a.
1332 1315999104000000 dread "Fixed in cset:860b357a2aec for release 1.4.3 and updated no.ckan.net.
This problem was due to {{{[1:authorization group]}}} being translated as {{{[autoriseringsgruppe]}}} and I corrected it to {{{[1:autoriseringsgruppe]}}}. This word has a bold tag, hence the brackets and the number is used to identify which bracketed word is which (when you have more than one).
It is a shame that Transifex doesn't draw attention to this problem and Babel doesn't spot the error on compilation."
1334 1316086680000000 dread cset:b01de680bf86 copes with the problem. Now to find out why revision.packages has some None values - very odd!
1334 1319798961000000 dread I think this has been a problem since 1.3.3. Fixed in v1.5.
1335 1315910117000000 dread "Done to a basic extent in branch feature-1211-drupal for release 1.4.3.
Further tickets:
* #1322: Action API improvements
* #1315: Document /api/action"
1338 1316951545000000 amercader Reopening, because another change is needed to support custom schemas
1338 1317117738000000 amercader Fixed in eca1edce3a0f
1339 1316010607000000 kindly "I have fixed the isodata and made a slightly modified int_converter for this case. In the correct place and raises invalids on not being an int. cset:a4af115116bb
The thinking was that the input of these fields would be through the api so the empty string case did not arise. These should clearly be converted to None (Null).
What issues in general?
Having done this lots of times before you always end up needing to write your own little validators as the standard ones never do what you want. Thats the point of them. Look in ckan/lib/validators if you need examples.
So what you did was correct..."
1340 1315991534000000 dread Fixed in default in cset:33856d3ce84e aimed at release 1.5
1341 1315996368000000 kindly "This was run to delete the users and their mistaken revisions that where created.
{{{
BEGIN;
delete from revision r where r.id in (
select r.id from ""user"" join revision r on r.author = ""user"".name
left join resource_revision rr on rr.revision_id = r.id
left join package_revision pr on pr.revision_id = r.id
left join group_revision gr on gr.revision_id = r.id
where ""user"".created between '2011-08-15' and '2011-09-06'
and gr.id is null and rr.id is null and pr.id is null
and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%')
and ""user"".name not like 'http%'
);
delete from ""user"" u where u.id in (
select ""user"".id from ""user""
left join revision r on r.author = ""user"".name
where r.id is null and ""user"".created between '2011-08-15' and '2011-09-06'
and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%')
and ""user"".name not like 'http%'
);
COMMIT;
}}}
"
1341 1319649573000000 dread The majority of our 6000 users are still spammers. Can we simply elimenate those who've not made any revisions? Even if we take a few genuine users with it, these are pretty dead accounts anyway.
1341 1319709997000000 dread Reopening - this is still a serious problem. We can't link to the user page because it is so embarrassing: http://thedatahub.org/user
1343 1316078440000000 dread I don't think email confirmation is necessary - see discussion at #1319
1343 1318528138000000 rgrp Remove from ckan-v1.5 since remaining non-complete items (user page improvements and modal login won't get done in v1.5.
1344 1316078757000000 kindly fixed cset:f08215845dab
1345 1337961353000000 nils.toedtmann This becomes an actual problem on the CKAN farm s057, see http://trac.okfn.org/ticket/1245
1345 1338551926000000 nils.toedtmann See #2485 for workarounds containing the leak(s).
1345 1340190737000000 ross Moved out of milestone so that it gets looked at again soon
1345 1343215489000000 amercader See https://github.com/okfn/ckan/pull/73
1346 1316086314000000 dread "Fixed in cset:df299e69d456 on default for release 1.5.
Problem exists on releases 1.4.3 - 1.3.1 and probably earlier too."
1346 1316094632000000 dread Also Transposed to 1.4.3.1.
1347 1316094608000000 dread Fixed on cset:7c3cc0fd6ae4 on branch release-v1.4.3.1.
1349 1316277490000000 zephod Done. cset:e074aee19923
1350 1316624288000000 zephod "Was going to reimplement an old workaround described by Friedrich, where a language.js file is served through the home controller (with appropriate settings in routing.py) and so ${_('string literals')} will be translated. See:
* https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/controllers/home.py#cl-48
* https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/templates/home/language.js
However the string-extraction mechanism seems geared towards Genshi templates and hacks are required to make it read a javascript file neatly (ie. without putting the *entire file* into a string for translation). A more elegant solution seems to be using an xi:include to pull in a genshi template which contains an html
}}}
where the jquery.html5.preie9fix.js is simply the [gist](https://gist.github.com/887560) published by Akkuma in a [jquery-tmpl issue (#36)](https://github.com/jquery/jquery-tmpl/issues/36#issuecomment-918495).
This script patches JQuery on the fly on IE < 9 to rework in memory html 5 blocks handled by JQuery, so that it doesn't break.
And this fixes the ""Upload a File"" button on IE.
I didn't commit/pull this specific fix for the following reasons:
* the script embeds innerShiv, which is a previous incarnation of html5shiv (now included in CKAN). It doesn't sound fine to include 2 pieces of code with similar effect, but I couldn't reproduce the fix with html5shiv!
* in my environment I have included this fix sitewide even if I didn't spot other problematic features..I don't know if it would be better to include it only on the dataset edit page in the official CKAN code.
Summary: I leave to you guys the choice on how to deal with this problem."
1788 1330624155000000 zydio "Erh...sorry for some formatting errors in my comment.
I used markdown formatting for links, and used the #NUMBER format to cite issues, which obviously is interpreted by Trac as references to his own issues...my bad."
1788 1330644056000000 rgrp Re the Upload file button not sure what's best. We do have plans to replace jquery-tmpl with mustache in the near future (but not clear exactly *how* near). We'll think about this and get back to you. Thanks for the detailed debug report.
1788 1333133365000000 zephod "I've been performing a full sweep of the site to rebuild the markup & stylesheets using Bootstrap, and in doing so have managed to close off this ticket. Bootstrap allows us to delegate lots of clever functionality and layouts to Twitter's tried-and-tested code, making it easier to be compatible with IE7.
It has brought up plenty of issues: Lots of Javascript was broken; plenty of pages generate broken doctypes causing the browser to go into quirks mode (which in turn causes the page to explode all over the screen); most dynamic content would fail to generate properly in one way or another.... (see the #1788 branch for a log of issues encountered).
I've been refactoring our JS and have managed to strip ckanjs, too, because the only significant element we'd used was the File Upload view (which also now works in IE7).
test.ckan.net has been redeployed on master; I'm browsing the site in IE7 and everything is looking great :-)"
1791 1329138315000000 dread Fixed in [release-v1.6 beaeaed]
1792 1329483249000000 toby package create update use correct schema in get_action
1792 1332755148000000 toby closing as done all that I was asked to do as part of this ticket
1792 1332872069000000 dread "What is the changeset for this? There is evidence of a branch ""enhancement-1792-api-logic"" but perhaps this name got fast-forwarded out through a bad merge? Is this merged into 1.6.1 release?"
1792 1332928847000000 toby This is in the 1.6.1 as far as I'm aware. The branch was deleted after the merge into master
1792 1333037710000000 dread "I have tracked down the changesets to:
* branched off master at dff9f7a8
* merged master in for the last time (and became the 'new master'?) at 1a9227d9ff73c
"
1796 1339586499000000 ross LXML removed, update requirements on the extensions
1797 1330550304000000 rgrp New API endpoint, Authorization and documentation done and merged to master. Time so far. 1.5d. https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24
1797 1330863639000000 rgrp Data Viewer support for new DataStore in https://github.com/okfn/ckan/commit/9ab8b0283bb086eb4cd663ff73c27066bdd3c79a
1797 1331412469000000 rgrp "All done!!
Help for data api done in https://github.com/okfn/ckan/commit/1d8e464f8542d4c33286bb93f4de50060665799f
Checkbox for datastore enabled in https://github.com/okfn/ckan/commit/3f1320cd92ae0e775fde1b5eada156260c55e0a6"
1797 1331412644000000 rgrp "Merge fix https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547
"
1798 1329395560000000 dread "Was fixed by Ian in https://github.com/okfn/ckan/commit/5a9054459e3833443bed3e118bbbb6c442e55b0b on branch feature-1453-flexible-tag-names.
This has gone into CKAN 1.6."
1799 1330000389000000 dread Fixed in [master ea2d824] for both logging in and registering whilst logged in. Cherry picked to release-v1.6
1801 1343144718000000 amercader Closing as both old and new theme now show password reset links
1802 1329409718000000 dread Current progress - I have got most of the dependencies installed - many by pip, but some I had to revert to Windows msi installers. Pad with details here: http://ckan.okfnpad.org/windows. So far done: 1 day.
1802 1329490081000000 dread "I've achieved CKAN running with paster and without SOLR (using simple_search).
Running locally on an old XP machine, performance seems fine - no noticeable issues.
I installed most things into a pyenv using pip. pyutilib, psycopg2 and lxml were slightly problematic, but I have workarounds.
I had problems pip and svn and the Cygwin window, so reverted to DOS prompt. I still had the Cygwin tools installed though, so next step is to try it without these at all - they are a big overhead."
1802 1329734716000000 dread Spent so far 1.75d
1802 1329830097000000 dread "* Refined install without using Cygwin - this seems a little easier.
* Added example Apache & modwsgi deployment.
* Documented extensively at: http://wiki.ckan.org/CKAN_install_on_Windows
* Have not tried SOLR on Windows - I'm not convinced this is high priority, and normal install should be fine as it is so separate from CKAN.
Spent another 0.5d. Total spent 2.25d"
1804 1329491156000000 toby This should be fixed in branch feature-1653-i18n_url_rewriting as we use different logic. However it may affect other return_to calls used elsewhere like adding a new package.
1804 1330000705000000 dread I'll fix this on release-v1.6 and then reassign to Toby to fix on feature-1653-i18n_url_rewriting too if the same issue crops up there.
1804 1330037202000000 dread "Fixed this on release-v1.6 in cset 557e72330db7. Spent 0.5h.
Reassigning for Toby to look at branch feature-1653-i18n_url_rewriting "
1804 1330347185000000 toby This has been fixed in master for a while and was only a 1.6 issue
1805 1329760795000000 toby this is fixed in branch feature-1653-i18n_url_rewriting
1806 1330942356000000 toby version one completed
1806 1331295310000000 toby closing as initial work has been done if we need to amend then create a new ticket
1807 1330809641000000 seanh "Implemented the recently-changed-datasets activity stream on branch feature-1807-recently-changed-datasets in CKAN and merged it into master (but the default templates don't make use of it).
Added the recently-changed-datasets activity stream to the front page template on branch master in ckanext-ecportal."
1809 1330528828000000 johnglover Caught exceptions in QA instead, error message stored as openness_score_reason
1810 1329918479000000 johnglover Deferred for now, moving to backlog.
1810 1335874745000000 johnglover Not using eurovoc tags.
1811 1329918517000000 johnglover Deferred for now, moving to backlog
1811 1335874765000000 johnglover Not using eurovoc tags.
1812 1335874864000000 johnglover Ross has fixed this via group authentication.
1816 1330863079000000 rgrp "As per detailed comments in http://wiki.ckan.org/Proposals#Apps_in_CKAN I think Apps is a confusing name. What we are talking about here is the ""Related Stuff"" item mentioned in that ticket.
Rather than update ckanext-apps I suggest we do a rewrite. I've created a detailed super ticket at #2204 and assigned to you Adria for review and thoughts. (While I understand #2204 may involve more work which we maybe cannot do before relaunch I'd still recommend starting on that rather than spending time upgrading this extension -- if we have to disable it for relaunch I don't think that is a big deal ...)"
1816 1331302835000000 amercader "+1 to a new, more flexible extension.
But in the meantime I've just spent a couple of hours making it work with latest CKAN, which was easier than expected, so we can deploy it with the new version of PDEU."
1819 1332163324000000 kindly Currently using package_show_rest. Should be moved to just use package_show but that is another ticket.
1828 1330525412000000 rgrp Fixed in https://github.com/okfn/ckan/commit/e6da48d9337575a83bc6a30896e68d1081b43847
1829 1330001828000000 dread "This was due to https://github.com/okfn/ckan/commit/f07bfdb9#diff-1 - string ""Language has been set to: English"" shouldn't have been translated. I've put in a patch to make the code intentions clearer and added a test: [release-v1.6 07d3a02]. This only affected the 1.6 beta, and is fixed for the release."
1830 1330089912000000 dread "Now if you set openid_enabled to false, you are not given the option in user settings to associate your account with openid. Cset: [master b295bde]
I looked at removing the openid middleware, but we use it for logging out users, whether they logged in or not - it looks like friendly form plugin is not as good as the openid one in this respect! We could hack the logout function into the friendly form to sort this out, but not sure the effort is worth it at the moment."
1831 1330956631000000 ross Already have a branch for this.
1831 1331549284000000 ross Email is not currently 'unique' in the user model. Find out if this is intentional or not.
1831 1331655458000000 ross Decided to delay this until later but code is in feature-1831-login-by-email
2198 1339771453000000 kindly Already in the docs.
2201 1330765400000000 rgrp Duplicate of #1171.
2202 1338204729000000 toby these are available through the pkg_dict
2202 1338204801000000 toby "re-reading this it's slightly different
re-opening"
2203 1348568961000000 dominik There is work on this in the dataviewer-iframe branch.
2204 1332341044000000 toby needs discussion - planned for 23-3-2012
2204 1332341080000000 toby http://ckan.okfnpad.org/feature-2204-related-stuff
2204 1332843232000000 toby still awaiting discussion
2204 1334077936000000 rgrp "Important comment: not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...
Also (@icmurray): does this interact with our desire to do embedding?"
2204 1334218735000000 ross "I'd be happier adding this to another ticket rather than moving the target for this ticket half way through so have created #2284
Should only be a short ticket in the next sprint."
2204 1335287636000000 aron.carroll "Making some notes of front-end changes that I think need to be implemented sooooon.
{{{
Implement better truncation (the expanded text should pop out on hover rather than expand the entire row)
Show the type of related item (an icon would be good here)
Test in IE
Make the form work without JavaScript by accepting x-www-form-urlencoded.
Add classes to first item in a row to clear the row (currently uses :nth-child CSS)
Touch gestures and re-ordering (David's request)
}}}"
2204 1335795665000000 ross Giving this ticket to Aron to fix the front end
2204 1337582843000000 rgrp "Surely this ticket was closed 3w ago?
@Ross: can you close ..."
2204 1337594723000000 ross Was waiting for documentation, can close now.
2206 1331046486000000 johnglover "Close enough for now. Still have to restyle language drop-down to match template, but will fix when updating the list of languages.
Also still to decide on menu styles/locations (for CKAN login and main menu, and ODP main menu)."
2208 1332251880000000 seanh "I've attached a ckan.pot file. To make it, I edited setup.py in CKAN core like this:
https://gist.github.com/2135849
and then ran python setup.py extract_messages. (And I had ckanext-ecportal installed at ~/pyenv/src/ckanext-ecportal.)
I guess I may need to run this again when the ckanext-ecportal source is finalised, so leaving this ticket open for now."
2209 1331547639000000 ross Option 4 is the most appealing at this point, although you probably should look at #2209
2209 1331547674000000 ross "Previous comment should read:
Option 4 is the most appealing at this point, although you probably should look at #1649 "
2209 1331551170000000 ross "Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template.
Pros:
Easy to implement
Easy to use
Not hard-coded as currently
Fast execution
Cons:
Requires knowledge of required RDF output if default is not useful
RDF and not any of the other formats yet.
Only works with package/resource/tags unless more work is done
"
2209 1331551393000000 ross "Can't edit comments .. so
Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template.
Pros:
Easy to implement
Easy to use
Not hard-coded as currently
Fast execution
Cons:
Requires knowledge of required RDF output if default is not useful
RDF and not any of the other formats yet.
Only works with package/resource/tags unless more work is done
"
2210 1331549350000000 ross This is now done through the logic layer member_create/member_list/member_delete
2212 1331719868000000 johnglover Added as a new paster command in ckanext-ecportal (run 'paster ecportal' for a list of commands and arguments).
2212 1331720191000000 johnglover Deployed on test server, where it imported 4087 datasets. A small number of datasets were not created as they failed CKAN validation - most of which had strange values such as 9999 and 0 for date fields (some also didn't have unique names).
2220 1331291241000000 toby "issues:
1) want to cache static files long term - need to add a ?ver=xxx to allow changes - this will need a change to all templates etc - could be seen as a good thing
2) we can only purge specific pages not whole cache so we need to know all the urls which want to be invalidated - including language variants
3) urls should be consistent so we should use id not name for datasets etc
4) for now we can just use short expiry times as this prevents some of these problems"
2220 1332950550000000 toby "Implementation of this has been carried out at the page level.
ckan.page_cache_enabled is config option to enable it
requires redis
need to document it somewhere
"
2221 1331561095000000 ross Fixed in https://github.com/okfn/ckan/compare/1eeda11...ad2358a
2223 1331558791000000 rgrp "Have introduced boostrap js and css on merge of #1797 https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547
However need to remove blueprint and make sure we keep our original theme."
2223 1332246030000000 zephod "I have been working overtime to try and get this change straightened out as it's suddenly on the critical path for something. Have a look here: https://github.com/okfn/ckan/tree/feature-2223-bootstrap
It's 80% of the way there; some form CSS needs overriding because Bootstrap tends to make things very large so they spill over. But on the whole the markup is in a steady state, which is what we need."
2223 1332842692000000 zephod Merged into master here: https://github.com/okfn/ckan/commit/0bf4b33955df3806785f9b42a55acaeb45596e25
2224 1332475944000000 rgrp Autocomplete based on bootstrap looks pretty non-trivial ...
2224 1338205358000000 ross Being done by Aron
2225 1343220502000000 aron.carroll There is a jshint.json config file that all JS in the new base theme conforms to.
2226 1332159640000000 seanh These deployment docs seem worth moving into sphinx as well: http://wiki.ckan.org/Deployment
2226 1332475468000000 rgrp All done (as of Tuesday in fact!). No doubt further work to do but that's a new ticket!
2227 1332843126000000 toby being done in other tickets
2232 1332331660000000 amercader "Fixed in 6b4c6c9
This was due to some characters not being supported by XML. We now strip them before indexing."
2234 1337237687000000 rgrp This looks interesting but is this a priority :-) (also no estimate as yet :-) )
2234 1337239588000000 ross Yes, definitely need a replacement for wordpresser and sooner rather than later. Doing some experimental noodling to work out an estimate.
2236 1332153688000000 rgrp Done. See https://github.com/okfn/ckan/commit/0d970d3a0e523a1695830e8c8c28c0ba10a8d8b9
2238 1343392681000000 seanh Ticket #2771 encompasses this and related changes. Vocabs probably won't be deployed to test.ckan.net but a working example extension using vocabs will be added to core instead. Closing this ticket as a dup
2239 1334581838000000 ross Changes to schema/rdf output
2240 1332408769000000 ross Distributed to ckan-coord
2242 1333375436000000 seanh Conclusion: jenkins should work, may need some jenkins plugin magic to get it to show what branch each build was run on.
2250 1332680690000000 rgrp Is this now how we are doing it? I thought we were integrating with RDF store (so it was more about sorting out the data viewer / recline stuff)?
2251 1332331650000000 rgrp Big +1: everyone wants to know page views. Would like detail of how this goes into interface. Downloads already being tracked. Also isn't this just an extension to ckanext-googleanalytics.
2251 1332332390000000 toby "not really part of ckanext-ga as we need to sit first in the middleware for speed reasons. I'm looking at recording the details and will do a summary page like ga extension.
As far as other uses like showing popular resources etc then we need to decide where the data will live - on the resource or separate.
Downloads tracked? where is this done in the code/data stored"
2251 1332340874000000 toby "notes from meeting 21-3-2012
US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.
US1a Next to a resource or a dataset see how often it has been downloaded / viewed
US1b I want to see datasets or resources ranked by most downloaded or viewed
US1c See a trend graph for a dataset (and resources)
Adminstrative Dashboard (?)
* I want to see the traffic breakdown by country to my site ...
* Ditto for browser type, language, etc etc
* I want to see it graphed over time ...
== Implementation Details ==
1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display
* Config option ckan.status.enabled = False (by default)
=== Storing Data ===
How does ckanext-googleanalytics do this? Current table:
package_id | count_recent | count_total
Move to a new stats_summary table
id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value
Do we store this data into the search (solr) so we can search by it?
=== Displaying Data ===
* Helper functions / dictize:
* Helper function: h.stats_get(object_type, id, stats_type)
* h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
* Change to dictize
* Location in the default theme (do we show for example in search results too!)
* Support for ranking by most popular in search?
=== Tracking Data ===
* Our own solution (just write to site_tracking)
* Google analytics (plus extension for retrieving data) <- would need a refactor
* Piwiki
==== Own Solution ====
site_tracking table
id | url | timestamp | action (page_view, resource_download) |
* Make javascript to make request to ckan to store clicks and page views.
* Add middleware so these requests do not go through pylons and just store data quickly.
"
2251 1332340947000000 toby "TODO
look at
1. How do we store this data in CKAN?
3. How do we display * Config option ckan.status.enabled = False (by default)
"
2251 1332524123000000 rgrp Update description in great detail.
2251 1332776926000000 toby "we want
resource show downloaded/viewed on resources
show on the dataset total/recent"
2251 1333535572000000 toby "we have various options on tracking unique users seems the sane approach.
* unique daily view - needed for nice graphing
* unique total views
* total unique daily views - higher numbers and feels more right - easier to calculate if original data is lost/or archived
for now will collect all and then decide which to use for display as gives us flexibility if needed
display
-------
why is group listing of datasets inconsistent with main search (TDH at least)?
should we make these consistent?
group listing allows downloading of resource without tracking (this should be fixed)
if we make listings uniform do we want the download links or not?
I don't like the term download when it is just a link (maybe that's just me - what would be better?)"
2251 1333644538000000 toby "most work completed: outstanding issues
* add a prune tracking_raw data to cli
* add indexes to tracking_raw
* add summary page for new data format
* add api counter
1) do we record the url or action etc (action seems more sensible)
2) add in __call__ or in action?
"
2251 1334074772000000 toby api counter moved to ticket #2282 as needs better specification
2251 1334144999000000 toby "For meeting today with Adria
* How to add tracking to solr index. Can the data live in tracking_summary? Do we need any extra indexes on table to help indexing?
* How will we add order by popularity to search?
* Can we add resources too?
* Will search be best way to find most popular packages/resources or should we get that pages data from tracking_summary?"
2252 1333108787000000 amercader "There is a new ticket for the same schema as forms issue: #2268
The exception was fixed on [https://github.com/okfn/ckanext-inspire/commit/0dff1f6 0dff1f6]"
2252 1333374836000000 amercader This needs more work on ckanext-dgu and testing.
2252 1334567495000000 amercader Fixed on 0dff1f6c2e
2253 1340038490000000 toby closing as nothing has ever happened with this ticket
2254 1332343198000000 toby "What do we want to be able to do?
what do we use the stream stuff for at the moment?
la la la"
2254 1332950678000000 toby "plan of attack
a) get removal of TDH from core - this will give a better idea of the issues involved
b) have a think"
2254 1338210316000000 toby "closing.
we have fanstatic integration in ckan.demo work
tdh is it's own extension
h.snippet is in 1.7
ITemplateHelpers in 1.7
This allows most of the issues resolved. Full removal of the IGenshiStream filter still neads some form of in template hooks but this is being looked at as a possiblity in ckan demo and can be considered out of scope for this ticket
closing"
2255 1332752936000000 rgrp "I would say Public is default for Organization with Private being an option (or do we make this a config option?).
What about ownership by users?
> New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below)
Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page."
2255 1332772340000000 ross "> What about ownership by users?
Good question. WIll add to description.
> > New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below)
> Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page.
The former, although currently we will have a slightly different form in the organizations extension, until such a time as we're ready to replace the existing one. Tried merging the two forms but it ended up with such a mess of logic that it was unmanageable.
"
2255 1332934041000000 ross Changed description based on docs of stuff to verify.
2255 1334582784000000 ross Review + restrict resource access (when private)
2256 1333620795000000 toby "I'd like to remove all 'custom' icons and just use bootstrap ones - is this reasonable? they are added in different ways but h.icon() could be easily adapted for this.
minimal .sj .css files"
2256 1333720237000000 rgrp Not sure what ticket means (remove TDH from core?). Also agree that we should glyphicons from bootstrap where possible going forward - though that would seem to fit more naturally with #2224 (Simplify javascript and css dependencies and add minified version) and have little to do with TDH versus core :-)
2256 1334051526000000 toby "@Rufus,
The idea of this ticket is to separate TDH into an extension rather than it being incorporated into core. This should help TDH by making it possible to push fixes without needing to do a new release of core. It should also help make core more stable and as slim line as possible. New features can be tested in TDH and then if they have a wider appeal they can be merged into core.
There is a strong desire for this within the dev team."
2256 1334051733000000 rgrp @toby: indeed and I'm strongly in favour of the basic idea too (and largely a dev question so up to you guys though feature / functionality implications would need to be talked through). My point was that custom icon stuff was #2224 :-)
2256 1334052087000000 toby "@Rufus,
Thanks, I think there is overlap and I'm happy to leave that bit of work to you (or future ticket owner) I was just adding some brief notes to this ticket on some of the outcomes I'd like to see and hence why I'd be happy to simplify the icons."
2256 1336046098000000 toby I'm closing this ticket now ckanext-datahub exists and is live - the other work is included in other tickets
2257 1332950416000000 toby pushed first round of changes into master
2257 1336731023000000 toby ckan.restrict_template_vars defaults to true for 1.8 release
2260 1333373978000000 johnglover Still some work to do on this, template needs to be finalised.
2260 1333554069000000 johnglover Finished for this iteration, waiting for design work to be completed before making further changes.
2261 1332838253000000 toby "new config option
ckan.root_path = /path/from/root/with/{{LANG}}/substitution
"
2261 1332840056000000 toby completed plus documentation updated
2261 1332840112000000 toby please test logging in/out as the redirect was not testable but should be correct
2264 1333635637000000 seanh "Many strings have been fixed (and committed on master). Still to go:
When uploading a file:
- The ""Browse..."" button. This comes from the user's browser?
- Hangs when I try to upload a file, so the rest of this UI is unchecked.
The licenses on the dataset add, edit and view pages, ""License Not Specified"" etc.
Dataset state on dataset view page, e.g. ""active"".
When adding or editing resources:
- The resource types (Data File, etc.) The strings seem to come from resource.resource_type.
- ""You can use Markdown formatting here"", ""Should a DataStore table and Data API be enabled for this resource?"", ""Dates are in ISO Format — eg. 2012-12-25 or 2010-05-31T14:30"". I can't get strings with hyperlinks in them to translate here.
Dataset view page:
- License not specified.
Dataset search page:
- ""Tags"", ""Res_format"" and ""Groups"" in the sidebar. Leaving because this code has been refactored on feature-1821-multilingual-extension. Fix after that branch has been merged.
Dataset view page:
- The resource types, e.g. ""plain text"".
- The field names under additional information: cache_last_updated etc.
Dataset history view: timestamps not localised.
Login page:
- ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"" This string is marked for translation but I can't seem to get translations to work for flash messages. Also ""Sean Hammond is now logged in"" flash message.
Register page:
- ""Missing value"" in error message when you try to register and don't give an email. The string seems to come from formencode. stdtrans() not working?
User view page:
- Activity streams strings. Activity streams genshi function needs a little refactoring to be i18n-friendly.
Users page:
- ""less than 1 month"". This string comes from Pylons' time_ago_in_words() function which doesn't appear to support i18n.
New auth group (and edit auth group) page:
- ""Unique identifier for group. 2+ chars, lowercase, using only 'a-z0-9' and '-_'"". This string is marked for translation but for some reason it isn't working.
- ""Name: Please enter a value"" (in an error msg)
- ""User name"". The string comes from class FormBuilder?
Also, need to check that all tooltips are translated!"
2264 1334348070000000 seanh "A few strings are still left to be marked for translation:
- http://trac.ckan.org/ticket/2289
- http://trac.ckan.org/ticket/2288
- http://trac.ckan.org/ticket/2287
- http://trac.ckan.org/ticket/2290"
2265 1332865837000000 dread "I started work on this and need to park it somewhere - here will do for now.
I was using the 'MLT Component' http://wiki.apache.org/solr/MoreLikeThis :
https://gist.github.com/2217684
But I think it would make more sense to use the 'MLT Handler' http://wiki.apache.org/solr/MoreLikeThisHandler which needs enabling by adding this to the config {{{/usr/share/solr/core1/conf/solrconfig.xml}}}:
{{{
}}}"
2266 1332932634000000 dread Fixed in [release-v1.6.1 ed2772b] with test.
2267 1343135541000000 toby this is not important as far as phase 2 is concerned - but you can do if you want
2267 1343229146000000 aron.carroll Fixed in eade4e3
2271 1334567582000000 amercader This was caused by an human error.
2275 1334159668000000 johnglover Done in branch feature-2275-group-logo, still need to decide on how to style.
2278 1333441177000000 rgrp https://github.com/okfn/ckan/commit/b7b32820a9132811028e7ccc243b2d2339492d4d
2281 1333967862000000 dread Done in [release-v1.6.1 9ecbfa5][release-v1.6.1 9ecbfa5]
2283 1340623843000000 kindly No super tickets anymore.
2284 1337583675000000 rgrp This needs connecting to the user stories being generated at http://ckan.okfnpad.org/dataexplorer-user-stories
2285 1334580266000000 rgrp "Recline improvements are done.
https://github.com/okfn/recline/issues/88"
2288 1338204815000000 ross Auth groups are being deprecated
2289 1340887101000000 toby needs docstrings
2292 1335021860000000 rgrp "This annoyed me so much I just fixed it: https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5
Issue was that our auto-completion depended on structure of the HTML which changes with bootstrapification.
BTW: have deployed this via copy and paste on datahub so i can use it. Will want to revert this when you do a deploy.
@Ross: I note that the same bug probably affects organization auto-complete."
2293 1339062853000000 ross "Intercepting the group name change and forcing a reindex of all of the datasets within that group seems like the obvious approach (rather than more lookups by ID after processing search results).
Should take less than a day."
2296 1334567141000000 icmurray "* ensure the read_ajax and history_ajax actions work in the routing.
* ensure the moderatededits.js works with the new templating
* ensure that the css works with the new templating
* ensure that the stream filters work with the new templating.
~ 6 days work"
2297 1334569632000000 seanh http://ckan.okfnpad.org/27
2298 1340285049000000 toby related to http://trac.ckan.org/ticket/2579 move sort functions to helpers
2302 1335516537000000 ross Will eventually want to allow customisation of different CSS but designs are currently under review/change.
2302 1342095074000000 toby "@ross
do you want to give this to aron or me?"
2302 1342097600000000 ross Have assigned it to Toby. Not sure how high priority this is, but only provides a very minimal level of configurability.
2302 1343315555000000 shevski "My ideas for this could include one day:
One page where the portal admin can choose and/or change the:
1. Portal name & tag line
2. Logo (to be displayed in header)
3. About page text
4. Featured dataset & text on homepage ?
5. Featured groups for homepage ?
6. Footer links
7. Colour scheme
8. Homepage background (?)"
2302 1343596669000000 shevski I reckon getting this live, esp with a easy to fill styled text header (see Sam's design) and a UI for changing About page text, featured dataset on homepage and welcome text is key
2302 1343636153000000 toby "@shevski,
please do not try to change the meaning of tickets/add features or they will just get closed as won't fix.
featured datasets is nothing whatsoever to do with this ticket so adding it to the ticket just causes me to get upset / demotivated etc. please do not do this. It ruins my day.
Also comments like see sams design is of no value without a url in fact it has negative value.
:(
I have changed the priority of this ticket to account for this but next time i'll just close it."
2302 1343820855000000 ross Just a quick note, if Rufus asks about editing about page, he's talking about this ticket.
2303 1334654727000000 ross feature-2303-rdf-export-command
2304 1335878372000000 seanh I'm setting the priority of this high just because it's a low-hanging fruit for this sprint, as it was almost done last sprint.
2304 1335885463000000 seanh Setting minor again after cleaning tickets out of this sprint
2304 1337073870000000 seanh Currently implemented in this branch https://github.com/okfn/ckan/compare/master...feature-2304-follow waiting for code review and merge into master
2304 1337158641000000 seanh Finished and merged into master
2305 1335879165000000 seanh Setting the priority to major as I think it's worth getting this done since the follow support (which is almost finished) is a bit pointless without this.
2305 1335885478000000 seanh Setting minor again after cleaning tickets out of this sprint
2305 1340879812000000 seanh Implemented by Sven
2311 1334837867000000 toby "most work has been completed
comments can be added to templates via ${h.disqus_comments()}
recent comments via ${h.disqus_recent()}
basic i18n works for languages except norwegian which reverts to english
needs ckan 1.7
awaiting clarification re urls changing from under us"
2312 1340188741000000 ross "We have a ticket for object ownership (group and package) #2548 which will make it much easier to implement this functionality of showing user owned datasets in a 'virtual group'.
Will close this in preference to the other ticket which is needed for other features too."
2313 1334756921000000 seanh "Note that c.new_facets has now been renamed c.search_facets.
Before this ticket is closed the old c.facets should also be removed from the template context. It's put there by ckan/logic/action/get.py:package_search(), which also puts the new c.search_facets there."
2315 1334919522000000 dread Done, with tests and bootstrapification of the form: [release-v1.6.1 b23b357]
2317 1340033433000000 kindly Getting replace as part of new theme.
2319 1340624083000000 ross Resolved with rework of UI
2322 1337339587000000 ross "+1 on these, although ../download instead of ../raw and ../view instead of ../viewer
"
2322 1337864855000000 ross "Viewer almost exists with the /embed link and so have just used that with a better default size to provide a prettier url. Can still use the width=&height= that are used in the /embed url.
"
2322 1339749795000000 rgrp When will this go live? Will this be in v1.7.1?
2322 1339750135000000 ross No, new new features in point releases, we only put bug features in unless there is an extremely pressing need.
2322 1339750208000000 ross "Going to try that last comment again.
No, no new features in point releases, we only put bug fixes in unless there is an extremely pressing need."
2322 1344086640000000 rgrp "Can I confirm this is in v1.8 ...
"
2323 1335516967000000 ross "Doesn't make sense in Organizations mode. Adding existing dataset would mean that dataset would have to already be in another organization.
Leaving this open though to fix in case we decide that datasets *can* be in more than one organization at a time."
2323 1340188400000000 ross Can create a new ticket if the requirements change.
2325 1335366188000000 dread Done in [release-v1.6.1 6343a9c]
2329 1346664271000000 ross Moved to 2.0 branch for new UI.
2330 1336149571000000 icmurray "Implemented, providing two methods of passing an action's parameters. Either, each parameter is specified as a url parameter, eg
{{{
curl http://127.0.0.1:5000/api/3/action/package_search?q=police
}}}
or, for cases where a more complex data_dict is required, the data_dict can be parsed as a json-encoded dict in a single url-parameter called ``data_dict``, ie:
{{{
curl http://127.0.0.1:5000/api/3/action/package_search?data_dict={'q':'police'}
}}}
As part of this ticket:
- I noticed that all module-public members of the action modules were being loaded as actions (including functions imported from other modules). So I've cleared up the action modules to prevent this.
- I've cleaned up the action modules to use ``get_or_bust()`` when accessing a passed in data_dict outside before it's been validated by a schema. This provides better error messages for clients.
- provided a decorator, ``@side_effect_free``, for users implementing their own actions through the IActions interface to be able to expose their custom/overridden actions as a GET request through the action API."
2330 1336387328000000 icmurray "Replying to [comment:3 icmurray]:
>
> or, for cases where a more complex data_dict is required, the data_dict can be parsed as a json-encoded dict in a single url-parameter called ``data_dict``, ie:
>
> {{{
> curl http://127.0.0.1:5000/api/3/action/package_search?data_dict={'q':'police'}
> }}}
>
Another option is to have the data_dict json-encoded in the GET request's body. This is possible, but
1) Although permissable, it's not, as I understand it, conventional to have a GET request's body form the options of the request.
2) Caching proxies won't inspect the body of the request, only the URL.
"
2331 1337782689000000 kindly "This is a wontfix. I think terms should be ored by default. All modern search engines work like this.
If there is an issue due to relevancy (i.e if you type mulitple words and your result not coming near the top) then we should use this examples so we can tweak the results."
2331 1338390995000000 rgrp "@kindly I really don't get this but I may understand poorly. Google does not work by or-ing does it (as i add things to the search query I get *less* results not more ...).
It also makes it really hard to find stuff: as I add query terms to my query I get more stuff not less which makes for terrible finding of stuff - I appreciate that there is a scoring aspect but that seems secondary."
2331 1338455981000000 kindly "Scoring is primary in my opinion. Who cares if you have 1000 results if the top few are yours.
If things are hard to find we need to change our relevancy first. So if you have examples of where you think the scoring is wrong then please make a ticket for that.
Google, I imagine, just has a cutoff of anything under a certain score not being shown. We could do that as well but it would take some working out what we wanted that score to be.
Full ""And"" queries also limit any accidental discovery, especially of rare terms. If you do not get your search exactly correct then you get nothing, which is bad.
Obviously you can still AND things or +things.
The correct solution to this is adding a minimum match parameter which is a middle ground. eg you can say that you want to match over half of the terms. ""thing1 thing2 thing3 thing3"" means you have to match at least 2. There are many options described here. http://wiki.apache.org/solr/DisMaxQParserPlugin. You can change this in an extension if you want it just requries adding an mm field to the before_search in the Ipackage controller. I do not personally think we should change it as default.
I am closing it as wont fix as its trivial to change and is a philosophical difference not a technical one."
2331 1343814758000000 rgrp I'm re-opening this. This is a reasonably significant UX problem -- not just a philosophical difference. It is just *weird* to get *more* results as I add search terms rather than less. Please can we have this fixed ...
2331 1343862230000000 kindly As I said I do not not agree with it being a UX problem or *wierd*. *wierd* is not acceptable in response to a thought out comment. So I am closing it again.
2331 1345191494000000 rgrp "Re-opening yet again. This is something brought up by entire client team on several occasions :-)
We really do need to fix this as it is really poor UX. To give, yet another, example:
http://datahub.io/dataset?q=thatcher+wages
This returns 15 datasets yet only one of them has any mention of thatchers (the first one!). (The others are there because they match wages). This isn't what I expect. I expect *only* to see datasets that match *both* terms ...
"
2331 1345192722000000 ross "This is the dullest game of ticket tennis ever :D
Google don't OR the results, they have an explicit operator for ORing query terms and default to ensuring that all terms are present (in one form or another) in the documents. I understand the argument about cut-offs and relevancy scores, but for a large search engine they have the benefit of scale both in resource and documents. I spent 6 months of my life tweaking weightings for n-dimensional vector space search, I don't think it is likely to be a number we can, or will find by accident.
Given that Marcus also got confused by the current behaviour I think we should either:
a) Make it configurable
b) Just change it to what has been asked for
Don't care which.
"
2331 1356474344000000 rgrp Now transferred to github at https://github.com/okfn/ckan/issues/258
2332 1337582891000000 rgrp @rossjones / @amercader: can you do a full review and generate sub-tickets as necessary (obviously may not all be this sprint ...)
2345 1335878420000000 seanh Setting priority high as we want this done by Friday for the 1.7 release.
2345 1336492725000000 seanh Work on this has started in branch feature-2345-action-api-autodocs
2345 1337962454000000 seanh Done but now I need to fix readthedocs again
2346 1338205553000000 johnglover Will create a separate ticket for 'organization' to 'publisher' renaming, the rest are done.
2347 1335962835000000 ross [ ] Check that dataset privacy can be taken into account.
2347 1336039458000000 ross Keeping track of UI changes required, and copied Aron
2347 1337238738000000 rgrp "This looks great Ross. Couple of things not included (which I have taken liberty of adding to the description as well - you will probably want to tidy up):
* Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. ""Apps, Ideas, Visualization and other material related to this dataset"") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!)
* Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says
* What it does
* How to enable (and perhaps includes a nice-screenshot!)
* How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...)
* Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...)
* Clearly mark this old extension as deprecated from v1.7 forward: https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7)
"
2347 1339602763000000 shevski "Name-wise: what do we think of 'Related Media' - that suggests news articles, apps & visualisations to me
I'n not sure about relating 'ideas' to specific datasets. Looking at the way ideas work in DGU & pd.eu - it's more general and portal-wide so should flesh out user stories for this before adding ideas to related extension"
2347 1340613938000000 ross Needs some UI work now, have changed status to blocker which appears to be the only way to denote something as blocked (awaiting resource).
2347 1343126819000000 icmurray "Small fix to create action, outlined in [1] to be done. Said I'd do it, so moved ticket over to myself.
[1] https://github.com/okfn/ckan/pull/62"
2348 1338193014000000 ross "A fairly good start has been made on this at https://github.com/okfn/CKANClient-J
Lowering priority for completion for now"
2349 1336037341000000 ross Openlink hoping to get RDF dump into Virtuoso today.
2349 1336555121000000 ross Chased Openlink today for update
2349 1338193148000000 ross Initial DNS change doesn't seem to have taken, chasing Nils to see if he can take a look.
2349 1339075361000000 ross "Created symlink from /var/www/nginx-default/dump to the rdf output folder, accessible via http://s055.okserver.org/dump/ with daily dump available at http://s055.okserver.org/dump/rdfexport.tgz
"
2349 1339428913000000 ross "Notified Hugh that http://datahub.io/dump/rdfexport.tgz will contain all of the RDF files every 24 hours if they want to fetch and import them.
Have asked to make sure that this is satisfactory."
2351 1340615641000000 icmurray Scripts and docs in https://github.com/okfn/ecportal-server-setup
2353 1336037264000000 ross Started the user stories at https://docs.google.com/document/d/185sWPEO9cn1SbquH83UFcLme7xKdgfhCNqOo7pdsRKw/edit
2355 1336731240000000 toby "ckan.plugins.toolkit introduced to help control access to ckan for extensions. This should reduce breakage for extensions using this.
ITemplateHelpers allows extensions to add helper functions so templates can be less reliant on Genshi filter"
2355 1338212946000000 toby "Need to do documentation which is #2362 so has own ticket
closing"
2358 1336641577000000 amercader Took longer than expected due to last minute changes and problems with the packaged version.
2361 1337016768000000 amercader I actually spent some more time of this because I upgraded PDEU to CKAN 1.7, which was needed to fix the search index and a good testing for the release as well.
2362 1341314244000000 toby "first part complete
part2 is the auto generation merge that is for 1.9"
2362 1342085420000000 toby need to get the auto-documentation done now
2364 1335890204000000 ross Review the FRY app and investigate alternative payment gateways.
2364 1336483769000000 ross Start of doc listing payment gateways is at https://docs.google.com/document/d/1E4lX6RnRLDO0ioieKRpVs9a-My9nh6lBjvVahWDm3SY/edit
2365 1336492342000000 ross See also #1165 and #1096
2365 1340266964000000 ross After discussion with DR, it was decided that multisite wasn't something we would approach now, instead going for individual (shared codebase) installs.
2366 1336395238000000 dread For my site I need the Authorization Groups sections gone from the authz pages, for v1.7.1, so I've made these check for the presence of any Authorization Groups in commit: [master 8947302]. So it's another useful stepping stone to getting rid of them cleanly.
2366 1336555441000000 ross I think the UI aspects have been removed in 1.7 with hopefully the rest of it going in 1.8 - assuming we have a clean migration script.
2366 1336560445000000 dread "Thanks Ross. It's the dataset authz and admin authz pages which still need Authorization Groups removing. e.g. http://thedatahub.org/dataset/authz/conservative-party-uk-donors-by-sector
and
http://thedatahub.org/en/ckan-admin/authz"
2366 1338215334000000 ross Need to make sure there is a clear migration between authgroups and organizations if desired.
2367 1336035031000000 aron.carroll Fixed in ckanext-datahub@4aa76740e3def64a80171df4417c195210f0eb25
2371 1338481174000000 toby implementation trial in ckan demo branch #2375
2371 1340038431000000 toby awaiting release and demo theme merge
2371 1345209675000000 toby closing as part of 2375 branches
2372 1343220160000000 aron.carroll These are now in https://github.com/okfn/ckan/blob/b31109b628a21072ec0c2dec2a5d5a1c2cc0ed9c/doc/coding-standards.rst#javascript-coding-standards
2373 1336142514000000 dread Fixed in https://github.com/okfn/ckan/commit/8b7141cd2ff619a0e35aaa31bbc1d17156647498 on master (and also on release-dgu1).
2374 1340287976000000 ross "count is returned at the top level of the dict (at the same level as results)
Was added by dread on 2012-05-04, must have been in a recent patch."
2375 1338292898000000 toby stealing this ticket
2375 1340710303000000 toby moving milestone
2379 1340033026000000 ross Implemented in master
2380 1337868169000000 ross "datagm is VERY broken with 1.7
Waiting for datasuite UI changes."
2380 1340627558000000 ross Dupe of #2545
2383 1337872916000000 icmurray "This helper function won't parse UTC offset if passed a UTC offset as ""+01:00""; but incorrectly parses the UTC offset ""+0100"" as 100 microseconds.
Resolution:
- I've decided to keep the behaviour of not parsing UTC offsets because we currently don't store timezone information in the database; and I think that either dropping the tzinfo, or adjusting for it would appear to be ""magic behaviour"" from the user's perspective. Also, as this is a helper function, I think it makes sense to keep the behaviour as is, even if it's slightly non-standard.
This fix [1] maintains all the old non-ISO behaviour:
- not allowing UTC offset to be specified.
- arbitrary delimiters between date and time values.
- microsecond precision.
But won't parse ""+0100"" as microsecond precision.
I tried using a couple of python modules for parsing date strings; but none maintained the same behaviour.
[1] https://github.com/okfn/ckan/commit/d6c9e48f8df5e926abd3fb1a1607f4f5a30075b4"
2384 1337100810000000 dread Done in 0bc06c6495c749d6b481164d59cceeca873acefb
2386 1337967006000000 seanh Done. ckanext-spatial and ckanext-qa are not activated because they don't work with the current version of CKAN. datastorer is not installed yet.
2389 1337237502000000 rgrp Just to note these were 2 *different* options. '''Either''' you do the fallback approach '''or''' we turn off DataStore by default and only use if enabled (which happens due to explicit enabling of DataStorer turning it on ...)
2390 1337100788000000 dread Done in d1c2ed08c90c2d94e1b2ced86e8758208a61ee96
2391 1338397982000000 seanh I've told them how to do it themselves and offered to help if they get stuck
2394 1337161917000000 dread Done in [master b486244]
2398 1337963342000000 seanh I'm closing this because we've got a whole bunch of tickets from the dev meetup about fixing the dependencies and requirements file that should deal with this problem
2399 1340706702000000 amercader Fixed in 493589aaf
2400 1338205947000000 toby In testing this is not replicable due to lack of info so closing
2401 1337273042000000 dread Done in master 112e405d0116bdcf196062520e3474e231337996
2402 1337302474000000 kindly cset: 12da5e9effeeb1aca0df321c355d8438647ef426
2403 1337302700000000 kindly cset: 05144f8621ee719c345373934e70719f46e87cf6
2405 1337583749000000 rgrp Assigning to ross for review and assignment.
2405 1338212357000000 ross Suggest Sam might be able to provide you the icon
2405 1338213564000000 icmurray "Questions for client team:
1. Did you have anything in mind for improving the ""Powered by CKAN"" text and icon? I'll sort out the vertical alignment of the text and icon; and see if Sam has a better icon.
2. The branded section itself is already customizable by a snippet (https://github.com/okfn/ckan/blob/master/ckan/templates/snippets/data-viewer-embed-branded-link.html). Is that enough? Or are you asking for something else?
3. Personally I think the URI should use the resource uuid, and not name. And from recent discussions on ckan-dev, I think the consensus is the same. How important is this for you? Perhaps it's something that could be overriden in the datahub extension."
2405 1340097846000000 icmurray Bump: Questions for client team above...
2406 1337766385000000 toby "I installed CKAN yesterday afternoon from source, using the instructions at
http://docs.ckan.org/en/ckan-1.7/install-from-source.html
Most things worked pretty smoothly and, luckily, when they didn't I
was lucky enough to have David Raznick on hand to help sort them out.
As a result I've made the following notes on where the doc could be
improved.
For some ghastly reason I am running Linux Mint, which is some variant
of Ubuntu. It's not 10.04, so no package, hence the source install.
Sec 2: Not wanting a mysterious file called 'pyenv' in my home
directory, I created a directory ~/ckan and typed 'virtualenv pyenv'
from there. Everything worked fine. But if I had needed the 'tip' it
would have been a little confusing. It would also be worth reminding
the user that the dubious package is one they've supposedly just
installed, so if that went OK, they won't need to worry about this.
The tip also isn't clear about under what circumstances I'd need to do
the 'easy_install pip' - I didn't need to (is that also only when
python-virtualenv is missing?)
Sec 4: This worked, but the example command for installing a
particular version is confusing. It says it is for 1.5.1, but
replacing '1.5.1' with '1.7' fails because there is a rogue 'c' in the
command.
Probably it would be better to use the latest version (1.7) as the
example anyway. Or is there a way of asking for the latest released
version?
I understand this checks out the whole Git repository with all history
- I can't imagine why I'd need to do this to build a particular
version. But David assures me it's very sensible.
Sec 6: This worked, but it wasn't very clear what the purpose of the
'List existing databases' step was.
Sec 8: So now I had to try to get Solr working using the instructions at
http://docs.ckan.org/en/ckan-1.7/solr-setup.html
This didn't work at all, in ways I can't now remember, to do with
Java. In the end David suggested using Tomcat rather than Jetty (don't
ask me what these do, but they have something to do with getting Solr
to work ...) and this turned out to be much more straightforward to
install and get working, with two caveats (below). David suggested
maybe Tomcat should be the default suggested in this doc.
- One problem with Jetty was to do with using the wrong version of
Java, which seemed to be fixed using ""sudo update-alternatives
--config java"". It didn't fix Jetty - not sure if it would also have
been needed with Tomcat.
- Soft linking the schema file didn't work - it seems Tomcat doesn't
like soft links. We made a copy.
After this everything worked, and I now have my very own CKAN. Hurrah!
Mark
"
2406 1337766413000000 toby "The 'post-installation setup' instructions at
http://docs.ckan.org/en/ckan-1.7/post-installation.html
claim to be for either package or source installations, but only work
as stated if you have done a package installation. E.g. after a source
install, ""paster --plugin=ckan user add admin
--config=/etc/ckan/std/std.ini"" doesn't work (the ini file is
somewhere completely different), but just typing ""paster user add
admin"" from the ckan directory is fine.
The instructions should also mention that the virtual environment
should be activated.
Mark
"
2407 1340633440000000 amercader This will require some thought. Bumping to 1.9.
2408 1341497601000000 aron.carroll This will be done as part of adding javascript in phase 2
2408 1343657014000000 aron.carroll Fixed in df502fd9269859c73458f4ad8de75939220e0315
2409 1337688965000000 ross This is in the Package controller and may be related to upcoming ticket on removal of autoneg.
2410 1338193633000000 ross Tim Lebo and Keith Alexander are both now looking at how they can implement this themselves, so for now this is wontfix. We can reopen this as part of any future work with VoID
2411 1338210872000000 toby This is a long term goal
2414 1337869578000000 ross "LXML is currently used for markdown_extract and check_solr_schema_version
Changing check_solr_schema_version should be possible,
Used in markdown_extract to just strip HTML tags."
2414 1338471374000000 ross Merged by kindly
2416 1338464207000000 toby commit 37622d5 does this but probably in the wrong place but works for now. Need to revisit at some point.
2416 1340710252000000 toby moving to phase 2 as not causing problems but still needs proper fix
2417 1337864155000000 toby fixed in demo-theme branch
2418 1337875172000000 toby "all facets now shown and selected have -
@aron styling of active facets with long names breaks. we could make the names shorter in snippet/facet_list.html via truncate or make the selected taggy thing wider"
2419 1337863024000000 toby "@aron
the reason for this is that controller/package.py renders the form `package/new_package_form.html` as part of it's work and then makes this available a `c.form` to the template. I'm not sure I like this as an approach but it is how it is currently done. If you split your form into a new template `package/new_package_form.html` then you get all the errors etc.
I think for now we should do this and then look at if doing it all in one template pass might be more sensible.
reassign to me if you want me to do this else I'll leave it with you"
2420 1337950422000000 toby merged into master
2422 1337945204000000 seanh Closing because I'm not able to reproduce and it turned out floapps had publisher auth on which was overriding his rights auth
2423 1338193199000000 ross Duplicate of #2414
2424 1338193269000000 ross Duplicate of #2413
2428 1341234933000000 seanh This awaiting merge, pull request: https://github.com/okfn/ckan/pull/46
2428 1341314475000000 seanh Merged. The only thing I didn't do was add a test case that runs pip freeze and tests that all the right versions of deps are installed.
2430 1338397815000000 seanh http://trac-hacks.org/wiki/MasterTicketsPlugin
2431 1338397906000000 seanh https://github.com/davglass/github-trac
2432 1338205526000000 aron.carroll "It would be great to use something like http://trac-hacks.org/wiki/SimpleTicketPlugin to remove most of the options when creating a ticket.
Suggestions to remove theme, repository, component and type unless they're actually useful."
2432 1338325120000000 shevski "Where do new tickets go to await triaging?
Is priority set by whoever creates the ticket? Does this get revised? Will we have a field for dev time estimate or where will this go?"
2433 1337960203000000 toby "the affected actions
===============
package_list
package_relationship_create
revision_show
group_list
package_update_validate
It would be nice to get this fixed and I'm loathed to suggest adding a version 4 hack.
If we make this change it will break things for existing users. This behaviour has existed since the action api was added as far as I'm aware
So should we
a) leave it forever and document
b) break things and fix it - when 1.7.1 or 1.8?"
2433 1338397268000000 toby after discussion it has been agreed to keep the existing behaviour and improve documentation as it is likely to break existing applications etc
2434 1341234522000000 seanh "We've moved to AWS hosting with docs generated by Jenkins instead, see:
http://trac.okfn.org/ticket/1254
There are still two issues to resolve:
1. Automatically updating the symlink so that docs.ckan.org always redirects to the latest stable version
2. Finding somewhere to put links to all the other versions"
2434 1350296272000000 amercader Seems to be working again, closing.
2437 1338484676000000 seanh "Please add to the file started to doc/coding-standards.rst
Also, for the Python coding standards see http://wiki.okfn.org/Coding_Standards as a starting point"
2437 1338487089000000 seanh "I've also added the commit message guidelines as they reference a trac plugin I'm going to get installed.
I'll leave the rest to you.
I remembered that http://wiki.ckan.org/Main_Page has a bunch of pages under Developer Resources that should probably be updated and merged into these coding standards and deleted from the wiki"
2438 1338212234000000 icmurray "Proposed solution:
Add a new q field which can be specified multiple times. Each q parameter is a string: `{field}:term`. `:` characters are escaped in the `{field}` by `\:`.
Mark the `fields` parameters as deprecated."
2438 1340736112000000 icmurray "Ready to be merged [1]
Brief synopsis of changes:
- altered return type of the action: from Resource domain objects to dictized resources. This is required if this action is to be made accessible through the action api as the domain object is not json-serializable. The 'old' return type is still available by passing in a flag in the context. In fact, this is required as the search layer uses this action, and deals with the domain objects directly.
- decided that escaping the ':' (as described in above comments) wasn't necessary as resource fields shouldn't have ':' characters in them anyway.
- improved action's docstring
- 'fields' parameter has been deprecated ( see #2603 ). Use of it by the search layer has been removed in favour of the new 'query' parameter. But it still works, and correctly maintains backward compatibility.
[1] https://github.com/okfn/ckan/pull/37"
2439 1338214158000000 icmurray "The `fields` argument doesn't provide any extra functionality that the `query` argument can't provide.
1. Figure out what the reason for this argument is, just to make sure.
2. Deprecate it, and change any internal uses of it to use `query` field instead."
2440 1339421746000000 aron.carroll "Has some styling as of b0153ba but needs content to be applied.
Toby, can you pull in the top 10 tags, as well as n of the most ""popular"" datasets and n of the most recent datasets. Actually just providing the functionality as helpers would be great.
Do we actually have ""featured"" datasets implemented at the moment?
Not sure what we're going to use for the box on the top left when on a new base CKAN instance."
2440 1339423966000000 toby "@aron,
tags what functionality do we want for this? I can just reuse the facet_list snippet but then we have all sorts of interactivity. Do we want this more a static list of tags (linked to somewhere but no add remove stuff)? seems more sensible I think
featured datasets I've no idea I'll try to discover"
2440 1339430956000000 toby "you may want to look at your css as the dataset is generated by the package_list snippet
I've made all packages popular for now"
2440 1339497700000000 aron.carroll Fixed this up, last thing I think we should do is ensure that the datasets passed to the homepage have descriptions.
2440 1339498655000000 toby "dataset descriptions is a data issue but I'll keep this open for now. We can probably do an exclude if needed.
"
2440 1339499074000000 aron.carroll Sure, but this is more about displaying nice complete examples on the homepage.
2440 1339581622000000 toby I am changing this to assigned not accepted as this makes trac easier to use
2440 1340373461000000 toby i'm closing we can make tickets for specific issues
2441 1338287243000000 aron.carroll Order by needs hooking up to form. Currently it's just a @
What Are Groups?
- Whilst tags are great at collecting datasets together, +Whilst tags are great at collecting datasets togethe
+
+
+ Creat
+
+
+ To create a new group, please first
${h.subnav_link(c, h.icon('group') + _('List Groups'), controller='grou
Create a new group To create a new group, please first log-in.
}}} {{{-
+ class=""${'fullyopen' if (package.isopen and package.resources) else None}"">
${h.link_to(package.title or package.name, h.url_for(controller='package', action='read', id=package.name))} @@ -60,7 +60,7 @@
-
-
-
@@ -72,7 +72,7 @@
- ${h.icon('lock')} Not Openly Licensed diff -r adf254b7c507 ckan/templates/home/index.html --- a/ckan/templates/home/index.html Sat Jul 23 18:16:15 2011 +0100 +++ b/ckan/templates/home/index.html Thu Jul 28 14:10:23 2011 +0000 @@ -64,6 +64,7 @@
- ${h.nav_link(c, _('Add a package'), controller='package', action='new', id=None)} +
- Upload Data
- ${h.nav_link(c, _('Tags'), controller='tag', action='index', id=None)}
- ${h.nav_link(c, _('Groups'), controller='group', action='index', id=None, highlight_actions = 'new index')}
- ${h.nav_link(c, _('About'), controller='home', action='about', id=None)} diff -r adf254b7c507 ckan/templates/package/read.html --- a/ckan/templates/package/read.html Sat Jul 23 18:16:15 2011 +0100 +++ b/ckan/templates/package/read.html Thu Jul 28 14:10:23 2011 +0000 @@ -56,7 +56,7 @@
-
-
This Package is Open
@@ -80,7 +80,7 @@This package is Not Open
Either because it is not openly licensed or is missing }}}" 1246 1311885360000000 pudo "I'm afraid to say I don't understand any of this: the faulty display of openness is triggered as removing the brackets turns the isopen() check into a check for the existence of the method. Reverting the changes on the server seems to work without generating exceptions. What exactly is the issue that was supposed to be fixed originally? " 1247 1311955176000000 rgrp Done. See http://ckan.readthedocs.org/ and http://docs.ckan.org/ will propagate soon. 1248 1319797763000000 dread "Fixed in cset:188d27a56c89c35f97f3a79fdd0a6460223f86cf on branch release-v1.4.3. This was only seen in CKAN 1.4.3beta." 1249 1311959356000000 pudo fixed in cset:0398243111d4 1250 1312370201000000 pudo "The issue is caused after search returns from solr and tries to get objects from the database. These are by default ordered by name and we need to restore the original order after fetching them from the database. Fixed in module." 1251 1314406332000000 rgrp Really non-urgent and can be done once #1252 is done (which is dependent on v1.5) 1251 1317315159000000 dread rgrp did this last weekend. 1252 1314406271000000 rgrp Move to v1.5 since this will coincide with release. 1252 1316876747000000 rgrp Announced. thedatahub.org now running for 2+ months. Today did wiki.ckan.net redirect and hope to do thedatahub.org redirect. 1252 1317315135000000 dread This is done now? 1252 1318160334000000 rgrp This was done ~2 weeks ago. 1256 1312409296000000 kindly This has also caused problems in authorization for deleted packages. 1256 1312813529000000 kindly fixed cset:357cf9377b25 1256 1319812556000000 dread It sounds like this became an issue with CKAN 1.4.3 and the fix goes into CKAN 1.5. 1258 1312813614000000 kindly cset:e22d4e385fc8 1258 1319812452000000 dread I believe this was broken with the introduction of moderated edits in CKAN 1.4.3 and this fix went into 1.5. 1262 1314029150000000 dread "Pudo - I'm not sure what's meant here. Are you talking about moderation for new users? Or some other sort of enforcement? Is it spam you want to target? The links you're concerned about - are they the spam ones in the User's 'about' field? If so, I think we *should* allow users to have links, although continue with applying them with 'nofollow' and attack spammers back in a concerted strategy across all editable fields - see ticket #1257." 1263 1314029279000000 dread This was fixed in cset:8d293393a9f2 for release 1.4.3. 1264 1314406294000000 rgrp Nothing urgent about this so moving to backlog. 1264 1318244619000000 zephod Created #1377 to handle bugs revealed while doing this work. 1264 1318245716000000 zephod Complete; cset:b216952644aa 1265 1312566052000000 dread Fixed in cset:a1b8dd169f4e on 1.4.3 release and merged to default. 1266 1312797477000000 dread Fixed in 987dc4fe078e on 1.4.3 branch and default. 1267 1312889130000000 dread John thinks it is a problem with WebOb 1.0.6 1267 1312890962000000 dread This error has been occurring on at.ckan.net which has webob 0.9.7.1-1 1267 1312894071000000 dread Running tests with WebOb 0.9.7.1, we see the error! Am now trying to fix. 1267 1312897724000000 dread "Fixed in cset:db5bf5b45b4f for ckan 1.4.3. Later versions of webob add the charset header for unicode errors (webob/exc.py:generate_response): {{{ extra_kw = {} if isinstance(body, unicode): extra_kw.update(charset='utf-8') resp = Response(body, status=self.status, headerlist=headerlist, content_type=content_type, **extra_kw ) }}} This was fixed in webob cset:25fc14d49623 which went into WebOb 1.0. This problem affects versions from CKAN 1.4.2 back to 1.3.1. We had been avoiding this in CKAN previously by only doing abort() with a plain string. " 1268 1317212499000000 kindly fixed cset:eebbe6071741 1270 1312912188000000 dread Fixed in cset:ca5295c9c8cf and released as ckanclient version 0.9 1271 1313753663000000 rgrp Closed in cset:749cb3a087e3 1272 1313745867000000 rgrp "Assume these are the 3 spatial fields listed on http://wiki.ckan.net/Geospatial_Capabilities. I'm a big +1 here. Starting point would be a bit more documentation on that wiki page. Questions: * Is this work going in ckanext-spatial? * What is being proposed here, I assume, is to automatically index those fields in our geospatial search. * Should we develop a simple display widget using google maps or similar (this is orthogonal to the WMS preview i think) which *just* uses the core fields. This is probably a separate ticket and possibly separate from ckanext-spatial extension." 1275 1314110016000000 johnglover "Completed in branch feature-1275-solr-search. Up to date with current default but not yet merged into default. John" 1275 1319812967000000 dread Going into release 1.5. 1276 1314886010000000 johnglover "Added to branch feature-1275-solr-search Don't seem to be able to apply weighting to fields at index time using solrpy (see open issue: http://code.google.com/p/solrpy/issues/detail?id=6). So, switched to using the newer Solr edismax query parser and have applied weights to fields at search time. Current weights are as follows: name: 4 title: 4 group: 2 tag: 2 text (everything else): 1" 1276 1315948394000000 rgrp Closing since done (though not yet in main!) 1277 1314868204000000 johnglover Done in branch feature-1275-solr-search for all package searches (which use Solr). Resource and Tag searches still use the old ckan query parser. 1277 1315297186000000 rgrp This just requires merging into main but has been deferred as once solr branch is merged solr is *required* to run CKAN at all (we are thinking of having basic support for sqlite/dbs out of the box ...) 1277 1315948417000000 rgrp Closing since done (though still need to merge into default) 1279 1314303732000000 rgrp Wow, that's a pretty big omission. authz groups are important and are used. I think we should discuss this in next planning meeting. 1279 1314304326000000 rgrp In addition it appears that authz group was left out of the form refactoring so new and edit page use old formalchemy stuff. Should fix this. 1279 1338212028000000 ross Duplicate of #2366 1280 1328786670000000 dread David tells me that this was fixed in CKAN when we moved to SQLAlchemy 0.7 #1433 which went into CKAN 1.5.1. 1281 1314021919000000 dread "Done by pudo in cset:eaf342823caf on default branch, so headed for release 1.4.4. ""Work around the fact that locale changes do not affect the running request and therefore led to an incorrect flashing message. Now manually changing language.""" 1282 1314404540000000 rgrp "Done in cset:4b6838a820bb. Have not done either of the optional items. JS testing for what we have is relatively hard and of dubious value. Re CDN stuff requires work and of doubtful of immediate value since can just configure cache or main site to serve static stuff directly. (Likely more value from compressing js and css ...)." 1283 1326720891000000 dread Since we switched completely to SOLR, this has been true. SOLR doesn't index deleted packages so you can't search them, but they can be read/edited by admins. 1283 1330083933000000 dread This went into CKAN 1.6 1284 1315212258000000 dread "Fixed by rgrp in cset:1952445d2802 on release-v1.4.3 and default. Comment: do not create revision for changes to users as they are not revisioned. * This had already been removed in cset:8d6fde0e2196 (then in user controller) but Adria seems to have reinstated this when refactoring things into the logic layer. This 'bug' explains the huge number of empty revisions we have been seeing on ckan.net." 1284 1315212603000000 dread I've added tests for this (to prevent yet another regression!) in cset:1b77ce8ce560. Closing. 1284 1315214358000000 rgrp I'm not sure this is the sort of thing we need tests for (we can have too many tests!). Something worse discussing ... 1284 1315215626000000 dread Really? How many times do you want to fix this bug again? For the sake of three lines of test... 1285 1314269414000000 dread Another instance is when the database tables are not created and you want to both display a sensible error and have an email alert. #1290 1285 1323173125000000 thejimmyg "I could write some middleware to handle this. Can we come up with a list of exceptions not to catch or should I create a base exception called ""NoEmailReportTriggeredException()"" which exceptions for this purpose have to be derived from or which code has to raise? " 1285 1323178678000000 dread Isn't the point is we don't to raise an exception at all - we want to call a method to send the email and then return from the controller a normal html page. Don't we just need a method to get the error email address from the config and send an email. 1286 1342436420000000 dread This can happen now, if it hasn't already. 1289 1317315211000000 dread Discussions have not resolved this either way. Decided to leave it like it is for now. 1290 1314270894000000 dread "Improved on option 2. For requests to Home controller, the first thing it does is check the user. I catch SQL errors that are to do with missing tables and that now produces a template-less error saying the site is off-line and then mentions database initialisation. Done in cset:484e50c4f989 on default for ckan 1.4.4." 1291 1315948475000000 rgrp @kindly: is this not done? (I know there is more work to be done here but though table creation was done ...). If not need to move to next milestone so we can close this one. 1292 1315999202000000 dread
i'm def +1 for removing API stuff entirely from i18n for the present 1292 1320145677000000 dread " Review process of releases and updating strings (pot) and translations (po) files c.f. http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html http://wiki.ckan.net/Release_process updated with basic process Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?) Welcome message is translated in all languages. Most language have upped their percentage translated significantly and Swedish and Polish added as well. Review long list of strings for improvement Consolidate any? Get rid of i18n for strings in api About 50% of the strings are templates, 30% are in the logic layer and 20% are in the controllers. Removed a handful from ckan/lib/base.py that were only displayed in the api, but the logic layer ones are generally errors that can be displayed in the Web UI as well as the API, so not ideal to remove. Try and remove javascript Tom did this. Add in extensions This could be done. Have raised ticket #1292 for this, next release. #1374 Fix switching to en from other default language Done #1417 Fix browser language detection Done" 1292 1320145732000000 dread These fixes all went into release 1.5. 1293 1314701675000000 johnglover "== Changes == * All templates * Routing: /package/ -> /dataset/ (including API) * Tests: h.url_for(controller='dataset') rather than package ... * One test to check redirect works? * Docs == Leave == * Stay with Package in code e.g. stay with 'PackageController' * also leave stuff like pkg = .... (gradual migration) * Leave extensions == Notes == * Package -> Dataset * Data Package -> Dataset * respect capitalization" 1293 1314905799000000 johnglover "Branch: feature-1293-rename-package-to-dataset Routing and test editing should be finished (all tests passings). Have not added test for redirect yet." 1293 1315214527000000 rgrp Reassign for completion (templates and docs) as john is away. 1293 1315246628000000 zephod "cset:https://bitbucket.org/okfn/ckan/changeset/4a98589af6c1 Fixed" 1294 1315948112000000 rgrp Removing #1311 as that is not directly related to dataset edit / create. 1294 1315948257000000 rgrp Done with merge into default in cset:8bb0720a2150 and deployment to http://thedatahub.org/ 1295 1315820564000000 rgrp This is mostly done but some final tweaks and name disabling to be done. See work in feature-1294-ux-improvements-dataset 1295 1315948009000000 rgrp Completed with merge into default in cset:8bb0720a2150 1296 1315403275000000 zephod No longer doing inline resource editing on dataset/view; instead use the edit form. 1296 1315821293000000 rgrp Have now done resource add pretty effectively and most of edit done (should complete today). 1296 1315948027000000 rgrp Completed with merge into default in cset:8bb0720a2150 1297 1315403878000000 zephod No longer doing inline editing on the view page. 1297 1315403974000000 zephod Largely done in cset:8901e722b158 1297 1315404322000000 zephod all done: cset:71096cd7b27d 1298 1321199769000000 seanh I've been working on this in this branch: https://github.com/seanh/ckan/commits/feature-1298-activities-table 1298 1323106307000000 seanh See the new super ticket and its branch: #1515 1298 1323710593000000 dread Sean says: This is done on the activity stream branch. Just would like David Raznick to review it. 1298 1324384191000000 seanh "David, Ignore my link above to a branch on my ckan fork. Now that I have permissions I'm pushing my branches to the okfn ckan repo on github. My super branch for the activity streams feature is '''feature-1515-activity-streams''' on the okfn ckan repo on github. This page comparing my branch to master is particularly useful: https://github.com/okfn/ckan/compare/master...feature-1515-activity-streams (click on the Files Changed tab) For reviewing this ticket, the relevant changes to review are: * ckan/lib/activity.py, all of it * ckan/model/activity.py, all of it * My changes to ckan/model/meta.py * My changes to ckan/model/package.py * My changes to ckan/model/resource.py * ckan/tests/models/test_activity.py, all of it The other changes on my branch are for other activity streams tickets that follow this one. The super ticket #1515 has an overview of it all. " 1299 1325866115000000 rgrp No estimate as yet? (Or is it in super ticket? (Link?)). Also isn't this a duplicate of #1510 (or vice-versa)? 1300 1315906283000000 rgrp page_heading and optional_footer done in #1108 1300 1315908961000000 rgrp Move of js to optional_footer done in cset:dcb21f0d976f. As part of #1294 have done a lot of css adding. Not sure how much more to do here. 1300 1319794520000000 rgrp Side bar menu is switchable (though not sure how -- zephod has done this!) CSS tidying and labelling of divs done in #1380. So closing! 1303 1314978842000000 dread Adria fixed this in cset:7637d8694388 on default. I transplanted it to release-1.4.3 and have updated thedatahub.org. 1305 1314971164000000 nils.toedtmann "Always use ""localhost"" as mail relay. On all servers, the local MTA should be configured correctly (e.g. in this case to relay via mail.fry-it.com), also to be able to send rootmail. I fixed it on IATI (s049.okserver.org = ex eu23) and thedatahub.org (s055.okserver.org). Please verify that it's working now." 1305 1315317033000000 nils.toedtmann [15:22:02] Adrià Mercader: Yes it looks like it works on both thedatahub.org and iatiregistry.org 1306 1319797204000000 dread These are expressed as functional API tests - see tests/functional/api/test_action.py. There's not a lot of difference (apart from Paste coming in there) so no real need to duplicate API and logic layer tests, or to switch them to talk directly to the logic layer. 1307 1315246706000000 zephod cset:2c31fe24e1c9 1307 1315247298000000 zephod "Replying to [comment:2 zephod]: Oops: cset:cbd6677ec41e " 1309 1315586335000000 dread Duplicate of #1318 1310 1315820419000000 dread This was fixed cset:e49781cb74fd for 1.4.3. 1312 1315327743000000 dread "The default Content-Type is ""application/x-www-form-urlencoded"" (that is what curl sends, and what CKAN defaults to). Therefore these characters need url encoding (""=&;"" -> ""%3D%26%3B""). Here is an example of it working: {{{ (pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test7"", ""title"": ""Test2 %3D%26%3B""}' -H ""Authorization:tester"" HTTP/1.0 201 Created Server: PasteWSGIServer/0.5 Python/2.6.5 Date: Tue, 06 Sep 2011 16:36:32 GMT Pragma: no-cache Cache-Control: no-cache Location: http://localhost:5000/api/rest/package/5866a0f5-905a-44ff-91c3-c1489374a73e Content-Type: application/json;charset=utf-8 Content-Length: 571 {""maintainer"": null, ""maintainer_email"": null, ""id"": ""5866a0f5-905a-44ff-91c3-c1489374a73e"", ""metadata_created"": ""2011-09-06T14:06:40.735488"", ""relationships"": [], ""metadata_modified"": ""2011-09-06T16:36:32.543461"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [], ""groups"": [], ""name"": ""test7"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""notes"": null, ""title"": ""Test2 =&;"", ""ratings_average"": null, ""extras"": {}, ""ratings_count"": 0, ""revision_id"": ""92ee0b2e-c5e4-47b9-b00c-b8e29662b950""} }}} Alternatively you can set the content type to ""application/json"" and you don't need to URL encode. e.g.: {{{ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test5"", ""title"": ""Test &""}' -H ""Authorization:tester"" -H ""Content-Type: application/json;charset=utf-8"" }}} To help work out these problems with encoding in the future, I've improved the error message to show the decoded JSON string. cset:55273629a552 which is headed for CKAN 1.5 release." 1313 1317076432000000 rgrp "Error was due to maintenance. Need both config options for recaptcha to work so either one can be used to check config. Re docs see #1358 (auto generation of docs on config from the config source). Given that only docs remain and that is part of #1358 am closing as duplicate." 1315 1315414218000000 dread Pretty good start added in cset:7a0e7a486df3 on default, aiming for release 1.5. Just needs some loose ends tidying up. 1315 1320173768000000 dread Tidied up in cset:39daf1afd64a for release 1.5. 1316 1338193724000000 kindly now get_or_bust function handles this. 1317 1315416649000000 dread "Ok, I think this example may be when searching for ""ira"". ""irina"" works me, but ""ira"" gives: {{{""ira"" matched several users}}}" 1317 1315822670000000 rgrp This is very minor IMO. I would defer this as part of a more general user system ux refactor. 1317 1325473762000000 rgrp Moving out v1.6 as minor. 1318 1315569362000000 dread "Problem tracked down to Adria's cset:4f1ca1eb2630 on branch feature-1229-db-out-of-controllers that missed out repoze.who in the redirects after login, when refactoring the user registration. This code went into default on 2011-07-28 for release 1.4.3. Fixed in cset:93731cd3b22e for release 1.4.3 and doesn't affect previous releases. Tests for user registration added too." 1319 1315476057000000 shevski Ideally we should also verify people's email addresses. Do we think that's overkill? 1319 1315904592000000 dread "No responses received about this on ckan-discuss, so I'm taking that as no objections to requiring email address on registration. I think that verifying email address is not worth it in this case, because the value of supplying an email address is only for resetting a password. The chance of needing a password reset AND having inputted an incorrect email address is very low, so I don't think it justifies everyone having to do this extra step. If we used email address more then that may change." 1319 1318426807000000 rgrp @dread: hope it is OK to re-assign to me but this is very similar to #1386 (and is a su bticket of #1343) 1319 1318527613000000 rgrp Fixed in https://bitbucket.org/okfn/ckan/changeset/130cad62163c 1319 1324036947000000 dread Goes into release 1.5.1 1320 1317391018000000 zephod Extend misc.py to regex find http and https, surrounding it with <>. ALSO note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package). 1320 1317730999000000 zephod "Slightly painful when links appear inside naughty HTML tags or in markdown tags. Restriction: Links are auto-linked if they appear at the start of a line or after whitespace. cset:f9dfb0506594" 1320 1318164603000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/f9dfb0506594 1321 1319558004000000 dread "We should not cache if there are flash messages to show (otherwise they get queued up for the next non-cached page view). Fixed in cset:945e2a6ce74f for release 1.5" 1323 1315594333000000 dread "thedatahub has a couple of auth groups with no name. The form allows it! Fixed in cset:bebd0a46ebc6 * form requires the name field * existing auth groups without name can be viewed/edited if necessary (referring by ID) This has probably been a problem since auth groups were added to CKAN." 1324 1315948336000000 rgrp Done and merged into main at cset:8bb0720a2150 (and deployed!) 1325 1315587113000000 dread Fixed in cset:0dd4d4457c97 for release 1.4.3. Previous Python 2.5 incompatible versions include 1.4.2 - 1.3.4.1 and maybe further. 1327 1320662446000000 rgrp Duplicate of #1397 1328 1340191065000000 ross Let's see if this is still an issue in 1.8 1329 1316624026000000 zephod "I've tried to do this wherever I could as I looked through ckan.pot for broken pieces of string. Generally it triggers errors unless things are very neatly structured (no nested links or ${a.variable} accesses, it seems) but things are quite a lot tidier. cset:d5bf4c7ba293 Does that count as 'fixed', or is this a permenant issue?" 1329 1317118323000000 dread I'll take a look to see if I can do any more and then will close it. 1329 1317404566000000 dread "POT looks far better. I couldn't see any outstanding problems. This has gone into ckan 1.5." 1330 1327060201000000 dread "This is simply incorrect. test_authz.py (I assume you mean the one in ckan/tests/functional?) tests that authz works all across the Web UI and API in enough detail. The 'total site lockdown' test - perhaps you mean the TestLockedDownViaRoles class which makes up a small part of test_authz? It was a test added due to an untested use case implemented in IATI, DataGM etc. These are currently active so still need testing. Regarding refactoring the tests to be ""proper"" then please open a new ticket, providing references to why it is not ""proper"" to test by searching for text in a web response." 1331 1325355631000000 dread This appears to be fixed in 1.5, 1.5.1 and 1.5.2a. 1332 1315999104000000 dread "Fixed in cset:860b357a2aec for release 1.4.3 and updated no.ckan.net. This problem was due to {{{[1:authorization group]}}} being translated as {{{[autoriseringsgruppe]}}} and I corrected it to {{{[1:autoriseringsgruppe]}}}. This word has a bold tag, hence the brackets and the number is used to identify which bracketed word is which (when you have more than one). It is a shame that Transifex doesn't draw attention to this problem and Babel doesn't spot the error on compilation." 1334 1316086680000000 dread cset:b01de680bf86 copes with the problem. Now to find out why revision.packages has some None values - very odd! 1334 1319798961000000 dread I think this has been a problem since 1.3.3. Fixed in v1.5. 1335 1315910117000000 dread "Done to a basic extent in branch feature-1211-drupal for release 1.4.3. Further tickets: * #1322: Action API improvements * #1315: Document /api/action" 1338 1316951545000000 amercader Reopening, because another change is needed to support custom schemas 1338 1317117738000000 amercader Fixed in eca1edce3a0f 1339 1316010607000000 kindly "I have fixed the isodata and made a slightly modified int_converter for this case. In the correct place and raises invalids on not being an int. cset:a4af115116bb The thinking was that the input of these fields would be through the api so the empty string case did not arise. These should clearly be converted to None (Null). What issues in general? Having done this lots of times before you always end up needing to write your own little validators as the standard ones never do what you want. Thats the point of them. Look in ckan/lib/validators if you need examples. So what you did was correct..." 1340 1315991534000000 dread Fixed in default in cset:33856d3ce84e aimed at release 1.5 1341 1315996368000000 kindly "This was run to delete the users and their mistaken revisions that where created. {{{ BEGIN; delete from revision r where r.id in ( select r.id from ""user"" join revision r on r.author = ""user"".name left join resource_revision rr on rr.revision_id = r.id left join package_revision pr on pr.revision_id = r.id left join group_revision gr on gr.revision_id = r.id where ""user"".created between '2011-08-15' and '2011-09-06' and gr.id is null and rr.id is null and pr.id is null and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%') and ""user"".name not like 'http%' ); delete from ""user"" u where u.id in ( select ""user"".id from ""user"" left join revision r on r.author = ""user"".name where r.id is null and ""user"".created between '2011-08-15' and '2011-09-06' and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%') and ""user"".name not like 'http%' ); COMMIT; }}} " 1341 1319649573000000 dread The majority of our 6000 users are still spammers. Can we simply elimenate those who've not made any revisions? Even if we take a few genuine users with it, these are pretty dead accounts anyway. 1341 1319709997000000 dread Reopening - this is still a serious problem. We can't link to the user page because it is so embarrassing: http://thedatahub.org/user 1343 1316078440000000 dread I don't think email confirmation is necessary - see discussion at #1319 1343 1318528138000000 rgrp Remove from ckan-v1.5 since remaining non-complete items (user page improvements and modal login won't get done in v1.5. 1344 1316078757000000 kindly fixed cset:f08215845dab 1345 1337961353000000 nils.toedtmann This becomes an actual problem on the CKAN farm s057, see http://trac.okfn.org/ticket/1245 1345 1338551926000000 nils.toedtmann See #2485 for workarounds containing the leak(s). 1345 1340190737000000 ross Moved out of milestone so that it gets looked at again soon 1345 1343215489000000 amercader See https://github.com/okfn/ckan/pull/73 1346 1316086314000000 dread "Fixed in cset:df299e69d456 on default for release 1.5. Problem exists on releases 1.4.3 - 1.3.1 and probably earlier too." 1346 1316094632000000 dread Also Transposed to 1.4.3.1. 1347 1316094608000000 dread Fixed on cset:7c3cc0fd6ae4 on branch release-v1.4.3.1. 1349 1316277490000000 zephod Done. cset:e074aee19923 1350 1316624288000000 zephod "Was going to reimplement an old workaround described by Friedrich, where a language.js file is served through the home controller (with appropriate settings in routing.py) and so ${_('string literals')} will be translated. See: * https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/controllers/home.py#cl-48 * https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/templates/home/language.js However the string-extraction mechanism seems geared towards Genshi templates and hacks are required to make it read a javascript file neatly (ie. without putting the *entire file* into a string for translation). A more elegant solution seems to be using an xi:include to pull in a genshi template which contains an html }}} where the jquery.html5.preie9fix.js is simply the [gist](https://gist.github.com/887560) published by Akkuma in a [jquery-tmpl issue (#36)](https://github.com/jquery/jquery-tmpl/issues/36#issuecomment-918495). This script patches JQuery on the fly on IE < 9 to rework in memory html 5 blocks handled by JQuery, so that it doesn't break. And this fixes the ""Upload a File"" button on IE. I didn't commit/pull this specific fix for the following reasons: * the script embeds innerShiv, which is a previous incarnation of html5shiv (now included in CKAN). It doesn't sound fine to include 2 pieces of code with similar effect, but I couldn't reproduce the fix with html5shiv! * in my environment I have included this fix sitewide even if I didn't spot other problematic features..I don't know if it would be better to include it only on the dataset edit page in the official CKAN code. Summary: I leave to you guys the choice on how to deal with this problem." 1788 1330624155000000 zydio "Erh...sorry for some formatting errors in my comment. I used markdown formatting for links, and used the #NUMBER format to cite issues, which obviously is interpreted by Trac as references to his own issues...my bad." 1788 1330644056000000 rgrp Re the Upload file button not sure what's best. We do have plans to replace jquery-tmpl with mustache in the near future (but not clear exactly *how* near). We'll think about this and get back to you. Thanks for the detailed debug report. 1788 1333133365000000 zephod "I've been performing a full sweep of the site to rebuild the markup & stylesheets using Bootstrap, and in doing so have managed to close off this ticket. Bootstrap allows us to delegate lots of clever functionality and layouts to Twitter's tried-and-tested code, making it easier to be compatible with IE7. It has brought up plenty of issues: Lots of Javascript was broken; plenty of pages generate broken doctypes causing the browser to go into quirks mode (which in turn causes the page to explode all over the screen); most dynamic content would fail to generate properly in one way or another.... (see the #1788 branch for a log of issues encountered). I've been refactoring our JS and have managed to strip ckanjs, too, because the only significant element we'd used was the File Upload view (which also now works in IE7). test.ckan.net has been redeployed on master; I'm browsing the site in IE7 and everything is looking great :-)" 1791 1329138315000000 dread Fixed in [release-v1.6 beaeaed] 1792 1329483249000000 toby package create update use correct schema in get_action 1792 1332755148000000 toby closing as done all that I was asked to do as part of this ticket 1792 1332872069000000 dread "What is the changeset for this? There is evidence of a branch ""enhancement-1792-api-logic"" but perhaps this name got fast-forwarded out through a bad merge? Is this merged into 1.6.1 release?" 1792 1332928847000000 toby This is in the 1.6.1 as far as I'm aware. The branch was deleted after the merge into master 1792 1333037710000000 dread "I have tracked down the changesets to: * branched off master at dff9f7a8 * merged master in for the last time (and became the 'new master'?) at 1a9227d9ff73c " 1796 1339586499000000 ross LXML removed, update requirements on the extensions 1797 1330550304000000 rgrp New API endpoint, Authorization and documentation done and merged to master. Time so far. 1.5d. https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24 1797 1330863639000000 rgrp Data Viewer support for new DataStore in https://github.com/okfn/ckan/commit/9ab8b0283bb086eb4cd663ff73c27066bdd3c79a 1797 1331412469000000 rgrp "All done!! Help for data api done in https://github.com/okfn/ckan/commit/1d8e464f8542d4c33286bb93f4de50060665799f Checkbox for datastore enabled in https://github.com/okfn/ckan/commit/3f1320cd92ae0e775fde1b5eada156260c55e0a6" 1797 1331412644000000 rgrp "Merge fix https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547 " 1798 1329395560000000 dread "Was fixed by Ian in https://github.com/okfn/ckan/commit/5a9054459e3833443bed3e118bbbb6c442e55b0b on branch feature-1453-flexible-tag-names. This has gone into CKAN 1.6." 1799 1330000389000000 dread Fixed in [master ea2d824] for both logging in and registering whilst logged in. Cherry picked to release-v1.6 1801 1343144718000000 amercader Closing as both old and new theme now show password reset links 1802 1329409718000000 dread Current progress - I have got most of the dependencies installed - many by pip, but some I had to revert to Windows msi installers. Pad with details here: http://ckan.okfnpad.org/windows. So far done: 1 day. 1802 1329490081000000 dread "I've achieved CKAN running with paster and without SOLR (using simple_search). Running locally on an old XP machine, performance seems fine - no noticeable issues. I installed most things into a pyenv using pip. pyutilib, psycopg2 and lxml were slightly problematic, but I have workarounds. I had problems pip and svn and the Cygwin window, so reverted to DOS prompt. I still had the Cygwin tools installed though, so next step is to try it without these at all - they are a big overhead." 1802 1329734716000000 dread Spent so far 1.75d 1802 1329830097000000 dread "* Refined install without using Cygwin - this seems a little easier. * Added example Apache & modwsgi deployment. * Documented extensively at: http://wiki.ckan.org/CKAN_install_on_Windows * Have not tried SOLR on Windows - I'm not convinced this is high priority, and normal install should be fine as it is so separate from CKAN. Spent another 0.5d. Total spent 2.25d" 1804 1329491156000000 toby This should be fixed in branch feature-1653-i18n_url_rewriting as we use different logic. However it may affect other return_to calls used elsewhere like adding a new package. 1804 1330000705000000 dread I'll fix this on release-v1.6 and then reassign to Toby to fix on feature-1653-i18n_url_rewriting too if the same issue crops up there. 1804 1330037202000000 dread "Fixed this on release-v1.6 in cset 557e72330db7. Spent 0.5h. Reassigning for Toby to look at branch feature-1653-i18n_url_rewriting " 1804 1330347185000000 toby This has been fixed in master for a while and was only a 1.6 issue 1805 1329760795000000 toby this is fixed in branch feature-1653-i18n_url_rewriting 1806 1330942356000000 toby version one completed 1806 1331295310000000 toby closing as initial work has been done if we need to amend then create a new ticket 1807 1330809641000000 seanh "Implemented the recently-changed-datasets activity stream on branch feature-1807-recently-changed-datasets in CKAN and merged it into master (but the default templates don't make use of it). Added the recently-changed-datasets activity stream to the front page template on branch master in ckanext-ecportal." 1809 1330528828000000 johnglover Caught exceptions in QA instead, error message stored as openness_score_reason 1810 1329918479000000 johnglover Deferred for now, moving to backlog. 1810 1335874745000000 johnglover Not using eurovoc tags. 1811 1329918517000000 johnglover Deferred for now, moving to backlog 1811 1335874765000000 johnglover Not using eurovoc tags. 1812 1335874864000000 johnglover Ross has fixed this via group authentication. 1816 1330863079000000 rgrp "As per detailed comments in http://wiki.ckan.org/Proposals#Apps_in_CKAN I think Apps is a confusing name. What we are talking about here is the ""Related Stuff"" item mentioned in that ticket. Rather than update ckanext-apps I suggest we do a rewrite. I've created a detailed super ticket at #2204 and assigned to you Adria for review and thoughts. (While I understand #2204 may involve more work which we maybe cannot do before relaunch I'd still recommend starting on that rather than spending time upgrading this extension -- if we have to disable it for relaunch I don't think that is a big deal ...)" 1816 1331302835000000 amercader "+1 to a new, more flexible extension. But in the meantime I've just spent a couple of hours making it work with latest CKAN, which was easier than expected, so we can deploy it with the new version of PDEU." 1819 1332163324000000 kindly Currently using package_show_rest. Should be moved to just use package_show but that is another ticket. 1828 1330525412000000 rgrp Fixed in https://github.com/okfn/ckan/commit/e6da48d9337575a83bc6a30896e68d1081b43847 1829 1330001828000000 dread "This was due to https://github.com/okfn/ckan/commit/f07bfdb9#diff-1 - string ""Language has been set to: English"" shouldn't have been translated. I've put in a patch to make the code intentions clearer and added a test: [release-v1.6 07d3a02]. This only affected the 1.6 beta, and is fixed for the release." 1830 1330089912000000 dread "Now if you set openid_enabled to false, you are not given the option in user settings to associate your account with openid. Cset: [master b295bde] I looked at removing the openid middleware, but we use it for logging out users, whether they logged in or not - it looks like friendly form plugin is not as good as the openid one in this respect! We could hack the logout function into the friendly form to sort this out, but not sure the effort is worth it at the moment." 1831 1330956631000000 ross Already have a branch for this. 1831 1331549284000000 ross Email is not currently 'unique' in the user model. Find out if this is intentional or not. 1831 1331655458000000 ross Decided to delay this until later but code is in feature-1831-login-by-email 2198 1339771453000000 kindly Already in the docs. 2201 1330765400000000 rgrp Duplicate of #1171. 2202 1338204729000000 toby these are available through the pkg_dict 2202 1338204801000000 toby "re-reading this it's slightly different re-opening" 2203 1348568961000000 dominik There is work on this in the dataviewer-iframe branch. 2204 1332341044000000 toby needs discussion - planned for 23-3-2012 2204 1332341080000000 toby http://ckan.okfnpad.org/feature-2204-related-stuff 2204 1332843232000000 toby still awaiting discussion 2204 1334077936000000 rgrp "Important comment: not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ... Also (@icmurray): does this interact with our desire to do embedding?" 2204 1334218735000000 ross "I'd be happier adding this to another ticket rather than moving the target for this ticket half way through so have created #2284 Should only be a short ticket in the next sprint." 2204 1335287636000000 aron.carroll "Making some notes of front-end changes that I think need to be implemented sooooon. {{{ Implement better truncation (the expanded text should pop out on hover rather than expand the entire row) Show the type of related item (an icon would be good here) Test in IE Make the form work without JavaScript by accepting x-www-form-urlencoded. Add classes to first item in a row to clear the row (currently uses :nth-child CSS) Touch gestures and re-ordering (David's request) }}}" 2204 1335795665000000 ross Giving this ticket to Aron to fix the front end 2204 1337582843000000 rgrp "Surely this ticket was closed 3w ago? @Ross: can you close ..." 2204 1337594723000000 ross Was waiting for documentation, can close now. 2206 1331046486000000 johnglover "Close enough for now. Still have to restyle language drop-down to match template, but will fix when updating the list of languages. Also still to decide on menu styles/locations (for CKAN login and main menu, and ODP main menu)." 2208 1332251880000000 seanh "I've attached a ckan.pot file. To make it, I edited setup.py in CKAN core like this: https://gist.github.com/2135849 and then ran python setup.py extract_messages. (And I had ckanext-ecportal installed at ~/pyenv/src/ckanext-ecportal.) I guess I may need to run this again when the ckanext-ecportal source is finalised, so leaving this ticket open for now." 2209 1331547639000000 ross Option 4 is the most appealing at this point, although you probably should look at #2209 2209 1331547674000000 ross "Previous comment should read: Option 4 is the most appealing at this point, although you probably should look at #1649 " 2209 1331551170000000 ross "Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template. Pros: Easy to implement Easy to use Not hard-coded as currently Fast execution Cons: Requires knowledge of required RDF output if default is not useful RDF and not any of the other formats yet. Only works with package/resource/tags unless more work is done " 2209 1331551393000000 ross "Can't edit comments .. so Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template. Pros: Easy to implement Easy to use Not hard-coded as currently Fast execution Cons: Requires knowledge of required RDF output if default is not useful RDF and not any of the other formats yet. Only works with package/resource/tags unless more work is done " 2210 1331549350000000 ross This is now done through the logic layer member_create/member_list/member_delete 2212 1331719868000000 johnglover Added as a new paster command in ckanext-ecportal (run 'paster ecportal' for a list of commands and arguments). 2212 1331720191000000 johnglover Deployed on test server, where it imported 4087 datasets. A small number of datasets were not created as they failed CKAN validation - most of which had strange values such as 9999 and 0 for date fields (some also didn't have unique names). 2220 1331291241000000 toby "issues: 1) want to cache static files long term - need to add a ?ver=xxx to allow changes - this will need a change to all templates etc - could be seen as a good thing 2) we can only purge specific pages not whole cache so we need to know all the urls which want to be invalidated - including language variants 3) urls should be consistent so we should use id not name for datasets etc 4) for now we can just use short expiry times as this prevents some of these problems" 2220 1332950550000000 toby "Implementation of this has been carried out at the page level. ckan.page_cache_enabled is config option to enable it requires redis need to document it somewhere " 2221 1331561095000000 ross Fixed in https://github.com/okfn/ckan/compare/1eeda11...ad2358a 2223 1331558791000000 rgrp "Have introduced boostrap js and css on merge of #1797 https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547 However need to remove blueprint and make sure we keep our original theme." 2223 1332246030000000 zephod "I have been working overtime to try and get this change straightened out as it's suddenly on the critical path for something. Have a look here: https://github.com/okfn/ckan/tree/feature-2223-bootstrap It's 80% of the way there; some form CSS needs overriding because Bootstrap tends to make things very large so they spill over. But on the whole the markup is in a steady state, which is what we need." 2223 1332842692000000 zephod Merged into master here: https://github.com/okfn/ckan/commit/0bf4b33955df3806785f9b42a55acaeb45596e25 2224 1332475944000000 rgrp Autocomplete based on bootstrap looks pretty non-trivial ... 2224 1338205358000000 ross Being done by Aron 2225 1343220502000000 aron.carroll There is a jshint.json config file that all JS in the new base theme conforms to. 2226 1332159640000000 seanh These deployment docs seem worth moving into sphinx as well: http://wiki.ckan.org/Deployment 2226 1332475468000000 rgrp All done (as of Tuesday in fact!). No doubt further work to do but that's a new ticket! 2227 1332843126000000 toby being done in other tickets 2232 1332331660000000 amercader "Fixed in 6b4c6c9 This was due to some characters not being supported by XML. We now strip them before indexing." 2234 1337237687000000 rgrp This looks interesting but is this a priority :-) (also no estimate as yet :-) ) 2234 1337239588000000 ross Yes, definitely need a replacement for wordpresser and sooner rather than later. Doing some experimental noodling to work out an estimate. 2236 1332153688000000 rgrp Done. See https://github.com/okfn/ckan/commit/0d970d3a0e523a1695830e8c8c28c0ba10a8d8b9 2238 1343392681000000 seanh Ticket #2771 encompasses this and related changes. Vocabs probably won't be deployed to test.ckan.net but a working example extension using vocabs will be added to core instead. Closing this ticket as a dup 2239 1334581838000000 ross Changes to schema/rdf output 2240 1332408769000000 ross Distributed to ckan-coord 2242 1333375436000000 seanh Conclusion: jenkins should work, may need some jenkins plugin magic to get it to show what branch each build was run on. 2250 1332680690000000 rgrp Is this now how we are doing it? I thought we were integrating with RDF store (so it was more about sorting out the data viewer / recline stuff)? 2251 1332331650000000 rgrp Big +1: everyone wants to know page views. Would like detail of how this goes into interface. Downloads already being tracked. Also isn't this just an extension to ckanext-googleanalytics. 2251 1332332390000000 toby "not really part of ckanext-ga as we need to sit first in the middleware for speed reasons. I'm looking at recording the details and will do a summary page like ga extension. As far as other uses like showing popular resources etc then we need to decide where the data will live - on the resource or separate. Downloads tracked? where is this done in the code/data stored" 2251 1332340874000000 toby "notes from meeting 21-3-2012 US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded. US1a Next to a resource or a dataset see how often it has been downloaded / viewed US1b I want to see datasets or resources ranked by most downloaded or viewed US1c See a trend graph for a dataset (and resources) Adminstrative Dashboard (?) * I want to see the traffic breakdown by country to my site ... * Ditto for browser type, language, etc etc * I want to see it graphed over time ... == Implementation Details == 1. How do we store this data in CKAN? 2. How do we track (and store)? 3. How do we display * Config option ckan.status.enabled = False (by default) === Storing Data === How does ckanext-googleanalytics do this? Current table: package_id | count_recent | count_total Move to a new stats_summary table id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value Do we store this data into the search (solr) so we can search by it? === Displaying Data === * Helper functions / dictize: * Helper function: h.stats_get(object_type, id, stats_type) * h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ... * Change to dictize * Location in the default theme (do we show for example in search results too!) * Support for ranking by most popular in search? === Tracking Data === * Our own solution (just write to site_tracking) * Google analytics (plus extension for retrieving data) <- would need a refactor * Piwiki ==== Own Solution ==== site_tracking table id | url | timestamp | action (page_view, resource_download) | * Make javascript to make request to ckan to store clicks and page views. * Add middleware so these requests do not go through pylons and just store data quickly. " 2251 1332340947000000 toby "TODO look at 1. How do we store this data in CKAN? 3. How do we display * Config option ckan.status.enabled = False (by default) " 2251 1332524123000000 rgrp Update description in great detail. 2251 1332776926000000 toby "we want resource show downloaded/viewed on resources show on the dataset total/recent" 2251 1333535572000000 toby "we have various options on tracking unique users seems the sane approach. * unique daily view - needed for nice graphing * unique total views * total unique daily views - higher numbers and feels more right - easier to calculate if original data is lost/or archived for now will collect all and then decide which to use for display as gives us flexibility if needed display ------- why is group listing of datasets inconsistent with main search (TDH at least)? should we make these consistent? group listing allows downloading of resource without tracking (this should be fixed) if we make listings uniform do we want the download links or not? I don't like the term download when it is just a link (maybe that's just me - what would be better?)" 2251 1333644538000000 toby "most work completed: outstanding issues * add a prune tracking_raw data to cli * add indexes to tracking_raw * add summary page for new data format * add api counter 1) do we record the url or action etc (action seems more sensible) 2) add in __call__ or in action? " 2251 1334074772000000 toby api counter moved to ticket #2282 as needs better specification 2251 1334144999000000 toby "For meeting today with Adria * How to add tracking to solr index. Can the data live in tracking_summary? Do we need any extra indexes on table to help indexing? * How will we add order by popularity to search? * Can we add resources too? * Will search be best way to find most popular packages/resources or should we get that pages data from tracking_summary?" 2252 1333108787000000 amercader "There is a new ticket for the same schema as forms issue: #2268 The exception was fixed on [https://github.com/okfn/ckanext-inspire/commit/0dff1f6 0dff1f6]" 2252 1333374836000000 amercader This needs more work on ckanext-dgu and testing. 2252 1334567495000000 amercader Fixed on 0dff1f6c2e 2253 1340038490000000 toby closing as nothing has ever happened with this ticket 2254 1332343198000000 toby "What do we want to be able to do? what do we use the stream stuff for at the moment? la la la" 2254 1332950678000000 toby "plan of attack a) get removal of TDH from core - this will give a better idea of the issues involved b) have a think" 2254 1338210316000000 toby "closing. we have fanstatic integration in ckan.demo work tdh is it's own extension h.snippet is in 1.7 ITemplateHelpers in 1.7 This allows most of the issues resolved. Full removal of the IGenshiStream filter still neads some form of in template hooks but this is being looked at as a possiblity in ckan demo and can be considered out of scope for this ticket closing" 2255 1332752936000000 rgrp "I would say Public is default for Organization with Private being an option (or do we make this a config option?). What about ownership by users? > New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below) Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page." 2255 1332772340000000 ross "> What about ownership by users? Good question. WIll add to description. > > New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below) > Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page. The former, although currently we will have a slightly different form in the organizations extension, until such a time as we're ready to replace the existing one. Tried merging the two forms but it ended up with such a mess of logic that it was unmanageable. " 2255 1332934041000000 ross Changed description based on docs of stuff to verify. 2255 1334582784000000 ross Review + restrict resource access (when private) 2256 1333620795000000 toby "I'd like to remove all 'custom' icons and just use bootstrap ones - is this reasonable? they are added in different ways but h.icon() could be easily adapted for this. minimal .sj .css files" 2256 1333720237000000 rgrp Not sure what ticket means (remove TDH from core?). Also agree that we should glyphicons from bootstrap where possible going forward - though that would seem to fit more naturally with #2224 (Simplify javascript and css dependencies and add minified version) and have little to do with TDH versus core :-) 2256 1334051526000000 toby "@Rufus, The idea of this ticket is to separate TDH into an extension rather than it being incorporated into core. This should help TDH by making it possible to push fixes without needing to do a new release of core. It should also help make core more stable and as slim line as possible. New features can be tested in TDH and then if they have a wider appeal they can be merged into core. There is a strong desire for this within the dev team." 2256 1334051733000000 rgrp @toby: indeed and I'm strongly in favour of the basic idea too (and largely a dev question so up to you guys though feature / functionality implications would need to be talked through). My point was that custom icon stuff was #2224 :-) 2256 1334052087000000 toby "@Rufus, Thanks, I think there is overlap and I'm happy to leave that bit of work to you (or future ticket owner) I was just adding some brief notes to this ticket on some of the outcomes I'd like to see and hence why I'd be happy to simplify the icons." 2256 1336046098000000 toby I'm closing this ticket now ckanext-datahub exists and is live - the other work is included in other tickets 2257 1332950416000000 toby pushed first round of changes into master 2257 1336731023000000 toby ckan.restrict_template_vars defaults to true for 1.8 release 2260 1333373978000000 johnglover Still some work to do on this, template needs to be finalised. 2260 1333554069000000 johnglover Finished for this iteration, waiting for design work to be completed before making further changes. 2261 1332838253000000 toby "new config option ckan.root_path = /path/from/root/with/{{LANG}}/substitution " 2261 1332840056000000 toby completed plus documentation updated 2261 1332840112000000 toby please test logging in/out as the redirect was not testable but should be correct 2264 1333635637000000 seanh "Many strings have been fixed (and committed on master). Still to go: When uploading a file: - The ""Browse..."" button. This comes from the user's browser? - Hangs when I try to upload a file, so the rest of this UI is unchecked. The licenses on the dataset add, edit and view pages, ""License Not Specified"" etc. Dataset state on dataset view page, e.g. ""active"". When adding or editing resources: - The resource types (Data File, etc.) The strings seem to come from resource.resource_type. - ""You can use Markdown formatting here"", ""Should a DataStore table and Data API be enabled for this resource?"", ""Dates are in ISO Format — eg. 2012-12-25 or 2010-05-31T14:30"". I can't get strings with hyperlinks in them to translate here. Dataset view page: - License not specified. Dataset search page: - ""Tags"", ""Res_format"" and ""Groups"" in the sidebar. Leaving because this code has been refactored on feature-1821-multilingual-extension. Fix after that branch has been merged. Dataset view page: - The resource types, e.g. ""plain text"". - The field names under additional information: cache_last_updated etc. Dataset history view: timestamps not localised. Login page: - ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"" This string is marked for translation but I can't seem to get translations to work for flash messages. Also ""Sean Hammond is now logged in"" flash message. Register page: - ""Missing value"" in error message when you try to register and don't give an email. The string seems to come from formencode. stdtrans() not working? User view page: - Activity streams strings. Activity streams genshi function needs a little refactoring to be i18n-friendly. Users page: - ""less than 1 month"". This string comes from Pylons' time_ago_in_words() function which doesn't appear to support i18n. New auth group (and edit auth group) page: - ""Unique identifier for group. 2+ chars, lowercase, using only 'a-z0-9' and '-_'"". This string is marked for translation but for some reason it isn't working. - ""Name: Please enter a value"" (in an error msg) - ""User name"". The string comes from class FormBuilder? Also, need to check that all tooltips are translated!" 2264 1334348070000000 seanh "A few strings are still left to be marked for translation: - http://trac.ckan.org/ticket/2289 - http://trac.ckan.org/ticket/2288 - http://trac.ckan.org/ticket/2287 - http://trac.ckan.org/ticket/2290" 2265 1332865837000000 dread "I started work on this and need to park it somewhere - here will do for now. I was using the 'MLT Component' http://wiki.apache.org/solr/MoreLikeThis : https://gist.github.com/2217684 But I think it would make more sense to use the 'MLT Handler' http://wiki.apache.org/solr/MoreLikeThisHandler which needs enabling by adding this to the config {{{/usr/share/solr/core1/conf/solrconfig.xml}}}: {{{ - @aron styling of active facets with long names breaks. we could make the names shorter in snippet/facet_list.html via truncate or make the selected taggy thing wider" 2419 1337863024000000 toby "@aron the reason for this is that controller/package.py renders the form `package/new_package_form.html` as part of it's work and then makes this available a `c.form` to the template. I'm not sure I like this as an approach but it is how it is currently done. If you split your form into a new template `package/new_package_form.html` then you get all the errors etc. I think for now we should do this and then look at if doing it all in one template pass might be more sensible. reassign to me if you want me to do this else I'll leave it with you" 2420 1337950422000000 toby merged into master 2422 1337945204000000 seanh Closing because I'm not able to reproduce and it turned out floapps had publisher auth on which was overriding his rights auth 2423 1338193199000000 ross Duplicate of #2414 2424 1338193269000000 ross Duplicate of #2413 2428 1341234933000000 seanh This awaiting merge, pull request: https://github.com/okfn/ckan/pull/46 2428 1341314475000000 seanh Merged. The only thing I didn't do was add a test case that runs pip freeze and tests that all the right versions of deps are installed. 2430 1338397815000000 seanh http://trac-hacks.org/wiki/MasterTicketsPlugin 2431 1338397906000000 seanh https://github.com/davglass/github-trac 2432 1338205526000000 aron.carroll "It would be great to use something like http://trac-hacks.org/wiki/SimpleTicketPlugin to remove most of the options when creating a ticket. Suggestions to remove theme, repository, component and type unless they're actually useful." 2432 1338325120000000 shevski "Where do new tickets go to await triaging? Is priority set by whoever creates the ticket? Does this get revised? Will we have a field for dev time estimate or where will this go?" 2433 1337960203000000 toby "the affected actions =============== package_list package_relationship_create revision_show group_list package_update_validate It would be nice to get this fixed and I'm loathed to suggest adding a version 4 hack. If we make this change it will break things for existing users. This behaviour has existed since the action api was added as far as I'm aware So should we a) leave it forever and document b) break things and fix it - when 1.7.1 or 1.8?" 2433 1338397268000000 toby after discussion it has been agreed to keep the existing behaviour and improve documentation as it is likely to break existing applications etc 2434 1341234522000000 seanh "We've moved to AWS hosting with docs generated by Jenkins instead, see: http://trac.okfn.org/ticket/1254 There are still two issues to resolve: 1. Automatically updating the symlink so that docs.ckan.org always redirects to the latest stable version 2. Finding somewhere to put links to all the other versions" 2434 1350296272000000 amercader Seems to be working again, closing. 2437 1338484676000000 seanh "Please add to the file started to doc/coding-standards.rst Also, for the Python coding standards see http://wiki.okfn.org/Coding_Standards as a starting point" 2437 1338487089000000 seanh "I've also added the commit message guidelines as they reference a trac plugin I'm going to get installed. I'll leave the rest to you. I remembered that http://wiki.ckan.org/Main_Page has a bunch of pages under Developer Resources that should probably be updated and merged into these coding standards and deleted from the wiki" 2438 1338212234000000 icmurray "Proposed solution: Add a new q field which can be specified multiple times. Each q parameter is a string: `{field}:term`. `:` characters are escaped in the `{field}` by `\:`. Mark the `fields` parameters as deprecated." 2438 1340736112000000 icmurray "Ready to be merged [1] Brief synopsis of changes: - altered return type of the action: from Resource domain objects to dictized resources. This is required if this action is to be made accessible through the action api as the domain object is not json-serializable. The 'old' return type is still available by passing in a flag in the context. In fact, this is required as the search layer uses this action, and deals with the domain objects directly. - decided that escaping the ':' (as described in above comments) wasn't necessary as resource fields shouldn't have ':' characters in them anyway. - improved action's docstring - 'fields' parameter has been deprecated ( see #2603 ). Use of it by the search layer has been removed in favour of the new 'query' parameter. But it still works, and correctly maintains backward compatibility. [1] https://github.com/okfn/ckan/pull/37" 2439 1338214158000000 icmurray "The `fields` argument doesn't provide any extra functionality that the `query` argument can't provide. 1. Figure out what the reason for this argument is, just to make sure. 2. Deprecate it, and change any internal uses of it to use `query` field instead." 2440 1339421746000000 aron.carroll "Has some styling as of b0153ba but needs content to be applied. Toby, can you pull in the top 10 tags, as well as n of the most ""popular"" datasets and n of the most recent datasets. Actually just providing the functionality as helpers would be great. Do we actually have ""featured"" datasets implemented at the moment? Not sure what we're going to use for the box on the top left when on a new base CKAN instance." 2440 1339423966000000 toby "@aron, tags what functionality do we want for this? I can just reuse the facet_list snippet but then we have all sorts of interactivity. Do we want this more a static list of tags (linked to somewhere but no add remove stuff)? seems more sensible I think featured datasets I've no idea I'll try to discover" 2440 1339430956000000 toby "you may want to look at your css as the dataset is generated by the package_list snippet I've made all packages popular for now" 2440 1339497700000000 aron.carroll Fixed this up, last thing I think we should do is ensure that the datasets passed to the homepage have descriptions. 2440 1339498655000000 toby "dataset descriptions is a data issue but I'll keep this open for now. We can probably do an exclude if needed. " 2440 1339499074000000 aron.carroll Sure, but this is more about displaying nice complete examples on the homepage. 2440 1339581622000000 toby I am changing this to assigned not accepted as this makes trac easier to use 2440 1340373461000000 toby i'm closing we can make tickets for specific issues 2441 1338287243000000 aron.carroll Order by needs hooking up to form. Currently it's just a @
Recently changed packages
+ ${package_list_from_dict(c.latest_packages)} -