{23} Trac comments (3729 matches)
Results (1 - 100 of 3729)
| Ticket | Posixtime | Author | Newvalue | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #1788 | 1330623913000000 | zydio | Replying to [comment:1 rgrp]: > Believe most of these are resolved by https://github.com/okfn/ckan/commit/27f4fc776b9199621d259749cf20787328df101f > > @zephod: could you check again and see if anything remains? I noticed that Thedatahub with latest code is still messed up..I was working with absolute paths in my environment without the h.url_for_static() to include the html5shiv.js script so I missed a problem, which can be fixed with my new [pull request #4](https://github.com/okfn/ckan/pull/4). I'm aware of only one problem left: when you hit the "Upload a File" button inside the Dataset Edit/Resources tab nothing happens in IE < 9. This is related to those IE versions not being able to dynamically insert HTML 5 tags, which breaks some jquery-tmpl (actually JQuery) operation (if you debug the code everything works in jquery-tmpl until one of the latest operation involving JQuery returns null instead of the expected object). I was able to fix the issue in my environment by adding the following conditional comment: {{{ </script> <!--[if lt IE 9]> <script type="text/javascript" src="/scripts/jquery.html5.preie9fix.js"> </script> <![endif]--> }}} where the jquery.html5.preie9fix.js is simply the [gist](https://gist.github.com/887560) published by Akkuma in a [jquery-tmpl issue (#36)](https://github.com/jquery/jquery-tmpl/issues/36#issuecomment-918495). This script patches JQuery on the fly on IE < 9 to rework in memory html 5 blocks handled by JQuery, so that it doesn't break. And this fixes the "Upload a File" button on IE. I didn't commit/pull this specific fix for the following reasons: * the script embeds innerShiv, which is a previous incarnation of html5shiv (now included in CKAN). It doesn't sound fine to include 2 pieces of code with similar effect, but I couldn't reproduce the fix with html5shiv! * in my environment I have included this fix sitewide even if I didn't spot other problematic features..I don't know if it would be better to include it only on the dataset edit page in the official CKAN code. Summary: I leave to you guys the choice on how to deal with this problem. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1788 | 1330624155000000 | zydio | Erh...sorry for some formatting errors in my comment. I used markdown formatting for links, and used the #NUMBER format to cite issues, which obviously is interpreted by Trac as references to his own issues...my bad. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #78 | 1314877151000000 | zephod | Fixed in cset:8638a0ac2255 (fifty year expiry time on the four cookies) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #978 | 1330547181000000 | zephod | This was not trivial to implement. The backend supports arbitrary key/value pairings on resources, and the frontend can now handle this. Add, edit and delete resource extras according to the form state. I had to make a modification to the backend: When saving a resource, you have to submit the complete set of extras. Unsubmitted extras are assumed to be deleted. (This matches the behaviour of the package form). https://github.com/okfn/ckan/commit/a41cd0c9b04c757f5fa37acaba6be71e345a9c1f#L0R39 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1004 | 1323194016000000 | zephod | Dread, that fix wouldn't work because the class attribute was being defined in Genshi and again in HTML, with slighly undefined behaviour (in this case Genshi overwrites HTML). I have performed a very very simple fix which modifies the navbar text when you're not logged in. https://github.com/okfn/ckan/commit/a89a48731ba548170045a60ac2930e0019c299c7 I don't think the link should be restored in the sidebar, it was explicitly removed as part of a site-wide sweep to make the sidebar a passive, helpful element rather than an active element with action links. Site looks great to me, I'm closing this ticket. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1032 | 1330348463000000 | zephod | Streamlining supertickets. #1506 is the new parent of #978; all tasks remaining in this ticket are complete. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1108 | 1314878772000000 | zephod | First, here is a repo with the new DataHub theme: https://bitbucket.org/okfn/ckanext-datahub-theme We plan to integrate this into ckan-core, thereby replacing the current theme. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1264 | 1318244619000000 | zephod | Created #1377 to handle bugs revealed while doing this work. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1264 | 1318245716000000 | zephod | Complete; cset:b216952644aa | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1293 | 1315246628000000 | zephod | cset:https://bitbucket.org/okfn/ckan/changeset/4a98589af6c1 Fixed | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1296 | 1315403275000000 | zephod | No longer doing inline resource editing on dataset/view; instead use the edit form. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1297 | 1315403878000000 | zephod | No longer doing inline editing on the view page. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1297 | 1315403974000000 | zephod | Largely done in cset:8901e722b158 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1297 | 1315404322000000 | zephod | all done: cset:71096cd7b27d | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1307 | 1315246706000000 | zephod | cset:2c31fe24e1c9 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1307 | 1315247298000000 | zephod | Replying to [comment:2 zephod]: Oops: cset:cbd6677ec41e | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1320 | 1317391018000000 | zephod | Extend misc.py to regex find http and https, surrounding it with <>. ALSO note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package). | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1320 | 1317730999000000 | zephod | Slightly painful when links appear inside naughty HTML tags or in markdown tags. Restriction: Links are auto-linked if they appear at the start of a line or after whitespace. cset:f9dfb0506594 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1329 | 1316624026000000 | zephod | I've tried to do this wherever I could as I looked through ckan.pot for broken pieces of string. Generally it triggers errors unless things are very neatly structured (no nested links or ${a.variable} accesses, it seems) but things are quite a lot tidier. cset:d5bf4c7ba293 Does that count as 'fixed', or is this a permenant issue? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1349 | 1316277490000000 | zephod | Done. cset:e074aee19923 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1350 | 1316624288000000 | zephod | Was going to reimplement an old workaround described by Friedrich, where a language.js file is served through the home controller (with appropriate settings in routing.py) and so ${_('string literals')} will be translated. See: * https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/controllers/home.py#cl-48 * https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/templates/home/language.js However the string-extraction mechanism seems geared towards Genshi templates and hacks are required to make it read a javascript file neatly (ie. without putting the *entire file* into a string for translation). A more elegant solution seems to be using an xi:include to pull in a genshi template which contains an html <script> tag. * cset:d5bf4c7ba293 Issue is resolved; strings are now translated. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1351 | 1316540182000000 | zephod | Fixed in cset:4eebcb118b09 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1369 | 1317732684000000 | zephod | cset 6904048ce970 Okfn needs to accept a github pull request to make this work. Currently CKAN live-links to the raw github script for ckanjs, which must be updated. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1377 | 1318245795000000 | zephod | Complete: cset:b216952644aa | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1378 | 1319567698000000 | zephod | https://bitbucket.org/okfn/ckan/changeset/494bbd08364f | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1379 | 1320155927000000 | zephod | Resolved in cset:af694663f30e Now allowing markdown to do its thing, then fixing remaining links afterwards. More success that way. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1379 | 1320161439000000 | zephod | actually here: https://bitbucket.org/okfn/ckan/changeset/607f3363a8c5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1380 | 1319627454000000 | zephod | https://bitbucket.org/okfn/ckan/changeset/f600e0d37f6c | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1409 | 1319567582000000 | zephod | https://bitbucket.org/okfn/ckan/changeset/c8c62bc8766d | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1410 | 1319213205000000 | zephod | Oops: #1409 double posted. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1425 | 1327605718000000 | zephod | Refactored it to be (a) easy to find, and (b) a two-stage process which has a little more gravity. Like Bitbucket and GitHub. https://github.com/okfn/ckan/commit/f3a4c3de70da08d396da4e65a5d286b253be0e0b | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1445 | 1323178176000000 | zephod | JohnGlover has done most of this and I've been adding aesthetic tweaks. The resource page is now tidy and has a similar look and feel to its parent Dataset view page. I'm closing this ticket and its sibling #1450; I am merging the feature-1450-dataset-view branch which contains all of this work. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1450 | 1323109665000000 | zephod | johnglover architected this and I've added aesthetic tweaks. Now pushed on branch: feature-1450-dataset-view. One bug ticket remains (#1517) before the work can be merged. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1517 | 1323175957000000 | zephod | After some research, I've decided to disable the iframe datapreview of unknown filetypes. We need a safer mechanism for verifying that it's a text file or web page; a blacklist of filetypes which won't be iframed (GZIP, ZIP, etc) will never be adequate, and accidently starting a binary file download is a dangerous thing. The fix is simple - commit: 17f33031f5e348e623146601f830b8c67462fe00 But more work will be required to relax this restriction. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1521 | 1327080136000000 | zephod | This function can be deprecated, as groups will use Solr searching to list their packages. https://github.com/okfn/ckan/blob/master/ckan/logic/action/get.py#L398 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1521 | 1327405394000000 | zephod | Resolved in this merge: https://github.com/okfn/ckan/commit/2337424dd4ebe3b81b6da454f2305031922d3d5c | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1528 | 1323198001000000 | zephod | Done here. https://github.com/okfn/ckan/commit/b2748e395760083be2071b007641e787a071f955 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1583 | 1327422386000000 | zephod | Working on this on branch feature-1583-qa. https://github.com/okfn/ckan/tree/feature-1583-qa Design work is done, it needs integrating with the live API. That's all in the QA extension. Suggest all my work should be moved into the ckanext-qa extension and plugged into the API available there. Shouldnt have developed this in core after all. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1583 | 1327582611000000 | zephod | Complete. CKAN: * https://github.com/okfn/ckan/commit/048f58748b052ecd9efaa0b6940d3c1659990494 CKANEXT-QA: * https://github.com/okfn/ckanext-qa/commit/dc040f9fdfe01cf420919b0779d61043b5ec76cb * https://github.com/okfn/ckanext-qa/commit/590150dcd947d4a12ff2ec7c396f5dee4cbbaee9 * https://github.com/okfn/ckanext-qa/commit/590150dcd947d4a12ff2ec7c396f5dee4cbbaee9 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1583 | 1327582702000000 | zephod | And this: CKAN: * https://github.com/okfn/ckan/commit/8f3c70c3f3008acf9e3bfcc813f40df07c6ee4c5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1648 | 1330632344000000 | zephod | Done in the #1506 branch | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1716 | 1327601196000000 | zephod | Easily done. https://github.com/okfn/ckan/commit/2bd50d92f48ca67edb69e51a7bcfc5009d77a785 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1717 | 1330088539000000 | zephod | 1. Already exists; "tags:csv" rather than "tag:csv" 2. Format is user selected; we just show the most popular choices. Perhaps an effort should be made to force people to homogenize their format choices (my work with automatic icons will do this to an extent). Also, res_format doesn't appear on thedatahub.org, but for some reason it does on test... 3. There are thousands. We just show the most popular/relevant tags for your search. This could break out into another ticket though - eg. autocompleting tag input on the page? 4. Can't remember what this is... 5. I did this a while ago and popular opinion was that it should change back. This gives us room for expanded options like on thedatahub's live search page right now, and matches the layout of the Groups view page. 6. They should maybe be ordered by relevance, which includes popularity in some metric? This should be broken into a seperate ticket if you want it to go ahead and given to a Solr expert. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1735 | 1328531105000000 | zephod | This is an edge case, and there's no simple fix -- you can't detect *why* the tags box has lost focus -- so I think let's not waste time inventing something complicated. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1744 | 1330349472000000 | zephod | Massive refactor of parent ticket mean this is all done and can be closed off. See #1506 for remaining UX work. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1788 | 1333133365000000 | zephod | I've been performing a full sweep of the site to rebuild the markup & stylesheets using Bootstrap, and in doing so have managed to close off this ticket. Bootstrap allows us to delegate lots of clever functionality and layouts to Twitter's tried-and-tested code, making it easier to be compatible with IE7. It has brought up plenty of issues: Lots of Javascript was broken; plenty of pages generate broken doctypes causing the browser to go into quirks mode (which in turn causes the page to explode all over the screen); most dynamic content would fail to generate properly in one way or another.... (see the #1788 branch for a log of issues encountered). I've been refactoring our JS and have managed to strip ckanjs, too, because the only significant element we'd used was the File Upload view (which also now works in IE7). test.ckan.net has been redeployed on master; I'm browsing the site in IE7 and everything is looking great :-) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #2223 | 1332246030000000 | zephod | I have been working overtime to try and get this change straightened out as it's suddenly on the critical path for something. Have a look here: https://github.com/okfn/ckan/tree/feature-2223-bootstrap It's 80% of the way there; some form CSS needs overriding because Bootstrap tends to make things very large so they spill over. But on the whole the markup is in a steady state, which is what we need. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #2223 | 1332842692000000 | zephod | Merged into master here: https://github.com/okfn/ckan/commit/0bf4b33955df3806785f9b42a55acaeb45596e25 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #941 | 1298886391000000 | wwitzel3 | Continued work on the community plugin. I am still learning the layout of templates and how they work within ckan and getting figuring out Genshi templates so this is where most of the delay has been. I've been able to determine a pretty good plugin layout for extensions that create models. I am currently focusing on getting the rest of the UI in place and trying to determine the best way to get colander to do the desired validation beyond ensuring the form has all the elements. After todays work, I will push what I've done and I would like to walk through the design with someone at some point. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #941 | 1300295545000000 | wwitzel3 | https://bitbucket.org/okfn/ckanext-community/changeset/c26cc663fa00 I need to update the README to explain how to enable Edit and Delete now that I've integrated it with authorization in CKAN. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #986 | 1297812401000000 | wwitzel3 | https://bitbucket.org/okfn/ckanext-qa/src/be57e20c60ef/ | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #90 | 1264158298000000 | wwaites | see http://semantic.ckan.net/ data can be loaded into the store easily enough (cron job for now?) http://semantic.ckan.net/data/package-name the RDF equivalent of http://ckan.net/package/package-name has sparql interface at http://semantic.ckan.net/sparql not so sure about the assertions Talis has made about the ckan namespace... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1274377385000000 | wwaites | Some more datapoints from Leigh Dodds of Talis: I'm still having no joy with this I'm afraid. I'm test parsing the data locally using the TDB command-line tools, specifically tdbcheck which will parse the data and generate warnings/exceptions. This uses the same parsing code, data and URI validation code as we're using on the Platform. Currently its giving me warnings for invalid lexical values for dates, e.g: Lexical not valid for datatype: "2008"^^http://www.w3.org/2001/XMLSchema#date While these aren't a major issue, looking at some of the data suggests that there are more underlying data problems that need checking and fixing up, e.g: Lexical not valid for datatype: "n/a"^^http://www.w3.org/2001/XMLSchema#date Lexical not valid for datatype: "27/04/2006 13:56"^^http://www.w3.org/2001/XMLSchema#date Lexical not valid for datatype: "Real time calculation"^^http://www.w3.org/2001/XMLSchema#date Lexical not valid for datatype: "varies by country"^^http://www.w3.org/2001/XMLSchema#date And there are still some invalid URIs, e.g: <https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%3&ref=3.02.16> Code: 30/ILLEGAL_PERCENT_ENCODING in QUERY: The host component a percent occurred without two following hexadecimal digits. Can I suggest you try running the converted data through tdbcheck to iron out any problems? Then I can push it into the Platform. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1276271343000000 | wwaites | url validation reputed to be here: http://www.livinglogic.de/Python/url/Howto.html | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1276438793000000 | wwaites | Some good news, ll.url seems to take bad urls and make them into good urls. viz: {{{ In [1]: from ll import url In [2]: print url.URL("https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%3&ref=3.02.16") ------> print(url.URL("https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%3&ref=3.02.16")) /Users/ww/Work/OKF/ckanrdf/lib/python2.6/site-packages/ll/url.py:2358: UserWarning: truncated escape at position 4 value = _unescape(namevalue[1].replace("+", " ")) https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%253&ref=3%2E02%2E16 }}} | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1276438832000000 | wwaites | Also fyi, getting ll.url is done like so {{ pip install ll-xist }}} | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1276438907000000 | wwaites | I've updated ckanrdf to strip out datatypes and use this ll.url on external references so that should be sufficient to hold off talis. Still need to work particularly on validating dates though... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1283179768000000 | wwaites | CO may not realise the implications when they said it was low priority. The implication of this lack of validation is that it is impossible to generate valid URIs in the RDF which means it cannot be imported by Talis. So until there is a solution to this, no RDF catalog. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #318 | 1296482049000000 | wwaites | We still require form validation to check URIs. They are not free-form strings. This is not the same as 852 or necessarily included in it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #433 | 1282081297000000 | wwaites | So one more modification, the datapkg_sources argument is no longer a dictionary but a string, similar to the metadata.txt but with the name as the section heading rather than [DEFAULT] so as to be able to support more than one download. As of now, this works:: {{{ This class treats an installed python package as a data index. For instructions on creating such a package, what needs to go in its setup.py and such, see :func:`datapkg.pypkgtools.datapkg_sources`. Here we are concerned with how to use such a package. An example of one such package can be installed like so:: % pip install hg+http://bitbucket.org/ww/ukgov_treasury_cra Once installed, datapkg can be used to inspect it and install parts wherever desired:: % datapkg list egg://ukgov_treasury_cra cra2009 -- Country and Regional Analysis 2009 % datapkg install egg://ukgov_treasury_cra/cra2009 file:///tmp [...] % ls -l /tmp/cra2009/ total 11112 -rw-r--r-- 1 ww wheel 5681852 May 12 15:48 cra_2009_db.csv -rw-r--r-- 1 ww wheel 292 Aug 17 22:37 metadata.txt }}} Of course the related python code and machinery hasn't been ported over to that package yet, but that's quite another matter. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #433 | 1283183548000000 | wwaites | changed datapkg_sources to datapkg_index and updated to work with the new changes to how the downloader works. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #489 | 1294416189000000 | wwaites | We still do need something like this. Right now the rdf generation is a cron job that crawls the API. Really we want two things: * only generate RDF for those packages that have changed * enable multiple client/worker processes, potentially run by third parties, to generate different parts of the RDF description. For example, it is reasonable that we generate the DCat representation ourselves, however the voiD authors (DCat is generic, voiD is about RDF datasets in particular) want to generate the voiD-specific annotations themselves and contribute it back to augment the catalogue. Another example: group curators may also want to annotate packages in their group with group-specific metadata. Yet another example: checks on the coherence, availability, quality, openness, etc. of a package should be done from time to time or when a package changes, which can result in further annotations (see the curate tool). Because of the "third party" aspect we cannot do this with the internal package representation and the rabbit queue. Most likely it is feasible to do this by looking at the revisions in the API to get all changes since the last time a script was run in which case most likely the answer is, yes, there is nothing to do here and the ticket can be closed. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #490 | 1283770756000000 | wwaites | Presuming this is for INSPIRE compliance, support for Opensearch should be added | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #537 | 1283325156000000 | wwaites | See #540 for a story about Varnish. Strongly favouring squid at this juncture. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #540 | 1283324947000000 | wwaites | Cut-and-paste from ckan-discuss: I had a look at Varnish and I agree that the configuration language is complicated. In fact by default Varnish disregards cache control headers and in general behaves in a very standards non-compliant way. I have no doubt that it is very fast -- if you are willing to spend the efford to customise its configuration for the exact layout of pages and headers and such that each web site it is going to be used with will use. In other words, there is a large administrative burden. So I decided to change tack and see where the Squid proxy has gotten to in the decade or so since I last met it. Squid is a general purpose caching proxy that can be configured as an http accelerator. The configuration is simple. You tell it where your web servers are for which sites. The web servers make sure to set the cache control headers appropriately. Here are some results from my testing, against http://de.ckan.net/package/list?page=B which is an example of a slow page. Except for the first, which only did 100 requests, the tests were set to 8 simultaneous connections and a total of 1000 requests. {{{ No caching of any kind: Requests per second: 0.44 [#/sec] (mean) Beaker Cache (filesystem): Requests per second: 43.16 [#/sec] (mean) SQUID setting cache control headers correctly: Requests per second: 421.33 [#/sec] (mean) }}} The results are clear. Using the application cache is about 100 times faster than doing nothing. Using squid is about 1000 times faster. (Doing both wouldn't necessarily help very much). I'm sure we could squeeze a bit more performance out of it if we used Varnish, but probably not an order of magnitude and I don't think it is worth the administrative burden. If we set up a production Squid instance (or farm), with a bare minimum of work it can cache for any number of sites, not just CKAN. For the python coders, here's what you have to do to set the headers properly so that squid will cache the page: {{{ del response.headers["Pragma"] del response.headers["Cache-Control"] from time import gmtime, strftime response.headers["Last-Modified"] = strftime("%a, %d %b %Y %H:%M:%S GMT", gmtime()) response.cache_expires(seconds=3600) }}} A further advantage is that the *browsers* will also understand these cache-control headers and do their own caching - just setting them properly without even using Squid should result in some subjective performance improvements. That's all for now, I suggest we dedicate a machine to just running squid, the more RAM the better and big discs are good, and put it between the world and the ckans. Oh, and comb through the controllers setting the headers correctly where appropriate... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #541 | 1294924872000000 | wwaites | this is implemented with the cookie javascript technique. not so suitable for pages where content is radically differerent depending on the logged in user but for small bits like the logged in status works perfectly well, likewise for edit links. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #542 | 1283278896000000 | wwaites | Implemented with: * cset:2bbc186459cb * cset:eba23cc027e5 * cset:e89e0b82b24e * cset:943ed812e237 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #569 | 1297177220000000 | wwaites | so this means a dependency on amara (finally). on the plus side it is one of the better xml python libraries, xpath, xslt, etc. on the down side it how many xml libraries do we depend on and use now? this is for the schematron... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #569 | 1297177522000000 | wwaites | *sigh* but it relies on 4suite in version 1.x and amara2 doesn't have the schematron anymore... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #569 | 1297177695000000 | wwaites | *sigh* again. the gemini schematron schema isn't compatible with the amara scimitar... despite the instructions to use the implementation from schematron.com... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #576 | 1284212162000000 | wwaites | * cset:9b1255e02e5e removes a reference to Package.c which is deprecated even in 0.4 * cset:d7e583948c95 wraps the session creation in if/else block on SQAlchemy version as the default behaviours and options have changed Unit Tests appear to work as before... Boy do they take a long time to run... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #576 | 1284215009000000 | wwaites | cset:7a04e78cec97 ''pool_threadlocal=True'' is no longer the default with 0.5. Setting it to ''False'' (the default) causes unit tests to hang. So explicitly set it to ''True'' to retain compatibility. It is an open question whether this is the ''correct'' setting, the defaults were changed as apparently they led to surprising behaviour. One hopes that our unit tests have already eliminated any such surprises. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #576 | 1284222948000000 | wwaites | Most of the way to 0.6 compatibility with cset:ce6b870570c1/vdm All tests still pass with 0.4 and 0.5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #662 | 1288624137000000 | wwaites | Ran into this with RDF export (that then updates the CKAN package with LOD2-compatible extras from the results). Cannot use ckanclient.package_entity_put(ckanclient.package_entity_get("ckan")). Is this related to the modifications htat are showing up in the changelog with no apparent package (the package that should be appearing there is "ckan" itself) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #693 | 1286831155000000 | wwaites | Fixed with cset:1e53d160abb6 If expires argument is set, Cache-Control headers get set -- so remove any values, default or otherwise from your configs. ETag gets set to last-modified. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #694 | 1287087801000000 | wwaites | Installed postgres client from lenny backports on eu3. Can now access without problem, web sites still appear to work (e.g. fr.ckan.net). Note that the backup process runs directly on eu5 now as mentioned by rgrp in email | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #728 | 1304934653000000 | wwaites | filters not properly supported by e.g. geonetwork. so we defer. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #851 | 1292860957000000 | wwaites | * link checker above uses the queue. queue not running generally * quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #851 | 1292892515000000 | wwaites | Ready to put into cron job. cf: * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions) * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions) * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi 1.2) * https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #851 | 1292894661000000 | wwaites | currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #851 | 1292941303000000 | wwaites | urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp. change the httpReq action to use http://curl.haxx.se/libcurl/python/ | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #851 | 1293025112000000 | wwaites | Implemented curlReq that does a curl request and returns statements similar to or analogous to httpReq. Require curate<=0.8 This in a cron job is sufficient to go through all the packages and update them with a broken link tag now: {{{ curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY }}} | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #852 | 1291723207000000 | wwaites | plugin documentation: http://packages.python.org/ckan/plugins.html | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #853 | 1291723143000000 | wwaites | done in http://bitbucket.org/ww/ofs need to merge back into main ofs repo | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #853 | 1294594581000000 | wwaites | we don't need a policy document exchange. it's simpler than that. the "server" instance has already permissions to upload. it just calculates the headers and such that are needed (based on the "client"'s initial headers) and gives them to the "client" the client then uploads without knowing the "server"'s credentials. The "client" never needs any of its own goostor credentials at all. the only separate step is to make the widget readable by the world. ticket #879 is to expose this as a small set of API calls. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #860 | 1291726067000000 | wwaites | http://bitbucket.org/ww/ckanrdf/changeset/67df6dc33ec4 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #862 | 1291726261000000 | wwaites | not to mention sparql endpoint... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #871 | 1292322361000000 | wwaites | Regarding rkhunter -- yes, eu1 appears to be clean Regarding the upgrade -- upgradede to 4.72 from backports which, looking more closely, appears to still have the privilege escalation bug but not the remote root exploit. Regarding exim on other hosts, there is no reason for them to be running a full mta, something like ssmtp should suffice. Also very worth the thought of moving to postfix. It's much easier to configure and I haven't known it to have any comparable bugs in the decade or so I've been running it. In fact I've never seen anyone actually use exim before... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #879 | 1294604066000000 | wwaites | first cut: https://bitbucket.org/ww/ckanext-storage/src | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #884 | 1293319984000000 | wwaites | Implemented in feature-884-rmsource branch at https://bitbucket.org/ww/ckan/changeset/50cbfc5a6bee | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #884 | 1294253563000000 | wwaites | see feature-884-rmsource in the main ckan repo deleting my own repo | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #885 | 1293278564000000 | wwaites | Friedrich had some mixed experiences with owslib and some German CSW endpoints: http://pudo.okfnpad.org/geodaten | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #885 | 1293314219000000 | wwaites | See: https://bitbucket.org/ww/ckanext-ows impossible as of yet to test against live server from the OS | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #885 | 1294253497000000 | wwaites | done with feature-885-owslib | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #908 | 1296403695000000 | wwaites | see also from http://knowledgeforge.net/okfn/tasks/ticket/485 {{{ (pyenv)okfn@eu7:~/var/srvc/ckan.net$ sudo uwsgi -C -iH /home/okfn/var/srvc/ckan.net/pyenv --paste config:ckan.net.ini --uid www-data -s 10.48.162.201:9001 *** Starting uWSGI 0.9.6.6 (32bit) on [Sun Jan 30 16:00:13 2011] *** compiled with version: 4.4.3 Python version: 2.6.5 (r265:79063, Apr 16 2010, 13:28:26) [GCC 4.4.3] uWSGI running as root, you can use --uid/--gid/--chroot options setuid() to 33 *** WARNING: you are running uWSGI without its master process manager *** your memory page size is 4096 bytes allocated 412 bytes (0 KB) for 1 request's buffer. Setting PythonHome to /home/okfn/var/srvc/ckan.net/pyenv... binding on TCP port: 9001 your server socket listen backlog is limited to 64 connections initializing hooks...done. Loading paste environment: config:ckan.net.ini Traceback (most recent call last): File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/__init__.py", line 3, in <module> from loadwsgi import * File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 8, in <module> import pkg_resources File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py", line 2691, in <module> add_activation_listener(lambda dist: dist.activate()) File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py", line 668, in subscribe callback(dist) File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py", line 2691, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py", line 2195, in activate map(declare_namespace, self._get_metadata('namespace_packages.txt')) File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py", line 1790, in declare_namespace _handle_ns(packageName, path_item) File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py", line 1761, in _handle_ns loader.load_module(packageName); module.__path__ = path File "/usr/lib/python2.6/pkgutil.py", line 238, in load_module mod = imp.load_module(fullname, self.file, self.filename, self.etc) File "/home/okfn/var/srvc/ckan.net/pyenv/src/ckanext-dataapi/ckanext/dataapi/__init__.py", line 36, in <module> from pylons import config File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/__init__.py", line 4, in <module> from pylons.config import config File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/config.py", line 2, in <module> from pylons.configuration import * File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/configuration.py", line 25, in <module> import pylons.legacy File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/legacy.py", line 11, in <module> from pylons.util import deprecated, func_move File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/util.py", line 18, in <module> from paste.script.appinstall import Installer File "/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/script/appinstall.py", line 23, in <module> from paste.deploy import appconfig ImportError: cannot import name appconfig }}} | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #908 | 1296660052000000 | wwaites | Not fixed. Try running with: {{{ uwsgi -C -iH /path/to/your/pyenv --paste config:ckan.net.ini -s 127.0.0.1:9000 }}} Still get giant traceback relating to a circular import centered on an arbitrary plugin | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #930 | 1296062391000000 | wwaites | This is for HMG to prove to them that requests are being serviced in a timely manner * move this functionality to a plugin * make it use rrdtool instead of lots of itty bitty files | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #930 | 1296070197000000 | wwaites | actually, this is a configuration variable ckan.enable_call_timing cleaned up the code a bit, make sure this variable unset or set to false or no and there should be no millions of files as it is unclear if this is actually used or necessary, not doing the rrdtool part just yet. if there is call for it, make a new ticket for this. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1088 | 1302630624000000 | wwaites | you use the negotiate function in autoneg.py with a cfg dictionary like this, {{{ cfg = { "mime_types": [ ("application/xhtml+xml", ["html"]), ("text/html", ["html"], ("application/rdf+xml", ["rdf", "owl"]), ("application/turtle", ["ttl"]), ("text/plain", ["nt"]), ("text/x-graphviz", ["dot"]), ] for content_type, exts in negotiate(cfg, accept_header): if "html" in exts: render page normally else: redirect to semantic.ckan.net } }}} | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| #1088 | 1303035487000000 | wwaites | fixed in 5c091fc08474 (bug-1088-autoneg-is-wonky) tip | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note:
See TracReports for help on using and creating reports.
