{23} Trac comments (3729 matches)

Results (1101 - 1200 of 3729)

← 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 →
Ticket Posixtime Author Newvalue
#693 1286831155000000 wwaites Fixed with cset:1e53d160abb6 If expires argument is set, Cache-Control headers get set -- so remove any values, default or otherwise from your configs. ETag gets set to last-modified.
#694 1287087801000000 wwaites Installed postgres client from lenny backports on eu3. Can now access without problem, web sites still appear to work (e.g. fr.ckan.net). Note that the backup process runs directly on eu5 now as mentioned by rgrp in email
#695 1287391751000000 pudo This is triggered by an issue where having local blinker notifications without asynchronous notifications will break indexing. The issues cause is still unknown, but one possible fix is running CKAN with a queue enabled, synchronous indexing on and no queue consumers attached. Not a real solution, but made possible in cset:a065dbc8041c
#695 1287677169000000 ollyc This is down to blinker not persisting the signal objects. Although the docs claim: "Every call to signal('name') returns the same signal object, allowing unconnected parts of code (different modules, plugins, anything) to all use the same signal without requiring any code sharing or special imports" This isn't true unless you maintain a reference to the signal object. To demonstrate this: {{{ >>> import gc >>> from blinker import signal >>> signal_id = id(signal('Package')) >>> gc.collect() 0 >>> id(signal('Package')) == signal_id False }}} The synchronous notifications code connects to signals without storing the signal objects, which are subsequently garbage collected and hence never fire. The async notifications stores references to all signals as a class attribute, so this problem is not seen when async notifications are enabled.
#695 1287677481000000 dread Superb finding Friedrich! Any chance of a quick fix for metastable then?
#695 1287677524000000 dread Sorry, I meant Olly!
#695 1287748240000000 ollyc Fix and tests for this are here: * http://bitbucket.org/ollyc/ckan/changeset/48234e36ee61 * http://bitbucket.org/ollyc/ckan/changeset/f1447bbc9d65 * http://bitbucket.org/ollyc/ckan/changeset/1d3cb5378a2f * http://bitbucket.org/ollyc/ckan/changeset/e8409c84683d * http://bitbucket.org/ollyc/ckan/changeset/8e89cbce15c7 * http://bitbucket.org/ollyc/ckan/changeset/d1d63db64585 * http://bitbucket.org/ollyc/ckan/changeset/3d9eb4172f48
#695 1287766973000000 dread These changesets do the trick. I've: * merged Ollie's changes into default and metastable branches. * updated ckan.net to latest metastable and reindexed it. * ckan.net now searches on latest packages, including sanskrit example. * test-hmg.ckan.net also updated to latest metastable for testing with UAT. * ckanhmg is to be updated next week and DGU ticket remains for this.
#696 1287392123000000 johnbywater Moved from sprint 1.2.8
#698 1291742752000000 Stiivi I see two possible options: Option A: store only mirrors of source files, have file format based plugins for querying files Option B: store mirrors of source files, have plugin based loading scripts into "common structured format", have single query module. I would go with option B as it is: - easier to implement - file format based transformations are simpler than file format based queries - more transparent data management process - only one simple query module (see attached ckan-srcmirror.png) The Option B will fit better to the broader data architecture context: http://democracyfarm.org/f/ckan/data_arch.png Concerning API I would suggest to try to be compatible with google spreadsheet API: http://code.google.com/apis/spreadsheets/data/3.0/reference.html
#698 1291832133000000 thejimmyg Actually we've implemented a first version which doesn't store the data. See this post: http://blog.ckan.org/2010/12/04/open-data-day-announcing-ckan-data-proxy/ You can get data like this: http://1.latest.jsonpdataproxy.appspot.com/?sheet=1&indent=4&url=http://research.dwp.gov.uk/asd/asd4/r1_values.xls
#698 1291851275000000 Stiivi @thejimmyg: It is neat simple solution. You have suggested a proxy API: ''There will be a new API at ``/api/spreadsheet?callback=jsonpcallback&url=`` '' There are two options: 1. Have public ckan data proxy as stand-alone service: I get package resource URL from CKAN and pass it to proxy 2. Have ckan data API (as ticket title suggests): If I am talking to CKAN, I am getting data from CKAN, I should not care about proxy or anything behind nor I should care about original data source - I care about resource data in a format that I can process (CSV/JSON). For CKAN data API I would suggest something like: {{{ /api/resource_data/RESOURCE_ID?... }}} or more human readable: {{{ /api/resource_data/PACKAGE_NAME/RESOURCE_NUMBER?... }}} This will allow others to get only CKAN resources. Moreover, allowing to get only resource data (not any URL data) would allow us to pre-process resources in the future. First version/implementation: pass each requested resource URL to your proxy service (external, not CKAN related), which determines file by file extension in URL, fail on unknown file or unprocessable file. /api/resource_data/PACKAGE/RESOURCE?output=jsonp&sheet=1... would be redirected to (for example): http://1.latest.jsonpdataproxy.appspot.com/?url=RESOURCE["URL"]&sheet=1... Second version/implementation: Determine file type in advance and pass to appropriate conversion service when requested If you upload document on scribd or slideshare it gets processed in the background. This can be done in CKAN after any resource change. We do not need to download the file at the moment, however what can be done is: 1. try a converter by URL file extension 2. try a converter by MIME type (content-type header) 3. brute-force try all converters No need to store copies of files, just store determined file type somewhere in the resource record (as mime type). Also, it would be nice if any data conversion service would provide output in both - JSON/CSV. Therefore we would be able to have "Download CSV" link directly in CKAN web page for browsing users: /api/resource_data/PACKAGE/RESOURCE?output=csv...
#698 1291858346000000 Stiivi I have created "proof of concept" implementation that will use external data proxy service when accessing: {{{ /api/data/PACKAGE_ID }}} like: {{{ http://127.0.0.1:5000/api/data/069c80f8-8476-452e-bfd4-0a9077666c14 }}} It just works and requires refactoring to match ckan standards. I would need help from soneone who knows ckan internals better.
#698 1291859298000000 Stiivi One more note: it would be good if packages had names/identifiers as well, as referencing internal IDs from outside world is not very good practice - they are quite volatile, mostly in regard to expected objects. PACKAGE/RESOURCE_REFERENCE Possible resource references: - 'default' - reserved keyword for 'the only one resource' if there is only one, or first resource if there are more or the one with flag 'default' - 'latest' - to be able to access 'latest' resource within package (or 'actual' or 'last'?) - alphanumeric identifier (not starting with number) - number - index of resource as human/visitor sees it on page (not the same as "position" attribute - as that one might contain gaps or be different (and it is in some cases)), index of resource should be something like: {{{ SELECT package_id, id, url, ROW_NUMBER() OVER (PARTITION BY package_id ORDER BY position) AS index FROM package_resource }}}
#698 1292001709000000 Stiivi 'draft": https://github.com/Stiivi/ckanext-dataapi requires that the client handles HTTP 302 Redirect correctly.
#698 1292239372000000 rgrp 1. move repo to bitbucket 2. clone james proxy code and modify to make google spreadsheets compatible (add a test ...) 3. update the ckanext to pass on parameters .... 4. Deploy all of this to test.ckan.net 5. Rufus: check redirects with javascript
#698 1292596589000000 Stiivi Here is the fork for (json) data proxy: https://bitbucket.org/Stiivi/dataproxy I've refactored it and moved transformations into separate modules. For each resource type there should be a module in transform/<type>_transform.py Each module should implement ``transform(flow, url, query)`` and should return a dictionary as a result. Existing modules: * transform/csv_transform - CSV files * transform/xls_transform - Excel XLS files if there is no resource_type module, HTTP 200 Error Resource type not supported is returned. You can override URL file extension or specify type if extension is missing through type= URL option. For example if you have any URL that contains CSV data however the url is just foo.com/data then you can pass: url=http://foo.com/data&type=csv Note: Source refactored/updated in example/dataproxy, being tested by running locally localhost:8000.
#698 1292781368000000 Stiivi pushed parameter passing; change handling of unknown reply type on proxy side: do not raise exception, but reply with 200 Error - unkown reply type, use json/jsonp
#698 1293472613000000 anonymous Data proxy documentation: http://democracyfarm.org/dataproxy/api.html (included in sources) Updated ('s' as in structured) data proxy app: http://sdataproxy.appspot.com
#698 1293649815000000 rgrp This ticket is complete: * ckanext-dataapi: working /api/data/{resource-id} with tests * https://bitbucket.org/okfn/dataproxy - the dataproxy code running at http://jsonpdataproxy.appspot.com * functioning but needs tests and improvements There a whole bunch of improvements to be done but these will be in ticket:888
#699 1294412194000000 thejimmyg I've just verified that the search pages navigation does not respect the checkboxes. Could you have a look please Friedrich?
#699 1296499061000000 pudo fixed in cset:81a0f5538a5b
#700 1288562082000000 rgrp I can verify that groups are getting lost on package preview on ckan.net (steps: 1. click edit, 2. preview 3. look at edit form and group has disappeared). Given this is a bug on a production service it is urgent this gets fixed. @pudo: please can you confirm your expected fix date on this?
#700 1289823108000000 dread This is still holding back releases. @pudo please can you look at it this
#703 1288355926000000 johnbywater Referencing ticket #700 has changed sprint.
#703 1288356182000000 johnbywater Referencing ticket #700 has changed sprint.
#703 1289909096000000 pudo fixed in cset:b23377108b6d
#704 1288355926000000 johnbywater Referencing ticket #700 has changed sprint.
#704 1288356182000000 johnbywater Referencing ticket #700 has changed sprint.
#705 1288356302000000 johnbywater Referencing ticket #701 has changed sprint.
#705 1288356319000000 johnbywater Referencing ticket #701 has changed sprint.
#706 1288355926000000 johnbywater Referencing ticket #700 has changed sprint.
#706 1288356181000000 johnbywater Referencing ticket #700 has changed sprint.
#706 1289909028000000 pudo fixed in cset:b23377108b6d
#707 1288355926000000 johnbywater Referencing ticket #700 has changed sprint.
#707 1288356181000000 johnbywater Referencing ticket #700 has changed sprint.
#708 1288356302000000 johnbywater Referencing ticket #701 has changed sprint.
#708 1288356319000000 johnbywater Referencing ticket #701 has changed sprint.
#711 1288014219000000 johnbywater See also #504.
#711 1294232521000000 thejimmyg At the moment DGU has locations such as Wales, England etc. This won't change and won't try to be merged with INSPIRE datasets or bounding boxes. What we do need is a flag for INSPIRE so that the different types of package can be highlighted.
#711 1296592589000000 thejimmyg Our new approach agreed with Rod is to start with a minimum set of package extras and expand them in line with feedback from the user group. We can now show the source GEMINI record as XML (and also a very basic HTML view) for anyone who wants the full detail. No need for this ticket anymore.
#712 1288038147000000 johnbywater Moved from sprint 1.3.0
#712 1294408364000000 thejimmyg There is no 0.4 metadata spec yet so we can't model it. If there was, the discussion would be on #711 anyway.
#713 1288026226000000 johnbywater Referencing ticket #712 has changed sprint.
#713 1288038147000000 johnbywater Referencing ticket #712 has changed sprint.
#714 1291733788000000 dread This was implemented with DGU ticket 614
#715 1289815657000000 johnbywater Moved from sprint 1.3.3
#715 1297796784000000 pudo fixed in cset:69c4210f635a
#718 1288195538000000 johnbywater Referencing ticket #716 has changed sprint.
#718 1288196547000000 johnbywater Referencing ticket #716 has changed sprint.
#718 1288197569000000 johnbywater Referencing ticket #716 has changed sprint.
#718 1288459823000000 pudo fixed http://bitbucket.org/okfn/ckanextiati/changeset/4bf10d3f26e7
#719 1288195538000000 johnbywater Referencing ticket #716 has changed sprint.
#719 1288196547000000 johnbywater Referencing ticket #716 has changed sprint.
#719 1288197569000000 johnbywater Referencing ticket #716 has changed sprint.
#720 1288195538000000 johnbywater Referencing ticket #716 has changed sprint.
#720 1288196547000000 johnbywater Referencing ticket #716 has changed sprint.
#720 1288197569000000 johnbywater Referencing ticket #716 has changed sprint.
#720 1288459344000000 pudo Customer has stated they do not want this in the current iteration.
#721 1288195538000000 johnbywater Referencing ticket #716 has changed sprint.
#721 1288196547000000 johnbywater Referencing ticket #716 has changed sprint.
#721 1288197569000000 johnbywater Referencing ticket #716 has changed sprint.
#721 1288459534000000 pudo Solved using javascript to toggle hide/show preview for each resource: http://bitbucket.org/okfn/ckanextiati/changeset/942cec028cf5
#722 1288195538000000 johnbywater Referencing ticket #716 has changed sprint.
#722 1288196547000000 johnbywater Referencing ticket #716 has changed sprint.
#722 1288197569000000 johnbywater Referencing ticket #716 has changed sprint.
#722 1288459703000000 pudo fixed: http://bitbucket.org/okfn/ckanextiati/changeset/fb166fde9d3e
#723 1288195538000000 johnbywater Referencing ticket #716 has changed sprint.
#723 1288196547000000 johnbywater Referencing ticket #716 has changed sprint.
#723 1288197569000000 johnbywater Referencing ticket #716 has changed sprint.
#723 1288459610000000 pudo Fixed in cset:78b23f3f6aaf
#725 1288195549000000 johnbywater Referencing ticket #724 has changed sprint.
#725 1288196716000000 johnbywater Referencing ticket #724 has changed sprint.
#725 1288197657000000 johnbywater Referencing ticket #724 has changed sprint.
#725 1289295966000000 pudo fixed in cset:ff4264eb609c (http://bitbucket.org/okfn/iati/changeset/ff4264eb609c)
#726 1288195549000000 johnbywater Referencing ticket #724 has changed sprint.
#726 1288196716000000 johnbywater Referencing ticket #724 has changed sprint.
#726 1288197657000000 johnbywater Referencing ticket #724 has changed sprint.
#727 1288195549000000 johnbywater Referencing ticket #724 has changed sprint.
#727 1288196716000000 johnbywater Referencing ticket #724 has changed sprint.
#727 1288197657000000 johnbywater Referencing ticket #724 has changed sprint.
#728 1294232752000000 thejimmyg WAF records may always need to be re-harvested to see if they have changed. Does CSW provide any functionality that allows us to see what has changed?
#728 1304934653000000 wwaites filters not properly supported by e.g. geonetwork. so we defer.
#728 1304936482000000 thejimmyg Actually, these are a different type of filter. We need to send a filter to only get modified documents, to save processing our end.
#728 1310124784000000 thejimmyg Isn't this already completed? Putting in the backlog but I think it is already implemented. Adria, could you please update accordingly?
#729 1288012854000000 rgrp Closed in cset:a9006fd56450
#730 1291139609000000 rgrp Duplicate of ticket:402
#731 1288015048000000 johnbywater Are the tasks for this user story outstading, or can we close them now?
#732 1287776309000000 johnbywater Moved from sprint 1.2.9
#732 1288021197000000 dread Couldn't reproduce
#733 1287776309000000 johnbywater Moved from sprint 1.2.9
#734 1288038147000000 johnbywater Moved from sprint 1.3.0
#734 1292587603000000 dread Found problems and ticketed: #872 #873
#735 1292957248000000 dread I have been through looking for package names with a trailing underscore checking if they should indeed be separate package from those without. #872 and #873 cover creation of the duplicates in the first place.
#736 1294409747000000 thejimmyg The tests we write for this would need to: * Submit harvest source form with valid CSW location * Submit harvest source form with invalid CSW location We can now close #790 and #791. Note that discussion on end-to-end testing is now at #784.
#736 1296592767000000 thejimmyg We also need to check WMS URLs added to package resources to check they are genuine WMSs.
#736 1304937755000000 thejimmyg We now check WMS links. The CSW link checking could be more robust. New proposal is: Check that a URL with a / is treated the same way as one without. Treated as the same if same port, scheme, domain and path (ie query string and fragment are ignored, port is normalised away)
#736 1304963598000000 amercader See https://bitbucket.org/okfn/ckanext-harvest/changeset/a7b6d8c0dde7 https://bitbucket.org/okfn/ckanext-harvest/changeset/40de12fada74
#737 1319730303000000 dread Something like this one on github: {{{ # This is an <h1> tag ## This is an <h2> tag ###### This is an <h6> tag Text styles *This text will be italic* _This will also be italic_ **This text will be bold** __This will also be bold__ *You **can** combine them* Lists Unordered * Item 1 * Item 2 * Item 2a * Item 2b Ordered 1. Item 1 2. Item 2 3. Item 3 * Item 3a * Item 3b Miscellaneous Images ![GitHub Logo](/images/logo.png) Format: ![Alt Text](url) Links http://github.com - automatic! [GitHub](http://github.com) Blockquotes As Kanye West said: > We're living the future so > the present is our past. }}}
#738 1294408864000000 thejimmyg This needs more discussion. It is a separate problem from deleting a harvest source from the admin interface which is dealt with in #884.
← 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 →
Note: See TracReports for help on using and creating reports.