{23} Trac comments (3729 matches)

Results (2601 - 2700 of 3729)

← 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 →
Ticket Posixtime Author Newvalue
#704 1288355926000000 johnbywater Referencing ticket #700 has changed sprint.
#704 1288356182000000 johnbywater Referencing ticket #700 has changed sprint.
#703 1288355926000000 johnbywater Referencing ticket #700 has changed sprint.
#703 1288356182000000 johnbywater Referencing ticket #700 has changed sprint.
#703 1289909096000000 pudo fixed in cset:b23377108b6d
#700 1288562082000000 rgrp I can verify that groups are getting lost on package preview on ckan.net (steps: 1. click edit, 2. preview 3. look at edit form and group has disappeared). Given this is a bug on a production service it is urgent this gets fixed. @pudo: please can you confirm your expected fix date on this?
#700 1289823108000000 dread This is still holding back releases. @pudo please can you look at it this
#699 1294412194000000 thejimmyg I've just verified that the search pages navigation does not respect the checkboxes. Could you have a look please Friedrich?
#699 1296499061000000 pudo fixed in cset:81a0f5538a5b
#698 1291742752000000 Stiivi I see two possible options: Option A: store only mirrors of source files, have file format based plugins for querying files Option B: store mirrors of source files, have plugin based loading scripts into "common structured format", have single query module. I would go with option B as it is: - easier to implement - file format based transformations are simpler than file format based queries - more transparent data management process - only one simple query module (see attached ckan-srcmirror.png) The Option B will fit better to the broader data architecture context: http://democracyfarm.org/f/ckan/data_arch.png Concerning API I would suggest to try to be compatible with google spreadsheet API: http://code.google.com/apis/spreadsheets/data/3.0/reference.html
#698 1291832133000000 thejimmyg Actually we've implemented a first version which doesn't store the data. See this post: http://blog.ckan.org/2010/12/04/open-data-day-announcing-ckan-data-proxy/ You can get data like this: http://1.latest.jsonpdataproxy.appspot.com/?sheet=1&indent=4&url=http://research.dwp.gov.uk/asd/asd4/r1_values.xls
#698 1291851275000000 Stiivi @thejimmyg: It is neat simple solution. You have suggested a proxy API: ''There will be a new API at ``/api/spreadsheet?callback=jsonpcallback&url=`` '' There are two options: 1. Have public ckan data proxy as stand-alone service: I get package resource URL from CKAN and pass it to proxy 2. Have ckan data API (as ticket title suggests): If I am talking to CKAN, I am getting data from CKAN, I should not care about proxy or anything behind nor I should care about original data source - I care about resource data in a format that I can process (CSV/JSON). For CKAN data API I would suggest something like: {{{ /api/resource_data/RESOURCE_ID?... }}} or more human readable: {{{ /api/resource_data/PACKAGE_NAME/RESOURCE_NUMBER?... }}} This will allow others to get only CKAN resources. Moreover, allowing to get only resource data (not any URL data) would allow us to pre-process resources in the future. First version/implementation: pass each requested resource URL to your proxy service (external, not CKAN related), which determines file by file extension in URL, fail on unknown file or unprocessable file. /api/resource_data/PACKAGE/RESOURCE?output=jsonp&sheet=1... would be redirected to (for example): http://1.latest.jsonpdataproxy.appspot.com/?url=RESOURCE["URL"]&sheet=1... Second version/implementation: Determine file type in advance and pass to appropriate conversion service when requested If you upload document on scribd or slideshare it gets processed in the background. This can be done in CKAN after any resource change. We do not need to download the file at the moment, however what can be done is: 1. try a converter by URL file extension 2. try a converter by MIME type (content-type header) 3. brute-force try all converters No need to store copies of files, just store determined file type somewhere in the resource record (as mime type). Also, it would be nice if any data conversion service would provide output in both - JSON/CSV. Therefore we would be able to have "Download CSV" link directly in CKAN web page for browsing users: /api/resource_data/PACKAGE/RESOURCE?output=csv...
#698 1291858346000000 Stiivi I have created "proof of concept" implementation that will use external data proxy service when accessing: {{{ /api/data/PACKAGE_ID }}} like: {{{ http://127.0.0.1:5000/api/data/069c80f8-8476-452e-bfd4-0a9077666c14 }}} It just works and requires refactoring to match ckan standards. I would need help from soneone who knows ckan internals better.
#698 1291859298000000 Stiivi One more note: it would be good if packages had names/identifiers as well, as referencing internal IDs from outside world is not very good practice - they are quite volatile, mostly in regard to expected objects. PACKAGE/RESOURCE_REFERENCE Possible resource references: - 'default' - reserved keyword for 'the only one resource' if there is only one, or first resource if there are more or the one with flag 'default' - 'latest' - to be able to access 'latest' resource within package (or 'actual' or 'last'?) - alphanumeric identifier (not starting with number) - number - index of resource as human/visitor sees it on page (not the same as "position" attribute - as that one might contain gaps or be different (and it is in some cases)), index of resource should be something like: {{{ SELECT package_id, id, url, ROW_NUMBER() OVER (PARTITION BY package_id ORDER BY position) AS index FROM package_resource }}}
#698 1292001709000000 Stiivi 'draft": https://github.com/Stiivi/ckanext-dataapi requires that the client handles HTTP 302 Redirect correctly.
#698 1292239372000000 rgrp 1. move repo to bitbucket 2. clone james proxy code and modify to make google spreadsheets compatible (add a test ...) 3. update the ckanext to pass on parameters .... 4. Deploy all of this to test.ckan.net 5. Rufus: check redirects with javascript
#698 1292596589000000 Stiivi Here is the fork for (json) data proxy: https://bitbucket.org/Stiivi/dataproxy I've refactored it and moved transformations into separate modules. For each resource type there should be a module in transform/<type>_transform.py Each module should implement ``transform(flow, url, query)`` and should return a dictionary as a result. Existing modules: * transform/csv_transform - CSV files * transform/xls_transform - Excel XLS files if there is no resource_type module, HTTP 200 Error Resource type not supported is returned. You can override URL file extension or specify type if extension is missing through type= URL option. For example if you have any URL that contains CSV data however the url is just foo.com/data then you can pass: url=http://foo.com/data&type=csv Note: Source refactored/updated in example/dataproxy, being tested by running locally localhost:8000.
#698 1292781368000000 Stiivi pushed parameter passing; change handling of unknown reply type on proxy side: do not raise exception, but reply with 200 Error - unkown reply type, use json/jsonp
#698 1293472613000000 anonymous Data proxy documentation: http://democracyfarm.org/dataproxy/api.html (included in sources) Updated ('s' as in structured) data proxy app: http://sdataproxy.appspot.com
#698 1293649815000000 rgrp This ticket is complete: * ckanext-dataapi: working /api/data/{resource-id} with tests * https://bitbucket.org/okfn/dataproxy - the dataproxy code running at http://jsonpdataproxy.appspot.com * functioning but needs tests and improvements There a whole bunch of improvements to be done but these will be in ticket:888
#696 1287392123000000 johnbywater Moved from sprint 1.2.8
#695 1287391751000000 pudo This is triggered by an issue where having local blinker notifications without asynchronous notifications will break indexing. The issues cause is still unknown, but one possible fix is running CKAN with a queue enabled, synchronous indexing on and no queue consumers attached. Not a real solution, but made possible in cset:a065dbc8041c
#695 1287677169000000 ollyc This is down to blinker not persisting the signal objects. Although the docs claim: "Every call to signal('name') returns the same signal object, allowing unconnected parts of code (different modules, plugins, anything) to all use the same signal without requiring any code sharing or special imports" This isn't true unless you maintain a reference to the signal object. To demonstrate this: {{{ >>> import gc >>> from blinker import signal >>> signal_id = id(signal('Package')) >>> gc.collect() 0 >>> id(signal('Package')) == signal_id False }}} The synchronous notifications code connects to signals without storing the signal objects, which are subsequently garbage collected and hence never fire. The async notifications stores references to all signals as a class attribute, so this problem is not seen when async notifications are enabled.
#695 1287677481000000 dread Superb finding Friedrich! Any chance of a quick fix for metastable then?
#695 1287677524000000 dread Sorry, I meant Olly!
#695 1287748240000000 ollyc Fix and tests for this are here: * http://bitbucket.org/ollyc/ckan/changeset/48234e36ee61 * http://bitbucket.org/ollyc/ckan/changeset/f1447bbc9d65 * http://bitbucket.org/ollyc/ckan/changeset/1d3cb5378a2f * http://bitbucket.org/ollyc/ckan/changeset/e8409c84683d * http://bitbucket.org/ollyc/ckan/changeset/8e89cbce15c7 * http://bitbucket.org/ollyc/ckan/changeset/d1d63db64585 * http://bitbucket.org/ollyc/ckan/changeset/3d9eb4172f48
#695 1287766973000000 dread These changesets do the trick. I've: * merged Ollie's changes into default and metastable branches. * updated ckan.net to latest metastable and reindexed it. * ckan.net now searches on latest packages, including sanskrit example. * test-hmg.ckan.net also updated to latest metastable for testing with UAT. * ckanhmg is to be updated next week and DGU ticket remains for this.
#694 1287087801000000 wwaites Installed postgres client from lenny backports on eu3. Can now access without problem, web sites still appear to work (e.g. fr.ckan.net). Note that the backup process runs directly on eu5 now as mentioned by rgrp in email
#693 1286831155000000 wwaites Fixed with cset:1e53d160abb6 If expires argument is set, Cache-Control headers get set -- so remove any values, default or otherwise from your configs. ETag gets set to last-modified.
#692 1294247841000000 thejimmyg CKAN doesn't need to implement this, Drupal does. Incidentally, the initial version is implemented on UAT anyway.
#691 1294412765000000 thejimmyg This is more a long term idea, taking ownership for the timebeing but happy for someone else to take this on.
#691 1294412929000000 thejimmyg If we did implement this we'd need to link new sample package to previous sample package of continuous series. So closing #748.
#691 1294413010000000 thejimmyg We'd also fold up continuous series in search results behind newest sample package. So closing #749.
#691 1294413993000000 thejimmyg We'd also discuss package relationships ideas with JF so closing #444.
#691 1295610145000000 rgrp Most of package relationships was implemented in ticket:253 and its subtickets. Therefore closing this as duplicate.
#690 1288013610000000 johnbywater This concern was resolved in favour of a converged set of attributes (#711).
#689 1287392122000000 johnbywater Referencing ticket #572 has changed sprint.
#689 1287776309000000 johnbywater Referencing ticket #572 has changed sprint.
#686 1286388346000000 dread Referencing ticket #683 has changed sprint.
#686 1287997047000000 dread Don't need this yet.
#685 1286388345000000 dread Referencing ticket #683 has changed sprint.
#684 1286388346000000 dread Referencing ticket #683 has changed sprint.
#683 1287142289000000 dread Done apart from ultrastable - not in use at moment.
#682 1286786420000000 johnbywater Moved from sprint 1.2.6
#682 1287392123000000 johnbywater Moved from sprint 1.2.8
#682 1287776309000000 johnbywater Moved from sprint 1.2.9
#682 1288038147000000 johnbywater Moved from sprint 1.3.0
#682 1297358266000000 dread Various improvements to ckanclient to enable this: cset:1bfefd7596d3 and cset:47fd07087547 and installed on buildbot now.
#681 1286786420000000 johnbywater Moved from sprint 1.2.6
#681 1287160548000000 dread Looks like integration with drupal is the problem here. I'm writing tests.
#681 1287392123000000 johnbywater Moved from sprint 1.2.8
#681 1287776309000000 johnbywater Moved from sprint 1.2.9
#681 1288036387000000 rgrp This has been fixed.
#681 1288036983000000 dread The code is fixed, but not yet deployed, which requires time.
#681 1290507180000000 dread <[email protected]> This has been deployed now.
#679 1286786420000000 johnbywater Moved from sprint 1.2.6
#679 1287392123000000 johnbywater Moved from sprint 1.2.8
#679 1287776309000000 johnbywater Moved from sprint 1.2.9
#679 1288038147000000 johnbywater Moved from sprint 1.3.0
#679 1292587426000000 dread James has started this, but still some things to add.
#679 1294166120000000 dread I added some extra bits in cset:1ca7ba29d409. Resource formats disagree between DGU and the FAQ - have sided with DGU for now as it's simpler. I think this ticket is complete now.
#678 1286786420000000 johnbywater Moved from sprint 1.2.6
#677 1286786420000000 johnbywater Moved from sprint 1.2.6
#677 1287392123000000 johnbywater Moved from sprint 1.2.8
#677 1287589651000000 dread Related to https://trac.dataco.coi.gov.uk/projects/datagov/ticket/645
#677 1287745278000000 dread Currently blocked, waiting for exact details of script.
#677 1287746221000000 johnbywater Just to say, in cases where something needs to be delivered by somebody else, it can be useful to create a separate task ticket (with undefined remaining time) to indicate what deliverable is being waited on, and in the sprint backlog to drag it above the ticket(s) that depend on the deliverable. That means we don't feel a need to factor unknowable waiting time into our own tasks. It also means impediments are more obviously displayed, so you don't lose track when waiting for a number of things. The two different objectives also well separated so there is no question that somebody is stuck and passing time on a task. I've done this a few times recently and it does help to keep things simple. :-)
#677 1287776309000000 johnbywater Moved from sprint 1.2.9
#677 1288038147000000 johnbywater Moved from sprint 1.3.0
#677 1292587315000000 dread Done a few weeks ago in dgu repo.
#676 1294413641000000 thejimmyg There was discussion of this on the list a few months ago, was there a conclusion?
#676 1295869107000000 rgrp Closing as wontfix. For dev/sysadmin documentation people can follow the link in the CKAN footer to ckan.org where we are consolidating documentation.
#675 1286786420000000 johnbywater Referencing ticket #568 has changed sprint.
#675 1287392122000000 johnbywater Referencing ticket #568 has changed sprint.
#675 1287776309000000 johnbywater Referencing ticket #568 has changed sprint.
#674 1286786419000000 johnbywater Referencing ticket #568 has changed sprint.
#673 1286786420000000 johnbywater Referencing ticket #568 has changed sprint.
#672 1286789428000000 johnbywater Moved from sprint 1.2.7
#672 1287392123000000 johnbywater Moved from sprint 1.2.8
#672 1287776309000000 johnbywater Moved from sprint 1.2.9
#672 1288038146000000 johnbywater Moved from sprint 1.3.0
#672 1291715338000000 pudo This is fixed for Solr now, plus a decision not to support postgres facets has been made.
#671 1286789428000000 johnbywater Moved from sprint 1.2.7
#671 1287392123000000 johnbywater Moved from sprint 1.2.8
#671 1287776309000000 johnbywater Moved from sprint 1.2.9
#671 1288038146000000 johnbywater Moved from sprint 1.3.0
#670 1286789428000000 johnbywater Moved from sprint 1.2.7
#670 1287392123000000 johnbywater Moved from sprint 1.2.8
#670 1287776309000000 johnbywater Moved from sprint 1.2.9
#670 1288038146000000 johnbywater Moved from sprint 1.3.0
#669 1286789428000000 johnbywater Moved from sprint 1.2.7
#669 1287392123000000 johnbywater Moved from sprint 1.2.8
#669 1287776309000000 johnbywater Moved from sprint 1.2.9
#669 1288038146000000 johnbywater Moved from sprint 1.3.0
#669 1291715237000000 pudo Important sub-task: check in schema.xml from eu4 somewhere.
#668 1294412060000000 thejimmyg Assigning this to myself but won't be looking at it any time soon, if anyone else wants to take it on I'd be very grateful.
#668 1311176649000000 thejimmyg This ticket is more than 6 months old so marking as invalid in line with our ticketing policy. The API v1 and 2 were completely re-written as part of the logic layer refactor so it is likely this issue is resolved anyway.
#667 1285925476000000 dread This was done when caching was switched off for a few hours it seems. Still, it is worth investigating this.
#667 1294414834000000 thejimmyg This is something we'll shortly be looking into for DGU work anyway once we switch to the "thin" system so don't need a separate ticket for it here too. Also, the upgrade of SQLAlchmey and test performance improvements may also make this less of a problem now anyway. Marking wontfix.
#667 1297069121000000 rgrp Actually marking as wontfix!
← 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 →
Note: See TracReports for help on using and creating reports.