{22} Trac tickets (2647 matches)

Results (1201 - 1300 of 2647)

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#860 defect wwaites wwaites ckan-v1.3 closed fixed full text indexing for semantic.ckan.net

...

1291726029000000 1291726067000000
#861 defect wwaites wwaites closed invalid unicode errors creating rdf output

http://bitbucket.org/ww/ckanrdf/changeset/63b60d3a253d

1291726125000000 1311168845000000
#862 defect wwaites wwaites ckan-v1.3 closed fixed deploy rdf on catalogue.data.gov.uk

handle various dgu extras, configure apache for autonegotiation, etc.

1291726237000000 1291726261000000
#863 enhancement memespring memespring closed wontfix prompt users for missing package information

e.g. license, owner etc

1291729787000000 1338206455000000
#864 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Search results UI changes

as per http://ckan.org/wiki/UIRedesignSearch

1291736441000000 1291741028000000
#865 defect pudo pudo ckan-v1.3-sprint-1 closed fixed Support external strings in JS
  • Provide and include a "language.js" file.
1291801938000000 1295259773000000
#866 enhancement memespring memespring ckan-v1.3-sprint-1 closed fixed Include download formats in search results

as above

1291812213000000 1292240191000000
#867 enhancement dread dread ckan-v1.3-sprint-1 closed fixed ckanclient raises exceptions

To be more pythonic, raise exceptions when ckanclient gets status which isn't 200.

1291915162000000 1299866685000000
#868 enhancement sebbacon thejimmyg closed fixed Test improvements

The tests currently take 41 mins to run on most laptops. This slows down development and discourages a test-driven approach.

We'd like to see the tests pass in 5 mins or less (but anything would be an improvement!)

Some suggestions for achieving this include:

  • Upgrading the entire codebase to SQLAlchemy 0.6 so that tests could run against an in-memory SQLite database
  • Not setting up and tearing down the database so frequently.
1291987811000000 1294753596000000
#869 enhancement rgrp rgrp ckan-v1.3-sprint-1 closed fixed Add plugin interface to allow editing and extending of config by extensions

Use case: extensions need to add to template and public search paths if they wish to have templates or extra static files (such as js etc).

Allowing plugins to edit the config dictionary on load would allow this (and more!).

Are there any security considerations here? (Is is bad to allow extensions to make arbitrary changes to the config information?)

1291989395000000 1292235262000000
#870 defect memespring ckan-v1.3-sprint-1 closed fixed Menu items not selecting correctly

Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly.

Specifically if Home / About or Add Package / Search are selected both menu items are highlighted.

This is because the code assumes only one top-level menu item per controller.

1292243583000000 1294914243000000
#871 defect nils.toedtmann closed invalid Check whether localhost-only exim installtions need upgrading too

The infamous exim bug only needs one mail with prepared headers to travel through a exim system infect it. All local processes could do that, and some services (e.g. cron, webapps) send messages and might be convinced by malicious remote users to produce evil headers.

We should either rule out that this could happen on our systems, or upgrade all exims regardless of whether they are localhost-only or not.

BTW did we already run a rootkit checker like Rootkit hunter on eu1? If not we should maybe do it now - there was already an exploit out in the wild. ByteMark? has (a) already observed infections and (b) notified us because they remotely fingerprinted our mailer to be exim<4.70 (our EHLO banner contains the exim version), just as anyone could.

1292264117000000 1296340558000000
#872 enhancement johnbywater dread closed fixed ONS loader assigns packages to same package even when dept name changes

Solutions: ResourceSeriesLoader? allows synonyms for values.

1292424112000000 1292957110000000
#873 defect dread dread closed fixed ONS loader assigns packages to same package even when dept name added manually

e.g. national_child_measurement_programme had no department but "Information Centre for Health and Social Care" for agency, and TNA inferred it should be under department DoH. But now the next ONS update to this series expects a blank department, so creates a new package national_child_measurement_programme_.

1292437253000000 1292939274000000
#874 enhancement pudo ckan-v1.3-sprint-2 closed fixed Extract solr search backend into an extension

Move the solr SearchBackend? to a seperate code base, remove any mentions of the CKAN model.

1292844422000000 1293057151000000
#875 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Search backend supports solr interface and query API mimics solr

Consolidate search API interface (and backend) on solr (solrpy) type interface.

  • Support for standard query structure
  • Support for facet options

Do not need to change response formats. (Or do we?)

2 options here for advanced features like facets in non-solr:

  1. Disable (happens automatically)
  2. Implement - suggest using group by etc

Extras

  • Front-page tag cloud: change this to use facets
    • Accept this means that if facets not functional in backend we have no tag cloud
1292844957000000 1297085261000000
#876 enhancement sebbacon rgrp ckan-v1.3-sprint-2 closed fixed Support sqlite as a database backend for CKAN

Among other things this will allow the tests to run much quicker.

1292858711000000 1294753889000000
#877 enhancement rgrp rgrp ckan-v1.4-sprint-2 closed fixed File upload in WUI (extension)

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

  • Implement as an Extension
  • Locate at /upload
  • Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
  • Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
  • This wants to be tied in to general package workflow. See parent ticket:852.
1292868146000000 1298624165000000
#878 enhancement rgrp rgrp ckan-sprint-2011-09-12 closed fixed Integrate file upload with workflow around package resources

Create/edit a package -> add a resource -> Oh, i haven't uploaded yet, 'upload here' -> /upload -> standard upload form

Do we need to create a resource out of the file after upload? Do we sync file metadata with local Resource metadata?

If yes suggests (need to ticket):

  • 'ResourceExtras?' for extra metadata on resources
  • Makes 'Resource' into first class object (and make available in WUI)?

Depends

  • #877 (Upload in WUI) - should be part of that extension
1292868269000000 1315820838000000
#879 enhancement wwaites rgrp ckan-v1.3 closed fixed Storage auth API

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

1292868509000000 1316030482000000
#880 defect dread ckan-v1.3-sprint-2 closed fixed ONS loader tripping up on deleted packages

Deleted packages with the same title and extra fields as an active package trip up the loader. It finds them in the search, but due to ckanclient not using the apikey when you 'get' the package to examine it further, it causes an exception.

1292939379000000 1292945137000000
#881 defect thejimmyg fccoelho closed invalid http authentication requirement when installing

I am having trouble installing Ckan with virtualenv and pip using pip-requirements.txt due to the Authentication requirements to download the source code.

1293021889000000 1296335072000000
#882 defect pudo pudo closed wontfix Update i18n user docs and backport genshi i18n:domain fixes from WDMMG 1293192044000000 1340626094000000
#883 enhancement wwaites wwaites ckan-v1.4 closed fixed uklii harvesting refactor

master ticket

  • #884 implement harvest job delete command for an easier life
  • #885 move csw code into dedicated ckanext plugin
  • #886 move forms api into dgu extension module
  • #887 move harvesting code to use generic harvesting module

1293277713000000 1300196622000000
#884 enhancement thejimmyg wwaites ckan-v1.3 closed fixed harvest job delete operation

implement paster subcommand to delete harvest jobs

1293277800000000 1296592140000000
#885 defect wwaites wwaites ckan-v1.3 closed fixed reimplement csw module to use owslib 1293278099000000 1294253508000000
#886 defect wwaites wwaites ckan-v1.3 closed duplicate move uklii forms api to dgu module

doesn't belong in ckan core

1293278145000000 1294916538000000
#887 enhancement wwaites wwaites ckan-v1.4 closed fixed move harvesting to generic module

in concert with pudo's work on harvesting different types of documents. should also make it possible to consume, e.g. dcat

1293278215000000 1300196600000000
#888 enhancement johnglover rgrp ckan-sprint-2011-10-28 closed fixed Improvements to the dataproxy and the data API

First version of dataproxy and data API working (ticket:698) but have identified a variety of important improvements. (Should split these into sub-tickets ...):

For dataproxy:

  • Testing for dataproxy
    • Can start by using known good remote urls (moving forward could switch to providing/mocking these locally)
  • Remove content-lenght for csv requirement: just read the first x rows (up to some configurable maximum)
  • Google docs style row/column selections
  • Use the swiss library - https://bitbucket.org/okfn/swiss
    • Support google docs spreadsheets (format = service/gdocs/ccc or gdocs/ccc or gdocs/spreadsheet)
  • Handle redirects for content-length?
  • Ignore resource type if not recognized and fall-back to trying to identify from extension (or mime-type?)

For dataapi:

  • Ensure we pass on resource format as part of redirect i.e. /api/data/{id} -> {dataproxy}?url={resource-url}&type={resource-type}
1293649783000000 1311773103000000
#889 enhancement rgrp rgrp ckan-v1.3-sprint-2 closed fixed Support extra footer material in config option (e.g. for google analytics)

Add a config option 'template_footer_end' which is inserted in layout_template just before </body>.

This allows sysadmins to add extra items, especially, scripts directly into site without having to do any theming and is especially useful for things like google analytics.

Aside: going forward may want to turn this into a extension.

Cost: 1h

1293713189000000 1293715109000000
#890 enhancement kindly pudo ckan-v1.6 closed invalid Introduce timed actions into ckanext-queue

The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows:

  • routing_key: Package
  • operation: daily
  • payload: pkg.as_dict()

Where operation is one of daily, weekly, monthly or any other interval.

1294050769000000 1318599247000000
#891 task johnglover pudo ckan-sprint-2011-11-07 closed fixed Resource download worker daemon

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

  • Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
  • Should we have clever ways to dump APIs? ANS: no.
  • Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
  • Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
    • Complete support for ETags
    • Expires, Max-Age etc.
  • Check

Functionality

  • Download files via HTTP, HTTPS (will not do FTP)

Process:

  1. [Archiver.Update checks queue (automated as part of celery)]
  2. Open url and get any info from resource on cache / content-length etc
    1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
    2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
  3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
    • Add cache url to resource and updated date
    • Add other relevant info to resource such as md5, content-type etc
  4. Update task_status

Optional functionality

  • If result object is HTML, search for references to "proper data" (CSV download pages etc.)
  • Download from POST forms (accepting licenses or weird proprietary systems)
  • Support running on Google Apps Engine to save traffic costs.

Existing work

1294052979000000 1320149841000000
#892 enhancement johnglover pudo ckan-sprint-2012-01-09 closed fixed Make stored data available in WUI - 0.5d

Once we have storage, make the data available in the following ways:

  • Now have a cached_url field can show in the frontend ...
  • Add a [<a href="${cached_url}">cached</a>] link to right of real url on resource listing on dataset page.
  • On resource page: will not add it yet.
    • At the moment no clear place to pu this given nice big download button (could put in list of items on left but that does not seem right and note that it will turn up in big list of info at bottom)
  • Add test (?)
  • Deploy
1294053293000000 1324402480000000
#893 defect pudo ckan-v1.4 closed wontfix ExtrasField should not overwrite more specific extras

At the moment, ExtrasField? cannot be used in conjunction with more specific extra fields, such as TextExtraField? or SuggestTextExtraField?.

1294135783000000 1298293527000000
#894 defect wwaites wwaites ckan-v1.4 closed invalid harvest from http://locationmetadataeditor.data.gov.uk/geonetwork/srv/csw 1294253876000000 1300196388000000
#896 story thejimmyg closed duplicate Distributed Data and Syncing Between CKAN Instances

Based on the ideas in:

Merging the tickets #296, #297, #298, #299 into this one.

Related ideas include:

  • Using dcat to exchange data
  • Using couchDB to have a stub package record synchronised between multiple CKAN instances, each linking to the main record in the correct instance

This should be considered a wishlist item for the timebeing.

1294406999000000 1300217375000000
#897 defect dread dread closed duplicate "New package" highlighted in Navigator when searching

On ckan.net 1.3.1b, click on "Search" in the Navigator. Now both "Search" and "New Package" are highlighted.

1294648815000000 1294914333000000
#898 defect rgrp dread closed fixed Changes stored indefinitely

Every change to every object is being stored in memory, which could add up to quite a lot of memory.

This fixes it by making sure the objects are in a weakref. https://bitbucket.org/kindly/vdm/changeset/8d5f91db641f

1294659490000000 1294662408000000
#899 enhancement rgrp dread closed fixed Optimise check_real_change

There is no need to re-query the database to detect if there has been a changed as sqlalchemy knows this. It saves about 15 secs in the ckan tests. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e

From David Raznick

1294659583000000 1294916615000000
#900 defect dread closed fixed Intermittent error with new revision not having id

There are still some intermittent errors due to objects not having a revision_id, if you save an object at the same time as the revision. We add the uuid earlier to fix this. https://bitbucket.org/kindly/vdm/changeset/0050d023ca4e.

From David Raznick

1294659696000000 1294916587000000
#901 enhancement rgrp dread closed fixed vdm support for sqla 0.6 and sqlite 1294660144000000 1294662466000000
#902 defect rgrp dread ckan-v1.4-sprint-1 closed fixed Package buttons not clickable / Improve submenu tabs

On a package page there are three buttons to the right - View, Edit, History. Each of these buttons contain an icon and the text, but you can only activate the button by clicking on the icon. When you click on the part of the button where the text is (about 50% of the button) then nothing happens.

Using: Chrome, Firefox

1294756898000000 1297078885000000
#903 defect pudo pudo ckan-v1.3 closed fixed Proper handling of deleted packages in Solr serps

Currently, deleted packages are filtered from search results after solr returns. That means result count can drop below the requested number, down to 0. Thus we need to filter before or in Solr.

1294831054000000 1296588070000000
#904 task rgrp Stiivi ckan-v1.4-sprint-3 closed fixed Review CKAN documentation

What's bad at the moment?

  • lack of documentation e.g. config (very poorly documented)
  • too many sources of documentation
  • no common theming

Sources:

Resulting meta-ticket with things to do: ticket:927

1294832610000000 1299840539000000
#905 defect thejimmyg Stiivi ckan-sprint-2012-02-06 closed worksforme Unable to search with accented characters in package names

I have a package with name: "Štatistické prehľady agendy cestovných pasov" and when I try to search for "Štatistické" there are no results.

CKAN setup:

  • psql DB with UTF-8 tables
  • psql search backend
  • ckan version: 1.3.1b

Example:

http://verejnedata.sk/package/agenda-cestovnch-pasov

try to search for the first word in the package name.

1294939120000000 1328638536000000
#906 enhancement thejimmyg Stiivi ckan-sprint-2012-03-05 closed fixed Ability to search without accents for accented words

In Slovakia users are expecting from sites to be able to search without typing accents, for example if they do not have SK keyboard, just US. For example searching for: 'Obyvateľstvo' and 'obyvatelstvo' (population) or 'štatistika' and 'statistika' (statistics) should yield same results.

This should work the other way around as well, as some people might enter entries without accents into CKAN instance and others might search with accents.

For SK language simple ASCII transliteration for searched term and indexed words is sufficient (iconv ASCIITRANSLIT).

1294939588000000 1330990400000000
#907 enhancement rgrp ckan-v1.3-sprint-3 closed fixed Data previews: preview package resource data in the WUI

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

  1. Need a 'data proxy' - see ticket:698 and ticket:888
  2. Preview using javascript in WUI - do this in an extension http://bitbucket.org/okfn/ckanext-dataapi
1295261173000000 1297072303000000
#908 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Issues deploying extensions with modwsgi / uwsgi

Some extensions import pylons features (such as request) at root.

Have encountered following issues:

  1. You need to import ckanext before any ckan stuff in your wsgi script (extension stats and dataapi)
  1. Running under wsgi daemon mode (normal mode is fine!) the front page (front page only) will not work (does not occur with disqus or synchronous_search but does with stats and dataapi extensions). You get errors like this in the log:
[Mon Jan 17 09:01:19 2011] [error] Exception KeyError: KeyError(-1218594160,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Mon Jan 17 09:01:20 2011] [error] Exception AttributeError: "'NoneType' object has no attribute 'clearing'" in <bound method PluginEnvironment.__del__ of  Services for Environment '<default>'
1295264624000000 1296730578000000
#910 enhancement pudo pudo ckan-v1.5 closed invalid Refactor CKANrdf to support extras

Extra fields should be represented in some manner.

1295266026000000 1310128544000000
#911 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Add pager to dcat-tools/rdfsolr 1295266211000000 1296467375000000
#912 defect pudo pudo ckan-v1.5 closed wontfix Rethink result row representation in dcat-tools/rdfsolr
  • Should license go in the bottom line?
  • Formats should be styled as in CKAN
1295266299000000 1306774876000000
#913 enhancement pudo pudo ckan-v1.5 closed wontfix Handle licenses in dcat-tools
  • Create a list of all valid predicates where licenses are stored.
  • Create a license normalization registry in GSpread
  • Fill spreadsheet from OKD/OSI lists.
1295266407000000 1306774962000000
#914 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Handle formats in dcat-tools
  • Create a list of valid predicates for format information (accessURL et al)
  • Execute #235 via Google Spreadsheets and include in DCat enrichment
1295266569000000 1297069958000000
#915 defect pudo pudo ckan-v1.3-sprint-3 closed fixed Deploy dcat-tools/rdfsolr to publicdata.eu as a placeholder

We should present something there, why not start with a meta-search.

1295267040000000 1296467518000000
#916 enhancement rgrp pudo vdm-0.8 closed duplicate Port new vdm to Mongodb

We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG.

1295277011000000 1297066902000000
#917 enhancement kindly kindly closed fixed Turning off Autoflush in SQLAlchemy

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

  • Most of the time nothing will change. A commit implies flush.
  • You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
  • A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
  • You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

1295279694000000 1295280232000000
#918 defect memespring dread closed invalid Package preview doesn't show tags or groups
  1. Edit any package
  2. Hit 'preview'

Notice that the preview at the bottom doesn't show what tags or groups the package is in.

This is because the new design of the package view only has this info in the sidebar, which is not displayed in the preview.

One might argue that this is ok because the tag is not a central part of the metadata, only used for searching, so it is ok not to preview it. But I wanted to check that this change was a conscious decision.

1295346496000000 1315911507000000
#919 enhancement dread dread ckan-v1.4-sprint-6 closed fixed Package preview contains API & datapkg instructions
  1. Edit a package
  2. Hit 'preview'.

The preview contains the section "CKAN API / datapkg" which seems irrelevant at this point. Looking at ckan.net previews, the Comments section only appears when you view a package and not when you preview it. This seems more sensible - can this be done for the "CKAN API / datapkg" section too?

1295346730000000 1303202627000000
#920 defect thejimmyg hellmann@… closed fixed empty tags are kind of confusing

http://ckan.net/tag/linguistsic doesn't have a single entry, but is still kept and displayed.

It should be purged !!!

1295355280000000 1300319140000000
#921 enhancement sebbacon thejimmyg closed fixed Refactor DGU Harvesting

Refactor harvesting code to simplify.

1295389237000000 1300197629000000
#922 enhancement kindly kindly closed duplicate Make Resources first class entity.

Why?

  • Will make it easier to give resources their own form, independent of packages.
  • Makes uploading, harvesting and previewing data cleaner as its separate from the packages.
  • Gives the option of sharing a resource between packages.
  • Leads the way to supporting storing richer semantic information against them.
1295713858000000 1320664187000000
#923 defect rgrp dread closed worksforme Search box doesn't work in leaderboard page
  1. Go to: http://ckan.net/stats/leaderboard#content
  2. In the far top-right of the browser, select the search box in 'water'.
  3. Press enter to search. Nothing happens.

Tried in: chrome, firefox

1295867328000000 1340632144000000
#925 defect dread ckan-backlog closed fixed Change the search box icon to remove the down arrow

Is there a good reason why the search box has a 'down arrow' icon when there is no drop-down menu? Or can this be usefully removed?

1295867593000000 1323168588000000
#926 enhancement sebbacon sebbacon ckan-v1.4-sprint-1 closed fixed Pick a simpler form framework

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

Review existing (and potentially non-existing) frameworks with a view to porting forms over to something more explicit and lightweight.

Implement the current Package forms as an example of how this would work.

Document and circulate.

Sub-ticket of #961 (form, validation, model sync meta-ticket)

1295869056000000 1298541597000000
#927 enhancement rgrp rgrp ckan-v1.4-sprint-3 closed fixed [super] Improvements to CKAN documentation

List of improvements to CKAN documentation (result of ticket:904). Should convert most items into sub-tickets as we go along.

Major

Miscellaneous

  • Contributors / credits page: ticket:928
  • (Data proxy documentation: move it to packages.python.org/dataproxy)
  • (List dataapi extensions on http://ckan.org/wiki/Extensions (Name, summary string + point to pypi page ...))
  • Document license configuration and license system - ticket:973
  • Refactor trac frontpage moving general feature overview to ckan.org
  • Create proper roadmap information on http://trac.ckan.org/
  • Add link to wiki.ckan.net to ckan.org
1295869492000000 1300105638000000
#928 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Documentation - CKAN credits / contributors page
  1. Remove contributors list from about page (out-of-date and does not belong there)
    • A "Powered by CKAN" link in footer is enough
  2. Add contributors / team / credits page to ckan.org
    • May want to separate credits (e.g. software use), from contributors (also is team different from contributors)
    • (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket)
  3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info.

Cost: 1h

1295872661000000 1297115136000000
#929 defect rgrp rene.kapusta ckan-v1.4 closed fixed Handle the case when the licenses service is down better

I'm using v 1.3.1a and get a lot of "WebApp? Error: <type 'exceptions.Exception'>: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>" error messages.

Module ckan.forms.package:87 in build_package_form
<<      # Options/settings
           builder.set_field_option('name', 'validate', package_name_validator)
           builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
           builder.set_field_option('state', 'dropdown', {'options':model.State.all})
           builder.set_field_option('notes', 'textarea', {'size':'60x15'})
>>  builder.set_field_option('license_id', 'dropdown', {'options':[('', None)] + model.Package.get_license_options()})
Module ckan.model.package:283 in get_license_options
<<      @classmethod
           def get_license_options(self):
               register = self.get_license_register()
               return [(l.title, l.id) for l in register.values()]
>>  register = self.get_license_register()
Module ckan.model.package:278 in get_license_register
<<      def get_license_register(self):
               if not hasattr(self, '_license_register'):
                   self._license_register = LicenseRegister()
               return self._license_register
>>  self._license_register = LicenseRegister()
Module ckan.model.license:46 in __init__
<<              from licenses.service import LicensesService2
                   self.service = LicensesService2(group_url)
                   entity_list = self.service.get_licenses()
               else:
                   from licenses import Licenses
>>  entity_list = self.service.get_licenses()
Module licenses.service:44 in get_licenses
<<          except Exception, inst:
                   msg = "Couldn't connect to licenses service: %s" % inst
                   raise Exception, msg
               try:
                   licenses = loads(response_body)
>>  raise Exception, msg
Exception: Couldn't connect to licenses service: <urlopen error (104, 'Connection reset by peer')>
1295995409000000 1299840884000000
#930 defect wwaites dread closed fixed call_timing files created for every request

On a production server we produce literally millions of little files saying how long every single request took to process.

They are here:

hmg.ckan.net.2/pylonsdata/call_timing

They were added by ckan/lib/base.py in cset:da438a9085d3.

I don't believe anyone uses these stats.

1296061721000000 1297066292000000
#931 enhancement dread dread ckan-v1.4-sprint-1 closed fixed Search results generator hides paging functionality

ckanclient's search results list only packages up to the 'limit'. It would be good to return a generator instead of a list. When the limit is reached on the generator then another 'page' is loaded automatically.

1296147360000000 1298379187000000
#932 enhancement dread dread ckan-v1.3 closed fixed Move to SQLAlchemy-Migrate 0.6

Basic steps from kindly:

  • changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions.
  • running the migrate script on the repository (or changing the repository scripts manually).
  • checking in both these changes at once.
1296156660000000 1297076710000000
#933 enhancement kindly ckan-v1.4 closed fixed get rid of self when it should be cls in classmethods

use pylint

1296174526000000 1297348975000000
#934 enhancement kindly closed invalid Key value store for extensions to store data

We need to allow plugins to store information.

Each plug-in should have its own name space and allow any data to be set.

There are two options.

  • make a key value table in ckan that will hold random information.
  • use redis (or other key value store)

If in sql suggest schema of form:

  • (namespace, obj_id, key, value [, value_type])
  • value is json
1296228758000000 1323171047000000
#935 defect pudo pudo closed fixed "groups" field bug in IATI

An internal server error was reported when trying to create a new package (not when editing):

WebApp? Error: <class 'sqlalchemy.exceptions.IntegrityError?'>: (IntegrityError?) null value in column "name" violates not-null constraint 'INSERT INTO package (id, name, title, version, url, author, author_email, maintainer, maintainer_email, notes, license_id, state, revision_id) VALUES (%(id)s, %(name)s, %(title)s, %(version)s, %(url)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(notes)s, %(license_id)s, %(state)s, %(revision_id)s)' {'maintainer': None, 'name': None, 'author': None, 'author_email': None, 'notes': None, 'title': None, 'maintainer_email': None, 'url': None, 'state': 'active', 'version': None, 'license_id': None, 'revision_id': '0141c09c-4eb3-4dda-a6b1-e07f2c73b4c5', 'id': '7fe16a29-f183-471a-99c5-2c2e81dd86e4'}

The error could not be reproduced in a non-apache/mod_wsgi environment, even with a dump of the live database. Further investigation by manual editing of the package form left the "groups" field as the likely cause. After updating the packages installed, and temporarily editing the package controller the problem vanished after a restart of the web server.

Possible causes include:

  • Stale pyc file
  • Threading issue
  • Dependency issue

As the problem is gone for now, it cannot be further investigated atm.

1296302915000000 1306778682000000
#936 enhancement johnglover rgrp ckan-v1.4-sprint-6 closed fixed Follow / watch package extension

As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). Decision: use follow

Implementation

Interface

Become a follower:

  • Follow button on packages (if already watching say 'unfollow')

Package-related changes:

  • Show number of followers on a package
  • List followers of a package at /package/{name}/followers
    • On a separate page

a package User-related changes:

  • List followed packages
    • Either on user's page on a separate 'following' page. (NB: called 'following')
  • Does watching involve notifications (by email)
    • Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
  • [Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page

Nitty-Gritty

  • Want to do this in ajax-y manner
  • API endpoint: /api/2/follower
  • Store data in a new follower table

API

/api/2/follower

follow => PUT / POST
{
   user_id
   object_type
   object_id
}

If this is submitted by a user with user.id != user_id => error (401)

unfollow => DELETE

/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

Table

Called 'follower'

user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]

Random Extras

  • What about following users as well
1296339079000000 1303838713000000
#937 enhancement sebbacon rgrp ckan-v1.4-sprint-5 closed fixed Record download stats for resources (extension)

As a User (especially as a Package Owner/Maintainer?) I want to know how many times a resource has been downloaded (and when).

So let's record download stats (as in clicks on the link for a resource).

Implementation

Old Spec (do it in CKAN)

  • Record info of form: resource id (or url?), timestamp
  • Do this via javascript capturing of onclick event talking to an api
  • API: /api/resource/{id}/download
    • POST to increment (how do we stop spamming -- could use a nonce setup with a random string set on each page load for the js)
    • GET to get data back { total: X, day_count: [ [yyyy-mm-dd, count], ... ] }

Questions

  • Do we record ip addresses (to handle de-botting etc)?
  • Do we count preview clicks as well?
1296340273000000 1302513831000000
#938 enhancement pudo rgrp ckan-v1.4-sprint-1 closed fixed Message flashing / notification in WUI

Good user interfaces provide feedback to users. We should provide more feedback about the success / failure of actions using message flashing (see http://flask.pocoo.org/docs/patterns/flashing/).

We already do some of this (slightly poorly), e.g. on adding an authorization the app says: "X authorization added".

Implementation

  • Standardize this process and add section to html for this purpose.
  • Already done in other OKFN apps e.g. yourtopia.net, wheredoesmymoneygo and commentonit
  • Testing: suggest keeping this very simple

Estimate

  • Cost: 0.5h (html etc) + 1h to do some integration
1296398920000000 1297077023000000
#939 enhancement rgrp closed fixed Notification bar at top of site shown to new users pointing them to simple instructions

On http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

  • Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
  • FAQ/instructions page (use the new wiki?)

Depends

Estimate

  • Cost: 1h
1296399008000000 1323171158000000
#940 defect pudo rgrp ckan-v1.3 closed wontfix OpenID login via google for same user generates 2 different openid urls

This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h

1296401521000000 1297071748000000
#941 enhancement thejimmyg rgrp ckan-v1.5 closed wontfix Submit apps or ideas for apps related to datasets (extension)

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

1296403099000000 1310134339000000
#942 defect pudo pudo ckan-v1.4-sprint-1 closed fixed Fix SERP pager argument passing

It only passes "q", but the whole query must be serialized.

1296469490000000 1297074807000000
#943 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Create CKAN Wiki

Set up a Mediawiki to contain info on CKAN.net (community) conventions etc.

1296469547000000 1297076498000000
#944 defect pudo pudo iati-4 closed fixed Upgrade IATI to CKAN 'default'
  • Move code base, fix any breakage.
  • Deploy to eu8
  • Test
  • Deploy to eu14

Why

Later CKAN codebase has a bunch of improvements / fixes that it would be good to have.

Estimate

  • 8h
1296469688000000 1306774766000000
#945 enhancement kindly kindly ckan-v1.6 closed fixed [super] Richer resources - Resource Groups, new fields, improved UI

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

  • New resource group table. #956
  • New kind field in resource. #957
  • UI for new kind field. #958
  • Resources in REST API ticket:358
  • Resources in WUI #1445
  • Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

1296475283000000 1325259350000000
#946 defect pudo dread ckan-v1.3 closed fixed No default system rights

CKAN currently comes out of the box being uneditable. We'd like to encourage people to have open CKANs and therefore by default init the db with this right:

visitor -> is editor on -> System system
1296499922000000 1296833383000000
#947 enhancement rgrp closed duplicate Move the assignment of roles to a package on package creation to an extension

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

1296557478000000 1314031310000000
#949 enhancement pudo pudo ckan-v1.4-sprint-1 closed fixed Generic username/password login

based on repoze.who-friendlyForms

1296658126000000 1297074827000000
#950 enhancement dread dread ckan-v1.3 closed fixed Move importer controller to ckanext-importer

ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext.

1296666038000000 1297079743000000
#951 defect adrian.pohl@… closed invalid Can't add a package to group

I can't add a package (e.g. http://ckan.net/package/ub-konstanz) to a group (e.g. http://ckan.net/group/bibliographic). It's neither possible when editing a package (the only group in drop down menu is "history") nor on the group page.

1296726886000000 1314031006000000
#952 task wwaites pudo ckan-backlog closed invalid CKAN should run under nginx/uswgi

second part of #908

1296730498000000 1310125204000000
#953 enhancement rgrp rgrp ckan-v1.4-sprint-1 closed fixed Add tagline/description to template and set in configuration

Have a tagline / description are under main title and set if from ckan.site_description config variable.

1296809808000000 1296809834000000
#954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3

Child tickets:

  • #1107 Move package autocomplete from package controller and move to API
  • #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

1296811899000000 1320142744000000
#955 defect dread dread ckan-v1.3 closed fixed Tidy up tests
  • Removal of init_db from every setup etc.
  • Get them working
1296819300000000 1297342534000000
#956 enhancement kindly kindly ckan-v1.4-sprint-2 closed fixed Add a resource group table.

We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package <o2m> resource_group <o2m> resource

The use cases to this are outlined in this email. http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

1296821112000000 1299489084000000
#957 enhancement kindly kindly closed wontfix new "kind " field added to resources

The new field was outlined in

http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.

1296821398000000 1314218701000000
#958 enhancement kindly ckan-v1.5 closed duplicate ui for new kind field

Ui for related new field in resource (#957)

1296822896000000 1320664462000000
#960 defect pudo rgrp ckan-v1.4-sprint-1 closed fixed Support non-ascii character in internal error messages

Just saw an error like the following which looks like it is to do with having a user object with non-ascii characters in it. Either get rid of 'cast' to str type here or do it in a unicode aware way.

Should do this not just here but everywhere we can find in the code base.

Module ckan.controllers.package:302 in edit
<<          am_authz = self.authorizer.am_authorized(c, model.Action.EDIT, pkg)
               if not am_authz:
                   abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
       
               auth_for_change_state = self.authorizer.am_authorized(c, model.Action.CHANGE_STATE, pkg)
>>  abort(401, str(gettext('User %r not authorized to edit %s') % (c.user, id)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Estimate

Cost: 1h

1297069721000000 1297169056000000
#961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

  1. Generating (or just filling) a form template with 'form data' (and errors)
  2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
  3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

  • #926 - Pick a simpler form framework
  • #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
    • #1079 Refactor API to use new logic layer and dictization
    • #1078 Refactor WUI controllers and forms to use logic layer
    • cf existing dumper and importer code
    • This will fix #662
  • [not ticketed yet] - validation layer (should work on serialized objects?)
  • #662 - Can't put entity that is returned by posting to package register (Defect)
  • #972 - Merge 'extras' into main package dict rather than having separate key
  • #1035 - Form impressions are given IDs
  • #810 - Move "add packages" field up in group form (easier to do this once forms are done)
1297069849000000 1310126100000000
#962 enhancement rgrp rgrp ckan-v1.4-sprint-5 closed fixed Improvements to data preview extension

Basic implementation done (and deployed):

https://bitbucket.org/okfn/ckanext-datapreview

However plenty to improve, e.g.

  • Support more formats (use external systems for preview?)
    • json (!)
    • html (trivial!)
    • sparql
    • ...
  • Do not display preview if no preview

Also suggest reworking to use external services rather than doing preview 'in house' (doing in house places heavy reliance on data proxy service and on converting data to a standard format).

1297072524000000 1301364987000000
#963 enhancement thejimmyg rgrp ckan-v1.4-sprint-1 closed fixed Package CKAN as a debian package

Package CKAN as a debian package so it is easier to install.

1297076364000000 1298284252000000
#964 enhancement rgrp rgrp datapkg-0.8 closed fixed Create Resource Downloader and make it pluggable

In downloading packages we need to download resources and we need a ResourceDownloader? object for this. these should be pluggable so that we can add support for different types of resources.

1297211004000000 1297211237000000
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Note: See TracReports for help on using and creating reports.