{22} Trac tickets (2647 matches)

Currently the default authz for a package is hard-coded to:

 <PackageRole user="visitor" role="editor" context="Package">,
 <PackageRole user="logged_in" role="editor" context="Package">,

This should be configurable in the config, so that you can have a more locked down instance etc.

When a user visits the site, he/she may only see a login box.

It's handy to be able to create users using the cli (e.g. dgu migration)

Report on errors by any worker daemons, send them out via E-Mail

We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package <o2m> resource_group <o2m> resource

The use cases to this are outlined in this email. ​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

I have a requirement to allow an external source to define the groups of which a user is a member.

I propose to create an IAuthzUserGroups plugin interface that allows an extension to arbitrarily extend the list of AuthzGroups? that a user is in.

We propose the following push approach using individual packages:

1. New, updated and deleted packages are pushed to a url endpoint on Wordpress (WPURL). This endpoint will be a configuration option on the CKAN side. A POST or PUT http method will get used. The payload will be a json document of the following form:

{

payload: The entity data as available from the REST API entity-type: 'Package', operation-type: 'create'|'update'|'delete' # one of these options

}

It will be sent as the body of the request with content-type set to application/json.

2. The wordpress side will provide a 200 on success. Any other response will be taken as a failure. On failure, submission will be archived and failure logged and notified to system administrator. Submission can be resent later automatically by sysadmin after review.

3. Pushes will happen continuously and approximately simultaneously with updates (a webhooks type model)

4. [optional] CKAN side will support configuring authorization for basic authentication if applied on WP side.

5. [optional] List queue status (including failures) on ckan adminstrative dashboard.

• examples of posting a new package
• example license_id - explain

Looking at: ​http://ckan.net/package/rkb-explorer-acm

'rkb-explorer-resex' gets display multiple times and shouldn't

Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

• lib/base.py: c.time_call_started
• ....

We're currently setting the user preferences links via a javascript snippet that also evals the name. This should be removed as we're not displaying the user name any longer.

We should also have page fragment caching in Genshi, which is not currently implmented.

cf. ​http://api.rubyonrails.org/classes/ActionController/Caching/Fragments.html

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

• Implement as an Extension
• Locate at /upload
• Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
• Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
• This wants to be tied in to general package workflow. See parent ticket:852.

Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end.

Here are the details:

• in CKAN's package view (in web interface, API and dump) no resources display
• but they WERE created in CKAN, as shown by the revision diffs.
• they are in the Drupal front-end

So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal.

Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism.

​http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py

This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed.

In order to test migrations properly, we need to test the upgraded database against *all* the tests.

To do this a special flag to nosetests should be added. nosetests --migrations

This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db.

At the moment we only use X-forwarded to determine remote user address when remote-addr header is 127.0.0.1. However if a site is behind a cache remote_addr will always be the IP of the cache.

We should fix this by using x-forwarded-for header whenever available and only use remote-addr when it is not available.

Cost: 10m

Should have the following features:

• A list of all packages with broken links on the resource
• Under each packages a description of any resources with broken links to include:
  • Link
  • Description
  • Format
  • First found to be broken date

[If you can't do the last one yet, don't worry]

Allow choice of language on ckan instance.

• Configuration variable listing language choices (could try doing this automatically but recommend against this)
• Switcher option in UI (perhaps on user profile page)
• Default it automatically based on browser settings (?)

Think this is high value (and cost is low)

Cost: low (4h)

Document how license system works and specifically how licenses can be configured.

Cost: 2h

• db_upgrade (in ckan/model/init) makes call to validate_authorization_setup but forgets about rest of initialisation.

• it uses self.metadata.bind.url when it should be self.metadata.bind it seems (confused by sqlmigrate update?)

use pylint

Currently placeholder text like 'Search ...' is not shown in some browsers (e.g. FF and IE).

This leads to poor UX, for example in top bar search where it unclear whether that box is for login or search.

We should fix this by finding a way to display placeholder attribute in all browsers.

• Removal of init_db from every setup etc.
• Get them working

Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h

In downloading packages we need to download resources and we need a ResourceDownloader? object for this. these should be pluggable so that we can add support for different types of resources.

Replace use of pastescript templates in PythonDistribution? (write) with of something simpler (allowing us to remove dependency on pastescript).

1. Remove contributors list from about page (out-of-date and does not belong there)
   • A "Powered by CKAN" link in footer is enough
2. Add contributors / team / credits page to ckan.org
   • May want to separate credits (e.g. software use), from contributors (also is team different from contributors)
   • (perhaps with photos!) so that people can see who is part of it and from which organisations. (this is medium-term, not really part of this ticket)
3. ? CREDITS.txt file to the source repository and add a link to it on bitbucket for more detailed credits info.

Cost: 1h

E.g. ​http://ckan.net/tag/lod contains musicbrainz_ (Musicbrainz) item (if you click you will get 403/login as deleted packages can only be seen by admins).

Also suggested that these are showing up in the API (courtesy of Anja)

Cost: 2h

ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext.

Because CKAN's worth it!

Basic steps from kindly:

• changing the setup.py dependencies for sqlalchemy-migrate and sqlalchemy to their latest versions.
• running the migrate script on the repository (or changing the repository scripts manually).
• checking in both these changes at once.

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

based on repoze.who-friendlyForms

• (Load cli commands from setuptools entry point name datapkg.cli)
• Document how other can write new commands
• Refactor base Command class to support this pluggability (if necessary)

Allow using different indexes, including ckan, file, dcat, ...

This has been noted twice in the last couple of weeks.

Checking around my guess is this is due to someone logging in to www.ckan.net and ckan.net. Google generates openid based on the RP (requesting party) which is the requesting domain url unless the openid.realm is set. See:

• ​http://groups.google.com/group/google-federated-login-api/web/the-most-important-technical-issue-in-using-the-google-accounts-api?pli=1
• ​http://stackoverflow.com/questions/3549953/how-to-get-unique-identifier-from-openid-google

To fix this we should set the openid.realm (not sure how atm).

If this is not possible an alternative is to make sure we redirect all users to a single site url (perhaps using ckan config option site_url, or doing this in apache).

Cost: 1h

See message and script: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-September/000569.html

1500 requests shouldn't take hours.

We should create a port of VDM 0.9 for MongoDB to support user editing in WDMMG.

At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)

On a production server we produce literally millions of little files saying how long every single request took to process.

They are here:

hmg.ckan.net.2/pylonsdata/call_timing

They were added by ckan/lib/base.py in cset:da438a9085d3.

I don't believe anyone uses these stats.

Have a tagline / description are under main title and set if from ckan.site_description config variable.

Issue 1. Help command without arguments fails:

$paster help --plugin ckan --config development.ini

gives:

EITHER

Command help not recognized

OR

paster help: error: no such option: --plugin

Issue 2: Harvester command with arguments fails.

$ paster harvester Traceback (most recent call last): ...

cmd = self.args[0]

IndexError?: list index out of range