{22} Trac tickets (2647 matches)

From Toby: make the context auto-populated with things like user/model/session if not user supplied - especially for extensions (pre 2.0) maybe via a helper or else a decorator on the action - helper seems nicer maybe done via get_action()

From Toby: make lib/base.py be more like it was originally intended eg defines stuff that is shared eg _ but maybe try to include stuff like render/get_action if the circular import stuff can be solved - which should be doable if hard and likely needs an extra file or two for the externals _ etc

WUI:

• Editable as part of package or separately? (e.g. like authz)
• Do we normalize to only one type name of the pair?
• Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?)

Storage represents something you can upload to (download is done in ticket:406).

• storage sub package with defined interface (maybe just use OFS)
• Only support uploading single files for the moment (though can do multiple files).
  • Relationship to build (ticket:306)
  • No support for syncing e.g. hg/git repos -- that's up to you as a user
• Upload command

At end of this we can deprecate Repository.

Having Repository encapsulate Index and Storage (without Storage being explicit) is not helpful and leads to duplication.

Split various jobs up into separate scripts. Change ONS importer to run remotely. Run ONS remotely. Run dump locally.

Requested by DGU.

Allow using different indexes, including ckan, file, dcat, ...

As a

user browsing a CKAN package which has a resource in a bad format

I want to

be alerted to the possibility of scraping it on scraperwiki

Implementation

This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it
Image: (Scraperwiki dumper truck)
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can ​request this data is scraped or even ​write the scraper yourself.

This would be an extension to the disqus plugin that would patch itself into home/index.

Requirements:

• cater naturally for multiple packages with multiple resources
• international encoding
• large number of packages

When performing maintenance on CKAN it may be necessary to make CKAN obviously read-only, telling the users and restricting access to 'edit' pages.

Examples of use:

• Administrator wants to upgrade CKAN or move it to another server. During this time the database is being administered and either edits are lost or can't be done.
• A CKAN is used just for distributing metadata and so is always read-only. Updates may still arrive through direct db manipulation, e.g.:
  • another (but writable) CKAN instance is connected to the same db
  • restoring database dumps from another CKAN db
• Should a security be breached, all editing could be stopped

CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache.

No need for requirement - closing in favour of #505

cf. Tariqs mail

Updated version: ​https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0

Original Version

Publishing Entity:
Publishing Entity Type: (Donor, Recipient, Community Data..)
Donor
Country
Activity period:
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated:
License: Need this field even if it may be a standard license

So naively mashing these together, we get something like:

Identifier
Title
Abstract
Donor
Country
Publisher
Publisher Type
Verification Status
Department
Contact
Contact e-mail
Licence
Resource format
Resource URL
Resource ID
Activity period
Number of activities
Date record updated
Date data updated

• Use what is on iatiregistry.org at the moment.

Read/write APIs for users and publishing entities to access and register data

• Users have read access
• Publishing entities have both (though can only write to 'their' records)
• Priority: 4 for write (read access is 3)

• So some kind of confirmation or approval by someone more senior?
• Alternative: just have an account request.
• Priority: 3

CKAN uses a database. We can put the database on another machine and then clone the CKAN machine. Then there are two machines calling the database and converting SQL results into HTTP responses.

How many machines can you have before the database becomes the bottleneck? At that point, what QoS can be obtained with normal hardware?

Afterwards, can CKAN be made to scale further than this with a little development? Can its Postgres database be mirrored in realtime? Or can we change SQL writes (create/updates) to write to many databases?

Are there any other options for scaling the persistence mechanism?

This is in IATI, would be nice to have in generic CKAN.

• ​http://bitbucket.org/okfn/iati/changeset/378431974c76

• copy exisiting tests, modify authz in setUp, adapt and extend tests
  • Problem: default_role_actions is read by init_db
  • Solution: nuke db after monkey-patching
  • role = model.Role('Reader'), del role.actions[...]

self.PRE_MAUTHZ_RULES = copy(mauthz.default_role_actions) mauthz.default_role_actions.remove((Role.READER, Action.CREATE)) #raise Exception(mauthz.default_role_actions) model.Session.remove() model.repo.rebuild_db()

3. Start from the functional
   • Both the api controllers and wui controllers to do
   • tests/function/test_authz.py extended to check CREATE in lockedDown mode. fails nicely. ​http://bitbucket.org/pudo/ckan-authz2

is_authorized(user, Action.Create, model.Package)

-> Doing this will put 'Package' in the context field of the user_object_role table. This will trigger SQLAlchemy to attempt a join towards PackageRole? in all queries. Since for class-level role assignments there never is a PackageRole? join table entry, this will never return any results.

• cf branch "classes": ​http://bitbucket.org/pudo/ckan-authz2/src/6fd0475e0c66
• No possibility of including "instance" column in inheritance decision.
• Even if this can be cirumvented, it must happen on a per-query level and would require a major re-modelling.

• have a ckan install that would not allow visitors to either list packages or list groups
  • two paths: create the listing, but for each group/pkg decide that you cannot show this
  • lock down the whole page (/package/list)
    • this is class-based, not object-based

is_authorized(user, Action.Package_Create, model.System()) is_authorized(user, Action.Group_Create, model.System())

[Separating Package and Group roles may be useful going forward: PackageEditor?, GroupEditor? etc]

4. Find a standard way to lock down classes
   • possibly add default rows in user_object_role
   • introduce lock-down into controllers: group new, package new, REST equivalents,
   • confirm tests

Much more natural place to do this, even at the risk of overloading the package edit form.

As an admin I want to check a CKAN instance works having just upgraded it or configured it.

Wish to fold service records into package resources. Trouble is that service records don't necessarily refer to a data record, so should this be a new package?

To do this for the UKLP, the Ordnance Survey would need to add to their map service an operation to return image for a given bounding box, and COI would need to present the image on the search result page.

Otherwise, it would be possible to use Google.

This seems to require doing something else with such a group's 'delete' button, such as showing it greyed, or perhaps not showing it at all.

Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please?

This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider.

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

A link “About CKAN” pointing to ​http://ckan.net/about should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is.

I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar.

But anywhere else would be fine as well.

There are some problems with the Resources table on package pages:

1. The column labelled “URL” doesn't contain URLs, but just the word “Download”.
2. The links labelled “Download” often are not download links, but something else (example links, API endpoints, documentation links).

Attached is a screenshot for a proposed redesign. The changes are:

1. Move the Format column to the right of the Description column (description is more important)
2. Make the description a clickable link
3. Drop the “URL/Download” column because it's now redundant
4. Rename “Description” to “Resource” (not that important)

... or, if the CKAN packages do contain architecture-specific binary code, build packages for i386 too.

Currently, ​http://apt-alpha.ckan.org/debian only offers packages for amd64, but e.g. "m1.small" EC2 instances are i386.

We would need this in order to migrate the community instances to a packaged based CKAN.

Rufus, pls prioritise.

Child tickets:

• #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

Requirements

We need to clone the demo ODS site for DataGM as soon as we have a mostly working demo so that the DataGM team can try it out and comment/feedback on functionality.

Interface

None

User Stories

None

Tasks

[ ] Deploy demo DataGM site.

Estimates

With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is:

• the fanstatic_resources defined in ckan/lib/fanstatic_resources.py (bottom of module) should be prepared, and minified.

This ticket is to:

• provide a paster command which when run will generate the minified javascript as css files.
• stop the auto-minification of files when CKAN starts up.
• remove reference to the minified files in the .gitignore file.
• add instruction to the release process to run this command, and check-in the minified files to the repo.

## Background

Currently, minification works seamlessly without the need for any preparation when CKAN is started in a development setup. But on a production site, the webserver will (almost certainly) not have write-access to the directories that will contain the minified files. And so the minification will fail, and the site will end up serving the un-minified media, or even *old* minified media.

One way around this would be to allow webserver write access to the directory its serving out of. But this is not generally considered good practice.

Another method would be to distribute the minified files with CKAN. This ticket describes how to do this without causing a lot of noise in the commit history of the repo.

The auto-minifcation occurs when importing ckan/lib/fanstatic_resources.py.

## Changing the process slightly

The reason for moving away from auto-minifying files at start-up, to minifying files when running a paster command is:

• we should be distributing the minified files when we make a release (as we do with translation files). This makes it easier to install CKAN from source.

• if we distribute the minified files, then they need to be checked-in to the repo.

• if they are auto-minified, whenever small changes are made, then this will create a lot of noise in the repo. It will be clearer to have the minified-files generated manually as part of the release process. (Or whever we make a deployment internally).