{22} Trac tickets (2647 matches)

need controller action and front-end method

reported as editing datasets incorrect

e.g. clicking on edit here ​http://s031.okserver.org:2375/dataset/ff takes you to create dataset page ​http://s031.okserver.org:2375/dataset/edit/ff

but this is the correct behaviour of a draft dataset

We need to show draft datasets correctly

proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc

Problem: Example group ( ​http://s031.okserver.org:2375/en/group/example-group ) has a broken image, image not found. Link either incorrect, or no image set, if no image set then shouldn't it show a default image? Expected: There should be an image.

There is some bizarre language flag issues. 1) Log in to ​http://s031.okserver.org:2375/ (I'm logged in as danieljohnlewis) 2) Visit the apps page: ​http://s031.okserver.org:2375/apps 3) Have a look at the languages drop down, flags look fine (although Deutsche does have an EU flag rather than the German flag.. but that's irrelevant) 4) Change filters to, say, type: Application and sort: Newest, and apply. Web address becomes: ​http://s031.okserver.org:2375/apps?page=1&type=application&sort=created_desc 5) Look at the languages drop down: Flags are largely incorrect, with more than one country having the same flag.

Please see the attached screenshot.

Note: this was originally tested with Google Chrome on Linux (Ubuntu).

the title for additional info should be translated

capitalised etc

Problem: In the English version (which has a UK flag, indicating British English), the word "Visualization" is used. For an example see the "Filter by type" drop down on the /apps page. Expected: This should be "Visualisation" in British English. Any instances of "Visualize" should be changed to "Visualise" too.

Problem: When creating a related item (e.g. a Visualisation), if you don't put in a URL it succeeds, but on the related items and apps pages it renders it as a link to the same page. Expected: Always require a URL and it should only submit if one is added

Over-enthusiastic removal of some model items

this breaks qa extension

3. Uploading a file behaves counter-intuitively (I would suggest wrongly).

When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen:

• the pathname of that file is filled in the box;
• nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc).

Instead of this,

• the file is immediately and, seemingly, irrevocably uploaded;
• the box is filled with a mysterious URL (​https://commondatastorage.googleapis.com/ckan-demo ...)
• the mysterious URL is so long I can't even see the final element, which is my file name.

The 'What are Groups?' text at <​http://demo.ckan.org/group> is a bit naff. Here is a suggested replacement:

What are groups?

Groups allow you to group users and data together so that they are easier to manage. For example datasets in a group could be visible only to group members, or publicly visible but only editable by group members.

Groups can be used to collect together datasets from a single publisher, or related to a particular project, for example.

Get ckan working on heroku

If you make your browser window short - say half screen height on a page with delete eg ​http://localhost:5000/dataset/edit/1-kmwaterfractionfromnationaltopographicdatabasemapscanada

the dialog box is partially off the screen - it should try to stay on screen

e.g. currently no way to read full description for the 'data explorer examples' group here:http://s031.okserver.org:2375/group/data-explorer

Can we add a 'read more' link that expands and contracts the sidebar box or similar?

on resource pages (e.g. ​http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360), same as old ckan for continuity and to highlight!

This is what we need the JSON to look like. Taken from the Jed documentation (​http://slexaxton.github.com/Jed/).

{

      // The empty string key is used as the configuration
      // block for each domain
      "" : {
        // Domain name
        "domain" : "the_domain",

        // Language code
        "lang" : "en",

        // Plural form function for language
        "plural_forms" : "nplurals=2; plural=(n != 1);"
      },

      // Other keys in a domain contain arrays as values
      // that map to the translations for that key.
      // Gettext suggests that you use english as your keys
      // in case the key isn't found, and it can just pass
      // the value directly through.
      // Note: by convention, the 0-index location of the translations
      // is never accessed. It's just a thing, I guess.
      "a key" : [ null, "the translation", "the plural translations", ... ],

      // The plural form string is converted into a function
      // and the value that's passed into the gettext call
      // is passed into the plural form function. It's result
      // (plus 1) is the index that the translation comes from.

      // We're using sprintf interpolation on our keys so we can
      // then sub in the _actual_ values into the result.
      "%d key" : [ null, "%d key", "%d keys" ],

      // Contexts are keys that are just prefixed with a context string
      // with a unicode \u0004 as the delimiter.
      // You can use it for anything. Usually it's just for being content aware
      // in some way (e.g. male vs. female, product vs. category)
      "context\u0004%d key": [ null, "context %d key", "context %d keys" ]
    }

This looks like an appropriate python library to do the formatting. ​http://pypi.python.org/pypi/pojson/ but I'll leave to your discretion.

Finally an API endpoint that returns the appropriate request would save us making requests to the i18n directory.

/api/i18n/{locale}

For example if the current url is ​http://localhost:5000/pl/dataset then the client would request:

http://localhost:5000/api/i18n/pl

This should return the Content-Type: application/json and a 200 with an appropriate JSON body (this can be the language requested or an appropriate fallback). I think that if the fallback is the default language (in most cases English) then we just return a 204 No Content to save data.

be good to do this in phase 3

this want to

a) be a link on the left of the form like deletes

b) needs a confirm like delete

c) wants to work with the autoform

A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data.

​http://s031.okserver.org:2375/dataset/resource-markdown/resource_edit/ba1b094d-697c-47c4-a72a-6c2557da5409

should have back to resource as a minimum

​http://s031.okserver.org:2375/ language dropdown expands downwards it should open upwards

select2 can do this see ​http://ivaynberg.github.com/select2/#documentation can you see if we can get this in ckan demo

This is a very slow call it would benefit from the sort of speed-ups that package-search received

for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization

can we just grab json blobs from solr?

also is it possible to specify a sort order/paging?

I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

e.g. if I click on download here: ​http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b can it take me to a new tab instead of navigating away from ckan?

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

use ​http://leaflet.cloudmade.com/ properly to generate map backgrounds for demo instances

Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?

apps dashboard available at ​http://localhost:5000/apps in main dev branch

I have got this to work in the demo however

a) it is a bit ugly

b) the template could do with some sorting eg the dropdowns should be being created in a nicer way

to use on demos instead of the ckan logo & tag line.

Design here: ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381 (First, text only one at the bottom)

We need to be able to do the following:

• Rip the localised strings from the JavaScript? source code. These are strings that look like _('String'), translate('String') and isPlural('String') .
• An endpoint that provides the translations in the correct locale for the user.
• The endpoint should provide the locale as JSON (possibly using ​http://jsgettext.berlios.de/doc/html/po2json.html or similar tool)

The JavaScript? can then read this file on page load.

Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first

Luca De Santis reported these strings not marked in 1.7.1:

• /user/login.html ("Remember me" & "Sign in")
• /package/search.html and template/package/group/read.html (Tags, Resource Formats, Groups)
• /package/resource_read.html (the "No description" message)
• the "Add related item" dialog (some strings are i18n savvy, others not)
• the stats page (labels on the graphs)
• the history page (as far as the log messages are concerned).

lets check that they are marked in 1.8

so that data proxy works

The new ckan creates datasets as part of a 3 phase process. To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'. These datasets should not be seen as active by the activity stream.

If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created. When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created

We do not want these adding. Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it.

finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream.

The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there.

Let me know if you need any help with this ticket or a better explanation of the problem.

Definitely do:

• Local file filestore: Remove pairtree (and OFS) and do something very simple

Options:

1. Local file only. Allow for uploaders to GS / S3 in the background
   • Advantages: simpler, if local upload can do progress bars etc etc
   • Disadvantages: strain for web-app (upload a 2GB file what happens)
     • This is probably fixable ...
2. Stick with how we are

New filestore without pairtree

Each uploaded file is located on disk at:

{uuid}/{filename}

Alternative:

yyyy/mm/dd/{uuid}/{filename}

Need the uuid to avoid collisions.

Metadata: Store no metadata.

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

see spacing between 'download' button and 'data api' button ​http://s031.okserver.org:2375/dataset/afterfibre/resource/66f5b980-06db-4ccd-a005-c18de4d68a4b

We get an error which should be improved the actual problem is this but not passed to user

⇝ KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

It doesn't pass extras_as_string

Specifically the 'add resource' button and the 'add item' button in related section

When navigating to ​http://s031.okserver.org:2375/group on the demo server, the Group link in the navbar is not highlighted in the same was as the other links are when visiting those pages.

we need to ensure that translations in .js files are ripper for translation

look at what is available as preference to creating our own

When editing a resource you see the current and any other existing resources in sidebar

(e.g. see ​http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page

Hey, this is still missing. It's the one with the add dataset text

​http://s031.okserver.org:2375/dataset/new

license dropdown - when open we need some line spacing at least in FF I think it'd be nice to fix this in phase 2

​http://localhost:5000/ckan-admin/authz

you need to be an admin

paster sysadmin add <username> -c dev.ini

like #2741 we need other admin links

you may want to make the inputs use your form macros - I've just done a basic port

also we will need to get all the tables done but looks like we need to get auth groups working for that

When adding a resource, the breadcrumb should say "add data" or "add resource" not "create dataset" as you see here: ​http://s031.okserver.org:2375/dataset/new_resource/frere

Why is datastore not enabled for this resource ​http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd ?

Is this going to be a site wide setting? How do new resources get added to datastore? Or are we just using the dataproxy for demo?

We need to check the archiver to make sure it works as we would expect after the changes in #2732:

• Receive notification of a file that needs processing
• Check the file's DB entry and upload the file to the configured remote service
• Update the URL of the file (both in the file's DB entry and the resource)
• Once we're sure the file is safely archived, we should mark the file as being archived

so that we know it can safely be deleted at some point.

Maybe this could/should be implemented as a service rather than a celery task?

Steps to reproduce:

1. go to edit a resource, e.g. ​http://s031.okserver.org:2375/dataset/ted/resource_edit/a621a6ad-6f1c-46d1-b858-03cdb7a25c07

2. Change the format field to something it doesn't get. e.g. add a comma "HTML,"

3. Get message: "The form contains invalid entries:

Last modified: Date format incorrect Size: Invalid integer"

4. Format field reverts, try to re-save/submit form - i.e. click on "update resource"

5. nothing happens (same if you even try changing the format field

We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is ​http://ckan.okfnpad.org/uploads

Simplifying uploads

Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

• File uploads themselves
• Storage of uploaded files
• Notification of the upload to other components

File uploads

Things file upload should do:

• Allow sysadmin to disable
• Allow auth'ed users to upload
• Store whatever they send on disk, and store DB entry linking the file to the person
• When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files.

Can we do the upload asynchronously and then associate the 
uploaded key with the resource before the save ? What happens 
if the user tries to submit before asymc upload finishes ? Should 
we delay them?

The upload workflow should look like...

1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.
2. 1. When called via ajax then the ID of the newly created file should be returned,
   2. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource.

This should enable:

• Separate file upload into a user's temporary store, either individually or as a batch.
• Creating resources and simply choosing from previously uploaded, unassigned files
• Adding files/data to a resource after the fact.

File storage

File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request.

File storage should:

• allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths).
• provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up.
• allow for collection of information (i.e. estimate of storate space used)
• check whether there is enough space and handling the conequences cleanly
• ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
• Have a configurable maximum accepted blob size during upload.
• Should store what meta-data was provided with the upload, such as mimetype.

Somewhere in the DB we should store ...

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.

Notifications

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).