{22} Trac tickets (2647 matches)

We need an incremental plan that connects with current state of DGU and reflects what is actually required by UKLP.

Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

• Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
• Should we have clever ways to dump APIs? ANS: no.
• Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
• Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
  • Complete support for ETags
  • Expires, Max-Age etc.
• Check

Functionality

• Download files via HTTP, HTTPS (will not do FTP)

Process:

1. [Archiver.Update checks queue (automated as part of celery)]
2. Open url and get any info from resource on cache / content-length etc
   1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
   2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
   • Add cache url to resource and updated date
   • Add other relevant info to resource such as md5, content-type etc
4. Update task_status

Optional functionality

• If result object is HTML, search for references to "proper data" (CSV download pages etc.)
• Download from POST forms (accepting licenses or weird proprietary systems)
• Support running on Google Apps Engine to save traffic costs.

Existing work

• ​https://bitbucket.org/okfn/ckanext-qa/overview
• out of date: ​https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental.
• out of date: ​https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc

second part of #908

This will involve running a sample of real requests synchronously against real data.

It's not good to have ckanext doing lots of different things with different dependencies. Split it off into:

• ckanext-importlib

etc.

And then deprecate the ckanext repo itself.

Only tests with failing --ckan-migration fail, due to authz not being initialised.

Upload to ckan.net: ​https://sites.google.com/site/scotlandsdata/

As part of the general documentation overhaul (ticket:1142) we (APS and RGRP) want to convert the current Sphinx docs into:

• An Admin Manual which is task-based and aims to cover everything a developer would need to know to set up and customize a CKAN install.
• A Reference Manual which is the primary source of reference for CKAN software - this includes API docs.

The current chapters of the Sphinx docs should be moved as follows:

• index.rst - copy some stuff from README.txt, keep short
• README.txt - split out installation - stop symlinking in, keep separate, write new intro in index.rst
• [NEW] install.rst - new file on installation
• [NEW] theming.rst - move over from wiki
• api - will stay (gradually move tutorials/getting started to wiki.ckan.net user guide)
• i18n.rst: internationlization. say we have x langauges. just set lang config option. if your lang not there yet see wiki page for how to prepare a translation file
• design.rst - REMOVE (can copy some over if you can be bothered to ​http://wiki.ckan.net/Vision (all philosophical stuff should go!))
• loaders.rst: move to wiki.ckan.net/Using_Loaders
• feeds.rst: move to User Manual (CKAN_Feeds or just Feeds)
• importer.rst: remove (don't even copy)
• forms.rst: remove (ping david read and ckan-dev asking for replacement documentation -- do we need this in core ckan -- do we want pure js ...?). Suggest we start with (stub out) ​http://wiki.ckan.net/Customizing_Forms
• forms-integration.rst: remove
• model.rst: leave but move to reference (only for core devs)
• load-testing.rst -> move to ​http://wiki.ckan.net/Load_Testing
• distributed.rst -> remove
• admin.html should consolidate with authorization.html (some of authorization.html is probably in ref section but howto in main manual) <-- high priority
• deb.html -> go into reference (query james gardner on list about moving this to wiki?)

Here is Rufus' plan for ckan.org reorganisation from ​http://ckan.okfnpad.org/documentation (22/7/11) (ticket made by dread):

/solutions
  /overview <--- current 'about' page (the software)
  /customers <-- say "in addition to our clients..."
  /features <-- needs most work (pictures etc)
/pricing/  <-- simplest thing possible ... talk with Ira (cover pricing and support)
/developers <--- links to http://docs.ckan.org/, getting the software, the wiki, the mailing list /contact
/blog

Tasks:

• add tagline in header - "the data portal software"
• make and add glossy PDF
• change "data hub" to "data portal" passim (if only using one term)
• add info@…
• image banner on home page - redo screenshots without menus

Compose a feature list for CKAN. Use the existing Feature Matrix. Basic explanation of terms. For use by customers.

Reasons: more people familiar with Git, bigger Open Source community on Github.

Keeping track of the move here: ​https://docs.google.com/spreadsheet/ccc?key=0AvCDqUH8jVN8dGNVNEpXLWk3UTlJZXhFMjBPbXVtRmc&hl=en_GB#gid=0

Because of #1514 we should just disable name changing, until #1514 is done.

We have 150 odd branches (git branch -a) - most of them old - we should prune them. At very least, branches that have been merged in should be deleted. Look at old branches that haven't been merged in and wonder why.

May be of some use:

git branch --merged
git branch --no-merged

Time estimate: 2d

• Fix logs (apache, ckan, harvest): rotate, set suitable levels
• Fix harvesting jobs: supervisord for gather consumer, cron job
• Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

Need to setup server in tenforce test server.

Tasks include:

Estimate 4d

• Update requirements (git)
• Search facets error
• Error on dataset read (extras in context)
• Url generator for rdf
• Make templates work with latest theme
• Use dicts in templates (isopen())

Various stuff related with release 1.7:

• Review patches sent to the release branch
• Regularly update TDH with latest changes
• write changelog, announce email, etc
• ...

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

• [ ] Write up python coding standards as discussed at dev-meetup into CKAN docs. (Mostly written-up as part of dev-meetup notes.)

• [ ] Collate frontend coding standards (​http://aron.github.com/ckan-style/styleguide/) into same doc(s).

The GRANT SELECT ON ALL and ALTER DEFAULT PRIVILEGES are not available in Postgres prior to version 9.0.