{22} Trac tickets (2647 matches)

Results (1001 - 1100 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#826 enhancement kindly dread ckan-v1.4-sprint-1 closed fixed Resource 'extra' fields

There are a number of extra properties we want to store in a resource:

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)

Implementation

I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra? code as much as possible. (Other much more radical alternative is to move model to RDF.)

1290611905000000 1306766057000000
#841 enhancement kindly dread ckan-v1.4-sprint-4 closed duplicate Caching docs (as a whole)

Documentation article on caching / improving performance. (To complement configuration docs.)

  • Different sorts of cache - beaker style, etags, package_dict in search results(?)
  • How each one affects performance
  • How to turn them on/off and configure them
  • Is it possible to bypass each of them in the browser or with wget/curl?
1291308879000000 1300364333000000
#890 enhancement kindly pudo ckan-v1.6 closed invalid Introduce timed actions into ckanext-queue

The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows:

  • routing_key: Package
  • operation: daily
  • payload: pkg.as_dict()

Where operation is one of daily, weekly, monthly or any other interval.

1294050769000000 1318599247000000
#917 enhancement kindly kindly closed fixed Turning off Autoflush in SQLAlchemy

David Raznik: I propose we remove autoflush by default in ckan. It caused a lot of bugs on the version upgrade due to its magical behavior.

I have a patch in https://bitbucket.org/kindly/ckan/ on branch feature_switch_autoflush_off_by_default.

This change simplifies code. It only took 7 extra flushes/commits to make work and there are over 40 instances that we remove autoflush, so as not to cause error (I have not removed them in the patch yet).

Things to know about using flush:

  • Most of the time nothing will change. A commit implies flush.
  • You only need to flush if you expect a query to return an object you just saved. i.e if you session.add(obj) and you expect obj to be in a session.query.
  • A flush is handy when you want to get the primary key out before you do a commit. i.e session.add(obj); session.flush(); obj.id is now generated.
  • You can use them as a limited form of nested transaction.

It increased the performance on the tests by about 5%. There are less flushes altogether so is faster.

I do not see any downsides.

1295279694000000 1295280232000000
#922 enhancement kindly kindly closed duplicate Make Resources first class entity.

Why?

  • Will make it easier to give resources their own form, independent of packages.
  • Makes uploading, harvesting and previewing data cleaner as its separate from the packages.
  • Gives the option of sharing a resource between packages.
  • Leads the way to supporting storing richer semantic information against them.
1295713858000000 1320664187000000
#945 enhancement kindly kindly ckan-v1.6 closed fixed [super] Richer resources - Resource Groups, new fields, improved UI

Super ticket: #1032

This is a meta ticket for changes that are going to happen in resources.

  • New resource group table. #956
  • New kind field in resource. #957
  • UI for new kind field. #958
  • Resources in REST API ticket:358
  • Resources in WUI #1445
  • Make Resources first class entity. #922 (duplicate?)

Background on this change can be found at:

1296475283000000 1325259350000000
#954 enhancement kindly rgrp ckan-v1.5 closed fixed [super] API version 3

Child tickets:

  • #1107 Move package autocomplete from package controller and move to API
  • #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

1296811899000000 1320142744000000
#956 enhancement kindly kindly ckan-v1.4-sprint-2 closed fixed Add a resource group table.

We need a way to group resources by various different factors. The most important of which is timeseries.

The model will look like

Package <o2m> resource_group <o2m> resource

The use cases to this are outlined in this email. http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

1296821112000000 1299489084000000
#957 enhancement kindly kindly closed wontfix new "kind " field added to resources

The new field was outlined in

http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.

1296821398000000 1314218701000000
#961 enhancement kindly rgrp ckan-v1.5 closed fixed [super] Refactoring of forms, validation and model synchronization

This is a meta-ticket to hold all of the work on refactoring forms, validation and model-synchronization in CKAN.

ckan-dev thread: http://lists.okfn.org/pipermail/ckan-dev/2011-January/000180.html

The Issue

From #926:

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

From http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html:

... FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved [in this area]:

  1. Generating (or just filling) a form template with 'form data' (and errors)
  2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
  3. Converting form data to model data (and validating) (inverse of previous step)

Related Tickets

  • #926 - Pick a simpler form framework
  • #1046 'dictization' and the logic layer - serialization / deserialization of package (and other domain objects) to standard intermediate format such as json-convertable python dict
    • #1079 Refactor API to use new logic layer and dictization
    • #1078 Refactor WUI controllers and forms to use logic layer
    • cf existing dumper and importer code
    • This will fix #662
  • [not ticketed yet] - validation layer (should work on serialized objects?)
  • #662 - Can't put entity that is returned by posting to package register (Defect)
  • #972 - Merge 'extras' into main package dict rather than having separate key
  • #1035 - Form impressions are given IDs
  • #810 - Move "add packages" field up in group form (easier to do this once forms are done)
1297069849000000 1310126100000000
#965 enhancement kindly kindly ckan-v1.4 closed fixed make migration testing plugin to nose

In order to test migrations properly, we need to test the upgraded database against *all* the tests.

To do this a special flag to nosetests should be added. nosetests --migrations

This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db.

1297213492000000 1298280637000000
#979 enhancement kindly dread ckan-backlog assigned Edit Resource extras in the API

Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API.

1297429777000000 1315222244000000
#981 defect kindly kindly ckan-v1.4-sprint-2 closed fixed fix authorization_group_user.id so it's added in migrate

This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up.

1297434480000000 1298278654000000
#984 defect kindly kindly ckan-v1.4-sprint-2 closed fixed Fix migration, where upgrade diverges from actual model.

The migration scripts have not been inline with the upgrade scripts. There are lots of discrepancies.

1297627447000000 1298280689000000
#989 enhancement kindly pudo ckan-future new Extending the model from plugins

We need to support extending the model from plugins. This could involve:

  • Adding a plugin hook to extend the mapper
  • Adding an upgrade hook for plugin schema migrations
  • Documenting how this is to be done
  • Find a way to avoid conflicts
1297689724000000 1340034311000000
#994 defect kindly kindly ckan-v1.4-sprint-3 closed fixed document a new migration methodology

It is too easy to make a mistake with the migrations at the moment. A more systematic way of doing them is needed and this needs to be documented.

1298281192000000 1298912842000000
#995 defect kindly kindly closed wontfix consolidate caching

Try and minimise the amount of different caching methodologies, especially for api calls.

1298283499000000 1311179009000000
#996 task kindly kindly ckan-v1.4 closed fixed Run some basic load testing.

This will involve running a sample of real requests synchronously against real data.

1298283994000000 1300364398000000
#997 defect kindly kindly closed duplicate change create on cli to upgrade

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

1298287981000000 1298288665000000
#998 defect kindly kindly closed fixed change create on cli to upgrade

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

1298287981000000 1300364423000000
#1000 enhancement kindly rgrp ckan-v1.4-sprint-3 closed fixed Remove call timings code

Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

  • lib/base.py: c.time_call_started
  • ....
1298489643000000 1298912726000000
#1006 enhancement kindly rgrp ckan-v1.4-sprint-4 closed fixed Deprecate stable branch

Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time).

Cost: 10m (giving high priority because of low cost)

(Assigning to dread as he has been managing the stable branch).

1298624996000000 1300372286000000
#1012 defect kindly kindly ckan-v1.4 closed fixed Add package revision history to api

Revision history information is not accessible, dgu want this.

1298887351000000 1301943113000000
#1015 defect kindly dread ckan-v1.4-sprint-3 closed fixed Editing group gives exception

Editing a group properties on ckan.net gives an exception when you submit the form. (Is this related to db migration issues?)

Example page: http://ckan.net/group/edit/civil-society

Exception:

WebApp Error: <class 'sqlalchemy.exc.IntegrityError'>: (IntegrityError) duplicate key value violates unique constraint "group_revision_pkey" 'INSERT INTO group_revision (id, name, title, description, created, state, revision_id, continuity_id) VALUES (%(id)s, %(name)s, %(title)s, %(description)s, %(created)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'description': u"A group for open data related to civil society supported by members of the interest group at: http://okfn.org/groups/civil-society\r\n\r\nCivil society is composed of the totality of voluntary civic and social organizations and institutions that form the basis of a functioning society, as distinct from the force-backed structures of a state (regardless of that state's political system) and commercial institutions of the market. \r\n\r\nhttp://en.wikipedia.org/wiki/Civil_society", 'created': datetime.datetime(2011, 2, 24, 14, 30, 53, 334842), 'title': u'Civil Society', 'state': u'active', 'continuity_id': u'f4f9f09
1298898588000000 1299788821000000
#1046 enhancement kindly thejimmyg ckan-v1.4-sprint-5 closed fixed Dictization and the new logic layer

The stages involved with doing this.

  • Convert model objects to standard dict format (DONE)
  • Convert standard dicts to current api formats (DONE)
  • Make standard dicts savable (DONE)
  • Validate standard dict format. (DONE)
  • Authorize actions
1300364694000000 1302777668000000
#1054 defect kindly dread ckan-v1.4-sprint-4 closed fixed Ordering of resources

Changing an old resource and creating a new resource on the end results in the old resource moving to the end in the ordering.

This breaks tests:

  • (ckanext-dgu) ckanext/dgu/tests/ons/test_ons_loader.py:TestOnsLoadBasic.test_fields
  • (ckanext-importlib) ckanext/importlib/tests/test_loader.py:TestLoaderInsertingResources.test_0_reload

You can make a ckan test break with this patch:

diff -r e6643cf1324c ckan/tests/models/test_resource.py
--- a/ckan/tests/models/test_resource.py        Wed Mar 23 13:25:52 2011 +0000
+++ b/ckan/tests/models/test_resource.py        Wed Mar 23 19:22:35 2011 +0000
@@ -297,6 +297,8 @@
                'url':self.urls[1], 'format':u'OTHER FORMAT',
                'description':self.description, 'hash':self.hash,
                'id':original_res_ids[2]},
+            { #new
+                'url':'new'},
            ]
        pkg.update_resources(res_dicts)
        model.repo.commit_and_remove()

There seems to be a problem with vdm creating a replacement Resource for the old resource - because it has a duplicate position it is put to the end by the SQLAlchemy ordering_list function.

1300969236000000 1301305615000000
#1077 enhancement kindly rgrp ckan-backlog new Move to simpler vdm system

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

  • The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset

for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

  • does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

  • Easier support for pending state and similar behaviour
  • No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

  • Migration is required
  • More difficult to query revision history.
    • Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
  • Performance (?)
    • Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
      • Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

1302304464000000 1340034345000000
#1078 enhancement kindly rgrp ckan-v1.5 closed fixed Refactors WUI controllers and forms to use logic layer
  • Deserialize forms to new dict format.
  • Replace controllers/forms to use dictization.
1302509347000000 1305828973000000
#1079 enhancement kindly rgrp ckan-v1.4-sprint-5 closed fixed Refactor API to use new logic layer and dictization
  • Convert current api saves to the new standard dict format.
1302509530000000 1302777504000000
#1092 defect kindly kindly ckan-v1.4-sprint-6 closed fixed refactor logic layer to seperate out api, form logic

The logic layer is a bit too api centric. Make the reusable parts separate in preparation for the wui refactor.

1302777929000000 1305570822000000
#1095 defect kindly kindly closed fixed add way to pass in schema to logic layer.

We need a way to pass in schemas to the logic layer to deal with edge cases.

1303221854000000 1310134959000000
#1109 defect kindly kindly closed fixed When extras has a value other than a string an integrity error occurs in the api.

This is a regression that happened after refactoring the api.

It was shown by

http://pastebin.com/2v7QasZy

1303839943000000 1305124697000000
#1110 enhancement kindly kindly closed wontfix profile ckan

We need to see what areas of ckan are slow.

1303840041000000 1340034394000000
#1113 defect kindly kindly closed fixed lists in extras serialized wrongly on get with the api.

Lists are being converted to unicode and then translated into a json when getting from the api.

1304017353000000 1304024611000000
#1135 enhancement kindly rgrp assigned Changeset model for vdm

Move to Changeset model for vdm.

A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned.

This change would also incorporate significant simplication of vdm.

1305209986000000 1340632267000000
#1136 enhancement kindly rgrp assigned Move to SessionExtension in vdm

When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that SessionExtension? exists we should use it. We can also follow the existing SQLAlchemy recipe: <http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>

1305210855000000 1340632980000000
#1137 enhancement kindly rgrp assigned Remove need for statefulness in vdm

Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is required because, atm, revision objects have FKs to continuity objects.

This ticket proposes the following changes:

NB: this could be limited just to case of join tables (leaving state stuff on other tables)

  • Remove FKs from revision to continuity (or allow for them to be nullable).
    • We could just limit this to m2m stuff
  • Delete of an object leads to:
    • Deletion of continuity object
    • Adding an entry in revision table with state set to deleted (we retain state on revision table)

If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain "active" relationships.

If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state).

The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects).

1305211628000000 1340631974000000
#1146 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed make logic layer control its own state

Logic layer should not use any vdm defined state and should manage it itself.

1305829117000000 1307957527000000
#1147 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Add expired_id to all revision tables.

Revision tables need expired_id to make querying history AND pending changes more efficient.

This involves making a session extension and a large table migration.

1305839833000000 1307957556000000
#1148 refactor kindly kindly ckan-v1.5-sprint-1 closed fixed test speed improvements and cleanup

The tests have been running slower recently and need fixing. They also could do with a bit more consistency to them.

1305969223000000 1305969925000000
#1149 enhancement kindly kindly ckan-v1.5-sprint-1 closed fixed Change domain object modification plugin to use Session extension.

This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.

1305969863000000 1306090663000000
#1193 enhancement kindly kindly closed fixed bug with new formalchemy causing all resouces to be deleted

The new formalchemy stores dict in a new format.

1308525655000000 1309768960000000
#1205 defect kindly dread closed fixed db upgrade problem on cz.ckan.net

When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade).

cz.ckan.net is not working properly until this is fixed.

okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File "/usr/bin/paster", line 18, in <module>
    command.run()
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command
    model.repo.upgrade_db()
  File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange
    change.run(self.engine, step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run
    script_func(engine)
  File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade
    migrate_engine.execute(q)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute
    params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context
    context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey"
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}
1309339700000000 1310568631000000
#1211 enhancement kindly kindly ckan-v1.5 closed fixed [super] Drupal integration

Sub-tasks / tickets:

  • #1335 Action API - access to any logic layer function
  • Standardize logic functions on ordering: (context, data_dict) and put parameters into data_dict.
  • Add package_create_validate & package_update_validate logic functions
  • Allow extensions to provide logic layer functions (IActions)
1309958442000000 1315948703000000
#1231 requirement kindly thejimmyg ckan-backlog closed wontfix [super] Management Information Reporting

Child tickets:

  • #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

1311173919000000 1325474447000000
#1238 defect kindly dread ckan-sprint-2011-10-28 closed invalid Viewing a package @date misses non-PackageRevisions

There's a problem with getting old versions of packages using @date. The query looks for the PackageRevision? related to the date, but ignores TagRevisions? etc.

So for example if you compare this table: http://ckan.net/package/history/osm with this view: http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627830 you actually see the 2010-01-13 11:13 revision - the wrong revision.

1311351303000000 1311759784000000
#1240 enhancement kindly rgrp ckan-backlog assigned [super] API v4

(Just creating this ticket as somewhere to keep notes)

  • Decide on REST api versus action API
    • Do we want to support both?
  • Tidying
    • Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
    • extras merged into normal field list in package
    • Get rid of /rest/ so just have api/v1/package
    • Get rid of separation of search api from 'rest' api
      • Propose that GET on REST index is search e.g. /package/?q=...
        • This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
  • Resource read/write in API (separate from package)
    • Does this need authorization work?
  • user/account API - read/write
  • Remove autocomplete -- can just use search
    • Do not worry about backwards compat as should only be used in our js (if others using it too bad!)
1311525660000000 1325473312000000
#1255 enhancement kindly kindly ckan-backlog assigned Drupal consistancy checks.

Make a robust way to make sure the drupal database is consistent with the ckan data.

1312219968000000 1313400054000000
#1256 defect kindly kindly ckan-sprint-2011-10-28 closed fixed fix modification extension for moderation

Solr is not working for moderation, pending changes are not being indexed.

1312280026000000 1319812556000000
#1258 enhancement kindly kindly ckan-sprint-2011-10-28 closed fixed Bring purge revision into ckan repo from vdm

In order to make purge revision work correctly with the moderated edits we need to modify purge revision in vdm. This is best modified in ckan so we will override the vdm one in the reposotory.

1312289539000000 1319812452000000
#1268 enhancement kindly rgrp ckan-sprint-2011-10-10 closed fixed Speed up listing users by revisions by adding db index

At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?

1312906592000000 1318279592000000
#1291 enhancement kindly kindly ckan-sprint-2011-09-12 closed fixed Add activity and activity detail tables.

Add tables described in http://ckan.okfnpad.org/notifications

1314267572000000 1315950394000000
#1298 enhancement kindly kindly ckan-sprint-2012-01-09 closed fixed Generate activites to be put into activities table.

This should be done from the logic layer or automatically from a session extension.

1314696442000000 1325591582000000
#1306 enhancement kindly rgrp ckan-backlog closed wontfix Tests for the logic layer

AFAICT there are no tests for the logic layer at the moment. I imagine this is an issue (if it is not please explain and close as wontfix).

1315153267000000 1319797204000000
#1316 defect kindly dread closed fixed /api/action uncaught exception if missing necessary parameter
curl http://ckan.net/api/action/package_show -d '{"name": "uk-quango-data"}'

This gives 500 error due to KeyError? (it looked for the 'id' param), rather than 400.

1315397224000000 1338193724000000
#1324 enhancement kindly kindly ckan-sprint-2011-09-12 closed fixed Add extra fields to resource table

add extra fields to resource as outlined in

http://wiki.ckan.net/Domain_Model/Resource

Specifically (all unicode strings except for those noted):

  • name: a name for this resource (could be used in a ckan url)
  • type: the type of the resource. One of: file | api | service | listing
  • mimetype: standard mimetype (e.g. for zipped csv would be application/zip)
  • mimetype_inner: mimetype of innermost object (so for example would be text/csv)
  • size: [integer] size of the resource (content length) in bytes. Usually only relevant for resources of type file.
  • last_modified: [datetime] the date when this resource's data was last modified (NB: not the date when the metadata was modified).
  • cache_url: url for cache of object in ckan storage
  • cache_last_updated: [datetime] timestamp when cached version was created
  • webstore_url:
  • webstore_last_updated: [datetime] timestamp when webstore was last updated
1315578976000000 1315948336000000
#1335 enhancement kindly dread closed fixed Action API - basic

Provide an API to the logic layer at /api/action.

1315910021000000 1315910117000000
#1339 enhancement kindly rgrp ckan-backlog closed fixed Issues / question re navl and data conversion

I ran into a bug with the size field on resources.

  • It would not accept an empty value from form (IMO this clearly equates to null/None)
  • This could be fixed via using ignore_empty instead of ignore_missing
  • However using this means there was no way to empty the field (e.g. i may just want to set the size field back to null not just change to another value)
  • similar issues could arise around other fields (such as last_modified ...)
    • cf cset:645031d07b60

To solve this (cset:58acdcfe6d4e) i created an int_converter temporarily in logic/schema.py (this is almost certainly the wrong place). But I think it raises a bigger issue about the conversion layer and how it works.

1315947022000000 1316014954000000
#1341 enhancement kindly kindly ckan-backlog reopened Delete spam users from ckan

Spam users where added to thedatahub and we need to clean them.

1315995034000000 1320141540000000
#1344 enhancement kindly kindly ckan-sprint-2011-10-10 closed fixed datetime error json conversion on search

Json decoding error on search, due to date in resources.

1316020195000000 1317211299000000
#1345 enhancement kindly nils.toedtmann closed fixed Investigate possible memory leak

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (ticket) we experienced extraordinary memory usage peaks (ticket). Here are the observed value with Apache default settings:

  • eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
  • s055, mpm-prefork: base level ~1GB, peaks up to 4GB
  • s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (changeset). This (together with two other tweaks) changed the situation drastically:

  • s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

1316041450000000 1343215489000000
#1356 enhancement kindly amercader ckan-sprint-2011-10-10 closed fixed Can not recreate a deleted extra

If you delete an extra and later on change your mind, you can not recreate it with the same value (Different value works fine).

1317034180000000 1318279617000000
#1364 enhancement kindly kindly ckan-sprint-2011-10-10 closed fixed If package in index that is not in ckan error is caused.

We should ignore these but log the packages that are causing this to investigate.

1317222206000000 1318279636000000
#1376 enhancement kindly kindly ckan-sprint-2011-10-10 closed fixed 404 raised when approving package

Regression found by updating datacatalogs.

1318088589000000 1318279651000000
#1383 enhancement kindly kindly ckan-sprint-2011-10-24 closed fixed Add IResourceChange IResourceUrlChange Interfaces.

Need to add way for extensions to trigger based on resource change and just resource url change.

1318342153000000 1320142305000000
#1397 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Resource archiving

We want to cache/archive data associated to a resource so it is available if the resource url disappears (and in order to support other processing we may wish to do e.g. webstorer ...)

Etherpad: http://ckan.okfnpad.org/queue (most relevant parts inlined here)

Preliminaries

  • Add task_status table to store qa/archiever/webstore information that does not need to be versioned. - #1363 (and #1371 - related logic functions)

Configuration setup for daemons

Pass config through to workers i.e site_url, user, api_key. Need to make site user account. #1408

celeryd config:

All providers of tasks will add an item to the following entry point:

[ckan.tasks]
name = ckanext.{name}.tasks:....

celeryconfig.py

from pkg_resources import iter_entry_points
for entry in iter_entry_points:
     celeryimports.appen(....)

CELERY_IMPORTS = celeryimports

Work Items

  • Resource change notifications in core - Make an IResourceChange and IResourceUrlChange. [1d] [0.75d] - #1383
  • Generate archiving request on resource url change [0.25d][0.25d] - #1399
  • Make site user account.
  • Make entry point system for celery config
  • Archiver daemon #891
    1. implement link-check function and task (point 2 from Archiver.update above) [1d] [0.5d]
    2. Rewrite archiver to use external storage. (decide how!)[3d][~2d]
  • Write to resource and task status table.[1d][0.75d]
  • [Required?] Make archived data available in WUI - #892
  • Documentation - #1400
1318529593000000 1338202920000000
#1398 enhancement kindly rgrp ckan-sprint-2011-11-21 closed fixed Automated conversion of resource data into webstore
  • Possible sibling / requirement: #1397 (resource archiving)
  • Setup test environment. 1d
  • Make task to upload files to webstorer. 3d
    • This includes writing back webstore_url and updated date. 0.5d
    • Type guessing of file format. 1d
    • Make cron to upload all un webstored resources. 0.5d
  • Testing for errors by running cron 2d
  • Make nice way to deploy celery 1d.
  • Documentation. 1d
  • Deployment. 1d
1318529724000000 1321827970000000
#1402 enhancement kindly rgrp ckan-v1.6 closed fixed Migrate repository from mercurial to git

Plan to migrate from mercurial git

Process:

  • Do trial run
  • Announce conversion date / time
  • Require everyone to have pushed all outstanding changes at that time
  • Do conversion
  • Test
  • Announce on list and ckan.org/
1318811651000000 1324334011000000
#1408 enhancement kindly kindly ckan-sprint-2011-10-24 closed fixed Make site user, so tasks can post information back to ckan.

Give site user system administrative rights.

1319019626000000 1320141847000000
#1433 enhancement kindly rgrp ckan-sprint-2011-11-21 closed fixed Support SQLAlchemy 0.7

Why: current stable version of sqlalchemy. geoalchemy stuff required 0.7 and likely that some other things will require it soon.

Probably requires work on vdm https://bitbucket.org/okfn/vdm

NB: should have discussion before making 0.7 the default required version in CKAN core.

1320143453000000 1324472583000000
#1447 defect kindly dread ckan-backlog assigned disk space leakage

Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25.

1320666843000000 1340727330000000
#1448 enhancement kindly kindly closed fixed Set up nice way to do celery deployment.

Celery is awkward to deply, need to find a way to do it more simply. i.e using celery-pylons and supervisor. A modified version of celery-pylons may be the best solution. 1d

1320666977000000 1325774155000000
#1475 enhancement kindly kindly ckan-sprint-2012-01-23 closed fixed Create documentation showing how to make an extension that use celery. 1321826681000000 1327576068000000
#1477 enhancement kindly kindly ckan-v1.6 closed fixed [Super] Group refactor

This is the implementation of the final draft in

http://wiki.ckan.org/Groups_Refactor

Steps to complete this will follow (somewhat) in the following order.

Change package_group table to membership. (#1478) 3d

  • model migration. (done)
  • simplify revisioning (done)
  • Remove any sqlalchemy releations from groups. (done)
  • change logic functions (done)

Single type per group (#1531)

  • migration add type to group table. (done) 1d

Add capacity table (#1522)

  • model migration. (done)
  • add to logic functions.

Allow other objects to be added to groups. (#1531)

  • change logic functions (done)

Adapt thedatahub.org to use new model.

  • move auth groups over to groups/members
  • change user_object roles to use groups? keep user object roles for time being.
  • make create update package/group put the correct items in the group/members/capacity tables.
  • work out how ui works.

--not doing the below as part of this super ticker now #1669.

Create model for dgu.

  • adapt auth functions to use auth.
  • adapt package_create/update group_create/update to fill in the correct capacties/members.

Adapt Iati, and potentially other custom instances.

Sub-tickets:

  • #1279 Deprecate Authorization Groups
  • #1198 Publisher hierarchy
1321959320000000 1328016209000000
#1478 enhancement kindly kindly ckan-sprint-2011-12-05 closed fixed change package_group to members table

This will comprise of the following steps.

  • model migration.
  • simplify revisioning
  • Remove any sqlalchemy relations from groups.
  • change logic functions

estimate 2d

1321959973000000 1323172654000000
#1485 enhancement kindly kindly closed fixed Package/Group form extension mechanism so you can add forms for particular package_types

We want to be able to change form depending on package type or group type.

This is dependent on a type field being added to the Package and the Group.

1322059169000000 1340034422000000
#1492 defect kindly dread closed wontfix Interference between extra and relationship fields in API

From Jan Kučera <elquenor@…>

I have also tried to use extras fields to store information about relationships between datasets. I found out that if you first add relationship using CKAN API into dataset relationships attribute and than you add extras field for the same relationship with key in format "parent_of:dataset_name" or "child_of:dataset_name", storing this extras field will clear all relationships in relationships attribute stored earlier.

I might be some kind of bug but I suppose you will know better. Is it possible to use relationships attribute and extras field for relationships at the same time?

1322481496000000 1340034528000000
#1495 enhancement kindly seanh ckan-sprint-2012-01-23 closed fixed API call for getting a user's public activity stream

An API call that returns a user's public activity stream in JSON format.

1322495549000000 1327576099000000
#1511 enhancement kindly seanh ckan-sprint-2012-01-23 closed fixed Logic function for getting a user's public activity stream

Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.

1323094207000000 1327576134000000
#1522 enhancement kindly kindly ckan-sprint-2011-12-19 closed fixed Add capacity to member table.

Need to add capacities to member tables.

1323172610000000 1324333827000000
#1531 enhancement kindly kindly ckan-sprint-2012-01-09 closed fixed Update group create/update so you can add capacities and group types.

The new members table needs a way so you can add arbitrary domain objects against them.

We need to extend the group schema to accept types, and instead of just being able to add packages to groups add members with their capacities that associated with different table rows.

4d

1323272500000000 1326155226000000
#1595 defect kindly rgrp ckan-v1.7 closed wontfix Editing of a Group leads to recording a change against all associated datasets

Or at least it appears that way from revision log which displays all Group datasets against the changelog, see http://thedatahub.org/revision/4cdeeb42-3281-4f53-a29d-c694e1fd9217 -- not apparent from that page but see my user page: http://thedatahub.org/user/rufuspollock

This may turn out to be a UX bug rather than a real bug ...

Assigning to kindly for review and comment.

1324504875000000 1338203959000000
#1605 enhancement kindly rgrp ckan-v1.7 closed fixed [super] Multilingual support in CKAN

Multi-language:

  • dataset and resource metadata (and other objects such as groups?)
  • Field values in taxonomy (e.g. country names - Eurovoc)
    • #1665 Research into Eurovoc
    • Display Taxonomy in different languages
  • Field values not in taxonomy (e.g. title & description)
    • Use extra fields e.g. _i18n_title_fr = Le data.
    • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)
  • EC extension (templates, form)
    • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?
1325473367000000 1338193641000000
#1612 enhancement kindly kindly ckan-sprint-2012-01-09 closed fixed Group view page slow

Group show that lists packages is slow due to not using query in pagination.

1325633737000000 1325688886000000
#1614 enhancement kindly kindly ckan-sprint-2012-01-09 closed fixed remove po files from git diff

Its a pain to see the difference between branches as there are normally a lot of po file transaction. Make the default be see that they have changed without actually show the diffs themselves.

1325686639000000 1325689136000000
#1652 enhancement kindly ross ckan-backlog assigned How we intergrate with Drupal Multiligual?
  • Analysis of how/where we can integrate with Drupal Multilingual
  • Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/...
  • Check with Ian for other issues.
1326709894000000 1340187535000000
#1689 enhancement kindly dread assigned List deleted datasets in API

The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets.

'state' could be a parameter on the package_list call

  • Easy to call and implement
  • But getting permissions right for the different state values may not be easy - maybe a separate logic function is better?
1327062214000000 1340190040000000
#1715 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Search index multilingual

Need to make solr schema work for many languages. Get stopwords and choose correct analysis for each.

1327598884000000 1329393450000000
#1736 enhancement kindly kindly ckan-v1.7 closed fixed [Super] Multilingual Metadata

Translate all areas of metatdata to selected language. This includes finding datasets in all languages.

Multilingal Metadata: Tasks:

  • Add translation table. 2d
  • Api to add translation 3d
  • Add extention point to dataset view. 3d
  • Add extension point to search index to add translations. 2d.
  • Add all eurovoc data to translations. 1d.
  • Make multilingual solr schema 2d #1715
1328012697000000 1338205016000000
#1738 enhancement kindly kindly ckan-sprint-2012-02-06 closed fixed Add extension point to search index, so can modify dataset dict before index.

It is useful for extensions to add or change items that go into the search index. Add an extension point for this.

1328031572000000 1330086105000000
#1739 enhancement kindly kindly ckan-sprint-2012-02-06 closed fixed Fix issues where tests where not run.

Test failures in master due to tom and rufus not running tests. 0.3d.

1328047129000000 1328495826000000
#1741 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Add extention point to dataset view.

Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown.

Tasks include:

  • Fix up data_dict and templates so that the pkg objects do not get used in the templates. This is to assure that all data passed is modifiable. (2d) (complete)
  • Add extension point to Ipackage controller (1d)
  • Test (1d)
1328105709000000 1329750838000000
#1753 enhancement kindly rgrp ckan-sprint-2012-03-05 closed fixed Last fixes of Webstorer + Deploy on the DataHub 1328527932000000 1330990061000000
#1777 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Write about multilingual features for ec.

Basic description of multilingual features intended to be supported.

1328542835000000 1329393295000000
#1779 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Add multilingual translation table.

A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination.

1328548631000000 1329393759000000
#1780 enhancement kindly kindly ckan-backlog closed duplicate Api to add translations.

Add api to translation to the term_translation table.

  • add to logic 2d
  • test 2d
1328548908000000 1328578479000000
#1781 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Api to add translations.

Add api to translation to the term_translation table.

  • add to logic 2d
  • test 1d
1328549031000000 1329393822000000
#1819 enhancement kindly kindly ckan-sprint-2012-04-02 closed fixed change search index code to use package_dictize instead of obj.to_dict

Search index code needs to use package_dictize so that it can get the new vocablurary information and is more consistant with the rest of the system. This is in preperation for their translation being put in the search indexer.

1329770555000000 1338193886000000
#1820 enhancement kindly kindly ckan-sprint-2012-03-19 closed fixed Index multilingual data when mulilingual extension is added.

All translated fields and vocabularies need to be added to search index in the correct fields.

1329770903000000 1332163374000000
#1821 enhancement kindly kindly ckan-sprint-2012-03-05 closed fixed create internal multilingual extension that brings together all multilingual features

This extension point needs to combine.

  • Making sure the correct data get put in search index.
  • Make the search queries are modified to weight multilingual search correctly.
  • View pages, including search results are translated.
  • Bring together all search schema modifications.
1329771267000000 1330990111000000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Note: See TracReports for help on using and creating reports.