{23} Trac comments (3729 matches)

Results (1401 - 1500 of 3729)

← 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 →
Ticket Posixtime Author Newvalue
#826 1297415095000000 pudo I want to strongly support david in his call for fully flexible extras: one of my use cases for them is to store the various bits of fallout from an archiving process, such as: * last-status * last-crawled * last-etag * last-expires * last-md5 * failure-count * fall-back-url These are things needed to really archive the data well, but they have nothing to do with CKAN core ops. Essentially, the archiver is a seperate concern and it need not appear in the CKAN config.
#826 1297416879000000 kindly There is nothing to stop anyone from putting any extra attributes in the extra_info field dict. So any have the flexibility you need. The config option is to add some fields that act in exactly the same way as python attributes, having the same semantics as them. i.e if you have an extra field called alturl, you can do obj.alturl = 'fdsffs'. This is the best of both worlds as far as I can tell.
#826 1297417900000000 kindly I forgot to mention that he main advantage of the fixed fields, is that we can make them properly searchable i.e the values searchable. This currently does not work for package extra values as they are jsons. I have added this searchability for the sql backend.
#826 1297421022000000 dread Ah - I understand. I'd like it if you could make all the extra fields work as attributes and be searchable - is that possible?
#826 1297423342000000 kindly This would be too much of a hack. You do not want users overwriting any attributes on the object. If they called the attribute "__init__" it would write over the actual __init__.
#826 1297429910000000 dread Merged into default in cset:613d7bd5fc96. Future tickets in this area include: * #978 Full edit in Web UI * #979 Edit Resource extras in the API
#826 1306766057000000 dread This went into ckan release 1.3.2.
#827 1290766239000000 rgrp Fixed in cset:63a0c0223da4
#828 1290696586000000 rgrp Closed in cset:66a3efe16ed4
#829 1290697386000000 dread Some work towards this in cset:066b1b553595. No exceptions and test skip in place. Looks like formalchemy changed its interface.
#829 1303118864000000 thejimmyg Is this still relevant?
#829 1303122056000000 dread Rufus originally specified it. Reassigning to him to decide whether we still want it or not.
#829 1303838115000000 rgrp This is a wontfix for the present.
#830 1311180263000000 thejimmyg We can now implement themes via extensions or change them via config options. See https://bitbucket.org/okfn/ckanext-exampletheme/ for an example.
#831 1290760864000000 rgrp Done over the last week (finally fixed weird trac no __dict__ bug today and enabled user accounts on wednesday).
#832 1296334980000000 rgrp Done (~4w ago). See https://bitbucket.org/okfn/ckanext-stats and remove from core cset:311313e4afdb.
#833 1295263447000000 rgrp Have repo https://bitbucket.org/okfn/ckanext-admin
#833 1298889104000000 rgrp In progress now (sysadmin view and update nearly done).
#833 1302276855000000 rgrp Authz subsystem complete.
#833 1303236364000000 rgrp Two main tickets done so closing.
#834 1291633657000000 rgrp Alexander: Documentation is here: http://packages.python.org/ckan/api/version2.html#search-api Those queries should work but I do suggest you use the stable version of CKAN which is v1.2 and we suggest you use that. If you still have problems please reopen.
#835 1291644820000000 pudo done in cset:d41b77099954
#836 1291851041000000 rgrp Fixed in cset:2f6d54341b47 and branch feature-286-siteurl
#836 1302882808000000 dread This has been a (minor) issue since release v1.3
#838 1291299716000000 memespring Package page redesign: http://ckan.org/ticket/839
#838 1291636351000000 memespring Merge css files http://ckan.org/ticket/846
#838 1291721955000000 memespring Show welcome message: http://ckan.org/ticket/850
#838 1291729819000000 memespring Prompt users to enter missing info - http://ckan.org/ticket/863
#838 1291736461000000 memespring Search results changes: http://ckan.org/ticket/864
#838 1291812318000000 memespring Add download formats to search results (http://ckan.org/ticket/866)
#838 1295259773000000 rgrp I'm going to close this ticket as almost everything done and remaining ticket can be done on its own.
#839 1291736541000000 memespring Done with the exception of the discuss/comments page. Plugin won't install on my setup. Fredrich looking into it.
#840 1291325251000000 dread dread: ] That's the proxy_cache decorator which sets a Beaker expiry time only. ] Whether that is on or off, you still end up run the ] "etag_cache(cache_key)" command in the package controller won't you? ] And won't that will either insert the Etags header or abort 304 for a ] repeat? I must admit I've not tried it, asking purely with the code in ] front of me and welcome you pointing out where I've gone wrong. ww: Right. So those etag calls predate the cache decorators and should probably be either moved up into the decorator (e.g. use @ckan_cache instead of @proxy_cache) or wrapped in a check for the config parameter. (And @ckan_cache could be changed to use pylons' etag_cache function rather than just setting the header...)
#840 1302694123000000 dread Basic on/off switch added, tested & documented in cset:0da189c9630e on default.
#841 1300364333000000 thejimmyg See #995
#842 1296468313000000 rgrp Change to awaitingtriage as definitely not critical.
#842 1303474131000000 thejimmyg As a user I come to a package: Have a todo count at that top that takes you down to the todo list (which may say nothing todo) At the bottom is a section of the package display titled "ToDo" where I see a list of all toDos for the package most recent at the top If I am logged in see a form for "Add to do" at the top of the todo section and can add one straight away I see a "now resolved" button next to each which goes green when you hover. When clicked the todo fades away. Otherwise I see a button that says "login to add todo" expands out the form The form One of the fields is category -> autocomplete the category (not constrained) Add a description Submit, the todo gets added via AJAX to the list at the top as the most recent todo Model: todo id package_id todo_category_id (required) description (required) date=NOW() resolved=False todo_category id name Prepopulate with: broken-resource-link, no-author, bad-format
#842 1303474228000000 thejimmyg > Otherwise I see a button that says "login to add todo" > > expands out the form > Actually rather than expanding the form, you will go away to the login page and come back to see the expanded form (question: how does this redirect you back to the bottom?)
#843 1291652696000000 pudo The real name is also requested via AX/SReg extensions in OpenID so for new users who are not using google, this should usually be filled in automatically.
#843 1319710380000000 dread * "we should show the openid as well to distinguish between users with the same name." - when "Full name" is not distinguishable, maybe best to display the unique 'name' field as a hover-over. * "on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name." - yes you always got take to the personal details page. We should use a flash message at this point if they have not filled in the "Full name" field to suggest they click edit and do this. * "List is to long" - this has been addressed - see http://thedatahub.org/user
#843 1319721601000000 dread Ok I misunderstood this ticket. This is referring to adding a user in e.g. http://thedatahub.org/group/authz/energy-data This UI seems to be updated. You start typing the name, full name or open id of the person and it has a dropdown that autocompletes. This seems to be sufficient for Will's points 1 and 3. Would be still good to have a flash message on account creation to encourage people to add personal their Full name. This is similar to #1413 so I'll close this ticket and add it there.
#844 1296340486000000 rgrp Looking at DNS this apparently has been fixed.
#845 1291723492000000 dread This was completed on the feature-845-required-fields branch and merged into default in cset:3b5635ffaa7d
#846 1291715212000000 rgrp Reopening and fix used illegal css ;) -- it is illegal to have an @import statement (as we do with extras.css) after normal css statements. We therefore need to move the core css back out into a ckan.css (or similar file).
#846 1291719074000000 memespring It's been done on a branch and pushed to bitbucket: http://bitbucket.org/memespring/ckan/src/9e74c40ff073/ckan/public/css/style.css
#847 1291639903000000 rgrp Done my best to pull all the solr related tickets and work together in one place.
#847 1292844627000000 rgrp Still have ticket:669 and ticket:874 to do (431 probably won't be done for a while).
#847 1295259902000000 pudo This is now functional, minus the index testing tool.
#849 1291715179000000 pudo == #846
#851 1292860957000000 wwaites * link checker above uses the queue. queue not running generally * quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest
#851 1292892515000000 wwaites Ready to put into cron job. cf: * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions) * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions) * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi 1.2) * https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update)
#851 1292894661000000 wwaites currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow...
#851 1292941303000000 wwaites urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp. change the httpReq action to use http://curl.haxx.se/libcurl/python/
#851 1293025112000000 wwaites Implemented curlReq that does a curl request and returns statements similar to or analogous to httpReq. Require curate<=0.8 This in a cron job is sufficient to go through all the packages and update them with a broken link tag now: {{{ curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY }}}
#852 1291723207000000 wwaites plugin documentation: http://packages.python.org/ckan/plugins.html
#852 1295259620000000 rgrp Removing from sprint and moving to release as a master ticket.
#852 1315821628000000 rgrp Moved dataset archiving into new ticket #1327 and this is therefore now done.
#853 1291723143000000 wwaites done in http://bitbucket.org/ww/ofs need to merge back into main ofs repo
#853 1294047550000000 pudo Looking at the changeset this cannot be functional yet: where is the implementation of the policy document exchange. It seems to me like this is currently adding the actual credentials to the request (self.ofs.conn.add_aws_auth_header(headers, 'PUT', path)). c.f.: * http://doc.s3.amazonaws.com/proposals/post.html#Form_Fields * http://code.google.com/apis/storage/docs/reference-methods.html#postobject
#853 1294594581000000 wwaites we don't need a policy document exchange. it's simpler than that. the "server" instance has already permissions to upload. it just calculates the headers and such that are needed (based on the "client"'s initial headers) and gives them to the "client" the client then uploads without knowing the "server"'s credentials. The "client" never needs any of its own goostor credentials at all. the only separate step is to make the widget readable by the world. ticket #879 is to expose this as a small set of API calls.
#854 1304351843000000 johnlawrenceaspden Coverage now up to 84% and 81%. Remaining untested code is error conditions, which we decided weren't worth the effort of locking down. fixed on feature-854-tests-for-authz-groups, now merged into default code.
#855 1311176988000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. The auth code is in the process of a major refactor anyway.
#856 1311177066000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
#857 1311177075000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
#858 1291734297000000 dread On branch feature-858-diff removed ckan.lib.diff as it was not being used. Merged into default in cset:f9ba1ae63ddd
#859 1311177461000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer.
#860 1291726067000000 wwaites http://bitbucket.org/ww/ckanrdf/changeset/67df6dc33ec4
#861 1311168845000000 thejimmyg This ticket has been open for than 6 months so I'm closing it.
#862 1291726261000000 wwaites not to mention sparql endpoint...
#863 1295259827000000 rgrp Removing milestone as not certain when we'll do this.
#863 1338206455000000 ross UI has changed rather a lot in last 18 months, so I killing this bug.
#864 1291741028000000 memespring Havent implemented all text changes because worried about impact on other themes.
#865 1294414639000000 thejimmyg Isn't this in there now? Can we close? Thanks
#865 1295259773000000 pudo This has been fixed on the respective branch and merged into default..
#867 1291916278000000 dread Done in ckanclient cset:d40fb101aba9
#867 1292410515000000 dread Also ckan cset:c5b018bfe9bb
#867 1299866685000000 rgrp This was a breaking change for loaders code. Obviously we don't have tests for that so would not have been noticed ... Fixed in cset:af81e54bd590/ckanclient
#868 1292890251000000 [email protected] Below is a patch to make the tests run at least 2.5 times faster (about 15 mins on my old laptop). Instead of dropping the tables each time, it just deletes everything in them, using a low level connection. All the tests pass this way. It's a surprisingly clean patch. Here are a few points concerning it. * I tested truncating the tables but it's slower. If there are any big tables in the tests this way is the fastest (faster than drop). * The sequences (id columns) will start from where they left off. * I also investigated making postgres template database and cloning it, but the complication was not worth it. * sqlalchemy iterates the tables in reverse dependency order, which make this possible. * I targeted rebuild_db as that what most of the tests I saw where using, however I have not checked all tests to see if they all are. * There is a slight hack on the repo object to make sure it knows that "clean_db" is coming from the tests. * I refactored init_db for code reuse. * I have not done a version check. sqlalchemy >= 0.5 do this in a different way as outlined in the comments. {{{ diff -r 7f2239b0f743 ckan/model/__init__.py --- a/ckan/model/__init__.py Fri Dec 17 10:34:47 2010 +0000 +++ b/ckan/model/__init__.py Mon Dec 20 23:25:04 2010 +0000 @@ -41,6 +41,9 @@ def init_db(self): super(Repository, self).init_db() + self.add_initial_data() + + def add_initial_data(self): # assume if this exists everything else does too if not User.by_name(PSEUDO_USER__VISITOR): visitor = User(name=PSEUDO_USER__VISITOR) @@ -69,6 +72,26 @@ import migrate.versioning.api as mig version = mig.version(self.migrate_repository) return version + + def clean_db(self): + # delete only added for tests + if hasattr(self, "delete_only") and self.delete_only: + self.delete_all() + else: + super(Repository, self).clean_db() + + def delete_all(self): + + self.session.remove() + ## use raw connection for performance + connection = self.session.connection() + ## sqla sorts in reverse dependancy order. + ## in >= 0.5 use reversed(metadata.sorted_tables()) instead of table_iterator + for table in self.metadata.table_iterator(): + connection.execute('delete from "%s"' % table.name) + self.session.commit() + + self.add_initial_data() def setup_migration_version_control(self, version=None): import migrate.versioning.exceptions diff -r 7f2239b0f743 ckan/tests/__init__.py --- a/ckan/tests/__init__.py Fri Dec 17 10:34:47 2010 +0000 +++ b/ckan/tests/__init__.py Mon Dec 20 23:25:04 2010 +0000 @@ -55,6 +55,7 @@ import ckan.model as model model.repo.rebuild_db() +model.repo.delete_only = True class BaseCase(object): }}}
#868 1292921428000000 sebbacon See also #867 Thanks for the patch.
#868 1292921456000000 sebbacon I mean #876, of course.
#868 1293401808000000 anonymous Attached are the timings I have for the tests after I upgraded to 0.57 and after a few simple test tweaks. They do not include setup and teardown time at the class level as they are not assignable to individual tests.
#868 1294753596000000 dread I've merged in David Raznick's patches: * no_autoflush_deletes.diff cset:2b9591172182 * postgres_speed.diff cset:fa1b7e3a4e0f * vdm_purge_no_autoflush.diff vdm cset 8accdd0b9b7f I've also merged in Seb's fork: cset:68d63fda4814 which closes this ticket, achieving test speeds of under 3 minutes!
#869 1292059662000000 rgrp Closed by cset:83734b5e251c which implements an IConfigurer interface.
#870 1294862485000000 [email protected] A patch is available here. https://bitbucket.org/kindly/ckan/changeset/9a1d6f55587b
#870 1294914243000000 anonymous Merged into default in cset:54ae110094be
#871 1292322361000000 wwaites Regarding rkhunter -- yes, eu1 appears to be clean Regarding the upgrade -- upgradede to 4.72 from backports which, looking more closely, appears to still have the privilege escalation bug but not the remote root exploit. Regarding exim on other hosts, there is no reason for them to be running a full mta, something like ssmtp should suffice. Also very worth the thought of moving to postfix. It's much easier to configure and I haven't known it to have any comparable bugs in the decade or so I've been running it. In fact I've never seen anyone actually use exim before...
#871 1292323656000000 nils.toedtmann Re postfix: I second ww. I like to run some super simple local MTA (e.g. "nullmailer") on all but one server, using a central postfix (or a send-only GMail account) as smarthost. Am happy with postfix for >10years, it's straightforward and rock solid.
#871 1296340558000000 rgrp This is not a ckan issue. should have been on http://knowledgeforge.net/okfn/tasks
#872 1292957110000000 dread Done in ckanext cset:8a7e931ef37c
#873 1292939274000000 dread Fixed in dgu cset:8bfb867e247a and ckanext cset:fabd31544a73
#875 1297085261000000 pudo So far opting for route 1) (not implementing facets), therefore this can be closed!
#876 1292891133000000 [email protected] I have looked into this already so I can give you a head start. I am working on a project that uses many backends so I have some experience. So here is what I have found so far. == nested transactions == VDM does not support sqlite, as it uses nested transactions. I do not think vdm needs nested transactions. It can use a flush instead. Here is the patch that works. All vdm tests pass. {{{ --- a/vdm/sqlalchemy/base.py Sat Sep 11 23:06:26 2010 +0000 +++ b/vdm/sqlalchemy/base.py Mon Dec 20 16:16:34 2010 +0000 @@ -40,9 +40,8 @@ self.setattr(session, 'HEAD', True) self.setattr(session, 'revision', revision) if revision.id is None: - session.begin_nested() session.add(revision) - session.commit() + session.flush() }}} == indexes == The index file 021_postgres_upgrade.sql in the migrate repository will not run as it uses syntax particular to postgres. Another will need to be made thats similar. sqlite does not support complex indexes like upper(text), so a work around will need to be found. == unicode == The harvesting returns utf8 encoded strings and pysqlite dbapi only supports python unicode objects (as far as I can tell). There will need to be a process in converting all strings that get into the database with string.decode("utf8") == dates == Have not looked into this one too much. However, as sqlite stores everything as strings the timestamps appear to be failing on conversion back into python. I have solved the above two issues before by adding attribute extensions to sqlalchemy mappers to do the conversions without effecting too much code. == in memory sqlite == Some tests need to change in order to make sure the database is created first because the database gets lost each time. In the tests that I have made pass, they run in about a seventh of the time as they do on postgres. == Other things to keep in mind. == * Need a new flag in test.ini to remove full text indexing completely, or always use it with solr. * There are enough incompatibilities between the databases that you would also want to test against postgres as well, at least before a release. * I would probably upgrade sqlalchemy first, so you will not have to the changes twice. The new versions are significantly faster too. * I have submitted a patch to #868 that makes the tests run about 2.5 times as fast and I think there are more low hanging fruit if the aim is test speed.
#876 1292924064000000 sebbacon Thanks for the info! Re. nested transactions. I am getting repeated non-deterministic test failures against sqlite (and indeed postgres, but these failures appear more frequent against sqlite). One of them I seemed to be able to get rid of by eliminating the savepoint as per your first point. However, it appears that sqlite [http://www.sqlite.org/lang_savepoint.html does] support savepoints; to demonstrate it, the following test code appears to work in the latest sqlalchemy: {{{ #!python from sqlalchemy import * from sqlalchemy.orm import * db = create_engine('sqlite:///') metadata = MetaData(db) users = Table('users', metadata, Column('user_id', Integer, primary_key=True), Column('name', String(40)),) users.create() class User(object): pass usermapper = mapper(User, users) Session = sessionmaker() session = Session() fred = User() fred.name = "Fred" sue = User() sue.name = "Sue" amy = User() amy.name = "Amy" session.add(fred) session.add(amy) session.begin_nested() session.add(sue) session.rollback() session.commit() assert session.query(User).count() == 2 print "OK" }}} So, while I agree they're not needed, I'm not sure they're a problem. What do you think? Also, have you seen non-deterministic test errors like this?
#876 1292939711000000 [email protected] I have read quite a lot of people having problems with savepoints with sqlite and thought they were not supported on sqlalchemy. They are at least not consistant with postgres ones. I may well be out of date on this. Here is an [http://groups.google.com/group/sqlalchemy/browse_thread/thread/dc9d1b61044bf730/65a62a33ec313842?lnk=gst&q=no+such+savepoint#65a62a33ec313842 example] even though its a bit old. I did get some non deterninistic errors, the above seemed to fix them. A failed subtransaction is not handled well by sqlalchemy and I think this causes knockon effects due to the unresolved transaction. I would stay well clear of them entirely if possible. What are the errors you are getting?? My 2 cents. ignore me at will... I would think about using a different backend for testing than production. [http://stackoverflow.com/questions/2716847/sqlalchemy-sqlite-for-testing-and-postgresql-for-development-how-to-port look here]. If you want to support both then you should test on both. There are simple ways to scrape a few more minutes off the tests. If you want real speed, then a multiprocess solution (with a database per core) would be sensible if a bit tricky.
#876 1293188088000000 anonymous Thanks for your feedback, very useful. I don't really agree with the people in the linked discussion who say it's pointless testing against a different database from production. The goal here is to make it easy enough for people to run as many tests as possible that they actually do so. Even 15 minutes is too long in that case. With sqlite we can get it in at under 5 minutes. I would also like to identify the longest running tests (which I would characterise as "functional" or "integration" tests and make them run as a separate suite, and then encourage a culture of writing true unit tests before functional tests, so that running unit tests can happen in 1 minute and be part of the regular development cycle. That's no replacement for also running *all* tests periodically, and also running tests under postgres, which we can continue to do on the continuous integration server. Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests.
#876 1293218714000000 anonymous I agree with all your points about testing apart form using sqlite, especially splitting out the functional tests and continuous integration. > Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests. Well there are two options 1. refactor the tests 2. refactor the code to use sqlite and postress It is a value judgment as to which is more complicated. I personally think 2 is more complicated but may be wrong on that. The real danger with 2 is that you are needlessly adding complication to production code, with 1 you are only changing the tests. Upgrading to sqlalchemy 0.5+ should happen first regardless. You will need upto date documentation. There is another option too. Put the postgres data directory on tempfs/ramfs and turn off durability [http://www.postgresql.org/docs/9.0/static/non-durability.html here]. We would need a way to db init before the tests where run (or) at boot). This may be the best of both worlds. Anyway Happy xmas!!
#876 1294753889000000 dread Seb and David have completed this I believe. I've merged the changes into core CKAN in cset:68d63fda4814.
#877 1297680579000000 rgrp Basic pass on an implementation (no permissions yet etc): https://bitbucket.org/okfn/ckanext-upload/changeset/9ae543f0645f
#877 1298624165000000 rgrp Various tidying in https://bitbucket.org/okfn/ckanext-upload/changeset/0fad7aa7aa97 (success messages, permissions on uploaded file - public-read) and completed permissions in https://bitbucket.org/okfn/ckanext-upload/changeset/a83ce00a1266. Still need to integrate into general workflow (e.g. create a Resource on successful upload) but that is a separate item so this ticket is now done.
#878 1314404621000000 rgrp A working version of this in ckanjs as of 2 weeks ago. Need to integrate into main CKAN.
← 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 →
Note: See TracReports for help on using and creating reports.