CKAN: Ticket #1447: disk space leakage

cc set

nils.toedtmann — Mon, 07 Nov 2011 16:49:43 GMT

cc nils.toedtmann@… added

dread — Fri, 25 Nov 2011 11:31:30 GMT

This appears to have happened again today on test.ckan.net and someone has sorted it.

The problem is visible on munin as inodes running out.

eu25 seems ready to fall over in about a week: http://munin.okfn.org/okfn.org/eu25.okfn.org-df_inode.html
thedatahub.org on s055 (and other fry instances) seem to have dynamically adjusted inode table size (by the kernel) so is less of a problem

dread — Wed, 30 Nov 2011 10:27:53 GMT

As predicted, this happened again today. From the following analysis it confirms that the problem is the cache growing and growing.

Disk usage in megabytes:

okfn@s025:~/var/srvc/publicdata.eu$ du -s -m /*
7	/bin
22	/boot
1	/dev
10	/etc
4157	/home
0	/initrd.img
0	/initrd.img.old
114	/lib
1	/lost+found
1	/media
1	/mnt
1	/opt
0	/proc
1	/root
7	/sbin
1	/selinux
1	/srv
0	/sys
1	/tmp
421	/usr
443	/var
0	/vmlinuz
0	/vmlinuz.old

okfn@s025:~/var/srvc/publicdata.eu$ du -s -m /home/okfn/var/srvc/publicdata.eu/*2173	/home/okfn/var/srvc/publicdata.eu/backup
1	/home/okfn/var/srvc/publicdata.eu/backup_RENAMED_TO_AVOID_MAYHEM.sh
1	/home/okfn/var/srvc/publicdata.eu/common.sh
1893	/home/okfn/var/srvc/publicdata.eu/data
1	/home/okfn/var/srvc/publicdata.eu/fetch.sh
1	/home/okfn/var/srvc/publicdata.eu/gather.sh
1	/home/okfn/var/srvc/publicdata.eu/pip-requirements.txt
1	/home/okfn/var/srvc/publicdata.eu/publicdata.eu.ini
86	/home/okfn/var/srvc/publicdata.eu/pyenv
1	/home/okfn/var/srvc/publicdata.eu/run.sh
1	/home/okfn/var/srvc/publicdata.eu/sstore
0	/home/okfn/var/srvc/publicdata.eu/who.ini

okfn@s025:~/var/srvc/publicdata.eu$ ls -l /home/okfn/var/srvc/publicdata.eu/backup
total 2224588
-rw-r--r-- 1 okfn okfn 343199744 2011-06-14 20:50 db-20110614-2050.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:51 db-20110614-2051.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:52 db-20110614-2052.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:53 db-20110614-2053.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:54 db-20110614-2054.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:55 db-20110614-2055.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:56 db-20110614-2056.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:57 db-20110614-2057.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:58 db-20110614-2058.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 20:59 db-20110614-2059.sql
-rw-r--r-- 1 okfn okfn   1036288 2011-06-14 22:00 db-20110614-2200.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:01 db-20110614-2201.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:02 db-20110614-2202.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:03 db-20110614-2203.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:04 db-20110614-2204.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:05 db-20110614-2205.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:06 db-20110614-2206.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:07 db-20110614-2207.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:08 db-20110614-2208.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:09 db-20110614-2209.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:10 db-20110614-2210.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:11 db-20110614-2211.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:12 db-20110614-2212.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:13 db-20110614-2213.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:14 db-20110614-2214.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:15 db-20110614-2215.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:16 db-20110614-2216.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:17 db-20110614-2217.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:18 db-20110614-2218.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:19 db-20110614-2219.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:20 db-20110614-2220.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:21 db-20110614-2221.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:22 db-20110614-2222.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:23 db-20110614-2223.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:24 db-20110614-2224.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:25 db-20110614-2225.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:26 db-20110614-2226.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:27 db-20110614-2227.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:28 db-20110614-2228.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:29 db-20110614-2229.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:30 db-20110614-2230.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:31 db-20110614-2231.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:32 db-20110614-2232.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:33 db-20110614-2233.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:34 db-20110614-2234.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:35 db-20110614-2235.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:36 db-20110614-2236.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:37 db-20110614-2237.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:38 db-20110614-2238.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:39 db-20110614-2239.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:40 db-20110614-2240.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:41 db-20110614-2241.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:42 db-20110614-2242.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:43 db-20110614-2243.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:44 db-20110614-2244.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:45 db-20110614-2245.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:46 db-20110614-2246.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:47 db-20110614-2247.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:48 db-20110614-2248.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:49 db-20110614-2249.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:50 db-20110614-2250.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:51 db-20110614-2251.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:52 db-20110614-2252.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:53 db-20110614-2253.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:54 db-20110614-2254.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:55 db-20110614-2255.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:56 db-20110614-2256.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:57 db-20110614-2257.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:58 db-20110614-2258.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-14 22:59 db-20110614-2259.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:00 db-20110615-0000.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:01 db-20110615-0001.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:02 db-20110615-0002.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:03 db-20110615-0003.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:04 db-20110615-0004.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:05 db-20110615-0005.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:06 db-20110615-0006.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:07 db-20110615-0007.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:08 db-20110615-0008.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:09 db-20110615-0009.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:10 db-20110615-0010.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:11 db-20110615-0011.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:12 db-20110615-0012.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:13 db-20110615-0013.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:14 db-20110615-0014.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:15 db-20110615-0015.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:16 db-20110615-0016.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:17 db-20110615-0017.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:18 db-20110615-0018.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:19 db-20110615-0019.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:20 db-20110615-0020.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:21 db-20110615-0021.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:22 db-20110615-0022.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:23 db-20110615-0023.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:24 db-20110615-0024.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:25 db-20110615-0025.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:26 db-20110615-0026.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:27 db-20110615-0027.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:28 db-20110615-0028.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:29 db-20110615-0029.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:30 db-20110615-0030.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:31 db-20110615-0031.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:32 db-20110615-0032.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:33 db-20110615-0033.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:34 db-20110615-0034.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:35 db-20110615-0035.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:36 db-20110615-0036.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:37 db-20110615-0037.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:38 db-20110615-0038.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:39 db-20110615-0039.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:40 db-20110615-0040.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:41 db-20110615-0041.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:42 db-20110615-0042.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:43 db-20110615-0043.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:44 db-20110615-0044.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:45 db-20110615-0045.sql
-rw-r--r-- 1 okfn okfn         0 2011-06-15 00:46 db-20110615-0046.sql
-rw-r--r-- 1 okfn okfn 483144447 2011-06-15 10:00 db-20110615-1000.sql
-rw-r--r-- 1 okfn okfn 482136064 2011-06-15 10:07 db-20110615-1007.sql
-rw-r--r-- 1 okfn okfn 483144447 2011-06-15 10:50 db-20110615-1050.sql
-rw-r--r-- 1 okfn okfn 483053568 2011-06-15 10:51 db-20110615-1051.sql

okfn@s025:~/var/srvc/publicdata.eu$ du -s -m /home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/*
117	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/0
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/1
117	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/2
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/3
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/4
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/5
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/6
117	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/7
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/8
117	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/9
117	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/a
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/b
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/c
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/d
117	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/e
116	/home/okfn/var/srvc/publicdata.eu/data/sessions/container_file/f

dread — Tue, 20 Dec 2011 09:45:24 GMT

Once again eu25 ran out of space again today.

dread — Wed, 11 Jan 2012 10:58:15 GMT

eu25 ran out of space again this weekend and eu8 (at/it/us_co) today.

nils.toedtmann — Wed, 11 Jan 2012 15:49:47 GMT

For time being, i created a cron script remove_old_files.

You could just copy it to /etc/cron.daily/, but i recommend to not run it as root: if it's misconfigured, it could wipe the system!

So you better copy it to /home/okfn/sbin/ (not /home/okfn/bin/ which often is the sysadmin HG repo), and add it to some unprivileged user's crontab. In most cases, the leftover files are owned by user "www-data", so

$ sudo crontab -e -u www-data

and then add something like

37 4 * * *  /home/okfn/sbin/remove_old_files

Don't forget to edit the script remove_old_files itself and list the directories you want to be cleaned up.

This is already done on s008/eu8 and s019/eu19. dread, do you want to do this for s025/eu25 and see how this goes?

Todo nils: verify tomorrow on s019 that it worked properly, e.g. this should show only a few files:

find /var/lib/ckan/nederland/data/sessions/ -type f -amin +$((7*24*60)) -ls

nils.toedtmann — Fri, 24 Feb 2012 11:23:56 GMT

I had forgotten to check s019 how well my cleanup script is working (and now s019 is gone), but at least it didn't destroy it :-)

You might want to give it a try on s025/PDEU. (Tell me if you want me to do that).

dread — Fri, 24 Feb 2012 11:26:48 GMT

Yes please Nils!

cc changed

nils.toedtmann — Fri, 24 Feb 2012 13:21:02 GMT

cc pudo added

OK i fixed a bug in my script and refactored it so that it can now be dropped into /etc/cron.daily/ while still deleting as unprivileged user.

It is now running on s025, removing everything older than 7 days. Please verify in 9 days or so that it's working.

Consider to add this cron job to the ckan deb package e.g. as "/etc/cron.daily/ckan-cleanup"

status changed; resolution set

nils.toedtmann — Fri, 23 Mar 2012 13:53:10 GMT

status changed from new to closed
resolution set to fixed

Just checked s025 (which is depricated now), looks like my script is working fine - nothing older than a week in /home/okfn/var/srvc/publicdata.eu/data/sessions/.

We should activate this script on other hosts as well, e.g. so55/thedatahub.

nils.toedtmann — Fri, 23 Mar 2012 13:55:13 GMT

Just to add: the remove_old_files script is only a workaround, not a fix. CKAN should clean up after itself. Feel free to re-open this ticket for a proper solution ;-)

status changed; resolution deleted

rgrp — Fri, 23 Mar 2012 14:05:02 GMT

status changed from closed to reopened
resolution fixed deleted

status changed; owner, milestone set

rgrp — Fri, 23 Mar 2012 14:05:44 GMT

owner set to kindly
status changed from reopened to assigned
milestone set to ckan-v1.7

@kindly: hope ok to assign to you (maybe just for review and thought on who would be best placed to look at ...)

nils.toedtmann — Fri, 23 Mar 2012 16:10:29 GMT

Ticket http://trac.okfn.org/ticket/1222 tracks the effort to push the clean-up script onto CKAN hosts.

milestone changed

kindly — Mon, 28 May 2012 10:55:04 GMT

milestone changed from ckan-v1.7 to ckan-backlog

nils.toedtmann — Tue, 26 Jun 2012 15:58:03 GMT

This is becoming painful for the sysadmins. Please fix.

dread — Tue, 26 Jun 2012 16:15:30 GMT

BTW on DGU I have set it up to use memcached for these sessions (v. easy) and I think it solves the problem.