docker占用磁盘空间清理
问题描述: 发现磁盘空间不够，分析磁盘情况发现docker占用了大量磁盘空间，下面分析如何找到谁是最大的磁盘占用者。
查看磁盘使用:
du -m --max-depth=1 ./
输出:
13 ./sbin
1 ./application
1 ./srv
1262 ./opt
160 ./root
13 ./bin
1 ./corefile
140932 ./home
4 ./lib32
1 ./dev
du: cannot access './run/user/1000/gvfs': Permission denied
11 ./run
1108 ./lib
1 ./cgroup
1 ./lib64
1 ./data
1 ./nomad
0 ./sys
231 ./boot
1 ./media
191261 ./var
3094 ./snap
1 ./cdrom
1 ./mnt
1 ./lost+found
9408 ./usr
1 ./.config
1 ./rootfs
1 ./tmp
16 ./etc
du: cannot access './proc/31724/task/31724/fd/3': No such file or directory
du: cannot access './proc/31724/task/31724/fdinfo/3': No such file or directory
du: cannot access './proc/31724/fd/4': No such file or directory
du: cannot access './proc/31724/fdinfo/4': No such file or directory
du: cannot access './proc/32032': No such file or directory
0 ./proc
347511 ./
可以看到var目录使用磁盘比较大。
cd var
du -m --max-depth=1 ./
输出:
1 ./opt
9 ./backups
1 ./metrics
153 ./cache
1 ./local
191011 ./lib
88 ./log
1 ./spool
1 ./snap
1 ./mail
1 ./crash
1 ./tmp
191261 ./
cd lib
du -m --max-depth=1 ./
输出
1 ./xkb
1 ./NetworkManager
1 ./dictionaries-common
1 ./ubiquity
1 ./man-db
1 ./initscripts
1 ./polkit-1
1 ./ieee-data
1 ./apparmor
1 ./urandom
1 ./rpm
119 ./dpkg
1 ./gems
1 ./update-rc.d
1 ./colord
1 ./fwupd
1 ./usb_modeswitch
1 ./os-prober
2 ./ureadahead
1 ./vim
1 ./certmonger
1 ./sudo
1 ./ipa-client
188219 ./docker
13 ./app-info
...
可以看到docker使用大部分磁盘空间。
/var/lib/docker# du -sh
184G .
分析docker的磁盘使用情况
docker system df
输出:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 30 10 13.76GB 6.882GB (50%)
Containers 10 10 20.05GB 0B (0%)
Local Volumes 5 2 845MB 624.6MB (73%)
Build Cache 0B 0B
实际使用并不是很高。
分析docker目录: /var/lib/docker
du -m --max-depth=1 ./
输出:
807 ./volumes
1 ./trust
1 ./builder
1 ./network
1 ./containerd
75 ./image
1 ./plugins
1 ./runtimes
64338 ./aufs
1 ./swarm
123000 ./containers
1 ./tmp
188219 ./
root@lap-131:/var/lib/docker#
最大的两块:
/var/lib/docker/aufs# du -sh
63G .
/var/lib/docker/containers# du -sh
121G .
分析容器占用最大的容器: gitlab
/var/lib/docker/containers# du -m --max-depth=1 ./
输出:
1 ./a3ea6a15d0e0f4f5e2d7c5e7faf7fa3c172659126e94f033600528318960c663
3 ./80039c421bb69a2d25a59be4a95a30ad8df5a3552371db3c47223298e4950775
1 ./b117d03cd46018dd71688d2b17bcb08e68f2f76c66ebf05a63a25dd99593be6e
1 ./8168081fb8241b776a87ff0bc5eaba59ce6ab49c2a35961cbfddc7c062b8c02a
16 ./d988b1bfbaf1bfcc6b1f0d4cbf07bdf1eff61cf88589bb853e3194adb8f321f9
236 ./348f2400b96eba5918959041007d4c08fbe75bb3b4d3021cc487f8f297be6a76
1 ./15bd5827152a4a01cb3d246aafba2f202a03e29561a59517872a719591d16a95
1 ./3c77f3562cf66a7e47726add293e5803d801b302d380a8a2fca6838a36c6a905
1 ./2136547f678cf92118b546f0eb67a1798313d7521d58ad392931435bb046eafb
122745 ./cce705b881997536dcb2455f2cc23986e06bc73a78eb0728a1182f7abe0bcc10
123000 ./
docker ps
输出:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cce705b88199 gitlab/gitlab-ce "/assets/wrapper" 2 years ago Up 4 hours (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:122->22/tcp, 0.0.0.0:1443->443/tcp gitlab
分析容器内占用磁盘最大的文件:
cd /var/lib/docker/containers/cce705b881997536dcb2455f2cc23986e06bc73a78eb0728a1182f7abe0bcc10
ll -h
输出:
total 120G
drwx------ 4 root root 4.0K 9月 15 15:23 ./
drwx------ 12 root root 76K 9月 15 15:01 ../
-rw-r----- 1 root root 120G 9月 15 15:24 cce705b881997536dcb2455f2cc23986e06bc73a78eb0728a1182f7abe0bcc10-json.log
drwx------ 2 root root 4.0K 11月 8 2018 checkpoints/
-rw------- 1 root root 28K 9月 15 15:23 config.v2.json
-rw-r--r-- 1 root root 1.5K 9月 15 15:23 hostconfig.json
-rw-r--r-- 1 root root 10 9月 15 11:35 hostname
-rw-r--r-- 1 root root 174 9月 15 11:35 hosts
drwx------ 3 root root 4.0K 11月 8 2018 mounts/
-rw-r--r-- 1 root root 212 9月 15 11:35 resolv.conf
-rw-r--r-- 1 root root 71 9月 15 11:35 resolv.conf.hash
最大的为docker容器的日志。
Docker限制容器日志大小
参考文档: https://blog.csdn.net/gulijiang2008/article/details/105269414
清理Docker容器log（治标）
cat /dev/null > /var/lib/docker/containers/容器id/容器id-json.log
示例:
cat /dev/null > /var/lib/docker/containers/cce705b881997536dcb2455f2cc23986e06bc73a78eb0728a1182f7abe0bcc10/cce705b881997536dcb2455f2cc23986e06bc73a78eb0728a1182f7abe0bcc10-json.log
设置Docker容器日志大小（治本）
设置一个容器服务的日志大小上限
通过配置容器docker-compose.yml的max-size选项来实现:
nginx:
image: nginx:1.12.1
restart: always
logging:
driver: "json-file"
options:
max-size: "5g"
全局设置
新建/etc/docker/daemon.json，若有就不用新建了
# vim /etc/docker/daemon.json
{
"registry-mirrors": ["http://f613ce8f.m.daocloud.io"],
"log-driver":"json-file",
"log-opts": {"max-size":"500m", "max-file":"3"}
}
说明:
- max-size=500m，意味着一个容器日志大小上限是500M，
- max-file=3，意味着一个容器有三个日志，分别是id+.json、id+1.json、id+2.json
注：设置后只对新添加的容器有效。
重启docker守护进程
systemctl daemon-reload
systemctl restart docker
启动容器命令行配置:
docker run --help
输出:
--log-opt list Log driver options
示例:
docker run -it --log-opt mode=non-blocking --log-opt max-buffer-size=4m alpine ping 127.0.0.1