I think it’s good news : you are not alone with this problem to solve
I really don’t understand your problems with Docker, never seen such behavior in my test VM …
Docker overlays are self-managed, normally you just have to prune container/images.
What is weird is that you can’t kill the container(s) !
Does it survive a reboot ? Did you try a systemctl kill docker
?
I’m wondering if an unusual growing of overlay storage may be caused by a missing mapping in your Docker launch command (the -v lines).
Some reading : Why is Docker filling up /var/lib/docker/overlay2? - Stack Overflow , server - Docker overlay2 eating Disk Space - Stack Overflow , Use the OverlayFS storage driver | Docker Documentation
Some details about my test VM, maybe it can help …
- Ubuntu 18 LST, not really up to date (I’m too lazy ) in a VMware WS VM (16 CPU, 24GB RAM)
- Docker installed as instructed by /e/ documentation
- system disk is 32 GB (1-partition scheme)
- build disk is an additional 350GB, mounted over /srv
- I run everything as root (it’s a test VM …)
- Docker launch script in /srv, launched with root in a text console (pts) :
root@ubuntu:/srv# cat docker.sh
#!/bin/bash
docker pull registry.gitlab.e.foundation/e/os/docker-lineage-cicd:community
docker run \
-v "/srv/src:/srv/src" \
-v "/srv/zips:/srv/zips" \
-v "/srv/logs:/srv/logs" \
-v "/srv/ccache:/srv/ccache" \
-v "/srv/mirror:/srv/mirror" \
-e "BRANCH_NAME=v0.18-q" \
-e "DEVICE_LIST=chiron" \
-e "REPO=https://gitlab.e.foundation/e/os/releases.git" \
-e "ANDROID_JACK_VM_ARGS=-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx20G" \
registry.gitlab.e.foundation/e/os/docker-lineage-cicd:community
- of course, I created src, zips,logs,ccache,mirror dirs before launching (755)
About the disk space problem in /srv :
- the attempt from last evening (after formatting /srv with EXT4) leads to the same result, and this is also good news : I can reproduce the problem every time
- as I already knew, ext filesystems are very bad when it comes to some usage : they can waste lots of disk space (I found some relevant readings about that : Same file and folders on same disk but different sizes)
- as I did for many production servers over the years, i formatted the /srv again with XFS, build is running … (some info here : https://unix.stackexchange.com/a/525630)
- at this time, overlay is stable at it’s usual size :
root@ubuntu:/srv# df -h |grep overlay
overlay 32G 18G 12G 60% /var/lib/docker/overlay2/5500dabcf9b85c08b8645f6a0f5e5244013510388afe0d0e208461874ff0d1f9/merged