Randomly some Docker-containers on a clients Linux machine wouldn’t start - they’d fail with the error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\"": unknown.
It didn’t matter what container it was, nor where or who started it. Sometimes it happened with Gitlab-CI jobs, sometimes with manually run containers. It also didn’t happen everytime. Sometimes it just worked, other times it didn’t work for hours and I’d ignore it.
The last time it happened I googled the error again and again found this huge open issue on Docker’s Github issue-tracker. This time I read every response in the thread and came upon this answer:
Okay, just found another interesting post on other forum. If you are running an vps which is virtualized with Virtuozzo you hosting provider maybe locked your tasks…
Im using strato and it seems to be that they have limited my server. Under /proc/user_beancounters you can find those settings. The numprocs is set to 700 and my actual held is 661. Starting an bigger docker stock seems to be impossible…
You can find more in this post https://serverfault.com/questions/1017994/docker-compose-oci-runtime-create-failed-pthread-create-failed/1018402
It seems to be there is no bug…
This sounded like my kind of problem as my client is using a Strato server, too!
Now my native language isn’t English, however I remembered the term “beancounter” from the BOFH and I really do hope that this feature originates somewhat from there (though I know that the term is older than the BOFH).
Looking at the servers beancounter gives me the following output:
root@h2939459:~# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
2939459: kmemsize 353345536 1649811456 9223372036854775807 9223372036854775807 0
lockedpages 0 32 9223372036854775807 9223372036854775807 0
privvmpages 5861268 6833651 9223372036854775807 9223372036854775807 0
shmpages 2222328 2255102 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 541 541 1100 1100 0
physpages 1461605 8319910 8388608 8388608 0
vmguarpages 0 0 9223372036854775807 9223372036854775807 0
oomguarpages 1525697 8388608 0 0 0
numtcpsock 0 0 9223372036854775807 9223372036854775807 0
numflock 0 0 9223372036854775807 9223372036854775807 0
numpty 1 2 9223372036854775807 9223372036854775807 0
numsiginfo 16 147 9223372036854775807 9223372036854775807 0
tcpsndbuf 0 0 9223372036854775807 9223372036854775807 0
tcprcvbuf 0 0 9223372036854775807 9223372036854775807 0
othersockbuf 0 0 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 0 9223372036854775807 9223372036854775807 0
numothersock 0 0 9223372036854775807 9223372036854775807 0
dcachesize 264458240 1497632768 9223372036854775807 9223372036854775807 0
numfile 4284 7930 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numiptent 1997 2000 2000 2000 74
Take a look at the last line:
uid resource held maxheld barrier limit failcnt
numiptent 1997 2000 2000 2000 74
This shows that the resource “numiptent” is currently using 1997 “units”, where 2000 can be held totally. “failcnt” shows the count of refused allocations. When I run another container, this count increased so it must be the problem!
A quick search revealed that “numiptent” are the number of NETFILTER (IP packet filtering) entries, or in simpler terms: the number of iptables rules.
I immediatly knew the reason for this: the client is using fail2ban which blocks IP-addresses that try to bruteforce ssh-logins on the server. Looking at the fail2ban-overview I noticed that 1900 IP-addresses where blocked, conspicuously close to the beancounter-limit of 2000!
Restarting fail2ban threw out all these IP-addresses which decreased the beancounter on “numiptent”.
Reason found! Now why Strato decided to limit the number of iptables-entries and what to do against this (apart from disabling fail2ban) - I don’t know yet.