docker - 运行时物理内存最小配置需>=2G
背景
最近在某云申请了一个免费试用的云服务器,默认配置是:CPU - 1Core, MEM - 1G, 使用 docker-compose 启动服务组,docker container 反复重启。。。
排查问题
使用 docker log 进入容器查看,未发现问题。。
使用 docker stats 查看容器使用资源的情况,发现问题: 内存溢出!!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | ubuntu@VM-0-3-ubuntu:~$ docker stats fatal error: runtime: out of memory runtime stack: runtime.throw(0x55fb06c3ca5c, 0x16) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/panic .go:617 +0x74 fp=0x7fff4a9245d0 sp=0x7fff4a9245a0 pc=0x55fb056605d4 runtime.sysMap(0xc000000000, 0x4000000, 0x55fb08bfefb8) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mem_linux .go:170 +0xc9 fp=0x7fff4a924610 sp=0x7fff4a9245d0 pc=0x55fb0564b8e9 runtime.(*mheap).sysAlloc(0x55fb08be5aa0, 0x2000, 0x55fb08be5ab0, 0x1) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/malloc .go:633 +0x1cf fp=0x7fff4a9246b8 sp=0x7fff4a924610 pc=0x55fb0563e6ff runtime.(*mheap).grow(0x55fb08be5aa0, 0x1, 0x0) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mheap .go:1222 +0x44 fp=0x7fff4a924710 sp=0x7fff4a9246b8 pc=0x55fb05658cf4 runtime.(*mheap).allocSpanLocked(0x55fb08be5aa0, 0x1, 0x55fb08bfefc8, 0x0) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mheap .go:1150 +0x381 fp=0x7fff4a924748 sp=0x7fff4a924710 pc=0x55fb05658be1 runtime.(*mheap).alloc_m(0x55fb08be5aa0, 0x1, 0x2a, 0x6e43a318) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mheap .go:977 +0xc6 fp=0x7fff4a924798 sp=0x7fff4a924748 pc=0x55fb05658236 runtime.(*mheap).alloc.func1() /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mheap .go:1048 +0x4e fp=0x7fff4a9247d0 sp=0x7fff4a924798 pc=0x55fb0568935e runtime.(*mheap).alloc(0x55fb08be5aa0, 0x1, 0x55fb0501002a, 0x7fff4a924870) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mheap .go:1047 +0x8c fp=0x7fff4a924820 sp=0x7fff4a9247d0 pc=0x55fb0565850c runtime.(*mcentral).grow(0x55fb08be68a0, 0x0) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mcentral .go:256 +0x97 fp=0x7fff4a924868 sp=0x7fff4a924820 pc=0x55fb0564b367 runtime.(*mcentral).cacheSpan(0x55fb08be68a0, 0x7ff734a23000) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mcentral .go:106 +0x301 fp=0x7fff4a9248c8 sp=0x7fff4a924868 pc=0x55fb0564ae71 runtime.(*mcache).refill(0x7ff734a23008, 0x2a) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/mcache .go:135 +0x88 fp=0x7fff4a9248e8 sp=0x7fff4a9248c8 pc=0x55fb0564a908 runtime.(*mcache).nextFree(0x7ff734a23008, 0x55fb08bdb92a, 0x7ff734a23008, 0x7ff734a23000, 0x8) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/malloc .go:786 +0x8a fp=0x7fff4a924920 sp=0x7fff4a9248e8 pc=0x55fb0563ef3a runtime.mallocgc(0x180, 0x55fb0792fe20, 0x1, 0x55fb08bff020) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/malloc .go:939 +0x780 fp=0x7fff4a9249c0 sp=0x7fff4a924920 pc=0x55fb0563f870 runtime.newobject(0x55fb0792fe20, 0x4000) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/malloc .go:1068 +0x3a fp=0x7fff4a9249f0 sp=0x7fff4a9249c0 pc=0x55fb0563fc7a runtime.malg(0x521c300008000, 0x55fb08be8110) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/proc .go:3220 +0x33 fp=0x7fff4a924a30 sp=0x7fff4a9249f0 pc=0x55fb05669a83 runtime.mpreinit(...) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/os_linux .go:311 runtime.mcommoninit(0x55fb08bdfd60) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/proc .go:618 +0xc6 fp=0x7fff4a924a68 sp=0x7fff4a924a30 pc=0x55fb056633f6 runtime.schedinit() /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/proc .go:540 +0x78 fp=0x7fff4a924ac0 sp=0x7fff4a924a68 pc=0x55fb05663088 runtime.rt0_go(0x7fff4a924bc8, 0x2, 0x7fff4a924bc8, 0x0, 0x7ff734055b97, 0x2, 0x7fff4a924bc8, 0x200008000, 0x55fb0568b3d0, 0x0, ...) /build/docker .io-YkakXX /docker .io-19.03.6 /go/src/runtime/asm_amd64 .s:195 +0x11e fp=0x7fff4a924ac8 sp=0x7fff4a924ac0 pc=0x55fb0568b4fe ubuntu@VM-0-3-ubuntu:~$ |
解决方法
一是增加系统内存,二是优化进程,使其占用内存降低。
考虑到云服务器只有1G内存,目前考虑升级硬件配置。
关于docker使用物理内存的情况,网上搜罗了一下,大体是说需要保证2G的物理内存保障docker的运行。。
I can also easily see this error, with docker 1.8.2, kernel 4.3-rc4 on a virtual machine with 1G RAM. I suppose it's a similar issue as moby/moby#14460. However, with docker 1.8.2 the crash seems to still happen. As a workaround, it would be possible to raise system memory to 2G or higher. That'll work according to my tests. In contrast to other issues like #489, it's interesting that kernel OOM killer is not necessarily involved in this case. So it might make sense to limit memory utilization on the docker side.