Mar 29 10:09:19 k8s-node01 kernel: java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=739
Mar 29 10:09:19 k8s-node01 kernel: java cpuset=docker-137e8874e5d7c4154f169d09759d6436cea52f706fc7b7272a46ede4cfa3ae26.scope mems_allowed=0
Mar 29 10:09:19 k8s-node01 kernel: CPU: 1 PID: 2167 Comm: java Tainted: G ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Mar 29 10:09:19 k8s-node01 kernel: Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
Mar 29 10:09:19 k8s-node01 kernel: ffff8801c9b46eb0 00000000fe11f473 ffff88022ef27c90 ffffffff816a3db1
Mar 29 10:09:19 k8s-node01 kernel: ffff88022ef27d20 ffffffff8169f1a6 ffff8800bb2ed680 0000000000000001
Mar 29 10:09:19 k8s-node01 kernel: 0000000000000000 ffff8800bb7f7d10 ffff88022ef27cd0 0000000000000046
Mar 29 10:09:19 k8s-node01 kernel: Call Trace:
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff816a3db1>] dump_stack+0x19/0x1b
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff8169f1a6>] dump_header+0x90/0x229
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff81185ee6>] ? find_lock_task_mm+0x56/0xc0
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff81186394>] oom_kill_process+0x254/0x3d0
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff811f52a6>] mem_cgroup_oom_synchronize+0x546/0x570
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff811f4720>] ? mem_cgroup_charge_common+0xc0/0xc0
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff81186c24>] pagefault_out_of_memory+0x14/0x90
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff8169d56e>] mm_fault_error+0x68/0x12b
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff816b0231>] __do_page_fault+0x391/0x450
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff810295da>] ? __switch_to+0x15a/0x510
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff816b03d6>] trace_do_page_fault+0x56/0x150
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff816afa6a>] do_async_page_fault+0x1a/0xd0
Mar 29 10:09:19 k8s-node01 kernel: [<ffffffff816ac578>] async_page_fault+0x28/0x30
Mar 29 10:09:19 k8s-node01 kernel: Task in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8f22f6_32f6_11e8_aacb_00163e04c6d6.slice/docker-137e8874e5d7c4154f169d09759d6436cea52f706fc7b7272a46ede4cfa3ae26.scope killed as a result of limit of /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8f22f6_32f6_11e8_aacb_00163e04c6d6.slice
当前问题共有如下(2)个解决方案
- ship哥
看报错是发生oom killer了,尝试把物理内存申请大一些,或者把进程内存设置小一些
尝试使用 k8s 的 liveness 随时监控应用情况,暂时无法解决的问题可以先让 pod 不断重启止损