es集群里有一臺機器，突然cpu load飆到 21左右（8core cpu），但是cpu使用率會變成0，且同時io 等使用率全部變?yōu)?.這種狀態(tài)不可以自己恢復，除非重啟。es沒有判斷出節(jié)點有問題，整個集群不可以訪問，所有操作都超時，包括cat集群信息等接口，直到重啟機器。這種情況下節(jié)點已經(jīng)死了，但是節(jié)點還能夠響應其master發(fā)出的transport ping消息，master不會將其從集群摘除。日志里提示

monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
還有
Failed to connect to nodexxxxtimeout

從監(jiān)控看，流量沒有突然上升，且 disk io ops也沒突然飆升。

且這個時候使用kill -9 殺死es進程，殺不掉，原先進程會變成僵尸進程。load一直下不來。

僵死的時候 dmsg 輸出了日志：

[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] kthreadd        D ffffffffffffffff     0     2      0 0x00000000
[Wed Jun 14 18:03:48 2017]  ffff881fd2a234e0 0000000000000046 ffff881fd2998b80 ffff881fd2a23fd8
[Wed Jun 14 18:03:48 2017]  ffff881fd2a23fd8 ffff881fd2a23fd8 ffff881fd2998b80 ffff881fd2a23628
[Wed Jun 14 18:03:48 2017]  ffff881fd2a23630 7fffffffffffffff ffff881fd2998b80 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff81631ada>] ? __slab_free+0x10e/0x277
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a666>] kernel_thread+0x26/0x30
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a65f2>] kthreadd+0x2b2/0x2f0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6340>] ? kthread_create_on_cpu+0x60/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6340>] ? kthread_create_on_cpu+0x60/0x60
[Wed Jun 14 18:03:48 2017] INFO: task kswapd0:244 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] kswapd0         D ffffffffffffffff     0   244      2 0x00000000
[Wed Jun 14 18:03:48 2017]  ffff881fcda7f6f0 0000000000000046 ffff881fcd9ba280 ffff881fcda7ffd8
[Wed Jun 14 18:03:48 2017]  ffff881fcda7ffd8 ffff881fcda7ffd8 ffff881fcd9ba280 ffff881fcda7f838
[Wed Jun 14 18:03:48 2017]  ffff881fcda7f840 7fffffffffffffff ffff881fcd9ba280 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bd008>] ? __enqueue_entity+0x78/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c34b7>] ? enqueue_entity+0x237/0x890
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5eb7>] ? vmpressure+0x87/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81180131>] balance_pgdat+0x4b1/0x5e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff811803d3>] kswapd+0x173/0x450
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jun 14 18:03:48 2017]  [<ffffffff81180260>] ? balance_pgdat+0x5e0/0x5e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017] INFO: task xfsaild/sdb1:781 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] xfsaild/sdb1    D ffffffffffffffff     0   781      2 0x00000000
[Wed Jun 14 18:03:48 2017]  ffff881fcf1afb20 0000000000000046 ffff881fcfbe7300 ffff881fcf1affd8
[Wed Jun 14 18:03:48 2017]  ffff881fcf1affd8 ffff881fcf1affd8 ffff881fcfbe7300 ffff881fcf1afc68
[Wed Jun 14 18:03:48 2017]  ffff881fcf1afc70 7fffffffffffffff ffff881fcfbe7300 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8108d6fe>] ? try_to_del_timer_sync+0x5e/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4720>] _xfs_log_force+0x70/0x290 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8108bf50>] ? internal_add_timer+0x70/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4966>] xfs_log_force+0x26/0x80 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02cf470>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02cf5c1>] xfsaild+0x151/0x5e0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02cf470>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1594 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1594   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fcf91f5a0 0000000000000086 ffff881fccdaf300 ffff881fcf91ffd8
[Wed Jun 14 18:03:48 2017]  ffff881fcf91ffd8 ffff881fcf91ffd8 ffff881fccdaf300 ffff881fcf91f6e8
[Wed Jun 14 18:03:48 2017]  ffff881fcf91f6f0 7fffffffffffffff ffff881fccdaf300 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c4a74>] ? find_busiest_group+0x114/0x910
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1595 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1595   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fceb935a0 0000000000000086 ffff881fccdadc00 ffff881fceb93fd8
[Wed Jun 14 18:03:48 2017]  ffff881fceb93fd8 ffff881fceb93fd8 ffff881fccdadc00 ffff881fceb936e8
[Wed Jun 14 18:03:48 2017]  ffff881fceb936f0 7fffffffffffffff ffff881fccdadc00 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff81631ada>] ? __slab_free+0x10e/0x277
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1598 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1598   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fbec535a0 0000000000000086 ffff881fbec21700 ffff881fbec53fd8
[Wed Jun 14 18:03:48 2017]  ffff881fbec53fd8 ffff881fbec53fd8 ffff881fbec21700 ffff881fbec536e8
[Wed Jun 14 18:03:48 2017]  ffff881fbec536f0 7fffffffffffffff ffff881fbec21700 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c4a74>] ? find_busiest_group+0x114/0x910
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1601 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1601   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fcef035a0 0000000000000082 ffff881fbec23980 ffff881fcef03fd8
[Wed Jun 14 18:03:48 2017]  ffff881fcef03fd8 ffff881fcef03fd8 ffff881fbec23980 ffff881fcef036e8
[Wed Jun 14 18:03:48 2017]  ffff881fcef036f0 7fffffffffffffff ffff881fbec23980 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1609 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1609   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fbee335a0 0000000000000082 ffff881fbec27300 ffff881fbee33fd8
[Wed Jun 14 18:03:48 2017]  ffff881fbee33fd8 ffff881fbee33fd8 ffff881fbec27300 ffff881fbee336e8
[Wed Jun 14 18:03:48 2017]  ffff881fbee336f0 7fffffffffffffff ffff881fbec27300 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task java:9265 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] java            D ffffffffffffffff     0  9265      1 0x00000180
[Wed Jun 14 18:03:48 2017]  ffff880f94f075a0 0000000000000086 ffff880fe98bdc00 ffff880f94f07fd8
[Wed Jun 14 18:03:48 2017]  ffff880f94f07fd8 ffff880f94f07fd8 ffff880fe98bdc00 ffff880f94f076e8
[Wed Jun 14 18:03:48 2017]  ffff880f94f076f0 7fffffffffffffff ffff880fe98bdc00 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa0295e47>] ? xfs_iext_bno_to_ext+0xa7/0x1a0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bd008>] ? __enqueue_entity+0x78/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c34b7>] ? enqueue_entity+0x237/0x890
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8bd3>] ? wake_up_process+0x23/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff81122b33>] ? __secure_computing+0x73/0x240
[Wed Jun 14 18:03:48 2017]  [<ffffffff8110b796>] ? __audit_syscall_exit+0x1e6/0x280
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645b12>] ? tracesys+0xdd/0xe2
[Wed Jun 14 18:03:48 2017] INFO: task java:28683 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] java            D ffffffffffffffff     0 28683      1 0x00000180
[Wed Jun 14 18:03:48 2017]  ffff880f6d8ebc20 0000000000000086 ffff881fc8ffdc00 ffff880f6d8ebfd8
[Wed Jun 14 18:03:48 2017]  ffff880f6d8ebfd8 ffff880f6d8ebfd8 ffff881fc8ffdc00 ffff880f6d8ebd68
[Wed Jun 14 18:03:48 2017]  ffff880f6d8ebd70 7fffffffffffffff ffff881fc8ffdc00 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff81639b12>] ? down_read+0x12/0x30
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02a74d0>] xfs_file_fsync+0x1b0/0x200 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8120f975>] do_fsync+0x65/0xa0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8120fc63>] SyS_fdatasync+0x13/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645b12>] tracesys+0xdd/0xe2

網(wǎng)上搜索一下資料，有兩種場景。 一種是因為內(nèi)核默認啟用了Linux NUMA優(yōu)化，關掉就好了。 第二種是當時使用了xfs，換回ext4就好了。 第二種情況根源當時也沒有找到，只是因為之前跑在ext4上一直穩(wěn)定，就換回去了。是否是xfs和linux內(nèi)核之間存在bug導致

看一下關閉NUMA是否能解決你的問題，如果不行的化，再考慮是否是xfs的問題。

關閉NUMA的方法:

在/etc/sysctl.conf 中增加配置vm.zone_reclaim_mode = 0 ， 然后執(zhí)行 sysctl -p 使配置生效。文章來源地址http://www.zghlxwxcb.cn/news/detail-741882.html