Crashes after stacktraces about flush and kswapd

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
 
(One intermediate revision not shown)
Line 1: Line 1:
== About ==
== About ==
-
* Kernel version: 2.6.32, 2.6.33
+
* Kernel version: 2.6.32, 2.6.33, 2.6.33.1
* [https://bugzilla.kernel.org/show_bug.cgi?id=15552 bug 15552]
* [https://bugzilla.kernel.org/show_bug.cgi?id=15552 bug 15552]
* [https://bugzilla.kernel.org/show_bug.cgi?id=15578 bug 15578]
* [https://bugzilla.kernel.org/show_bug.cgi?id=15578 bug 15578]
Line 9: Line 9:
* Something similar to the following appears in the logs:
* Something similar to the following appears in the logs:
<pre>
<pre>
-
Mar 11 06:45:14 river kernel: [40200.628071] kswapd0      D 0000000000000002    0    47      2 0x00000000
+
kswapd0      D 0000000000000002    0    47      2 0x00000000
-
Mar 11 06:45:14 river kernel: [40200.628076] ffff88022f073880 0000000000000046 0000000000000000 ffffffff810114ce
+
  ffff88022f073880 0000000000000046 0000000000000000 ffffffff810114ce
-
Mar 11 06:45:14 river kernel: [40200.628080] ffffffffb468199e 000000000000f8a0 ffff88022c8e3fd8 00000000000155c0
+
  ffffffffb468199e 000000000000f8a0 ffff88022c8e3fd8 00000000000155c0
-
Mar 11 06:45:14 river kernel: [40200.628084] 00000000000155c0 ffff88022f135bd0 ffff88022f135ec8 0000000100000000
+
  00000000000155c0 ffff88022f135bd0 ffff88022f135ec8 0000000100000000
-
Mar 11 06:45:14 river kernel: [40200.628088] Call Trace:
+
Call Trace:
-
Mar 11 06:45:14 river kernel: [40200.628112] [<ffffffff810114ce>] ? common_interrupt+0xe/0x13
+
  [<ffffffff810114ce>] ? common_interrupt+0xe/0x13
-
Mar 11 06:45:14 river kernel: [40200.628116] [<ffffffff81098e5e>] ? delayacct_end+0x74/0x7f
+
  [<ffffffff81098e5e>] ? delayacct_end+0x74/0x7f
-
Mar 11 06:45:14 river kernel: [40200.628129] [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
+
  [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628134] [<ffffffff812ee03d>] ? io_schedule+0x73/0xb7
+
  [<ffffffff812ee03d>] ? io_schedule+0x73/0xb7
-
Mar 11 06:45:14 river kernel: [40200.628140] [<ffffffffa038dc41>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
+
  [<ffffffffa038dc41>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628143] [<ffffffff812ee53d>] ? __wait_on_bit+0x41/0x70
+
  [<ffffffff812ee53d>] ? __wait_on_bit+0x41/0x70
-
Mar 11 06:45:14 river kernel: [40200.628148] [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
+
  [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
-
Mar 11 06:45:14 river kernel: [40200.628154] [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
+
  [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628157] [<ffffffff812ee5d7>] ? out_of_line_wait_on_bit+0x6b/0x77
+
  [<ffffffff812ee5d7>] ? out_of_line_wait_on_bit+0x6b/0x77
-
Mar 11 06:45:14 river kernel: [40200.628161] [<ffffffff81064a64>] ? wake_bit_function+0x0/0x23
+
  [<ffffffff81064a64>] ? wake_bit_function+0x0/0x23
-
Mar 11 06:45:14 river kernel: [40200.628168] [<ffffffffa0391bc3>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
+
  [<ffffffffa0391bc3>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628175] [<ffffffffa0391d84>] ? nfs_wb_page+0x94/0xc3 [nfs]
+
  [<ffffffffa0391d84>] ? nfs_wb_page+0x94/0xc3 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628179] [<ffffffff810b4968>] ? __remove_from_page_cache+0x33/0xb6
+
  [<ffffffff810b4968>] ? __remove_from_page_cache+0x33/0xb6
-
Mar 11 06:45:14 river kernel: [40200.628185] [<ffffffffa0384e54>] ? nfs_release_page+0x3a/0x57 [nfs]
+
  [<ffffffffa0384e54>] ? nfs_release_page+0x3a/0x57 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628189] [<ffffffff810bd244>] ? shrink_page_list+0x481/0x617
+
  [<ffffffff810bd244>] ? shrink_page_list+0x481/0x617
-
Mar 11 06:45:14 river kernel: [40200.628192] [<ffffffff8101166e>] ? apic_timer_interrupt+0xe/0x20
+
  [<ffffffff8101166e>] ? apic_timer_interrupt+0xe/0x20
-
Mar 11 06:45:14 river kernel: [40200.628195] [<ffffffff810bc317>] ? isolate_pages_global+0x1a0/0x20f
+
  [<ffffffff810bc317>] ? isolate_pages_global+0x1a0/0x20f
-
Mar 11 06:45:14 river kernel: [40200.628198] [<ffffffff810bdaf1>] ? shrink_list+0x44a/0x725
+
  [<ffffffff810bdaf1>] ? shrink_list+0x44a/0x725
-
Mar 11 06:45:14 river kernel: [40200.628206] [<ffffffffa0276828>] ? jbd2_journal_release_jbd_inode+0x55/0x10e [jbd2]
+
  [<ffffffffa0276828>] ? jbd2_journal_release_jbd_inode+0x55/0x10e [jbd2]
-
Mar 11 06:45:14 river kernel: [40200.628211] [<ffffffff810e2fd7>] ? add_partial+0x11/0x58
+
  [<ffffffff810e2fd7>] ? add_partial+0x11/0x58
-
Mar 11 06:45:14 river kernel: [40200.628214] [<ffffffff810be04c>] ? shrink_zone+0x280/0x342
+
  [<ffffffff810be04c>] ? shrink_zone+0x280/0x342
-
Mar 11 06:45:14 river kernel: [40200.628216] [<ffffffff810be24f>] ? shrink_slab+0x141/0x153
+
  [<ffffffff810be24f>] ? shrink_slab+0x141/0x153
-
Mar 11 06:45:14 river kernel: [40200.628219] [<ffffffff810bea71>] ? kswapd+0x4b9/0x683
+
  [<ffffffff810bea71>] ? kswapd+0x4b9/0x683
-
Mar 11 06:45:14 river kernel: [40200.628222] [<ffffffff810bc177>] ? isolate_pages_global+0x0/0x20f
+
  [<ffffffff810bc177>] ? isolate_pages_global+0x0/0x20f
-
Mar 11 06:45:14 river kernel: [40200.628224] [<ffffffff81064a36>] ? autoremove_wake_function+0x0/0x2e
+
  [<ffffffff81064a36>] ? autoremove_wake_function+0x0/0x2e
-
Mar 11 06:45:14 river kernel: [40200.628227] [<ffffffff810be5b8>] ? kswapd+0x0/0x683
+
  [<ffffffff810be5b8>] ? kswapd+0x0/0x683
-
Mar 11 06:45:14 river kernel: [40200.628229] [<ffffffff81064769>] ? kthread+0x79/0x81
+
  [<ffffffff81064769>] ? kthread+0x79/0x81
-
Mar 11 06:45:14 river kernel: [40200.628232] [<ffffffff81011baa>] ? child_rip+0xa/0x20
+
  [<ffffffff81011baa>] ? child_rip+0xa/0x20
-
Mar 11 06:45:14 river kernel: [40200.628234] [<ffffffff810646f0>] ? kthread+0x0/0x81
+
  [<ffffffff810646f0>] ? kthread+0x0/0x81
-
Mar 11 06:45:14 river kernel: [40200.628237] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
+
  [<ffffffff81011ba0>] ? child_rip+0x0/0x20
</pre>
</pre>
<pre>
<pre>
-
Mar 11 06:45:14 river kernel: [40200.628409] flush-0:24    D 0000000000000002    0  4682      2 0x00000000
+
flush-0:24    D 0000000000000002    0  4682      2 0x00000000
-
Mar 11 06:45:14 river kernel: [40200.628412] ffff88022f0754c0 0000000000000046 0000000000000000 ffff88020bfbf1ac
+
  ffff88022f0754c0 0000000000000046 0000000000000000 ffff88020bfbf1ac
-
Mar 11 06:45:14 river kernel: [40200.628415] 0000000000000000 000000000000f8a0 ffff88020bfbffd8 00000000000155c0
+
  0000000000000000 000000000000f8a0 ffff88020bfbffd8 00000000000155c0
-
Mar 11 06:45:14 river kernel: [40200.628418] 00000000000155c0 ffff880203973170 ffff880203973468 0000000200000000
+
  00000000000155c0 ffff880203973170 ffff880203973468 0000000200000000
-
Mar 11 06:45:14 river kernel: [40200.628421] Call Trace:
+
Call Trace:
-
Mar 11 06:45:14 river kernel: [40200.628423] [<ffffffff81098e5e>] ? delayacct_end+0x74/0x7f
+
  [<ffffffff81098e5e>] ? delayacct_end+0x74/0x7f
-
Mar 11 06:45:14 river kernel: [40200.628430] [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
+
  [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628433] [<ffffffff812ee03d>] ? io_schedule+0x73/0xb7
+
  [<ffffffff812ee03d>] ? io_schedule+0x73/0xb7
-
Mar 11 06:45:14 river kernel: [40200.628439] [<ffffffffa038dc41>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
+
  [<ffffffffa038dc41>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628442] [<ffffffff812ee53d>] ? __wait_on_bit+0x41/0x70
+
  [<ffffffff812ee53d>] ? __wait_on_bit+0x41/0x70
-
Mar 11 06:45:14 river kernel: [40200.628445] [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
+
  [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
-
Mar 11 06:45:14 river kernel: [40200.628451] [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
+
  [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628454] [<ffffffff812ee5d7>] ? out_of_line_wait_on_bit+0x6b/0x77
+
  [<ffffffff812ee5d7>] ? out_of_line_wait_on_bit+0x6b/0x77
-
Mar 11 06:45:14 river kernel: [40200.628456] [<ffffffff81064a64>] ? wake_bit_function+0x0/0x23
+
  [<ffffffff81064a64>] ? wake_bit_function+0x0/0x23
-
Mar 11 06:45:14 river kernel: [40200.628464] [<ffffffffa0391bc3>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
+
  [<ffffffffa0391bc3>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628471] [<ffffffffa0391d84>] ? nfs_wb_page+0x94/0xc3 [nfs]
+
  [<ffffffffa0391d84>] ? nfs_wb_page+0x94/0xc3 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628473] [<ffffffff810b4968>] ? __remove_from_page_cache+0x33/0xb6
+
  [<ffffffff810b4968>] ? __remove_from_page_cache+0x33/0xb6
-
Mar 11 06:45:14 river kernel: [40200.628479] [<ffffffffa0384e54>] ? nfs_release_page+0x3a/0x57 [nfs]
+
  [<ffffffffa0384e54>] ? nfs_release_page+0x3a/0x57 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628482] [<ffffffff810bd244>] ? shrink_page_list+0x481/0x617
+
  [<ffffffff810bd244>] ? shrink_page_list+0x481/0x617
-
Mar 11 06:45:14 river kernel: [40200.628486] [<ffffffff812564c7>] ? sch_direct_xmit+0x7f/0x14c
+
  [<ffffffff812564c7>] ? sch_direct_xmit+0x7f/0x14c
-
Mar 11 06:45:14 river kernel: [40200.628489] [<ffffffff810bc317>] ? isolate_pages_global+0x1a0/0x20f
+
  [<ffffffff810bc317>] ? isolate_pages_global+0x1a0/0x20f
-
Mar 11 06:45:14 river kernel: [40200.628492] [<ffffffff810bdaf1>] ? shrink_list+0x44a/0x725
+
  [<ffffffff810bdaf1>] ? shrink_list+0x44a/0x725
-
Mar 11 06:45:14 river kernel: [40200.628495] [<ffffffff810b9dfc>] ? determine_dirtyable_memory+0xd/0x1d
+
  [<ffffffff810b9dfc>] ? determine_dirtyable_memory+0xd/0x1d
-
Mar 11 06:45:14 river kernel: [40200.628498] [<ffffffff810b9e74>] ? get_dirty_limits+0x1d/0x259
+
  [<ffffffff810b9e74>] ? get_dirty_limits+0x1d/0x259
-
Mar 11 06:45:14 river kernel: [40200.628500] [<ffffffff810be04c>] ? shrink_zone+0x280/0x342
+
  [<ffffffff810be04c>] ? shrink_zone+0x280/0x342
-
Mar 11 06:45:14 river kernel: [40200.628504] [<ffffffff810c63ac>] ? zone_statistics+0x3c/0x5d
+
  [<ffffffff810c63ac>] ? zone_statistics+0x3c/0x5d
-
Mar 11 06:45:14 river kernel: [40200.628507] [<ffffffff810bf110>] ? try_to_free_pages+0x232/0x38e
+
  [<ffffffff810bf110>] ? try_to_free_pages+0x232/0x38e
-
Mar 11 06:45:14 river kernel: [40200.628510] [<ffffffff810bc177>] ? isolate_pages_global+0x0/0x20f
+
  [<ffffffff810bc177>] ? isolate_pages_global+0x0/0x20f
-
Mar 11 06:45:14 river kernel: [40200.628512] [<ffffffff810b92c5>] ? __alloc_pages_nodemask+0x3bb/0x5ce
+
  [<ffffffff810b92c5>] ? __alloc_pages_nodemask+0x3bb/0x5ce
-
Mar 11 06:45:14 river kernel: [40200.628516] [<ffffffff810e5190>] ? new_slab+0x42/0x1ca
+
  [<ffffffff810e5190>] ? new_slab+0x42/0x1ca
-
Mar 11 06:45:14 river kernel: [40200.628519] [<ffffffff810e5508>] ? __slab_alloc+0x1f0/0x39b
+
  [<ffffffff810e5508>] ? __slab_alloc+0x1f0/0x39b
-
Mar 11 06:45:14 river kernel: [40200.628526] [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
+
  [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628530] [<ffffffff810e67eb>] ? __kmalloc+0xf1/0x141
+
  [<ffffffff810e67eb>] ? __kmalloc+0xf1/0x141
-
Mar 11 06:45:14 river kernel: [40200.628532] [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
+
  [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
-
Mar 11 06:45:14 river kernel: [40200.628539] [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
+
  [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628546] [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
+
  [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628553] [<ffffffffa0392501>] ? nfs_flush_one+0x14/0xce [nfs]
+
  [<ffffffffa0392501>] ? nfs_flush_one+0x14/0xce [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628559] [<ffffffffa038daf0>] ? nfs_pageio_doio+0x2a/0x51 [nfs]
+
  [<ffffffffa038daf0>] ? nfs_pageio_doio+0x2a/0x51 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628566] [<ffffffffa038dbdc>] ? nfs_pageio_add_request+0xc5/0xd5 [nfs]
+
  [<ffffffffa038dbdc>] ? nfs_pageio_add_request+0xc5/0xd5 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628573] [<ffffffffa0390dfd>] ? nfs_do_writepage+0x100/0x122 [nfs]
+
  [<ffffffffa0390dfd>] ? nfs_do_writepage+0x100/0x122 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628580] [<ffffffffa0391316>] ? nfs_writepages_callback+0xf/0x21 [nfs]
+
  [<ffffffffa0391316>] ? nfs_writepages_callback+0xf/0x21 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628583] [<ffffffff810b9b69>] ? write_cache_pages+0x20b/0x327
+
  [<ffffffff810b9b69>] ? write_cache_pages+0x20b/0x327
-
Mar 11 06:45:14 river kernel: [40200.628590] [<ffffffffa0391307>] ? nfs_writepages_callback+0x0/0x21 [nfs]
+
  [<ffffffffa0391307>] ? nfs_writepages_callback+0x0/0x21 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628597] [<ffffffffa03912c6>] ? nfs_writepages+0xef/0x130 [nfs]
+
  [<ffffffffa03912c6>] ? nfs_writepages+0xef/0x130 [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628604] [<ffffffffa03924ed>] ? nfs_flush_one+0x0/0xce [nfs]
+
  [<ffffffffa03924ed>] ? nfs_flush_one+0x0/0xce [nfs]
-
Mar 11 06:45:14 river kernel: [40200.628607] [<ffffffff81064957>] ? bit_waitqueue+0x10/0xa0
+
  [<ffffffff81064957>] ? bit_waitqueue+0x10/0xa0
-
Mar 11 06:45:14 river kernel: [40200.628611] [<ffffffff8110637e>] ? writeback_single_inode+0xe7/0x2da
+
  [<ffffffff8110637e>] ? writeback_single_inode+0xe7/0x2da
-
Mar 11 06:45:14 river kernel: [40200.628614] [<ffffffff81107057>] ? writeback_inodes_wb+0x423/0x4fe
+
  [<ffffffff81107057>] ? writeback_inodes_wb+0x423/0x4fe
-
Mar 11 06:45:14 river kernel: [40200.628617] [<ffffffff8110725e>] ? wb_writeback+0x12c/0x1ab
+
  [<ffffffff8110725e>] ? wb_writeback+0x12c/0x1ab
-
Mar 11 06:45:14 river kernel: [40200.628620] [<ffffffff811073f2>] ? wb_do_writeback+0x73/0x15b
+
  [<ffffffff811073f2>] ? wb_do_writeback+0x73/0x15b
-
Mar 11 06:45:14 river kernel: [40200.628622] [<ffffffff8110750b>] ? bdi_writeback_task+0x31/0x9d
+
  [<ffffffff8110750b>] ? bdi_writeback_task+0x31/0x9d
-
Mar 11 06:45:14 river kernel: [40200.628626] [<ffffffff810c7c1e>] ? bdi_start_fn+0x0/0xca
+
  [<ffffffff810c7c1e>] ? bdi_start_fn+0x0/0xca
-
Mar 11 06:45:14 river kernel: [40200.628628] [<ffffffff810c7c8e>] ? bdi_start_fn+0x70/0xca
+
  [<ffffffff810c7c8e>] ? bdi_start_fn+0x70/0xca
-
Mar 11 06:45:14 river kernel: [40200.628631] [<ffffffff810c7c1e>] ? bdi_start_fn+0x0/0xca
+
  [<ffffffff810c7c1e>] ? bdi_start_fn+0x0/0xca
-
Mar 11 06:45:14 river kernel: [40200.628633] [<ffffffff81064769>] ? kthread+0x79/0x81
+
  [<ffffffff81064769>] ? kthread+0x79/0x81
-
Mar 11 06:45:14 river kernel: [40200.628636] [<ffffffff81011baa>] ? child_rip+0xa/0x20
+
  [<ffffffff81011baa>] ? child_rip+0xa/0x20
-
Mar 11 06:45:14 river kernel: [40200.628638]  [<ffffffff810646f0>] ? kthread+0x0/0x81
+
[<ffffffff810646f0>] ? kthread+0x0/0x81
-
Mar 11 06:45:14 river kernel: [40200.628640]  [<ffffffff81011ba0>] ? child_rip+0x0/0x20
+
[<ffffffff81011ba0>] ? child_rip+0x0/0x20
 +
</pre>
 +
<pre>
 +
INFO: task crond:3509 blocked for more than 120 seconds.
 +
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 +
crond        D ffff880008c0f948    0  3509      1 0x00000000
 +
ffff880221ae3498 0000000000000086 ffff880221ae3460 ffff880221ae345c
 +
0000000000000000 ffff880008c0f948 ffff880008d152c0 ffff88022504df70
 +
00000001002ef773 ffff880221ae3fd8 ffff880221ae2000 ffff880221ae2000
 +
Call Trace:
 +
[<ffffffffa0554290>] ? nfs_wait_bit_uninterruptible+0x0/0x10 [nfs]
 +
  [<ffffffff8135970e>] io_schedule+0x6e/0xb0
 +
[<ffffffffa0554299>] nfs_wait_bit_uninterruptible+0x9/0x10 [nfs]
 +
[<ffffffff81359e77>] __wait_on_bit+0x57/0x80
 +
[<ffffffffa0554290>] ? nfs_wait_bit_uninterruptible+0x0/0x10 [nfs]
 +
[<ffffffff81359f13>] out_of_line_wait_on_bit+0x73/0x90
 +
[<ffffffff81073090>] ? wake_bit_function+0x0/0x40
 +
[<ffffffffa0554287>] nfs_wait_on_request+0x27/0x30 [nfs]
 +
[<ffffffffa05593d0>] nfs_sync_mapping_wait+0x120/0x280 [nfs]
 +
[<ffffffffa0559ba9>] nfs_wb_page+0x79/0xd0 [nfs]
 +
[<ffffffffa05488bf>] nfs_release_page+0x5f/0x80 [nfs]
 +
[<ffffffff810dabfb>] try_to_release_page+0x2b/0x50
 +
[<ffffffff810e8af3>] shrink_page_list+0x453/0x5f0
 +
[<ffffffff810e8f59>] shrink_inactive_list+0x2c9/0x770
 +
[<ffffffff810088ee>] ? __switch_to+0x29e/0x360
 +
[<ffffffff810e3e45>] ? determine_dirtyable_memory+0x15/0x30
 +
[<ffffffff810e3ee2>] ? get_dirty_limits+0x22/0x2e0
 +
[<ffffffff810e9671>] shrink_zone+0x271/0x460
 +
[<ffffffff810e9a2a>] ? shrink_slab+0x13a/0x170
 +
[<ffffffff810ea47e>] do_try_to_free_pages+0xde/0x420
 +
[<ffffffff810ea94b>] try_to_free_pages+0x6b/0x70
 +
[<ffffffff810e7df0>] ? isolate_pages_global+0x0/0x270
 +
[<ffffffff810e2b6d>] __alloc_pages_nodemask+0x41d/0x700
 +
[<ffffffff810e4c90>] __do_page_cache_readahead+0x110/0x280
 +
[<ffffffff810e4e1c>] ra_submit+0x1c/0x20
 +
[<ffffffff810dc7df>] filemap_fault+0x37f/0x440
 +
[<ffffffff810f625f>] __do_fault+0x4f/0x590
 +
[<ffffffff8107735c>] ? lock_hrtimer_base+0x2c/0x60
 +
[<ffffffff810f8a0b>] handle_mm_fault+0x19b/0xa20
 +
[<ffffffff8135af60>] ? do_nanosleep+0x40/0xd0
 +
[<ffffffff81077a5f>] ? hrtimer_nanosleep+0xbf/0x180
 +
[<ffffffff81031683>] do_page_fault+0x133/0x320
 +
[<ffffffff8135c965>] page_fault+0x25/0x30
 +
</pre>
 +
<pre>
 +
INFO: task rsync:17708 blocked for more than 600 seconds.
 +
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 +
rsync        D 0000000000000002    0 17708  17707 0x00000000
 +
ffff880c7cfb6200 0000000000000086 0000000000000000 ffff88044b409334
 +
0000000000000286 000000000000f8e0 ffff88044b409fd8 0000000000015680
 +
0000000000015680 ffff88067bc6e900 ffff88067bc6ebf0 0000000181066af9
 +
Call Trace:
 +
[<ffffffff8100f789>] ? read_tsc+0xa/0x20
 +
  [<ffffffff8109411e>] ? delayacct_end+0x74/0x7f
 +
[<ffffffffa03c3c04>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 +
[<ffffffff812ed1cb>] ? io_schedule+0x73/0xb7
 +
[<ffffffffa03c3c0d>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
 +
[<ffffffff812ed6f6>] ? __wait_on_bit+0x41/0x70
 +
[<ffffffff81183d3c>] ? __lookup_tag+0xad/0x11b
 +
[<ffffffffa03c3c04>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 +
[<ffffffff812ed790>] ? out_of_line_wait_on_bit+0x6b/0x77
 +
[<ffffffff8105f300>] ? wake_bit_function+0x0/0x23
 +
[<ffffffffa03c7b87>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
 +
[<ffffffffa03c7d48>] ? nfs_wb_page+0x94/0xc3 [nfs]
 +
[<ffffffffa03badfc>] ? nfs_release_page+0x3a/0x57 [nfs]
 +
[<ffffffff810b9382>] ? shrink_page_list+0x48d/0x617
 +
[<ffffffff8103a9a1>] ? pick_next_task_fair+0xca/0xd5
 +
[<ffffffff810b843d>] ? isolate_pages_global+0x1a0/0x20f
 +
[<ffffffff810b9ee3>] ? shrink_zone+0x710/0xae9
 +
[<ffffffff810b52ba>] ? __alloc_pages_nodemask+0x2a7/0x5e1
 +
[<ffffffff81279ab1>] ? tcp_established_options+0x2d/0xa9
 +
[<ffffffff8103dcac>] ? find_busiest_group+0x3b0/0x875
 +
[<ffffffff8100f789>] ? read_tsc+0xa/0x20
 +
[<ffffffff810bb121>] ? do_try_to_free_pages+0x1ce/0x31e
 +
[<ffffffff810bb396>] ? try_to_free_pages+0x72/0x78
 +
[<ffffffff810b829d>] ? isolate_pages_global+0x0/0x20f
 +
[<ffffffff810b53cb>] ? __alloc_pages_nodemask+0x3b8/0x5e1
 +
[<ffffffff81053aaf>] ? lock_timer_base+0x26/0x4b
 +
[<ffffffff810e1314>] ? new_slab+0x42/0x1ca
 +
[<ffffffff810e168c>] ? __slab_alloc+0x1f0/0x3a2
 +
[<ffffffff810b0a58>] ? mempool_alloc+0x55/0x106
 +
[<ffffffff810b0a58>] ? mempool_alloc+0x55/0x106
 +
[<ffffffff810e1d66>] ? kmem_cache_alloc+0x7f/0xf0
 +
[<ffffffff810b0a58>] ? mempool_alloc+0x55/0x106
 +
[<ffffffff8105f2d2>] ? autoremove_wake_function+0x0/0x2e
 +
[<ffffffffa03c840e>] ? nfs_writedata_alloc+0x19/0x98 [nfs]
 +
[<ffffffffa03c84a1>] ? nfs_flush_one+0x14/0xce [nfs]
 +
[<ffffffffa03c3abc>] ? nfs_pageio_doio+0x2a/0x51 [nfs]
 +
[<ffffffffa03c3ba8>] ? nfs_pageio_add_request+0xc5/0xd5 [nfs]
 +
[<ffffffffa03c6e01>] ? nfs_do_writepage+0xe0/0x103 [nfs]
 +
[<ffffffffa03c72e7>] ? nfs_writepages_callback+0xf/0x21 [nfs]
 +
[<ffffffff810b5c38>] ? write_cache_pages+0x1be/0x2a4
 +
[<ffffffffa03c72d8>] ? nfs_writepages_callback+0x0/0x21 [nfs]
 +
[<ffffffffa03c7297>] ? nfs_writepages+0xde/0x11f [nfs]
 +
[<ffffffffa03c848d>] ? nfs_flush_one+0x0/0xce [nfs]
 +
[<ffffffffa03c8336>] ? nfs_write_mapping+0x55/0x8e [nfs]
 +
[<ffffffffa03bb273>] ? nfs_do_fsync+0x1c/0x3c [nfs]
 +
[<ffffffff810e820a>] ? filp_close+0x37/0x62
 +
[<ffffffff810e82c9>] ? sys_close+0x94/0xcd
 +
[<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
</pre>
</pre>

Latest revision as of 19:15, 22 October 2010

Contents

About

  • Kernel version: 2.6.32, 2.6.33, 2.6.33.1
  • bug 15552
  • bug 15578
  • Reported by: a.radke@arcor.de (March 17, 2010), lkolbe@techfak.uni-bielefeld.de (March 19, 2010)
  • Fixed by: Trond Myklemust (March 19, 2010)

Symptoms

  • Something similar to the following appears in the logs:
kswapd0       D 0000000000000002     0    47      2 0x00000000
 ffff88022f073880 0000000000000046 0000000000000000 ffffffff810114ce
 ffffffffb468199e 000000000000f8a0 ffff88022c8e3fd8 00000000000155c0
 00000000000155c0 ffff88022f135bd0 ffff88022f135ec8 0000000100000000
Call Trace:
 [<ffffffff810114ce>] ? common_interrupt+0xe/0x13
 [<ffffffff81098e5e>] ? delayacct_end+0x74/0x7f
 [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 [<ffffffff812ee03d>] ? io_schedule+0x73/0xb7
 [<ffffffffa038dc41>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
 [<ffffffff812ee53d>] ? __wait_on_bit+0x41/0x70
 [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
 [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 [<ffffffff812ee5d7>] ? out_of_line_wait_on_bit+0x6b/0x77
 [<ffffffff81064a64>] ? wake_bit_function+0x0/0x23
 [<ffffffffa0391bc3>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
 [<ffffffffa0391d84>] ? nfs_wb_page+0x94/0xc3 [nfs]
 [<ffffffff810b4968>] ? __remove_from_page_cache+0x33/0xb6
 [<ffffffffa0384e54>] ? nfs_release_page+0x3a/0x57 [nfs]
 [<ffffffff810bd244>] ? shrink_page_list+0x481/0x617
 [<ffffffff8101166e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff810bc317>] ? isolate_pages_global+0x1a0/0x20f
 [<ffffffff810bdaf1>] ? shrink_list+0x44a/0x725
 [<ffffffffa0276828>] ? jbd2_journal_release_jbd_inode+0x55/0x10e [jbd2]
 [<ffffffff810e2fd7>] ? add_partial+0x11/0x58
 [<ffffffff810be04c>] ? shrink_zone+0x280/0x342
 [<ffffffff810be24f>] ? shrink_slab+0x141/0x153
 [<ffffffff810bea71>] ? kswapd+0x4b9/0x683
 [<ffffffff810bc177>] ? isolate_pages_global+0x0/0x20f
 [<ffffffff81064a36>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff810be5b8>] ? kswapd+0x0/0x683
 [<ffffffff81064769>] ? kthread+0x79/0x81
 [<ffffffff81011baa>] ? child_rip+0xa/0x20
 [<ffffffff810646f0>] ? kthread+0x0/0x81
 [<ffffffff81011ba0>] ? child_rip+0x0/0x20
flush-0:24    D 0000000000000002     0  4682      2 0x00000000
 ffff88022f0754c0 0000000000000046 0000000000000000 ffff88020bfbf1ac
 0000000000000000 000000000000f8a0 ffff88020bfbffd8 00000000000155c0
 00000000000155c0 ffff880203973170 ffff880203973468 0000000200000000
Call Trace:
 [<ffffffff81098e5e>] ? delayacct_end+0x74/0x7f
 [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 [<ffffffff812ee03d>] ? io_schedule+0x73/0xb7
 [<ffffffffa038dc41>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
 [<ffffffff812ee53d>] ? __wait_on_bit+0x41/0x70
 [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
 [<ffffffffa038dc38>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 [<ffffffff812ee5d7>] ? out_of_line_wait_on_bit+0x6b/0x77
 [<ffffffff81064a64>] ? wake_bit_function+0x0/0x23
 [<ffffffffa0391bc3>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
 [<ffffffffa0391d84>] ? nfs_wb_page+0x94/0xc3 [nfs]
 [<ffffffff810b4968>] ? __remove_from_page_cache+0x33/0xb6
 [<ffffffffa0384e54>] ? nfs_release_page+0x3a/0x57 [nfs]
 [<ffffffff810bd244>] ? shrink_page_list+0x481/0x617
 [<ffffffff812564c7>] ? sch_direct_xmit+0x7f/0x14c
 [<ffffffff810bc317>] ? isolate_pages_global+0x1a0/0x20f
 [<ffffffff810bdaf1>] ? shrink_list+0x44a/0x725
 [<ffffffff810b9dfc>] ? determine_dirtyable_memory+0xd/0x1d
 [<ffffffff810b9e74>] ? get_dirty_limits+0x1d/0x259
 [<ffffffff810be04c>] ? shrink_zone+0x280/0x342
 [<ffffffff810c63ac>] ? zone_statistics+0x3c/0x5d
 [<ffffffff810bf110>] ? try_to_free_pages+0x232/0x38e
 [<ffffffff810bc177>] ? isolate_pages_global+0x0/0x20f
 [<ffffffff810b92c5>] ? __alloc_pages_nodemask+0x3bb/0x5ce
 [<ffffffff810e5190>] ? new_slab+0x42/0x1ca
 [<ffffffff810e5508>] ? __slab_alloc+0x1f0/0x39b
 [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
 [<ffffffff810e67eb>] ? __kmalloc+0xf1/0x141
 [<ffffffff8118a10f>] ? __lookup_tag+0xad/0x11b
 [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
 [<ffffffffa03924c9>] ? nfs_writedata_alloc+0x74/0x98 [nfs]
 [<ffffffffa0392501>] ? nfs_flush_one+0x14/0xce [nfs]
 [<ffffffffa038daf0>] ? nfs_pageio_doio+0x2a/0x51 [nfs]
 [<ffffffffa038dbdc>] ? nfs_pageio_add_request+0xc5/0xd5 [nfs]
 [<ffffffffa0390dfd>] ? nfs_do_writepage+0x100/0x122 [nfs]
 [<ffffffffa0391316>] ? nfs_writepages_callback+0xf/0x21 [nfs]
 [<ffffffff810b9b69>] ? write_cache_pages+0x20b/0x327
 [<ffffffffa0391307>] ? nfs_writepages_callback+0x0/0x21 [nfs]
 [<ffffffffa03912c6>] ? nfs_writepages+0xef/0x130 [nfs]
 [<ffffffffa03924ed>] ? nfs_flush_one+0x0/0xce [nfs]
 [<ffffffff81064957>] ? bit_waitqueue+0x10/0xa0
 [<ffffffff8110637e>] ? writeback_single_inode+0xe7/0x2da
 [<ffffffff81107057>] ? writeback_inodes_wb+0x423/0x4fe
 [<ffffffff8110725e>] ? wb_writeback+0x12c/0x1ab
 [<ffffffff811073f2>] ? wb_do_writeback+0x73/0x15b
 [<ffffffff8110750b>] ? bdi_writeback_task+0x31/0x9d
 [<ffffffff810c7c1e>] ? bdi_start_fn+0x0/0xca
 [<ffffffff810c7c8e>] ? bdi_start_fn+0x70/0xca
 [<ffffffff810c7c1e>] ? bdi_start_fn+0x0/0xca
 [<ffffffff81064769>] ? kthread+0x79/0x81
 [<ffffffff81011baa>] ? child_rip+0xa/0x20
 [<ffffffff810646f0>] ? kthread+0x0/0x81
 [<ffffffff81011ba0>] ? child_rip+0x0/0x20
INFO: task crond:3509 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
crond         D ffff880008c0f948     0  3509      1 0x00000000
 ffff880221ae3498 0000000000000086 ffff880221ae3460 ffff880221ae345c
 0000000000000000 ffff880008c0f948 ffff880008d152c0 ffff88022504df70
 00000001002ef773 ffff880221ae3fd8 ffff880221ae2000 ffff880221ae2000
Call Trace:
 [<ffffffffa0554290>] ? nfs_wait_bit_uninterruptible+0x0/0x10 [nfs]
 [<ffffffff8135970e>] io_schedule+0x6e/0xb0
 [<ffffffffa0554299>] nfs_wait_bit_uninterruptible+0x9/0x10 [nfs]
 [<ffffffff81359e77>] __wait_on_bit+0x57/0x80
 [<ffffffffa0554290>] ? nfs_wait_bit_uninterruptible+0x0/0x10 [nfs]
 [<ffffffff81359f13>] out_of_line_wait_on_bit+0x73/0x90
 [<ffffffff81073090>] ? wake_bit_function+0x0/0x40
 [<ffffffffa0554287>] nfs_wait_on_request+0x27/0x30 [nfs]
 [<ffffffffa05593d0>] nfs_sync_mapping_wait+0x120/0x280 [nfs]
 [<ffffffffa0559ba9>] nfs_wb_page+0x79/0xd0 [nfs]
 [<ffffffffa05488bf>] nfs_release_page+0x5f/0x80 [nfs]
 [<ffffffff810dabfb>] try_to_release_page+0x2b/0x50
 [<ffffffff810e8af3>] shrink_page_list+0x453/0x5f0
 [<ffffffff810e8f59>] shrink_inactive_list+0x2c9/0x770
 [<ffffffff810088ee>] ? __switch_to+0x29e/0x360
 [<ffffffff810e3e45>] ? determine_dirtyable_memory+0x15/0x30
 [<ffffffff810e3ee2>] ? get_dirty_limits+0x22/0x2e0
 [<ffffffff810e9671>] shrink_zone+0x271/0x460
 [<ffffffff810e9a2a>] ? shrink_slab+0x13a/0x170
 [<ffffffff810ea47e>] do_try_to_free_pages+0xde/0x420
 [<ffffffff810ea94b>] try_to_free_pages+0x6b/0x70
 [<ffffffff810e7df0>] ? isolate_pages_global+0x0/0x270
 [<ffffffff810e2b6d>] __alloc_pages_nodemask+0x41d/0x700
 [<ffffffff810e4c90>] __do_page_cache_readahead+0x110/0x280
 [<ffffffff810e4e1c>] ra_submit+0x1c/0x20
 [<ffffffff810dc7df>] filemap_fault+0x37f/0x440
 [<ffffffff810f625f>] __do_fault+0x4f/0x590
 [<ffffffff8107735c>] ? lock_hrtimer_base+0x2c/0x60
 [<ffffffff810f8a0b>] handle_mm_fault+0x19b/0xa20
 [<ffffffff8135af60>] ? do_nanosleep+0x40/0xd0
 [<ffffffff81077a5f>] ? hrtimer_nanosleep+0xbf/0x180
 [<ffffffff81031683>] do_page_fault+0x133/0x320
 [<ffffffff8135c965>] page_fault+0x25/0x30
INFO: task rsync:17708 blocked for more than 600 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsync         D 0000000000000002     0 17708  17707 0x00000000
 ffff880c7cfb6200 0000000000000086 0000000000000000 ffff88044b409334
 0000000000000286 000000000000f8e0 ffff88044b409fd8 0000000000015680
 0000000000015680 ffff88067bc6e900 ffff88067bc6ebf0 0000000181066af9
Call Trace:
 [<ffffffff8100f789>] ? read_tsc+0xa/0x20
 [<ffffffff8109411e>] ? delayacct_end+0x74/0x7f
 [<ffffffffa03c3c04>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 [<ffffffff812ed1cb>] ? io_schedule+0x73/0xb7
 [<ffffffffa03c3c0d>] ? nfs_wait_bit_uninterruptible+0x9/0xd [nfs]
 [<ffffffff812ed6f6>] ? __wait_on_bit+0x41/0x70
 [<ffffffff81183d3c>] ? __lookup_tag+0xad/0x11b
 [<ffffffffa03c3c04>] ? nfs_wait_bit_uninterruptible+0x0/0xd [nfs]
 [<ffffffff812ed790>] ? out_of_line_wait_on_bit+0x6b/0x77
 [<ffffffff8105f300>] ? wake_bit_function+0x0/0x23
 [<ffffffffa03c7b87>] ? nfs_sync_mapping_wait+0xfa/0x227 [nfs]
 [<ffffffffa03c7d48>] ? nfs_wb_page+0x94/0xc3 [nfs]
 [<ffffffffa03badfc>] ? nfs_release_page+0x3a/0x57 [nfs]
 [<ffffffff810b9382>] ? shrink_page_list+0x48d/0x617
 [<ffffffff8103a9a1>] ? pick_next_task_fair+0xca/0xd5
 [<ffffffff810b843d>] ? isolate_pages_global+0x1a0/0x20f
 [<ffffffff810b9ee3>] ? shrink_zone+0x710/0xae9
 [<ffffffff810b52ba>] ? __alloc_pages_nodemask+0x2a7/0x5e1
 [<ffffffff81279ab1>] ? tcp_established_options+0x2d/0xa9
 [<ffffffff8103dcac>] ? find_busiest_group+0x3b0/0x875
 [<ffffffff8100f789>] ? read_tsc+0xa/0x20
 [<ffffffff810bb121>] ? do_try_to_free_pages+0x1ce/0x31e
 [<ffffffff810bb396>] ? try_to_free_pages+0x72/0x78
 [<ffffffff810b829d>] ? isolate_pages_global+0x0/0x20f
 [<ffffffff810b53cb>] ? __alloc_pages_nodemask+0x3b8/0x5e1
 [<ffffffff81053aaf>] ? lock_timer_base+0x26/0x4b
 [<ffffffff810e1314>] ? new_slab+0x42/0x1ca
 [<ffffffff810e168c>] ? __slab_alloc+0x1f0/0x3a2
 [<ffffffff810b0a58>] ? mempool_alloc+0x55/0x106
 [<ffffffff810b0a58>] ? mempool_alloc+0x55/0x106
 [<ffffffff810e1d66>] ? kmem_cache_alloc+0x7f/0xf0
 [<ffffffff810b0a58>] ? mempool_alloc+0x55/0x106
 [<ffffffff8105f2d2>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffffa03c840e>] ? nfs_writedata_alloc+0x19/0x98 [nfs]
 [<ffffffffa03c84a1>] ? nfs_flush_one+0x14/0xce [nfs]
 [<ffffffffa03c3abc>] ? nfs_pageio_doio+0x2a/0x51 [nfs]
 [<ffffffffa03c3ba8>] ? nfs_pageio_add_request+0xc5/0xd5 [nfs]
 [<ffffffffa03c6e01>] ? nfs_do_writepage+0xe0/0x103 [nfs]
 [<ffffffffa03c72e7>] ? nfs_writepages_callback+0xf/0x21 [nfs]
 [<ffffffff810b5c38>] ? write_cache_pages+0x1be/0x2a4
 [<ffffffffa03c72d8>] ? nfs_writepages_callback+0x0/0x21 [nfs]
 [<ffffffffa03c7297>] ? nfs_writepages+0xde/0x11f [nfs]
 [<ffffffffa03c848d>] ? nfs_flush_one+0x0/0xce [nfs]
 [<ffffffffa03c8336>] ? nfs_write_mapping+0x55/0x8e [nfs]
 [<ffffffffa03bb273>] ? nfs_do_fsync+0x1c/0x3c [nfs]
 [<ffffffff810e820a>] ? filp_close+0x37/0x62
 [<ffffffff810e82c9>] ? sys_close+0x94/0xcd
 [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b

Cause

  • A deadlock caused by freeing a page with __GFP_FS set

Resolution

commit d812e575822a2b7ab1a7cadae2571505ec6ec2bd
Author: Trond Myklebust <Trond.Myklebust@netapp.com>
Date:   Fri Mar 19 13:55:17 2010 -0400

    NFS: Prevent another deadlock in nfs_release_page()
    
    We should not attempt to free the page if __GFP_FS is not set. Otherwise we
    can deadlock as per
    
      http://bugzilla.kernel.org/show_bug.cgi?id=15578
    
    Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
    Cc: stable@kernel.org

Personal tools