[Gluster-users] bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) ... timeout = 1800

Fri Jun 14 09:28:06 UTC 2013

Hi, everyone:

We deploy GlusterFS3.2.7 with 24 servers and each server has 12 disks using
for brick server.

The volume type is DHT + AFR(replica=3) and transport type is socket.

Use native client and each server has 8 mount points.

The gluster client will log some error once in a while, following is an
example.

We google these error, but not find any method to resolve it. Can give me
some help, thanks!

[2013-06-14 15:58:41.913210] W [client-handshake.c:264:client_ping_cbk]
0-gfs1-client-174: timer must have expired

[2013-06-14 15:58:41.913439] E [rpc-clnt.c:341:saved_frames_unwind]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d]
(-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde])))
0-gfs1-client-184: forced unwinding frame type(GlusterFS Handshake)
op(PING(3)) called at 2013-06-14 15:57:36.516579

[2013-06-14 15:58:41.913456] W [client-handshake.c:264:client_ping_cbk]
0-gfs1-client-184: timer must have expired

[2013-06-14 15:58:41.913526] E [rpc-clnt.c:341:saved_frames_unwind]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d]
(-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde])))
0-gfs1-client-187: forced unwinding frame type(GlusterFS Handshake)
op(PING(3)) called at 2013-06-14 15:57:36.516591

[2013-06-14 15:58:41.913542] W [client-handshake.c:264:client_ping_cbk]
0-gfs1-client-187: timer must have expired

[2013-06-14 15:58:41.913662] E [rpc-clnt.c:341:saved_frames_unwind]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d]
(-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde])))
0-gfs1-client-170: forced unwinding frame type(GlusterFS Handshake)
op(PING(3)) called at 2013-06-14 15:57:36.516554

[2013-06-14 15:58:41.913678] W [client-handshake.c:264:client_ping_cbk]
0-gfs1-client-170: timer must have expired

[2013-06-14 15:58:41.914089] E [rpc-clnt.c:341:saved_frames_unwind]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d]
(-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde])))
0-gfs1-client-180: forced unwinding frame type(GlusterFS Handshake)
op(PING(3)) called at 2013-06-14 15:57:36.516543

[2013-06-14 15:58:41.914115] W [client-handshake.c:264:client_ping_cbk]
0-gfs1-client-180: timer must have expired

[2013-06-14 16:16:36.467651] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11081156x
sent = 2013-06-14 15:46:27.573575. timeout = 1800

[2013-06-14 16:16:36.467723] E
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote
operation failed: Transport endpoint is not connected

[2013-06-14 16:32:07.91967] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115092x
sent = 2013-06-14 16:02:02.712824. timeout = 1800

[2013-06-14 16:32:07.92014] E [client3_1-fops.c:1264:client3_1_finodelk_cbk]
0-gfs1-client-48: remote operation failed: Transport endpoint is not
connected

[2013-06-14 16:35:42.221471] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115696x
sent = 2013-06-14 16:05:41.199990. timeout = 1800

[2013-06-14 16:35:42.221586] E
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote
operation failed: Transport endpoint is not connected

[2013-06-14 16:36:12.766153] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115817x
sent = 2013-06-14 16:06:02.592259. timeout = 1800

[2013-06-14 16:36:12.766201] E
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote
operation failed: Transport endpoint is not connected

[2013-06-14 16:47:32.203692] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11095024x
sent = 2013-06-14 16:17:31.966654. timeout = 1800

[2013-06-14 16:47:32.203766] E
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote
operation failed: Transport endpoint is not connected

[2013-06-14 17:17:42.983248] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11112188x
sent = 2013-06-14 16:47:32.210391. timeout = 1800

[2013-06-14 17:17:42.983337] E
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote
operation failed: Transport endpoint is not connected

[2013-06-14 17:17:42.983413] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51:
bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11112186x
sent = 2013-06-14 16:47:32.206834. timeout = 1800

[2013-06-14 17:17:42.983449] E
[client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote
operation failed: Transport endpoint is not connected

[2013-06-14 17:19:15.201853] W [dict.c:418:dict_unref]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x32f320e4e5]
(-->/usr/lib/glusterfs/3.2.7/xlator/protocol/client.so(client3_1_fstat_cbk+0
x343) [0x7fd4d4c4a373]
(-->/usr/lib/glusterfs/3.2.7/xlator/cluster/replicate.so(afr_sh_data_fstat_c
bk+0x1eb) [0x7fd4d49ed90b]))) 0-dict: dict is NULL

Volume info as below:

[root at bj-nx-cip-w83 ~]# gluster volume info

Volume Name: gfs1

Type: Distributed-Replicate

Status: Started

Number of Bricks: 94 x 3 = 282

Transport-type: tcp

Bricks:

Brick1: 10.0.11.81:/xmail/disk1/gfs1

Brick2: 10.0.12.71:/xmail/disk1/gfs1

Brick3: 10.0.13.91:/xmail/disk1/gfs1

Brick4: 10.0.11.82:/xmail/disk1/gfs1

Brick5: 10.0.12.72:/xmail/disk1/gfs1

Brick6: 10.0.13.92:/xmail/disk1/gfs1

……

Brick280: 10.0.13.88:/xmail/disk12/gfs1

Brick281: 10.0.11.78:/xmail/disk12/gfs1

Brick282: 10.0.12.68:/xmail/disk12/gfs1

Options Reconfigured:

performance.read-ahead: on

performance.io-thread-count: 16

diagnostics.client-log-level: WARNING

diagnostics.brick-log-level: WARNING

nfs.disable: on

performance.io-cache: off

performance.write-behind: off

--------------------------

Cailiang song

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130614/02b5df9c/attachment.html>