As part of my job, I am doing some tests with the Red Hat Storage Software Appliance. The current version of RHSSA is based on Gluster 3.2.5. After some experiments, it seem that Gluster is pretty cool and surprisingly easy to setup.
In order to see what is going happening on the network, I captured some tcpdumps. Reading the files in Wireshark, does not show any Gluster specifics. It seems that Wireshark does not know how to decode (or rather dissect) the Gluster traffic. Very unfortunate, as quite some future troubleshooting and performance analysis may require investigating the network packets.
Luckily the Wireshark Developer’s guide contains a chapter on Adding a basic dissector. After writing some code and tests, I now have some Wireshark packages that recognize some Gluster communication. The RPMs are available for testing, feedback over email is appreciated.
With the updated packages, the output of tshark (the terminal version of Wireshark) identifies some Gluster packets:
$ tshark -r gluster-communication.cap 'tcp.len > 0' | head
7 0.002572 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
8 0.002633 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
11 0.002909 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 7)
12 0.002918 192.168.122.137 -> 192.168.122.184 Gluster Dump V1 DUMP Reply (Call In 8)
15 0.003104 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
16 0.003158 192.168.122.184 -> 192.168.122.137 Gluster Portmap V1 PORTBYBRICK Call
17 0.003298 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 15)
18 0.003310 192.168.122.137 -> 192.168.122.184 Gluster Portmap V1 PORTBYBRICK Reply (Call In 16)
31 3.013909 192.168.122.184 -> 192.168.122.137 Gluster Dump V1 DUMP Call
32 3.013965 192.168.122.184 -> 192.168.122.137 PCEP Unknown Message (0).
As with several other protocols, Wireshark detects some packets as non-gluster ones. In this tcpdump, there surely is no PCEP traffic (last line in the above output). Each dissector for a protocol should do some sanity checks to see if a packet belongs to its protocol. These checks are not easy to do, and hence quite some protocols detect packets from Gluster as their communication stream.
Luckily it is possible to disable a protocol in the ~/.wireshark/disabled_protos file. Finding the correct names of a protocol isn’t always straight forward. Use Wireshark to graphically create the file is the easiest, it also takes care of disabling the protocols that are possibly encapsulated. In Wireshark
After these steps, tshark should recognize all traffic to and from port 24007 as belonging to one of the Gluster protocols. I have only tested the Wireshark dissectors on Gluster 3.2.5, later releases use some newer versions of some protocols and these may not be detected yet.
2020 has not been a year we would have been able to predict. With a worldwide pandemic and lives thrown out of gear, as we head into 2021, we are thankful that our community and project continued to receive new developers, users and make small gains. For that and a...
It has been a while since we provided an update to the Gluster community. Across the world various nations, states and localities have put together sets of guidelines around shelter-in-place and quarantine. We request our community members to stay safe, to care for their loved ones, to continue to be...
The initial rounds of conversation around the planning of content for release 8 has helped the project identify one key thing – the need to stagger out features and enhancements over multiple releases. Thus, while release 8 is unlikely to be feature heavy as previous releases, it will be the...