<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Which application is filling memory? <br>
<br>
If it's a brick (glusterfsd) then stopping and starting a brick
("kill" and "gluster volume start ... force") will not waste
cycles re-healing any files that are healthy. Any heals of an
individual file that were not complete will be restarted as well
as any files that were changed while the brick was offline.<br>
</p>
If it's glusterd, that can be restarted at any time without
interfering with the volume.<br>
<br>
If it's glustershd ("/usr/bin/glusterfs -s localhost --volfile-id
gluster/glustershd ..."), you can restart that with "gluster volume
start ... force" (even if the volume is already started).<br>
<br>
<br>
<div class="moz-cite-prefix">On 12/29/2016 02:27 PM, Jackie Tung
wrote:<br>
</div>
<blockquote cite="mid:5FAD1025-9B21-4DF2-A30F-6837920929AB@drive.ai"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class="">Ravi,</div>
<div class=""><br class="">
</div>
Got it thanks. I’ve kicked this off, it seems be doing OK.
<div class=""><br class="">
</div>
<div class="">I am a little concerned about a slow creep of memory
usage:</div>
<div class=""><br class="">
</div>
<div class="">* swap (64GB) completed filled up on server_1</div>
<div class="">* general memory usage creeping up slowly over time.</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-size: 10px; line-height: normal;
font-family: Monaco; color: rgb(204, 204, 204);
background-color: rgba(0, 0, 0, 0.85098);" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">$
free -m</span></div>
<div style="margin: 0px; font-size: 10px; line-height: normal;
font-family: Monaco; color: rgb(204, 204, 204);
background-color: rgba(0, 0, 0, 0.85098);" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">
total used free shared
buff/cache available</span></div>
<div style="margin: 0px; font-size: 10px; line-height: normal;
font-family: Monaco; color: rgb(204, 204, 204);
background-color: rgba(0, 0, 0, 0.85098);" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Mem:
128829 55596 614 53
72618 71783</span></div>
<div style="margin: 0px; font-size: 10px; line-height: normal;
font-family: Monaco; color: rgb(204, 204, 204);
background-color: rgba(0, 0, 0, 0.85098);" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Swap:
61034 61034 0</span></div>
</div>
<div class=""><br class="">
</div>
<div class="">Similar issue on server_2, though lower starting
memory usage:</div>
<div class=""><br class="">
</div>
<div class="">The “available” number is slowly going down - at
this rate, probably will go to 0 before heal is done.</div>
<div class=""><br class="">
</div>
<div class="">We are actually running 3.8.6, I’d like to try to
pause the heal, upgrade to 3.8.7, and resume. Is this possible
heal suspend/resume possible or advisable?</div>
<div class=""><br class="">
</div>
<div class="">The upgrade idea came from this on bugzilla (not
100% if it will help my leak):</div>
<div class=""><br class="">
</div>
<div class=""><a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1400927"
class="">https://bugzilla.redhat.com/show_bug.cgi?id=1400927</a></div>
<div class=""><br class="">
</div>
<div class="">Even without doing the upgrade, I may need to
restart glusterfs-server anyway to reset memory usage.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Jackie</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 28, 2016, at 9:40 PM, Ravishankar N
<<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="moz-cite-prefix">On 12/29/2016 10:46 AM,
Jackie Tung wrote:<br class="">
</div>
<blockquote
cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com"
type="cite" class="">
<div dir="auto" class="">Thanks very much for the
advice.
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Would you mind elaborating
on the "no io" recommendation? It's somewhat hard
for me to guarantee this without a long
maintenance window.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">What is the consequence of
having IO at point of add-brick, and for the heal
period afterwards?</div>
</div>
</blockquote>
<br class="">
Sorry I wasn't clear. Since you're running 16
distribute legs (16x2), a lot of self-heals would be
running and there is a chance that clients might
experience slowness due to the self-heals. Other than
that it should be fine.<br class="">
Thanks,<br class="">
Ravi<br class="">
<br class="">
<blockquote
cite="mid:CAG0reA6KWA3C8xxjw6iUFyvRuYuFED7-XyqU1x5MppNifZ=7jQ@mail.gmail.com"
type="cite" class="">
<div dir="auto" class="">
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Dec 28, 2016 8:27 PM,
"Ravishankar N" <<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" class="">ravishankar@redhat.com</a>>
wrote:<br type="attribution" class="">
<blockquote class="quote" style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="">
<div class="elided-text">
<div
class="m_5051585420273836920moz-cite-prefix">On
12/29/2016 07:30 AM, Jackie Tung wrote:<br
class="">
</div>
<blockquote type="cite" class="">
<div dir="auto" class="">Version is 3.8.7
on Ubuntu xenial.</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Dec 28, 2016
5:56 PM, "Jackie Tung" <<a
moz-do-not-send="true"
href="mailto:jackie@drive.ai"
target="_blank" class="">jackie@drive.ai</a>>
wrote:<br type="attribution" class="">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="auto" class="">If someone
has experience to share in this
area, i'd be grateful. I have an
existing distributed replicated
volume, 2x16.
<div dir="auto" class=""><br
class="">
</div>
<div dir="auto" class="">We have a
third server ready to go.
Redhat docs say just run add
brick replica 3, then run
rebalance.</div>
<div dir="auto" class=""><br
class="">
</div>
<div dir="auto" class="">The
rebalance step feels a bit off
to me. Isn't some kind of heal
operation in order rather than
rebalance?</div>
<div dir="auto" class=""><br
class="">
</div>
<div dir="auto" class="">No
additional usable space will be
introduced, only replica count
increase from 2 to 3.</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
</div>
You don't need to run re-balance for
increasing the replica count. Heals should
automatically be triggered when you run
'gluster vol add-brick <volname> replica
3 <list of bricks for the 3rd replica>`.
It is advisable to do this when there is no
I/O happening on the volume. You can verify
that files are getting populated in the newly
added bricks post running the command. <br
class="">
<br class="">
-Ravi<br class="">
<blockquote type="cite" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="auto" class="">
<div dir="auto" class=""><br
class="">
</div>
<div dir="auto" class="">Thanks</div>
<div dir="auto" class="">Jackie</div>
</div>
</blockquote>
</div>
</div>
<br class="">
<p class=""><font class="" face="Courier
New" color="#808080" size="1">The
information in this email is
confidential and may be legally
privileged. It is intended solely for
the addressee. Access to this email by
anyone else is unauthorized. If you are
not the intended recipient, any
disclosure, copying, distribution or any
action taken or omitted to be taken in
reliance on it, is prohibited and may be
unlawful.</font></p>
<br class="">
<fieldset
class="m_5051585420273836920mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="">______________________________<wbr class="">_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_5051585420273836920moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr class="">mailman/listinfo/gluster-users</a></pre>
</blockquote><div class="">
</div>
</div>
</blockquote></div>
</div><p class=""><font class="" face="Courier New" color="#808080" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>
</blockquote><div class="">
</div></div></div></blockquote></div>
</div>
<p><font face="Courier New" color="#808080" size="1">The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.</font></p>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</body></html>