[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy

Shaik Salam shaik.salam at tcs.com
Mon Jan 28 08:27:25 UTC 2019


Hi Raghavendra,

Could you please have look my issue If you have time.
Thanks in advance.

BR
Salam



From:   Shaik Salam <shaik.salam at tcs.com>
To:     rtalur at redhat.com
Cc:     "gluster-users at gluster.org List" <gluster-users at gluster.org>, John 
Mulligan <jmulliga at redhat.com>, Michael Adam <madam at redhat.com>
Date:   01/28/2019 12:42 PM
Subject:        Re: [Gluster-users] Failed to provision volume with 
StorageClass "glusterfs-storage": glusterfs: server busy
Sent by:        gluster-users-bounces at gluster.org



"External email. Open with Caution"
  
Hi Raghavendra, 
We are also facing following issue which is mentioned in case on openshift 
origin while we are creating pvc for pods.  (Please provide workaround to 
move further (pod restart doesn't workout) 
https://bugzilla.redhat.com/show_bug.cgi?id=1630117 
https://bugzilla.redhat.com/show_bug.cgi?id=1636912

Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume Server busy. Retry 
operation later.. 
at a time to create only one volume when in-flight operations are zero. 
Once volume requested it reaches to 8. 
Now single volume not able to create and we are till now mostly 10 volumes 
are created.

Please find heketidb dump and log 

[negroni] Completed 200 OK in 98.699µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 106.654µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 185.406µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 102.664µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 192.658µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 198.611µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 124.254µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 101.491µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 116.997µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 100.171µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 109.238µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/28 06:50:57 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 191.118µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 188.791µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 94.436µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 110.893µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 112.132µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 96.15µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 112.682µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 140.543µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 182.066µs 
[negroni] Started GET /queue/2604dd5965445711a4b6bc28592cb0f6 
[negroni] Completed 200 OK in 151.572µs 

BR 
Salam 



From:        Shaik Salam/HYD/TCS 
To:        "John Mulligan" <jmulliga at redhat.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com>, Madhu Rajanna <mrajanna at redhat.com> 
Date:        01/25/2019 04:03 PM 
Subject:        Re: [Gluster-users] Failed to provision volume with 
StorageClass "glusterfs-storage": glusterfs: server busy 


Hi John, 

Could you please have look my issue If you have time (atleast provide 
workaround). 
Thanks in advance. 

BR 
Salam 




From:        "Shaik Salam" <shaik.salam at tcs.com> 
To:         
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com> 
Date:        01/25/2019 02:55 PM 
Subject:        Re: [Gluster-users] Failed to provision volume with 
StorageClass "glusterfs-storage": glusterfs: server busy 
Sent by:        gluster-users-bounces at gluster.org 



"External email. Open with Caution" 
Hi John, 

Please find db dump and heketi log. Here kernel version. Please let me 
know If you need more information. 

[root at app2 ~]# uname -a 
Linux app2.matrix.nokia.com 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 
16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux 

Hardware: HP GEN8 

OS; NAME="CentOS Linux" 
VERSION="7 (Core)" 
ID="centos" 
ID_LIKE="rhel fedora" 
VERSION_ID="7" 
PRETTY_NAME="CentOS Linux 7 (Core)" 
ANSI_COLOR="0;31" 
CPE_NAME="cpe:/o:centos:centos:7" 
HOME_URL="https://www.centos.org/" 
BUG_REPORT_URL="https://bugs.centos.org/" 

CENTOS_MANTISBT_PROJECT="CentOS-7" 
CENTOS_MANTISBT_PROJECT_VERSION="7" 
REDHAT_SUPPORT_PRODUCT="centos" 
REDHAT_SUPPORT_PRODUCT_VERSION="7" 

 




From:        "Madhu Rajanna" <mrajanna at redhat.com> 
To:        "Shaik Salam" <shaik.salam at tcs.com>, "John Mulligan" 
<jmulliga at redhat.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com> 
Date:        01/24/2019 10:52 PM 
Subject:        Re: Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 



"External email. Open with Caution" 
Adding John who is having more idea about how to debug this one. 

@Shaik Salam can you some more info on the hardware on which you are 
running heketi (kernel details) 

On Thu, Jan 24, 2019 at 7:42 PM Shaik Salam <shaik.salam at tcs.com> wrote: 
Hi Madhu, 

Sorry to disturb could you please provide atleast work around (to clear 
requests which stuck) to move further. 
We are also not able to find root cause from glusterd logs. Please find 
attachment. 

BR 
Salam 

 



From:        Shaik Salam/HYD/TCS 
To:        "Madhu Rajanna" <mrajanna at redhat.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com> 
Date:        01/24/2019 04:12 PM 
Subject:        Re: Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 


Hi Madhu, 

Please let me know If any other information required. 

BR 
Salam 




From:        Shaik Salam/HYD/TCS 
To:        "Madhu Rajanna" <mrajanna at redhat.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com> 
Date:        01/24/2019 03:23 PM 
Subject:        Re: Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 


Hi Madhu, 

This is complete one after restart of heketi pod and process log. 

BR 
Salam 

[attachment "heketi-pod-complete.log" deleted by Shaik Salam/HYD/TCS] 
[attachment "ps-aux.txt" deleted by Shaik Salam/HYD/TCS] 




From:        "Madhu Rajanna" <mrajanna at redhat.com> 
To:        "Shaik Salam" <shaik.salam at tcs.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com> 
Date:        01/24/2019 01:55 PM 
Subject:        Re: Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 



"External email. Open with Caution" 
the logs you provided is not complete, not able to figure out which 
command is struck, can you reattach the complete output of `ps aux` and 
also attach complete heketi logs. 

On Thu, Jan 24, 2019 at 1:41 PM Shaik Salam <shaik.salam at tcs.com> wrote: 
Hi Madhu, 

Please find requested info. 

BR 
Salam 

 



From:        Madhu Rajanna <mrajanna at redhat.com> 
To:        Shaik Salam <shaik.salam at tcs.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
Michael Adam <madam at redhat.com> 
Date:        01/24/2019 01:33 PM 
Subject:        Re: Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 



"External email. Open with Caution" 
the heketi logs you have attached is not complete i believe, can you 
povide  the complete heketi logs 
and also an we get the output of "ps aux" from the gluster pods ? I want 
to see if any lvm commands or gluster commands are "stuck". 


On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <shaik.salam at tcs.com> wrote: 
Hi Madhu. 

I tried lot of times restarted heketi pod but not resolved. 

sh-4.4# heketi-cli server operations info 
Operation Counts: 
 Total: 0 
 In-Flight: 0 
 New: 0 
 Stale: 0 

Now you can see all operations are zero. Now I try to create single volume 
below is observation in-flight reaching slowly to 8. 

sh-4.4# heketi-cli server operations infoCLI_SERVER=http://localhost:8080 
; export HEKETI_CLI_USE                                       Operation 
Counts: 
 Total: 0 
 In-Flight: 6 
 New: 0 
 Stale: 0 
sh-4.4# heketi-cli server operations info 
Operation Counts: 
 Total: 0 
 In-Flight: 7 
 New: 0 
 Stale: 0 
sh-4.4# heketi-cli server operations info 
Operation Counts: 
 Total: 0 
 In-Flight: 7 
 New: 0 
 Stale: 0 
sh-4.4# heketi-cli server operations info 
Operation Counts: 
 Total: 0 
 In-Flight: 7 
 New: 0 
 Stale: 0 
sh-4.4# heketi-cli server operations info 
Operation Counts: 
 Total: 0 
 In-Flight: 7 
 New: 0 
 Stale: 0 

[negroni] Completed 200 OK in 186.286µs 
[negroni] Started POST /volumes 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 166.294µs 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 186.411µs 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 179.796µs 
[negroni] Started POST /volumes 
[negroni] Started POST /volumes 
[negroni] Started POST /volumes 
[negroni] Started POST /volumes 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 131.108µs 
[negroni] Started POST /volumes 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 111.392µs 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 265.023µs 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 179.364µs 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 295.058µs 
[negroni] Started GET /operations 
[negroni] Completed 200 OK in 146.857µs 
[negroni] Started POST /volumes 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 403.166µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 193.554µs 


But for pod volume is not creating. 
1:15:36 PM 
Warning
Provisioning failed  
Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume Server busy. Retry 
operation later.. 
9 times in the last 2 minutes 
1:13:21 PM 
Warning
Provisioning failed  
Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume . 
8 times in the last 









From:        "Madhu Rajanna" <mrajanna at redhat.com> 
To:        "Shaik Salam" <shaik.salam at tcs.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org>, 
"Michael Adam" <madam at redhat.com> 
Date:        01/24/2019 12:51 PM 
Subject:        Re: Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 



"External email. Open with Caution" 
HI Shaik, 

  can you provide me the outpout of $heketi-cli server operations info 
from heketi pod 
as a workround you can try restarting the heketi pod. This will cause the 
current  operations to go stale, but other pending pvcs may go to Bound 
state 
Regards, 
Madhu R 

On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <shaik.salam at tcs.com> wrote: 
H Madhu, 

Could you please have look my issue If you have time (atleast workaround). 

I am unable to send mail to "John Mulligan" <John_Mulligan at redhat.com>" 
who is currently handling issue 
https://bugzilla.redhat.com/show_bug.cgi?id=1636912 

BR 
Salam 


From:        Shaik Salam/HYD/TCS 
To:        "John Mulligan" <John_Mulligan at redhat.com>, "Michael Adam" <
madam at redhat.com>, "Madhu Rajanna" <mrajanna at redhat.com> 
Cc:        "gluster-users at gluster.org List" <gluster-users at gluster.org> 
Date:        01/24/2019 12:21 PM 
Subject:        Failed to provision volume with StorageClass 
"glusterfs-storage": glusterfs: server busy 


 

Hi All, 

We are facing also following issue on openshift origin while we are 
creating pvc for pods.  (atlease provide workaround to move further) 

Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume Server busy. Retry 
operation later.. 

Please find heketidb dump and log 

[negroni] Completed 429 Too Many Requests in 250.763µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 169.08µs 
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c 
[negroni] Completed 404 Not Found in 148.125µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 496.624µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 101.673µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 209.681µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 103.595µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 297.594µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 96.75µs 
[negroni] Started POST /volumes 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 477.007µs 
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 165.38µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 488.253µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 171.836µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 208.59µs 
[negroni] Started POST /volumes 
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds 
limit (8) 
[negroni] Completed 429 Too Many Requests in 125.141µs 
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 
[negroni] Completed 404 Not Found in 138.687µs 
[negroni] Started POST /volumes 


BR 
Salam 
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you 


-- 
Madhu Rajanna 
Software Engineer 
Red Hat Bangalore, India 
mrajanna at redhat.com    M: +91-9741133155 



-- 
Madhu Rajanna 
Software Engineer 
Red Hat Bangalore, India 
mrajanna at redhat.com    M: +91-9741133155 



-- 
Madhu Rajanna 
Software Engineer 
Red Hat Bangalore, India 
mrajanna at redhat.com    M: +91-9741133155 



-- 
Madhu Rajanna 
Software Engineer 
Red Hat Bangalore, India 
mrajanna at redhat.com    M: +91-9741133155 
[attachment "heketi-complete.log.txt" deleted by Shaik Salam/HYD/TCS] 
[attachment "heketi-gluster.db.txt" deleted by Shaik Salam/HYD/TCS] 
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users [attachment 
"heketi-complete.log" deleted by Shaik Salam/HYD/TCS] [attachment 
"heketi-dump.db" deleted by Shaik Salam/HYD/TCS] 
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190128/ac69f675/attachment.html>


More information about the Gluster-users mailing list