[Gluster-users] gluster local vs local = gluster x4 slower

Jeremy Enos jenos at ncsa.uiuc.edu
Wed Mar 24 14:47:52 PDT 2010


Good suggestion- I hadn't tried that yet.  It brings them much closer.

##############GLUSTER SINGLE DISK##############
[root at ac33 gjenos]# time (tar xzf 
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )

real    0m32.089s
user    0m6.516s
sys     0m3.177s
##############DIRECT SINGLE DISK##############
[root at ac33 gjenos]# cd /export/jenos/
[root at ac33 jenos]# time (tar xzf 
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )

real    0m25.089s
user    0m6.850s
sys     0m2.058s
##############DIRECT SINGLE DISK CACHED##############
[root at ac33 jenos]# time (tar xzf 
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz )

real    0m8.955s
user    0m6.785s
sys     0m1.848s


Oddly, I'm seeing better performance on the gluster system than previous 
tests too (used to be ~39 s).  The direct disk time is obviously 
benefiting from cache.  There is still a difference, but it appears most 
of the difference disappears w/ the cache advantage removed.  That said- 
the relative performance issue then still exists with Gluster.  What can 
be done to make it benefit from cache the same way direct disk does?
thx-

     Jeremy

P.S.
I'll be posting results w/ performance options completely removed from 
gluster as soon as I get a chance.

     Jeremy

On 3/24/2010 4:23 PM, Bryan Whitehead wrote:
> I'd like to see results with this:
>
> time ( tar xzf /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz&&  sync )
>
> I've found local filesystems seem to use cache very heavily. The
> untarred file could mostly be sitting in ram with local fs vs going
> though fuse (which might do many more sync'ed flushes to disk?).
>
> On Wed, Mar 24, 2010 at 2:25 AM, Jeremy Enos<jenos at ncsa.uiuc.edu>  wrote:
>    
>> I also neglected to mention that the underlying filesystem is ext3.
>>
>> On 3/24/2010 3:44 AM, Jeremy Enos wrote:
>>      
>>> I haven't tried all performance options disabled yet- I can try that
>>> tomorrow when the resource frees up.  I was actually asking first before
>>> blindly trying different configuration matrices in case there's a clear
>>> direction I should take with it.  I'll let you know.
>>>
>>>     Jeremy
>>>
>>> On 3/24/2010 2:54 AM, Stephan von Krawczynski wrote:
>>>        
>>>> Hi Jeremy,
>>>>
>>>> have you tried to reproduce with all performance options disabled? They
>>>> are
>>>> possibly no good idea on a local system.
>>>> What local fs do you use?
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Stephan
>>>>
>>>>
>>>> On Tue, 23 Mar 2010 19:11:28 -0500
>>>> Jeremy Enos<jenos at ncsa.uiuc.edu>    wrote:
>>>>
>>>>          
>>>>> Stephan is correct- I primarily did this test to show a demonstrable
>>>>> overhead example that I'm trying to eliminate.  It's pronounced enough
>>>>> that it can be seen on a single disk / single node configuration, which
>>>>> is good in a way (so anyone can easily repro).
>>>>>
>>>>> My distributed/clustered solution would be ideal if it were fast enough
>>>>> for small block i/o as well as large block- I was hoping that single
>>>>> node systems would achieve that, hence the single node test.  Because
>>>>> the single node test performed poorly, I eventually reduced down to
>>>>> single disk to see if it could still be seen, and it clearly can be.
>>>>> Perhaps it's something in my configuration?  I've pasted my config files
>>>>> below.
>>>>> thx-
>>>>>
>>>>>       Jeremy
>>>>>
>>>>> ######################glusterfsd.vol######################
>>>>> volume posix
>>>>>     type storage/posix
>>>>>     option directory /export
>>>>> end-volume
>>>>>
>>>>> volume locks
>>>>>     type features/locks
>>>>>     subvolumes posix
>>>>> end-volume
>>>>>
>>>>> volume disk
>>>>>     type performance/io-threads
>>>>>     option thread-count 4
>>>>>     subvolumes locks
>>>>> end-volume
>>>>>
>>>>> volume server-ib
>>>>>     type protocol/server
>>>>>     option transport-type ib-verbs/server
>>>>>     option auth.addr.disk.allow *
>>>>>     subvolumes disk
>>>>> end-volume
>>>>>
>>>>> volume server-tcp
>>>>>     type protocol/server
>>>>>     option transport-type tcp/server
>>>>>     option auth.addr.disk.allow *
>>>>>     subvolumes disk
>>>>> end-volume
>>>>>
>>>>> ######################ghome.vol######################
>>>>>
>>>>> #-----------IB remotes------------------
>>>>> volume ghome
>>>>>     type protocol/client
>>>>>     option transport-type ib-verbs/client
>>>>> #  option transport-type tcp/client
>>>>>     option remote-host acfs
>>>>>     option remote-subvolume raid
>>>>> end-volume
>>>>>
>>>>> #------------Performance Options-------------------
>>>>>
>>>>> volume readahead
>>>>>     type performance/read-ahead
>>>>>     option page-count 4           # 2 is default option
>>>>>     option force-atime-update off # default is off
>>>>>     subvolumes ghome
>>>>> end-volume
>>>>>
>>>>> volume writebehind
>>>>>     type performance/write-behind
>>>>>     option cache-size 1MB
>>>>>     subvolumes readahead
>>>>> end-volume
>>>>>
>>>>> volume cache
>>>>>     type performance/io-cache
>>>>>     option cache-size 1GB
>>>>>     subvolumes writebehind
>>>>> end-volume
>>>>>
>>>>> ######################END######################
>>>>>
>>>>>
>>>>>
>>>>> On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote:
>>>>>            
>>>>>> On Tue, 23 Mar 2010 02:59:35 -0600 (CST)
>>>>>> "Tejas N. Bhise"<tejas at gluster.com>     wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Out of curiosity, if you want to do stuff only on one machine,
>>>>>>> why do you want to use a distributed, multi node, clustered,
>>>>>>> file system ?
>>>>>>>
>>>>>>>                
>>>>>> Because what he does is a very good way to show the overhead produced
>>>>>> only by
>>>>>> glusterfs and nothing else (i.e. no network involved).
>>>>>> A pretty relevant test scenario I would say.
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Am I missing something here ?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Tejas.
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: "Jeremy Enos"<jenos at ncsa.uiuc.edu>
>>>>>>> To: gluster-users at gluster.org
>>>>>>> Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai, Kolkata,
>>>>>>> Mumbai, New Delhi
>>>>>>> Subject: [Gluster-users] gluster local vs local = gluster x4 slower
>>>>>>>
>>>>>>> This test is pretty easy to replicate anywhere- only takes 1 disk, one
>>>>>>> machine, one tarball.  Untarring to local disk directly vs thru
>>>>>>> gluster
>>>>>>> is about 4.5x faster.  At first I thought this may be due to a slow
>>>>>>> host
>>>>>>> (Opteron 2.4ghz).  But it's not- same configuration, on a much faster
>>>>>>> machine (dual 3.33ghz Xeon) yields the performance below.
>>>>>>>
>>>>>>> ####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
>>>>>>> [root at ac33 jenos]# time tar xzf
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>>
>>>>>>> real    0m41.290s
>>>>>>> user    0m14.246s
>>>>>>> sys     0m2.957s
>>>>>>>
>>>>>>> ####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
>>>>>>> [root at ac33 jenos]# cd /export/jenos/
>>>>>>> [root at ac33 jenos]# time tar xzf
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>>
>>>>>>> real    0m8.983s
>>>>>>> user    0m6.857s
>>>>>>> sys     0m1.844s
>>>>>>>
>>>>>>> ####THESE ARE TEST FILE DETAILS####
>>>>>>> [root at ac33 jenos]# tar tzvf
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz  |wc -l
>>>>>>> 109
>>>>>>> [root at ac33 jenos]# ls -l
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>> -rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>> [root at ac33 jenos]#
>>>>>>>
>>>>>>> These are the relevant performance options I'm using in my .vol file:
>>>>>>>
>>>>>>> #------------Performance Options-------------------
>>>>>>>
>>>>>>> volume readahead
>>>>>>>      type performance/read-ahead
>>>>>>>      option page-count 4           # 2 is default option
>>>>>>>      option force-atime-update off # default is off
>>>>>>>      subvolumes ghome
>>>>>>> end-volume
>>>>>>>
>>>>>>> volume writebehind
>>>>>>>      type performance/write-behind
>>>>>>>      option cache-size 1MB
>>>>>>>      subvolumes readahead
>>>>>>> end-volume
>>>>>>>
>>>>>>> volume cache
>>>>>>>      type performance/io-cache
>>>>>>>      option cache-size 1GB
>>>>>>>      subvolumes writebehind
>>>>>>> end-volume
>>>>>>>
>>>>>>> What can I do to improve gluster's performance?
>>>>>>>
>>>>>>>        Jeremy
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>              
>>>>          
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>      
>    


More information about the Gluster-users mailing list