[Gluster-users] Gluster-users Digest, Vol 20, Issue 22
harsha at gluster.com
Wed Jan 6 03:48:48 UTC 2010
* replies inline *
On Wed, Jan 6, 2010 at 7:07 AM, Liam Slusser <lslusser at gmail.com> wrote:
> Arvids & Larry,
> Interesting read, Arvids. And i completely agree. On our large raid6
> array it takes 1 week to rebuild the array from any ONE drive failure.
> Its a scary time when doing a rebuild because of the decreased
> performance from the array and the increased chance of a full raid
> failure if we lost another two drives. Makes for a very long week of
> nail biting.
> Larry brought up some great points. I, also, have been burned way to
> many times by raid5 and only use it if i absolutely have to. I
> normally stick to raid1/10 or raid6/60. Even with my huge raid6
> rebuild time of a week, its still faster to do that then have gluster
> resync everything. The raid rebuild does affect the performance of
> the box, however, so would a gluster rebuild.
> As for Larry's point #4 i duplicate the data across two boxes using
> cluster/replication on top of raid6. So each box has a large raid6
> set and then dup the data between the two. So, for whatever reason,
> if i did loose a whole raid array i can still recover with Gluster.
> I've also been frowned on for using desktop drives in our servers -
> but on the bright side i've had very little problems with them. Of
> course it did take buying a bunch of different raid cards and drives
> before finding a combination that played well together. We currently
> have 240 Seagate 1.5tb desktop drives in our two gluster clusters and
> have only had to replaced three in the last year - two that just died
> and one started to get smart errors so it was replaced. I haven't had
> a problem getting Seagate to replace the drives - as they fail i ship
> them off to Seagate and they send me a new one. I did figure we would
> have to do support in house so we bought lots of spare parts when we
> ordered everything. It was still way cheaper to buy desktop drives
> and Supermicro servers with lots of spare parts than shopping at Dell,
> HP or Sun - by more than half.
> Honestly my biggest peeve of Gluster is the rebuild process. Take the
> OneFS file system in Isilon clusters - they are able to rebuild at a
> block level - only replicating information which has changed. So even
> with one node being offline all day - a rebuild/resync operation is
> very quick. And have 30 billion files or 10 huge ones makes no
> difference on resync speed. While with Gluster a huge directory
> tree/number of files can take days if not weeks to finish. Of course
> being that Gluster runs on top of a normal filesystem such as
> xfs/ext3/zfs having access to block level replication may be tricky.
> I honestly would not be against having the Gluster team modifying the
> xfs/ext3/whatever filesystem so they could tailer it more for their
> own needs - which of course would make it far less portable and much
> more difficult to install and configure...
> GlusterFS does checksum based self-heal since the 3.0 release, i would
believe your experiences are from 2.0? which has issues of doing a full file
self-heal which will a lot of time. But i would suggest an upgrade with
release which is due Feb 1st week for your cluster. 3.x releases with new
self-heal you should get very less rebuild times. If its possible to compare
3.0.1 rebuild times with the One-FS from Isilon should help us improve it
> Whatever the solution is i can tell you that the rebuild issues will
> only get worse as drives continue to get larger and the number of
> files/directories continue to grow. Sun's ZFS filesystem goes along
> way to fix some of these problems, i just wish they would port it over
> to Linux.
I would suggest wait for "brtfs".
> On Tue, Jan 5, 2010 at 2:17 PM, Arvids Godjuks <arvids.godjuks at gmail.com>
> > Consider this - a rebuild of 1.5-2 TB HDD in raid5/6 array can easily
> > take up to few days to complete. At that moment your storage at that
> > node will not perform well. I read a week ago very good article with
> > research of this area, only thing it's in russian, but it mentions a
> > few english sources too. Maybe google translate will help.
> > Here's the original link: http://habrahabr.ru/blogs/hardware/78311/
> > Here's the google translate version:
> > (looks quite neet by the way)
> > 2010/1/5 Liam Slusser <lslusser at gmail.com>:
> >> Larry & All,
> >> I would much rather rebuild a bad drive with a raid controller then
> >> have to wait for Gluster to do it. With a large number of files doing
> >> a ls -aglR can take weeks. Also you don't NEED enterprise drives with
> >> a raid controller, i use desktop 1.5tb Seagate drives which happy as a
> >> clam on a 3ware SAS card under a SAS expander.
> >> liam
> Gluster-users mailing list
> Gluster-users at gluster.org
More information about the Gluster-users