[luau] Linux Fibre Channel Server?

Fri Feb 27 11:16:01 PST 2004

Brian,

Realistically I am six months away from doing anything.  I am just
starting research right now.  Thanks for the offer.  Once I get something
going you can deffinately come check it out.

Can I direct connect two FC HBAs or do I need a FC switch/hub?

Mahalo,
Dusty

> Well how about mulitple gbe adapters and use load balancing....that should
> work....linux supports it as well as windoze...not sure about OSx
> though....got a book on it for linux somewhere...I'll try to find it if
> you
> decide to think in this direction....not to mention that if you're ONLY
> running load balanced GBE on this switched network....you can run jumbo
> frames. That will help if it's large blocks...if LOTS of small ones, then
> FC
> is the answer.
>
> Otherwise....I THINK??? I got a couple 2gb FC adapters and a 2gb FC hub
> laying around that I would be willing to trade....how many ports do you
> need?
>
> You WILL have to get a couple of 2gb FC gbics...and they ain't cheap....I
> also have editorial discounts for some gear....if you let me play too,
> I'll
> be happy to ask around.
>
> What time frame are you looking at? I'm on the N+I NOC team so I can sniff
> around the show for you if that helps????
>
> /brian chee
>
> University of Hawaii ICS Dept
> Advanced Network Computing Lab
> 1680 East West Road, POST rm 311
> Honolulu, HI  96822
> 808-956-5797 voice, 877-284-1934 fax
>
> ----- Original Message -----
> From: "Dustin Cross" <dusty at sandust.com>
> To: <luau at videl.ics.hawaii.edu>
> Sent: Thursday, February 26, 2004 4:39 PM
> Subject: Re: [luau] Linux Fibre Channel Server?
>
>
>> Brian,
>>
>> Thanks, but Gbe is TOO SLOW for Hi Def.  2Gbit FC is the minimum.  I
>> have
>> actually been talking to 10Gbit Infiniband vendors about using nfs or
>> smb
>> over "IP over Infiniband", but Infinibad isn't really designed to work
>> like that.  To do Hi Def we need a minimum of 150MBytes (~1500Mbits) for
>> each system.  I am looking for anything more flexible than direct
>> attached
>> raid for each system, but I think that is what is going to be the most
>> cost effective.
>>
>> When I talk to the big shops on the mainland they all seem to use half
>> million dollar SGI FC systems with CXFS for 6Tbytes of storage.  I can
>> build a 6TB SATA RAID Linux system for less than $15K.   Now I just have
>> to figure out how to get the data to the hosts (Windows & OSX)?
>>
>> What I really need is 10Gbit Ethernet, but that is expensive!!!!
>>
>>
>> Mahalo,
>> Dusty
>>
>>
>>
>> > oooohhhhh.....don't do it!!!!! FC is the most expensive way to doing
> this
>> > possible short of a proprietary solution.
>> >
>> > You REALLY want iscsi and that runs over el'cheapo gbe cards.....just
> get
>> > an
>> > el'cheapo copper gbe switch and use that ONLY for iscsi and you will
>> get
>> > performance approaching FC for a very small fraction of the cost.
>> It's
>> > also
>> > easier to setup and doesn't require proprietary solutions like FC.
>> >
>> > /brian chee
>> >
>> > University of Hawaii ICS Dept
>> > Advanced Network Computing Lab
>> > 1680 East West Road, POST rm 311
>> > Honolulu, HI  96822
>> > 808-956-5797 voice, 877-284-1934 fax
>> >
>> > ----- Original Message -----
>> > From: "Dustin Cross" <dusty at sandust.com>
>> > To: <luau at videl.ics.hawaii.edu>
>> > Sent: Thursday, February 26, 2004 3:46 PM
>> > Subject: Re: [luau] Linux Fibre Channel Server?
>> >
>> >
>> >> Brian,
>> >>
>> >> Thanks for the info.
>> >>
>> >> I guess my question is poorly formed, because everyone keeps telling
>> me
>> >> what FC is.  I know that part.
>> >>
>> >> I think I have answered my question with this Mayastor software.
>> >>
>> >> Right now I don't have any FC hardware and that is why I didn't
>> mention
>> >> any brands.
>> >>
>> >> Basically I want to take a Linux system with a SATA RAID array, put a
> FC
>> >> HBA  card in it and then share the SATA RAID array across FC to
>> another
>> >> system.  The only FC part in the Linux system will be the FC HBA.  It
>> >> sounds like Mayastor does exactly this.  Now I just have to come up
> with
>> >> a
>> >> couple cheap HBAs to test it.
>> >>
>> >> My end goal is to build a multi TB SATA RAID Linux file server
>> sharing
>> >> volumes across 2Gb Fibre Channel.
>> >>
>> >> We do video editing and are looking to get into Hi Def.  That means
>> we
>> >> need tons of disk space and a minimum of 150MBytes per sec of disk
>> I/O.
>> >>
>> >>
>> >> Thanks,
>> >> Dusty
>> >>
>> >>
>> >>
>> >> > ok....basically the gist as I understand is that an FC array once
>> >> > configured
>> >> > with the HBA utilities looks like a SCSI drive....VERY similar to a
>> >> SCSI
>> >> > RAID array. The key is the HBA utilities that it lets you carve up
> the
>> >> > array
>> >> > into LUNs (logical units) that then can be assigned to drive
>> pointers
>> >> or
>> >> > multiple hosts over an FC network...but FC switches cost a kings
>> >> ransom
>> >> > and
>> >> > you really don't want to use a hub for anything other than a single
>> >> > system.
>> >> > FC comes both copper and fiber with the copper using a DB-9 varient
>> >> that
>> >> > is
>> >> > WELL shielded.
>> >> >
>> >> > Now if you want to export arrays, take a look at the iscsi
>> >> stuff....very
>> >> > cool and it doesn't matter what kind of drives you use....the
>> >> > Networld+Interop iLabs guys did a raid5 array of thumb drives
> exported
>> > via
>> >> > iscsi over 802.11a to a win2k laptop that mounted them as dynamic
>> >> > drives....
>> >> >
>> >> > Since you didn't include which HBA you're trying to use, nor the
> drive
>> >> > array
>> >> > type, nor the type of FC hub/switch...I just picked an HBA at
>> random
>> >> and
>> >> > assume you're using a reasonable array. I attached the readme file
>> >> from
>> > a
>> >> > qlogic (my only experience driving an install alone) HBA. Such
>> >> readme's
>> >> > are
>> >> > typically found with the card and are VERY specific to the HBA,
>> since
>> >> > there
>> >> > is quite a bit more to an FC array than just the driver. You MUST
> have
>> >> > access to some sort of utility that can setup the FC array, or
>> you're
>> >> > going
>> >> > nowhere. Opensource hasn't quite caught up on FC yet, and this is
>> why
>> > I've
>> >> > stuck to qlogic so far...it's also why I'm leaning away from FC and
>> >> more
>> >> > towards SATA.
>> >> >
>> >> > /brian chee
>> >> >
>> >> >
>> >> > University of Hawaii ICS Dept
>> >> > Advanced Network Computing Lab
>> >> > 1680 East West Road, POST rm 311
>> >> > Honolulu, HI  96822
>> >> > 808-956-5797 voice, 877-284-1934 fax
>> >> >
>> >> > ----- Original Message -----
>> >> > From: "Dustin Cross" <dusty at sandust.com>
>> >> > To: <luau at videl.ics.hawaii.edu>
>> >> > Sent: Thursday, February 26, 2004 2:23 PM
>> >> > Subject: Re: [luau] Linux Fibre Channel Server?
>> >> >
>> >> >
>> >> >> Brian,
>> >> >>
>> >> >> I understand how Fibre Channel works and the cost involved with
>> >> buying
>> >> >> the
>> >> >> HBAs and switch.  What I am trying to get information on is can I
> use
>> > my
>> >> >> existing Linux file server, put a Fibre Channel HBA in it and
>> share
>> >> >> disks
>> >> >> out to other systems on the Fibre Channel SAN?
>> >> >>
>> >> >> It sounds like this Mayastor software
>> >> >> (http://www.pavitrasoft.com/mayastor/) does exactly what I am
> looking
>> >> >> for.
>> >> >>
>> >> >> I don't want to invest thousands of dollars in something like an
>> >> Apple
>> >> >> Xraid if I can accomplish the same thing with my existing Linux
>> File
>> >> >> Server.
>> >> >>
>> >> >>
>> >> >> Mahalo,
>> >> >> Dusty
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Hi Dusty,
>> >> >> >
>> >> >> > I have setup many systems with FiberChannel and GigE networks
>> >> (mainly
>> >> >> > FreeBSD).
>> >> >> > There is really not much to it other than installing the
> Fiber/GigE
>> >> >> NIC
>> >> > in
>> >> >> > your server. Before you buy
>> >> >> > a NIC it would be a good idea to make sure the drivers for the
>> NIC
>> >> >> exist
>> >> >> > for
>> >> >> > Linux. I don't know what type of file server software you are
> using
>> >> >> but
>> >> >> > installing FiberChannel NIC may or may not help you. If all the
>> >> hosts
>> >> >> on
>> >> >> > your network are connected via 100Mbps connections to the file
>> >> server
>> >> > and
>> >> >> > the current NIC on your file server isn't maxing out on
>> throughput
>> >> at
>> >> > any
>> >> >> > given time, then you will not notice much of a performance
>> > difference.
>> >> >> > Fiber
>> >> >> > Channel can also be expensive, remember you will need a
>> >> FiberChannel
>> >> >> > switch
>> >> >> > too. I have no idea what type of data you share over your
>> network,
>> > but
>> >> > if
>> >> >> > it
>> >> >> > isn't extremely brandwidth intensive you may want to look at a
> GigE
>> >> >> > (10/100/1000 Mbps) solution instead. This route may save you
>> quite
>> >> a
>> >> >> bit
>> >> >> > of
>> >> >> > cash.
>> >> >> >
>> >> >> > --Brian
>> >> >> >
>> >> >> > ----- Original Message -----
>> >> >> > From: "Dustin Cross" <dusty at sandust.com>
>> >> >> > To: <luau at videl.ics.hawaii.edu>
>> >> >> > Sent: Thursday, February 26, 2004 12:55 PM
>> >> >> > Subject: [luau] Linux Fibre Channel Server?
>> >> >> >
>> >> >> >
>> >> >> >> Aloha,
>> >> >> >>
>> >> >> >> I am trying to find out if it is possible to put a Fibre
>> Channel
>> > card
>> >> > in
>> >> >> >> my Linux File Server and start sharing disk space over 2Gb
>> fibre
>> >> >> >> Channel.
>> >> >> >> Does anyone have any knowledge of doing this?
>> >> >> >>
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Dusty
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> LUAU mailing list
>> >> >> >> LUAU at videl.ics.hawaii.edu
>> >> >> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
>> >> >> >>
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > LUAU mailing list
>> >> >> > LUAU at videl.ics.hawaii.edu
>> >> >> > http://videl.ics.hawaii.edu/mailman/listinfo/luau
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> _______________________________________________
>> >> >> LUAU mailing list
>> >> >> LUAU at videl.ics.hawaii.edu
>> >> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
>> >> >
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> LUAU mailing list
>> >> LUAU at videl.ics.hawaii.edu
>> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
>> >
>> > _______________________________________________
>> > LUAU mailing list
>> > LUAU at videl.ics.hawaii.edu
>> > http://videl.ics.hawaii.edu/mailman/listinfo/luau
>> >
>>
>>
>>
>> _______________________________________________
>> LUAU mailing list
>> LUAU at videl.ics.hawaii.edu
>> http://videl.ics.hawaii.edu/mailman/listinfo/luau
>>
>
> _______________________________________________
> LUAU mailing list
> LUAU at videl.ics.hawaii.edu
> http://videl.ics.hawaii.edu/mailman/listinfo/luau
>