[luau] Linux Fibre Channel Server?

[Brian Chee]

Supposedly you can just flip the tx/rx fibers around and go from there. The
good news is that 2gb FC is mini-gbic based and they're less expensive than
traditional gbics. The other hassle is that you can't play the generic gbic
game anymore. Seems most vendors are now checking serial numbers for
"certified" gbics...thusly before we were able to use the el'cheapo gbics
(1gbe) in either an FC switch/hub or a gig ethernet switch....but alas this
has changed. I don't know for sure, but if I got time laters, I'll bring up
a couple and see if it's possible...I used to be able to get HP 1gbe gbics
at dot com depot for $75/each.

If you can get your hands on a copper FC card....then you gotta play some
wierd games with modifying the cable....fiber FC you can just swap the tx/rx
fibers around....if you need help, we build our own fiber here...most times
better than the factory.

Anyways....like I said...I'm confirming my airfare to Lost Wages now for
N+I. There have been proprietary NICs in the past that did some VERY
interesting load balancing....so I'll go look for that...in the past I've
seen a dual gbe that supported load balanced/channel bonding to certain GBE
switches and Windows 2k/AS would allow us to do four of those....you in
theory (I've never tried it) you may be able to do 8gbe to the switch and
just eat up gbe ports.

Rumor has it that 10gbe nics are getting close, but you're going to need one
heck of a frontside bus to handle this and at least a 64bit PCI slot for the
I/O....remember there is an uppper limit on Bus I/O and you really ought to
be doing this on a machine that has as many separate PCI bus structures as
possible. I'd go with something that has at least three PCI bridge chips or
the floppy/keyboard/mouse is going to drag at least one PCI bus down to ISA
speeds part of the time....this is why my speed freak servers (too bad I
don't get to build them too often) all have USB keyboard/mouse and no
floppy. This is how the old NetFrames were constructed and they were the
only machines certified to run 1000 user netware in their days...

/brian chee


/brian chee

University of Hawaii ICS Dept
Advanced Network Computing Lab
1680 East West Road, POST rm 311
Honolulu, HI  96822
808-956-5797 voice, 877-284-1934 fax

----- Original Message ----- 
From: "Dustin Cross" <dusty at sandust.com>
To: <luau at videl.ics.hawaii.edu>
Sent: Friday, February 27, 2004 11:23 AM
Subject: Re: [luau] Linux Fibre Channel Server?


> Brian,
>
> Realistically I am six months away from doing anything.  I am just
> starting research right now.  Thanks for the offer.  Once I get something
> going you can deffinately come check it out.
>
> Can I direct connect two FC HBAs or do I need a FC switch/hub?
>
>
> Mahalo,
> Dusty
>
>
> > Well how about mulitple gbe adapters and use load balancing....that
should
> > work....linux supports it as well as windoze...not sure about OSx
> > though....got a book on it for linux somewhere...I'll try to find it if
> > you
> > decide to think in this direction....not to mention that if you're ONLY
> > running load balanced GBE on this switched network....you can run jumbo
> > frames. That will help if it's large blocks...if LOTS of small ones,
then
> > FC
> > is the answer.
> >
> > Otherwise....I THINK??? I got a couple 2gb FC adapters and a 2gb FC hub
> > laying around that I would be willing to trade....how many ports do you
> > need?
> >
> > You WILL have to get a couple of 2gb FC gbics...and they ain't
cheap....I
> > also have editorial discounts for some gear....if you let me play too,
> > I'll
> > be happy to ask around.
> >
> > What time frame are you looking at? I'm on the N+I NOC team so I can
sniff
> > around the show for you if that helps????
> >
> > /brian chee
> >
> > University of Hawaii ICS Dept
> > Advanced Network Computing Lab
> > 1680 East West Road, POST rm 311
> > Honolulu, HI  96822
> > 808-956-5797 voice, 877-284-1934 fax
> >
> > ----- Original Message -----
> > From: "Dustin Cross" <dusty at sandust.com>
> > To: <luau at videl.ics.hawaii.edu>
> > Sent: Thursday, February 26, 2004 4:39 PM
> > Subject: Re: [luau] Linux Fibre Channel Server?
> >
> >
> >> Brian,
> >>
> >> Thanks, but Gbe is TOO SLOW for Hi Def.  2Gbit FC is the minimum.  I
> >> have
> >> actually been talking to 10Gbit Infiniband vendors about using nfs or
> >> smb
> >> over "IP over Infiniband", but Infinibad isn't really designed to work
> >> like that.  To do Hi Def we need a minimum of 150MBytes (~1500Mbits)
for
> >> each system.  I am looking for anything more flexible than direct
> >> attached
> >> raid for each system, but I think that is what is going to be the most
> >> cost effective.
> >>
> >> When I talk to the big shops on the mainland they all seem to use half
> >> million dollar SGI FC systems with CXFS for 6Tbytes of storage.  I can
> >> build a 6TB SATA RAID Linux system for less than $15K.   Now I just
have
> >> to figure out how to get the data to the hosts (Windows & OSX)?
> >>
> >> What I really need is 10Gbit Ethernet, but that is expensive!!!!
> >>
> >>
> >> Mahalo,
> >> Dusty
> >>
> >>
> >>
> >> > oooohhhhh.....don't do it!!!!! FC is the most expensive way to doing
> > this
> >> > possible short of a proprietary solution.
> >> >
> >> > You REALLY want iscsi and that runs over el'cheapo gbe cards.....just
> > get
> >> > an
> >> > el'cheapo copper gbe switch and use that ONLY for iscsi and you will
> >> get
> >> > performance approaching FC for a very small fraction of the cost.
> >> It's
> >> > also
> >> > easier to setup and doesn't require proprietary solutions like FC.
> >> >
> >> > /brian chee
> >> >
> >> > University of Hawaii ICS Dept
> >> > Advanced Network Computing Lab
> >> > 1680 East West Road, POST rm 311
> >> > Honolulu, HI  96822
> >> > 808-956-5797 voice, 877-284-1934 fax
> >> >
> >> > ----- Original Message -----
> >> > From: "Dustin Cross" <dusty at sandust.com>
> >> > To: <luau at videl.ics.hawaii.edu>
> >> > Sent: Thursday, February 26, 2004 3:46 PM
> >> > Subject: Re: [luau] Linux Fibre Channel Server?
> >> >
> >> >
> >> >> Brian,
> >> >>
> >> >> Thanks for the info.
> >> >>
> >> >> I guess my question is poorly formed, because everyone keeps telling
> >> me
> >> >> what FC is.  I know that part.
> >> >>
> >> >> I think I have answered my question with this Mayastor software.
> >> >>
> >> >> Right now I don't have any FC hardware and that is why I didn't
> >> mention
> >> >> any brands.
> >> >>
> >> >> Basically I want to take a Linux system with a SATA RAID array, put
a
> > FC
> >> >> HBA  card in it and then share the SATA RAID array across FC to
> >> another
> >> >> system.  The only FC part in the Linux system will be the FC HBA.
It
> >> >> sounds like Mayastor does exactly this.  Now I just have to come up
> > with
> >> >> a
> >> >> couple cheap HBAs to test it.
> >> >>
> >> >> My end goal is to build a multi TB SATA RAID Linux file server
> >> sharing
> >> >> volumes across 2Gb Fibre Channel.
> >> >>
> >> >> We do video editing and are looking to get into Hi Def.  That means
> >> we
> >> >> need tons of disk space and a minimum of 150MBytes per sec of disk
> >> I/O.
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Dusty
> >> >>
> >> >>
> >> >>
> >> >> > ok....basically the gist as I understand is that an FC array once
> >> >> > configured
> >> >> > with the HBA utilities looks like a SCSI drive....VERY similar to
a
> >> >> SCSI
> >> >> > RAID array. The key is the HBA utilities that it lets you carve up
> > the
> >> >> > array
> >> >> > into LUNs (logical units) that then can be assigned to drive
> >> pointers
> >> >> or
> >> >> > multiple hosts over an FC network...but FC switches cost a kings
> >> >> ransom
> >> >> > and
> >> >> > you really don't want to use a hub for anything other than a
single
> >> >> > system.
> >> >> > FC comes both copper and fiber with the copper using a DB-9
varient
> >> >> that
> >> >> > is
> >> >> > WELL shielded.
> >> >> >
> >> >> > Now if you want to export arrays, take a look at the iscsi
> >> >> stuff....very
> >> >> > cool and it doesn't matter what kind of drives you use....the
> >> >> > Networld+Interop iLabs guys did a raid5 array of thumb drives
> > exported
> >> > via
> >> >> > iscsi over 802.11a to a win2k laptop that mounted them as dynamic
> >> >> > drives....
> >> >> >
> >> >> > Since you didn't include which HBA you're trying to use, nor the
> > drive
> >> >> > array
> >> >> > type, nor the type of FC hub/switch...I just picked an HBA at
> >> random
> >> >> and
> >> >> > assume you're using a reasonable array. I attached the readme file
> >> >> from
> >> > a
> >> >> > qlogic (my only experience driving an install alone) HBA. Such
> >> >> readme's
> >> >> > are
> >> >> > typically found with the card and are VERY specific to the HBA,
> >> since
> >> >> > there
> >> >> > is quite a bit more to an FC array than just the driver. You MUST
> > have
> >> >> > access to some sort of utility that can setup the FC array, or
> >> you're
> >> >> > going
> >> >> > nowhere. Opensource hasn't quite caught up on FC yet, and this is
> >> why
> >> > I've
> >> >> > stuck to qlogic so far...it's also why I'm leaning away from FC
and
> >> >> more
> >> >> > towards SATA.
> >> >> >
> >> >> > /brian chee
> >> >> >
> >> >> >
> >> >> > University of Hawaii ICS Dept
> >> >> > Advanced Network Computing Lab
> >> >> > 1680 East West Road, POST rm 311
> >> >> > Honolulu, HI  96822
> >> >> > 808-956-5797 voice, 877-284-1934 fax
> >> >> >
> >> >> > ----- Original Message -----
> >> >> > From: "Dustin Cross" <dusty at sandust.com>
> >> >> > To: <luau at videl.ics.hawaii.edu>
> >> >> > Sent: Thursday, February 26, 2004 2:23 PM
> >> >> > Subject: Re: [luau] Linux Fibre Channel Server?
> >> >> >
> >> >> >
> >> >> >> Brian,
> >> >> >>
> >> >> >> I understand how Fibre Channel works and the cost involved with
> >> >> buying
> >> >> >> the
> >> >> >> HBAs and switch.  What I am trying to get information on is can I
> > use
> >> > my
> >> >> >> existing Linux file server, put a Fibre Channel HBA in it and
> >> share
> >> >> >> disks
> >> >> >> out to other systems on the Fibre Channel SAN?
> >> >> >>
> >> >> >> It sounds like this Mayastor software
> >> >> >> (http://www.pavitrasoft.com/mayastor/) does exactly what I am
> > looking
> >> >> >> for.
> >> >> >>
> >> >> >> I don't want to invest thousands of dollars in something like an
> >> >> Apple
> >> >> >> Xraid if I can accomplish the same thing with my existing Linux
> >> File
> >> >> >> Server.
> >> >> >>
> >> >> >>
> >> >> >> Mahalo,
> >> >> >> Dusty
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> > Hi Dusty,
> >> >> >> >
> >> >> >> > I have setup many systems with FiberChannel and GigE networks
> >> >> (mainly
> >> >> >> > FreeBSD).
> >> >> >> > There is really not much to it other than installing the
> > Fiber/GigE
> >> >> >> NIC
> >> >> > in
> >> >> >> > your server. Before you buy
> >> >> >> > a NIC it would be a good idea to make sure the drivers for the
> >> NIC
> >> >> >> exist
> >> >> >> > for
> >> >> >> > Linux. I don't know what type of file server software you are
> > using
> >> >> >> but
> >> >> >> > installing FiberChannel NIC may or may not help you. If all the
> >> >> hosts
> >> >> >> on
> >> >> >> > your network are connected via 100Mbps connections to the file
> >> >> server
> >> >> > and
> >> >> >> > the current NIC on your file server isn't maxing out on
> >> throughput
> >> >> at
> >> >> > any
> >> >> >> > given time, then you will not notice much of a performance
> >> > difference.
> >> >> >> > Fiber
> >> >> >> > Channel can also be expensive, remember you will need a
> >> >> FiberChannel
> >> >> >> > switch
> >> >> >> > too. I have no idea what type of data you share over your
> >> network,
> >> > but
> >> >> > if
> >> >> >> > it
> >> >> >> > isn't extremely brandwidth intensive you may want to look at a
> > GigE
> >> >> >> > (10/100/1000 Mbps) solution instead. This route may save you
> >> quite
> >> >> a
> >> >> >> bit
> >> >> >> > of
> >> >> >> > cash.
> >> >> >> >
> >> >> >> > --Brian
> >> >> >> >
> >> >> >> > ----- Original Message -----
> >> >> >> > From: "Dustin Cross" <dusty at sandust.com>
> >> >> >> > To: <luau at videl.ics.hawaii.edu>
> >> >> >> > Sent: Thursday, February 26, 2004 12:55 PM
> >> >> >> > Subject: [luau] Linux Fibre Channel Server?
> >> >> >> >
> >> >> >> >
> >> >> >> >> Aloha,
> >> >> >> >>
> >> >> >> >> I am trying to find out if it is possible to put a Fibre
> >> Channel
> >> > card
> >> >> > in
> >> >> >> >> my Linux File Server and start sharing disk space over 2Gb
> >> fibre
> >> >> >> >> Channel.
> >> >> >> >> Does anyone have any knowledge of doing this?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> Dusty
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> _______________________________________________
> >> >> >> >> LUAU mailing list
> >> >> >> >> LUAU at videl.ics.hawaii.edu
> >> >> >> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >> >> >> >>
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > LUAU mailing list
> >> >> >> > LUAU at videl.ics.hawaii.edu
> >> >> >> > http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> LUAU mailing list
> >> >> >> LUAU at videl.ics.hawaii.edu
> >> >> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> LUAU mailing list
> >> >> LUAU at videl.ics.hawaii.edu
> >> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >> >
> >> > _______________________________________________
> >> > LUAU mailing list
> >> > LUAU at videl.ics.hawaii.edu
> >> > http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >> >
> >>
> >>
> >>
> >> _______________________________________________
> >> LUAU mailing list
> >> LUAU at videl.ics.hawaii.edu
> >> http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >>
> >
> > _______________________________________________
> > LUAU mailing list
> > LUAU at videl.ics.hawaii.edu
> > http://videl.ics.hawaii.edu/mailman/listinfo/luau
> >
>
>
>
> _______________________________________________
> LUAU mailing list
> LUAU at videl.ics.hawaii.edu
> http://videl.ics.hawaii.edu/mailman/listinfo/luau