[luau] Proposal for HOSEF Servers

[Warren Togami]

On Fri, 2003-12-05 at 11:52, R.Scott Belford wrote:
> This was not how this thread was supposed to evolve.  I am, by all 
> definitions, only intermediate in my evolving knowledge of Linux, and I 
> have never claimed differently.  I am an impassioned student, though, 
> and I am devoted to making my "knowledge" as knowable as possible by as 
> many as possible.  Hence my 400 + hours devoted to HOSEF and the 
> promotion and de-mystification of Linux to those who lack "sound 
> technical knowledge."  I am average, but I am trying to build our local 
> community in my own humble way.

I now realize it was wrong of me to attack.  We are all trying our best,
and too often the stresses are really getting to me.  Keep up the good
work and I will continue to try to do the same.

> Some of you may know that we are in the process of adding a dual 
> athlon, 1gb ram, mega-box to our arsenal.  This is an older, spare 
> machine at Pricebusters that I am buying from Pricebusters and giving 
> to HOSEF.  Once in the rack, thanks to the kind efforts of Brian Chee, 
> it will be offloading a lot of Videl's burden.  At this point, we are 
> due for some server downtime while drives and mirrors are moved.  A 
> good time, it seemed, to add the yet-to-be-given matching 160gb drive.
> 

Due to the magic of vserver, there need not be much of any downtime at
all from the perspective of the Internet.  

Proposal for New Server Launch:
* Several days before the new server is installed, 128.171.104.146 is
assigned to a dedicated vserver on videl for HOSEF's httpd and mysql,
then hosef.org points to the new IP address.  mysql needs to be copied
from Videl's shared mysql.  DNS MX needs to continue pointing at
videl.ics.hawaii.edu for the domain aliases to continue to work.

(Brian, can you rename quack.ics.hawaii.edu to pan.ics.hawaii.edu?)

Several Days Later:
* The new server has 2x120GB drives, which can be setup as RAID1.
* Add vserver kernel
* shutdown the HOSEF vserver on videl and immediately tarball
* copy to new server, transfer 128.171.104.146 config
* untarball vserver and startup
* HOSEF is back online with less than 5 minutes of downtime
* Do the same thing for all other vservers on Videl.  This will
temporarily run all vservers on the new server during the reinstall.
* Shutdown videl, rip out disks, install new disks, reinstall, secure,
reconfigure vserver.

Now to discuss our disk purchase and use options...

I personally have received $241 in donations in total from four
contributors, the largest donation was $200 from a David Rees, Fedora
user in California who was also impressed by HOSEF's work of putting
Linux into Hawaii schools.  Several other list members have contacted me
directly and we have been discussing further donations that may total
another $400-500.  I *really* want the 1TB mirror to happen, so I am
considering covering the rest out of pocket.

My quibbling over one new 60GB drive (~$99) vs. 2x160GB drives (~$350?)
was mostly out of surprise that it happened with very little
discussion.  I hope that we can constructively discuss our options
here.  First my proposal for the goals:

1) videl.ics.hawaii.edu needs to be back with RAID1 mirror on the system
drives.
2) I wish us to have substantial storage capacity so we can mirror all
requests that our users use.  As the only substantial Linux/OSS mirror
in Hawaii, it is of great benefit to our community in mirroring these
things.
3) I wish to have 100GB+ storage at the Oceanic mirror in order to
mirror the most popular data.  There are many of us heavy mirror users
on Oceanic's network, and it would greatly improve efficiency to pull
from Oceanic's network rather than saturate HIX.

#2 I am willing to compromise if 1TB is too high a fund raising goal. 
We would do fine with 50-75% that capacity.

Several parts of your proposal indicated taking the 160GB and/or 120GB
disk for the new server.  This seemed to me like an emotional outcry
rather than from any technical reason, and there are much better ways in
doing this.  Reasons:

1) The 120GB drive is an older 5400rpm, thus would be a mismatch living
in a storage array with newer, faster disks.  This makes it absolutely
perfect living alone in the Oceanic mirror.
2) The 160GB disk was also indicated as serving the basis of the Linux
mirror.  There is a problem here in greater capacity/cost ratio with
larger disks, and the fact that only a limited number of disks can be
mounted within the server case.  We should today not buy relatively
small sizes like 160GB in the name of conserving this growth potential. 
For these reasons I really hope we can pool the donations and buy the
largest disks with the best possible cost/capacity/space resource ratio.

If the above is reasonable, then this leaves us with three options for
repairing Videl:
1) 60GB + 60GB (~$100)
2) 160GB + 60GB (~$170?)
Mismatch but yes, that is okay.
3) 160GB + 160GB (~$350?)

Plan #3 is currently possible with the current level of donations.  I
however personally hope we can go with plan #1 because the the extra
storage is nowhere near needed for the system drive, and the extra money
would be spent more effectively as part of the mirror array.  Plan #2
makes me twitch, but we can do it if the 160GB disk cannot be exchanged.

Now this quibbling over saving about $150 is moot if suddenly another
$1,000 of donations pour in from our community.  *HINT* *HINT*

One more thing to discuss in this proposal:
All Servers are Generic
=======================
Under my plan, all of the HOSEF servers are generic.  Each runs a
vserver kernel, and is capable of running any vserver within it.  At any
time a vserver can be shut down and moved to another physical server,
with only an extremely small downtime window.  This allows great
flexibility in administering the various services that HOSEF operates. 

In the below theoretical breakdown, any ISOLATED vserver has its own
server instance and IP address.  An isolated vserver can be easily moved
to another physical server in order to help balance CPU or memory
usage.  There are a few exceptions marked with asterisks and explained
below the breakdown.

Theoretical Breakdown:
----------------------
videl.ics.hawaii.edu	
	shared web server (ISOLATED)
		fedora.us
		uhband.hawaii.edu
		macromedia.mplug.org
	download.fedora.us (ISOLATED)
	mailman.hosef.org (ISOLATED)
		fedora.us mailing lists
		hosef.org mailing lists
		monmotha's mailing lists
		flash-plugin mailing list
		radeonigp mailing list
	wwwjdic.ics.hawaii.edu (ISOLATED)
pan.ics.hawaii.edu (New Server)
	hosef.org (ISOLATED)
	Linux Mirror, perhaps alias mirror.hawaii.edu *
fedora.ics.hawaii.edu **
	bugzilla.fedora.us (ISOLATED)
	cvs.fedora.us (ISOLATED)
	RPM Build Server

Note that the above breakdown only shows the previously proposed
services that the new server would run.  Given that the new server has
substantially more CPU and RAM than Videl, we should perhaps move more
isolated vservers to it by default.  Perhaps wwwjdic and mailman would
be ideal.  We can shuffle around the vservers easily enough so the
specifics of this plan doesn't matter at the moment.

This concept of generic servers abstracts the physical server from the
jobs that it does.  This also means that server names are meaningless,
because the actual Internet visible names are movable transparently.

* The mirror cannot physically move from this box since the storage
capacity is married to the physical server... unless somebody wants to
donate $5,000 so we can use an independent fiber channel array shared
between the servers. =) j/k
** Administrator access for fedora.ics.hawaii.edu cannot be given to
anyone outside of the Fedora leadership due to the high security risk.