- September 1, 2002.
Contact by Jerry Eichhoefer, head of CS at Greenville College (IL).
- September 4, 2002.
KDG reports that Gaussian 98 is still not running in parallel mode;
moreover, he has discovered that it will only run successfully if the
user running g98 is the owner of the gaussian distribution.
This has broken WebMO.
KDG begins drafting another letter to their tech support,
to be CC'd to their President, VP, Director, etc, etc.
- September 5, 2002.
Letter sent to Gaussian.
Replacement IBM hard drive arrives. It fails (#7) on bootup.
- September 9, 2002.
Receive quote from Joe Lippman (JL) at
Western Scientific
for their Tornado F3 and a 2-CPU
(AMD Athlon MP) head node.
- September 10, 2002.
Response received from Gaussian. Unfortunately, the response is
"stock" suggestions that we've already tried.
- September 11, 2002.
More interaction with Gaussian, and a recognition on their part that
we're having a problem that "stock" answers won't resolve.
- September 12, 2002.
KDG creates user account for Gaussian's Dr Fox, so that he can
investigate first-person (remotely) via ssh.
We determine that the Tyan motherboard on the headnode quoted from
Western Scientific is the wrong one (S2468GN) with no on-board SCSI.
It does have 2 64-bit PCI slots though, that can handle our SK NICs.
We ask them about substituting the S2468UGN instead.
- September 13, 2002.
Western Scientific provides updated quote using correct motherboard
(Tyan S2468UGN).
- September 16, 2002.
We start comparing the Western Scientific RAID (IDE-to-SCSI)
and the
Adaptec
Durastore (SCSI-to-SCSI) devices.
- September 19, 2002.
We learn that the Adaptec Durastore "solution" has
dual SCSI channels, but they are designed for fault-tolerant
fail-over behavior, not for simultaneous use by 2 masters,
so it won't solve our problem.
- September 20, 2002.
Randy Brouwer submits abstract of EE student work to
the Argonne Labs Conference.
We make arrangements to start rotating backup tapes through
CIT's firesafe.
Dr. Fox from Gaussian reports that he's still not isolated
our problem, but name resolution has been ruled out.
- September 26, 2002.
We order a Tornado F3 and head node from Western Scientific.
- September 27, 2002.
Randy Brouwer provides an abstract of EE's paper for Argonne,
describing their system which they call FireEyez.
- October 2, 2002.
Vic delivers 36 additional Syskonnect SK-9821 NICs, for
- extending our ring topology into a hypercube (34 NICs), and
- adding gigabit NICs to the RAID array's head node (2 NICs).
- October 4, 2002.
KDG reports that the 34 hypercube NICs are installed,
and that his scripts to reconfigure the (logical) topology
on the fly are finished and working correctly.
- October 7, 2002.
KDG reports that it appears as though two motherboards
have failed, and need to be replaced, one with an EIDE southbridge
problem, and one with a PCI problem..
JL reports that the RAID array and head node are scheduled to
ship the next day.
KDG wonders if we can install 2 NICs in the 1U head node?
JCA follows up with JL.
- October 8-9, 2002.
JL reports that the 1U chassis only allows a single NIC.
Switching to a 2U chassis will cost an extra $250
(labor, chassis, 2 riser cards).
JCA oks the change to our order.
- October 9, 2002.
A/C accidently shut down overnight; on arrival,
room temperature 76 F,
slave 9 temperature alarm sounding.
Resetting thermostat fixes problem.
- October 16, 2002.
JL reports the RAID array and head node ship today.
- October 17-18, 2002.
Digital certificate for secure socket layer (SSL) usage
requested and received from IPSCA.
- October 18, 2002.
JCA begins annual NSF report for year 2.
KDG begins work on poster for Calvin College Science Division Fair.
- October 22, 2002.
The RAID array and head node arrive from Western Scientific.
KDG begins the installation.
- October 23, 2002.
RAID array initialized, replacement rails needed (and ordered from Vic).
KDG installed the riser card in the 2U head node so that we can fit
two Syskonnect gigabit NICs in the head node and use channel bonding
to double the bandwidth between the head node and the switch.
The head node boots fine with one NIC in place, it will not boot
with two NICs in place.
JCA contacts Western Scientific;
KDG begins planning migration of the backup, HTTP, DNS, and DHCP
services to the headnode from master1.
- October 24, 2002.
Requested an upgrade of
various Calvin DNS entries to provide convenient abbreviations
for the Ohm master nodes.
- October 25, 2002.
Ohm Poster on display at Calvin College Science Division fair.
The RAID array is configured, up, and running, but using just 1 NIC.
- October 26, 2002.
Jeremy Andrus and Jeremy Heersink present their summer research, titled
FireEyez, A Beowulf Monitoring System,
at the
Thirteenth Annual Argonne Symposium for Undergraduates in Science,
Engineering and Mathematics, at
Argonne National Laboratory, and
report on their experience.
- October 26-31, 2002.
Lots of back-and-forth e-mail to Western Scientific trying to resolve
the RAID array's problem with 2 NICs.
Problem still not resolved.
- October 31, 2002.
JCA contacted by Benjamin Crill (a senior at Huntington College, IN)
for information about Ohm and our project.
Sent him an overview of our project and link to its website.
- November 1, 2002.
The hard drive in master1 fails.
More e-mail exchanges with Western Scientific.
- November 5, 2002.
Another hard drive fails. Both this one and that one that failed
11/1 were RMA replacements from IBM. 8^(
- November 6, 2002.
Ohm powered down until replacement hardware (2 disk drives,
2 motherboards, 5 fans, ...) arrive.
- November 7, 2002.
One of the defective motherboards begins working again.
The system elves are apparently up to their old tricks again.
- November 8, 2002.
JCA and KDG begin work on abstract for the
Michigan Academy,
detailing KDG's summer work on dynamic reconfiguration of Ohm.
- November 11, 2002.
JCA begins advertising for KDG's replacement.
KDG finishes rackmounting the new head node and RAID array.
- November 13, 2002.
Jeremy Andrus and Jeremy Heersink give Calvin seminar on FireEyez,
their custom PCI monitoring board.
- November 14, 2002.
A final abstract
is sent to Michigan Academy.
Title is: On-the-fly Beowulf Cluster Topology Reconfiguration.
- November 20, 2002.
Replacement hard drives arrive from IBM; KDG installs them.
Cluster up and running again.
- November 21, 2002.
Eliot Eshelman hired as cluster administrator for 2003/4.
- November 22, 2002.
New head node crashed unexpectedly.
KDG suspects a problem with the RAID array.
- November 22-25, 2002.
Chemistry 104 lab using WebMO for visualizing molecules.
Java applets in latest version of WebMO are causing browsers to crash:
IE 6.0 -> WebMO's Java applets crashed the browser
NS 6.2 -> WebMO's Java applets crashed the browser
Moz 1.2b -> WebMO's Java applets malfunctioned, but did not crash the browser
NS 4.7 -> WebMO worked flawlessly (so used _this_ in the lab)
- November 27, 2002.
NSF Annual Report submitted.
- December 1, 2002.
Notified by
Michigan Academy that our abstract/submission on dynamic reconfiguration
was accepted for presentation at its
March 2003
Saturday morning session.
- December 9, 2002.
Head node crashed again.
Mysterious: everything locks up to the point that you must cycle the power
switch; then there are no diagnostics in the error logs...
- December 10, 2002.
Tape drive is not working with the head node.
KDG trying to track down the problem.
- December 11, 2002.
KDG gets the RAID array working again: the problem seems to be a loose
connection on the SCSI cable.
KDG reports that the tape drive works flawlessly on his Win2K machine,
but he's still having problems with it on the cluster (running Linux).
KDG also has finished setting up the accounts for the 2003 January-term course in
High Performance Computing.
Mackenzie Cummings (MC) has started porting Jeremy Frens'
matrix-factorization programs to the cluster and reports problems using MPI.
It turns out the problem is that he was logged into the head node
(which is dedicated to providing http, nfs, dhcp, ... services)
instead of a master node.
- December 17, 2002.
Randy Moeller (RM) begins running dnetc -- a
distributed.net
client for cracking RC5-72 -- on Ohm.
- December 20, 2002.
JG sets up
ohm1 and ohm2 as DNS aliases for
ohm-master1.calvin.edu and ohm-master2.calvin.edu,
respectively.
Vic returns the last motherboard, which he found to be operating normally.
- January 6, 2003.
Another hard drive fails.
Kevin and Eliot will work together on it, to help Eliot learn the ropes.
- January 7, 2003.
January term 2003 begins, with fourteen students signed up to take
High Performance Computing,
including Jin Yi, a computer science and cognitive science graduate
from the University of Michigan.
- January 13, 2003.
Head node crashes again.
KDG restricts access to passwd to force students to change
their password on the head node, and on the head node
creates a replacement script that changes a user's password universally.
- January 17-20, 2003.
Students reporting X11 problems on Ohm.
These turn out to be false alarms; they are neglecting to link in the
right libraries (in their Makefiles).
- January 23, 2003.
JCA follows up with Western Scientific about getting the two gigabit NICs to
work with the head node.
(Apparently a circular wait was occurring.)
- January 24 - Februrary 5, 2003.
KDG and "Hank" from Western Scientific play phone tag over the 2-NIC problem.
- Februrary 5, 2003.
KDG hands over administration duties to Eliot Eshelman (EE).
- Februrary 7, 2003.
EE reports the hard drive in node 12 has crashed,
and that slave 8 crashed but appears to be "ok" on reboot.
- Februrary 17-25, 2003.
KDH and Hank from WM talk and exchange e-mails regarding the 2-NIC problem.
Western Scientific's records indicate that they shipped us two
"flex risers" whereas we received a single "fixed riser",
so there was a glitch in the packing.
They ship us the correct item and we return the incorrect item.
- Februrary 26, 2003.
KDG and EE install the two gigabit NICs using the flex riser:
We connected the topmost gigabit card to the first
PCI slot (64-bit, white), and the other gigabit card (in the middle slot)
to the second PCI slot (64-bit, green), via the flex riser cables.
Bootup is completely normal, and the kernel sees both cards just fine, as
indicated by 'dmesg' and 'lspci'. Bringing them up (via 'ifconfig')
works fine as well. We now have eth0 and eth1 (the gigabit cards) and
eth2/eth3 (the onboard 3com interfaces).
EE begins work on channel bonding the two NICs and our switch.
- Februrary 28, 2003.
EE reports that the power supply failed in node 13.
The replacement hard drive has arrived for node 12,
and it seems to be running fine again.
Node 5 had a memory failure; EE has gotten it going using
memory from the spare node.
Ohm is fully functional again!
- March 4, 2003.
JCA (in Pennsylvania for father's funeral) gives
updated version of SIGCSE 2002 talk
at Geneva College.
- March 8-20, 2003.
EE experiments with jumbo frames to see if enabling them affects performance;
it does not seem to.
- March 22, 2003.
KDG presents
Dynamic Beowulf Topology:
Realtime Supercomputer Network Reconfiguration at the
2003
Michigan Academy meeting.
Kevin does an excellent job, and the project generates much discussion
among those present.
- April 1, 2003.
JCA contacted by Ben Meisgeier, a student at Central College in Pella, IA
with questions about buiding a cluster there.
- April 3, 2003.
JCA contacted by Peter McDevitt, an instructor at
New Brunswick Community
College -- Saint Johns Campus
with questions about building a cluster for an upcoming IT fair there.
about buiding a cluster there.
- April 5-8, 2003.
JCA contacted by Brett van de Sande who is building a Beowulf cluster at
Geneva College for feedback on his
design.
Brett suggests that the problems we're having with Stan Haan's large
matrix programs (seg faults when matrices larger than 1024 are used)
may be related to a stack size limitation in the shell.
This value can be minipulated via ulimit in bash,
and via limit in csh and its descendents.
After some experimentation, it appears that
$ulimit -s unlimited
eliminates the problem with C programs;
it remains to be seen if this will solve Haan's problems
(we'll find out summer 2003,
but we're optimistic!
- April 21, 2003.
EE reports he's gotten the tape drive working cleanly,
using these two resources:
http://www.linux1onstream.nl
http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html
EE speculates that the problem may have been usage of a block size
other than 32K blocks.
- April 22, 2003.
EE reports memory errors in node 8.
Thankfully, our Crucial memory has a lifetime warranty!
- April 28, 2003.
Peter McDevitt (see April 3) that he and three students have built a
20-node cluster.
They started on April 14 and had everything up and running by April 25,
just in time for their IT fair.
- May 8, 2003.
EE reports that the backup sequence is now completely automated
and running on a regular rotation.
He also reports that he's located a suite of benchmarks at NASA
(the NAS Parallel
Benchmarks) that look like they will serve as part of our
summer 2003 test suite.
- May 12, 2003.
JCA contacted by Tom Frahm from Atipa
Technologies, a vendor selling cluster technology.
- May 17-26, 2003.
JCA and EE begin discussing exactly how to categorize applications
as coarse, medium, or fine-grained.
The problem is to come up with an empirical means of categorizing
an application as coarse, medium, or fine-grained.
The logical way to do it seems to be to measure each application's
network behavior.
Eliot suggests two means of doing so:
-
Using SNMP and our switch, we can collect a summary of the
running application's network behavior once each 14 seconds; or
-
By writing some scripts that use a utility like ntop,
we can record each node's network behavior and then tabulate
all of the results.
Since it will accomplish the same thing without our having to write
custom scripts, we decide to use the SNMP/switch approach.
JCA suggests a multi-phase approach for the summer:
- Phase 1. We run each application in the test suite
and measure its network behavior,
in order to empirically classify it as fine, medium, or coarse-grained.
- Phase 2. We run each application three times
-- once under each topology -- and time how long each takes.
- Phase 3. We analyze the timing data we collected
in phase 2, looking for consistency or trends within granularity
categories.
- Phase 4. Based on what we discover in phase 3,
we perform a cost analysis of our three topologies.
- Phase 5. Write up our results.
- May 29, 2003.
EE reports that the CPU fan on node 5 was not spinning
(with no temperature warnings).
He requests a new fan from Vic and begins checking all nodes just in case...
EE also reports on his granularity measurements.
Using a spreadsheet, he's prepared charts/graphs of the various
application's network behavior over the course of the computation.
While the different applications generate network traffice at differing rates,
they behave VERY consistently over time (perhaps because we are recording
minute-by-minute).
- May 30, 2003.
EE computes and reports median values for the number of bytes/minute
each of the various applications generates:
7.0 x 10^2 - DPMTA
5.0 x 10^2 - EP
2.0 x 10^5 - FastDNAml
1.5 x 10^7 - LU
2.0 x 10^7 - BT
2.5 x 10^7 - MG, SP
3.5 x 10^7 - NAMD
7.5 x 10^7 - MDynamix, CG
1.0 x 10^8 - IS
Thanks to the applications' consistent behavior over time,
we can use these values to define "granularity thresholds"
between fine, medium, and coarse-grained computations.
For example, we might define any computation that generates below 10^3
chars/minute as coarse-grained, any computation that generates 10^6
or more chars/minute as fine-grained, and anything in between as medium-grained.
This scheme creates a "diameter" of three orders of magnitude
across each of our granularity categories.
- June 2, 2003.
JCA provides a set of links EE can check to look for more applications,
to beef up the number of entries in our medium- and coarse-grained categories.
- June 3, 2003.
EE reports he's found another medium-grained application: PVMPOV,
that generates behavior similar to that of FastDNAml.
If our scheme is valid, this gives us two coarse-grained apps,
two medium-grained apps. and eight fine-grained apps.
EE also reports his results of computing the average number of bytes/sec
each application transfers across the network during the computation:
EP: 5.5 * 10^1
Dnetc: 7.6 * 10^1
DPMTA: 5.8 * 10^2
FastDNAml: 1.5 * 10^4
PVMPOV: 1.2 * 10^4
LU: 1.0 * 10^6
Parmetis: 1.2 * 10^6
BT: 1.3 * 10^6
MG: 1.4 * 10^6
SP: 1.7 * 10^6
NAMD: 2.2 * 10^6
CG: 5.2 * 10^6
MDynamix: 5.2 * 10^6
IS: 6.0 * 10^6
These results are consistent with our median bytes/minute measurement
(see May 30 above), with the applications "clumping" into three groups:
-
Those below 10^3 bytes/sec: EP, Dnetc, DPMTA
-
Those between 10^3 and 10^6 bytes/sec: FastDNAml, PVMPOV
-
Those above 10^6 bytes/sec: LU, Parmetis, BT, MG, SP, NAMD, CG, MDynamix, IS
- July 14, 2003. EE has compiled and installed several additional applications. The groups of applications are:
-
Those below 10^3 bytes/sec: EP, Dnetc, DPMTA
-
Those between 10^3 and 10^6 bytes/sec: FastDNAml, PVMPOV, Gamess, Gadget, HPLinpack
-
Those above 10^6 bytes/sec: LU, Parmetis, BT, MG, SP, NAMD, CG, MDynamix, IS, TPM, GROMACS
EE has spent the last few weeks timing each application's performance on our three topologies: star, ring-star and hypercube-star. Each application was run 10 times on each topology, and the results averaged. A full analysis of the data has not yet been done.
- July 15, 2003a.
Discussed with EE:
building a Java applet to present our performance/price results.
Discussed with Paul Harper (Physics):
getting
gridMathematica from
Wolfram Research for the cluster.
Paul is interested in writing high performance programs using Mathematica,
and gridMathematica seems to provide the most convenient way to do so
while taking advantage of the cluster.
Paul will follow-up with CIT about how best to go about this,
given Calvin's site-license for Mathematica.
- July 16, 2003a.
EE reports that most of the applications he has found but not been able to
compile mention that they are compiled using the Portland Group's compiler.
It looks like that would be a good component to get if we want to include
those applications in our testing.
Will wait to see how much money remains in the budget in September.
- July 18, 2003.
EE reports that Prof. Haan's "small" program runs fine with a stack
as small as 128; his "large" program seg faults even with an "unlimited"
stack size.
- July 21, 2003.
Rob Bobeldyk (CIT) forwards a message from Kelvin Mischo (Wolfram)
indicating that getting gridMathematica will just require us to
adjust our licensing agreement with Wolfram and pay
an increase of about $300/year.
Heard from O. William McClung at
Nebraska Wesleyan University.
They are interested in building their own cluster on an HHMI grant.
Forwarded them a copy of the .PDF files I uploaded for the grant.
They are interested in running Gaussian.
Since we have upgraded our OS version (to RH 7.3) since we tried to install it,
I wonder if Gaussian might now run correctly?
- July 23, 2003.
Discussed with EE and Jeremy Frens how to best analyze the data we are getting.
Decided that our performance/price ratio should be speedup/price,
where an application's speedup is its time-for-1-CPU/time-for-N-CPUs.
This means we will need to rerun each of our applications using a single CPU,
in order to get its time-for-1-CPU.
Since we already have the times for N-CPUs on each topology,
those times will let us compute each application's speedup for a given topology.
We can then divide that topology's speedup by the price for that topology,
to get its performance/price ratio.
For a given application, whichever topology maximizes this ratio is the
best topology on which to run the application.
Since these are parallel applications, it may take some time to get these
time-for-1-CPU values...
- July 30, 2003.
EE reports that he has a
working version of the results applet,
into which he can plug the time-for-1-CPU values as they complete.
It presents its results using our year-2000 prices.
Discussed providing input boxes whereby a user could enter current prices
to see how the performance/price ratio has changed.
This will provide an easy way for users of one of these applications
(or of applications similar to these) to apply our results to their situation.
Also discussed using tabbed panes to break down the results into
granularity categories.
- August 4, 2003.
EE has refined
the results applet
as we discussed.
He reports that all applications (but not all benchmarks) have
completed their 1-CPU executions and been added to the applet.
He also reports that the large fan in s2 was beginning to whine,
so he replaced it with the spare node's fan.
In s10, the large fan's warning light is on, though the fan is still
spinning quietly.
Looks like we need some new fans.
He also reported that he has begun work on his poster for Calvin's
Science Division Poster Session next fall.
- August 5, 2003.
Reviewed first draft of poster.
Also reviewed EE's close-to-final version of
the results applet
-- it's looking pretty good!
- August 6, 2003.
Proof-read second draft of poster.
- August 7, 2003.
EE working on installing an upgrade for WebMO --
the web-based front-end for Gaussian.
Once all tests have completed, he will re-install Gaussian.
- August 8, 2003.
EE reports that gridMathematica has arrived and he is in the process
of installing it.
- August 19, 2003.
EE reports:
-
The 1-CPU tests have all completed, and he is adding them to
the results applet.
-
gridMathematica, Gaussian, and WebMO
have all been installed and seem to be working correctly.
-
The hard drive in s7 has died.
This is the last straw.
We decide to begin the process of reconfiguring all of our nodes
(except the head node that serves access to the RAID array)
as diskless nodes that boot across the network from a floppy.
These lousy disks and the case-fans are the only mechanical components
in the cluster, and they have far and away been the most failure-prone.
The only drawback we can see is that the /swap partition
will now be across the network, and thus take longer.
However with 1GB RAM in each node, we have never seen the hard drive
access light go on, once the node has booted, for any of our applications.
As a result, we don't think this will be much of a problem --
if you never have to swap, it doesn't matter how much longer it will take...
To anyone who reads this: consider seriously designing your system to have
diskless nodes, and put the $$ you would have spent on the disks into RAM.
Minimize the mechanical (translation: failure-prone) components in
your system.
Also sent a note to Bill McClung at Nebraska Wesleyan U. giving him the good
news about Gaussian. In his reply, he thanked us for this journal,
which has been very useful to them in planning their project.
- August 20, 2003.
With Gaussian working now, we decide to add a couple of Gaussian computations
to our list of applications.
More specifically, we choose two of the Gaussian computations (231 and 358)
more or less at random, to see how they perform on the differing topologies.
EE begins work on figuring out how to reconfigure Ohm to have/use diskless nodes.
- August 27, 2003.
EE reports:
-
He has successfully created a boot floppy to boot a node across the network.
So far, each node needs its own boot floppy.
We discuss modifying its scripts and the head/server content to identify
the nodes using their MAC address,
so that a single floppy will suffice to boot any node.
-
So long as we're reconfiguring the cluster, EE suggests that we consider
going with the
Gentoo distribution of Linux, in order to simplify updates/maintenance,
and ensure that our binaries are optimized.
Redhat's package manager can be installed on top of Gentoo,
for apps that are only distributed that way.
(For those distributed as source rpms, they can be converted to .tar files
and then installed via Gentoo's normal installer.)
EE is going to back everything up, give it a try and report any problems.
-
The two Gaussian computations seem to perform almost the same regardless of
the network topology, so their change in speedup is basically zero as we
vary the topology.
Their bandwidth use is roughly 10-3 bytes/sec, placing them in the
course-grained category.
-
The poster for the science division poster fair is in final form.
- August 29, 2003.
EE reports the following data from the Gaussian computations:
Gaus-231:
Star: 1172
Ring: 1173
Hyper: 1171
Single: 1349
Speedup: ~1.15 for all of the topologies
Gaus-358:
Star: 415
Ring: 414
Hyper: 414
Single: 1297
Speedup: ~3.12 for all of the topologies
- August 31, 2003.
End of project.
For those interested, our results are online, as follows:
-
The final version of our results applet,
showing the performance-price ratio for each topology for:
-
the averages for each of our granularity categories,
-
the performance/price ratios for our coarse-grained applications,
-
the performance/price ratios for our medium-grained applications,
-
the performance/price ratios for our fine-grained applications, and
-
the performance/price ratios for all of our applications;
-
The raw data results of our bandwidth tests,
that we used to categorize applications as coarse, medium, or fine-grained;
-
The results of our single-node timing experiments
(in both raw and summarized forms)
that we used for the numerators in our speedup computations; and
-
The results of our mutiple node timing experiments
(in both raw and summarized forms)
that we used as the denominators in our speedup computation.
We'll be writing these up for submission to a conference or journal,
as soon as we finish our final project report.
Our thanks to the
National Science Foundation,
for making it all possible!