• extending our ring topology into a hypercube (34 NICs), and
• adding gigabit NICs to the RAID array's head node (2 NICs).

KDG reports that the tape drive works flawlessly on his Win2K machine, but he's still having problems with it on the cluster (running Linux).

KDG also has finished setting up the accounts for the 2003 January-term course in High Performance Computing.

Mackenzie Cummings (MC) has started porting Jeremy Frens' matrix-factorization programs to the cluster and reports problems using MPI. It turns out the problem is that he was logged into the head node (which is dedicated to providing http, nfs, dhcp, ... services) instead of a master node.

Vic returns the last motherboard, which he found to be operating normally.

KDG restricts access to passwd to force students to change their password on the head node, and on the head node creates a replacement script that changes a user's password universally.

    We connected the topmost gigabit card to the first
    PCI slot (64-bit, white), and the other gigabit card (in the middle slot)
    to the second PCI slot (64-bit, green), via the flex riser cables.

    Bootup is completely normal, and the kernel sees both cards just fine, as
    indicated by 'dmesg' and 'lspci'.  Bringing them up (via 'ifconfig')
    works fine as well.  We now have eth0 and eth1 (the gigabit cards) and
    eth2/eth3 (the onboard 3com interfaces).
   

Brett suggests that the problems we're having with Stan Haan's large matrix programs (seg faults when matrices larger than 1024 are used) may be related to a stack size limitation in the shell. This value can be minipulated via ulimit in bash, and via limit in csh and its descendents. After some experimentation, it appears that

         $ulimit -s unlimited
    

He also reports that he's located a suite of benchmarks at NASA (the NAS Parallel Benchmarks) that look like they will serve as part of our summer 2003 test suite.

• Using SNMP and our switch, we can collect a summary of the running application's network behavior once each 14 seconds; or
• By writing some scripts that use a utility like ntop, we can record each node's network behavior and then tabulate all of the results.

JCA suggests a multi-phase approach for the summer:

1. Phase 1. We run each application in the test suite and measure its network behavior, in order to empirically classify it as fine, medium, or coarse-grained.
2. Phase 2. We run each application three times -- once under each topology -- and time how long each takes.
3. Phase 3. We analyze the timing data we collected in phase 2, looking for consistency or trends within granularity categories.
4. Phase 4. Based on what we discover in phase 3, we perform a cost analysis of our three topologies.
5. Phase 5. Write up our results.

EE also reports on his granularity measurements. Using a spreadsheet, he's prepared charts/graphs of the various application's network behavior over the course of the computation. While the different applications generate network traffice at differing rates, they behave VERY consistently over time (perhaps because we are recording minute-by-minute).

    7.0 x 10^2 - DPMTA
    5.0 x 10^2 - EP
    2.0 x 10^5 - FastDNAml
    1.5 x 10^7 - LU
    2.0 x 10^7 - BT
    2.5 x 10^7 - MG, SP
    3.5 x 10^7 - NAMD
    7.5 x 10^7 - MDynamix, CG
    1.0 x 10^8 - IS
  

EE also reports his results of computing the average number of bytes/sec each application transfers across the network during the computation:

   EP:         5.5 * 10^1
   Dnetc:      7.6 * 10^1
   DPMTA:      5.8 * 10^2
   FastDNAml:  1.5 * 10^4
   PVMPOV:     1.2 * 10^4
   LU:         1.0 * 10^6
   Parmetis:   1.2 * 10^6
   BT:         1.3 * 10^6
   MG:         1.4 * 10^6
   SP:         1.7 * 10^6
   NAMD:       2.2 * 10^6
   CG:         5.2 * 10^6
   MDynamix:   5.2 * 10^6
   IS:         6.0 * 10^6
  

• Those below 10^3 bytes/sec: EP, Dnetc, DPMTA
• Those between 10^3 and 10^6 bytes/sec: FastDNAml, PVMPOV
• Those above 10^6 bytes/sec: LU, Parmetis, BT, MG, SP, NAMD, CG, MDynamix, IS

• Those below 10^3 bytes/sec: EP, Dnetc, DPMTA
• Those between 10^3 and 10^6 bytes/sec: FastDNAml, PVMPOV, Gamess, Gadget, HPLinpack
• Those above 10^6 bytes/sec: LU, Parmetis, BT, MG, SP, NAMD, CG, MDynamix, IS, TPM, GROMACS

EE has spent the last few weeks timing each application's performance on our three topologies: star, ring-star and hypercube-star. Each application was run 10 times on each topology, and the results averaged. A full analysis of the data has not yet been done.

Discussed with Paul Harper (Physics): getting gridMathematica from Wolfram Research for the cluster. Paul is interested in writing high performance programs using Mathematica, and gridMathematica seems to provide the most convenient way to do so while taking advantage of the cluster. Paul will follow-up with CIT about how best to go about this, given Calvin's site-license for Mathematica.

Heard from O. William McClung at Nebraska Wesleyan University. They are interested in building their own cluster on an HHMI grant. Forwarded them a copy of the .PDF files I uploaded for the grant. They are interested in running Gaussian. Since we have upgraded our OS version (to RH 7.3) since we tried to install it, I wonder if Gaussian might now run correctly?

Since these are parallel applications, it may take some time to get these time-for-1-CPU values...

He also reports that the large fan in s2 was beginning to whine, so he replaced it with the spare node's fan. In s10, the large fan's warning light is on, though the fan is still spinning quietly. Looks like we need some new fans.

He also reported that he has begun work on his poster for Calvin's Science Division Poster Session next fall.

Also reviewed EE's close-to-final version of the results applet -- it's looking pretty good!

• The 1-CPU tests have all completed, and he is adding them to the results applet.
• gridMathematica, Gaussian, and WebMO have all been installed and seem to be working correctly.
• The hard drive in s7 has died. This is the last straw. We decide to begin the process of reconfiguring all of our nodes (except the head node that serves access to the RAID array) as diskless nodes that boot across the network from a floppy. These lousy disks and the case-fans are the only mechanical components in the cluster, and they have far and away been the most failure-prone.

  The only drawback we can see is that the /swap partition will now be across the network, and thus take longer. However with 1GB RAM in each node, we have never seen the hard drive access light go on, once the node has booted, for any of our applications. As a result, we don't think this will be much of a problem -- if you never have to swap, it doesn't matter how much longer it will take...

  To anyone who reads this: consider seriously designing your system to have diskless nodes, and put the $$ you would have spent on the disks into RAM. Minimize the mechanical (translation: failure-prone) components in your system.

EE begins work on figuring out how to reconfigure Ohm to have/use diskless nodes.

• He has successfully created a boot floppy to boot a node across the network. So far, each node needs its own boot floppy. We discuss modifying its scripts and the head/server content to identify the nodes using their MAC address, so that a single floppy will suffice to boot any node.
• So long as we're reconfiguring the cluster, EE suggests that we consider going with the Gentoo distribution of Linux, in order to simplify updates/maintenance, and ensure that our binaries are optimized. Redhat's package manager can be installed on top of Gentoo, for apps that are only distributed that way. (For those distributed as source rpms, they can be converted to .tar files and then installed via Gentoo's normal installer.) EE is going to back everything up, give it a try and report any problems.
• The two Gaussian computations seem to perform almost the same regardless of the network topology, so their change in speedup is basically zero as we vary the topology. Their bandwidth use is roughly 10^-3 bytes/sec, placing them in the course-grained category.
• The poster for the science division poster fair is in final form.

   Gaus-231:
       Star:   1172
       Ring:   1173
       Hyper:  1171
       Single: 1349
      Speedup: ~1.15 for all of the topologies
   Gaus-358:
       Star:   415
       Ring:   414
       Hyper:  414
       Single: 1297
      Speedup: ~3.12 for all of the topologies
   

1. The final version of our results applet, showing the performance-price ratio for each topology for:
   • the averages for each of our granularity categories,
   • the performance/price ratios for our coarse-grained applications,
   • the performance/price ratios for our medium-grained applications,
   • the performance/price ratios for our fine-grained applications, and
   • the performance/price ratios for all of our applications;
2. The raw data results of our bandwidth tests, that we used to categorize applications as coarse, medium, or fine-grained;
3. The results of our single-node timing experiments (in both raw and summarized forms) that we used for the numerators in our speedup computations; and
4. The results of our mutiple node timing experiments (in both raw and summarized forms) that we used as the denominators in our speedup computation.