From mprinkey at aeolusresearch.com Mon Sep 1 16:26:17 2003 From: mprinkey at aeolusresearch.com (Michael T. Prinkey) Date: Mon, 1 Sep 2003 16:26:17 -0400 (EDT) Subject: Has Any Tested These? Message-ID: Adaptec's new gigabit ethernet card with TCP offloading: http://www.eweek.com/article2/0,3959,1228684,00.asp Any word on pricing? Thanks, Mike Prinkey Aeolus Research, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From johnt at quadrics.com Tue Sep 2 07:56:52 2003 From: johnt at quadrics.com (John Taylor) Date: Tue, 2 Sep 2003 12:56:52 +0100 Subject: Has Any Tested These? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA7CCDF25@stegosaurus.bristol.quadrics.com> The article suggests pricing at the end of about 800-900 USD about the same price as NICS for other high performance products which offer even better b/w and latency, notwithstanding switch costs of course. Be interested in performance of this. John > -----Original Message----- > From: Michael T. Prinkey [mailto:mprinkey at aeolusresearch.com] > Sent: 01 September 2003 21:26 > To: beowulf at beowulf.org > Subject: Has Any Tested These? > > > > Adaptec's new gigabit ethernet card with TCP offloading: > > http://www.eweek.com/article2/0,3959,1228684,00.asp > > Any word on pricing? > > Thanks, > > Mike Prinkey > Aeolus Research, Inc. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wade.hampton at nsc1.net Tue Sep 2 14:04:19 2003 From: wade.hampton at nsc1.net (Wade Hampton) Date: Tue, 02 Sep 2003 14:04:19 -0400 Subject: e1000 and Scyld 28cz Message-ID: <3F54DBA3.8060608@nsc1.net> G'day, I'm upgrading my first test cluster to 1G ethernet using the onboard e1000 parts on my nodes (Tyan motherboards with eepro100 and e1000). I have not gotten the nodes to boot and recognize the e1000 cards using a beoboot floppy. When I tried the latest driver from Intel (5.1.13), I could not get an e1000 module for my node boot floppy until I did the following: CFLAGS=$(KCFLAGS) make KCFLAGS="-D__BOOT_KERNEL_H_ -D__module__beoboot mv e1000.o /lib/modules/2.2.19-14.beobeoboot/net I also added the following to /etc/beowulf/config pci 0x8086 0x1008 e1000 (reference: http://www.beowulf.org/pipermail/beowulf/2002-September/004575.html) When I try to boot this floppy, it stopped booting after loading the e1000 driver and appeared to hang. I then removed the e1000.o modules and tried the Scyld RPM for 4.4.12. This seemed to hang at the same location. Any help in this matter would be appreciated. -- Wade Hampton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Tue Sep 2 22:57:47 2003 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Tue, 02 Sep 2003 21:57:47 -0500 Subject: Oh great pool of hardware and driver knowledge.... Message-ID: <3F5558AB.2010605@tamu.edu> OK, so this is off-topic again, but this group is likely the most knowledgable source of info likely to have a handle on something like this. I've got a pair of servers directly off a router/switch. There is one problemmatic server/port interface, as shown in the MRTG page at http://page4.tamu.edu/mrtg/atmres/165.91.1.44_40.html If you notice, connectivity appears to drop every 30 min. although subjectively it appears to happen more often (say, every 3 min or so...) for periods of 30-45 sec. Hardware here is a Dell PwerEdge 650 with the onboard Intel Pro/E1000 adapter and a 2nd Intel E1000 dual port copper adapter. Software is RH9 with all current patches, Apache and the Minnesota Mapserver, plus PostGres/PostGIS running. NFS services to the box die periodically and users are sayingthe web response stinks. I've taken a couple of decent shots at the problem. I'm hoping someone here might have a potential solution I can check out! Thanks! gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lehi.gracia at amd.com Wed Sep 3 16:16:07 2003 From: lehi.gracia at amd.com (lehi.gracia at amd.com) Date: Wed, 3 Sep 2003 15:16:07 -0500 Subject: Oh great pool of hardware and driver knowledge.... Message-ID: <99F2150714F93F448942F9A9F112634C07BE62FE@txexmtae.amd.com> I would try Dell drivers instead of the Intel drivers, I don't know if that is what you are using or not, but Dell does some driver development for their systems and might be different than the Intel ones. Sorry can't help any more but there is not a lot of info on your system's configuration, although it might not make a lot of difference because I haven't touch a PE650 for a few months now. Try different switch port, different switch, different network board, if you are trunking try without, try the latest DSA cd from Dell, it might have some new patches in it. Lehi -----Original Message----- From: Gerry Creager N5JXS [mailto:gerry.creager at tamu.edu] Sent: Tuesday, September 02, 2003 9:58 PM To: Beowulf Subject: Oh great pool of hardware and driver knowledge.... OK, so this is off-topic again, but this group is likely the most knowledgable source of info likely to have a handle on something like this. I've got a pair of servers directly off a router/switch. There is one problemmatic server/port interface, as shown in the MRTG page at http://page4.tamu.edu/mrtg/atmres/165.91.1.44_40.html If you notice, connectivity appears to drop every 30 min. although subjectively it appears to happen more often (say, every 3 min or so...) for periods of 30-45 sec. Hardware here is a Dell PwerEdge 650 with the onboard Intel Pro/E1000 adapter and a 2nd Intel E1000 dual port copper adapter. Software is RH9 with all current patches, Apache and the Minnesota Mapserver, plus PostGres/PostGIS running. NFS services to the box die periodically and users are sayingthe web response stinks. I've taken a couple of decent shots at the problem. I'm hoping someone here might have a potential solution I can check out! Thanks! gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joachim at ccrl-nece.de Thu Sep 4 03:57:50 2003 From: joachim at ccrl-nece.de (Joachim Worringen) Date: Thu, 4 Sep 2003 09:57:50 +0200 Subject: Intel acquiring Pallas In-Reply-To: <1062089290.4363.22.camel@localhost.localdomain> References: <1062089290.4363.22.camel@localhost.localdomain> Message-ID: <200309040957.50028.joachim@ccrl-nece.de> Vann H. Walke: > Pallas's main HPC product is Vampir/Vampirtrace. These are performance > analysis tools. As such they would only be peripherally useful for > compiler design (perhaps to measure the effects of certain changes). > Even for this purpose, Vampir/Vampirtrace doesn't provide the amount of > detail that Intel's own V-Tune product does. I agree with Vann. I assume they just want to extend V-Tune upwards, and get a fully integrated product. And maybe some additional knowledge on MPI (Pallas implemented/maintains the MPI for Fujitsu machines and maybe others). I wonder if Pallas will continue to sell PGI compilers (which are optimized for AMD/Opteron...). BTW, Intel also owns 25% of Scali. And Etnus, which produce TotalView, which Pallas sells, belongs to Dolphin, which in turn make a lot of business with Scali. Whatever that means... but it's a small world. Joachim -- Joachim Worringen - NEC C&C research lab St.Augustin fon +49-2241-9252.20 - fax .99 - http://www.ccrl-nece.de _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Thu Sep 4 11:32:22 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 04 Sep 2003 11:32:22 -0400 Subject: perl-bproc bindings In-Reply-To: References: Message-ID: <1062689542.2576.12.camel@roughneck> On Thu, 2003-08-28 at 14:35, Donald Becker wrote: > On 28 Aug 2003, Nicholas Henke wrote: > > > On Thu, 2003-08-28 at 01:49, Donald Becker wrote: > > > On Tue, 26 Aug 2003, Daniel Widyono wrote: > > > > > > > Anyone out there besides Dan Ridge from Scyld make Perl bindings for bproc, > > > > something more recent than spring of 2001? > > > > > > There are updated bindings, and a small example, at > > > ftp://www.scyld.com/pub/beowulf-components/bproc-perl/bproc-perl.tar.gz > > > > Any chance you guys have updated python bindings as well? > > 0.9-8 is the current version -- which are you using? > The last bugfix was logged in October of 2003. I am using pybproc-0.8. Where can I find the version you are referring to? > > The next planned refresh has added bindings for the Beostat statistics > library, Beomap job mapping and BBQ job scheduling systems. Awesome -- i look forward to using them. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ktaka at clustcom.com Thu Sep 4 10:49:40 2003 From: ktaka at clustcom.com (Kimitoshi Takahashi) Date: Thu, 04 Sep 2003 23:49:40 +0900 Subject: e1000 and Scyld 28cz In-Reply-To: <3F54DBA3.8060608@nsc1.net> References: <3F54DBA3.8060608@nsc1.net> Message-ID: <200309041449.AA00364@jack2.clustcom.com> Hello, >From my limited experience, I can think of the followings, 1. If your NIC chip is 82547(Intel's giga nic chip for CSA connection), the line in the /etc/beowulf/config should be, pci 0x8086 0x1019 e1000 You have to find out what your NIC's pci ID is. In the case of recent RedHat, see /etc/sysconfig/hwconfig. 2. You might have to rebuild phase 2 boot image with new e1000.o using beoboot. I hope these help. Kimitoshi Takahashi Wade Hampton ????????: >G'day, > >I'm upgrading my first test cluster to 1G ethernet >using the onboard e1000 parts on my nodes (Tyan >motherboards with eepro100 and e1000). > >I have not gotten the nodes to boot and recognize >the e1000 cards using a beoboot floppy. > >When I tried the latest driver from Intel (5.1.13), >I could not get an e1000 module for my node boot >floppy until I did the following: > > CFLAGS=$(KCFLAGS) > make KCFLAGS="-D__BOOT_KERNEL_H_ -D__module__beoboot > mv e1000.o /lib/modules/2.2.19-14.beobeoboot/net > >I also added the following to /etc/beowulf/config > pci 0x8086 0x1008 e1000 > > (reference: >http://www.beowulf.org/pipermail/beowulf/2002-September/004575.html) > >When I try to boot this floppy, it stopped booting after loading the >e1000 driver and appeared to hang. > >I then removed the e1000.o modules and tried the Scyld RPM >for 4.4.12. This seemed to hang at the same location. > >Any help in this matter would be appreciated. >-- >Wade Hampton > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tod at gust.sr.unh.edu Thu Sep 4 14:24:41 2003 From: tod at gust.sr.unh.edu (Tod Hagan) Date: 04 Sep 2003 14:24:41 -0400 Subject: Semaphore controlling access to resource in OpenPBS? SGE? Message-ID: <1062699881.452.22.camel@haze.sr.unh.edu> Hi, I'm a guest on a system running OpenPBS where I'd like to schedule a bunch of test jobs using OpenPBS. All the jobs will execute out of the same directory. Rather than duplicating the directory for each job, or using OpenPBS's support for dependency lists and rigidly specifying the execution order myself, what I really want is to declare a semaphore/mutex locking access to the directory so only one of these jobs will run at a time, and then let the batch system figure out which job is best to run given the other jobs on the cluster. Is there any way to declare this kind of semaphore or lock with OpenPBS? How about SGE? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Thu Sep 4 17:17:18 2003 From: rossini at blindglobe.net (A.J. Rossini) Date: Thu, 04 Sep 2003 14:17:18 -0700 Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? Message-ID: <85k78oi78h.fsf@blindglobe.net> I'm in the process of putting together another small compute cluster, and while I've solved the node configuration issues, I've been unable to come to any definitive solutions for networking. Because of the problems I work on (trivial or coarse parallelization, and at the most, moderate between-node communication), I've focused on copper-based gigabit. Now the hard part -- can anyone suggest NICs (Intel's e1000 is currently leading my list) and switches to uses? (Netgear GS508T/GS108, HP ProCurve 2708, etc...?). Throughput is of interest, though I've got to balance it with price... Or even better, point to comparison WWW sites? (I've been looking on Google and Tom's, but havn't really found the right query incantation yet, sigh...). best, -tony -- A.J. Rossini rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW : FAX=206-543-3461 | moving soon to a permanent office FHCRC: 206-667-7025 FAX=206-667-4812 | Voicemail is pretty sketchy/use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Johnson.Jeffre at epamail.epa.gov Thu Sep 4 16:40:53 2003 From: Johnson.Jeffre at epamail.epa.gov (Johnson.Jeffre at epamail.epa.gov) Date: Thu, 04 Sep 2003 13:40:53 -0700 Subject: Microway Opteron Cluster Message-ID: Hello; We are contemplating purchasing a small Opteron-based cluster from Microway, and I am curious as to whether anyone has any thoughts about Microway and/or Opterons. Thank you. Jeffre C. Johnson Research Chemist U.S. EPA, ORD, NERL, HEASD, EDRB (not quite the entire alphabet) Fabulous Las Vegas, Nevada _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lehi.gracia at amd.com Thu Sep 4 17:42:38 2003 From: lehi.gracia at amd.com (lehi.gracia at amd.com) Date: Thu, 4 Sep 2003 16:42:38 -0500 Subject: Microway Opteron Cluster Message-ID: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> Depending on how small is small you might want to look at Angstrom, for higher density, if realstate is a concern. http://www.angstrom.com/ Lehi -----Original Message----- From: Johnson.Jeffre at epamail.epa.gov [mailto:Johnson.Jeffre at epamail.epa.gov] Sent: Thursday, September 04, 2003 3:41 PM To: beowulf at beowulf.org Subject: Microway Opteron Cluster Hello; We are contemplating purchasing a small Opteron-based cluster from Microway, and I am curious as to whether anyone has any thoughts about Microway and/or Opterons. Thank you. Jeffre C. Johnson Research Chemist U.S. EPA, ORD, NERL, HEASD, EDRB (not quite the entire alphabet) Fabulous Las Vegas, Nevada _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctierney at hpti.com Thu Sep 4 17:57:37 2003 From: ctierney at hpti.com (Craig Tierney) Date: 04 Sep 2003 15:57:37 -0600 Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <85k78oi78h.fsf@blindglobe.net> References: <85k78oi78h.fsf@blindglobe.net> Message-ID: <1062712656.2998.12.camel@woody> On Thu, 2003-09-04 at 15:17, A.J. Rossini wrote: > I'm in the process of putting together another small compute cluster, > and while I've solved the node configuration issues, I've been unable > to come to any definitive solutions for networking. Because of the > problems I work on (trivial or coarse parallelization, and at the > most, moderate between-node communication), I've focused on > copper-based gigabit. > > Now the hard part -- can anyone suggest NICs (Intel's e1000 is > currently leading my list) and switches to uses? (Netgear > GS508T/GS108, HP ProCurve 2708, etc...?). Throughput is of interest, > though I've got to balance it with price... I haven't tried it yet, but the 12 and 24 port gigE switches from Dell look attractive. The switches provide full bandwidth and have several other good features. They are under $100 per port ($1199 and $2199 respectively). Craig > > Or even better, point to comparison WWW sites? (I've been looking on > Google and Tom's, but havn't really found the right query incantation > yet, sigh...). > > best, > -tony -- Craig Tierney _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Thu Sep 4 18:40:36 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Thu, 04 Sep 2003 18:40:36 -0400 Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <85k78oi78h.fsf@blindglobe.net> References: <85k78oi78h.fsf@blindglobe.net> Message-ID: <3F57BF64.1010008@bellsouth.net> Tony, SMC recently announced a low-cost 8 port GigE switch that can handle jumbo frames. http://www.smc.com/index.cfm?sec=About-SMC&pg=Press-Release-Details&pr_id=129&site=c They also announced GigE NICs for $30. I don't know about Linux support. If not, here's a website that some some decent information about copper GigE: http://www.cs.uni.edu/~gray/gig-over-copper/ Good Luck! Jeff >I'm in the process of putting together another small compute cluster, >and while I've solved the node configuration issues, I've been unable >to come to any definitive solutions for networking. Because of the >problems I work on (trivial or coarse parallelization, and at the >most, moderate between-node communication), I've focused on >copper-based gigabit. > >Now the hard part -- can anyone suggest NICs (Intel's e1000 is >currently leading my list) and switches to uses? (Netgear >GS508T/GS108, HP ProCurve 2708, etc...?). Throughput is of interest, >though I've got to balance it with price... > >Or even better, point to comparison WWW sites? (I've been looking on >Google and Tom's, but havn't really found the right query incantation >yet, sigh...). > >best, >-tony > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From angel at wolf.com Thu Sep 4 19:48:37 2003 From: angel at wolf.com (Angel Rivera) Date: Thu, 04 Sep 2003 23:48:37 GMT Subject: Microway Opteron Cluster In-Reply-To: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> References: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> Message-ID: <20030904234837.22385.qmail@houston.wolf.com> I love the angstrom, just be aware that if anything happens you will always be affecting two nodes. lehi.gracia at amd.com writes: > Depending on how small is small you might want to look at Angstrom, for higher density, if realstate is a concern. > http://www.angstrom.com/ > > Lehi > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sgoel at angstrom.com Thu Sep 4 20:00:20 2003 From: sgoel at angstrom.com (Sachin Goel) Date: Thu, 4 Sep 2003 20:00:20 -0400 Subject: Microway Opteron Cluster References: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> <20030904234837.22385.qmail@houston.wolf.com> Message-ID: <027a01c37340$b539a160$ae01a8c0@sachinlaptop> thanks for your appreciation Angel, but our blades are independent of each other if anything fails...so 1 blade failure affects only one blade.... cheers sachin Sachin Goel Angstrom Microsystems 27 Dry Dock Avenue, 8 Floor Boston, MA 02210 Tel: 617-695-0137 Ext 27 Cell: 617-429-2237 Fax: 617-695-0984 www.angstrom.com ----- Original Message ----- From: "Angel Rivera" To: Cc: ; Sent: Thursday, September 04, 2003 7:48 PM Subject: Re: Microway Opteron Cluster > I love the angstrom, just be aware that if anything happens you will always > be affecting two nodes. > > > lehi.gracia at amd.com writes: > > > Depending on how small is small you might want to look at Angstrom, for higher density, if realstate is a concern. > > http://www.angstrom.com/ > > > > Lehi > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From angel at wolf.com Thu Sep 4 21:31:58 2003 From: angel at wolf.com (Angel Rivera) Date: Fri, 05 Sep 2003 01:31:58 GMT Subject: Microway Opteron Cluster In-Reply-To: <027a01c37340$b539a160$ae01a8c0@sachinlaptop> References: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> <20030904234837.22385.qmail@houston.wolf.com> <027a01c37340$b539a160$ae01a8c0@sachinlaptop> Message-ID: <20030905013158.29603.qmail@houston.wolf.com> We run them hard and they are fast. We don't have the blades-will have to see how much money we can get out of them. :: Sachin Goel writes: > thanks for your appreciation Angel, but our blades are independent of each > other if anything fails...so 1 blade failure affects only one blade.... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Thu Sep 4 20:43:54 2003 From: becker at scyld.com (Donald Becker) Date: Thu, 4 Sep 2003 20:43:54 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <3F57BF64.1010008@bellsouth.net> Message-ID: On Thu, 4 Sep 2003, Jeffrey B. Layton wrote: > SMC recently announced a low-cost 8 port GigE switch that can handle > jumbo frames. > > http://www.smc.com/index.cfm?sec=About-SMC&pg=Press-Release-Details&pr_id=129&site=c That's very aggressive pricing for a switch with jumbo frame support. The previous lowest-cost GbE switches have a street price of $160 for 8 ports, but the per-port price goes up dramatically for larger switches. > They also announced GigE NICs for $30. I don't know about Linux > support. If not, here's a website that some some decent information > about copper GigE: My guess is that they are using Realtek rtl8169 chips. The rtl8169 is not as sophisticated as the e1000, but is still a reasonable performer. It's a much better design than the justly-maligned rtl8139. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Sep 4 22:08:12 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Fri, 5 Sep 2003 10:08:12 +0800 (CST) Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <1062699881.452.22.camel@haze.sr.unh.edu> Message-ID: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> If you use SGE, this howto will provide the info: http://gridengine.sunsource.net/project/gridengine/howto/resource.html I hope OpenPBS has similar features, but given that OpenPBS is basically dead (bugs unfixed, no new ports, very light mailing list traffic), is there a reason to keep using it? At least use Scablable PBS or SGE. Andrew. --- Tod Hagan ???? > Hi, > > I'm a guest on a system running OpenPBS where I'd > like to schedule a > bunch of test jobs using OpenPBS. All the jobs will > execute out of the > same directory. Rather than duplicating the > directory for each job, or > using OpenPBS's support for dependency lists and > rigidly specifying the > execution order myself, what I really want is to > declare a > semaphore/mutex locking access to the directory so > only one of these > jobs will run at a time, and then let the batch > system figure out which > job is best to run given the other jobs on the > cluster. > > Is there any way to declare this kind of semaphore > or lock with OpenPBS? > > How about SGE? > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Thu Sep 4 22:16:18 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 5 Sep 2003 12:16:18 +1000 Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> References: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> Message-ID: <200309051216.19692.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 5 Sep 2003 12:08 pm, Andrew Wang wrote: [OpenPBS dead] > very light mailing list traffic ...if you are lucky enough to be allowed onto it. It appears that they are ignoring attempts to subscribe these days (the list requires approval from the maintainers to join). Scalable PBS is certainly the way to go as far as we are concerned. - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/V/HyO2KABBYQAh8RAkCmAKCH1BJRYcE3P3VbEIMiZ/r/7doJtgCgg6qP V/lANRb6qLB4qVcyY5YJjDc= =VVIx -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sandeep at earl.met.fsu.edu Thu Sep 4 23:01:49 2003 From: sandeep at earl.met.fsu.edu (sandeep) Date: Thu, 04 Sep 2003 23:01:49 -0400 Subject: MMINPUT codes Message-ID: <5.2.0.9.2.20030904225959.029ddc10@earl.met.fsu.edu> Dear Beowulf, Myself in need of a code which could able to read the MMINPUT_DOMAIN files if you have the same ,could you kindly help me out . looking forward to hear from you soon. thanking you san _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 5 04:52:12 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 5 Sep 2003 10:52:12 +0200 (CEST) Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <200309051216.19692.csamuel@vpac.org> Message-ID: I'll speak up in favout of Gridengine. I'm learning more about it. The subject of Gridengine on Opteron came up recently, on a thread started by Mikhail. My experience recently was of setting up Gridengine on an Opteron cluster. Nodes run the SuSE 8.2 for x86_64. The 32 bit version of Gridengine seems to run OK. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Fri Sep 5 06:26:19 2003 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Fri, 5 Sep 2003 12:26:19 +0200 (CEST) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <1062712656.2998.12.camel@woody> Message-ID: On 4 Sep 2003, Craig Tierney wrote: > I haven't tried it yet, but the 12 and 24 port gigE switches from > Dell look attractive. The switches provide full bandwidth and have > several other good features. Except the management interface which I find horrible. Well, comparing it with the 3Com SuperStack and BayNetworks switches that I have... Have you tested the bandwidth or is there some published test about this ? I don't have enough GigE nodes to fill one... The documentation implies that there are 2 chips inside each responsible for 12 ports. I wasn't worried about on-chip communication, but inter-chip one. -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sgaudet at wildopensource.com Fri Sep 5 09:03:33 2003 From: sgaudet at wildopensource.com (Stephen Gaudet) Date: Fri, 05 Sep 2003 09:03:33 -0400 Subject: Microway Opteron Cluster In-Reply-To: References: Message-ID: <3F5889A5.4030104@wildopensource.com> Hello Jeffre, Johnson.Jeffre at epamail.epa.gov wrote: > > > > Hello; > > We are contemplating purchasing a small Opteron-based cluster from > Microway, and I am curious as to whether anyone has any thoughts about > Microway and/or Opterons. Thank you. While Microway is a good company, you might want to check out Angstrom Microsystems. http://www.Angstrom.com I just read this article on Angstrom. http://msnbc-cnet.com.com/2100-1010_3-5071318.html?part=msnbc-cnet&tag=alert&form=feed&subj=cnetnews I have a friend that worked there for over a year and to my surprise Angstrom NEVER had one RMA. I never believed it, until I heard it from some of their customers. I might add, Lalit Jain the CEO, graduate from MIT, builds the best AMD solution of all the AMD partners. No one spends more time making sure the box is cool enough and the power is clean. Check them out, can't go wrong. Cheers, Steve Gaudet Wild Open Source (home office) ---------------------- Bedford, NH 03110 pH:603-488-1599 cell:603-498-1600 http://www.wildopensource.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Fri Sep 5 10:11:08 2003 From: deadline at plogic.com (Douglas Eadline) Date: Fri, 5 Sep 2003 10:11:08 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Thu, 4 Sep 2003, Donald Becker wrote: > On Thu, 4 Sep 2003, Jeffrey B. Layton wrote: > > > SMC recently announced a low-cost 8 port GigE switch that can handle > > jumbo frames. > > > > http://www.smc.com/index.cfm?sec=About-SMC&pg=Press-Release-Details&pr_id=129&site=c > > That's very aggressive pricing for a switch with jumbo frame support. > > The previous lowest-cost GbE switches have a street price of $160 for 8 > ports, but the per-port price goes up dramatically for larger switches. > > > They also announced GigE NICs for $30. I don't know about Linux > > support. If not, here's a website that some some decent information > > about copper GigE: > > My guess is that they are using Realtek rtl8169 chips. The rtl8169 is > not as sophisticated as the e1000, but is still a reasonable performer. > It's a much better design than the justly-maligned rtl8139. Based on what I read at: http://www.cs.uni.edu/~gray/gig-over-copper/ (note: read the paper http://www.cs.uni.edu/~gray/gig-over-copper/hsln-lcn.ps) I purchased two Netgear 302T nics for about $30 each. They are 32 bit cards that will run at 66 Mhz, have a Broadcom chipset, and use the tg3 driver. Here as some _preliminary_ results using netpipe(tcp) PCI Speed Latency Max Throughput 33 28 700 Mbits/sec (87.5 Mbytes/sec) 66 30 840 Mbits/sec (105 Mbytes/sec) The tests were done in dual PIII-1.266 Serverworks LE (Supermicro P3TDLE) using the kernel 2.40.20, 1500 byte MTU (they support Jumbos, however) While not as sexy as a PCI-X 64 bit card, at $30 a card, the performance is pretty good. Doug Doug > > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lehi.gracia at amd.com Fri Sep 5 10:48:31 2003 From: lehi.gracia at amd.com (lehi.gracia at amd.com) Date: Fri, 5 Sep 2003 09:48:31 -0500 Subject: Microway Opteron Cluster Message-ID: <99F2150714F93F448942F9A9F112634C07BE6312@txexmtae.amd.com> Hello again Jeffre, >From my prior posting I just wanted is to point out that there are options that you could fit more systems in a rack "if" you need more processors per square foot and "if" you require many systems in your cluster. Since you have been considering already a company perhaps you should contact them and ask them about their options and they may have the best solution for you and/or point you in the right direction. Lehi -----Original Message----- From: Johnson.Jeffre at epamail.epa.gov [mailto:Johnson.Jeffre at epamail.epa.gov] Sent: Thursday, September 04, 2003 3:41 PM To: beowulf at beowulf.org Subject: Microway Opteron Cluster Hello; We are contemplating purchasing a small Opteron-based cluster from Microway, and I am curious as to whether anyone has any thoughts about Microway and/or Opterons. Thank you. Jeffre C. Johnson Research Chemist U.S. EPA, ORD, NERL, HEASD, EDRB (not quite the entire alphabet) Fabulous Las Vegas, Nevada _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Fri Sep 5 11:03:05 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Fri, 05 Sep 2003 17:03:05 +0200 Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> References: <1062699881.452.22.camel@haze.sr.unh.edu> <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> Message-ID: <20030905170305H.hanzl@unknown-domain> >> semaphore/mutex locking access to the directory > If you use SGE, this howto will provide the info: > > http://gridengine.sunsource.net/project/gridengine/howto/resource.html but to create true mutex you want slightly different configuration: - 'consumable' resource, with the overall amount to consume being just '1' - each job needs amount '1' of this resource so this resource is exhausted by the first job requesting it and available again once it finished, and this decision happens in central scheduler so it is a safe mutex. I think this howto is a closer match to this task: http://gridengine.sunsource.net/project/gridengine/howto/consumable.html (It takes some time to get used to GE's system of resources, consumables, various default values etc. but it is well worth it, they are nice building blocks for many useful things.) I would however still worry about coherency of things cached by network filesystem although some slow setups of say NFS are probably nearly safe. I personally think there is one great way of doing this: - Scheduler should know that one job needs to see filesystem changes made by another job (there are job dependencies specified via -hold_jid or there is a 'directory mutex') - Scheduler should be able to ask network filesystem "please propagate cached changes made on node A so as they are visible on node B" This would IMHO solve speed/consistency dilema for many practical purposes. It should be easy to implement the SGE part of this trick but I am not aware of any network filesystem being able to "cache a lot, propagate on demand" as described above. Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brichard at clusterworldexpo.com Fri Sep 5 12:40:35 2003 From: brichard at clusterworldexpo.com (Bryan Richard) Date: Fri, 5 Sep 2003 12:40:35 -0400 Subject: Clusters in the Finance Industry? Message-ID: <20030905164035.GJ79048@clusterworldexpo.com> Hello, I am doing research on HPC Beowulf clusters in the financial sector for the upcoming ClusterWorld and I was hoping the list might be able to help me out. I am interesting in learning more about, * How the finance industry utilizes clusters: risk management, modeling, analysis, &c. * What type of systems are being utilized * What applications are common to this industry * Who are the key individuals in HPC finance Our ultimate goal would be to include a Finance track in the ClusterWorld technical program -- something beyond Deep Green case studies -- that addresses the clustering needs of the HPC finance community. Happy to speak to anyone either on or off the list. Given the tight-lipped nature of this industry, any pointers or assistance would be greatly helpful. Thanks. Be well, Bryan Richard Conference Director QuarterPower Media +(913) 254 0592 voice +(913) 961 3601 mobile brichard at clusterworldexpo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Fri Sep 5 12:26:35 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Fri, 5 Sep 2003 09:26:35 -0700 (PDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Fri, 5 Sep 2003, Bogdan Costescu wrote: > On 4 Sep 2003, Craig Tierney wrote: > > > I haven't tried it yet, but the 12 and 24 port gigE switches from > > Dell look attractive. The switches provide full bandwidth and have > > several other good features. > > Except the management interface which I find horrible. Well, comparing it > with the 3Com SuperStack and BayNetworks switches that I have... The dells have a kind of brain-dead cisco-style cli. if you've used cat-ios you'll mostly be slightly annoyed by the things it doesn't do, if you're not a cat-ios user it'll probably just anoy the heck out of you... The switches themselves are made by accton for what it's worth. > Have you tested the bandwidth or is there some published test about this ? > I don't have enough GigE nodes to fill one... The documentation implies > that there are 2 chips inside each responsible for 12 ports. I wasn't > worried about on-chip communication, but inter-chip one. the 24 port models are two broadcom 5690's connected back to back using the 10Gb/s xaui port on each core. > -- > Bogdan Costescu > > IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen > Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY > Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 > E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tod at gust.sr.unh.edu Fri Sep 5 14:27:09 2003 From: tod at gust.sr.unh.edu (Tod Hagan) Date: 05 Sep 2003 14:27:09 -0400 Subject: MMINPUT codes In-Reply-To: <5.2.0.9.2.20030904225959.029ddc10@earl.met.fsu.edu> References: <5.2.0.9.2.20030904225959.029ddc10@earl.met.fsu.edu> Message-ID: <1062786430.2227.3.camel@haze.sr.unh.edu> On Thu, 2003-09-04 at 23:01, sandeep wrote: > Dear Beowulf, > > Myself in need of a code which could able to read the MMINPUT_DOMAIN files > if you have the same ,could you kindly help me out . Sandeep, You are more likely to get an answer for your question by asking on the mm5-users mailing list: http://mailman.ucar.edu/mailman/listinfo/mm5-users _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 5 14:43:46 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 5 Sep 2003 11:43:46 -0700 Subject: Filesystem In-Reply-To: <200308282056.54106.exa@kablonet.com.tr> References: <200308282056.54106.exa@kablonet.com.tr> Message-ID: <20030905184345.GA1378@greglaptop.internal.keyresearch.com> > I basically think ext3 and ext2 are a joke and we use XFS on the nodes with no > performance problem. Excellent reliability! Filesystem religion is fun, but keep in mind that ext2/3, XFS and Reiserfs are used in production by some pretty big communities -- Redhat, SGI, and Suse are not in the business of shipping software that fails for the common case. ext2 is somewhat annoying in that a big cluster of machines with local disks that has a power failure is likely to have at least one node that asks for a manual fsck. but reports of actal data loss are few and far between. greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mprinkey at aeolusresearch.com Fri Sep 5 13:08:50 2003 From: mprinkey at aeolusresearch.com (Michael T. Prinkey) Date: Fri, 5 Sep 2003 13:08:50 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: > (note: read the paper > http://www.cs.uni.edu/~gray/gig-over-copper/hsln-lcn.ps) > > I purchased two Netgear 302T nics for about $30 each. They are 32 bit > cards that will run at 66 Mhz, have a Broadcom chipset, and use > the tg3 driver. Here as some _preliminary_ results using netpipe(tcp) > > PCI Speed Latency Max Throughput > 33 28 700 Mbits/sec (87.5 Mbytes/sec) > 66 30 840 Mbits/sec (105 Mbytes/sec) > > The tests were done in dual PIII-1.266 Serverworks LE (Supermicro P3TDLE) > using the kernel 2.40.20, 1500 byte MTU (they support Jumbos, however) > While not as sexy as a PCI-X 64 bit card, at $30 a card, the performance > is pretty good. > > Doug > I will second this. I have been using Netgear 302Ts since late last year. I have been using them on dual Xeon boards with on-board Intel Gbit NICs. I run the on-board to a switch and use a pair of Netgears to build nearest neighbor links. It makes for a very cost effective way to speed up CFD runs that use 1D decompositions. The latency and bandwith of these cards are very impressive considering their low cost and 32-bittedness. Here is a tar file with full netpipe results for the onboard Intel thru a switch and two Netgear 302Ts back to back: http://aeolusres.homestead.com/files/np_bench.tar.gz Mike Prinkey Aeolus Research, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eccf at super.unam.mx Fri Sep 5 15:35:44 2003 From: eccf at super.unam.mx (Eduardo Cesar Cabrera Flores) Date: Fri, 5 Sep 2003 14:35:44 -0500 (CDT) Subject: Checkpoint? In-Reply-To: <200309051909.h85J9fw05503@NewBlue.Scyld.com> Message-ID: Is there any good SW to make a checkpoint in a cluster env for serial and parallel (mpi,pvm,OpenMP) application? thanks a lot cafe _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Fri Sep 5 15:53:20 2003 From: raysonlogin at yahoo.com (Rayson Ho) Date: Fri, 5 Sep 2003 12:53:20 -0700 (PDT) Subject: Checkpoint? In-Reply-To: Message-ID: <20030905195320.3145.qmail@web11404.mail.yahoo.com> http://www.checkpointing.org/ Rayson --- Eduardo Cesar Cabrera Flores wrote: > > Is there any good SW to make a checkpoint in a cluster env for serial > and > parallel (mpi,pvm,OpenMP) application? > > thanks a lot > > > cafe > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Fri Sep 5 17:31:36 2003 From: raysonlogin at yahoo.com (Rayson Ho) Date: Fri, 5 Sep 2003 14:31:36 -0700 (PDT) Subject: Mac clusters... Message-ID: <20030905213136.91079.qmail@web11406.mail.yahoo.com> 1100 dual G5s to build a supercomputer: http://www.computerweekly.com/articles/article.asp?liArticleID=124559&liFlavourID=1&sp=1 And this cluster is interesting too: --- "Hunt, Derek" wrote: > Hello all, this is in response to Charles D. Norton's post regarding > Xserve clusters. > > Site: Yale School Of Management > Yale University > Hardware Vendor: Apple Computer > Number of nodes: 45 > Number of processors: 90 > Total Memory: 90GB > Interconnect Technology: Gigabit Ethernet > OS: 10.2.6 with current patches > Comm Software: MPICH, Sun Gridengine > Application Software: GridMathematica, Matlab, C/C++, Fortran > Mail application area Scientific Computing: Financial Research, > Statistical Analysis > Year installed/upgraded: 2003 > > We are in the initial stages of deployment now. > > on a side note, I will be giving a talk on September 25th in > Rochester, > MN at the linux users group (more details at http://www.k-lug.org) if > anyone is interested in hearing the various trials we encountered > during > installation/deployment. > > - Derek __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Sat Sep 6 08:07:12 2003 From: deadline at plogic.com (Douglas Eadline) Date: Sat, 6 Sep 2003 08:07:12 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Fri, 5 Sep 2003, Michael T. Prinkey wrote: > > (note: read the paper > > http://www.cs.uni.edu/~gray/gig-over-copper/hsln-lcn.ps) > > > > I purchased two Netgear 302T nics for about $30 each. They are 32 bit > > cards that will run at 66 Mhz, have a Broadcom chipset, and use > > the tg3 driver. Here as some _preliminary_ results using netpipe(tcp) > > > > PCI Speed Latency Max Throughput > > 33 28 700 Mbits/sec (87.5 Mbytes/sec) > > 66 30 840 Mbits/sec (105 Mbytes/sec) > > > > The tests were done in dual PIII-1.266 Serverworks LE (Supermicro P3TDLE) > > using the kernel 2.40.20, 1500 byte MTU (they support Jumbos, however) > > While not as sexy as a PCI-X 64 bit card, at $30 a card, the performance > > is pretty good. > > > > Doug > > > > I will second this. I have been using Netgear 302Ts since late last year. > I have been using them on dual Xeon boards with on-board Intel Gbit NICs. > I run the on-board to a switch and use a pair of Netgears to build nearest > neighbor links. It makes for a very cost effective way to speed up CFD > runs that use 1D decompositions. The latency and bandwith of these cards > are very impressive considering their low cost and 32-bittedness. > > Here is a tar file with full netpipe results for the onboard Intel thru a > switch and two Netgear 302Ts back to back: > > http://aeolusres.homestead.com/files/np_bench.tar.gz Mike, a few questions, if you do not mind kernel and driver versions, model of switch, motherboard? Have you run the 302t's thought the switch or the intel's as xover? Just curious. Sounds like you are have built this machine to solve a specific type of problem. How is it working? Doug > > Mike Prinkey > Aeolus Research, Inc. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From iosephus at sgirmn.pluri.ucm.es Sat Sep 6 10:48:31 2003 From: iosephus at sgirmn.pluri.ucm.es (=?iso-8859-1?Q?Jos=E9_M=2E_P=E9rez_S=E1nchez?=) Date: Sat, 06 Sep 2003 16:48:31 +0200 Subject: Processor technology choice Message-ID: <20030906144831.GA1100@sgirmn.pluri.ucm.es> Hi people: We are planning to install a small cluster (around 12 processors). We mainly are going to do pattern recognition, and will run applications using artificial neural-network and genetic algorithms. We want to put around 6 dual processor machines running GNU/Linux with OpenMosix, and MPI on top of it. One server with SCSI storage and the rest of the nodes will be diskless. The main design uncertainty we have is what processor technology to use. Our choices are Intel Xeon, AMD Athlon XP/MP and Opteron. We are particularly interested in Opteron even if we have to put a couple of nodes less. My questions are: - Will we have some computing power gain using Opteron with the same budget? - What about long term maintenance cost, for how long will be 32 bit technology components generally available at normal price, now that we have Opteron and Itanium? - Most vendors say it's better to buy Intel (I personally think AMD are equivalent and cheaper), are they just trying to sell the most expensive? Any known real trouble with cooling for AMDs, or just bad case design? - We plan to use Debian, anyone with experience on it for 64 bit platforms. Thanks in advance, Jose M. P?rez. Madrid. Spain. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Sat Sep 6 12:57:17 2003 From: rossini at blindglobe.net (A.J. Rossini) Date: Sat, 06 Sep 2003 09:57:17 -0700 Subject: Processor technology choice In-Reply-To: <20030906144831.GA1100@sgirmn.pluri.ucm.es> (Jos's message of "Sat, 06 Sep 2003 16:48:31 +0200") References: <20030906144831.GA1100@sgirmn.pluri.ucm.es> Message-ID: <851xutonwy.fsf@blindglobe.net> Jos? M. P?rez S?nchez writes: > - We plan to use Debian, anyone with experience on it for 64 bit platforms. With respect to the Opteron, better review the Debian-AMD64 archives. It'll be a while (next year? longer) before it's really up and running and tested (esp for 32/64 combos). Just to make sure it'll be where you want it to be by the time you are ready to roll... best, -tony -- A.J. Rossini rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW : FAX=206-543-3461 | moving soon to a permanent office FHCRC: 206-667-7025 FAX=206-667-4812 | Voicemail is pretty sketchy/use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Sat Sep 6 12:23:25 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Sat, 6 Sep 2003 11:23:25 -0500 (CDT) Subject: Processor technology choice In-Reply-To: <20030906144831.GA1100@sgirmn.pluri.ucm.es> Message-ID: Opteron prices are on par with the Xeon's. I see no reason for anyone, anywhere to ever buy Athlon MP's ever again. The processor core is almost identical to the Opterons except that the Opterons have the 64-bit extensions. The opterons get a great benefit from the Hypertransport bus. They have great memory bandwidth and interprocess latency. Debain for amd64 is still under development. I occasionaly talk with the guys who are doing the port, and I dont think it is quite ready for prime-time yet. If you really know what you are doing, you can probably make it work, but it is not going to be trivial. Debian would be great on Xeon/Athlon/Itainium2 though. the Opteron's will give you a bit less potential raw FLOPS, but you will most likely achieve a greater effiency from those processors (versus Xeon) due to the faster memory bus. The best way is to test your code on each -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' On Sat, 6 Sep 2003, Jos? M. P?rez S?nchez wrote: > Hi people: > > We are planning to install a small cluster (around 12 processors). > We mainly are going to do pattern recognition, and will run > applications using artificial neural-network and genetic algorithms. > > We want to put around 6 dual processor machines running GNU/Linux with > OpenMosix, and MPI on top of it. One server with SCSI storage and the > rest of the nodes will be diskless. > > The main design uncertainty we have is what processor technology to use. > Our choices are Intel Xeon, AMD Athlon XP/MP and Opteron. We are > particularly interested in Opteron even if we have to put a couple of > nodes less. My questions are: > > - Will we have some computing power gain using Opteron with the same > budget? > - What about long term maintenance cost, for how long will be 32 bit > technology components generally available at normal price, now that we > have Opteron and Itanium? > - Most vendors say it's better to buy Intel (I personally think AMD are > equivalent and cheaper), are they just trying to sell the most > expensive? Any known real trouble with cooling for AMDs, or just bad > case design? > - We plan to use Debian, anyone with experience on it for 64 bit platforms. > > Thanks in advance, > > Jose M. P?rez. > Madrid. Spain. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mprinkey at aeolusresearch.com Sat Sep 6 12:30:19 2003 From: mprinkey at aeolusresearch.com (Michael T. Prinkey) Date: Sat, 6 Sep 2003 12:30:19 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Sat, 6 Sep 2003, Douglas Eadline wrote: > Mike, a few questions, if you do not mind kernel and driver versions, > model of switch, motherboard? 2.4.18 for those results, I believe. I saw about the same with 2.4.19. I haven't tested 2.4.20. I used the tg3 drivers as packaged in the kernel. I used the e1000 drivers for the onboards. I think the motherboards were INTEL SE7500CW2. CPUs were 2.4 GHz Xeons. The switch was a DLink DGS1016T. I have built 3 or 4 clusters using similar configurations. More recent ones have used the Tyan Tiger i7501 motherboard instead. > > Have you run the 302t's thought the switch or the intel's > as xover? Just curious. > No. I only had time to benchmark while I was burning the units in and didn't take the time to rewire the network. Wanted to, but didn't find time. > Sounds like you are have built this machine to solve a > specific type of problem. How is it working? > All of the clusters are running a commercial CFD package (the big one!) and are typically used for relatively small runs (2 to 16 CPUs). For these runs, principle axis decomposition is the default and about 98% of the traffic crosses the low-latency nearest neighbor links. For this application, I think such a configuration is near optimal. One day I hope to have a long enough lead time on delivery to actually benchmark with and without the xover links. For now, my only performance metric is anecdotal...I have several satisfied customers. Best, Mike Prinkey Aeolus Research, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sun Sep 7 07:57:50 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sun, 7 Sep 2003 13:57:50 +0200 (CEST) Subject: Processor technology choice In-Reply-To: Message-ID: Re. the Opteron, I'm getting experience now with installing/running them. I've been looking forward to this ever since a talk I heard by Bo Thorsen, then of SusE, a good couple of years ago. Currently running kernel 2.4.22 on them, and Sun Gridengine as the queuing system. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Sun Sep 7 14:39:13 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 07 Sep 2003 13:39:13 -0500 Subject: request for long running parallel applications. In-Reply-To: References: Message-ID: <1062959953.21346.106.camel@terra> On Sun, 2003-09-07 at 12:57, Timothy M Collins wrote: > Dear all, > I need long running parallel application(s) that I can using for testing on > my beowulf cluster. Possibly something that runs in a loop. (small or large) > Please help. Ideas well come. > regards > Collins > Try something like NAMD (http://www.ks.uiuc.edu/Research/namd/) which is a molecular dynamics code. There are benchmarks for it and you can make it run long by bumping up the timesteps of the benchmark. It will eat as much horsepower as you can throw at it. -- -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dmcollins79 at hotmail.com Sun Sep 7 13:57:02 2003 From: dmcollins79 at hotmail.com (Timothy M Collins) Date: Sun, 07 Sep 2003 18:57:02 +0100 Subject: request for long running parallel applications. Message-ID: Dear all, I need long running parallel application(s) that I can using for testing on my beowulf cluster. Possibly something that runs in a loop. (small or large) Please help. Ideas well come. regards Collins http://www.lanideas.com London. N17 _________________________________________________________________ Hotmail messages direct to your mobile phone http://www.msn.co.uk/msnmobile _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Mon Sep 8 00:02:20 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Mon, 8 Sep 2003 00:02:20 -0400 (EDT) Subject: Processor technology choice In-Reply-To: Message-ID: > Debian would be great on Xeon/Athlon/Itainium2 though. and opterons in 32b ("fast xeon") mode, I believe (no prob here with RH.) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From David_Walters at sra.com Mon Sep 8 09:04:40 2003 From: David_Walters at sra.com (Walters, David) Date: Mon, 8 Sep 2003 09:04:40 -0400 Subject: Mac clusters... Message-ID: <0EB5C81FE6FE5A4F8D1FEBF59C6C7BAA0A9FB7@durham.sra.com> Does anyone know where their funding came from? Did they choose Apple based on a cost/benefit analysis, or is Apple paying them? I am thinking back to the numbers that came out of the Cornell Theory Center here... Same scenario? Dave Walters Project Manager/Sr. Management Analyst, EPA IIASC SRA International, Inc. 919-474-4318 david_walters at sra.com -----Original Message----- From: Rayson Ho [mailto:raysonlogin at yahoo.com] Sent: Friday, September 05, 2003 5:32 PM To: beowulf Subject: Mac clusters... 1100 dual G5s to build a supercomputer: http://www.computerweekly.com/articles/article.asp?liArticleID=124559&liFlav ourID=1&sp=1 And this cluster is interesting too: --- "Hunt, Derek" wrote: > Hello all, this is in response to Charles D. Norton's post regarding > Xserve clusters. > > Site: Yale School Of Management > Yale University > Hardware Vendor: Apple Computer > Number of nodes: 45 > Number of processors: 90 > Total Memory: 90GB > Interconnect Technology: Gigabit Ethernet > OS: 10.2.6 with current patches > Comm Software: MPICH, Sun Gridengine > Application Software: GridMathematica, Matlab, C/C++, Fortran Mail > application area Scientific Computing: Financial Research, Statistical > Analysis Year installed/upgraded: 2003 > > We are in the initial stages of deployment now. > > on a side note, I will be giving a talk on September 25th in > Rochester, MN at the linux users group (more details at > http://www.k-lug.org) if anyone is interested in hearing the various > trials we encountered during > installation/deployment. > > - Derek __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Mon Sep 8 11:29:02 2003 From: deadline at plogic.com (Douglas Eadline) Date: Mon, 8 Sep 2003 11:29:02 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Sat, 6 Sep 2003, Michael T. Prinkey wrote: > On Sat, 6 Sep 2003, Douglas Eadline wrote: > > > Mike, a few questions, if you do not mind kernel and driver versions, > > model of switch, motherboard? > > 2.4.18 for those results, I believe. I saw about the same with 2.4.19. > I haven't tested 2.4.20. I used the tg3 drivers as packaged in the > kernel. I used the e1000 drivers for the onboards. I think the > motherboards were INTEL SE7500CW2. CPUs were 2.4 GHz Xeons. The > switch was a DLink DGS1016T. I have built 3 or 4 clusters using similar > configurations. More recent ones have used the Tyan Tiger i7501 > motherboard instead. > > > > > Have you run the 302t's thought the switch or the intel's > > as xover? Just curious. > > > > No. I only had time to benchmark while I was burning the units in and > didn't take the time to rewire the network. Wanted to, but didn't find > time. > > > Sounds like you are have built this machine to solve a > > specific type of problem. How is it working? > > > > All of the clusters are running a commercial CFD package (the big one!) > and are typically used for relatively small runs (2 to 16 CPUs). For > these runs, principle axis decomposition is the default and about 98% of > the traffic crosses the low-latency nearest neighbor links. For this > application, I think such a configuration is near optimal. One day I hope > to have a long enough lead time on delivery to actually benchmark with and > without the xover links. For now, my only performance metric is > anecdotal...I have several satisfied customers. That is what counts. > > Best, > > Mike Prinkey > Aeolus Research, Inc. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rkane at scs.carleton.ca Mon Sep 8 09:58:25 2003 From: rkane at scs.carleton.ca (Robert Kane) Date: Mon, 08 Sep 2003 09:58:25 -0400 Subject: CPUs for a Beowulf Message-ID: <3F5C8B01.5AD8BA4@scs.carleton.ca> Good morning, If anyone doesn't mind, may I ask a few questions. When given a specific application for which a cluster is being built it should be relatively simple to look are the requirements of the problem and the available hardware, and then determine which hardware solution is best for the problem. However, if the cluster is being built as a general purpose cluster for research, things become a bit more difficult, as (as I far as I can tell) there is no one answer. But, if anyone has any insight into the following problems it would be greatly appreciated. 1. Single versus Dual CPUs? Both of these choices have their pros and cons and are each best suited for different types of problems. Given that the cluster will be used for a variety of problems, is there one which would be a better choice? Is there a particular configuration for which the majority of problems will run better? Is there a solution that on average provides more performance per dollar? 2. CPU Type Intel and AMD's new 64-bit processors are finally beginning to become more common it appears. And from what I've seen the benchmarks are rather impressive. However, there seems to be a significant price increase going from previous generation chips (ie Xeon) to the new 64-bit chips. In general is the increased performance worth the money invested, or would a larger number of slower chips be effective cost/performance wise? Apart from the increased electricial, A/C costs of course. Thank you for any information concerning these issues, whether information be answers or links to good resources, Robert Kane _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From canon at nersc.gov Mon Sep 8 11:52:48 2003 From: canon at nersc.gov (canon at nersc.gov) Date: Mon, 08 Sep 2003 08:52:48 -0700 Subject: Processor technology choice In-Reply-To: Message from Rocky McGaugh of "Sat, 06 Sep 2003 11:23:25 CDT." Message-ID: <200309081552.h88FqmLu023230@pookie.nersc.gov> Rocky, Not to nit pick, but I think the first statement in your second paragraph and your last statement are at direct odds with each other. rocky at atipa.com said: > I see no reason for anyone, anywhere to ever buy Athlon MP's ever > again. rocky at atipa.com said: > The best way is to test your code on each When we benchmarked our codes and looked at price performance last month, we still found a slight edge to the Athlon MP's. This was too bad because we REALLY wanted an excuse to buy 70 Opterons. The reason was we made a few assumptions which are valid for our users, but would likely be untrue for others. 1. The codes would not likely be optimized for a specific platform. This would even include good choices of compiler flags. This obviously hurt the Opteron since you want to take advantage of the extra registers and some of the SSE/2 capabilities. Our users are part of a larger collaboration and they often value consistency over performance. 2. The codes don't look like they will pass the 2 GB barrier any time soon. So, while the 64 bit capability is cool, its not need...yet. Also, we are just struggling with how to move past RH 7.x, so moving to a 64 bit enabled version would be tough. This is for compatibility with the other collaborators, not because the transition itself is hard. We benchmarked on systems with dual 2200+ Athlon MP, 2.2 GHz Xeon, and Opteron 240. At the time of the benchmarks, the Opteron 240 was the only reasonably priced Opteron chip. We then assumed we would purchase 2600+, 2.6 GHz Xeon, or the Opteron 240. Even with out the clock adjustments, the Athlon and Xeon's were beating the Opteron. Of course if you turned on the right compiler flags, the Opteron fared much better. In our case the slightly higher clock speeds of the Athlons and Xeons offset the better efficiency of the Opteron. Obviously the faster Opteron's would have probably done much better, but would have cost 50% more per system. I think this will change very soon (if it hasn't already) as the Opteron CPUs drop in price, and I suspect our next big purchase will be Opteron based. So like Rocky said rocky at atipa.com said: > The best way is to test your code on each --Shane ------------------------------------------------------------------------ Shane Canon PSDF Project Lead National Energy Research Scientific Computing Center ------------------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Mon Sep 8 12:30:27 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Mon, 08 Sep 2003 09:30:27 -0700 Subject: CPUs for a Beowulf In-Reply-To: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: <5.2.0.9.2.20030908092157.018a8cf8@mailhost4.jpl.nasa.gov> RGB's book at the Duke Brahma site (I'm sure he'll post the URL) covers some of these tradeoffs.. You've posed an interesting question, because it's in the generic "what's the best way to get lowest dollars per instruction executed" way, but trickier.. It's that sticky word "performance" which is the problem. Wwe can all agree on what dollars mean and how much they are worth now, and in the future. However, performance is different things to different people. Is time worth money? (that is, is there a "wall clock time" as well as a CPU cycles aspect... Older computers are cheaper in terms of executing a particular number of instructions, but consume more support infrastructure (cooling, staff time, etc.) because, if nothing else, they have to run longer...) Is capital cost important, and, are intermediate results of interest.. There's a well known example where you have a "really big problem" that you could either spend some money now, and compute for the next two years, or save your money, wait a year, buy the (twice as fast) computers for the same price and do the computation then, finishing at the same time. Of course, you don't get any results during the first year, and for many applications, the partial results early are used to guide the later work. What's your particular labor/hardware maintenance/capital investment tradeoff.. If you have copious FREE and SKILLED labor, the trade is different... Likewise, if you have a "hard" reliability requirement and can't tolerate partial (or complete) downtime, the trade is different. Cluster computing, in some forms, also lends itself to "stealth, below-the-funding-watchdog-radar" work. You can buy, borrow, collect, etc., CPUs and gradually add them to an ever growing configuration. I notice though, that the whole cluster computing thing is a validated way to work, these days, and anyone with real work to do, and the budget to pay for it, just goes out and builds a real Beowulf. At 09:58 AM 9/8/2003 -0400, Robert Kane wrote: >Good morning, > > If anyone doesn't mind, may I ask a few questions. When given a >specific application for which a cluster is being built it should be >relatively simple to look are the requirements of the problem and the >available hardware, and then determine which hardware solution is best >for the problem. However, if the cluster is being built as a general >purpose cluster for research, things become a bit more difficult, as (as >I far as I can tell) there is no one answer. But, if anyone has any >insight into the following problems it would be greatly appreciated. > >1. Single versus Dual CPUs? > > Both of these choices have their pros and cons and are each best >suited for different types of problems. Given that the cluster will be >used for a variety of problems, is there one which would be a better >choice? Is there a particular configuration for which the majority of >problems will run better? Is there a solution that on average provides >more performance per dollar? > >2. CPU Type > > Intel and AMD's new 64-bit processors are finally beginning to become >more common it appears. And from what I've seen the benchmarks are >rather impressive. However, there seems to be a significant price >increase going from previous generation chips (ie Xeon) to the new >64-bit chips. In general is the increased performance worth the money >invested, or would a larger number of slower chips be effective >cost/performance wise? Apart from the increased electricial, A/C costs >of course. > > >Thank you for any information concerning these issues, whether >information be answers or links to good resources, > >Robert Kane >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Sep 8 13:00:59 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 8 Sep 2003 13:00:59 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <5.2.0.9.2.20030908092157.018a8cf8@mailhost4.jpl.nasa.gov> Message-ID: On Mon, 8 Sep 2003, Jim Lux wrote: > RGB's book at the Duke Brahma site (I'm sure he'll post the URL) covers > some of these tradeoffs.. http://www.phy.duke.edu/brahma But I think that Jim is optimistic that it does any better than his own stuff below. It hasn't been updated to take new architectural improvments into account, and I REALLY need to add more infrastructure stuff there as well. However, other links on the brahma site and my personal home page do address infrastructure at least some, as does an article I wrote for Linux Magazine back in June. Not so much single vs dual per se, but in general. Jim's refocussing your energy on overall infrastructure rather than CPU architecture per se is dead on the money, though. rgb > > You've posed an interesting question, because it's in the generic "what's > the best way to get lowest dollars per instruction executed" way, but > trickier.. > > It's that sticky word "performance" which is the problem. Wwe can all > agree on what dollars mean and how much they are worth now, and in the > future. However, performance is different things to different people. > > Is time worth money? (that is, is there a "wall clock time" as well as a > CPU cycles aspect... Older computers are cheaper in terms of executing a > particular number of instructions, but consume more support infrastructure > (cooling, staff time, etc.) because, if nothing else, they have to run > longer...) > > Is capital cost important, and, are intermediate results of interest.. > There's a well known example where you have a "really big problem" that you > could either spend some money now, and compute for the next two years, or > save your money, wait a year, buy the (twice as fast) computers for the > same price and do the computation then, finishing at the same time. Of > course, you don't get any results during the first year, and for many > applications, the partial results early are used to guide the later work. > > What's your particular labor/hardware maintenance/capital investment > tradeoff.. If you have copious FREE and SKILLED labor, the trade is > different... Likewise, if you have a "hard" reliability requirement and > can't tolerate partial (or complete) downtime, the trade is different. > > > Cluster computing, in some forms, also lends itself to "stealth, > below-the-funding-watchdog-radar" work. You can buy, borrow, collect, > etc., CPUs and gradually add them to an ever growing configuration. I > notice though, that the whole cluster computing thing is a validated way to > work, these days, and anyone with real work to do, and the budget to pay > for it, just goes out and builds a real Beowulf. > > > At 09:58 AM 9/8/2003 -0400, Robert Kane wrote: > >Good morning, > > > > If anyone doesn't mind, may I ask a few questions. When given a > >specific application for which a cluster is being built it should be > >relatively simple to look are the requirements of the problem and the > >available hardware, and then determine which hardware solution is best > >for the problem. However, if the cluster is being built as a general > >purpose cluster for research, things become a bit more difficult, as (as > >I far as I can tell) there is no one answer. But, if anyone has any > >insight into the following problems it would be greatly appreciated. > > > >1. Single versus Dual CPUs? > > > > Both of these choices have their pros and cons and are each best > >suited for different types of problems. Given that the cluster will be > >used for a variety of problems, is there one which would be a better > >choice? Is there a particular configuration for which the majority of > >problems will run better? Is there a solution that on average provides > >more performance per dollar? > > > >2. CPU Type > > > > Intel and AMD's new 64-bit processors are finally beginning to become > >more common it appears. And from what I've seen the benchmarks are > >rather impressive. However, there seems to be a significant price > >increase going from previous generation chips (ie Xeon) to the new > >64-bit chips. In general is the increased performance worth the money > >invested, or would a larger number of slower chips be effective > >cost/performance wise? Apart from the increased electricial, A/C costs > >of course. > > > > > >Thank you for any information concerning these issues, whether > >information be answers or links to good resources, > > > >Robert Kane > >_______________________________________________ > >Beowulf mailing list, Beowulf at beowulf.org > >To change your subscription (digest mode or unsubscribe) visit > >http://www.beowulf.org/mailman/listinfo/beowulf > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Sep 8 12:44:42 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 8 Sep 2003 12:44:42 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: On Mon, 8 Sep 2003, Robert Kane wrote: > Good morning, > > If anyone doesn't mind, may I ask a few questions. When given a > specific application for which a cluster is being built it should be > relatively simple to look are the requirements of the problem and the > available hardware, and then determine which hardware solution is best > for the problem. However, if the cluster is being built as a general > purpose cluster for research, things become a bit more difficult, as (as > I far as I can tell) there is no one answer. But, if anyone has any > insight into the following problems it would be greatly appreciated. > > 1. Single versus Dual CPUs? > > Both of these choices have their pros and cons and are each best > suited for different types of problems. Given that the cluster will be > used for a variety of problems, is there one which would be a better > choice? Is there a particular configuration for which the majority of > problems will run better? Is there a solution that on average provides > more performance per dollar? You're going to get a lot of "it depends" answers because it does. However: a) Historically dual packaging is marginally very slightly cheaper, per CPU, than single packaging. The difference is pretty much the duplicated parts -- two cases instead of one, two power supplies, two hard disks. So if your programs are expected to be purely CPU bound, you'll get a bit more CPU for your dollar in dual packaging. b) OTOH if your task(s) are likely to be memory bound, dual packagings typically oversubscribe the memory bus. That is if both CPUs are trying to read/write to memory as fast as they can, one will often be blocked waiting for the other. c) This latter problem projects down to other resources as well. For example two CPUs might end up sharing a single NIC, or two NICs might end up sharing a bus. Anywhere you have a memory/speed hierarchy that is shared by the CPUs on a dual but that has its own private resource on a single, you can end up with one CPU/task blocking while waiting for the other to free the resource. In many cases the problems described in b and c are minimal or relatively rare. In others they can significantly degrade performance. I think that's the most that can be said without a deeper knowledge of the problem space and the rest of your architecture, e.g. the network, the memory we're talking about, the CPU's, the overall architecture. With both 32 and 64 bit architectures and with a variety of FSB and memory and CPU speeds these days, something that might be a problem with one architecture might be perfectly ok with a different one, so just "single" and "dual" aren't enough data to answer your question, if they ever were. You'll probably have to do some sort of cost benefit analysis with SOME sort of idea of the boundaries of your problem space to proceed. For example, you might select duals knowing they won't scale perfectly for certain problems, because they'll outperform (dollarwise) on the others. Or you might go with singles to get the best scaling on the former, at the expense of the latter. These days the marginal cost difference isn't so great that either path is likely to be a huge win or loss, and it may be other considerations such as CPU density in a rackmount that become more important than performance per se. > 2. CPU Type > > Intel and AMD's new 64-bit processors are finally beginning to become > more common it appears. And from what I've seen the benchmarks are > rather impressive. However, there seems to be a significant price > increase going from previous generation chips (ie Xeon) to the new > 64-bit chips. In general is the increased performance worth the money > invested, or would a larger number of slower chips be effective > cost/performance wise? Apart from the increased electricial, A/C costs > of course. This is plain old impossible to answer without a knowledge of the problem space. Never one to let that stop me, let me pronounce that at the moment AMD's 64 bit chips are probably worth considering, and Intel's are not. Yet. And I could be behind the times on the yet, as well. Note that I say considering. I honestly think that the only way to answer a question like this is to get loaners of two or three candidate systems and run benchmarks. There are also additional costs outside of the raw hardware to consider, in particular a certain degree of weakness in mainline linux distribution support for the 64 bit systems. You're a bit closer to the beta level there and might well end up "paying" a bit extra in screwing-around-with-crap costs administratively to get the full benefit of the CPUs for a while yet. Again, though, the cost differentials are getting so low for AMD's that their 64 bit systems don't look TOO horrible run as 32 bit systems, and the OS situation can only improve. There is also little doubt that 64 bit support will in fact come to pass -- I know some folks that got eaten alive by Alphas when they bought them for their speed and had to deal with it when their OS support more or less evaporated, leaving them struggling and burning admin FTE with simple issues like scalable installation and mandatory security upgrades of important packages. Administrative costs can easily be THE dominant cost in a public cluster of the sort you describe, with benefits that are difficult to quantify. This tends to bias one towards conservative solutions more or less guaranteed to minimize human management time, which in turn biases one towards older "guaranteed to work" hardware, straight off the shelf commercial linux distros (or a supported cluster package like Scyld at moderate expense in dollars but presumed savings in human time), hardware from a vendor that does on site service as part of the up-front cost, and things like PXE-capable NICs, kickstart, yum. 64 bit systems would, I think, require a bit more human effort and skill in the administrative chain to make work. rgb > > > Thank you for any information concerning these issues, whether > information be answers or links to good resources, > > Robert Kane > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Sep 8 14:18:17 2003 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 8 Sep 2003 20:18:17 +0200 (CEST) Subject: CPUs for a Beowulf In-Reply-To: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: On Mon, 8 Sep 2003, Robert Kane wrote: > 2. CPU Type > > 64-bit chips. In general is the increased performance worth the money > invested, or would a larger number of slower chips be effective > cost/performance wise? Hmmmmm. Say, didn't a bunch of folks at a NASA lab do that a few years ago? Big :-) as I am slightly pulling your leg, but hopefully in a nice wayB. That is the beauty of the Beowulf idea - use cheap as chips stuff. Your point is well made. Seriously though, you might find that 64bit AMDs aren't that expensive. I think as somone pointed out recently that requiring large amounts of RAM starts to dominate the costs these days. Very much IMHO. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From iosephus at sgirmn.pluri.ucm.es Mon Sep 8 16:20:22 2003 From: iosephus at sgirmn.pluri.ucm.es (=?iso-8859-1?Q?Jos=E9_M=2E_P=E9rez_S=E1nchez?=) Date: Mon, 08 Sep 2003 22:20:22 +0200 Subject: SCSI Card Message-ID: <20030908202022.GC12233@sgirmn.pluri.ucm.es> Hi, Has anyone tested the UW-320 SCSI card Adaptec 29320-R? The strongest candidate to be purchased as cluster master node for our cluster comes with that card and 10000 RPM Fujitsu hard disks. I took a look at the Adaptec page and they say it's not RAID capable on linux. Shouldn't be hardware controlled RAID OS independent, as long as you have support for the card? What's the driver module for that card? AHA-something, AIC-something? Regards, Jose M. Perez. Madrid. Spain. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Sep 8 17:00:40 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 8 Sep 2003 17:00:40 -0400 (EDT) Subject: SCSI Card In-Reply-To: <20030908202022.GC12233@sgirmn.pluri.ucm.es> Message-ID: On Mon, 8 Sep 2003, [iso-8859-1] Jos? M. P?rez S?nchez wrote: > Hi, > > Has anyone tested the UW-320 SCSI card Adaptec 29320-R? The strongest > candidate to be purchased as cluster master node for our cluster comes with > that card and 10000 RPM Fujitsu hard disks. > > I took a look at the Adaptec page and they say it's not RAID capable on > linux. Shouldn't be hardware controlled RAID OS independent, as long as > you have support for the card? md (software) raid should work on anything, and works remarkably well on most things. We use it in large and small scale production and it performs like a champion. I've run a system six months in degraded mode at home (couldn't find time to go get a replacement disk) and still lost no data. RH's Kickstart can even kickstart install a ready-to-run md raid. As long as the card itself works under linux, building a RAID should be no problem. Hardware RAIDs often do require drivers (or that you work the RAID through a BIOS layer preboot) and leave you at the mercy of the RAID firmware designers as far as function and recovery are concerned. Adaptec (after a somewhat rocky start some years ago) has been pretty good about linux support so I'd guess drivers will be rapidly forthcoming, unless they are trying to pull the "proprietary" trick with a binary only driver (a really, really bad idea with something like a RAID controller). rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Mon Sep 8 17:51:17 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Mon, 8 Sep 2003 14:51:17 -0700 (PDT) Subject: SCSI Card In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Robert G. Brown wrote: > On Mon, 8 Sep 2003, [iso-8859-1] Jos? M. P?rez S?nchez wrote: > > > Hi, > > > > Has anyone tested the UW-320 SCSI card Adaptec 29320-R? The strongest > > candidate to be purchased as cluster master node for our cluster comes with > > that card and 10000 RPM Fujitsu hard disks. the 29320-r comes with a driver for windows that provides what they call hostraid support. something like an adaptec 2120 or 2200s is really a hardware raid controller with a full feature-set (uses aacraid rather than aic7xxx). linux will support this card but not it's partial raid functionality. linux software raid will happily work on top of all of this and probably run faster on a raid5 stripe than the 2120 to boot. the 29320-r doens't support raid-5 at all reagrdless of os. > > I took a look at the Adaptec page and they say it's not RAID capable on > > linux. Shouldn't be hardware controlled RAID OS independent, as long as > > you have support for the card? > > md (software) raid should work on anything, and works remarkably well on > most things. We use it in large and small scale production and it > performs like a champion. I've run a system six months in degraded mode > at home (couldn't find time to go get a replacement disk) and still lost > no data. RH's Kickstart can even kickstart install a ready-to-run md > raid. As long as the card itself works under linux, building a RAID > should be no problem. > > Hardware RAIDs often do require drivers (or that you work the RAID > through a BIOS layer preboot) and leave you at the mercy of the RAID > firmware designers as far as function and recovery are concerned. > Adaptec (after a somewhat rocky start some years ago) has been pretty > good about linux support so I'd guess drivers will be rapidly > forthcoming, unless they are trying to pull the "proprietary" trick with > a binary only driver (a really, really bad idea with something like a > RAID controller). > > rgb > > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 8 19:09:46 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 9 Sep 2003 09:09:46 +1000 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <200309090909.47342.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > Seriously though, you might find that 64bit AMDs aren't that expensive. My understanding is that memory bandwidth with Opterons scales with the number of CPUs due to the HyperTransport architecture, which could be a big plus to some people. I'd be interested to hear whether that turns out to be correct or not in practice! Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/XQw6O2KABBYQAh8RAo1DAJwIIoMRQEySoCGAPbjV8ZAsAR2FdQCfbUCh MSrYfESYyD7o9PEOFbsStl8= =thrl -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Mon Sep 8 21:31:30 2003 From: bill at math.ucdavis.edu (Bill Broadley) Date: Mon, 8 Sep 2003 18:31:30 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> References: <200309090909.47342.csamuel@vpac.org> Message-ID: <20030909013130.GA29417@sphere.math.ucdavis.edu> On Tue, Sep 09, 2003 at 09:09:46AM +1000, Chris Samuel wrote: > On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > > > Seriously though, you might find that 64bit AMDs aren't that expensive. > > My understanding is that memory bandwidth with Opterons scales with the number > of CPUs due to the HyperTransport architecture, which could be a big plus to > some people. This is true. > I'd be interested to hear whether that turns out to be correct or not in > practice! http://www.math.ucdavis.edu/~bill/dual-240-4xpc2700-icc.png http://www.math.ucdavis.edu/~bill/4x842-icc.png I suspect if I used the latest kernel tweaks to pin processes to a specific cpu and allocate only local memory the scaling would be even better. Oh, to compare to a P4 (or dual athlon): http://www.math.ucdavis.edu/~bill/dual-p4-2.4-icc.png -- Bill Broadley Mathematics UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Mon Sep 8 21:44:49 2003 From: becker at scyld.com (Donald Becker) Date: Mon, 8 Sep 2003 21:44:49 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> Message-ID: On Tue, 9 Sep 2003, Chris Samuel wrote: > On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > > > Seriously though, you might find that 64bit AMDs aren't that expensive. > > My understanding is that memory bandwidth with Opterons scales with > the number of CPUs due to the HyperTransport architecture, which > could be a big plus to some people. Hmmm, I would describe this a little differently. Each Opteron processor has an on-die memory controller and full bandwidth to its local memory banks. The only external connection is through Hyper Transport (HT) channels. If you want more memory banks, and more bandwidth, you have to add more memory controllers through HT. And for now, the only way to get another memory controller is to buy the one on the same die as... another Opteron. Again HyperTransport is the only way to connect the Opteron to other processors and, I/O busses Only some of the HT channels can connect CPUs -- this first type is more complex than the I/O-only HT channels. Note that the additional CPUs/memory controllers can only attach to the "better" HT channels. Now for the software issue. For a few processors, the HT-based cache coherency is fast enough to be ignored. But for larger systems the only way avoid a big latency hit from the coherency traffic is to minimize the coherency traffic by having only the applications utilize it. That means running the kernel from local memory, sharing only the locks and data structures needed for global I/O. Gee, suddenly the system starts looking like a cluster. Sure, the application can now use global shared memory, and it's now efficient to implement a single file system view. But all the approaches to making a cluster efficiently scalable can be directly applied to this type of SMP... -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 8 22:34:53 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 9 Sep 2003 12:34:53 +1000 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> References: <200309090909.47342.csamuel@vpac.org> Message-ID: <200309091234.59961.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 9 Sep 2003 09:09 am, Chris Samuel wrote: > I'd be interested to hear whether that turns out to be correct or not in > practice! Bill, Donald, Thanks very much for that useful feedback, much appreciated! Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/XTxSO2KABBYQAh8RAkgDAJ4nW2cOF4x0jKKc9M9wH84pGNSd5ACghg/i v1y+exu3FLkJd8YoblzIeNE= =ql/2 -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jim at ks.uiuc.edu Mon Sep 8 22:47:42 2003 From: jim at ks.uiuc.edu (Jim Phillips) Date: Mon, 8 Sep 2003 21:47:42 -0500 (CDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Donald Becker wrote: > For a few processors, the HT-based cache coherency is fast enough to be > ignored. But for larger systems the only way avoid a big latency hit > from the coherency traffic is to minimize the coherency traffic by > having only the applications utilize it. That means running the kernel > from local memory, sharing only the locks and data structures needed for > global I/O. > > Gee, suddenly the system starts looking like a cluster. Sure, the > application can now use global shared memory, and it's now efficient to > implement a single file system view. But all the approaches to making a > cluster efficiently scalable can be directly applied to this type of > SMP... If you could connect those HyperTransport channels into a scalable mesh then this sounds a lot like the good old Cray T3E design. Drool... It's too bad that Red Storm won't be taking that route. The T3E was so nice. -Jim _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Tue Sep 9 00:02:50 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Tue, 09 Sep 2003 00:02:50 -0400 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <1063080170.10843.108.camel@protein.scalableinformatics.com> One of the interesting aspects of the HT is that it makes it a good building block for bigger systems. Of course you have to be careful what you ask for ... The HT design reminds me of the old SGI XBOW internal fabric. Also formerly known by the (marketing/misleading) label Cray-link. Now called NUMA-link or something like that. On Mon, 2003-09-08 at 22:47, Jim Phillips wrote: > On Mon, 8 Sep 2003, Donald Becker wrote: > > > For a few processors, the HT-based cache coherency is fast enough to be > > ignored. But for larger systems the only way avoid a big latency hit > > from the coherency traffic is to minimize the coherency traffic by > > having only the applications utilize it. That means running the kernel > > from local memory, sharing only the locks and data structures needed for > > global I/O. > > > > Gee, suddenly the system starts looking like a cluster. Sure, the > > application can now use global shared memory, and it's now efficient to > > implement a single file system view. But all the approaches to making a > > cluster efficiently scalable can be directly applied to this type of > > SMP... I remember the early days of trying to get applications to scale on SGI Origins. The issues that Don alludes to here were largely present there. Some of the big issues I remember were memory hotspot identification, page migration thrashing, and a host of other fun items. Customers complained when applications took non-deterministic time to execute due to odd physical->virtual page layouts. Sure, go-ahead and malloc that 20 GB. Hope it doesnt all land on 10 processors... I remember playing with DPLACE directives on benchmarks to tweak this stuff. > If you could connect those HyperTransport channels into a scalable mesh > then this sounds a lot like the good old Cray T3E design. Drool... It's > too bad that Red Storm won't be taking that route. The T3E was so nice. You could build inexpensive Origin's or similar out of these things. You just need the OS/compiler support. That is unfortunately not a trivial thing. > > -Jim -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 9 01:06:28 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 9 Sep 2003 01:06:28 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Jim Phillips wrote: > > Gee, suddenly the system starts looking like a cluster. Sure, the > > application can now use global shared memory, and it's now efficient to > > implement a single file system view. But all the approaches to making a > > cluster efficiently scalable can be directly applied to this type of > > SMP... > > If you could connect those HyperTransport channels into a scalable mesh > then this sounds a lot like the good old Cray T3E design. Drool... It's > too bad that Red Storm won't be taking that route. The T3E was so nice. For those unfamiliar with the history and relationship The Cray T3D/T3E used Alphas, EV4 and EV5 generations HT was designed as the communication architecture for the EV7+ generation Alpha processor. AMD bought the HT technology from API, Alpha Processor Inc., later API Networks just before shutting down. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Tue Sep 9 02:37:35 2003 From: john.hearns at clustervision.com (John Hearns) Date: Tue, 9 Sep 2003 08:37:35 +0200 (CEST) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> Message-ID: On Tue, 9 Sep 2003, Chris Samuel wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > > > Seriously though, you might find that 64bit AMDs aren't that expensive. > > My understanding is that memory bandwidth with Opterons scales with the number > of CPUs due to the HyperTransport architecture, which could be a big plus to > some people. > Not really an answer to your question, but I found this article on the Hammer architecture (from DEcember last year). http://www.digit-life.com/articles2/amd-hammer-family/index.html I think it is good - has discussionof of memory timings, caches, HyperTransport etc. Worth a read. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Tue Sep 9 02:43:31 2003 From: john.hearns at clustervision.com (John Hearns) Date: Tue, 9 Sep 2003 08:43:31 +0200 (CEST) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Jim Phillips wrote: > On Mon, 8 Sep 2003, Donald Becker wrote: > > If you could connect those HyperTransport channels into a scalable mesh > then this sounds a lot like the good old Cray T3E design. Drool... It's > too bad that Red Storm won't be taking that route. The T3E was so nice. The article I quoted from Digit Life floats the idea that AMD may be able to make for Cray an Opteron with 4 Hypertransports, making a 3D mesh possible. Is this just supposition? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Tue Sep 9 04:31:38 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Tue, 09 Sep 2003 10:31:38 +0200 Subject: CPUs for a Beowulf In-Reply-To: References: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: <20030909103138Q.hanzl@unknown-domain> > Administrative costs can easily be THE dominant cost Yes, yes, yes. Anybody has different experience for his/her FIRST cluster? (If somebody does, he is genius and his labor has much higher value then estimated and I am right anyway :-) One year later the price of your hardware may be say half the price today. Divide this halve by 365 to get an idea how much costs each day of problems. Consider appropriate cost of human resources to counterfight this loss... If you want to get most power for the money you have, make it simple, do choices which make the system simple for you. If you want to make research how OTHERS could save, that is another story... (And that is what many people sometimes unwillingly do on this list, including myself :) Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Tue Sep 9 09:15:39 2003 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Tue, 9 Sep 2003 10:15:39 -0300 (ART) Subject: SSH is default(SuSE 8.2) Message-ID: <20030909131539.50851.qmail@web12206.mail.yahoo.com> I install the SUSE 8.2 distro with the mpich package that came with it. I made all the configurantion to use rsh, but when run a example application, it try to use ssh. What can I do to change it. _______________________________________________________________________ Desafio AntiZona: participe do jogo de perguntas e respostas que vai dar um Renault Clio, computadores, c?meras digitais, videogames e muito mais! www.cade.com.br/antizona _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Tue Sep 9 09:27:46 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Tue, 9 Sep 2003 09:27:46 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: > > If you could connect those HyperTransport channels into a scalable mesh > > then this sounds a lot like the good old Cray T3E design. Drool... It's > > too bad that Red Storm won't be taking that route. The T3E was so nice. > > The article I quoted from Digit Life floats the idea that AMD may be able > to make for Cray an Opteron with 4 Hypertransports, making a 3D mesh > possible. Is this just supposition? it would be astonishing if AMD hadn't prototyped this already. one sticking point is that the current architecture is based on broadcast for managing coherence. a big-scalable approach would need to be directory-based, perhaps assisted by the OS. I dimly recall that the current packet format has just three bits for node ID. I have an even more dim recollection that current HT is limited to 20cm; perhaps that would be enough for a non-torus mesh. but, as Dell points out, the market for such things is rather small. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rbw at ahpcrc.org Tue Sep 9 10:48:18 2003 From: rbw at ahpcrc.org (Richard Walsh) Date: Tue, 9 Sep 2003 09:48:18 -0500 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) Message-ID: <200309091448.h89EmIQ14373@mycroft.ahpcrc.org> On Tue Sep 9 2003, John Hearns wrote: >On Mon, 8 Sep 2003, Jim Phillips wrote: > >> On Mon, 8 Sep 2003, Donald Becker wrote: >> >> If you could connect those HyperTransport channels into a scalable mesh >> then this sounds a lot like the good old Cray T3E design. Drool... It's >> too bad that Red Storm won't be taking that route. The T3E was so nice. > >The article I quoted from Digit Life floats the idea that AMD may be able >to make for Cray an Opteron with 4 Hypertransports, making a 3D mesh >possible. Is this just supposition? Mmm ... with just 3 HT links per chip (one of which is attenutated .. IO) I don't see how you can do a full 2D torus let alone a 3D which requires 6 per node. Perhaps mesh here does not refer to the T3E's torus interconnect? The 8-way SMP Opteron layouts that I have seen have edges. Not withstanding this, Cray is custom engineering an MPP system with the Opteron as the base microprocessor and a proprietary interconnect for Sandia(?). Anyone have more detail on this machine? Regards, rbw #--------------------------------------------------- # Richard Walsh # Project Manager, Cluster Computing, Computational # Chemistry and Finance # netASPx, Inc. # 1200 Washington Ave. So. # Minneapolis, MN 55415 # VOX: 612-337-3467 # FAX: 612-337-3400 # EMAIL: rbw at networkcs.com, richard.walsh at netaspx.com # rbw at ahpcrc.org # #--------------------------------------------------- # "Why waste time learning when ignornace is # instantaneous?" -Thomas Hobbes #--------------------------------------------------- # "Life is lived between two opposing poles, the one # defines that which must be true (truism) and the # other that which is true (truth). -Max Headroom #--------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 9 12:06:40 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 9 Sep 2003 12:06:40 -0400 (EDT) Subject: Baltimore-Washington Beowulf Users Group meeting Sept. 9, 2003 Message-ID: http://BWBUG.org The September BWBUG meeting will be at the Northrop Grumman location: 7501 Greenway Center Drive, Suite 1000 Greenbelt , Maryland. on 9 Sept 2003 at 3:00 PM. Note that the fall series is continuing the spring series meeting pattern. We will be alternating venues between Greenbelt and McLean. This month's meeting is in GREENBELT. Steve Duchene Beowulf, design expert at Freddie Mac, will be our quest speaker to discuss and to demonstrate the Oscar Cluster Management Software. Steve has spoken at a number of Beowulf Cluster events such as the Large Scale Cluster Computing Workshop" at FERMILAB on May 2001. Steve was the key participate as the Atlanta Enthusiasts Representative at the 1997 Linux Expo in Atlanta. Steve was the principle design engineer at VA Linux responsible for their Beowulf Clusters. Mike Fitzmaurice http://BWBUG.org 703-205-3132 -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Tue Sep 9 13:36:34 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Tue, 9 Sep 2003 10:36:34 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: <200309090909.47342.csamuel@vpac.org> Message-ID: <20030909173634.GA2341@greglaptop.internal.keyresearch.com> While we're reinventing the T3E, it's worth noting: 1) HT's current generation doesn't have ECC, it just has a CRC-check and crashes when the check fails. That's kind-of OK for small systems, but doesn't scale. 2) AMD's cache coherence protocol is snoopy, which doesn't scale. I'd never heard the rumor about HT being the EV7+ bus (and I'd really doubt it due to (1)), but I do know that AMD only bought the API Networks technical team, not their technology. Digital/Compaq did have a long-term technical collaboration with Samsung and AMD regarding the EV6 bus. And yes, using the Linux NUMA memory placement stuff is a substantial improvement in Opteron performance for real apps. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Tue Sep 9 13:41:33 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Tue, 9 Sep 2003 10:41:33 -0700 Subject: CPUs for a Beowulf In-Reply-To: <20030909103138Q.hanzl@unknown-domain> References: <3F5C8B01.5AD8BA4@scs.carleton.ca> <20030909103138Q.hanzl@unknown-domain> Message-ID: <20030909174133.GB2341@greglaptop.internal.keyresearch.com> On Tue, Sep 09, 2003 at 10:31:38AM +0200, hanzl at noel.feld.cvut.cz wrote: > > Administrative costs can easily be THE dominant cost > > Yes, yes, yes. Anybody has different experience for his/her FIRST > cluster? (If somebody does, he is genius and his labor has much higher > value then estimated and I am right anyway :-) Clusters built by people who already have experience administering large numbers of machines tend to be pretty cost-efficient on the admin side. There isn't much crossover between the "Beowful" community and the LISA community, but there should be. My first non-small HPC cluster was 64 nodes, and it took 28 hours of labor to set up, of which only 8 hours was my time, and 20 hours was less experienced people wielding screwdrivers. Let's see, if you think $100 per node is a reasonable setup fee, and the 20 hours cost $10 each, then I was worth $775/hour. :-) (No, I'm not serious.) -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Tue Sep 9 16:53:43 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Tue, 9 Sep 2003 13:53:43 -0700 Subject: CPUs for a Beowulf In-Reply-To: <20030909174133.GB2341@greglaptop.internal.keyresearch.com> References: <3F5C8B01.5AD8BA4@scs.carleton.ca> <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> Message-ID: <20030909205343.GA3316@greglaptop.internal.keyresearch.com> On Tue, Sep 09, 2003 at 10:41:33AM -0700, Greg Lindahl wrote: > "Beowful" community Old English is weird, but not *that* weird... hm. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Tue Sep 9 19:01:26 2003 From: csamuel at vpac.org (Chris Samuel) Date: Wed, 10 Sep 2003 09:01:26 +1000 Subject: CPUs for a Beowulf In-Reply-To: <20030909205343.GA3316@greglaptop.internal.keyresearch.com> References: <3F5C8B01.5AD8BA4@scs.carleton.ca> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030909205343.GA3316@greglaptop.internal.keyresearch.com> Message-ID: <200309100901.29492.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 10 Sep 2003 06:53 am, Greg Lindahl wrote: > On Tue, Sep 09, 2003 at 10:41:33AM -0700, Greg Lindahl wrote: > > "Beowful" community > > Old English is weird, but not *that* weird... hm. I think you've just invented a new word. :-) Beowful: To use (or be full of) Beowulf clusters. Q: Do you use Linux clusters ? A: Yes, we're Beowful. :-) - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/XlvGO2KABBYQAh8RAsdTAKCYYiBqfO/qDYLu0KgFuQQ4vzlqSwCfaw0z 27RV6USmLJLESnLz6xYV9+o= =blp9 -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From timm at fnal.gov Tue Sep 9 22:34:41 2003 From: timm at fnal.gov (Steven Timm) Date: Tue, 09 Sep 2003 21:34:41 -0500 Subject: CPUs for a Beowulf In-Reply-To: <200309100901.29492.csamuel@vpac.org> Message-ID: > > Beowful: To use (or be full of) Beowulf clusters. > > Q: Do you use Linux clusters ? > A: Yes, we're Beowful. > I thought it was a cross between Beowulf + Awful. :) Steve _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 9 22:20:04 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 9 Sep 2003 22:20:04 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <20030909173634.GA2341@greglaptop.internal.keyresearch.com> Message-ID: On Tue, 9 Sep 2003, Greg Lindahl wrote: > While we're reinventing the T3E, it's worth noting: > > 1) HT's current generation doesn't have ECC, it just has a CRC-check > and crashes when the check fails. I think that it's more precisely a longitudinal check. "CRC" usually means a polynomial calculation over all of the bits, where any bit flip might change any bit in the final check word. A longitudinal-only check means that a bit flip only impacts the check word in that bit position. Longitudinal checks are much easier to implement in very high speed systems because you don't have to handle data skew combined with different length logic paths. But they catch fewer errors precisely because they are easier to implement -- they don't combine as many source bits into the result. > That's kind-of OK for small systems, > but doesn't scale. Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and correct most multibit errors, and most HT errors will be multibit. It's better to fail on corruption than to silently further corrupt. > 2) AMD's cache coherence protocol is snoopy, which doesn't scale. That's a point often missed: in order to implement a HT switch for a SMP system, you need to implement something like a cache directory. > I'd never heard the rumor about HT being the EV7+ bus (and I'd really > doubt it due to (1)), but I do know that AMD only bought the API > Networks technical team, not their technology. Digital/Compaq did have > a long-term technical collaboration with Samsung and AMD regarding the > EV6 bus. Remeber the goal of a single commodity motherboard supporting either an AMD x86 chip or an Alpha? Look where we are today... -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kdunder at sandia.gov Tue Sep 9 23:31:54 2003 From: kdunder at sandia.gov (Keith D. Underwood) Date: 09 Sep 2003 21:31:54 -0600 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <1063164714.18381.27.camel@thinkpad> > I think that it's more precisely a longitudinal check. "CRC" usually > means a polynomial calculation over all of the bits, where any bit flip > might change any bit in the final check word. A longitudinal-only check > means that a bit flip only impacts the check word in that bit position. > > Longitudinal checks are much easier to implement in very high speed > systems because you don't have to handle data skew combined with > different length logic paths. But they catch fewer errors precisely > because they are easier to implement -- they don't combine as many > source bits into the result. As I read the spec, it's actually a little weirder than that: "A 32-bit cyclic redundancy code (CRC) covers all HyperTransportTM links. The CRC is calculated on each 8-bit lane independently and covers the link as a whole, not individual packets." So, for a 16 bit link (such as on an Opteron), there are two concurrently running CRC's, one each for the two 8 bit lanes. Now, the strange part is that the CRC's cover the link and not the packets. So, a CRC is transmitted every 512 bit times (and hypertransport packets aren't that big). That means that you don't know which packets had a bad bit. > > That's kind-of OK for small systems, > > but doesn't scale. > > Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and > correct most multibit errors, and most HT errors will be multibit. > It's better to fail on corruption than to silently further corrupt. The problem is that there is no sane mechanism to know which packets are corrupted (and to therefore retransmit). At scale, that doesn't really work. e.g. if you built a Red Storm scale system using just these links, it would crash frequently because a CRC error would happen somewhere and there wouldn't be a recovery mechanism. (BTW, for those suggesting mimicking the T3E with these things, the T3E wasn't cache coherent. It just had a shared address space mechanism of sorts.) Someone asked something about Red Storm - here is a public link that includes public presentations on the topic: http://www.cs.sandia.gov/presentations/ Keith _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 10 03:01:34 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 10 Sep 2003 00:01:34 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: <20030909173634.GA2341@greglaptop.internal.keyresearch.com> Message-ID: <20030910070134.GA1731@greglaptop.greghome.keyresearch.com> On Tue, Sep 09, 2003 at 10:20:04PM -0400, Donald Becker wrote: > > That's kind-of OK for small systems, > > but doesn't scale. > > Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and > correct most multibit errors, and most HT errors will be multibit. > It's better to fail on corruption than to silently further corrupt. Well, I said "kind of OK" because you won't notice the failures on small systems until you have a bunch of them. That's not really acceptable, but then again lots of people seem willing to bet their data on systems where they haven't really thought about errors at all. I was speaking loosely when I said ECC. The CRC being used will (in theory) catch most multi-bit errors, but it's always scary to not know the pattern of the errors when you chose the CRC. The system does crash quickly enough that it's unlikely that your bad data makes it onto a network or disk. I get a kick out of asking link layer people, "So, what bad packets have you observed in the wild?" They give me the blankest looks... > Remeber the goal of a single commodity motherboard supporting either an > AMD x86 chip or an Alpha? Look where we are today... It was a great idea until the mechnical problems nuked it: Slot A vs. Socket A. I was not impressed by the reliability of Slot A. Sometimes it's the little things that cause the biggest problems. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joachim at ccrl-nece.de Wed Sep 10 03:39:26 2003 From: joachim at ccrl-nece.de (Joachim Worringen) Date: Wed, 10 Sep 2003 09:39:26 +0200 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <200309100939.26394.joachim@ccrl-nece.de> Donald Becker: > > 2) AMD's cache coherence protocol is snoopy, which doesn't scale. > > That's a point often missed: in order to implement a HT switch for a SMP > system, you need to implement something like a cache directory. Yes, but only if you really want SMP characteristics (sequential consistency). This is not mandatory for a HPC system if you can live with some kind of message passing programming model (MPI). Joachim -- Joachim Worringen - NEC C&C research lab St.Augustin fon +49-2241-9252.20 - fax .99 - http://www.ccrl-nece.de _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Wed Sep 10 04:58:48 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Wed, 10 Sep 2003 10:58:48 +0200 Subject: CPUs for a Beowulf In-Reply-To: <20030909174133.GB2341@greglaptop.internal.keyresearch.com> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> Message-ID: <20030910105848W.hanzl@unknown-domain> > > > Administrative costs can easily be THE dominant cost > > > > Yes, yes, yes. Anybody has different experience for his/her FIRST > > cluster? (If somebody does, he is genius and his labor has much higher > > value then estimated and I am right anyway :-) > > My first non-small HPC cluster was 64 nodes, and it took 28 hours of > labor to set up, of which only 8 hours was my time And the first small one? (I basically wanted to scare people who are just building the very first cluster and want to be very innovative at the same time - it is fine and a lot of fun but quite likely it is not a way to save yet. Maybe innovative soul can save a lot on the second one...) Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Wed Sep 10 09:56:57 2003 From: becker at scyld.com (Donald Becker) Date: Wed, 10 Sep 2003 09:56:57 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <1063164714.18381.27.camel@thinkpad> Message-ID: On 9 Sep 2003, Keith D. Underwood wrote: > > I think that it's more precisely a longitudinal check. "CRC" usually ... > > Longitudinal checks are much easier to implement in very high speed . > As I read the spec, it's actually a little weirder than that: > > "A 32-bit cyclic redundancy code (CRC) covers all HyperTransportTM > links. The CRC is calculated on each 8-bit lane independently and > covers the link as a whole, not individual packets." Ahhh, that implies that it uses a group longitudinal check. That has much better checking than a serial single bit check, and by limiting the width the reduce the skew and logic path problem of a wider check. > So, for a 16 bit link (such as on an Opteron), there are two > concurrently running CRC's, one each for the two 8 bit lanes. Now, the > strange part is that the CRC's cover the link and not the packets. So, > a CRC is transmitted every 512 bit times (and hypertransport packets > aren't that big). That means that you don't know which packets had a > bad bit. That's an excellent design decision. Putting a check word on each packet means - the physical encoding layer need to know about packetization - a packet must be held until the check passes - the tiny packets grow - to do anything with the per-packet info, packet copies must be kept These all add complexity and latency to the highest speed path. By putting the check on fixed block boundaries you can still detect and fail an unreliable link > > > That's kind-of OK for small systems, but doesn't scale. > > Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and > > correct most multibit errors, and most HT errors will be multibit. I'll repeat: ECC Bad! ECC Slow! > corrupted (and to therefore retransmit). At scale, that doesn't really > work. e.g. if you built a Red Storm scale system using just these > links, it would crash frequently because a CRC error would happen > somewhere and there wouldn't be a recovery mechanism. If you are getting CRC errors, you very likely have errors that ECC (*) would silently pass or further corrupt. * Any high-speed ECC implementation. It's possible to keep adding check bits, but anything past SECDED Single Error Correction, Double Error Detection becomes time consuming and expensive. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kdunder at sandia.gov Wed Sep 10 10:18:14 2003 From: kdunder at sandia.gov (Keith D. Underwood) Date: 10 Sep 2003 08:18:14 -0600 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <1063203494.18382.44.camel@thinkpad> > That's an excellent design decision. > Putting a check word on each packet means > - the physical encoding layer need to know about packetization > - a packet must be held until the check passes > - the tiny packets grow > - to do anything with the per-packet info, packet copies must be kept > These all add complexity and latency to the highest speed path. > > By putting the check on fixed block boundaries you can still detect and > fail an unreliable link All very true when you have 1, 2, 4, even 8 HT links that could cause a system to crash. And I'm not suggesting that ECC would be better (that was Greg's statement), but.... if you had 10000 HT links running their maximum distance (if you used HT links to build a mesh at Red Storm scale) and any bit error on any of them causes an app to fail because you don't know which packet had an error... That would be bad. Supposedly, this is going to be fixed in HT 2.0 (I wouldn't know since the spec isn't freely available). Keith _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From molson at edgetechcomputing.com Wed Sep 10 11:19:52 2003 From: molson at edgetechcomputing.com (Myles Olson) Date: Wed, 10 Sep 2003 09:19:52 -0600 Subject: data storage location Message-ID: <000601c377ae$fc0a3c90$54aa6f90@TOSHIBER> I'm new at this so please excuse any errors on my part. I need to process datasets around 100GB. What would be the best way for the Beowulf cluster to access this data? Should the host node connect to a disk array over FC-AL? Should it be a fiber connection to a SAN? Or does all this depend on how the application breaks the data down for processing? Thanks ? Myles Olson Edgetech Computing - Technical Services Office: 403.237.5331 Mobile: 403.875.6630 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Wed Sep 10 12:15:25 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Wed, 10 Sep 2003 09:15:25 -0700 (PDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <20030910070134.GA1731@greglaptop.greghome.keyresearch.com> Message-ID: On Wed, 10 Sep 2003, Greg Lindahl wrote: > > Remeber the goal of a single commodity motherboard supporting either an > > AMD x86 chip or an Alpha? Look where we are today... > > It was a great idea until the mechnical problems nuked it: Slot A > vs. Socket A. I was not impressed by the reliability of Slot A. > Sometimes it's the little things that cause the biggest problems. the processor cards for our alpha server 7000's (21164 433mhz) were something like 12"x16" the l2 cache alone is like 20 square inches. I still have one of them hanging on my wall. the larger alpha building blocks had and really continue to suffer from a serious form factor problem. The gs80 is basically stil about 3/4 of a rack for an 8cpu box. > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 13:20:52 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 10:20:52 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <1063203494.18382.44.camel@thinkpad> References: Message-ID: <5.2.0.9.2.20030910101023.018a99f8@mailhost4.jpl.nasa.gov> At 08:18 AM 9/10/2003 -0600, Keith D. Underwood wrote: > > That's an excellent design decision. > > Putting a check word on each packet means > > - the physical encoding layer need to know about packetization > > - a packet must be held until the check passes > > - the tiny packets grow > > - to do anything with the per-packet info, packet copies must be kept > > These all add complexity and latency to the highest speed path. > > > > By putting the check on fixed block boundaries you can still detect and > > fail an unreliable link > >All very true when you have 1, 2, 4, even 8 HT links that could cause a >system to crash. And I'm not suggesting that ECC would be better (that >was Greg's statement), but.... if you had 10000 HT links running their >maximum distance (if you used HT links to build a mesh at Red Storm >scale) and any bit error on any of them causes an app to fail because >you don't know which packet had an error... That would be bad. Any time you're looking at large distributed systems, one needs to plan for and be able to handle failures. If a single hit causes the entire shooting match to collapse, it's never going to work. In fact, I'd claim that what "scalability" really means is that the inevitable errors have limited propagation. Otherwise, as you increase the number of "widgets" in the system, the probability of any one failing starts to get close to one. The real design decisions come in when deciding at what level to handle the error. Detecting is straightforward at the bottom level, but error handling may be best dealt with at a high level. Perhaps the overall performance of the ensemble is better if everyone goes in lockstep, rather than retrying the failed communication. Compare, for example, 11 for 8 Hamming coding at the byte level (low latency, poor rate efficiency), and CRC error detection and retries at a higher level, and then, all the sorts of block interleaving schemes which turn burst errors into isolated (and correctable on the fly) errors. Sometimes you trade determinism for performance, or latency for overall bit error rate. A lot depends on your error statistics, and such is grist for much communications theory analysis, and keeps coding specialists employed. Consider also, things like algorithm robustness to bit flips in RAM. On one system I worked on, it was faster to do the calculations three times and vote the results, with no ECC, than to do them once with ECC, because of the increased latency of the ECC logic, and the adverse interaction between ECC and cache. There is a fair amount of literature on this... The Byzantine Generals problem (unreliable communication between multiple masters) is a good example. Fault robustness/tolerance in large cluster configurations is a subject that is near and dear to my heart because I want to fly clusters in space, where errors are a given, and maintenance is impossible. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Sep 10 13:07:57 2003 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 10 Sep 2003 11:07:57 -0600 Subject: CPUs for a Beowulf In-Reply-To: <20030910105848W.hanzl@unknown-domain> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> Message-ID: <3F5F5A6D.2050201@lanl.gov> hanzl at noel.feld.cvut.cz wrote: >>>>Administrative costs can easily be THE dominant cost >> >>My first non-small HPC cluster was 64 nodes, and it took 28 hours of >>labor to set up, of which only 8 hours was my time > > And the first small one? Anyone getting started on clusters has to negotiate a learning curve, regardless of whether they build it themselves or buy a turn-key system. This part of the administrative cost is unavoidable for a new cluster user. For lots of people, learning on the system you built yourself is preferable. Manpower costs are dominant in most projects, but most manpower costs are related to using the machinery, not administering it. When administrative costs exceed the cost of equipment, people tend to buy different stuff with lower overall life cycle costs. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 10 12:39:13 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 10 Sep 2003 09:39:13 -0700 Subject: CPUs for a Beowulf In-Reply-To: <20030910105848W.hanzl@unknown-domain> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> Message-ID: <20030910163913.GA3486@greglaptop.greghome.keyresearch.com> > > My first non-small HPC cluster was 64 nodes, and it took 28 hours of > > labor to set up, of which only 8 hours was my time > > And the first small one? > > (I basically wanted to scare people who are just building the very > first cluster You shouldn't use me as an example, then: that was my first HPC cluster, but I had set several groups of systems before which had related configs, and worked on clusters up to ~ 130 systems in size while working in industry. It all depends on what your experience is. If you haven't set up any kind of cluster before, you're asking for trouble. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 10 14:21:38 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 10 Sep 2003 14:21:38 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <20030910163913.GA3486@greglaptop.greghome.keyresearch.com> Message-ID: On Wed, 10 Sep 2003, Greg Lindahl wrote: > > > My first non-small HPC cluster was 64 nodes, and it took 28 hours of > > > labor to set up, of which only 8 hours was my time > > > > And the first small one? > > > > (I basically wanted to scare people who are just building the very > > first cluster > > You shouldn't use me as an example, then: that was my first HPC > cluster, but I had set several groups of systems before which had > related configs, and worked on clusters up to ~ 130 systems in size > while working in industry. > > It all depends on what your experience is. If you haven't set up > any kind of cluster before, you're asking for trouble. Ya, which returns nicely to the original point. There are two or three elements of cluster engineering that can bite you if you've never done large scale cluster (or LAN) design or management before. Hardware, installation and maintenance, and infrastructure might be easy (but probably incomplete) categories. COTS is good, COTS is the point, but reliability becomes increasingly important as the number of systems scales up and somewhere in there it is no longer wise to get CHEAP COTS systems, as in lowest-bid vanilla boxes with no provisions for service. Hardware maintenance can eat you alive if you end up with a large batch of flaky hardware, even if you do have a service deal of some sort as it still costs you all sorts of time for every failure. Clusters eat electricity and excrete it as heat. Cluster nodes need safe homes and like to talk. An eight-node toy cluster can often be set up "anywhere" and just plugged in to a handy wall receptacle (although even 8 nodes can make a room pretty hot without enough A/C). A larger cluster often requires planning and remodeling, even engineering, of the space the nodes are to live in. This can EASILY cost more than the hardware you put into that space! It is easy to use bad methodology to install and maintain a cluster. Not as easy as it once was -- it is really pretty easy to use GOOD methodology these days it the (awesome) tools are there and are even tolerably well-documented -- but if your experience with linux is installing it off of CD's onto a few systems one at a time, you've got some learning to do. If you're really a Unix novice and don't have a decent grasp of TCP/IP, ethernet (at least), scalable administrative tools and services such as but not limited to NIS, NFS, and a whole lot more, you've got a LOT of learning to do. You, Greg are very much in the expert category -- no, I'd have to go beyond that to "trans-professional superguru" category -- with a vast experience in all of the above and then some. Besides that, you're pretty smart;-) You should be wearing a special medal or insignia -- the Royal Order of the 'Wulf or the like:-) Others (including me if it came to it) sometimes are very good at figuring out what went wrong -- after the fact -- so it helps for them/us to proceed cautiously. Even something like a cheap, non-PXE fast ethernet NIC vs a more expensive, PXE-supporting NIC may not seem important (and hey, going COTS cheap saves you MONEY, right?), until you find yourself carrying a floppy around to 64 systems to install them one at a time, or have a two-post rack collapse because it isn't adequately anchored. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Wed Sep 10 15:01:55 2003 From: gmpc at sanger.ac.uk (Guy Coates) Date: Wed, 10 Sep 2003 20:01:55 +0100 (BST) Subject: data storage location In-Reply-To: <200309101726.h8AHQId22018@NewBlue.Scyld.com> References: <200309101726.h8AHQId22018@NewBlue.Scyld.com> Message-ID: As always, it depends on what the data access pattern like, and how many nodes you have. If the data access is read-only, *and* your node number is small (<10), *and* you have a fast network then you *might* be able to get away with NFS. However, for read/write access or more than a few nodes, you need to be a bit more creative. Fibre channel + switch may look good on paper but have their problems (not least of which is cost). If you have a moderate amount of fibre clients trying to access the same disk controller you can easily saturate the fibre ISLs or disk controllers. For example, 4 clients doing 50 Mbytes/S sustained data access are easily going to saturate the fibre to your disk controller, whether switched or on FC-AL. If your data access is bursty then you might do better. Furthermore, if you want all your clients to access the same filesystem on the your fibre channel disks then you are also going to need a clustered filesystem. The alternative approach is to keep copies of the data on local disk on each node. This gives you good IO rates, but you then have a substantial data management problem; how to you copy 100Gb to each node in your cluster in a sensible amount of time, and how do you update the data and make sure it is kept consistent? The commonest approach to data distribution is to do some sort of cascading rsync/rcp which follows the topology of your network. If your dataset is larger than the amount of local disk on your nodes, you then have to partition your data up, and integrate that with your queuing system, so that jobs which need a certain bit of the data end up on a node which actually holds a copy. Recently we've had some success using multicast methods to distribute data to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of the better multicast codes, but you'll need to put some wrappers around it to make it useful. The multicast method is substantially faster than rsyncing data on large clusters. Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 10 16:58:19 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 10 Sep 2003 13:58:19 -0700 Subject: CPUs for a Beowulf In-Reply-To: <3F5F5A6D.2050201@lanl.gov> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> <3F5F5A6D.2050201@lanl.gov> Message-ID: <20030910205819.GA2023@greglaptop.internal.keyresearch.com> On Wed, Sep 10, 2003 at 11:07:57AM -0600, Josip Loncaric wrote: > hanzl at noel.feld.cvut.cz wrote: > >>>>Administrative costs can easily be THE dominant cost > >> > >>My first non-small HPC cluster was 64 nodes, and it took 28 hours of > >>labor to set up, of which only 8 hours was my time > > > >And the first small one? > > Anyone getting started on clusters has to negotiate a learning curve, > regardless of whether they build it themselves or buy a turn-key system. Josip, My point was that some admins with zero "beowulf" cluster experience have plenty of large system Unix experience. LANL used to have several such admins, and they built some large clusters for their first attempt, and it went very well. They even used cfengine, which is a tool commonly used in the large Unix system arena. Several of them probably still work for LANL; the one I know best, Susan Coghlan, is now at Argonne. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 17:00:03 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 14:00:03 -0700 Subject: cluster using handhelds Message-ID: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Has anyone tried building a cluster using handheld computers (i.e. iPaq or Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is there an implementation of MPI that will work in that environment (WinCE, PalmOS) (perhaps MPICH in some form?) Yes, I know the performance will be hideous. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Wed Sep 10 17:22:16 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Wed, 10 Sep 2003 14:22:16 -0700 (PDT) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: I could trivially do it with both my zauruses right now... the real question is why... On Wed, 10 Sep 2003, Jim Lux wrote: > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > there an implementation of MPI that will work in that environment (WinCE, > PalmOS) (perhaps MPICH in some form?) > > Yes, I know the performance will be hideous. > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 17:49:29 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 14:49:29 -0700 Subject: cluster using handhelds In-Reply-To: References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> At 02:22 PM 9/10/2003 -0700, Joel Jaeggli wrote: >I could trivially do it with both my zauruses right now... the real >question is why... Which model of Zaurus, and what sort of wireless network interface (CF or PCMCIA)? I'm looking at a distributed measurement application where each processor needs to be able to talk to some local hardware and to share measurements with other processors. I also need to distribute some computing across the processors. I'd rather not have to rewrite all the computing as platforms change, so a standardized interface like MPI is attractive. Obviously, an alternative is a battery powered single board PC with a suitable display and user interface, but, in the case of the handhelds, someone else has already dealt with the hardware integration issues, and the purchase price is generally lower than one would spend on all the pieces. I don't want to be in the PC design business, if I can avoid it. I note that the various variants of Linux at handhelds.org (formerly, and possibly still, supported by Compaq) don't support the current versions of iPaqs being sold by HP. This, in itself, is kind of a worrying trend, because I've already got enough orphan hardware and software sitting around. (I've got ISA bus machines running win95 down in the lab, because that's what's needed to run the incircuit emulators for the "no longer sold in the commercial market but still sold for spaceflight" DSPs) >On Wed, 10 Sep 2003, Jim Lux wrote: > > > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > > there an implementation of MPI that will work in that environment (WinCE, > > PalmOS) (perhaps MPICH in some form?) > > > > Yes, I know the performance will be hideous. > > > > > > -------------------------------------------------------------------------- >Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu >GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Wed Sep 10 18:22:19 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Wed, 10 Sep 2003 15:22:19 -0700 (PDT) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > At 02:22 PM 9/10/2003 -0700, Joel Jaeggli wrote: > >I could trivially do it with both my zauruses right now... the real > >question is why... > > Which model of Zaurus, and what sort of wireless network interface (CF or > PCMCIA)? 5500sl it/they have cf wireless cards... you can pretty much get the whole arm build enivironment and toolchain on a large mmc card in the sd card slot. you can run open zaurus or the debian arm distro on them in a pretty simple fashion... They also have serial port pinout on the jack on the bottom which you can get a cable for so serial consoles are feasable. also they have keyboards so failing around on a text consoles is feasable. > I'm looking at a distributed measurement application where each processor > needs to be able to talk to some local hardware and to share measurements > with other processors. I also need to distribute some computing across the > processors. I'd rather not have to rewrite all the computing as platforms > change, so a standardized interface like MPI is attractive. > > Obviously, an alternative is a battery powered single board PC with a > suitable display and user interface, but, in the case of the handhelds, > someone else has already dealt with the hardware integration issues, and > the purchase price is generally lower than one would spend on all the > pieces. I don't want to be in the PC design business, if I can avoid it. basically the things that make me say why.... are the mips/watt equation on a pda is skewed by things like the active matrix display which is still power hungry even if the backlight is off. internal batteries, particularly with a wireless card inserted don't last useful lengths of time. in the zaurus's case about and hour and 45 minutes with the card in or maybe 6 without the card and the backlight turned all the way down. then we have the under-powered cpu's which for the 200mhz make it sufficent to compare to a cca 1993 pc which isn't bad but it's not huge either... We do a substantial amount of work around some embeded pc boards made by a company called soekris engineering http://www.soekris.com/ that come with nice cases that might be appropiate for you application... in particular it makes porting stuff a non-issue since i586 or 486 code will run happily on them. I actually run a somewhat condensed version of redhat 8.0 on a couple of their boxes. a number of people use them as wireless accesspoints amount other things so they may be useful for you application. they also have useful features that's pda's don't like gpio pins. some of the models (4521) have very wide dc voltage operating ranges (11-56 volts) to accomidate things like power over ethernet. http://www.soekris.com/net4521.htm > I note that the various variants of Linux at handhelds.org (formerly, and > possibly still, supported by Compaq) don't support the current versions of > iPaqs being sold by HP. This, in itself, is kind of a worrying trend, > because I've already got enough orphan hardware and software sitting > around. the life-cyle in the pda market, especially the wince pda market is very very short. > (I've got ISA bus machines running win95 down in the lab, because > that's what's needed to run the incircuit emulators for the "no longer sold > in the commercial market but still sold for spaceflight" DSPs) > > > > > >On Wed, 10 Sep 2003, Jim Lux wrote: > > > > > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > > > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > > > there an implementation of MPI that will work in that environment (WinCE, > > > PalmOS) (perhaps MPICH in some form?) > > > > > > Yes, I know the performance will be hideous. > > > > > > > > > -------------------------------------------------------------------------- > >Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu > >GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 18:29:36 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 15:29:36 -0700 Subject: cluster using handhelds In-Reply-To: References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> At 04:48 PM 9/10/2003 -0500, you wrote: >On Wed, 10 Sep 2003, Jim Lux wrote: > > > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > > there an implementation of MPI that will work in that environment (WinCE, > > PalmOS) (perhaps MPICH in some form?) > > > > Yes, I know the performance will be hideous. > >Ok, I can not help but to comment. > >Once upon a time, just for S&G's, we spec'ed out a T-FLOP cluster out of >PalmPilots. Assuming no space for wiring, we figgured we could fit >about 144 of my Palms into a cubic foot. > >Crammed floor to ceiling, had our building had 1,500 stories, it would >have almost held the thing. > >Sorry for my silliness, but i still find it funny. You may laugh, but consider a structure that looks like a floppy ball some 100 meters in radius with sensing elements scattered every 10-20 cm across the surface, each with it's own processor that needs to talk to some (but not all) of the other elements to make the measurements. Perhaps not TFLOPS, but a huge pile of small processors, nonetheless. Now consider that before anybody would give you the tens of millions of dollars required to actually build such a thing, they'd want to see some sort of credible demonstration that it's feasible to do the work with processors of limited capability, but, yet, still general purpose and programmable. Further, one probably doesn't want to spend a huge amount of time designing processor nodes, but wants to buy something off the shelf, preferably consumer mass market. One would also prefer something where the internal architecture is published and understandable so that one could make credible estimates of what that fancy custom element is going to need. Hence the question about clusters of handhelds: handhelds are consumer mass market and cheap linux on a handheld is "open and visible" as far as internals go WinCE might be (depending on licensing, etc. It's been a long time since I was an "embedded windows" (forerunner to WinCE) developer) 802.11 is a wireless comm scheme with known properties. The eventual system would probably use something else (Snaggletooth, Forkbeard (Bluetooth's son), Zigbee (802.15.4), CCSDS proximity link, or whatever) MPI because I'm lazy, and I already know how to use MPI for stuff. By the way, handhelds that can talk to a cmos/ccd camera and grab frames on demand would be even better. >-- >Rocky McGaugh >Atipa Technologies >rocky at atipatechnologies.com >rmcgaugh at atipa.com >1-785-841-9513 x3110 >http://67.8450073/ >perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Wed Sep 10 17:48:26 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Wed, 10 Sep 2003 16:48:26 -0500 (CDT) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > there an implementation of MPI that will work in that environment (WinCE, > PalmOS) (perhaps MPICH in some form?) > > Yes, I know the performance will be hideous. Ok, I can not help but to comment. Once upon a time, just for S&G's, we spec'ed out a T-FLOP cluster out of PalmPilots. Assuming no space for wiring, we figgured we could fit about 144 of my Palms into a cubic foot. Crammed floor to ceiling, had our building had 1,500 stories, it would have almost held the thing. Sorry for my silliness, but i still find it funny. -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Sep 10 20:42:20 2003 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 10 Sep 2003 18:42:20 -0600 Subject: CPUs for a Beowulf In-Reply-To: <20030910205819.GA2023@greglaptop.internal.keyresearch.com> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> <3F5F5A6D.2050201@lanl.gov> <20030910205819.GA2023@greglaptop.internal.keyresearch.com> Message-ID: <3F5FC4EC.2090001@lanl.gov> Greg Lindahl wrote: > On Wed, Sep 10, 2003 at 11:07:57AM -0600, Josip Loncaric wrote: > >>Anyone getting started on clusters has to negotiate a learning curve, >>regardless of whether they build it themselves or buy a turn-key system. > > My point was that some admins with zero "beowulf" cluster experience > have plenty of large system Unix experience. Good point. It sure helps to start high on the learning curve... I'd only like to add that administrative costs for a functional cluster are *way* lower than for the same number of office PCs. There are many reasons for this so I do not want to enumerate them here. You probably have a better feel for this, but my guess is that a capable Linux cluster administrator can manage several hundred nodes. In other words, although support costs for an N-processor cluster scale with N, the scaling constant is fairly reasonable. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Wed Sep 10 22:04:12 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 11 Sep 2003 10:04:12 +0800 (CST) Subject: Mac clusters... In-Reply-To: <0EB5C81FE6FE5A4F8D1FEBF59C6C7BAA0A9FB7@durham.sra.com> Message-ID: <20030911020412.29314.qmail@web16808.mail.tpe.yahoo.com> Sounds to me they chose Apple due to the better cost/performance, but you can get more information from the presentation: http://www.chaosmint.com/mac/vt-supercomputer/ One thing which is not mentioned is the batch system, are they going to use PBS or SGE, anyone? Andrew. --- "Walters, David" ????> Does anyone know where their funding came from? Did > they choose Apple based > on a cost/benefit analysis, or is Apple paying them? > I am thinking back to > the numbers that came out of the Cornell Theory > Center here... Same > scenario? > > Dave Walters > Project Manager/Sr. Management Analyst, EPA IIASC > SRA International, Inc. > 919-474-4318 > david_walters at sra.com > > > -----Original Message----- > From: Rayson Ho [mailto:raysonlogin at yahoo.com] > Sent: Friday, September 05, 2003 5:32 PM > To: beowulf > Subject: Mac clusters... > > > 1100 dual G5s to build a supercomputer: > > http://www.computerweekly.com/articles/article.asp?liArticleID=124559&liFlav > ourID=1&sp=1 > > And this cluster is interesting too: > > --- "Hunt, Derek" wrote: > > Hello all, this is in response to Charles D. > Norton's post regarding > > Xserve clusters. > > > > Site: Yale School Of Management > > Yale University > > Hardware Vendor: Apple Computer > > Number of nodes: 45 > > Number of processors: 90 > > Total Memory: 90GB > > Interconnect Technology: Gigabit Ethernet > > OS: 10.2.6 with current patches > > Comm Software: MPICH, Sun Gridengine > > Application Software: GridMathematica, Matlab, > C/C++, Fortran Mail > > application area Scientific Computing: Financial > Research, Statistical > > Analysis Year installed/upgraded: 2003 > > > > We are in the initial stages of deployment now. > > > > on a side note, I will be giving a talk on > September 25th in > > Rochester, MN at the linux users group (more > details at > > http://www.k-lug.org) if anyone is interested in > hearing the various > > trials we encountered during > > installation/deployment. > > > > - Derek > > > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free, easy-to-use web site > design software > http://sitebuilder.yahoo.com > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bari at onelabs.com Wed Sep 10 23:05:34 2003 From: bari at onelabs.com (Bari Ari) Date: Wed, 10 Sep 2003 22:05:34 -0500 Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: <3F5FE67E.2090002@onelabs.com> Jim Lux wrote: > > You may laugh, but consider a structure that looks like a floppy ball > some 100 meters in radius with sensing elements scattered every 10-20 > cm across the surface, each with it's own processor that needs to talk > to some (but not all) of the other elements to make the measurements. > Perhaps not TFLOPS, but a huge pile of small processors, nonetheless. > > Now consider that before anybody would give you the tens of millions > of dollars required to actually build such a thing, they'd want to see > some sort of credible demonstration that it's feasible to do the work > with processors of limited capability, but, yet, still general purpose > and programmable. > > Further, one probably doesn't want to spend a huge amount of time > designing processor nodes, but wants to buy something off the shelf, > preferably consumer mass market. One would also prefer something where > the internal architecture is published and understandable so that one > could make credible estimates of what that fancy custom element is > going to need. Take a look at http://www.eu.renesas.com/products/mpumcu/32bit/sh-microprocessors/sh4/index.html or http://www.superh.com/products/sh5.htm The SH-4's perform at around 1GFLOP/Watt and the SH-5's at 3GFLOP/Watt. SH Linux has been around for some time. http://www.sh-linux.org/ There are a few vendors who make PC-104ish size SH based boards. You could build a TFLOP cluster using the SH-5's that would fit into a single 42-U rack and only consume around 1KW to 3KW depending on the amount of memory used. Memory would actually consume more power than the cpu's. For fixed point only apps. consider some of the boards that use the PXA or X-Scale cpu's. They cost about the same as an high end PXA based PDA. Bari _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 11 05:28:10 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 11 Sep 2003 11:28:10 +0200 (CEST) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > there an implementation of MPI that will work in that environment (WinCE, > PalmOS) (perhaps MPICH in some form?) > If my memory serves me, I remember a presentation at the FOSDEM 2002 conference from a French lab which did something along these lines. A qucik Google doesn't find this. IT might have been CNRS, _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 11 08:11:35 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 11 Sep 2003 08:11:35 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <3F5FC4EC.2090001@lanl.gov> Message-ID: On Wed, 10 Sep 2003, Josip Loncaric wrote: > I'd only like to add that administrative costs for a functional cluster > are *way* lower than for the same number of office PCs. There are many > reasons for this so I do not want to enumerate them here. > > You probably have a better feel for this, but my guess is that a capable > Linux cluster administrator can manage several hundred nodes. In other > words, although support costs for an N-processor cluster scale with N, > the scaling constant is fairly reasonable. Agreed. The scaling constant itself can even get to where it is determined primarily by hardware installation and maintenance alone, although this is also an important determinant in any LAN. With PXE/diskless operation, or PXE/kickstart installation, yum or various clone methods for keeping software up to date, a single person can "take care of" (in the sense of install and maintain OS's on) a staggering number of systems, if they never physically break. In at least some cluster environments, there aren't many users and the users themselves are pretty unix-capable and need little "support" (often a if not the major office LAN cost). You can't get out of paying for power and cooling, but once a suitable space is constructed power and cooling become a fairly predictable expense, roughly $1 per watt per year. Hardware remains the joker in the deck. Hardware support costs can be hideously variable. Bad electricity or cooling or both can cause nodes to break (done that one). Inadequate cooling fans or other node design flaws can cause nodes to break (done that one). Poor quality node components (e.g. the newer 1 yr. warranty IDE drives, a "bad batch" of nearly any component) can have a relatively high probability of failure that is low (per day) for any single system but high (per day) for 512 systems (to a certain extent unavoidable, so sure, done that). A "new" motherboard with a great clock and slick features can turn out to have a piece of s**t bios that requires two or three reflashes before it finally settles down to function, or it can just plain never stabilize and ultimately have to be replaced (times N, mind you -- and yes, done both of these). And then even really good, gold-plated name brand COTS hardware breaks with fixed probabilities and a roughly poissonian distribution (with those lovely little clusters of events) so it isn't like one hard disk fails every two weeks, its more like no disks fail for a couple of months and then four fail within a couple of days of one another. A hardware failure also can have nonlinear costs (beyond mere downtime, human time involved in fixing it, and maybe the cost of a new component) in productivity, if the computation that is proceeding at the time of failure isn't checkpointed and is tightly coupled, so a node failure brings down the whole run. At least THIS one doesn't bother me -- my own computations tend to be EP.:-) The moral of the story being -- COTS hardware, sure, but for larger clusters especially (event frequency scaling linearly with the number of nodes, disaster severity scaling NONlinearly with the number of nodes) you should PROTOTYPE and GET HIGH QUALITY NODES, and meditate carefully upon the issue of possibly onsite support contracts. Onsite support reduces the amount of time YOU (the systems managers) have to screw around with broken hardware, although a lot of people do get by with e.g. overnight or second day replacement, possibly buffering the most likely parts to fail with spares, if they have the local staff to handle it. You can trade off local human effort (opportunity cost allocation of existing staff time) with prepaid remote human effort (service contracts and so forth) depending on how it is easier to budget and how much "disposable" local FTE you wish to dispose of in this way. One way or another, node (un)reliability equals time equals money, and this "hidden" cost has to be balanced against the cost of the nodes themselves as raw hardware when computing (hands on your wallets, please:-) "Total Cost of OwnerShip" (TCOS). [As a parenthetical insertion of the sort I'm doubtless (in)famous for, has anybody noticed how TCOS is an acronym-anagram for COTS? Well, really it isn't but since I added the S to TCO it is. Spooky, huh....] I think that this is a lesson most large scale cluster managers (including those that have previously managed large scale LANs) have already learned the hard way, to the point where they are bored with the enumeration above. Cluster/LAN newbies, though, especially those who may have played with or be preparing to engineer a "toy" cluster with just a few nodes, may not have thought about all of this, so it is worth it to hammer on the point just a wee bit. After all, many of us on the list learned these lessons the hard way (go on, raise those hands, don't be shy or ashamed -- look, my hand is WAY up:-) and the least we can do is try to spare those newbies our pain. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From scheinin at crs4.it Thu Sep 11 08:33:42 2003 From: scheinin at crs4.it (Alan Scheinine) Date: Thu, 11 Sep 2003 14:33:42 +0200 Subject: Tyan 2880 memory configuration Message-ID: <200309111233.h8BCXgx05965@crs4.it> Where can I get information concerning the configuration of memory for the Tyan 2880? (I ask in this mailing list because this boards seems a likely candidate for a do-it-yourself Opteron cluster.) The documentation from Tyan does not give useful advice. Page 29 of the manual shows different ways of inserting memory but does not give advice. In particular, one reads, "memory in CPU2 DIMM1-2 is not required when running dual CPU configuration" but the diagram on page 9 indicates that access to memory would be divided between two CPUs if the DIMM sockets CPU1 and CPU2 are both used. The table on page 29 shows putting 4 memory modules in the area CPU1 as a possibility without saying whether (as I would imagine) it is better to follow the other option of putting two in CPU1 and two in CPU2. Moreover, the BIOS documentation of the 2880 manual gives various options for the use of memory but does not explain the terminology and does not explain which choice would give better performance (page 54 of the manual). I do not intend to request that someone write a detailed reply to the mailing list, it would be enough to know where to look to find the information. best regards, Alan Scheinine _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Thu Sep 11 08:59:33 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Thu, 11 Sep 2003 08:59:33 -0400 (EDT) Subject: Tyan 2880 memory configuration In-Reply-To: <200309111233.h8BCXgx05965@crs4.it> Message-ID: > inserting memory but does not give advice. In particular, > one reads, "memory in CPU2 DIMM1-2 is not required when running > dual CPU configuration" but the diagram on page 9 indicates that > access to memory would be divided between two CPUs if the DIMM > sockets CPU1 and CPU2 are both used. The table on page 29 shows > putting 4 memory modules in the area CPU1 as a possibility without > saying whether (as I would imagine) it is better to follow the > other option of putting two in CPU1 and two in CPU2. my first opteron system arrived with 4 dimms in the first CPU's bank of memory, and bank interleaving turned off in bios. turning on bank interleaving gave me around 15% speedup on stream. moving two dimms to the other CPU gave me about 25% speedup. ideally, of course, both CPUs would have two full banks (4 dimms apiece), with both node and bank interleaving turned on in bios. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Thu Sep 11 09:25:29 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Thu, 11 Sep 2003 15:25:29 +0200 (CEST) Subject: data storage location In-Reply-To: Message-ID: On Wed, 10 Sep 2003, Guy Coates wrote: [...] > Recently we've had some success using multicast methods to distribute data > to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of > the better multicast codes, but you'll need to put some wrappers around it > to make it useful. The multicast method is substantially faster than > rsyncing data on large clusters. I had two problems when testing UDPcast: - A cheaper ATI CentreCom Fast-Ethernet switch multicasted data only with the speed of the slowest connected machine, even if that machine was *not* part of the multicast group. I had to unplug all 10 Mbps links to speed up UDPcast above 1 MByte/s. - With Gigabit Ethernet and a smaller cluster of 1 GHz PentiumIII machines I had to set a rate limitation on the sender. Otherwise the sender and the receivers lost synchronization and the transmission didn't work. However, if you forgive me a short bit of advertising, we use the "Dolly" programm to replicate large amounts of data in our clusters: http://www.cs.inf.ethz.ch/CoPs/patagonia/#dolly It replicates data with a multi-drop chain over TCP and scales quite nicely. We only had performance problems on a switch with a rather limited backplane, but otherwise we use it regularly in our 16- and 128-node clusters. - Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jhalpern at howard.edu Thu Sep 11 09:34:24 2003 From: jhalpern at howard.edu (Joshua Halpern) Date: Thu, 11 Sep 2003 08:34:24 -0500 Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: <3F6079E0.1020704@howard.edu> IF you really wanted to do this, one way would be to contact a large organiztion that had a lot of the same kind of laptops with broken display screens/keyboards...much more bang for the buck at less power and they probably WANT to give them to you. Of course the power brick farm would be interesting. Josh Halpern Jim Lux wrote: > At 04:48 PM 9/10/2003 -0500, you wrote: > > >> On Wed, 10 Sep 2003, Jim Lux wrote: >> >> > Has anyone tried building a cluster using handheld computers (i.e. >> iPaq or >> > Palm type) that use wireless LAN (802.11) or IR for the >> interconnect? Is >> > there an implementation of MPI that will work in that environment >> (WinCE, >> > PalmOS) (perhaps MPICH in some form?) >> > >> > Yes, I know the performance will be hideous. >> >> Ok, I can not help but to comment. >> >> Once upon a time, just for S&G's, we spec'ed out a T-FLOP cluster out of >> PalmPilots. Assuming no space for wiring, we figgured we could fit >> about 144 of my Palms into a cubic foot. >> >> Crammed floor to ceiling, had our building had 1,500 stories, it would >> have almost held the thing. >> >> Sorry for my silliness, but i still find it funny. > > > You may laugh, but consider a structure that looks like a floppy ball > some 100 meters in radius with sensing elements scattered every 10-20 > cm across the surface, each with it's own processor that needs to talk > to some (but not all) of the other elements to make the measurements. > Perhaps not TFLOPS, but a huge pile of small processors, nonetheless. > > Now consider that before anybody would give you the tens of millions > of dollars required to actually build such a thing, they'd want to see > some sort of credible demonstration that it's feasible to do the work > with processors of limited capability, but, yet, still general purpose > and programmable. > > Further, one probably doesn't want to spend a huge amount of time > designing processor nodes, but wants to buy something off the shelf, > preferably consumer mass market. One would also prefer something where > the internal architecture is published and understandable so that one > could make credible estimates of what that fancy custom element is > going to need. > > Hence the question about clusters of handhelds: > handhelds are consumer mass market and cheap > linux on a handheld is "open and visible" as far as internals go > WinCE might be (depending on licensing, etc. It's been a long time > since I was an "embedded windows" (forerunner to WinCE) developer) > 802.11 is a wireless comm scheme with known properties. The eventual > system would probably use something else (Snaggletooth, Forkbeard > (Bluetooth's son), Zigbee (802.15.4), CCSDS proximity link, or whatever) > MPI because I'm lazy, and I already know how to use MPI for stuff. > > By the way, handhelds that can talk to a cmos/ccd camera and grab > frames on demand would be even better. > > > > >> -- >> Rocky McGaugh >> Atipa Technologies >> rocky at atipatechnologies.com >> rmcgaugh at atipa.com >> 1-785-841-9513 x3110 >> http://67.8450073/ >> perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sp at scali.com Thu Sep 11 00:34:05 2003 From: sp at scali.com (Steffen Persvold) Date: Thu, 11 Sep 2003 06:34:05 +0200 Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> Message-ID: <3F5FFB3D.7070208@scali.com> Jim Lux wrote: > At 02:22 PM 9/10/2003 -0700, Joel Jaeggli wrote: > >> I could trivially do it with both my zauruses right now... the real >> question is why... > > > Which model of Zaurus, and what sort of wireless network interface (CF > or PCMCIA)? > > I'm looking at a distributed measurement application where each > processor needs to be able to talk to some local hardware and to share > measurements with other processors. I also need to distribute some > computing across the processors. I'd rather not have to rewrite all the > computing as platforms change, so a standardized interface like MPI is > attractive. > > Obviously, an alternative is a battery powered single board PC with a > suitable display and user interface, but, in the case of the handhelds, > someone else has already dealt with the hardware integration issues, and > the purchase price is generally lower than one would spend on all the > pieces. I don't want to be in the PC design business, if I can avoid it. > > I note that the various variants of Linux at handhelds.org (formerly, > and possibly still, supported by Compaq) don't support the current > versions of iPaqs being sold by HP. This, in itself, is kind of a > worrying trend, because I've already got enough orphan hardware and > software sitting around. (I've got ISA bus machines running win95 down > in the lab, because that's what's needed to run the incircuit emulators > for the "no longer sold in the commercial market but still sold for > spaceflight" DSPs) > > Jim, Beeing the Linux fan I am, I recently tried to load Linux on my HP iPAQ 5450 (a relatively new one with bluetooth, 802.11b, irda and.. fingerprint scanner..). It worked ! I reached the commandline prompt with the "Familiar" development distribution. However, I did not try it out much since I needed the PDA for more productive things. It was fun though :) If you take a look at the http://www.handhelds.org/projects/h5400.html page you'll see the progress of the project. I've been thinking of contributing, but I haven't got the time at the moment... Regards, -- Steffen Persvold Senior Software Engineer mob. +47 92 48 45 11 tel. +47 22 62 89 50 fax. +47 22 62 89 51 Scali - http://www.scali.com High Performance Clustering _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Thu Sep 11 09:39:45 2003 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Thu, 11 Sep 2003 08:39:45 -0500 Subject: data storage location In-Reply-To: References: Message-ID: <3F607B21.5010904@tamu.edu> We have seen less than stellar performance with the older/cheaper ATI Centrecom switches. We were almost completely unsuccessful with multicast tests. And RFC2544 packet testing was the pits. So: small packets are bad, and multicast appears problemmatic. At least on these devices. gerry Felix Rauch wrote: > On Wed, 10 Sep 2003, Guy Coates wrote: > [...] > >>Recently we've had some success using multicast methods to distribute data >>to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of >>the better multicast codes, but you'll need to put some wrappers around it >>to make it useful. The multicast method is substantially faster than >>rsyncing data on large clusters. > > > I had two problems when testing UDPcast: > > - A cheaper ATI CentreCom Fast-Ethernet switch multicasted data only > with the speed of the slowest connected machine, even if that > machine was *not* part of the multicast group. I had to unplug all > 10 Mbps links to speed up UDPcast above 1 MByte/s. > > - With Gigabit Ethernet and a smaller cluster of 1 GHz PentiumIII > machines I had to set a rate limitation on the sender. Otherwise the > sender and the receivers lost synchronization and the transmission > didn't work. > > However, if you forgive me a short bit of advertising, we use the > "Dolly" programm to replicate large amounts of data in our clusters: > http://www.cs.inf.ethz.ch/CoPs/patagonia/#dolly > > It replicates data with a multi-drop chain over TCP and scales quite > nicely. We only had performance problems on a switch with a rather > limited backplane, but otherwise we use it regularly in our 16- and > 128-node clusters. > > - Felix > -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Thu Sep 11 10:00:42 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Thu, 11 Sep 2003 16:00:42 +0200 (CEST) Subject: data storage location In-Reply-To: <3F607B21.5010904@tamu.edu> Message-ID: On Thu, 11 Sep 2003, Gerry Creager N5JXS wrote: > We have seen less than stellar performance with the older/cheaper ATI > Centrecom switches. We were almost completely unsuccessful with > multicast tests. And RFC2544 packet testing was the pits. So: small > packets are bad, and multicast appears problemmatic. At least on these > devices. I forgot to mention that on the same, little switch we had perfect scalability and performance with Dolly [1] (at least for large packets, which are the interesting ones for large data replications). - Felix [1] http://www.cs.inf.ethz.ch/~rauch/tmp/FE.CentreCom742i.Agg.pdf -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 11 10:27:08 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 11 Sep 2003 16:27:08 +0200 (CEST) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > > By the way, handhelds that can talk to a cmos/ccd camera and grab frames on > demand would be even better. Your wish is my command :-) http://www.interpocket.co.uk/ Probably you would have to team this with a cheap USB camera and a dual Compact Flash jacket. Total cost is of course higher. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathog at mendel.bio.caltech.edu Thu Sep 11 10:53:58 2003 From: mathog at mendel.bio.caltech.edu (David Mathog) Date: Thu, 11 Sep 2003 07:53:58 -0700 Subject: Kill the power faster than poweroff? Message-ID: Is there a faster way to shutdown a system and kill the power than "poweroff"? This is for use in emergency overheating conditions, where something that's effectively the equivalent of: % sync [turn off the power supply] is desired. lm_sensors and mondo are running on our Athlon 2200MP cluster and I'm concerned that in the event of a catastrophic failure of the cooling system the CPUs might overheat before the poweroff sequence completes. Not necessarily flame out immediately, but at least get hot enough so that they no longer run correctly, then poweroff wouldn't be able to complete, and then it's roasted CPU time. Tyan 2466 motherboards. This is only for emergency overheating, poweroff is fine for a normal shutdown. Thanks, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From asabigue at fing.edu.uy Thu Sep 11 06:19:38 2003 From: asabigue at fing.edu.uy (Ariel Sabiguero) Date: Thu, 11 Sep 2003 13:19:38 +0300 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <3F604C3A.3060008@fing.edu.uy> How about halt -p -f or poweroff -f .... I tried it and it takes 3s instead of 20 or more :-) From the man page: NOTES Under older sysvinit releases , reboot and halt should never be called directly. From release 2.74 on halt and reboot invoke shutdown(8) if the system is not in runlevel 0 or 6. This means that if halt or reboot cannot find out the cur? rent runlevel (for example, when /var/run/utmp hasn't been initialized correctly) shutdown will be called, which might not be what you want. Use the -f flag if you want to do a hard halt or reboot. It seems that it performs the sync call BEFORE killing the system but I think you should be careful. Regards Ariel David Mathog wrote: >Is there a faster way to shutdown a system and kill >the power than "poweroff"? This is for use in >emergency overheating conditions, where something >that's effectively the equivalent of: > > % sync > [turn off the power supply] > >is desired. > >lm_sensors and mondo are running on our Athlon 2200MP >cluster and I'm concerned that in the event of a >catastrophic failure of the cooling system the CPUs >might overheat before the poweroff sequence completes. >Not necessarily flame out immediately, but at least >get hot enough so that they no longer run correctly, >then poweroff wouldn't be able to complete, and then >it's roasted CPU time. > >Tyan 2466 motherboards. > >This is only for emergency overheating, poweroff is >fine for a normal shutdown. > >Thanks, > >David Mathog >mathog at caltech.edu >Manager, Sequence Analysis Facility, Biology Division, Caltech >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From asabigue at fing.edu.uy Thu Sep 11 07:04:18 2003 From: asabigue at fing.edu.uy (Ariel Sabiguero) Date: Thu, 11 Sep 2003 14:04:18 +0300 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <3F6056B2.2050303@fing.edu.uy> David Mathog wrote: >However it doesn't do so cleanly, neither >does: > > % sync ; poweroff -f > >In both cases there were inodes to fix and the disks were >rescanned on the next boot. Which is fine - much easier >to handle that sort of problem than fried hardware. > That depends on the data on disk. Sometimes it is cheaper to waste a CPU than to restore a highly complex DB. Maybe a journaled filesystem (jfs,ext3,reiser) could help at this point to give your filesystem the coherence that might get lost due to the poweroff. Good luck! Ariel > >Thanks, > >David Mathog >mathog at caltech.edu >Manager, Sequence Analysis Facility, Biology Division, Caltech > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathog at mendel.bio.caltech.edu Thu Sep 11 12:53:43 2003 From: mathog at mendel.bio.caltech.edu (David Mathog) Date: Thu, 11 Sep 2003 09:53:43 -0700 Subject: Kill the power faster than poweroff? Message-ID: > How about > > halt -p -f > > or > > poweroff -f > .... > I tried it and it takes 3s instead of 20 or more :-) You're right, that brings it down quickly, which is what is needed here. However it doesn't do so cleanly, neither does: % sync ; poweroff -f In both cases there were inodes to fix and the disks were rescanned on the next boot. Which is fine - much easier to handle that sort of problem than fried hardware. Thanks, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alan at dasher.wustl.edu Thu Sep 11 13:15:18 2003 From: alan at dasher.wustl.edu (alan at dasher.wustl.edu) Date: Thu, 11 Sep 2003 12:15:18 -0500 Subject: Kill the power faster than poweroff? In-Reply-To: Your message of "Thu, 11 Sep 2003 14:04:18 +0300." <3F6056B2.2050303@fing.edu.uy> Message-ID: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> :>However it doesn't do so cleanly, neither :>does: :> :> % sync ; poweroff -f :> :>In both cases there were inodes to fix and the disks were :>rescanned on the next boot. Which is fine - much easier :>to handle that sort of problem than fried hardware. I've read in other places that you need to repeat the sync command a number of times (I seem to remember the article saying 3 times in particular, though I can't remember how they arrived at that number) in order to guarantee writing. Even then it seems like there ought to be race-condition like issues. Still, have you tried sync; sleep 1; sync; poweroff -f Alan Grossfield --------------------------------------------------------------------------- | Only when we are willing to accept the universe for what it is, without | | myth or fear or prejudice, can we hope to build a truly just society. | | Lawrence Krauss, New York Times, 4/22/03 | --------------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 11 13:25:18 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 11 Sep 2003 10:25:18 -0700 Subject: cluster using handhelds In-Reply-To: References: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030911102311.018aaca0@mailhost4.jpl.nasa.gov> Most excellent.. Very intriguing, although I do note that the interpocket website doesn't mention having tested it with a camera... I would think that products like this will become more common, as the "cellphone camera" craze propagates.. At 04:27 PM 9/11/2003 +0200, John Hearns wrote: >On Wed, 10 Sep 2003, Jim Lux wrote: > > > > > By the way, handhelds that can talk to a cmos/ccd camera and grab > frames on > > demand would be even better. > >Your wish is my command :-) > >http://www.interpocket.co.uk/ > >Probably you would have to team this with a cheap USB camera and a dual >Compact Flash jacket. Total cost is of course higher. > > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 11 14:01:21 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 11 Sep 2003 11:01:21 -0700 Subject: cluster using handhelds In-Reply-To: <20030911175818.BBFF227C67@email.iwon.com> Message-ID: <5.2.0.9.2.20030911105852.030d2878@mailhost4.jpl.nasa.gov> At 01:58 PM 9/11/2003 -0400, eoster at iwon.com wrote: >James, > > > > >A good starting point is the Center for Embedded Netowrk Sensing (CENS) at >http://www.cens.ucla.edu/ . Indeed.. the paper in IEEE proceedings from CENS is what got me started thinking about iPAQs. >Depending on what you are looking for there might already be >communities. CENS has done projects on iPAQs (in particular) that >includes (but is in no way limited to) running FFTs. There's even a >development framework called Em* being offered for development on these >platforms. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Thu Sep 11 13:21:30 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 11 Sep 2003 12:21:30 -0500 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <1063300889.24932.20.camel@caffeine> On Thu, 2003-09-11 at 11:53, David Mathog wrote: > You're right, that brings it down quickly, which is what > is needed here. However it doesn't do so cleanly, neither > does: > > % sync ; poweroff -f > > In both cases there were inodes to fix and the disks were > rescanned on the next boot. Which is fine - much easier > to handle that sort of problem than fried hardware. Sounds like a good case for journalled filesystems. ;-) -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eoster at iwon.com Thu Sep 11 13:58:18 2003 From: eoster at iwon.com (eoster at iwon.com) Date: Thu, 11 Sep 2003 13:58:18 -0400 (EDT) Subject: cluster using handhelds Message-ID: <20030911175818.BBFF227C67@email.iwon.com> James, You might want to take a look at sensor networks. There's actually a lot of effort going into connecting small devices (among whom are specifically iPAQs) and doing distributed computing (in-network collaboration and processing). A good starting point is the Center for Embedded Netowrk Sensing (CENS) at http://www.cens.ucla.edu/ . Depending on what you are looking for there might already be communities. CENS has done projects on iPAQs (in particular) that includes (but is in no way limited to) running FFTs. There's even a development framework called Em* being offered for development on these platforms. Good luck, Eric =================== Has anyone tried building a cluster using handheld computers (i.e. iPaq or Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is there an implementation of MPI that will work in that environment (WinCE, PalmOS) (perhaps MPICH in some form?) Yes, I know the performance will be hideous. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 11 15:53:10 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 11 Sep 2003 15:53:10 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, David Mathog wrote: > You're right, that brings it down quickly, which is what > is needed here. However it doesn't do so cleanly, neither > does: > > % sync ; poweroff -f > > In both cases there were inodes to fix and the disks were > rescanned on the next boot. Which is fine - much easier > to handle that sort of problem than fried hardware. Inodes to fix on ext3? After a sync? Inquiring minds like to know... I'd expect a forced poweroff to be no worse than (and very similar to) just punching the power switch -- the sync should get the fs consistent EXCEPT for race condition output on open files belonging to active processes. Other ways of halting try to kill all those processes politely, in principle causing them to close all files that belong to them before the final sync. I'd also expect to not infrequently end up with a bunch of those little .nfs20800200 leftovers if the systems in question have active nfs mounts. rgb > > Thanks, > > David Mathog > mathog at caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From M.Arndt at science-computing.de Thu Sep 11 10:56:04 2003 From: M.Arndt at science-computing.de (Michael Arndt) Date: Thu, 11 Sep 2003 16:56:04 +0200 Subject: cluster using handhelds In-Reply-To: <3F6079E0.1020704@howard.edu>; from jhalpern@howard.edu on Thu, Sep 11, 2003 at 08:34:24AM -0500 References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> <3F6079E0.1020704@howard.edu> Message-ID: <20030911165604.A97778@blnsrv1.science-computing.de> someone asked related to "handheld" cluster http://www.handhelds.org/projects/skiffcluster.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Thu Sep 11 11:30:18 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Thu, 11 Sep 2003 10:30:18 -0500 (CDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: nice little snmpset to an APC Masterswitch will do the trick. -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' On Thu, 11 Sep 2003, David Mathog wrote: > Is there a faster way to shutdown a system and kill > the power than "poweroff"? This is for use in > emergency overheating conditions, where something > that's effectively the equivalent of: > > % sync > [turn off the power supply] > > is desired. > > lm_sensors and mondo are running on our Athlon 2200MP > cluster and I'm concerned that in the event of a > catastrophic failure of the cooling system the CPUs > might overheat before the poweroff sequence completes. > Not necessarily flame out immediately, but at least > get hot enough so that they no longer run correctly, > then poweroff wouldn't be able to complete, and then > it's roasted CPU time. > > Tyan 2466 motherboards. > > This is only for emergency overheating, poweroff is > fine for a normal shutdown. > > Thanks, > > David Mathog > mathog at caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ashley at pittman.co.uk Thu Sep 11 13:03:30 2003 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 11 Sep 2003 18:03:30 +0100 Subject: data storage location In-Reply-To: References: <200309101726.h8AHQId22018@NewBlue.Scyld.com> Message-ID: <1063299810.1150.287.camel@ashley> > The alternative approach is to keep copies of the data on local disk on > each node. This gives you good IO rates, but you then have a substantial > data management problem; how to you copy 100Gb to each node in your > cluster in a sensible amount of time, and how do you update the data and > make sure it is kept consistent? > > The commonest approach to data distribution is to do some sort of > cascading rsync/rcp which follows the topology of your network. I've often wondered why there isn't some kind of a 'mpicp' program to do just this. I'd imagine the command line to be something like $ mpirun -allcps mpicp node0:~/myfile.dat /tmp/ This would then use MPI_Bcast to send the data to all the nodes. The assumption here is that MPI_Bcast is fairly efficient anyway so it's best to use it rather than writing your own cascading rsync algorithm. Ashley, _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From purikk at hotmail.com Thu Sep 11 12:47:48 2003 From: purikk at hotmail.com (purushotham komaravolu) Date: Thu, 11 Sep 2003 12:47:48 -0400 Subject: Beowulf And jvm Message-ID: Hi, I am a newbie to beowulf. I have a question about running Java on a Beowulf cluster. Is it possible that I can start only one JVM on one machine and the task be run distributed on the cluster? It is a multi-threaded application. Like say, I have an application with 100 threads. Can I have 50 threads run on one machine and 50 on another by launching the application(jvm) on one machine? I dont want to use MPI or any special code. Has anyone tried something like this before? Thanks Regards, Puru -------------- next part -------------- An HTML attachment was scrubbed... URL: From becker at scyld.com Thu Sep 11 15:27:11 2003 From: becker at scyld.com (Donald Becker) Date: Thu, 11 Sep 2003 15:27:11 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> Message-ID: On Thu, 11 Sep 2003 alan at dasher.wustl.edu wrote: > I've read in other places that you need to repeat the sync command a > number of times (I seem to remember the article saying 3 times in > particular, though I can't remember how they arrived at that number) in > order to guarantee writing. Even then it seems like there ought to be > race-condition like issues. I was told over two decades ago. The principle was that the first sync queued the pages to write, and the second sync didn't start until the first write list had finished. The third was just to make sure. You know: if two is good, three must be better. I don't believe that a second 'sync' will have any beneficial effect with Linux, or any modern Unix. Somewhat over ten years ago most filesystems added an "I was unmounted cleanly" bit. What you really need to do unmount the file systems, which will have the effect of really doing what "sync" is supposed to do. > Still, have you tried > sync; sleep 1; sync; poweroff -f -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Thu Sep 11 16:03:45 2003 From: jakob at unthought.net (Jakob Oestergaard) Date: Thu, 11 Sep 2003 22:03:45 +0200 Subject: Kill the power faster than poweroff? In-Reply-To: References: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> Message-ID: <20030911200345.GB19677@unthought.net> On Thu, Sep 11, 2003 at 03:27:11PM -0400, Donald Becker wrote: > On Thu, 11 Sep 2003 alan at dasher.wustl.edu wrote: > > I've read in other places that you need to repeat the sync command a > > number of times (I seem to remember the article saying 3 times in > > particular, though I can't remember how they arrived at that number) in > > order to guarantee writing. Even then it seems like there ought to be > > race-condition like issues. > > I was told over two decades ago. The principle was that the first sync > queued the pages to write, and the second sync didn't start until the > first write list had finished. The third was just to make sure. You > know: if two is good, three must be better. The story I heard was, that sync could complete before all data had actually reached the platters (hardware write buffers or whatever...). Therefore, you had to *type* sync three times. No cheap scripting tricks - the whole idea of the two last syncs is to get some delay before you actually power off. > > I don't believe that a second 'sync' will have any beneficial effect > with Linux, or any modern Unix. And there is no state. Thus, the second sync must do exactly the same as the first sync. There is no way that it can have another effect. If you type sync three times, whatever write buffers there may be which the sync commands cannot sync, will have had time to flush. You could also type ]$ sync ]$ my ]$ disks ]$ please It just gives errors and makes you look stupid, if someone is watching over your shoulder ;) -- ................................................................ : jakob at unthought.net : And I see the elder races, : :.........................: putrid forms of man : : Jakob ?stergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Thu Sep 11 16:44:52 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Thu, 11 Sep 2003 13:44:52 -0700 Subject: Kill the power faster than poweroff? In-Reply-To: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> References: <3F6056B2.2050303@fing.edu.uy> <200309111715.h8BHFIXn000648@blitzen.wustl.edu> Message-ID: <20030911204452.GD1246@greglaptop.internal.keyresearch.com> On Thu, Sep 11, 2003 at 12:15:18PM -0500, alan at dasher.wustl.edu wrote: > I've read in other places that you need to repeat the sync command a > number of times (I seem to remember the article saying 3 times in > particular, though I can't remember how they arrived at that number) This is old advice, and it doesn't really apply to modern Linux or Unix. In Linux, the sync syscall and utility returns immediately after scheduling all the writes, instead of after completing them. In the good old days, there was probably a worry that if it took a while to write all the dirty blocks, there would be new dirty blocks, so 3 syncs in a row would pretty much have everything clean. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 11 17:13:10 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 11 Sep 2003 17:13:10 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, Donald Becker wrote: > Somewhat over ten years ago most filesystems added an "I was unmounted > cleanly" bit. What you really need to do unmount the file systems, > which will have the effect of really doing what "sync" is supposed to do. But I >>think<< that unless you kill all processes that own open files on the filesystem (or otherwise force a close) umount will complain about open files and not proceed even with e.g. the -f flag. If you do kill all processes to be sure you can run umount, then you're basically doing a full shutdown and back where you started. Or you can run lsof, filter out the files that are open for writing (and maybe reading -- don't know how umount manages files open for reading) and kill their controlling process. Quickly umount to avoid the a race on new processes opening files (which you could still lose, especially if the file owning processes are children of a process waiting to open another file owning process when its children terminate), cross your fingers, power down. Not horribly safe or elegant, but ought to get a good fraction of all open files especially on a node with a very controlled task list that owns open writeable files in the first place. I personally agree with other earlier remarks: use a journalling filesystem, do a sync to minimize at least some problems, and then just power off. You may well lose data in the write queues of open processes (as you would likely do anyway if you killed the processes) but you shouldn't end up with the system in a state that needs an fsck any more than you would if you just powered off without even the sync. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Sep 11 17:40:04 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Thu, 11 Sep 2003 17:40:04 -0400 Subject: Beowulf And jvm In-Reply-To: References: Message-ID: <1063316404.21466.7.camel@protein.scalableinformatics.com> Hi Puru: Java uses shared memory threads. It is unlikely that the beowulf architecture would be suitable for this. However, if you spawn multiple processes and create some sort of IPC mechanism (MPI is but one known functional instance of this), you might be able to get some benefit out of running on multiple CPUs across multiple nodes. If your threads are not latency sensitive, you could use a variety of IPC mechanisms. On Thu, 2003-09-11 at 12:47, purushotham komaravolu wrote: > Hi, > I am a newbie to beowulf. > I have a question about running Java on a Beowulf cluster. > Is it possible that I can start only one JVM on one machine and the > task > be run distributed on the cluster? It is a multi-threaded application. > Like say, I have an application with 100 threads. > Can I have 50 threads run on one machine and 50 on another by > launching > the application(jvm) on one machine? I dont want to use MPI or any > special code. > > Has anyone tried something like this before? > Thanks > Regards, > Puru -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Thu Sep 11 19:42:04 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Thu, 11 Sep 2003 16:42:04 -0700 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <20030911234204.GC2336@greglaptop.internal.keyresearch.com> On Thu, Sep 11, 2003 at 05:13:10PM -0400, Robert G. Brown wrote: > But I >>think<< that unless you kill all processes that own open files > on the filesystem (or otherwise force a close) umount will complain > about open files and not proceed even with e.g. the -f flag. If you do > kill all processes to be sure you can run umount, then you're basically > doing a full shutdown and back where you started. Yes, but. Linux shutdown kills everyone nicely. Kill -9 is much faster, and allows the umount. The *BSDs use death and distruction on the way down, and last I looked at it, took a couple of seconds to run their entire shutdown. No waiting for all the apps to call free() on all their data structures... swap in all their old, unused data pages... -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Fri Sep 12 00:35:28 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 12 Sep 2003 14:35:28 +1000 Subject: Kill the power faster than poweroff? In-Reply-To: <20030911200345.GB19677@unthought.net> References: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> <20030911200345.GB19677@unthought.net> Message-ID: <200309121435.30380.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 12 Sep 2003 06:03 am, Jakob Oestergaard wrote: > Therefore, you had to *type* sync three times. No cheap scripting > tricks - the whole idea of the two last syncs is to get some delay > before you actually power off. This certainly used to be the case. It's just a handy habit and if it doesn't cost you anything then why not. I know my 2.4.x Linux boxes show disk activity when I type sync, so it's certainly not worthless. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/YU0RO2KABBYQAh8RAnArAJ4uWHJsim9RmqTQJNh09dW/i9ZqTgCeOKmY qJOxQPL8X33HWpWJO4O3cwk= =YnHT -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Fri Sep 12 05:13:16 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Fri, 12 Sep 2003 11:13:16 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, Robert G. Brown wrote: > But I >>think<< that unless you kill all processes that own open files > on the filesystem (or otherwise force a close) umount will complain > about open files and not proceed even with e.g. the -f flag. If you do > kill all processes to be sure you can run umount, then you're basically > doing a full shutdown and back where you started. If your processes only *read* data, you could also try mount -o ro,/path to remount the filesystem read-only before turning the power off. This should write all pending blocks to disk and set the file system's clean bit. I haven't tried it, so it might fail when there are still processes with files open for writing on that file system. - Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 12 05:53:57 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 12 Sep 2003 11:53:57 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, David Mathog wrote: > Is there a faster way to shutdown a system and kill > the power than "poweroff"? This is for use in > emergency overheating conditions, where something > that's effectively the equivalent of: > You can use APC power distribution units. These are network-addressable and can switch on and off power to a set of outlets. We use them on our clusters. Quick look at the the Opteron cluster next to me - model number is AP9212. (I'm on customer site today) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 12 06:00:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 12 Sep 2003 12:00:11 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: ps. given all the discussion on syncing and journalled filesystems, I should say that I mention the APC units in direct response to the original question on alternative ways for fast shutdown. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Fri Sep 12 07:39:59 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Fri, 12 Sep 2003 13:39:59 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Fri, 12 Sep 2003, Juan Gallego wrote: > On 2003-09-12 11:13+0200, Felix Rauch wrote: > | mount -o ro,/path > > `mount -o ro,remount' /path maybe? Uhm, exactly, that's what I actually wanted to write... Thanks and regards, Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Fri Sep 12 06:00:57 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Fri, 12 Sep 2003 11:00:57 +0100 Subject: Kill the power faster than poweroff? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE11F@stegosaurus.bristol.quadrics.com> David, For information, both flavours of Opteron that I have here have an out-of-band BMC that I can issue remote power-on/ power-off commands to over ethernet.. Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- > -----Original Message----- > From: John Hearns [mailto:john.hearns at clustervision.com] > Sent: 12 September 2003 10:54 > To: beowulf at beowulf.org > Subject: Re: Kill the power faster than poweroff? > > > On Thu, 11 Sep 2003, David Mathog wrote: > > > Is there a faster way to shutdown a system and kill > > the power than "poweroff"? This is for use in > > emergency overheating conditions, where something > > that's effectively the equivalent of: > > > You can use APC power distribution units. > These are network-addressable and can switch on and off power to > a set of outlets. We use them on our clusters. > > > Quick look at the the Opteron cluster next to me - > model number is AP9212. (I'm on customer site today) > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Sep 12 06:55:43 2003 From: becker at scyld.com (Donald Becker) Date: Fri, 12 Sep 2003 06:55:43 -0400 (EDT) Subject: Beowulf And jvm In-Reply-To: <1063316404.21466.7.camel@protein.scalableinformatics.com> Message-ID: On Thu, 11 Sep 2003, Joseph Landman wrote: > Java uses shared memory threads. ...and it doesn't structure the sharing of information between threads. > It is unlikely that the beowulf architecture would be suitable for > this. There are Network Virtual Memory (*) systems that will emulate shared memory using virtual memory page protections. But they are very slow unless you restructure your application to avoid page bounces, and when you do that you end up with a system that is equivalent to doing message passing. (*) AKA DSM -- Distributed Shared Memory, but that name is also used for hardware-based, per-word memory coherency) > However, if you spawn multiple > processes and create some sort of IPC mechanism (MPI is but one known > functional instance of this), you might be able to get some benefit out > of running on multiple CPUs across multiple nodes. If your threads are > not latency sensitive, you could use a variety of IPC mechanisms. There are several Java-specific systems, such as JavaSpaces, that provide communication libraries. In general - You must still restructure the code - Support for low latency interconnects does not exist + They usually have a clean Java interface (vs. a MPI wrapper) My impression from others (disclaimer: not being a Java programmer myself), is that cluster Java applications are written with explicit distributed layout. Independent address spaces, which may be multithreaded, are run on the nodes and communicate without using a cluster-specific library. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Sep 12 09:54:17 2003 From: becker at scyld.com (Donald Becker) Date: Fri, 12 Sep 2003 09:54:17 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: <010C86D15E4D1247B9A5DD312B7F5AA78DE11F@stegosaurus.bristol.quadrics.com> Message-ID: On Fri, 12 Sep 2003, Daniel Kidger wrote: > For information, both flavours of Opteron that I have here have an > out-of-band BMC that I can issue remote power-on/ power-off commands to over > ethernet.. I'm a big fan of network IPMI 1.5, which is usually implemented by a Baseboard Management Controller (BMC). The BMC is a microcontroller running on stand-by power. It can control power, redirect the "serial" console to the network, monitor temp/voltage/switches, configure the BIOS and do a few other things. The cleanest IPMI implementation uses an enhanced Ethernet controller that redirects IPMI packets to the BMC instead of the OS. This allows installing a machine by plugging in only a power cord and Ethernet cable. Everything else may be done over the network. But my understanding is that the issue here is the software shutdown time. The big culprit in the slow shutdown is the daemons. We solved this problem for cluster compute nodes in the Scyld system by developing a unique system that does not require daemons. Thus we can cleanly shut down a slave node almost instantly. But we want a standard, full-featured Linux installation as a cluster master, and have the same slow shutdown issue there. So why does a 3GHz machine takes 10X more time to boot and shutdown than a slow machine with an earlier Linux installation? The biggest culprits are /etc/rc.d/init.d/functions /etc/rc.d/init.d/halt which do explicit sleeps and have very poor scaling. Fixing the fundamental problems is difficult (you can just revert to an older version), but it's easy to pick better numbers for the 'usleep' and 'sleep' calls. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Fri Sep 12 10:06:39 2003 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Fri, 12 Sep 2003 09:06:39 -0500 Subject: java and cluster Message-ID: <20030912090639.G5739@mikee.ath.cx> Accidently sent this directly to Donald Becker: On Fri, 12 Sep 2003, Donald Becker wrote: > My impression from others (disclaimer: not being a Java programmer > myself), is that cluster Java applications are written with explicit > distributed layout. Independent address spaces, which may be > multithreaded, are run on the nodes and communicate without using a > cluster-specific library. I used a JNI wrapper around PVM for my applications. Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathog at mendel.bio.caltech.edu Fri Sep 12 11:11:12 2003 From: mathog at mendel.bio.caltech.edu (David Mathog) Date: Fri, 12 Sep 2003 08:11:12 -0700 Subject: Kill the power faster than poweroff? Message-ID: On Fri, 12 Sep 2003 becker at scyld.com wrote: > The biggest culprits are > /etc/rc.d/init.d/functions > /etc/rc.d/init.d/halt > which do explicit sleeps and have very poor scaling. Replace the explicit seconds/microseconds with variables and conditionally set those variables to zero if "/fastshutdown" exists? Then a regular "poweroff" would execute without all the sleeps. I wonder though if it would work, since some of those sleeps are in there to let daemons shut down nicely before going on to the next thing. I'll have to try this at some point nd see what happens. Regards, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Fri Sep 12 11:34:42 2003 From: jcownie at etnus.com (James Cownie) Date: Fri, 12 Sep 2003 11:34:42 -0400 Subject: Beowulf And jvm In-Reply-To: Message from Donald Becker of "Fri, 12 Sep 2003 06:55:43 EDT." Message-ID: <200309121534.h8CFYgc32508@alien.etnus.com> If you want to rewrite code into explicitly parallel Java there's Berkeley's Titanium... http://www.cs.berkeley.edu/Research/Projects/titanium/ Titanium is an explicitly parallel dialect of Java developed at UC Berkeley to support high-performance scientific computing on large-scale multiprocessors, including massively parallel supercomputers and distributed-memory clusters with one or more processors per node. Other language goals include safety, portability, and support for building complex data structures. ... etc ... -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From abijur at craft-tech.com Fri Sep 12 12:18:29 2003 From: abijur at craft-tech.com (Ashwin Bijur) Date: Fri, 12 Sep 2003 12:18:29 -0400 Subject: channel bonding/trunking and diskless booting Message-ID: <3F61F1D5.5040504@craft-tech.com> Hi, Can anybody help me with the steps required to configure a linux cluster node with channel bonding (or trunking) and diskless booting? We use Redhat Linux 8.0, Pentium III dual processor PCs. Any help is greatly appreciated. Thanks, Ashwin Bijur Asst. Systems Administrator. Combustion Research and Flow Technology, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Fri Sep 12 12:43:27 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Fri, 12 Sep 2003 18:43:27 +0200 Subject: data storage location In-Reply-To: References: <200309101726.h8AHQId22018@NewBlue.Scyld.com> Message-ID: <20030912184327Q.hanzl@unknown-domain> > The alternative approach is to keep copies of the data on local disk on > each node. This gives you good IO rates, but you then have a substantial > data management problem; how to you copy 100Gb to each node in your > cluster in a sensible amount of time, and how do you update the data and > make sure it is kept consistent? > > ... > > If your dataset is larger than the amount of local disk on your nodes, you > then have to partition your data up, and integrate that with your queuing > system, so that jobs which need a certain bit of the data end up on a node > which actually holds a copy. This is exactly what we do. But moving the right data at right place and doing good job scheduling at the same time is not easy. Ideally we would like to automate it via huge caches on local disks: - central server has some 400GB of read-only data - nodes cache them on their harddisks as needed - queing system preferes some regular patterns in job/node assignment to make cache hits likely Cache-like behavior would save a lot of manual work but unfortunately I am not aware of any working solution for linux, I want something like cachefs (nonexistent for linux) or caching ability of AFS/Coda (too cumbersome for cluster) or theoretical features of Intermezzo (still and maybe forever unfinished). At the moment we work on small kernel hack to solve this, unfortunately repeating for the n-th time what many others once did and never maintained. Maybe genome research will generate more need for this data access pattern and more chance for re-usable software solution? Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Fri Sep 12 13:04:14 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Fri, 12 Sep 2003 13:04:14 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: > I'm a big fan of network IPMI 1.5, which is usually implemented by a > Baseboard Management Controller (BMC). The BMC is a microcontroller > running on stand-by power. It can control power, redirect the "serial" IPMI and BMC are attractive, but as far as I can tell, completely non-viable for moderately-priced nodes. I mean, if my 1-u dual costs $US 2k, adding a BMC probably means a 35-50% boost in price. or are there cheap BMC options? otherwise, it seems like a holdover from the world of "enterprise" gold-plated servers, where $1k on a $50K server is in the noise... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Fri Sep 12 13:20:34 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Fri, 12 Sep 2003 10:20:34 -0700 (PDT) Subject: data storage location In-Reply-To: <20030912184327Q.hanzl@unknown-domain> Message-ID: On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > The alternative approach is to keep copies of the data on local disk on > > each node. This gives you good IO rates, but you then have a substantial > > data management problem; how to you copy 100Gb to each node in your > > cluster in a sensible amount of time, and how do you update the data and > > make sure it is kept consistent? > > > > ... > > > > If your dataset is larger than the amount of local disk on your nodes, you > > then have to partition your data up, and integrate that with your queuing > > system, so that jobs which need a certain bit of the data end up on a node > > which actually holds a copy. > > This is exactly what we do. But moving the right data at right place > and doing good job scheduling at the same time is not easy. Ideally we > would like to automate it via huge caches on local disks: > > - central server has some 400GB of read-only data > - nodes cache them on their harddisks as needed > - queing system preferes some regular patterns in job/node assignment > to make cache hits likely > > Cache-like behavior would save a lot of manual work but unfortunately > I am not aware of any working solution for linux, I want something > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > (too cumbersome for cluster) or theoretical features of > Intermezzo (still and maybe forever unfinished). it's not clear to me that coda is too cumbersome for clusters... but the other approach to scale the storage infrastructure to the point where it's performance approaches that of locally attached disk. there's a lot of work that's been done on san and iscsi architecture for storage clusters that seems relevant to the issues of compute clusters as well... > At the moment we work on small kernel hack to solve this, > unfortunately repeating for the n-th time what many others once did > and never maintained. > > Maybe genome research will generate more need for this data access > pattern and more chance for re-usable software solution? > > Regards > > Vaclav Hanzl > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeff at aslab.com Fri Sep 12 16:03:59 2003 From: jeff at aslab.com (Jeff Nguyen) Date: Fri, 12 Sep 2003 13:03:59 -0700 Subject: Kill the power faster than poweroff? References: Message-ID: <20a001c37969$01b3bf70$6502a8c0@jeff> > > I'm a big fan of network IPMI 1.5, which is usually implemented by a > > Baseboard Management Controller (BMC). The BMC is a microcontroller > > running on stand-by power. It can control power, redirect the "serial" > > IPMI and BMC are attractive, but as far as I can tell, completely > non-viable for moderately-priced nodes. I mean, if my 1-u dual costs > $US 2k, adding a BMC probably means a 35-50% boost in price. > > or are there cheap BMC options? otherwise, it seems like a holdover > from the world of "enterprise" gold-plated servers, where $1k on a > $50K server is in the noise... There is low cost IPMI/BMC interface at a fraction of the system cost. Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Fri Sep 12 13:23:43 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Fri, 12 Sep 2003 18:23:43 +0100 Subject: Kill the power faster than poweroff? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE129@stegosaurus.bristol.quadrics.com> > > I'm a big fan of network IPMI 1.5, which is usually implemented by a > > Baseboard Management Controller (BMC). The BMC is a microcontroller > > running on stand-by power. It can control power, redirect > the "serial" > > IPMI and BMC are attractive, but as far as I can tell, completely > non-viable for moderately-priced nodes. I mean, if my 1-u dual costs > $US 2k, adding a BMC probably means a 35-50% boost in price. > > or are there cheap BMC options? otherwise, it seems like a holdover > from the world of "enterprise" gold-plated servers, where $1k on a > $50K server is in the noise... I would suggest that the majority of new dual CPU rack mounted nodes have BMCs on the MoBo. Certainly all the 'big band-name' ones seem to. Intel ones tend to have the BMC hijacking the main eth0. Others (Serverworks, Khepri, AMD, ZX1 et al) tend to use a seperate ethernet socket or hijack the serial port. Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Sep 12 17:50:35 2003 From: becker at scyld.com (Donald Becker) Date: Fri, 12 Sep 2003 17:50:35 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Fri, 12 Sep 2003, Mark Hahn wrote: > > I'm a big fan of network IPMI 1.5, which is usually implemented by a > > Baseboard Management Controller (BMC). The BMC is a microcontroller > > running on stand-by power. It can control power, redirect the "serial" > > IPMI and BMC are attractive, but as far as I can tell, completely > non-viable for moderately-priced nodes. I mean, if my 1-u dual costs > $US 2k, adding a BMC probably means a 35-50% boost in price. I wouldn't be a big fan if either it was too expensive to be common it was proprietary I've seen quotes for around $30 extra when bundled with the system board. Of course finding a BMC daughterboard later is difficult and likely expensive, so you have to specify it with the system. > or are there cheap BMC options? otherwise, it seems like a holdover > from the world of "enterprise" gold-plated servers, where $1k on a > $50K server is in the noise... And you thought the option -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 13 05:34:12 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 13 Sep 2003 11:34:12 +0200 (CEST) Subject: data storage location In-Reply-To: <20030912184327Q.hanzl@unknown-domain> Message-ID: On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > Cache-like behavior would save a lot of manual work but unfortunately > I am not aware of any working solution for linux, I want something > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > (too cumbersome for cluster) or theoretical features of Why do you say AFS is too cumbersome? I'm just saying that to provoke a debate - I've never actually set up an AFS infrastructure from scratch (kerberos, kernel patches etc...) but I know its not an afternoon's work. I have had call to work closely with AFS, doing a bit of performance tuning for caches etc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 13 05:39:02 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 13 Sep 2003 11:39:02 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Fri, 12 Sep 2003, Donald Becker wrote: HAs anyone got specifics for BMCs for HDAMA motherboards? Are they now available, cost etc. In fact, a nice summary of BMC capabilities, and which motherboards have or support them would be excellent. I guess it may be a little early in the game for that, though pointers welcome. Also, if anyoen has got lm_sensors working on HDAMA with a 2.4.22 kernel I'd be interested to chat off-list. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Sat Sep 13 07:56:38 2003 From: becker at scyld.com (Donald Becker) Date: Sat, 13 Sep 2003 07:56:38 -0400 (EDT) Subject: data storage location In-Reply-To: <20030912184327Q.hanzl@unknown-domain> Message-ID: On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > The alternative approach is to keep copies of the data on local disk on > > each node. This gives you good IO rates, but you then have a substantial > > data management problem; how to you copy 100Gb to each node in your > > cluster in a sensible amount of time, and how do you update the data and > > make sure it is kept consistent? One significant question is the access pattern and requirement. Is there a single or multiple application access patterns? Does the application read all or only a subset of the data? Is the subset predictable? Externally by a scheduler? Does the application step linearly through the data, or randomly access? If linear stepping, is the state space of the application small? If small, as with a best-match search, the processing time per file byte tends to be small. We recommend a static split of the data across machines and migrating the process instead. In the case of a single file read we can often do this without modifying the application, or localize the changes to a few lines around the read loop. If large, e.g. building a tree in memory from the data, what is the per-byte processing time? across machines and migrate the process. How is the data set updated? A read-only data set allows many file handling options. If files are updated as a whole, you may use a caching versioned file system. That is specialized, but provides many opportunities for optimization Handling arbitrary writes in the middle of files requires a consistent file system, and the cost for consistency is very high. > Cache-like behavior would save a lot of manual work but unfortunately > I am not aware of any working solution for linux, Several exist. It depends on the semantics you need, as doing this efficiently requires making assumptions about the access patterns and desired semantics. > or caching ability of AFS/Coda > (too cumbersome for cluster) or theoretical features of > Intermezzo (still and maybe forever unfinished). ...Declare it a success and move on -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Sat Sep 13 09:16:28 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 13 Sep 2003 09:16:28 -0400 Subject: data storage location In-Reply-To: References: Message-ID: <1063458985.31562.9.camel@QUIGLEY.LINIAC.UPENN.EDU> On Sat, 2003-09-13 at 05:34, John Hearns wrote: > On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > I am not aware of any working solution for linux, I want something > > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > > (too cumbersome for cluster) or theoretical features of > Why do you say AFS is too cumbersome? > I'm just saying that to provoke a debate - I've never actually set up > an AFS infrastructure from scratch (kerberos, kernel patches etc...) > but I know its not an afternoon's work. > I have had call to work closely with AFS, doing a bit of performance > tuning for caches etc. Ok, I'll bite :) We have been exploring filesystem options here, trying to get away from NFS. Our 'architecture' is 128 nodes with FastE, and 4 IO servers, each with ~600 GB of RAID5 and GigE. At first glance, AFS, really OpenAFS in this case, seemed like a dream come true. There was the ability to have mutliple file servers in a global namespace, support for doing amanda backups, could let users export their own data to any machine that is an AFS client -- even across the country/world. Now... for the practical side. When I went to implement it, I was unaware how much a pain it could be to setup the kerberos stuff. At Penn we have a campus wide krb5 setup, and getting users to use krb5 logins was not a problem. However, openAFS needs a krb4 ticket to work, and Penn will not add the krb524 translator. To get around this, it was necessary to do a bunch of hacks to store the krb4 ticket on the machine itself, and tell krb5 that localhost was a valid krb524 server. This may not sound too bad, but to figure it out, and then to implement correctly is such a major PITA, I almost dumped the project there. So, now that it is talkint to the kerberos setup just fine, it was very apparent that having your ticket timeout was a very annoying problem. -- Basically you have to do some really ugly script/hack to have a program remember all of the user's passwords, either in memory or in a gpg protected file, and periodically check for a ticket that was about to expire, and re klogin for that user. Not pretty. The second issue, and the real deal killer is the lack of support for >2GB files. Absolutely unacceptable. We do natural language processing on one cluster, and genomics on another, with files that regularily exceed 2GB. So... in conclusion. IF openAFS could support >2GB files, and the krb5 mess could be cleanedup, it might work. For now, we will be installing & playing with Lustre. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 13 10:13:54 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 13 Sep 2003 10:13:54 -0400 (EDT) Subject: data storage location In-Reply-To: Message-ID: On Sat, 13 Sep 2003, John Hearns wrote: > On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > I am not aware of any working solution for linux, I want something > > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > > (too cumbersome for cluster) or theoretical features of > Why do you say AFS is too cumbersome? > I'm just saying that to provoke a debate - I've never actually set up > an AFS infrastructure from scratch (kerberos, kernel patches etc...) > but I know its not an afternoon's work. > I have had call to work closely with AFS, doing a bit of performance > tuning for caches etc. AFS is in wide use at Duke (it is the basis of the students' globally shared home directory system on campus). It is doable administratively, although as you note it isn't an afternoon's work. However, it isn't likely to be a really good cluster fs. Reasons: a) It is even less efficient and more resource intensive than NFS (unsurprising, as it does more) b) It can (unless you take care with e.g. kerberos tickets) produce "odd" and undesirable behavior such as losing the ability to write to a file you are holding open after x hours unless/until you re-authenticate, blocking the process that owns the open file c) Its caching behavior is insane and (in my opinion) unreliable, at least as of the last time I tried it. By this I mean that I have directly experienced the following AFS bug/feature: System A opens file output in AFS directory System A writes lots of stuff into output System A fflushes and keeps running, output is still open System B wants to graze the current contents of output, so it opens output for reading. What does system B find? According to standard/reliable programming conventions, the use of fflush should force a write of all cached data back to the file. AFS, however, only flushes the data (apparently) back to its LOCAL cache/image of the file on system A, and only resync's with the global image when the file is closed. So System B will read nothing of what System A has written until A closes the file. So sure, system A could close and open the file repeatedly, but this adds a lot of overhead as open/close is expensive (open requires a full stat, checking permissions, etc.). Most of this is manageable and one can learn how to work with the system in a cluster environment for at least certain classes of task, but it is (as the man says:-) "cumbersome", as to a certain extent is administration (file acls are more powerful but also require more thought and human energy, etc). Too cumbersome is up to personal judgement. Many clusters use a user filesystem only to launch tasks and permit a single results file to eventually be written back -- "anything" can be made to work there, and AFS would work as well as NFS or whatever. Other cluster tasks might well be unsuitable as in my system A/B example, although with foreknowledge this can be coped with. Hope this helps. I personally no longer use AFS so perhaps the fflush "bug" has been fixed, and we have/do/are still meditating upon using AFS (since we have it running) in at least some capacity on a public cluster we're engineering, but it isn't a knee-jerk thing to do EVEN if it is already there, and will definitely require a bit of consideration if you have to set it up from scratch to use it at all. rgb > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rodmur at maybe.org Sat Sep 13 10:48:14 2003 From: rodmur at maybe.org (Dale Harris) Date: Sat, 13 Sep 2003 07:48:14 -0700 Subject: data storage location In-Reply-To: <1063458985.31562.9.camel@QUIGLEY.LINIAC.UPENN.EDU> References: <1063458985.31562.9.camel@QUIGLEY.LINIAC.UPENN.EDU> Message-ID: <20030913144814.GB6831@maybe.org> On Sat, Sep 13, 2003 at 09:16:28AM -0400, Nicholas Henke elucidated: > > Ok, I'll bite :) We have been exploring filesystem options here, trying > to get away from NFS. Our 'architecture' is 128 nodes with FastE, and 4 > IO servers, each with ~600 GB of RAID5 and GigE. > Any impressions on NFS4? I notice it's included, at least experimentally, in the 2.6 kernel. -- Dale Harris rodmur at maybe.org /.-) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From epaulson at cs.wisc.edu Sat Sep 13 11:10:11 2003 From: epaulson at cs.wisc.edu (Erik Paulson) Date: Sat, 13 Sep 2003 10:10:11 -0500 Subject: data storage location In-Reply-To: ; from rgb@phy.duke.edu on Sat, Sep 13, 2003 at 10:13:54AM -0400 References: Message-ID: <20030913101008.A3238@chopin.cs.wisc.edu> On Sat, Sep 13, 2003 at 10:13:54AM -0400, Robert G. Brown wrote: > On Sat, 13 Sep 2003, John Hearns wrote: > > > On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > > > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > > I am not aware of any working solution for linux, I want something > > > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > > > (too cumbersome for cluster) or theoretical features of > > Why do you say AFS is too cumbersome? > > I'm just saying that to provoke a debate - I've never actually set up > > an AFS infrastructure from scratch (kerberos, kernel patches etc...) > > but I know its not an afternoon's work. > > I have had call to work closely with AFS, doing a bit of performance > > tuning for caches etc. > > AFS is in wide use at Duke (it is the basis of the students' globally > shared home directory system on campus). It is doable administratively, > although as you note it isn't an afternoon's work. However, it isn't > likely to be a really good cluster fs. Reasons: > > a) It is even less efficient and more resource intensive than NFS > (unsurprising, as it does more) > > b) It can (unless you take care with e.g. kerberos tickets) produce > "odd" and undesirable behavior such as losing the ability to write to a > file you are holding open after x hours unless/until you > re-authenticate, blocking the process that owns the open file > First of all, you should be relaying on your batch system to manage the credentials for your job for you. Then, you really should be prepared to deal with that anyway - an NFS server can go away at any time too. > c) Its caching behavior is insane and (in my opinion) unreliable, at > least as of the last time I tried it. By this I mean that I have > directly experienced the following AFS bug/feature: > > System A opens file output in AFS directory > System A writes lots of stuff into output > System A fflushes and keeps running, output is still open > System B wants to graze the current contents of output, so it opens > output for reading. > > What does system B find? According to standard/reliable programming > conventions, the use of fflush should force a write of all cached data > back to the file. AFS, however, only flushes the data (apparently) back > to its LOCAL cache/image of the file on system A, and only resync's with > the global image when the file is closed. So System B will read nothing > of what System A has written until A closes the file. > > So sure, system A could close and open the file repeatedly, but this > adds a lot of overhead as open/close is expensive (open requires a full > stat, checking permissions, etc.). > > Most of this is manageable and one can learn how to work with the system > in a cluster environment for at least certain classes of task, but it is > (as the man says:-) "cumbersome", as to a certain extent is > administration (file acls are more powerful but also require more > thought and human energy, etc). Too cumbersome is up to personal > judgement. Many clusters use a user filesystem only to launch tasks and > permit a single results file to eventually be written back -- "anything" > can be made to work there, and AFS would work as well as NFS or > whatever. Other cluster tasks might well be unsuitable as in my system > A/B example, although with foreknowledge this can be coped with. > > Hope this helps. I personally no longer use AFS so perhaps the fflush > "bug" has been fixed, None of the AFS people will tell you it's a bug. AFS is a naturally more file-oriented system - AFS caches whole files, not subblocks of the file, so it makes sense that changes are propgated back to the server only at close() (and hey, watch out - close can fail - how many people actually check the return code from close?). AFS is probably a big win for regular users, where there's not much active sharing between processes. However, if you're using the filesystem for coordination between multiple processes, AFS is not for you. > and we have/do/are still meditating upon using AFS > (since we have it running) in at least some capacity on a public cluster > we're engineering, but it isn't a knee-jerk thing to do EVEN if it is > already there, and will definitely require a bit of consideration if you > have to set it up from scratch to use it at all. > For a shared global namespace filesystem that actually does some sort of authentication, AFS is really the only game in town. I can't imagine doing a campus-wide NFS or PVFS setup... -Erik _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bernd-schubert at web.de Sat Sep 13 11:43:26 2003 From: bernd-schubert at web.de (Bernd Schubert) Date: Sat, 13 Sep 2003 17:43:26 +0200 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <200309131743.26320.bernd-schubert@web.de> On Thursday 11 September 2003 16:53, David Mathog wrote: > Is there a faster way to shutdown a system and kill > the power than "poweroff"? This is for use in > emergency overheating conditions, where something > that's effectively the equivalent of: > > % sync > [turn off the power supply] > Hi, since linux-2.4.21 the magic-sysrq's can be trigger via keyboard. echo s > /proc/sysrq-trigger echo u >/proc/sysrq-trigger echo o /proc/sysrq-trigger takes less then a second to turn my system off. Cheers, Bernd _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 13 17:34:13 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 13 Sep 2003 17:34:13 -0400 (EDT) Subject: data storage location In-Reply-To: <20030913101008.A3238@chopin.cs.wisc.edu> Message-ID: On Sat, 13 Sep 2003, Erik Paulson wrote: > None of the AFS people will tell you it's a bug. AFS is a naturally > more file-oriented system - AFS caches whole files, not subblocks of the > file, so it makes sense that changes are propgated back to the server only > at close() (and hey, watch out - close can fail - how many people actually > check the return code from close?). AFS is probably a big win for regular > users, where there's not much active sharing between processes. However, > if you're using the filesystem for coordination between multiple processes, > AFS is not for you. Well, I did put bug/feature together, or used quotes, because I know they don't think it is a bug, but I still get/got somewhat annoyed when standard systems calls behave(d) in unexpected ways. Technically, of course, fflush only flushes user level buffers to the kernel, but leaves it to the kernel to flush them back to the actual file (which it does according to the dictates of the filesystem and so forth. So technically it isn't a real bug, just an arguable/debateable design decision. Some stuff like this made more sense a decade ago than it does now, as well, as overall capacities of a cheap >>client<< now exceed the capacity of most of the most expensive >>servers<< in use back then by a Moore's Law Mile. The feature can bite the hell out of you anyway, giving a whole new meaning to the term "race condition" if you're not aware of the possibility. As in one usually thinks of a race condition as being "short", and NFS is very good, really, at ensuring that any cached writebacks are written physically to disk (or serviced out of its cache) if you write on A and read on B. It is "difficult" to catch the system in the window in between a write and the point where the read will read the correct data. With AFS the race isn't just tortoise and hare, it can actually be against a creature that is virtually sessile... It's fixed now anyway, or there is a workaround (I just logged into two systems that share my campus AFS home directory to try it out). If you do int fd; fd = open("dummy",O_WRONLY); printf("Writing out to dummy.\n"); write(fd,"This is written out.",21); fsync(fd); fprintf(stdout,"Done.\n"); sleep(30); on system A now, the fsync actually forces a kernel level write through. fflush on files opened with fopen still doesn't work. My recollection of the last time I tried this (quite a few years ago) is that neither of them would work, which I would have (and did) call a bug. So now you can force a writeback at the expense of using low level file I/O. Unless somebody knows how to get the low level file descriptor associated with a high level file pointer (if there is any such beast to get) -- I've never tried to do this. > > and we have/do/are still meditating upon using AFS > > (since we have it running) in at least some capacity on a public cluster > > we're engineering, but it isn't a knee-jerk thing to do EVEN if it is > > already there, and will definitely require a bit of consideration if you > > have to set it up from scratch to use it at all. > > > > For a shared global namespace filesystem that actually does some sort of > authentication, AFS is really the only game in town. I can't imagine doing a > campus-wide NFS or PVFS setup... No, we agree, which is why we're thinking seriously about it. We may end up doing both -- permit AFS access to a "head node" so people can connect their own account space to the cluster across the campus WAN, but use something else on the "inside" of the head node/firewall. Or something else entirely. I'd welcome suggestions -- we're having a fairly major meeting on the project in a week or two. Real Life experiences of people who use AFS across a cluster would be lovely. Especially experiences or tests/benchmarks regarding things like efficiency and system overhead/load/requirements, and any caveats or gotchas to be overcome (given by hypothesis a longstanding, well managed campus-wide AFS setup -- at least seven or eight years old, but I don't really remember when they started using AFS and it might be a fair bit longer). rgb > > -Erik > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 13 19:17:27 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sun, 14 Sep 2003 01:17:27 +0200 (CEST) Subject: data storage location In-Reply-To: Message-ID: On Sat, 13 Sep 2003, Robert G. Brown wrote: > On Sat, 13 Sep 2003, Erik Paulson wrote: > > Or something else entirely. I'd welcome suggestions -- we're having a > fairly major meeting on the project in a week or two. Real Life > experiences of people who use AFS across a cluster would be lovely. > Especially experiences or tests/benchmarks regarding things like > efficiency and system overhead/load/requirements, and any caveats or > gotchas to be overcome (given by hypothesis a longstanding, well managed AFS is heavily used at CERN. They have requirements for access to an international base of users, and have a kerberos infrastructure in place. IIRC, they still use V4 for the reasons mentioned earlier in the thread. AFS is used across the computational cluster, which is where I met it. The cluster is batch oriented - not a parallel Beowulf type. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From iosephus at sgirmn.pluri.ucm.es Sun Sep 14 05:40:36 2003 From: iosephus at sgirmn.pluri.ucm.es (=?iso-8859-1?Q?Jos=E9_M=2E_P=E9rez_S=E1nchez?=) Date: Sun, 14 Sep 2003 11:40:36 +0200 Subject: Beowulf And jvm In-Reply-To: References: Message-ID: <20030914094036.GA19325@sgirmn.pluri.ucm.es> On Thu, Sep 11, 2003 at 12:47:48PM -0400, purushotham komaravolu wrote: > Hi, > I am a newbie to beowulf. > I have a question about running Java on a Beowulf cluster. > Is it possible that I can start only one JVM on one machine and the task > be run distributed on the cluster? It is a multi-threaded application. > Like say, I have an application with 100 threads. > Can I have 50 threads run on one machine and 50 on another by launching > the application(jvm) on one machine? I dont want to use MPI or any special code. > > Has anyone tried something like this before? > Thanks > Regards, > Puru A colleague of mine who makes parallel Java programming uses the 'Colt Distribution', maybe it can help you: http://hoschek.home.cern.ch/hoschek/colt/ Regards, Jose M. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bvds at bvds.geneva.edu Sat Sep 13 17:43:46 2003 From: bvds at bvds.geneva.edu (bvds at bvds.geneva.edu) Date: Sat, 13 Sep 2003 17:43:46 -0400 (EDT) Subject: channel bonding/trunking and diskless booting Message-ID: <20030913214346.61A67E2468@bvds.geneva.edu> I looked into this and decided that channel bonding could not be done in a system with diskless booting unless there was a separate network to handle the booting. >Can anybody help me with the steps required to configure a linux cluster >node with channel bonding (or trunking) and diskless booting? Brett van de Sande _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Sun Sep 14 16:54:14 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Sun, 14 Sep 2003 22:54:14 +0200 Subject: Diskless Beowulf Message-ID: <20030914205154.M46076@ug.ms.wits.ac.za> I know this may seem as a trivial question but can anyone please give me a guideline on where to start in setting up a diskless Beowulf cluster.I need something that will assist a Linux dummy. Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel.pfenniger at obs.unige.ch Mon Sep 15 03:51:29 2003 From: daniel.pfenniger at obs.unige.ch (Daniel Pfenniger) Date: Mon, 15 Sep 2003 09:51:29 +0200 Subject: channel bonding/trunking and diskless booting In-Reply-To: <20030913214346.61A67E2468@bvds.geneva.edu> References: <20030913214346.61A67E2468@bvds.geneva.edu> Message-ID: <3F656F81.8070508@obs.unige.ch> bvds at bvds.geneva.edu wrote: > I looked into this and decided that channel bonding could > not be done in a system with diskless booting unless there was a > separate network to handle the booting. Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 (say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be used just for the booting network? What is the obstacle for such a configuration? Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Florent.Calvayrac at univ-lemans.fr Mon Sep 15 06:05:59 2003 From: Florent.Calvayrac at univ-lemans.fr (Florent Calvayrac) Date: Mon, 15 Sep 2003 12:05:59 +0200 Subject: channel bonding/trunking and diskless booting In-Reply-To: <3F656F81.8070508@obs.unige.ch> References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> Message-ID: <3F658F07.2030101@univ-lemans.fr> Daniel Pfenniger wrote: > > > bvds at bvds.geneva.edu wrote: > >> I looked into this and decided that channel bonding could >> not be done in a system with diskless booting unless there was a >> separate network to handle the booting. > > > Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 > (say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be > used just > for the booting network? What is the obstacle for such a configuration? > > Dan > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > then you should rewrite the PXE rom to support channel bonding... why not have the last machine of your cluster with a disk to use as a BOOTP server, then once all the cluster is booted boot it as a node ? Or use an old Pentium 75 laying around as a BOOTP server downloading a channel bonding enabled kernel ? -- Florent Calvayrac | Tel : 02 43 83 26 26 Laboratoire de Physique de l'Etat Condense | Fax : 02 43 83 35 18 UMR-CNRS 6087 | http://www.univ-lemans.fr/~fcalvay Universite du Maine-Faculte des Sciences | 72085 Le Mans Cedex 9 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Mon Sep 15 07:19:10 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Mon, 15 Sep 2003 07:19:10 -0400 Subject: howto calculate pi to large (eg 10^6) places Message-ID: <3F65A02E.58937413@epa.gov> I was reading Petr Beckmann's book on the history of pi, which includes a chapter on the digit hunters (people who calculated pi to large numbers of digits). He says that (repeatedly) calculating pi to large numbers of places is a good way of testing whether a machine is functioning accurately. I just looked with google for such a program but couldn't find anything that went beyond the bit width (32 or 64) of the host machine. Does anyone know where I can get the code? How do you do a calculation to 10^6 places on a 32 bit machine? Thanks Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Mon Sep 15 07:58:32 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Mon, 15 Sep 2003 07:58:32 -0400 Subject: howto calculate pi to large (eg 10^6) places In-Reply-To: <3F65A02E.58937413@epa.gov> References: <3F65A02E.58937413@epa.gov> Message-ID: <1063627112.20882.83.camel@protein.scalableinformatics.com> Hi Joe: There are many methods available. You can exploit various of the expansions which yield pi or 1/pi. You will need an extended precision library, such as David Bailey's MP, or similar. As for algorithms, you have several choices. The Bailey-Borwein-Plouffe method will give you hexadecimal digits, though it might not be as compute intensive as you wish. See http://www.cecm.sfu.ca/organics/papers/borwein/paper/html/paper.html for some nice details on some of this. Note: some of the good folks at Cray used to do this with their machines. One of the "burn-in" tests was generating a few hundred-million digits. Joe On Mon, 2003-09-15 at 07:19, Joseph Mack wrote: > I was reading Petr Beckmann's book on the history of pi, which includes a chapter > on the digit hunters (people who calculated pi to large numbers of digits). > He says that (repeatedly) calculating pi to large numbers of places is a good way > of testing whether a machine is functioning accurately. > > I just looked with google for such a program but couldn't find anything that went > beyond the bit width (32 or 64) of the host machine. Does anyone know where > I can get the code? How do you do a calculation to 10^6 places on a 32 bit > machine? > > Thanks Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Mon Sep 15 08:53:50 2003 From: gmpc at sanger.ac.uk (Guy Coates) Date: Mon, 15 Sep 2003 13:53:50 +0100 (BST) Subject: data storage location In-Reply-To: References: Message-ID: > Does the application read all or only a subset of the data? > Is the subset predictable? Externally by a scheduler? Automatic data replication tied into the schedular would be really nice; data-grid for beowulf-clusters. (Buzzword overload?) The schedular would examine the jobs waiting in the queue, and then populate the cluster with the required data. Frequently accessed datasets would get replicated to more nodes than less popular ones. Of course, the devil is always in the details. One can imagine all sorts of nasty scenarios, such as someone submitting a mass of short running jobs, which kicks off a mass of replication events, which then crushes your network. Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Mon Sep 15 11:02:57 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 15 Sep 2003 11:02:57 -0400 Subject: data storage location In-Reply-To: References: Message-ID: <1063638177.26176.26.camel@roughneck> On Sat, 2003-09-13 at 19:17, John Hearns wrote: > On Sat, 13 Sep 2003, Robert G. Brown wrote: > > > On Sat, 13 Sep 2003, Erik Paulson wrote: > > > > Or something else entirely. I'd welcome suggestions -- we're having a > > fairly major meeting on the project in a week or two. Real Life > > experiences of people who use AFS across a cluster would be lovely. > > Especially experiences or tests/benchmarks regarding things like > > efficiency and system overhead/load/requirements, and any caveats or > > gotchas to be overcome (given by hypothesis a longstanding, well managed > > AFS is heavily used at CERN. They have requirements for access to an > international base of users, and have a kerberos infrastructure in place. > IIRC, they still use V4 for the reasons mentioned earlier in the thread. > AFS is used across the computational cluster, which is where I met it. > The cluster is batch oriented - not a parallel Beowulf type. Does anyone have any suggestions on how a batch system can handle the ticket timeout problem gracefully? Nic BTW -- Does the 2GB file limit matter to anybody but me ? -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ilopez at millicom.com.ar Sun Sep 14 17:59:58 2003 From: ilopez at millicom.com.ar (Ignacio =?iso-8859-1?q?L=F3pez?=) Date: Sun, 14 Sep 2003 23:59:58 +0200 Subject: Diskless Beowulf In-Reply-To: <20030914205154.M46076@ug.ms.wits.ac.za> References: <20030914205154.M46076@ug.ms.wits.ac.za> Message-ID: <200309142359.58948.ilopez@millicom.com.ar> El Dom 14 Sep 2003 22:54, masego-otlhe_segone escribi?: > I know this may seem as a trivial question but can anyone please give me a > guideline on where to start in setting up a diskless Beowulf cluster.I need > something that will assist a Linux dummy. > > Masego-otlhe Segone Well...you can start by taking a look at Linux Terminal Server Project (www.ltsp.org)...in my opinion it's a great way of setting diskless linux clients...of course that you have to do much more to boild a beowulf, but that'll get you started ;-) > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ic02 at uow.edu.au Sun Sep 14 23:31:45 2003 From: ic02 at uow.edu.au (Iwan Maximus Cornelius) Date: Mon, 15 Sep 2003 13:31:45 +1000 Subject: pvm crashing Message-ID: Hi, I am having some problems running a Monte Carlo physics simulation on a 9 machine (all linux redhat 9.0) cluster using pvm (version 3.4.4). The application I am running is composed of a parent application and 9 identical daughter applications. The parent essential does house keeping work, starting the pvm daemon, adding machine etc, sets catchout and spawns the daughters. It then sits in a loop, listening for messages from the daughters. Being a Monte Carlo simulation this can result in 10,000,000 or so messages being passed between daughters and the parent. Well it should but the pvmd crashes mysteriously after about 3,000,000 with the following error message: libpvm [t40001]: mxfer() mxinput bad return on pvmd sock For some reason the pvmd is crashing. Could anybody suggest if something here is obvious, or if not, could suggest a way of debugging such a problem? Thanks for your time, Iwan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Mon Sep 15 11:40:05 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Mon, 15 Sep 2003 17:40:05 +0200 Subject: data storage location In-Reply-To: References: <20030912184327Q.hanzl@unknown-domain> Message-ID: <20030915174005Q.hanzl@unknown-domain> > Why do you say AFS is too cumbersome? - cannot export normal filesystem like ext2/3 (which existed before) - tickets which expire - something like five daemons on server - semantics different from UNIX filesystem - I am not sure about big partitions (say 100-200GB) - on this list it was often discouraged for beowulf Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Mon Sep 15 11:44:58 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Mon, 15 Sep 2003 17:44:58 +0200 Subject: data storage location In-Reply-To: References: Message-ID: <20030915174458A.hanzl@unknown-domain> > AFS is heavily used at CERN. They have requirements for access to an > international base of users, and have a kerberos infrastructure in place. > IIRC, they still use V4 for the reasons mentioned earlier in the thread. > AFS is used across the computational cluster, which is where I met it. > The cluster is batch oriented - not a parallel Beowulf type. Are there setup details anywhere on the www? I guess this experience is rare... Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tegner at nada.kth.se Mon Sep 15 03:02:22 2003 From: tegner at nada.kth.se (tegner at nada.kth.se) Date: Mon, 15 Sep 2003 09:02:22 +0200 (MEST) Subject: Diskless Beowulf In-Reply-To: <20030914205154.M46076@ug.ms.wits.ac.za> References: <20030914205154.M46076@ug.ms.wits.ac.za> Message-ID: <64835.150.227.16.253.1063609342.squirrel@webmail.nada.kth.se> Check http://www.linuxnetmag.com/en/issue5/m5diskless1.html > I know this may seem as a trivial question but can anyone please give me > a guideline on where to start in setting up a diskless Beowulf cluster.I > need something that will assist a Linux dummy. > > Masego-otlhe Segone > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Mon Sep 15 11:29:12 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Mon, 15 Sep 2003 17:29:12 +0200 Subject: data storage location In-Reply-To: References: <20030912184327Q.hanzl@unknown-domain> Message-ID: <20030915172912S.hanzl@unknown-domain> > A read-only data set allows many file handling options. > If files are updated as a whole, you may use a caching versioned file > system. That is specialized, but provides many opportunities for > optimization > ... > > > Cache-like behavior would save a lot of manual work but unfortunately > > I am not aware of any working solution for linux, > > Several exist. It depends on the semantics you need, as doing this > efficiently requires making assumptions about the access patterns and > desired semantics. Please what do you include in these 'several'? Am I right that none of them is opensource AND freely downloadable ? I try hard to find out what does this list in your head include, please correct my thoughts: - you certainly did not qualify InterMezzo and Lustre as working - old dead projects like ClusterNFS or amd hacks are also out of question - I don't quite believe you meant AFS or Coda either - you could consider some non-free software - you could consider something you use at Scyld, being even opensource but not freely downloadable anywhere (if I got right one of your older posts) Basically I just want confirmation for this one bit of information: "There is no reliable opensource freely downloadable filesystem (or addition to filesystem) for (contemporary) linux which would use local harddisk as a cache, even for the most relaxed semantics one can imagine (whole file caching, read-only, nearly no synchronisation)." Or better yet, tell me where it is, I'll be happy to be wrong. Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Mon Sep 15 14:12:03 2003 From: becker at scyld.com (Donald Becker) Date: Mon, 15 Sep 2003 14:12:03 -0400 (EDT) Subject: data storage location In-Reply-To: <20030915172912S.hanzl@unknown-domain> Message-ID: On Mon, 15 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > A read-only data set allows many file handling options. > > If files are updated as a whole, you may use a caching versioned file > > system. That is specialized, but provides many opportunities for > > optimization > > ... > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > > I am not aware of any working solution for linux, > > > > Several exist. It depends on the semantics you need, as doing this > > efficiently requires making assumptions about the access patterns and > > desired semantics. > > Please what do you include in these 'several'? Am I right that none of > them is opensource AND freely downloadable ? You've instantly moved from a technical agenda to a political one. "Open Source" is a technical attribute, while "free" and "no cost online access" is different. The approaches I am talking about are a mix of - the Transparent File System, originally intended for caching in front of mounted CDs, but improved to sit in front of NFS. It was first implemented in Linux around 1993, but I haven't seen any updates - several academic projects, two of which I've seen at recent LinuxWorlds. There was an interesting related paper at ClusterWorld, which focused on co-servers -- using peers instead of the main file server. - and, of course, the implementation that we have. > I try hard to find out what does this list in your head include, > please correct my thoughts: The challenge here is that the kind of file server I'm talking about does not cleanly fit with Unix end-user semantics. Thus they are often hidden or not considered a file system. Specifically, the following conflict with Unix expectations Using versions while allowing multiple open versions with the same name. Prohibiting anything but whole-file updates Not providing consistent readdir() and access() results, as there is no fixed list of files. > - you certainly did not qualify InterMezzo and Lustre as working That's a political minefield. > - old dead projects like ClusterNFS or amd hacks are also out... Those are new rules. Caching file systems are very ad hoc, and thus used for only one purpose. There is not the broad user base motivation to make them general or keep them current. That doesn't mean they don't exist. And yes, some of the systems I'm referring to are related to automount daemons (AMD). I'm especially familar with those, as many are implemented with NFS servers. I wrote one of the first user-level NFS servers in the late '80s, and that NFS code became the basis for implementing several quirky, ad hoc filesystems. (For those that don't know how AMDs work, they mount a server or create a file in a local directory and return a symbolic link pointing to it.) One specifically was a FTP filesystem. When a file block was requested, the whole file was moved locally. It had the advantage over most caching systems of also being able to support readdir(). Why use the NFS / automount approach? Because it's the only user-level interface to the filesystem NFS is already quirky > Basically I just want confirmation for this one bit of information: > > "There is no reliable opensource freely downloadable filesystem (or > addition to filesystem) for (contemporary) linux which would use local > harddisk as a cache, even for the most relaxed semantics one can > imagine (whole file caching, read-only, nearly no synchronisation)." -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bvds at bvds.geneva.edu Mon Sep 15 12:11:48 2003 From: bvds at bvds.geneva.edu (bvds at bvds.geneva.edu) Date: Mon, 15 Sep 2003 12:11:48 -0400 (EDT) Subject: channel bonding/trunking and diskless booting In-Reply-To: <3F656F81.8070508@obs.unige.ch> (message from Daniel Pfenniger on Mon, 15 Sep 2003 09:51:29 +0200) References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> Message-ID: <20030915161148.87ECBE2468@bvds.geneva.edu> > >Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 >(say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be used just >for the booting network? What is the obstacle for such a configuration? > I am *really* hoping that someone would tell me how to do diskless nodes plus channel bonding. Adding an additional machine to act as DHCP server and tftp server is out (for me) because I don't don't have any free ports left on my switch. Can you really set up eth0 so that it is both bonded and unbonded? How do you do this? What is this eth0:0 thing? (I display my ignorance) Brett van de Sande _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bvds at bvds.geneva.edu Mon Sep 15 14:16:46 2003 From: bvds at bvds.geneva.edu (bvds at bvds.geneva.edu) Date: Mon, 15 Sep 2003 14:16:46 -0400 (EDT) Subject: channel bonding/trunking and diskless booting In-Reply-To: <3F656F81.8070508@obs.unige.ch> (message from Daniel Pfenniger on Mon, 15 Sep 2003 09:51:29 +0200) References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> Message-ID: <20030915181646.B6C6CE246B@bvds.geneva.edu> > >Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 >(say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be used just >for the booting network? What is the obstacle for such a configuration? > I just found an old post from Rob Latham asking the same question: http://www.beowulf.org/pipermail/beowulf/2001-June/000391.html He tried this and he said it didn't work: > shows what i know :>. it looks to me like the bonding tricks hold on > pretty tight to the slave interfaces and won't let me trick it into > sending trafic over just one nic when i want to. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Mon Sep 15 15:38:04 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Mon, 15 Sep 2003 21:38:04 +0200 Subject: howto calculate pi to large (eg 10^6) places References: <3F65A02E.58937413@epa.gov> <1063627112.20882.83.camel@protein.scalableinformatics.com> Message-ID: <3F66151C.8060604@moene.indiv.nluug.nl> Joseph Landman wrote: > Note: some of the good folks at Cray used to do this with their > machines. One of the "burn-in" tests was generating a few > hundred-million digits. Heh, the way we (previous employer) burnt in our new Cray was to have CWI, Amsterdam (www.cwi.nl) run one of their award-winning crypto programs "for free" before going operational ... -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From serguei.patchkovskii at sympatico.ca Mon Sep 15 15:32:55 2003 From: serguei.patchkovskii at sympatico.ca (serguei.patchkovskii at sympatico.ca) Date: Mon, 15 Sep 2003 15:32:55 -0400 Subject: channel bonding/trunking and diskless booting Message-ID: <20030915193255.KDRQ26607.tomts24-srv.bellnexxia.net@[209.226.175.20]> ----- Original Message ----- From: To: Cc: "Ashwin Bijur" Sent: Saturday, September 13, 2003 5:43 PM Subject: channel bonding/trunking and diskless booting > I looked into this and decided that channel bonding > could not be done in a system with diskless booting > unless there was a separate network to handle the > booting. Not true. I configured a 90-node cluster at the University of Calgary last fall to do exactly that. The system has two 100Mbit networks between the nodes, with only one of those attached to the front node. This "front" network is used for remote boot, then gets bonded with the the "back" network for the internode traffic. Getting this to work requires a bit of trickery with the network startup scripts. You have to create a temporary ram filesystem, copy -statically linked- ifenslave, ifconfig, route, shell, and the setup script to that filesystem, then activate channel bonding and set up the routing. The key with routing in such network is that it is perfectly possible to direct outgoing packets to systems, which are not fully attached to both networks, to the appropriate slaved interface. For a case where both networks are attached to the front node, there is an additional complication, as you do not necessarily know which interface PXE will try first. This can be solved by assigning unique subnets to both interfaces, which are -different- from the bonded subnet. Having done this, you can use set up IP routing on the front node such that it will respond through the correct interface while PXE is active. Debugging such setup is a bitch - but once it works, it works really well! Serguei _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Mon Sep 15 16:19:13 2003 From: lathama at yahoo.com (Andrew Latham) Date: Mon, 15 Sep 2003 13:19:13 -0700 (PDT) Subject: Diskless Cluster Message-ID: <20030915201913.20695.qmail@web80510.mail.yahoo.com> diskless is my preference. simply speaking you need some good hardware before you worry about software. The software is easy simply us PXE and TFTP to get a kernel to your node. Then Have that kernel load a NFS root on a file server or down load one to RAM. Start your system. ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 15 19:35:03 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 16 Sep 2003 09:35:03 +1000 Subject: data storage location In-Reply-To: <1063638177.26176.26.camel@roughneck> References: <1063638177.26176.26.camel@roughneck> Message-ID: <200309160935.05024.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 16 Sep 2003 01:02 am, Nicholas Henke wrote: > BTW -- Does the 2GB file limit matter to anybody but me ? It would if we used AFS, but we don't, we use NFS. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/ZkynO2KABBYQAh8RAhbpAKCO//vP6dn6Qos7Y+zR9WfoRMxSSACfZLBZ Rv6CONuRhmVRGbnM7MaeOGk= =6W96 -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 15 19:57:59 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 16 Sep 2003 09:57:59 +1000 Subject: channel bonding/trunking and diskless booting In-Reply-To: <20030915161148.87ECBE2468@bvds.geneva.edu> References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> <20030915161148.87ECBE2468@bvds.geneva.edu> Message-ID: <200309160958.00864.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 16 Sep 2003 02:11 am, bvds at bvds.geneva.edu wrote: > I am *really* hoping that someone would tell me how to do > diskless nodes plus channel bonding. The only idea I have would be whether doing a LinuxBIOS on the system would help - but then of course there's an even worse bootstrapping problem of installing LinuxBIOS on all your systems, not to mention small things like invalidating warranties, risk of blowing your systems up, etc.. :-) - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD4DBQE/ZlIHO2KABBYQAh8RAlDHAJdGn8u15uRzQI+yutR1Y0bTy+t1AJwJS/TC QMkuLi66IoW7dD3/JF9RDg== =7bnL -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From egan at sense.net Mon Sep 15 20:41:00 2003 From: egan at sense.net (egan at sense.net) Date: Mon, 15 Sep 2003 18:41:00 -0600 (MDT) Subject: howto calculate pi to large (eg 10^6) places In-Reply-To: <3F65A02E.58937413@epa.gov> References: <3F65A02E.58937413@epa.gov> Message-ID: <1063672860.3f665c1c22c2c@www.sense.net> Quoting Joseph Mack : > I can get the code? How do you do a calculation to 10^6 places on a 32 > bit > machine? echo "scale=1000000;a(1)*4" | bc -l _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 15 19:53:41 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 16 Sep 2003 09:53:41 +1000 Subject: Lustre Message-ID: <200309160953.42528.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Anyone tried out Lustre [1] ? What did you think ? - From what I've read it looks interesting, but I've not heard good things regarding stability. Chris [1] - http://www.lustre.org/ - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/ZlEFO2KABBYQAh8RAvySAJ9wJD62ogP/2WvADgqXI5qV+5z5mACaApZZ 8vvqRbBSeQ7b5LB0/xkqvUE= =h4et -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rouds at servihoo.com Tue Sep 16 06:06:42 2003 From: rouds at servihoo.com (RoUdY) Date: Tue, 16 Sep 2003 14:06:42 +0400 Subject: mpich2 In-Reply-To: <200309151829.h8FITmd25993@NewBlue.Scyld.com> Message-ID: Is their an expert in MPICH2-0.93??? Please mail me, I need some help in some issues Thanks Roudy -------------------------------------------------- Get your free email address from Servihoo.com! http://www.servihoo.com The Portal of Mauritius _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nican at nsc.liu.se Tue Sep 16 07:40:28 2003 From: nican at nsc.liu.se (Niclas Andersson) Date: Tue, 16 Sep 2003 13:40:28 +0200 (CEST) Subject: ANNOUNCEMENT: Workshop on Linux Clusters for Super Computing Message-ID: ============================================================== # 4th Annual Workshop on LINUX CLUSTERS FOR SUPER COMPUTING # Clusters for High Performance Computing and Grid Solutions # # October 22-24, 2003 # Hosted by # National Supercomputer Centre (NSC) # Link?ping University, Sweden ============================================================== The fast development of the processing power of high-end PCs and workstations, together with the availability of open source operating systems such as Linux have made it possible to build very cost effective clusters of commodity hardware, also known as beowulfs. With the addition of high bandwidth networks with low latency, open source based beowulfs has become the most common platform for high performance computing. Assembling a beowulf is, however, a non-trivial task due to the fast moving hardware market as well as the rapid development of Linux and available tools for clustered computing. The aim of the workshop is to gather together both people with experience of building and maintaining clusters and people using or interested in using cluster resources. The workshop will address the hardware and software issues encountered during design as well as deployed applications' efficiency, portability and utilization. We intend to cover the following topics: o Storage solutions for clusters and Grids. o Distributed computing and Grid technologies o Applications adapted for using parallel and distributed resources o Software and tools for using and managing clusters o Scalability issues, benchmarking and tuning o Job scheduling strategies o Key application areas o Update on the latest hardware developments In addition to invited talks, there will be tutorials (Oct. 22), exhibits and vendor presentations. The complete list of speakers and programme will be available at http://www.nsc.liu.se/lcsc/. If you wish to present your own cluster, Grid project, software development or give a talk on a topic related to clusters or Grid computing you are most welcome to contact Niclas Andersson E-mail: Phone: +46 13 281464 Fax: +46 13 282535 If you want to present your products in a vendor session or in our exhibit outside the auditorium, please contact Torgny Fax?n E-mail: Phone: +46 13 285798 Fax: +46 13 282535 CONFERENCE DETAILS Date October 22-24, 2003 Location The conference will take place at Collegium, Link?ping close to Link?ping University. Tutorials will be held at the University. Registration On-line registration at http://www.nsc.liu.se/lcsc Last date: 10 October. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gropp at mcs.anl.gov Tue Sep 16 08:23:37 2003 From: gropp at mcs.anl.gov (William Gropp) Date: Tue, 16 Sep 2003 07:23:37 -0500 Subject: mpich2 In-Reply-To: References: <200309151829.h8FITmd25993@NewBlue.Scyld.com> Message-ID: <5.1.1.6.2.20030916072231.02e51d88@localhost> At 02:06 PM 9/16/2003 +0400, RoUdY wrote: >Is their an expert in MPICH2-0.93??? >Please mail me, I need some help in some issues >Thanks Bug reports and questions on mpich2 should be sent to mpich2-maint at mcs.anl.gov . You may also wish to try the current version, 0.94 . Bill _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lusk at mcs.anl.gov Tue Sep 16 08:40:47 2003 From: lusk at mcs.anl.gov (Rusty Lusk) Date: Tue, 16 Sep 2003 07:40:47 -0500 Subject: mpich2 In-Reply-To: Message from "RoUdY" of "Tue, 16 Sep 2003 14:06:42 +0400." Message-ID: <200309161240.h8GCemL23134@shakey.mcs.anl.gov> | Is their an expert in MPICH2-0.93??? | Please mail me, I need some help in some issues | Thanks | Roudy The current release of MPICH2 is 0.94. Please send your problem, inlcuding whether you are using the Windows or Unix version of MPICH2, and the exact nature of the problem, to mpich2-maint at mcs.anl.gov. Regards, Rusty Lusk _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeffrey.b.layton at lmco.com Tue Sep 16 08:38:16 2003 From: jeffrey.b.layton at lmco.com (Jeff Layton) Date: Tue, 16 Sep 2003 08:38:16 -0400 Subject: Desired to Test on Opterons Message-ID: <3F670438.5050309@lmco.com> Hello, I'd like to test our codes on a small Opteron cluster, preferrably with Infiniband. We're trying to test with a vendor but we're having lots of problems with them, so we'd like to find a reliable vendor who can put together a small (4-8 node) dual Opteron cluster with at least GigE and if at all possible IB. Each node should have at least 2 Gigs of memory. The vendor needs to have the cluster up and running within 10 days of signing a Non-Disclosure Agreement (NDA) with Lockheed. The pot of gold at the end of the rainbow is that we're looking for several clusters next year. The first one we hope to order in November for a Jan. delivery. No guarantees though :) If you are interested, please contact me ASAP. Thanks! Jeff Layton Lockheed-Martin Aeronautics Company _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Tue Sep 16 09:50:20 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Tue, 16 Sep 2003 09:50:20 -0400 Subject: Lustre References: <200309160953.42528.csamuel@vpac.org> Message-ID: <3F67151C.A5D41000@epa.gov> Chris Samuel wrote: > Anyone tried out Lustre [1] ? There was a talk at OLS this year http://www.linuxsymposium.org/2003/ As usual when given by a person involved in the development, it's always presented as being easy to install and running smoothly. However they do have it running on two kilonode production clusters and have won the contract for the filesystem for some new cluster that is currently being bid on. So it's in production in some places Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Tue Sep 16 09:34:24 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Tue, 16 Sep 2003 09:34:24 -0400 Subject: howto calculate pi to large (eg 10^6) places References: <3F65A02E.58937413@epa.gov> <1063672860.3f665c1c22c2c@www.sense.net> Message-ID: <3F671160.D543756A@epa.gov> egan at sense.net wrote: > > > How do you do a calculation to 10^6 places on a 32 bit machine? > echo "scale=1000000;a(1)*4" | bc -l :-) I didn't know you could do this (still waiting for the prompt to return). Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Tue Sep 16 11:32:38 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Tue, 16 Sep 2003 17:32:38 +0200 Subject: data storage location In-Reply-To: References: <20030915172912S.hanzl@unknown-domain> Message-ID: <20030916173238X.hanzl@unknown-domain> > Caching file systems are very ad hoc, and thus used for only one > purpose. There is not the broad user base motivation to make them > general or keep them current. I believe cachefs for linux would meet enough demand to be viable. Or little kernel module intercepting open(2) and friends and letting userland to decide when these calls should go stright through and when they should ask for userland action (like fetching out-of-cache file) - an autofs-style variant of old amd-hacks game. Yes, you are right, nothing simple/universal managed to catch up so far. > You've instantly moved from a technical agenda to a political one. > "Open Source" is a technical attribute, while "free" and "no cost online > access" is different. For me "Open Source & free & no cost online access" means big user base and good chance of fixing problems via community collaboration - strong technical reason (with political side-effect). (I however understand and appreciate that your work extends technical possibilities even beyond the above limits, i.e. that one can even buy good solutions :) Best Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 16 12:40:31 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 16 Sep 2003 12:40:31 -0400 (EDT) Subject: Next Beowulf training class, September 24-25 in San Francisco Message-ID: Scyld is offering detailed training in the installation, configuration, administration and programming of Beowulf clusters. I will be the instructor for several of the upcoming courses including the next session, to be held in San Francisco on September 24th and 25th. If you'd like more information or want to register please visit http://www.scyld.com/training.html We typically alternate training between the east and west coast (U.S.), but other locations may be added to schedule based on demand. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Tue Sep 16 12:47:09 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Tue, 16 Sep 2003 09:47:09 -0700 (PDT) Subject: howto calculate pi to large (eg 10^6) places In-Reply-To: <3F671160.D543756A@epa.gov> Message-ID: gnu bc is a useful tool to have. it may take a while for the prompt to come back... ;) 1000 places takes around 3 seconds on a 900mhz pIII joelja On Tue, 16 Sep 2003, Joseph Mack wrote: > egan at sense.net wrote: > > > > > How do you do a calculation to 10^6 places on a 32 bit machine? > > > echo "scale=1000000;a(1)*4" | bc -l > > :-) I didn't know you could do this (still waiting for the prompt to return). > > Joe > > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Tue Sep 16 17:27:29 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Tue, 16 Sep 2003 23:27:29 +0200 Subject: Diskless Beowulf Message-ID: <20030916212730.M21242@ug.ms.wits.ac.za> Actually I alredy have a working diskless cluster, I need details on how to setup the beowulf and to configurations I need to do to get MPI working and to start running parallel programs. Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian.dobbins at yale.edu Tue Sep 16 20:16:09 2003 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Tue, 16 Sep 2003 20:16:09 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. Message-ID: Hi guys, Has anyone on this list been involved in the construction of a small 'machine room' for their clusters? We're beginning to look at creating an enclosure with an attached AC unit and, possibly, a raised floor. I've started poking around the 'net a bit, but was interested in hearing any experiences or tips from the people here. Anything about materials to use, people to talk to, hidden costs to watch out for, etc. The overall goal is to have a cool, clean, local systems 'room' for our computational facilities. As for specifics for size, that's unknown at the moment... perhaps, just for the sake of discussion, say 15' by 25' (by however tall we need, at least 8.5'). We'd like to plan for a capacity of up to 4 racks (w/ 32 nodes each), plus a few miscellaneous desksides lying around. (While we do have a central machine room on campus, our network is isolated for security reasons, and additionally we tend to run GigE between our newest machines -even some desktops- and we certainly can't get that sort of throughput from the central machine room! Thus, even though this might be costly, it may be worth it in the end.) Thanks for any pointers whatsoever! It's much appreciated! Cheers, - Brian Brian Dobbins Yale Mechanical Engineering ------------------------------------------------------------------ "Be nice to other people.. they outnumber you six billion to one." _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 16 21:57:11 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 16 Sep 2003 21:57:11 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: Message-ID: On Tue, 16 Sep 2003, Brian Dobbins wrote: > > Hi guys, > > Has anyone on this list been involved in the construction of a small > 'machine room' for their clusters? We're beginning to look at creating an > enclosure with an attached AC unit and, possibly, a raised floor. > > I've started poking around the 'net a bit, but was interested in hearing > any experiences or tips from the people here. Anything about materials to > use, people to talk to, hidden costs to watch out for, etc. The overall > goal is to have a cool, clean, local systems 'room' for our computational > facilities. There are a variety things you'll want to consider, especially: a) power; There have been some very extensive discussions on the list about power gotchas associated with driving a large number of non-power-factor corrected switching power supplies. There are vendors out there who sell server-room sized main transformers that compensate for the 180 Hz harmonic line distortion that can occur. You can also consider line conditioning and UPS at the same time, depending on your needs and budget. You'll need to locate power poles or other receptacle sets convenient to the racks. b) capacity and airflow patterns of your air conditioning compared to your power supply and expected loading; What goes in must come out, basically, with enough spare capacity to allow for future increases in rackmount power density and to keep the room COOL, not just "not hot". c) structural integrity; Loaded racks are >>heavy<< -- they can weigh 800 lbs or even more if they contain UPS batteries, on less than a square meter of floor space. Two post racks also are often under considerable torque when loaded, and need to be fastened carefully to the floor and/or equipped with cases with center mounts that balance. d) sound and light; Four full racks plus AC will sound like a small jet taxiing. All the time. Tolerable for short stretches but not a good place to work. Besides it's cold, right? Having enough light, effectively located, to be able to work with is also good if you are an Old Guy with eyes that get stressed by reading little bitty etched print upside down in the dark. e) network; Cable trays, rackspace or wall racks for network POPs and switches. WAN (organizational level) connections as well as the actual cluster LAN. f) security; Lessee, a room with 160 or so $2000 or so boxes = $320K. Locating the space in a room with a convenient door out to your dark and unattended loading dock, however convenient it is for delivery, is ill-advised. Locks, booby traps, x10 video cams, alarms, and armed guards optional. Find a comfort level here. g) comfort/convenience; We find things like a communal jacket on the back of the cluster room door, a workbench with tools (rechargable screwdriver), portable KVM, maybe a laptop for serial port access (for certain motherboards), a comfy workchair or two. Sound excluding/noise reducing stereo headphones can be nifty (and probably OSHA approved). > As for specifics for size, that's unknown at the moment... perhaps, just > for the sake of discussion, say 15' by 25' (by however tall we need, at > least 8.5'). We'd like to plan for a capacity of up to 4 racks (w/ 32 > nodes each), plus a few miscellaneous desksides lying around. If you mean a few tower-mounted servers, that's great, but see notes above about suitability as a real workspace. It is typically loud, cold, locked (or should be), and a bad place to take cups of coffee or food. So it's not a great place to locate humans for more than the times required to work on node installation and maintenance. > (While we do have a central machine room on campus, our network is > isolated for security reasons, and additionally we tend to run GigE > between our newest machines -even some desktops- and we certainly can't > get that sort of throughput from the central machine room! Thus, even > though this might be costly, it may be worth it in the end.) > > Thanks for any pointers whatsoever! It's much appreciated! Only two. Other folks will probably add some to bigger/better locations, but you can take a "walking tour in pictures" of our brahma cluster on http://www.phy.duke.edu/brahma. You can also find a book on engineering clusters and a few other documents which address infrastructure. Hope this helps. Oh, and I'd strongly suggest getting a professional architect (one with experience doing computer machine rooms) to do your plans. And avoid bozo electricians who try e.g. wiring three phase circuits with a single common neutral. rgb > > Cheers, > - Brian > > > Brian Dobbins > Yale Mechanical Engineering > ------------------------------------------------------------------ > "Be nice to other people.. they outnumber you six billion to one." > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Tue Sep 16 22:51:44 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Wed, 17 Sep 2003 04:51:44 +0200 Subject: rsh Message-ID: <20030917024414.M10506@ug.ms.wits.ac.za> I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I just realised that I cannot communicate with other hosts due to some rsh setting that I need to do. I tried running tstmachines and I got "Errors while trying to run rsh hostname -n true Unexpected response from hostname: ---> Permission denied ..." Please help with the rsh settings that I need to inorder to get the rsh to work. Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amazonia at myrealbox.com Wed Sep 17 05:01:18 2003 From: amazonia at myrealbox.com (amazonia) Date: Wed, 17 Sep 2003 03:01:18 -0600 Subject: Need Rocks cluster feedback Message-ID: <1063789278.82f649c0amazonia@myrealbox.com> Hi, We will be soon implementing a new 32 node gigabit cluster based on x86 arch. The bosses are looking at rockscluster or RH9+scripts or RH7.x+oscar. (My favourite is FAI, I will be probably part time admining the cluster) I would like to know the experience of listmembers and pros/cons of Rocks Cluster distr v3.0 for x86. I heard that rocks people advocate a quick reinstall of faulty node without any analysis. Is it true? I wouldn't prefer that. What is the best cluster mgmt tool? (Oscar etc). Which is the most popular? Regards, Matt _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rkane at scs.carleton.ca Wed Sep 17 09:32:03 2003 From: rkane at scs.carleton.ca (Robert Kane) Date: Wed, 17 Sep 2003 09:32:03 -0400 Subject: Need Rocks cluster feedback References: <1063789278.82f649c0amazonia@myrealbox.com> Message-ID: <3F686253.F7ABBF70@scs.carleton.ca> The experience I've had with Rocks has been fairly positive. For what it's worth though we're still using v2.3.2 and v3.0. The pros for Rocks are generally the ease of use. Everything (OS and tools [excluding lam-mpi]) are included in one very straightforward package. Getting a cluster up and running is fairly quick. A quick linux install on the frontend and then a couple minutes for each node to install. Assuming no problems, getting our 64 node cluster from scratch to up and running shouldn't take more than half-day. The cons. The above time estimate assumed that no problems occurred. When upgrading recently we did have some form of inexplicable glitch that caused the post-install configuration on the frontend to fail and have to be configured manually. I should point out that this error was the only time something like that has been known to happen. I should also point out that another pro to using Rocks is the helpful advice of the Rocks developers. When we had the above mentioned glitch, one of the developers graciously spent a fair amount of time making sure everything was working fine. Robert Kane _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jac67 at georgetown.edu Wed Sep 17 09:21:50 2003 From: jac67 at georgetown.edu (Jess Cannata) Date: Wed, 17 Sep 2003 09:21:50 -0400 Subject: Need Rocks cluster feedback In-Reply-To: <1063789278.82f649c0amazonia@myrealbox.com> References: <1063789278.82f649c0amazonia@myrealbox.com> Message-ID: <3F685FEE.7040107@georgetown.edu> We've been using Oscar on all of our clusters for a year now and have been satisfied with it. Our only complaint was Oscar's lack of support for RedHat 8.0/9.0, and that has since been remedied with Oscar 2.3. It is easy to install, and it has a good set of management tools. Jess Georgetown University amazonia wrote: >Hi, > >We will be soon implementing a new 32 node gigabit cluster based on x86 arch. > >The bosses are looking at rockscluster or RH9+scripts or RH7.x+oscar. (My favourite is FAI, I will be probably part time admining the cluster) > >I would like to know the experience of listmembers and pros/cons of Rocks Cluster distr v3.0 for x86. > >I heard that rocks people advocate a quick reinstall of faulty node without any analysis. Is it true? I wouldn't prefer that. > >What is the best cluster mgmt tool? (Oscar etc). Which is the most popular? > >Regards, >Matt > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From pu at ku.ac.th Wed Sep 17 10:37:48 2003 From: pu at ku.ac.th (Putchong Uthayopas) Date: Wed, 17 Sep 2003 21:37:48 +0700 Subject: Diskless Cluster References: <20030915201913.20695.qmail@web80510.mail.yahoo.com> Message-ID: <003f01c37d29$47d7ec10$0e01010a@hpcncd.cpe.ku.ac.th> Hi, That seems to be quite simple. Anyway, we found that: 1. This eat up a lot of RAM. So, it will not work well if your node does not have large memory. 2. NFS root create a lot of overhead and does not seems to scale well. Putchong ----- Original Message ----- From: "Andrew Latham" To: Cc: "beowulf" Sent: Tuesday, September 16, 2003 3:19 AM Subject: RE: Diskless Cluster > diskless is my preference. simply speaking you need some good hardware before > you worry about software. The software is easy simply us PXE and TFTP to get a > kernel to your node. Then Have that kernel load a NFS root on a file server or > down load one to RAM. Start your system. > > ===== > Andrew Latham > > Penguin loving, moralist agnostic. > > LathamA.com - (lay-th-ham-eh) > lathama at lathama.com - lathama at yahoo.com > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sean at asacomputers.com Wed Sep 17 11:28:45 2003 From: sean at asacomputers.com (Sean) Date: Wed, 17 Sep 2003 08:28:45 -0700 Subject: Need Rocks cluster feedback In-Reply-To: <1063789278.82f649c0amazonia@myrealbox.com> Message-ID: <5.1.0.14.2.20030917082343.02e5aca0@pop.asacomputers.com> Matt, We have done several Rocks cluster installation and it is very easy and straight forward. Linux comes embedded in it and this is an added advantage. C-3 is a widely accepted cluster management tool. At 03:01 AM 9/17/03 -0600, amazonia wrote: >Hi, > >We will be soon implementing a new 32 node gigabit cluster based on x86 arch. > >The bosses are looking at rockscluster or RH9+scripts or RH7.x+oscar. (My >favourite is FAI, I will be probably part time admining the cluster) > >I would like to know the experience of listmembers and pros/cons of Rocks >Cluster distr v3.0 for x86. > >I heard that rocks people advocate a quick reinstall of faulty node >without any analysis. Is it true? I wouldn't prefer that. > >What is the best cluster mgmt tool? (Oscar etc). Which is the most popular? > >Regards, >Matt > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf Thanks and Regards Sean ASA Computers Inc. 2354, Calle Del Mundo Santa Clara CA 95054 Telephone : (408) 654-2901 xtn 205 (408) 654-2900 ask for Sean (800) REAL-PCS (1-800-732-5727) Fax: (408) 654-2910 E-mail : sean at asacomputers.com URL : http://www.asacomputers.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jim at ks.uiuc.edu Wed Sep 17 11:52:51 2003 From: jim at ks.uiuc.edu (Jim Phillips) Date: Wed, 17 Sep 2003 10:52:51 -0500 (CDT) Subject: Diskless Cluster In-Reply-To: <003f01c37d29$47d7ec10$0e01010a@hpcncd.cpe.ku.ac.th> Message-ID: Another option is a bproc-based system like Scyld or Clustermatic. You don't get a full Linux install on the slaves, but if you can live with that limitation administration is pretty easy. We're currently booting our diskless slaves off of floppies, which is OK with bproc because the actual runtime kernel is loaded off of the network during the boot process. -Jim On Wed, 17 Sep 2003, Putchong Uthayopas wrote: > Hi, > > That seems to be quite simple. Anyway, we found that: > > 1. This eat up a lot of RAM. So, it will not work well if your node does not > have large memory. > 2. NFS root create a lot of overhead and does not seems to scale well. > > Putchong > ----- Original Message ----- > From: "Andrew Latham" > To: > Cc: "beowulf" > Sent: Tuesday, September 16, 2003 3:19 AM > Subject: RE: Diskless Cluster > > > > diskless is my preference. simply speaking you need some good hardware > before > > you worry about software. The software is easy simply us PXE and TFTP to > get a > > kernel to your node. Then Have that kernel load a NFS root on a file > server or > > down load one to RAM. Start your system. > > > > ===== > > Andrew Latham > > > > Penguin loving, moralist agnostic. > > > > LathamA.com - (lay-th-ham-eh) > > lathama at lathama.com - lathama at yahoo.com > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mas at ucla.edu Wed Sep 17 12:09:01 2003 From: mas at ucla.edu (Michael Stein) Date: Wed, 17 Sep 2003 09:09:01 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: ; from rgb@phy.duke.edu on Tue, Sep 16, 2003 at 09:57:11PM -0400 References: Message-ID: <20030917090901.B4711@mas1.ats.ucla.edu> Just some additions, good coverage.. - - - Access to both front and back of racks... > a) power; > There have been some very extensive discussions on the list about power > gotchas associated with driving a large number of non-power-factor > corrected switching power supplies. There are vendors out there who > sell server-room sized main transformers that compensate for the 180 Hz > harmonic line distortion that can occur. The 1U dual Xeon machines we've seen have power factor correcting power supplies so the harmonic problems might not exist. At 250 W/machine and 42 in a rack you do need an impressive amount of power, over 10 KW/rack. Watch out, future machines may need more... Watch the head/air flow. Typical rack mount machines want cool air in the front and exhaust hot air out the back. A rack full will heat a rack size volume of air all at once (something like 75 F in 99 F out all at once!). Don't run this hot air into the front of the next row of racks... Watch out -- the power usages varies depending on the CPU loading. A quick measurement of an idle machine will be very misleading: 1U Dual Xeon 2.66 Ghz 2 GB ram (& Kill-A-Watt power meter) idle: 109 W 1 CPU 174 W (1 x burnP6) 2 CPU 254 W (4 x burnP6) Make sure the electricians know the power you intend to use -- they need to size the circuits *larger* than this by some factor (from code?) -- you can't put a 20 A load on a 20 A breaker. > c) structural integrity; > > Loaded racks are >>heavy<< -- they can weigh 800 lbs or even more if > they contain UPS batteries 30 lb 1U machine * 42 in rack -> 1260 lbs or were they 40 lb machines? 40 lb 1U machine * 42 in rack -> 1680 lbs This does NOT include the weight of the rack, cables and any other stuff you manage to hang on the rack (rack rails?). > Oh, and I'd strongly suggest getting a professional architect (one with > experience doing computer machine rooms) to do your plans. And avoid > bozo electricians who try e.g. wiring three phase circuits with a single > common neutral. I'd strongly suggest that (in addition) you also run the numbers yourself: - power - heat flow (and power for it?) just to make sure (these numbers are large for more than a small single digit size cluster and get absolutely huge for 1000 nodes). Discovering a miscalculation here early is no big deal, late it's a disaster... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 17 14:06:16 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 17 Sep 2003 14:06:16 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <20030917090901.B4711@mas1.ats.ucla.edu> Message-ID: On Wed, 17 Sep 2003, Michael Stein wrote: > Just some additions, good coverage.. Some good additions to my additions, and they reminded me of still more:-) > Make sure the electricians know the power you intend to use -- they > need to size the circuits *larger* than this by some factor (from > code?) -- you can't put a 20 A load on a 20 A breaker. Also make sure that they put a distribution panel (or two) IN THE SPACE. Long runs between panel and receptacle carrying near-peak currents eat voltage, which in turn requires still more current at the lower voltages. Overwiring (using a higher gauge wire than strictly necessary for the run length) is better than marginal or underwiring. > > Oh, and I'd strongly suggest getting a professional architect (one with > > experience doing computer machine rooms) to do your plans. And avoid > > bozo electricians who try e.g. wiring three phase circuits with a single > > common neutral. > > I'd strongly suggest that (in addition) you also run the numbers yourself: > > - power > - heat flow (and power for it?) > > just to make sure (these numbers are large for more than a small > single digit size cluster and get absolutely huge for 1000 nodes). > > Discovering a miscalculation here early is no big deal, late it's > a disaster... Very good advice. I also forgot to mention two other important, even critical issues and a Useful Rule of Thumb. a) Thermal kill switch. You may well want the room to be equipped with one. This is a thermostatted switch that kills all room power if the ambient temperature exceeds some threshold, e.g. 90F. The idea is that if AC fails and you don't get down there to shut down nodes fast enough, the kill switch kills power all at once rather than permit the nodes to overheat to where they physically break (as they will if they get hot enough). Remember, at 10 KW/rack, four racks is 40 KW all being released in your itty bitty 15x25' room. The room will go from ambient cool to meltdown in a matter of minutes if AC fails, and (Murphy's law being what it is) it WILL FAIL sooner or later. b) When working out the AC, remember that in many locations, especially ones where it gets cold in the winter, physical plant people often engage in the unhealthy practice of turning off their chillers or putting them in a winter standby mode. After all, its bloody cold outside! Who needs a chiller! You do. Believe me. If the chilled water (or whatever) entering your machine room stops being (say) 42F and starts being the standby temperature of (say) 70F, the ambient air going into the front of your nodes will rapidly go up to (say) 85F or even 90F, trip your kill switch or break your nodes because it ALMOST trips and stays hot for days, and make your life a living hell as you try to convince the physical plant people to turn the air conditioning back on and they tell you that they aren't budgeted for year round operation and besides it will ice up their coils. Been there, done that. So be sure to get your physical plant people (or whoever is to run your actual air conditioning chiller/blower system) to sign a contract written in blood and promising you their firstborn child if they do a silly little thing like turn the air conditioning off the first time the outdoor temperature drops to ten below zero or something. Then be prepared to firebomb their houses when they do it anyway (just kidding, just kidding...:-). c) Don't forget to budget in the COST OF OPERATION for all of this infrastructure when you go with your hat in your hands looking for money. The remodelling will be a lot more than you think it will be by the time you get the 16 or so tons of AC you need, the huge blower, the chiller lines, and all that harmonically mitigated, line conditioned power. But THEN you get to buy the electricity and cooling. A reasonable rule of thumb estimate for electrical power is: $1 per watt per year of 24x7 operation. This includes both the power itself and the cost of the AC required to remove that power continuously. Note that it is only an estimate, and reality could be lower by a bit or higher by a factor of two, depending on local utility costs. For your presumed 40 KW cluster, though, you're looking at $40K/year just to keep the thing plugged in and running, AFTER paying off the renovation costs and buying all the hardware. I didn't address the raised floor issue here or before, by the way, but yes there are some good things about a raised floor design. For example, it is relatively easy to deliver AC and power directly to the racks from underneath. On the other hand, the raised floor has to support the racks and they are (as noted) likely to be HEAVY and to TORQUE on their floor support as well as just press down on it. Also, the hidden wiring is an advantage when it doesn't need to be worked on, then it becomes a small disadvantage relative to more open and accessible trays. I suspect it costs more too. Somebody who has tried both kinds of floor and prefers raised may be able to do better than this. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laurenceliew at yahoo.com.sg Wed Sep 17 14:31:14 2003 From: laurenceliew at yahoo.com.sg (Laurence Liew) Date: 18 Sep 2003 02:31:14 +0800 Subject: Need Rocks cluster feedback In-Reply-To: <3F686253.F7ABBF70@scs.carleton.ca> References: <1063789278.82f649c0amazonia@myrealbox.com> <3F686253.F7ABBF70@scs.carleton.ca> Message-ID: <1063823473.2962.408.camel@scalable> Hi, You can get lam-mpi for Rocks from www.x2ca.com download section... Cheers! Laurence On Wed, 2003-09-17 at 21:32, Robert Kane wrote: > The experience I've had with Rocks has been fairly positive. For what > it's worth though we're still using v2.3.2 and v3.0. > > The pros for Rocks are generally the ease of use. Everything (OS and > tools [excluding lam-mpi]) are included in one very straightforward > package. Getting a cluster up and running is fairly quick. A quick linux > install on the frontend and then a couple minutes for each node to > install. Assuming no problems, getting our 64 node cluster from scratch > to up and running shouldn't take more than half-day. > > The cons. The above time estimate assumed that no problems occurred. > When upgrading recently we did have some form of inexplicable glitch > that caused the post-install configuration on the frontend to fail and > have to be configured manually. I should point out that this error was > the only time something like that has been known to happen. I should > also point out that another pro to using Rocks is the helpful advice of > the Rocks developers. When we had the above mentioned glitch, one of the > developers graciously spent a fair amount of time making sure everything > was working fine. > > Robert Kane > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mas at ucla.edu Wed Sep 17 14:53:42 2003 From: mas at ucla.edu (Michael Stein) Date: Wed, 17 Sep 2003 11:53:42 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: ; from rgb@phy.duke.edu on Wed, Sep 17, 2003 at 02:06:16PM -0400 References: <20030917090901.B4711@mas1.ats.ucla.edu> Message-ID: <20030917115342.A5030@mas1.ats.ucla.edu> > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). > > Remember, at 10 KW/rack, four racks is 40 KW all being released in your > itty bitty 15x25' room. The room will go from ambient cool to meltdown > in a matter of minutes if AC fails, and (Murphy's law being what it is) > it WILL FAIL sooner or later. At that size/power a lot less than minutes. 1 1U machine might output about 30 CFM of 99 F air. 4 racks full of them (42 * 4) would be about 5000 CFM of 99 F air. You have 15 x 25 x 8? so about 3000 cubic feet of 75 F air to start. At 5000 CFM through the machines the whole room will be 99 F in about 36 seconds. After that it gets interesting (the machines are now taking in 99 F air)... And this assumes uniform/perfect air flow (no hot/cold spots). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 17 16:00:09 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 17 Sep 2003 16:00:09 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <20030917115342.A5030@mas1.ats.ucla.edu> Message-ID: On Wed, 17 Sep 2003, Michael Stein wrote: > > a) Thermal kill switch. You may well want the room to be equipped > > with one. This is a thermostatted switch that kills all room power if > > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > > that if AC fails and you don't get down there to shut down nodes fast > > enough, the kill switch kills power all at once rather than permit the > > nodes to overheat to where they physically break (as they will if they > > get hot enough). > > > > Remember, at 10 KW/rack, four racks is 40 KW all being released in your > > itty bitty 15x25' room. The room will go from ambient cool to meltdown > > in a matter of minutes if AC fails, and (Murphy's law being what it is) > > it WILL FAIL sooner or later. > > At that size/power a lot less than minutes. > > 1 1U machine might output about 30 CFM of 99 F air. 4 racks full of > them (42 * 4) would be about 5000 CFM of 99 F air. > > You have 15 x 25 x 8? so about 3000 cubic feet of 75 F air to start. > At 5000 CFM through the machines the whole room will be 99 F in about > 36 seconds. After that it gets interesting (the machines are now taking > in 99 F air)... > > And this assumes uniform/perfect air flow (no hot/cold spots). Ahh, you theorists. My claim of "minutes" was experimental. Alas. (Just kidding, I'm really a theorist myself;-) The point being that minutes or seconds, a thermal kill switch is very much a good idea. So are things like netbotz, other machine readable thermal monitors, or the use of lmsensors and so forth of you've got 'em. An automated kill SCRIPT that triggers before the kill SWITCH permits clean shutdown instead of instant loss of power (good even though with e.g. ext3 the latter is no longer quite so worrisome). Anyway, in our direct experience it isn't QUITE the "less than a minute" a pessimistic estimate yields, probably because there is a certain amount of thermal ballast in the room itself and the cases and the air (I think you're overestimating the perfection of the circulation and mixing process, for example), the walls and floor and ceiling do let some heat out, especially if they start good and cold (concrete has a fairly high specific heat), and if it is "just" the chiller that goes out but the blower keeps running there is a bit more "stored cold" (I know, stored absence of heat:-) in the AC coils and ductwork. We've found semi-empirically (the hard way) that it takes between five and fifteen minutes for the room space to really get over 90 on a full AC kill, admittedly starting an easy 10F colder than 75 -- barely enough time to run a distributed shutdown that heads off some of the meltdown/thermal kill process, or run down and open the door to the hall and throw a big fan in if one happens to be paying close attention. In fact, I'd also generally recommend holding the room ambient temperature well under 70, (as low as 60F or lower if you can stand it) partly to increase this window of where you can intervene less destructively than with a thermal kill, partly because computer components tend to give up roughly a year of projected lifetime for every 10F increase in ambient operating temperature. So a cluster with ambient air at 75F will have a LOT more hardware problems and a shorter lifetime than one at 65F, and one with ambient 85F air will just plain break all the time, including bitlevel errors that don't actually permanently break the hardware but cause a node to freeze up. Our cluster operated for a fairly extended period (several weeks) with ambient air in the high 70's - low 80's when they first turned off the chiller last winter before we convinced them that several hundred thousand dollars worth of equipment was at risk and that they'd PROMISED to keep the room cold all year long back in our original design meeting for the space. We got to see a whole lot of these problems firsthand -- I still have broken nodes lying around that are good only for spare parts, and everybody experienced systems crashes and lost work. We keep the incoming air pretty cold, just about as cold as we possibly can. After mixing, the air ends up cold but not unbearably cold ambient, a bit colder in front of racks (where the air vents are directed). In front it is actively uncomfortable due to a cold draft (low grade wind) you can catch cold of it. Behind a rack the air is a balmy 75F or so after mixing (not right in front of the vents but a few inches away). Warm but not hot. This is probably what extends our meltdown time out to minutes, and is also why we keep a jacket down there for general use -- in the summertime, shorts, sandles, and a hawaiian shirt outside don't translate well into cluster room garb;-) rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Wed Sep 17 17:23:09 2003 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 17 Sep 2003 14:23:09 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. References: Message-ID: <000401c37d62$02b773b0$35a8a8c0@laptop152422> > > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). Don't have the kill turn off ALL the power.. Only turn off the power to the computers, but leave the lights and some receptacles alive. More than once, I had to stumble around in a computer room that was totally dark after someone hit the Emergency OFF accidentally. Bear in mind that if you have all that fancy UPS gear, the kill switch needs to turn off the output of the UPS, not the line side. There are actually code requirements for some installations (mostly aimed at helping out firement, who don't want to hit the Emergency OFF, and have the equipment still energized). This will typically be done by things like "shunt trips".... your competent A&E person should know what's supposed to be done. You also don't want the thermal kill to shut down the power to the blowers, now, do you? There are books on computer room design around, and while not everything is applicable, a lot is. It might be worth finding one at the library or technical bookstore and browsing through it so that you can be an "educated consumer". _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Wed Sep 17 14:24:50 2003 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Wed, 17 Sep 2003 14:24:50 -0400 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: <20030917090901.B4711@mas1.ats.ucla.edu> Message-ID: <20030917182450.GA15185@central.cis.upenn.edu> Raised floor vs. overhead cable trays: raised floor requires careful routing underneath -- you don't want to keep lifting tiles and snaking things back and forth when changing cabling; also, it can get cramped in there unless you're talking 6' raised floors (ahhhh, to work for the government) overhead cable trays are a pain to route as well, but end routing is much easier to manipulate. depending on AC with raised floor means you must keep the apertures to a minimum (both size and number) else you lose cooling air flow everywhere raised floor? make sure you leave enough room between rack rows to be able to pull at least two tiles in between (back to careful routing of cables) dell's 42" racks fit nicely in 1x2 tile areas, fwiw. torque on raised floor? we use 4-post racks exclusively, no connecting to the tiles/floor, just screw the extensions down to floor level I'm not sure of the advantage of hidden wiring except if you're comparing raised floor to running cables on the floor (the latter being a really bad idea regardless of comparison). BTW we always have at least two A/C's for each room, one semi-redundant. Regards, Dan W. > I didn't address the raised floor issue here or before, by the way, but > yes there are some good things about a raised floor design. For > example, it is relatively easy to deliver AC and power directly to the > racks from underneath. On the other hand, the raised floor has to > support the racks and they are (as noted) likely to be HEAVY and to > TORQUE on their floor support as well as just press down on it. Also, > the hidden wiring is an advantage when it doesn't need to be worked on, > then it becomes a small disadvantage relative to more open and > accessible trays. I suspect it costs more too. > > Somebody who has tried both kinds of floor and prefers raised may be > able to do better than this. > > rgb -- -- Daniel Widyono http://www.cis.upenn.edu/~widyono -- Liniac Project, CIS Dept., SEAS, University of Pennsylvania -- Mail: CIS Dept, 302 Levine 3330 Walnut St Philadelphia, PA 19104 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Wed Sep 17 16:28:09 2003 From: lathama at yahoo.com (Andrew Latham) Date: Wed, 17 Sep 2003 13:28:09 -0700 (PDT) Subject: Building a small machine room? Message-ID: <20030917202809.1339.qmail@web80501.mail.yahoo.com> A book and Ideas. first get the National Electric Code. it applies to everything and is what the fire code is biased off of. second contact a Halon installation company (chemical fire stopper - sprinklers are bad) look at Hoffman enclosures and B-line ladders (possible to use these to get away from dedicated room and stay cool) ask the physics department for help on heat dispersal there are much more efficient methods putting up expanded metal under or between sheet-rock can reduce RF. If a new structure. try buried hoses for a geothermal effect. ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shuntim.luk at polyu.edu.hk Thu Sep 18 00:18:54 2003 From: shuntim.luk at polyu.edu.hk (LUK ShunTim) Date: Thu, 18 Sep 2003 12:18:54 +0800 Subject: Searchable beowulf list archive Message-ID: <3F69322E.9060504@polyu.edu.hk> Hello, There used to be a searchable archive for this list at http://www.supercomputer.org/Search/ but when I tried to use it just now, it is no longer there. Is there other alternative location? Regards, ST -- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From fluffie at gmx.net Thu Sep 18 07:13:45 2003 From: fluffie at gmx.net (Fluffie) Date: Thu, 18 Sep 2003 13:13:45 +0200 (MEST) Subject: PVM: problems with 'add host' Message-ID: <21108.1063883625@www28.gmx.net> Hi to all, I'm running PVM 3.4.3 on a Linux 6.4 machine and on a Windows 2000 machine. On the Linux machine I tried to add the Windows machine like this: pvm> add XYZ.XYZ.XYZ.XYZ add XYZ.XYZ.XYZ.XYZ 0 successful HOST DTID XYZ.XYZ.XYZ.XYZ Can't start pvmd Auto-Diagnosing Failed Hosts... XYZ.XYZ.XYZ.XYZ... Verifying Local Path to "rsh"... Rsh found in /usr/bin/rsh - O.K. Testing Rsh/Rhosts Access to Host "XYZ.XYZ.XYZ.XYZ"... Rsh/Rhosts Access is O.K. Checking O.S. Type (Unix test) on Host "XYZ.XYZ.XYZ.XYZ"... Checking O.S. Type (Win test) on Host "XYZ.XYZ.XYZ.XYZ"... Host XYZ.XYZ.XYZ.XYZ is Windows-based. Checking %PVM_ROOT% on Host "XYZ.XYZ.XYZ.XYZ"... %PVM_ROOT% on XYZ.XYZ.XYZ.XYZ Appears O.K. ("C:\Programme\PVM3.4 ") Verifying Location of PVM Daemon Script on Host "XYZ.XYZ.XYZ.XYZ"... PVM Daemon Script Found ("pvmd.bat") Determining PVM Architecture on Host "XYZ.XYZ.XYZ.XYZ"... %PVM_ARCH% on XYZ.XYZ.XYZ.XYZ set to WIN32 Verifying Existence of PVM Daemon Executable on Host "XYZ.XYZ.XYZ.XYZ"... PVM Daemon Executable Found ("pvmd3.exe") Determining PVM Temporary Directory on Host "XYZ.XYZ.XYZ.XYZ"... %PVM_TMP% on XYZ.XYZ.XYZ.XYZ set to c:\tmp Checking for Leftover PVM Daemon Files on Host "XYZ.XYZ.XYZ.XYZ"... No PVM Daemon Files Found. Host XYZ.XYZ.XYZ.XYZ Appears to Be Correctly Configured. Please check your local /tmp/pvml.501 log file for error messages, or else email "pvm at msr.epm.ornl.gov" for further assistance. The "pvmd3.exe" process has been started on the win2K machine, but the Linux machine does not recognize it. On the contrary, the auto diagnose starts after a certain time as seen above. The content of the pvml.501 log file is: [t80040000] 09/03 11:14:05 (127.0.0.2:1028) LINUX 3.4.3 [t80040000] 09/03 11:14:05 ready Wed Sep 3 11:14:05 2003 [t80000000] 09/03 11:14:21 stderr at XYZ.XYZ.XYZ.XYZ: The command "$PVM_ROOT" [t80000000] 09/03 11:14:21 stderr at XYZ.XYZ.XYZ.XYZ: could not be found. [t80000000] 09/03 11:14:21 stdout at XYZ.XYZ.XYZ.XYZ: EOF [t80000000] 09/03 11:15:21 pl_startup() giving up on host XYZ.XYZ.XYZ.XYZ after 60 secs [t80040000] 09/03 11:15:21 startack() host XYZ.XYZ.XYZ.XYZ expected version, got "PvmCantStart" What else can be wrong??? Stefan -- +++ GMX - die erste Adresse f?r Mail, Message, More! +++ Getestet von Stiftung Warentest: GMX FreeMail (GUT), GMX ProMail (GUT) (Heft 9/03 - 23 e-mail-Tarife: 6 gut, 12 befriedigend, 5 ausreichend) Jetzt selbst kostenlos testen: http://www.gmx.net _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmarch at prodigy.net Thu Sep 18 02:57:22 2003 From: jmarch at prodigy.net (Jim March) Date: Wed, 17 Sep 2003 23:57:22 -0700 Subject: Need the help of a Beowulf rig for a political problem... References: <20030917090901.B4711@mas1.ats.ucla.edu> <20030917115342.A5030@mas1.ats.ucla.edu> Message-ID: <00b801c37db2$1e98ffe0$dcc74fd1@me> Folks, I realize this will probably be about the weirdest thing to hit this mailing list. Short form: we have a real-life encyption problem with serious implications on our hands. We need to break a ZIP password that we suspect has up to 15 characters in the password, and only one file. More or less a worst-case scenario. There is a ZIP password breaker application at http://zipcracker.bengburken.net/ - with versions compatible with BeoWulf. The password-protected ZIP file in question is at: http://www.equalccw.com/ATL-TSRepair.zip We *think* it was zipped and encrypted with WinRAR: http://www.rarlab.com/download.htm Right, so what's going on here? This file was one of 40,000 pulled off of an open FTP site mis-managed by incompetents at Diebold Elections Systems. DES sells the computers and software to run elections; among their customers are the entire state of Georgia, at least 12 counties in California and scads of others. The software is proprietary, the source code is held as tightly as MS holds their stuff, and in the case of their new "touchscreen" product, there's no paper trail at all. All of the security measures are electronic. Only one company, a testing lab chosen by the Federal Elections Commission ("Metamor", now known as "Ciber Inc") got access to the source code; state and county-level elections officials and Secretaries of State were supposed to approve voting products while able to see how compiled and running systems work, but with no access to the source code themselves. The president of Diebold corporation is a Republican party activist and fundraiser who has been quoted as saying he's "going to deliver Ohio to George Bush", while at the same time submitting bids on voting systems in that state. http://www.portclintonnewsherald.com/news/stories/20030827/localnews/140871.html Any alarms going off yet? It gets worse. WAY worse. http://www.scoop.co.nz/mason/stories/HL0307/S00065.htm - this is the results of testing of the actual Diebold compiled Windows code. This sucker is rigged for vote fraud six ways from Sunday. Guys, they used MS-Access as a database engine, THEN deliberately crippled the security in at least two different ways. It's utter madness. Wanna test it for yourself? Here's your kit - download the executables and live voting data which Diebold conveniently left up on their site back in January: http://www.equalccw.com/dieboldtestnotes.html Since we (a pretty "diverse" bunch of activists) started showing people how the Deibold code works, Diebold insiders have been leaking internal memos and manuals. Y'all literally won't believe some of this stuff - they KNEW exactly what they were doing the whole time: http://workersrighttovote.org/bbv/diebold-memos-1.htm - we have a HUGE archive of Diebold Elections division EMail traffic - this file is a "best of" selection of that. Horrifying stuff, including deception of the Federally-approved testing lab ("Metamor", now called "Ciber, Inc"). http://www.scoop.co.nz/mason/stories/HL0309/S00150.htm - New Zealand mirror to the above... http://www.equalccw.com/smokinggun.pdf - if you want to see what the most damning of these internal EMails looked like as they originally appeared on Diebold's internal system... http://www.scoop.co.nz/mason/stories/HL0309/S00157.htm - Full copy plus the comical highlights of a Diebold internal manual on how to run elections-day procedures, written for the hapless Diebold (Canadian) tech staff. This material was NEVER intended for the public, and in places is drop-dead funny. http://www.equalccw.com/ElectionSupportGuide.pdf - same internal manual in it's original, unmodified form complete with corporate logos, fancy formatting and such. If you're going to show a copy to reporters/politicians/etc, use this one as it's clearly genuine. OK, so what's with the ATL-TSRepair.zip file? It's a Microsoft Access database (.MDB internally) dated just four days AFTER the November of 2002 general elections - elections in which a huge number of Republican "surprise victories" broke out. We don't know what the hell is on it, but it's fishy as hell - "ATL" probably refers to Atlanta. Why would they even have a small voting data file created after the election, unless it was to hide something nasty? Another problem: the filename is a lie. "TS" means "TouchScreen", the voting terminals. But those systems don't USE .MDB files - data is transmitted to the central box running GEMS ("Global Elections Management Software") right after the polls close, but not in Access format. So no possible "TouchScreen Repair File" could involve MS-Access data. Prior to the elections, a huge number of poorly-tested patches for the touchscreen terminals (running Windows CE!) were being passed around - it appears this file was designed to mimic those, but with ZIP encryption, you can still load the file enough to see the full filename and original date inside the compressed ZIP file. We want to know what's in there. If you have a beefy BeoWulf cluster PVM 3.4.2 or greater, and this project is of interest, drop me a line. VIA PGP :) unless you like getting Diebold cease'n'desist orders :). My real name is Jim March, email is jmarch at prodigy.net and my public key is registered at the default PGP keyserver at ldap://certserver.pgp.com - if you see two entries, use the SECOND (or later-date one, they're a week apart, only the 2nd one works). Sorry for the intrusion and this'll be my last posting to your list. But y'all gotta admit, it's one hell of a cool real-world encryption problem :). I'm BCCing a small number of activists involved in this, including Bev Harris, the lady who did the original documentation on the matter and is basically the leader (as much as we have one...). Jim March _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 18 08:50:37 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 18 Sep 2003 08:50:37 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <000401c37d62$02b773b0$35a8a8c0@laptop152422> Message-ID: On Wed, 17 Sep 2003, Jim Lux wrote: > You also don't want the thermal kill to shut down the power to the blowers, > now, do you? You don't but ours does, I'm pretty sure. I'm not certain what the precise motivation was -- I believe it was the issue of those firemen, not knowing what circuits in the room were on and what were off. > There are books on computer room design around, and while not everything is > applicable, a lot is. It might be worth finding one at the library or > technical bookstore and browsing through it so that you can be an "educated > consumer". Do you have any specific ones to recommend? Amazon is around the metaphorical corner, and it seems like this would be a useful addition to my bookshelf. Although at this point I've learned a lot on my own the very hardest of ways;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 18 09:25:56 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 18 Sep 2003 15:25:56 +0200 (CEST) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: Message-ID: On Wed, 17 Sep 2003, Robert G. Brown wrote: > > This is probably what extends our meltdown time out to minutes, and is > also why we keep a jacket down there for general use -- in the > summertime, shorts, sandles, and a hawaiian shirt outside don't > translate well into cluster room garb;-) Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. What would it consist of then? As you say a fleece for warmth. Lots of pockets for tools and screws. Ideas? Then again, we would have to get systems engineers to model it on the catwalk... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Thu Sep 18 09:49:59 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 18 Sep 2003 08:49:59 -0500 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: Message-ID: <1063892998.1106.86.camel@caffeine> On Thu, 2003-09-18 at 08:25, John Hearns wrote: > On Wed, 17 Sep 2003, Robert G. Brown wrote: > > > > This is probably what extends our meltdown time out to minutes, and is > > also why we keep a jacket down there for general use -- in the > > summertime, shorts, sandles, and a hawaiian shirt outside don't > > translate well into cluster room garb;-) > > Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. > What would it consist of then? > As you say a fleece for warmth. Lots of pockets for tools and screws. > Ideas? > > Then again, we would have to get systems engineers to model it on the > catwalk... > What next, a computer room edition of "Trading Spaces"? -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at lathama.com Thu Sep 18 09:03:10 2003 From: lathama at lathama.com (Andrew Latham) Date: Thu, 18 Sep 2003 06:03:10 -0700 (PDT) Subject: Building a small machine room? In-Reply-To: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <20030918130310.99878.qmail@web80507.mail.yahoo.com> Halon is used in situations where water is a threat. If a sprinkle head starts dumping water on a system pluged into a UPS it will get ugly. Halon is not banned from telcom rooms so just sell it to the fire marshal as a telcom room. I was under the impression that the room would have no people permanently stationed in it. Halon does not kill people. It replaces Oxygen. It can injure or kill a person after large and lengthy exposure but not likely. --- Daniel Kidger wrote: > > > > second contact a Halon installation company > > (chemical fire stopper - sprinklers are bad) > > Unfortunately in many places, halon is now banned from new machine room > installations - upon the mistaken belief that human lives are worth more > than the computers. > > > Daniel. > > -------------------------------------------------------------- > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > ----------------------- www.quadrics.com -------------------- > ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Thu Sep 18 09:59:35 2003 From: shewa at inel.gov (Andrew Shewmaker) Date: Thu, 18 Sep 2003 07:59:35 -0600 Subject: Searchable beowulf list archive In-Reply-To: <3F69322E.9060504@polyu.edu.hk> References: <3F69322E.9060504@polyu.edu.hk> Message-ID: <3F69BA47.4050509@inel.gov> LUK ShunTim wrote: > Hello, > > There used to be a searchable archive for this list at > http://www.supercomputer.org/Search/ > but when I tried to use it just now, it is no longer there. > > Is there other alternative location? The Mailing list ARChives at AIMS are nice. http://marc.theaimsgroup.com/?l=beowulf&r=1&w=2v -Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 18 10:43:05 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 18 Sep 2003 10:43:05 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <1063892998.1106.86.camel@caffeine> Message-ID: On 18 Sep 2003, Dean Johnson wrote: > On Thu, 2003-09-18 at 08:25, John Hearns wrote: > > On Wed, 17 Sep 2003, Robert G. Brown wrote: > > > > > > This is probably what extends our meltdown time out to minutes, and is > > > also why we keep a jacket down there for general use -- in the > > > summertime, shorts, sandles, and a hawaiian shirt outside don't > > > translate well into cluster room garb;-) > > > > Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. > > What would it consist of then? > > As you say a fleece for warmth. Lots of pockets for tools and screws. > > Ideas? > > > > Then again, we would have to get systems engineers to model it on the > > catwalk... > > > > What next, a computer room edition of "Trading Spaces"? Well, we have the makings of a great plot in the form of helping to crack an encrypted file that could contain evidence of political crimes that would make watergate pale in comparison. I doubt that it needs a cluster -- anybody that would leave incriminating information around (encrypted or not) in a publically accessible place is bound to be stupid enough to choose a crypt key like "elephant_man" or "republicans_rule". Or the name of their collie. crack is overkill. Fifteen minutes of hunt and peck and they'll have it. The only question is -- comedy or tragedy? I wanna be played by John Malkovich. rgb > > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Thu Sep 18 11:13:00 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Thu, 18 Sep 2003 08:13:00 -0700 (PDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <1063892998.1106.86.camel@caffeine> Message-ID: On 18 Sep 2003, Dean Johnson wrote: > On Thu, 2003-09-18 at 08:25, John Hearns wrote: > > On Wed, 17 Sep 2003, Robert G. Brown wrote: > > > > > > This is probably what extends our meltdown time out to minutes, and is > > > also why we keep a jacket down there for general use -- in the > > > summertime, shorts, sandles, and a hawaiian shirt outside don't > > > translate well into cluster room garb;-) > > > > Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. > > What would it consist of then? > > As you say a fleece for warmth. Lots of pockets for tools and screws. > > Ideas? > > > > Then again, we would have to get systems engineers to model it on the > > catwalk... > > > > What next, a computer room edition of "Trading Spaces"? can't get much for $1000 > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sth at ceh.ac.uk Thu Sep 18 11:57:11 2003 From: sth at ceh.ac.uk (Steve Hindmarsh) Date: Thu, 18 Sep 2003 16:57:11 +0100 Subject: Building a small machine room? Message-ID: Halon replacements are now available here in the UK at least - we have recently replaced our server room halon system with FM200 which is supposed to be as effective but safer for humans (and the environment). I don't think you are allowed to install new Halon systems in the UK now... Steve >>> Andrew Latham 18/09/2003 14:03:10 >>> Halon is used in situations where water is a threat. If a sprinkle head starts dumping water on a system pluged into a UPS it will get ugly. Halon is not banned from telcom rooms so just sell it to the fire marshal as a telcom room. I was under the impression that the room would have no people permanently stationed in it. Halon does not kill people. It replaces Oxygen. It can injure or kill a person after large and lengthy exposure but not likely. --- Daniel Kidger wrote: > > > > second contact a Halon installation company > > (chemical fire stopper - sprinklers are bad) > > Unfortunately in many places, halon is now banned from new machine room > installations - upon the mistaken belief that human lives are worth more > than the computers. > > > Daniel. > > -------------------------------------------------------------- > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > ----------------------- www.quadrics.com -------------------- > ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Sep 18 11:20:01 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Thu, 18 Sep 2003 11:20:01 -0400 Subject: more NCBI BLAST testing on opterons Message-ID: <1063898401.4635.179.camel@protein.scalableinformatics.com> Folks: We have been doing more testing on Opterons. This time a 246 unit. For the particular NCBI BLAST job we are using (blastx of 74 sequences vs nr from July, will happily supply both the database and the test scripts to anyone in exchange for their results and machine details), the dual Opteron is showing an execution time with 2 threads giving the best results, and the Xeon 3 GHz system showing best results with 4 threads (odd, HT must be working). What we are seeing is that comparing the best efforts of these two platforms, using the binary suitable with gcc on both platforms, that the 246 completes the run in 586 seconds, while the Xeon completes it in 1006 seconds. These are wall clock seconds, as measured by the start and end times of the script. Note: We are trying to work with the Intel compiler as well to see if it helps. We are using sse2 on gcc, and m64 mode on the Opteron. We are also seeing some interesting affinity bits. It looks like we need a way to pin memories and CPUs together for scheduling for best performance. Has anyone successfully built the NCBI toolkit using the ICC? If so, would you be willing to share your linux.ncbi.mk file? Thanks! Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Thu Sep 18 11:47:38 2003 From: jcownie at etnus.com (James Cownie) Date: Thu, 18 Sep 2003 16:47:38 +0100 Subject: Building a small machine room? In-Reply-To: Message from Andrew Latham of "Thu, 18 Sep 2003 06:03:10 PDT." <20030918130310.99878.qmail@web80507.mail.yahoo.com> Message-ID: <1A010M-7bE-00@etnus.com> > Halon is not banned from telcom rooms so just sell it to the fire marshal as a > telcom room. Maybe not in the US, but some of us are in Europe (at least Dan and I)... e.g. http://www.groundworkwales.org.uk/business/oct2002.htm#Halon Fire suppression using Halon will be banned under EU Regulation 2037/2000 from 31st December 2002. From 31st December 2003, all existing systems and extinguishers must be decommissioned, except for those industries that are designated as critical users e.g. aircraft compartments, the Channel Tunnel, and military vessels. The full regulation is here http://europa.eu.int/smartapi/cgi/sga_doc?smartapi!celexapi!prod!CELEXnumdoc&lg=EN&numdoc=32000R2037&model=guichett -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Thu Sep 18 11:14:20 2003 From: lathama at yahoo.com (Andrew Latham) Date: Thu, 18 Sep 2003 08:14:20 -0700 (PDT) Subject: Q: Building a small machine room? Message-ID: <20030918151420.59284.qmail@web80504.mail.yahoo.com> Keep in mind that the below mentioned Disconnects are required by the National Electric Code. The NEC has an ever growing section on computers (low voltage telecommunication devices & rooms) I friend who taught me the NEC is also a writer for it. He being an old electrician himself stated that the code is for saving lives and should be read as such. Yes he has witnessed larger buildings burn because of faulty wiring of only low voltage systems (it was a small office telephone company that messed up). LathamA Subject: Re: Q: Building a small machine room? Materials/costs/etc. Date: Wed, 17 Sep 2003 14:23:09 -0700 > > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). Don't have the kill turn off ALL the power.. Only turn off the power to the computers, but leave the lights and some receptacles alive. More than once, I had to stumble around in a computer room that was totally dark after someone hit the Emergency OFF accidentally. Bear in mind that if you have all that fancy UPS gear, the kill switch needs to turn off the output of the UPS, not the line side. There are actually code requirements for some installations (mostly aimed at helping out firement, who don't want to hit the Emergency OFF, and have the equipment still energized). This will typically be done by things like "shunt trips".... your competent A&E person should know what's supposed to be done. You also don't want the thermal kill to shut down the power to the blowers, now, do you? There are books on computer room design around, and while not everything is applicable, a lot is. It might be worth finding one at the library or technical bookstore and browsing through it so that you can be an "educated consumer". ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 18 12:18:37 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 18 Sep 2003 09:18:37 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: <000401c37d62$02b773b0$35a8a8c0@laptop152422> Message-ID: <5.2.0.9.2.20030918091503.02f397b8@mailhost4.jpl.nasa.gov> At 08:50 AM 9/18/2003 -0400, Robert G. Brown wrote: >On Wed, 17 Sep 2003, Jim Lux wrote: > > > You also don't want the thermal kill to shut down the power to the blowers, > > now, do you? > >You don't but ours does, I'm pretty sure. I'm not certain what the >precise motivation was -- I believe it was the issue of those firemen, >not knowing what circuits in the room were on and what were off. > I suppose this is why you want someone who has done this sort of design before... There's no real electrical reason why you can't have a thermal kill separate from the Emergency OFF kill. All a matter of relays, subpanels, etc. There IS a reason to shut down blowers when overtemp hits.. Most fire alarm systems (in big installations) are tied to the forced air ventilation system to prevent a fire from being pushed by the HVAC. This was generally motivated by the experience in the MGM fire disaster a few decades back (which is also responsible for things like "plenum rated" cabling, etc.) So, what you probably want is a sort of staged setup.. moderate overtemp shuts down computer power, bigger overtemp (i.e. a fire) shuts down the blowers, pressing the big red button next to the door shuts down all the power. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From johnb at quadrics.com Thu Sep 18 09:39:22 2003 From: johnb at quadrics.com (John Brookes) Date: Thu, 18 Sep 2003 14:39:22 +0100 Subject: Q: Building a small machine room? Materials/costs/etc. Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA7E5E2E7@stegosaurus.bristol.quadrics.com> Just a couple of quick points on raised versus, erm, not-raised floors. A raised floor is far and away the better idea for a nice, stable system, IMHO. It's easier to dress off your cables nicely and neatly if you've got a nice bit of under-floor space to store any excess. If the machine room will house only clusters that form single rows, then you'd be wasting cash getting a raised floor, to an extent. If you're going to have multiple, interconnected rows, though, then the cables'd have to go overhead without that space. Overheads are at least as difficult to sort out as under-floor (YMMV - I'm a bit of a shorta**e :) So you're left with one major difference: PRICE. A raised floor to support that kind of weight is not inexpensive, I would imagine. Just my tuppence. John Brookes Quadrics > -----Original Message----- > From: Robert G. Brown [mailto:rgb at phy.duke.edu] > Sent: 17 September 2003 19:06 > To: Michael Stein > Cc: Brian Dobbins; beowulf at beowulf.org > Subject: Re: Q: Building a small machine room? Materials/costs/etc. > > > On Wed, 17 Sep 2003, Michael Stein wrote: > > > Just some additions, good coverage.. > > Some good additions to my additions, and they reminded me of still > more:-) > > > Make sure the electricians know the power you intend to use -- they > > need to size the circuits *larger* than this by some factor (from > > code?) -- you can't put a 20 A load on a 20 A breaker. > > Also make sure that they put a distribution panel (or two) IN > THE SPACE. > Long runs between panel and receptacle carrying near-peak currents eat > voltage, which in turn requires still more current at the lower > voltages. Overwiring (using a higher gauge wire than > strictly necessary > for the run length) is better than marginal or underwiring. > > > > Oh, and I'd strongly suggest getting a professional > architect (one with > > > experience doing computer machine rooms) to do your > plans. And avoid > > > bozo electricians who try e.g. wiring three phase > circuits with a single > > > common neutral. > > > > I'd strongly suggest that (in addition) you also run the > numbers yourself: > > > > - power > > - heat flow (and power for it?) > > > > just to make sure (these numbers are large for more than a small > > single digit size cluster and get absolutely huge for 1000 nodes). > > > > Discovering a miscalculation here early is no big deal, late it's > > a disaster... > > Very good advice. > > I also forgot to mention two other important, even critical > issues and a > Useful Rule of Thumb. > > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). > > Remember, at 10 KW/rack, four racks is 40 KW all being > released in your > itty bitty 15x25' room. The room will go from ambient cool > to meltdown > in a matter of minutes if AC fails, and (Murphy's law being > what it is) > it WILL FAIL sooner or later. > > b) When working out the AC, remember that in many locations, > especially ones where it gets cold in the winter, physical > plant people > often engage in the unhealthy practice of turning off their > chillers or > putting them in a winter standby mode. After all, its bloody cold > outside! Who needs a chiller! > > You do. Believe me. If the chilled water (or whatever) entering your > machine room stops being (say) 42F and starts being the standby > temperature of (say) 70F, the ambient air going into the front of your > nodes will rapidly go up to (say) 85F or even 90F, trip your kill > switch or break your nodes because it ALMOST trips and stays hot for > days, and make your life a living hell as you try to convince the > physical plant people to turn the air conditioning back on > and they tell > you that they aren't budgeted for year round operation and besides it > will ice up their coils. Been there, done that. > > So be sure to get your physical plant people (or whoever is > to run your > actual air conditioning chiller/blower system) to sign a contract > written in blood and promising you their firstborn child if they do a > silly little thing like turn the air conditioning off the > first time the > outdoor temperature drops to ten below zero or something. Then be > prepared to firebomb their houses when they do it anyway > (just kidding, > just kidding...:-). > > c) Don't forget to budget in the COST OF OPERATION for all of this > infrastructure when you go with your hat in your hands looking for > money. The remodelling will be a lot more than you think it > will be by > the time you get the 16 or so tons of AC you need, the huge > blower, the > chiller lines, and all that harmonically mitigated, line conditioned > power. But THEN you get to buy the electricity and cooling. A > reasonable rule of thumb estimate for electrical power is: > > $1 per watt per year of 24x7 operation. > > This includes both the power itself and the cost of the AC required to > remove that power continuously. Note that it is only an estimate, and > reality could be lower by a bit or higher by a factor of two, > depending > on local utility costs. For your presumed 40 KW cluster, > though, you're > looking at $40K/year just to keep the thing plugged in and running, > AFTER paying off the renovation costs and buying all the hardware. > > I didn't address the raised floor issue here or before, by > the way, but > yes there are some good things about a raised floor design. For > example, it is relatively easy to deliver AC and power directly to the > racks from underneath. On the other hand, the raised floor has to > support the racks and they are (as noted) likely to be HEAVY and to > TORQUE on their floor support as well as just press down on it. Also, > the hidden wiring is an advantage when it doesn't need to be > worked on, > then it becomes a small disadvantage relative to more open and > accessible trays. I suspect it costs more too. > > Somebody who has tried both kinds of floor and prefers raised may be > able to do better than this. > > rgb > > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Thu Sep 18 12:08:31 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Thu, 18 Sep 2003 09:08:31 -0700 (PDT) Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> Message-ID: well actually halon is banned most places for new installations because it's a cfc... http://www.epa.gov/ozone/defns.html#halon It is possible to be grandfathered or get an exclusion but there's really no reason any more. In some installations osha rules requires the availablity of scba gear (self contained breathing aparatus) in the facility. our telco switchroom is setup in such a fashion... There are a number of acceptable replacemnts for total flood dry chemical fire suppression systems... which would include thing like intergen ig-541, hcfc-22, ammonium phosphate evirogel, halotron, or even good old CO2 On Thu, 18 Sep 2003, Andrew Latham wrote: > Halon is used in situations where water is a threat. If a sprinkle head starts > dumping water on a system pluged into a UPS it will get ugly. > > Halon is not banned from telcom rooms so just sell it to the fire marshal as a > telcom room. I was under the impression that the room would have no people > permanently stationed in it. > > Halon does not kill people. It replaces Oxygen. It can injure or kill a person > after large and lengthy exposure but not likely. > > --- Daniel Kidger wrote: > > > > > > > second contact a Halon installation company > > > (chemical fire stopper - sprinklers are bad) > > > > Unfortunately in many places, halon is now banned from new machine room > > installations - upon the mistaken belief that human lives are worth more > > than the computers. > > > > > > Daniel. > > > > -------------------------------------------------------------- > > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > > ----------------------- www.quadrics.com -------------------- > > > > ===== > Andrew Latham > > Penguin loving, moralist agnostic. > > LathamA.com - (lay-th-ham-eh) > lathama at lathama.com - lathama at yahoo.com > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 18 12:22:36 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 18 Sep 2003 09:22:36 -0700 Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> References: < <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <5.2.0.9.2.20030918091902.018acd08@mailhost4.jpl.nasa.gov> At 06:03 AM 9/18/2003 -0700, Andrew Latham wrote: >Halon is used in situations where water is a threat. If a sprinkle head >starts >dumping water on a system pluged into a UPS it will get ugly. Precisely why they require the emergency off to kill the UPS load side power! >Halon is not banned from telcom rooms so just sell it to the fire marshal as a >telcom room. I was under the impression that the room would have no people >permanently stationed in it. > >Halon does not kill people. It replaces Oxygen. It can injure or kill a person >after large and lengthy exposure but not likely. Halon is now incredibly expensive and an "ozone depleter", so that first "proof of performance" test after installation will set you back a pretty penny. Back in 1974, I saw a demo at the National Computer Conference from the mfrs of Halon systems where the sales guy stood in a big phone booth like tube, poured solvent all over his clothes, lit it on fire, and had the Halon discharge and put the fire out, all while he was talking through the presentation. One of the selling points was that it was much safer than CO2, which, til then, was the popular "non-water" fire extinguishing agent. >--- Daniel Kidger wrote: > > > > > > > second contact a Halon installation company > > > (chemical fire stopper - sprinklers are bad) > > > > Unfortunately in many places, halon is now banned from new machine room > > installations - upon the mistaken belief that human lives are worth more > > than the computers. > > > > > > Daniel. > > James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Thu Sep 18 08:18:23 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Thu, 18 Sep 2003 13:18:23 +0100 Subject: Building a small machine room? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> > second contact a Halon installation company > (chemical fire stopper - sprinklers are bad) Unfortunately in many places, halon is now banned from new machine room installations - upon the mistaken belief that human lives are worth more than the computers. Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Thu Sep 18 12:32:43 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 18 Sep 2003 11:32:43 -0500 Subject: Building a small machine room? In-Reply-To: References: Message-ID: <1063902762.1120.96.camel@caffeine> I have an alternative (albeit silly) solution, but mind you it doesn't make sense in quite a few cases. Take the money you were going to spend on clever (and expensive) fire suppression systems in the bank and use the ordinary building sprinkler systems. In the highly unlikely event that you have a serious fire (you have an offsite data backup, right?), you take the money you banked and buy much better gear for cheaper. For department level COTS clusters, this might be the most cost effective method. -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Wally-Edmondson at utc.edu Thu Sep 18 12:47:22 2003 From: Wally-Edmondson at utc.edu (Wally Edmondson) Date: Thu, 18 Sep 2003 12:47:22 -0400 Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> References: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <5.1.1.6.0.20030918115942.00b4b280@imap.utc.edu> An alternative to Halon is FM-200 gas. It is safer for humans and won't harm any equipment. The worst that it can do is give you frostbite if you are standing directly under one of the nozzles. Of course, there are flashing lights and beeping noises before it dumps, giving everyone a chance to evacuate (or cancel the dump with a switch near the door). It is installed in my server room. We have never had the misfortune to use it, but it lets me sleep peacefully at night. Here is a link comparing it to Halon: http://www.fireturnkey.com/sys_halonalt.htm. At 06:03 AM 9/18/2003 -0700, Andrew Latham wrote: >Halon is used in situations where water is a threat. If a sprinkle head >starts >dumping water on a system pluged into a UPS it will get ugly. > >Halon is not banned from telcom rooms so just sell it to the fire marshal as a >telcom room. I was under the impression that the room would have no people >permanently stationed in it. > >Halon does not kill people. It replaces Oxygen. It can injure or kill a person >after large and lengthy exposure but not likely. > >--- Daniel Kidger wrote: > > > > > > > second contact a Halon installation company > > > (chemical fire stopper - sprinklers are bad) > > > > Unfortunately in many places, halon is now banned from new machine room > > installations - upon the mistaken belief that human lives are worth more > > than the computers. > > > > > > Daniel. > > > > -------------------------------------------------------------- > > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > > ----------------------- www.quadrics.com -------------------- > > > >===== >Andrew Latham > >Penguin loving, moralist agnostic. > >LathamA.com - (lay-th-ham-eh) >lathama at lathama.com - lathama at yahoo.com >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 18 12:25:21 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 18 Sep 2003 12:25:21 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <5.2.0.9.2.20030918091503.02f397b8@mailhost4.jpl.nasa.gov> Message-ID: On Thu, 18 Sep 2003, Jim Lux wrote: > So, what you probably want is a sort of staged setup.. moderate overtemp > shuts down computer power, bigger overtemp (i.e. a fire) shuts down the > blowers, pressing the big red button next to the door shuts down all the power. We have some of these (including the big red switch by the door) but not all. Maybe soon -- the first is really a matter of sensors and scripts. rgb > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Thu Sep 18 09:24:54 2003 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Thu, 18 Sep 2003 09:24:54 -0400 Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> References: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> <20030918130310.99878.qmail@web80507.mail.yahoo.com> Message-ID: <20030918132454.GA10234@central.cis.upenn.edu> > Halon does not kill people. It replaces Oxygen. It can injure or kill a > person after large and lengthy exposure but not likely. http://www.umich.edu/~oseh/guidhalo.pdf OSEH Health Guideline for Halogenated Fire Extinguishing Systems Dan W. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From frank at joerdens.de Thu Sep 18 12:28:31 2003 From: frank at joerdens.de (Frank Joerdens) Date: Thu, 18 Sep 2003 18:28:31 +0200 Subject: Beowulfery under CYGWIN Message-ID: <20030918182831.B26638@superfly.archi-me-des.de> I've got a wee little 10-node Athlon 1800+ cluster here in the office which is sitting idle a lot of the time. It's designated purpose is Cinema4D rendering, which only runs under Windoze, unfortunately (I *really* tried to get it to work under WINE, it's no-go: some arcane bug relating to wsock32, it seems, where I'm totally out of depth as to fixing it, and the WINE people weren't interested enough in the problem), so I can't run Linux on it (dual-boot I looked into also; it's impractical in our scenario here). Hence my thinking, seeing that I'd like to learn some Beowulfery, that it might be possible to do it under CYGWIN. Has anyone tried yet? Would it be worth having a go at it, or is it clear from the start that you'd have to hack the sources quite extensively? Regards, Frank _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Thu Sep 18 14:43:30 2003 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Thu, 18 Sep 2003 13:43:30 -0500 Subject: Beowulfery under CYGWIN In-Reply-To: <20030918182831.B26638@superfly.archi-me-des.de>; from frank@joerdens.de on Thu, Sep 18, 2003 at 06:28:31PM +0200 References: <20030918182831.B26638@superfly.archi-me-des.de> Message-ID: <20030918134330.F26572@mikee.ath.cx> On Thu, 18 Sep 2003, Frank Joerdens wrote: > I've got a wee little 10-node Athlon 1800+ cluster here in the office > which is sitting idle a lot of the time. It's designated purpose is > Cinema4D rendering, which only runs under Windoze, unfortunately (I > *really* tried to get it to work under WINE, it's no-go: some arcane bug > relating to wsock32, it seems, where I'm totally out of depth as to > fixing it, and the WINE people weren't interested enough in the > problem), so I can't run Linux on it (dual-boot I looked into also; it's > impractical in our scenario here). > > Hence my thinking, seeing that I'd like to learn some Beowulfery, that > it might be possible to do it under CYGWIN. > > Has anyone tried yet? Would it be worth having a go at it, or is it > clear from the start that you'd have to hack the sources quite > extensively? I don't know about the full architecture, but I do know that sshd runs just fine as a service under cygwin. Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Thu Sep 18 15:29:51 2003 From: deadline at plogic.com (Douglas Eadline) Date: Thu, 18 Sep 2003 15:29:51 -0400 (EDT) Subject: Informal Survey In-Reply-To: Message-ID: Back in July, I asked people about how they find information about clusters (see below). I finally got around to collecting the answers. You may view them on http://www.cluster-rant.com/ as part of the "New Poll" story. While you are there, take the cluster size poll. I think the results will be interesting. Thanks, Doug On Wed, 9 Jul 2003, Douglas Eadline wrote: > > I am curious where everyone gets information on clusters. > Obviously this list is one source, but what about other > sources. In addition, what kind of information do people most > want/need about clusters. Please comment on the following > questions if you have the time. You can respond to me directly > and I will summarize the results for the list. > > 1. Where do you find "howto" information on clusters > (besides this list) > > a) Google > b) Vendor > c) Trade Show > d) News Sites (what news sites are there?) > e) Other > > 2. If there were a subscription print/web magazine on clusters, what kind > of coverage would you want? > > a) howto information > b) new products > c) case studies > d) benchmarks > e) other > > > Thanks, > > Doug > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Thu Sep 18 16:13:01 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Thu, 18 Sep 2003 15:13:01 -0500 (CDT) Subject: Informal Survey In-Reply-To: Message-ID: On Thu, 18 Sep 2003, Douglas Eadline wrote: > Back in July, I asked people about how they find information about > clusters (see below). I finally got around to collecting the answers. You > may view them on http://www.cluster-rant.com/ as part of the "New Poll" > story. > > While you are there, take the cluster size poll. I think the results > will be interesting. > > Thanks, > > Doug It's too bad the size of the results was so small. Lets try to guess who the 5 people were that knew everything..:) -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Thu Sep 18 16:25:58 2003 From: deadline at plogic.com (Douglas Eadline) Date: Thu, 18 Sep 2003 16:25:58 -0400 (EDT) Subject: Informal Survey In-Reply-To: Message-ID: On Thu, 18 Sep 2003, Rocky McGaugh wrote: > On Thu, 18 Sep 2003, Douglas Eadline wrote: > > > Back in July, I asked people about how they find information about > > clusters (see below). I finally got around to collecting the answers. You > > may view them on http://www.cluster-rant.com/ as part of the "New Poll" > > story. > > > > While you are there, take the cluster size poll. I think the results > > will be interesting. > > > > Thanks, > > > > Doug > > > It's too bad the size of the results was so small. > > Lets try to guess who the 5 people were that knew everything..:) Or the same person who voted 5 times. I would be great to get more input. The readership of cluster rant is growing, so maybe the polls with start to have some slight significance. They should be taken with a grain of salt however, I like to think of them as ball park view of what is going on. Doug > > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Thu Sep 18 16:57:24 2003 From: lars at meshtechnologies.com (Lars Henriksen) Date: 18 Sep 2003 22:57:24 +0200 Subject: Beowulfery under CYGWIN In-Reply-To: <20030918182831.B26638@superfly.archi-me-des.de> References: <20030918182831.B26638@superfly.archi-me-des.de> Message-ID: <1063918644.2273.11.camel@fermi> On Thu, 2003-09-18 at 18:28, Frank Joerdens wrote: > I've got a wee little 10-node Athlon 1800+ cluster here in the office > which is sitting idle a lot of the time. It's designated purpose is > Cinema4D rendering, which only runs under Windoze, unfortunately (I > *really* tried to get it to work under WINE, it's no-go: some arcane bug > relating to wsock32, it seems, where I'm totally out of depth as to > fixing it, and the WINE people weren't interested enough in the > problem), so I can't run Linux on it (dual-boot I looked into also; it's > impractical in our scenario here). > > Hence my thinking, seeing that I'd like to learn some Beowulfery, that > it might be possible to do it under CYGWIN. You might wanna consider doing it the other way round. Running Linux on the nodes, and on top of that VMWare with the appropriate Windows and your application. Gather the virtual windows desktops in a single VNC desktop under linux eases the pain of running multiple windows machines :-) I've done this with some success in a 12 node setup. Best regards Lars -- Lars Henriksen | MESH-Technologies ApS Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rburns at cs.ucsb.edu Thu Sep 18 19:40:21 2003 From: rburns at cs.ucsb.edu (Ryan Burns) Date: Thu, 18 Sep 2003 16:40:21 -0700 Subject: Cluster distributions? Message-ID: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Hello, I've been running a very small 4 node beowulf cluster for a project. Right now I'm running Debian on the cluster and I'm not very happy with it or the tools available through the distribution. Does anyone have any suggestions of a good distribution? What about MSC.Linux? any good? I am also looking for tools to automate software installations across all the nodes, etc. I like the Debian package system, I just wish there was an easy way to have the process replicate across all the computers. Thanks, Ryan Burns _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Thu Sep 18 20:13:43 2003 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Thu, 18 Sep 2003 17:13:43 -0700 (PDT) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: hi ya ryan fun stuff.... On Thu, 18 Sep 2003, Ryan Burns wrote: > Hello, > > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? other cluster distro and installers http://Www.Linux-Consulting.com/Cluster/ > I am also looking for tools to automate software installations across all > the nodes, etc. installing -- cdrom, network install, rsync updating -- create your-local*.rpm and each node downloads new "required" updates that has been tested on other "test clusters" - or rsync or *.tgz or ?? - and a package/scripting status files to say which machined applied which patches to itself > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. there is .. lots of easy ways ... - "easy" is all relative to the user's ideas ?? personally, i don't like debians installer or *.deb packages or any other *.rpm packages but use the same equivalent "my-apt-get.pl *.tgz" and i am happy as a clam c ya alvin _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 18 19:58:21 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 18 Sep 2003 16:58:21 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: <5.2.0.9.2.20030918091503.02f397b8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030918164810.018b2108@mailhost4.jpl.nasa.gov> At 12:25 PM 9/18/2003 -0400, Robert G. Brown wrote: >On Thu, 18 Sep 2003, Jim Lux wrote: > > > So, what you probably want is a sort of staged setup.. moderate overtemp > > shuts down computer power, bigger overtemp (i.e. a fire) shuts down the > > blowers, pressing the big red button next to the door shuts down all > the power. > >We have some of these (including the big red switch by the door) but not >all. Maybe soon -- the first is really a matter of sensors and scripts. > > rgb Personally, I prefer not relying on a computer (that's doing anything else) to shut down the computer for a serious overtemp. I'd rather see a sensor and a hardwired connection to some sort of relay. OK, I might go for a dedicated PLC (programmable logic controller), but I'd want it to be rock solid and robust in a variety of environments, something that no PC is ever going to be (PCs are just too complex with too many logic gates internally). Sometimes simple is good. The sensors and scripts would be good for the graceful shutdown as you get close to the "yellow limit". Anyone who is enamored of the total software control of things that can cause physical damage should remember one word: "THERAC-25"... Nancy Leveson at MIT has a site at http://sunnyday.mit.edu/ where there is much about software safety. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Luc.Vereecken at chem.kuleuven.ac.be Fri Sep 19 06:57:18 2003 From: Luc.Vereecken at chem.kuleuven.ac.be (Luc Vereecken) Date: Fri, 19 Sep 2003 12:57:18 +0200 Subject: Building a small machine room? In-Reply-To: <5.2.0.9.2.20030918091902.018acd08@mailhost4.jpl.nasa.gov> References: <20030918130310.99878.qmail@web80507.mail.yahoo.com> < <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <3.0.6.32.20030919125718.00930560@arrhenius.chem.kuleuven.ac.be> >At 06:03 AM 9/18/2003 -0700, Andrew Latham wrote: >>Halon is not banned from telcom rooms so just sell it to the fire marshal as a >>telcom room. I was under the impression that the room would have no people >>permanently stationed in it. >> >>Halon does not kill people. It replaces Oxygen. It can injure or kill a person >>after large and lengthy exposure but not likely. > Actually, Halon does not replace oxygen. CO2 extinguishers work by replacing oxygen, depriving the flames of their oxidant. Halon molecules (which contain halogen atoms, primarily Bromine) decompose in the flame, and the Bromine atoms catalyse the recombination of radicals in the flame, causing the flame to die from a lack of active radicals. This is why you need clouds of CO2, but only a small percentage of halons (< 0.5% ? ) in the air to stop a fire. The O2-concentration is hardly affected by this small amount of halon. At 09:22 AM 9/18/03 -0700, Jim Lux wrote: >Halon is now incredibly expensive and an "ozone depleter", . Because the bromine atoms, released after decomposition in the stratosphere by hard radiation, are efficient radical quenchers, they efficiently start to quench our ozone (O3 radicals). Bummer. Which is why you should never use Halon, even if it would have been legal to do so, and certainly not use any loopholes in the law to use it anyway. Putting the entire world population at risk because it is convenient for yourself is not acceptable. A number of alternatives have already been mentioned in other posts. :-) Luc Vereecken (who uses his cluster to work on combustion chemistry (-: ) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 19 10:34:43 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 19 Sep 2003 16:34:43 +0200 (CEST) Subject: Building a small machine room? In-Reply-To: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: On Thu, 18 Sep 2003, Daniel Kidger wrote: > > Unfortunately in many places, halon is now banned from new machine room > installations - upon the mistaken belief that human lives are worth more > than the computers. That's sad. Where will BOFH be without his Halon hold-off button to torture unsuspecting service engineers and IT managers? http://www.theregister.co.uk/content/30/30779.html "No, I mean, lift a couple of floor tiles, tie some cat-5 cable at ankle height between racks, unscrew the Halon Hold-off knobs, remove the doorknobs from the inside of the computer room doors and hide the oxygen masks." _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 19 10:44:20 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 19 Sep 2003 16:44:20 +0200 (CEST) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: On Thu, 18 Sep 2003, Ryan Burns wrote: > Hello, > > I am also looking for tools to automate software installations across all > the nodes, etc. If you like apt-get, check out apt-get for RPM at www.freshrpms.net And Yum at http://linux.duke.edu/projects/yum/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Sep 19 12:12:54 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 19 Sep 2003 12:12:54 -0400 (EDT) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: On Thu, 18 Sep 2003, Ryan Burns wrote: > Hello, > > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? > I am also looking for tools to automate software installations across all > the nodes, etc. > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. I think that your best choices are likely to be one of the following: Scyld ("free" version of commercial open source, probably, for such a small cluster) or Clustermatic (fully GPL). Red Hat with kickstart and yum (kickstart is precisely an easy way to install boilerplate scripted systems, although yes you have to learn to use it). If your systems have PXE-capable NICs you can get it to where you can (re)install a system over the network by just turning it on, or turning it on and selecting kickstart at a boot prompt. Yum then keeps the systems up to date, provided that you mirror your base repository from a site that mirrors a site that mirrors Red Hat that keeps the repository up to date with regard to at least security fixes and major bugfixes. Oscar, Rocks or other cluster toolsets (as discussed on the list just last week, check list archives) have tools to facilitate this kind of thing as well, often on top of one or more of the major distributions. Don't be too hasty to drop Debian -- you may just not be using it right. Check out the Debian FAI (Fully Automated Installation) site. All the links you might need for this stuff (in addition to being readily available via google) are collected on: http://www.phy.duke.edu/brahma/Resources/links.php along with a lot of other stuff, of course. Some of which you might also find interesting, but which isn't directly related to cluster installation methodologies. rgb > > > Thanks, > Ryan Burns > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rkane at scs.carleton.ca Fri Sep 19 11:48:38 2003 From: rkane at scs.carleton.ca (Robert Kane) Date: Fri, 19 Sep 2003 11:48:38 -0400 (EDT) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: If no one has already mentioned it, Rocks (www.rocksclusters.org) is a nice distribution to use. It's a customized distribution of Redhat 7.x. It's stable, runs on a widely used and supported version of Linux, and has all sorts of cluster tools (monitors, schedulers, mpi) integrated. It's generally quick to install, and easy to manage. Robert Kane > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? > I am also looking for tools to automate software installations across all > the nodes, etc. > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Sep 19 12:29:50 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 19 Sep 2003 12:29:50 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <5.2.0.9.2.20030918164810.018b2108@mailhost4.jpl.nasa.gov> Message-ID: On Thu, 18 Sep 2003, Jim Lux wrote: > At 12:25 PM 9/18/2003 -0400, Robert G. Brown wrote: > >On Thu, 18 Sep 2003, Jim Lux wrote: > > > > > So, what you probably want is a sort of staged setup.. moderate overtemp > > > shuts down computer power, bigger overtemp (i.e. a fire) shuts down the > > > blowers, pressing the big red button next to the door shuts down all > > the power. > > > >We have some of these (including the big red switch by the door) but not > >all. Maybe soon -- the first is really a matter of sensors and scripts. > > > > rgb > > Personally, I prefer not relying on a computer (that's doing anything else) > to shut down the computer for a serious overtemp. I'd rather see a sensor > and a hardwired connection to some sort of relay. OK, I might go for a > dedicated PLC (programmable logic controller), but I'd want it to be rock > solid and robust in a variety of environments, something that no PC is ever > going to be (PCs are just too complex with too many logic gates > internally). Sometimes simple is good. I had in mind something like one or more "server" class hosts with sensors connected that they poll every minute or five minutes or so (depending on how small your space is and how paranoid you are). This just gives you access to the room temperature at the sensor locations inside e.g. scripts. At that point, you can do anything you like -- mail out a warning at threshold A, initiate automated powerdowns by banks or all together at temperaure B (saving servers for last so all writebacks can occur) -- whatever. netbotz are expensive, but they provide such an interface (and more) completely canned and SNMP or web-accessible and even have internal mailers to do the mailing at an alarm threshold. And expensive is cheap for people who don't want to DIY and do want to protect a few hundred thousand dollars of hardware. Beyond that it isn't horribly easy to find attachable (e.g.) RS232 readable thermal sensors, but it can be done. Some are linked to brahma/Resources/vendors.php): the ibutton, sensorsoft, pico tech. There are likely more if one is good at googling. One of these days I'm going to get this set up for the cluster/server room. It actually already has lots of built in monitoring and automated emergency action, but it is all at the University level and the cluster control is all local, so I think it will be good to have the intermediate layer under our own immediate automagic control. > The sensors and scripts would be good for the graceful shutdown as you get > close to the "yellow limit". Precisely. > Anyone who is enamored of the total software control of things that can > cause physical damage should remember one word: "THERAC-25"... This is what I'm concerned about the other way, actually. We're already plugged into a University level IT sensor structure beyond our immediate control AND that provides us with no immediate information at the systems level in the room itself. By the time even their 24 hour service responds to a thermal problem, it is likely to be WAY too late because whether the meltdown (or rather, thermal kill) time is less than a minute or less then fifteen minutes, it is >>short<< once AC is really interrupted. We just got grazed by a potentially interrupting event (and yes, damn it, I've been Franned again and my house has no electricity and probably won't for several days, if the past is any guide). I'd really prefer the yellow limit scripts, sorted out so that nodes shut down first and maybe even only. If the room is large enough, it can sometimes manage to handle the heat output of only a few servers via thermal ballast and losses through walls and floor and so forth, especially if one can provide even an ordinary rotating fan to move the air. rgb > > Nancy Leveson at MIT has a site at http://sunnyday.mit.edu/ where there is > much about software safety. > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jsims at csiopen.com Fri Sep 19 13:24:33 2003 From: jsims at csiopen.com (Joey Sims) Date: Fri, 19 Sep 2003 13:24:33 -0400 Subject: Gigabit Switch Recommendation Message-ID: <812B16724C38EE45A802B03DD01FD5472605E7@exchange.concen.com> http://www.dninetworks.com/prod_lan_feature.asp?id=43 12port - GbE Jumbo Frames Support ---------------------------------------------------- |Joey P. Sims 800.995.4274 x 242 |Sales Manager 770.442.5896 - Fax |HPC/Storage Division jsims at csiopen.com |Concentric Systems,Inc. www.csilabs.net ---------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From julien.leduc at imag.fr Fri Sep 19 15:05:21 2003 From: julien.leduc at imag.fr (Julien Leduc) Date: Fri, 19 Sep 2003 19:05:21 +0000 Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> References: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: <20030919190521.718f8407.julien.leduc@imag.fr> Hi, > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? You should have a try with CLIC distribution or mandrake clustering, it automates deployment of the nodes and setup everything (NIS, dhcp, pxe ...) you need to run the cluster. Deployment is very fast (I am working on it, intensively use it), and reliable (for example, it takes me 3 minutes to clone 15 nodes, including the reboot). > I am also looking for tools to automate software installations across all > the nodes, etc. There is also urpmi parallel, that allow you to install packages over the cluster (you can even install a single machine with compiled sources and deploy it on the cluster). > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. If you want, you can only use the ka-tools, with a "correctly" configured environement (ie ssh to any node from a server without pass authentification, relying on keys for ex), you can execute your aptget on all your nodes (the problem will be the bandwidth throughput here, but I don't think it is an issue for a 4 nodes cluster...). You can try to use tools to spread your .deb packages efficiently over the network (mput for example, from the ka-tools, so that you won't have to compile anything but these tools to optimize your already existing installation). These tools are developped in my lab, and are very performant (and scalable) and also opensource. So you should have a closer look at http://ka-tools.sourceforge.net/ the are in deepth explanations about the method used. Regards, Julien Leduc Developper & Cluster admin Informatique et distribution Web: http://www-id.imag.fr _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Fri Sep 19 14:39:08 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Fri, 19 Sep 2003 20:39:08 +0200 Subject: Q: Building a small machine room? Materials/costs/etc. References: Message-ID: <3F6B4D4C.8070506@moene.indiv.nluug.nl> Robert G. Brown wrote: > We have some of these (including the big red switch by the door) but not > all. Maybe soon -- the first is really a matter of sensors and scripts. Do you also have the "door-opening-switch" right next to the BRS ? That can lead to very funny mistakes (especially if you're not the one making the mistake). "I'll see you out !". Wham ! SSSSsssssssshhhhhhhh :-) -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From glen at cert.ucr.edu Fri Sep 19 15:58:58 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Fri, 19 Sep 2003 12:58:58 -0700 Subject: opteron problems Message-ID: <3F6B6002.2020606@cert.ucr.edu> Hi, We've purchased a few Opteron's to play with, and they're turning out to not be very stable. The motherboard in the systems is the Tyan Thunder K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 beta on the other two. Each machine has the latest version of the bios from Tyan's website. I'm running the setiathome client, and a few of our own applications to stress test the systems. All the machines seem to either lock up, crash with a kernel panic, or reboot out of the blue. The lock ups, crashes, and reboots seem to happen at random about once a day, to once every 3 days. So, I was wondering if anyone else has had similar issues, and if you figured out any way to overcome them? Thanks for your time, Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian at chpc.utah.edu Fri Sep 19 16:43:07 2003 From: brian at chpc.utah.edu (Brian D. Haymore) Date: Fri, 19 Sep 2003 14:43:07 -0600 (MDT) Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> Message-ID: I have a couple dozen opterons in house on the arima board and have had no such issues as you are reporting. We did have a tyan in house for a couple days but did not give it a thorough test to see what your are seeing. We have been running Suse 8.1 and 8.2 beta on them as well as the beta of RedHat WS (Taroon) for x86-64 on those that we have. On Fri, 19 Sep 2003, Glen Kaukola wrote: > Hi, > > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios > from Tyan's website. I'm running the setiathome client, and a few of > our own applications to stress test the systems. All the machines seem > to either lock up, crash with a kernel panic, or reboot out of the > blue. The lock ups, crashes, and reboots seem to happen at random about > once a day, to once every 3 days. > > So, I was wondering if anyone else has had similar issues, and if you > figured out any way to overcome them? > > Thanks for your time, > Glen > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112-0190 Email: brian at chpc.utah.edu - Phone: (801) 585-1755 - Fax: (801) 585-5366 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Fri Sep 19 16:39:09 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Fri, 19 Sep 2003 13:39:09 -0700 (PDT) Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> Message-ID: On Fri, 19 Sep 2003, Glen Kaukola wrote: > Hi, > > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios > from Tyan's website. I'm running the setiathome client, and a few of > our own applications to stress test the systems. All the machines seem > to either lock up, crash with a kernel panic, or reboot out of the > blue. The lock ups, crashes, and reboots seem to happen at random about > once a day, to once every 3 days. > > So, I was wondering if anyone else has had similar issues, and if you > figured out any way to overcome them? we have 4 rioworks hdama boards with two 240s ea and two 1GB pc 2110 dimms interleaved in the primary cpu's bank 0 and 2. They've been pretty stable so far (about 6weeks). > Thanks for your time, > Glen > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 19 18:51:21 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 19 Sep 2003 15:51:21 -0700 Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> References: <3F6B6002.2020606@cert.ucr.edu> Message-ID: <20030919225121.GA5148@greglaptop.internal.keyresearch.com> On Fri, Sep 19, 2003 at 12:58:58PM -0700, Glen Kaukola wrote: > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. You didn't comment on what kernel you're running. 2.4.22 has some patches for the ia32 mode on the opteron, and these are probably the same patches already in 2.6. We have found apps that reliably crash our Opterons without this patch, but moving to 2.6 was much more stable. We haven't tried 2.4.22. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From glen at cert.ucr.edu Fri Sep 19 21:46:18 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Fri, 19 Sep 2003 18:46:18 -0700 Subject: opteron problems In-Reply-To: <20030919225121.GA5148@greglaptop.internal.keyresearch.com> References: <3F6B6002.2020606@cert.ucr.edu> <20030919225121.GA5148@greglaptop.internal.keyresearch.com> Message-ID: <3F6BB16A.4000409@cert.ucr.edu> Greg Lindahl wrote: >You didn't comment on what kernel you're running. 2.4.22 has some >patches for the ia32 mode on the opteron, and these are probably the >same patches already in 2.6. We have found apps that reliably crash >our Opterons without this patch, but moving to 2.6 was much more >stable. We haven't tried 2.4.22. > > On the redhat systems, I'm just going with the lastest kernel rpm package, kernel-2.4.20-20.9. And on Suse, it's k_smp-2.4.19-237. I hate to not use rpm, but I guess I'll try 2.4.22 and see what that does for me. Thanks for the advice. Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 20 05:39:05 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 20 Sep 2003 11:39:05 +0200 (CEST) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: Message-ID: On Fri, 19 Sep 2003, Robert G. Brown wrote: > netbotz are expensive, but they provide such an interface (and more) > completely canned and SNMP or web-accessible and even have internal > mailers to do the mailing at an alarm threshold. And expensive is cheap > for people who don't want to DIY and do want to protect a few hundred You could add these to your links page: http://web6.scan.co.uk/Products/Info.asp?WPID=42160 Maybe a good alternative for Europeans. Haven't ever used them - I just saw them at RAL last week. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 20 05:31:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 20 Sep 2003 11:31:11 +0200 (CEST) Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> Message-ID: On Fri, 19 Sep 2003, Glen Kaukola wrote: > Hi, > > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios We're running SuSE 8.2 beta on a cluster of dual Opterons. The motherboards are Rioworks HDAMA. We're not seeing the problems you describe - they generall seem stable. Recently upgraded to the latest BIOS from the Rioworks site. The main problem we are seeing is occasinally nodes halting with the message: hda: dma_timer_expiry: dma status == 0x21 I'm sure this is not a ahrdware problem with the disks. Have run kernels 2.4.22 and the SuSE patched 2.4.21. Anyone else seen this? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 20 05:59:36 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 20 Sep 2003 11:59:36 +0200 (CEST) Subject: opteron problems In-Reply-To: <3F6BB16A.4000409@cert.ucr.edu> Message-ID: On Fri, 19 Sep 2003, Glen Kaukola wrote: > On the redhat systems, I'm just going with the lastest kernel rpm > package, kernel-2.4.20-20.9. And on Suse, it's k_smp-2.4.19-237. I > hate to not use rpm, but I guess I'll try 2.4.22 and see what that does > for me. You can build a new kernel as an RPM. In the kernel source directory, do a 'make rpm'. (Or something like that - you get the idea). Also if you are trying 2.4.22, apply the ACPI hotfix patch from ftp://ftp.x86-64.org/pub/linux-x86_64/v2.4 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Sat Sep 20 13:10:12 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Sat, 20 Sep 2003 13:10:12 -0400 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> Message-ID: <3F6C89F4.303@bellsouth.net> Wow! It looks like they're only getting 12 nodes per rack. That's only 24 CPUs per rack (or for a 42U rack it's like 1.75U per CPU). That's like a 4U to hold a dual CPU node (welcome to the good-old days!). Yes, I know this is the only way to get dual G5's right now, but wow (hey look around this website. There's a seminar in the UK on 23 Sept. to discuss the G5 Xserve). So, for 1,100 G5 nodes, it will take at least 92 racks! I hope they have the floor space, power, and AC ready. Cool fans on the top of the racks though. Hey! If you poke around the website, you'll find some notes from the VT presentation. Here are some comments about the facilities: * 3 MW power, double redundant with backups - UPS and diesel o 1.5 MW reserved for the TCF * 2+ million BTUs of cooling capacity using Liebert's extremedensity cooling (rack mounted cooling via liquid refrigerant) o traditional methods [fans] would have produced windspeeds of 60+ MPH * Racks were custom designed for this system Yep, inexpensive, _custom_ racks (as much an oxymoron as military intelligence as I've heard). If they charge $1000 a rack (pretty cheap IMHO), then there's almost $1 million for the racks out of a stated $5 million. OK, should we have a pool or a poll on who's bankrolling this thing? Jeff P.S. Andrew - thanks for the link. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sat Sep 20 12:38:51 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sun, 21 Sep 2003 00:38:51 +0800 (CST) Subject: Virginia Tech PowerMac G5 Cluster Photos Message-ID: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> http://www.chaosmint.com/mac/techclusterphotos/ Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 20 15:23:04 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 20 Sep 2003 15:23:04 -0400 (EDT) Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <3F6C89F4.303@bellsouth.net> Message-ID: On Sat, 20 Sep 2003, Jeffrey B. Layton wrote: > take at least 92 racks! ... > Yep, inexpensive, _custom_ racks (as much an oxymoron > as military intelligence as I've heard). If they charge $1000 > a rack (pretty cheap IMHO), then there's almost $1 million Ummm, 92 x 1000 = ? Don't feel bad. I do the same thing all the time. > for the racks out of a stated $5 million. OK, should we have > a pool or a poll on who's bankrolling this thing? Somebody with very deep pockets. Liquid cooled racks are probably a lot more expensive than $1000 each, but one really should likely lump the power, cooling, racks, and room all together as "base infrastructure" and sure, $1M isn't a crazy estimate for that. That leaves $4M for systems and (possibly) for humans, service agreements, electricity, and more. The power capacity also does not compute wrt their existing configuration. 1000 300W nodes is only 300 KW. Even if the nodes run at 500W each (which would be insanely hot IMHO at 250 W/processor -- about as much as a loaded dual athlon PER PROCESSOR, and enough to make even an Alpha seem cool) it is only 500 KW. Add half again for AC and miscellaneous, and they are still only up to 750 KW (which costs them roughly $500,000/year as an ongoing operating expense. They've got a factor of four surplus power capacity (if not more like six). They must be planning ahead to upgrade those racks with real rackmounts at quadruple the power/compute density during the lifetime of the facility. Assuming a three year grant cycle, $2M for infrastructure and power and (probably) salaries, they've got $3M left for nodes, which works out order of $3K/node. This doesn't sound too insane, but I don't really know the prices of these nodes. Apple has been pushing these nodes for doing bioinformatics and gene cataloguing, so my dollar is on NIH. I doubt NSA or DOE or NSF -- its a bit rich for NSF, DOE tends to be conservative, NSA tends to run their own stuff. DOD is always a possibility, but any of the latter agencies I think would be less inclined to fund an Apple cluster with such a ticklish (liquid cooled) base design. Whoever it is, they expect to go back to the well, I betcha, when rackmounts become available, or else their architect's brother sells big generators and overkill facilities plans. rgb > > Jeff > > P.S. Andrew - thanks for the link. > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Sat Sep 20 17:48:52 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Sat, 20 Sep 2003 17:48:52 -0400 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: References: Message-ID: <3F6CCB44.2020409@bellsouth.net> Robert G. Brown wrote: >On Sat, 20 Sep 2003, Jeffrey B. Layton wrote: > > > >>take at least 92 racks! >> >> >... > > >>Yep, inexpensive, _custom_ racks (as much an oxymoron >>as military intelligence as I've heard). If they charge $1000 >>a rack (pretty cheap IMHO), then there's almost $1 million >> >> > >Ummm, 92 x 1000 = ? > >Don't feel bad. I do the same thing all the time. > Sorry. Homer Simpson moment. >>for the racks out of a stated $5 million. OK, should we have >>a pool or a poll on who's bankrolling this thing? >> >> > >Somebody with very deep pockets. Liquid cooled racks are probably a lot >more expensive than $1000 each, but one really should likely lump the >power, cooling, racks, and room all together as "base infrastructure" >and sure, $1M isn't a crazy estimate for that. > >That leaves $4M for systems and (possibly) for humans, service >agreements, electricity, and more. > >The power capacity also does not compute wrt their existing >configuration. 1000 300W nodes is only 300 KW. Even if the nodes run >at 500W each (which would be insanely hot IMHO at 250 W/processor -- >about as much as a loaded dual athlon PER PROCESSOR, and enough to make >even an Alpha seem cool) it is only 500 KW. Add half again for AC and >miscellaneous, and they are still only up to 750 KW (which costs them >roughly $500,000/year as an ongoing operating expense. They've got a >factor of four surplus power capacity (if not more like six). They must >be planning ahead to upgrade those racks with real rackmounts at >quadruple the power/compute density during the lifetime of the facility. > >Assuming a three year grant cycle, $2M for infrastructure and power and >(probably) salaries, they've got $3M left for nodes, which works out >order of $3K/node. This doesn't sound too insane, but I don't really >know the prices of these nodes. > >Apple has been pushing these nodes for doing bioinformatics and gene >cataloguing, so my dollar is on NIH. I doubt NSA or DOE or NSF -- its a >bit rich for NSF, DOE tends to be conservative, NSA tends to run their >own stuff. DOD is always a possibility, but any of the latter agencies >I think would be less inclined to fund an Apple cluster with such a >ticklish (liquid cooled) base design. Whoever it is, they expect to go >back to the well, I betcha, when rackmounts become available, or else >their architect's brother sells big generators and overkill facilities >plans. > I wasn't thinking so much about NIH, but rather an individual running a company. Perhaps a couple of companies? The notes I saw said it was a collection from several schools within the university. Anyway, it's always fun to speculate (heck, those old retired guys get _paid_ to do it on TV everytime we have a war or a crisis). When RGB starts his new clothing line, perhaps I'll take up beowulf fashion commentary :) Thanks! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Sat Sep 20 20:25:13 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Sat, 20 Sep 2003 17:25:13 -0700 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <3F6C89F4.303@bellsouth.net> References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> <3F6C89F4.303@bellsouth.net> Message-ID: <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: > * 2+ million BTUs of cooling capacity using Liebert's extremedensity > cooling (rack mounted cooling via liquid refrigerant) > o traditional methods [fans] would have produced windspeeds of > 60+ MPH This is kind of weird, given that their energy density isn't that high: 2 cpus in 4U. I've seen machine rooms which have 8 times the energy density. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Sun Sep 21 00:16:04 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 20 Sep 2003 23:16:04 -0500 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> <3F6C89F4.303@bellsouth.net> <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> Message-ID: <1064117764.1291.4.camel@terra> On Sat, 2003-09-20 at 19:25, Greg Lindahl wrote: > On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: > > > * 2+ million BTUs of cooling capacity using Liebert's extremedensity > > cooling (rack mounted cooling via liquid refrigerant) > > o traditional methods [fans] would have produced windspeeds of > > 60+ MPH > > This is kind of weird, given that their energy density isn't that > high: 2 cpus in 4U. I've seen machine rooms which have 8 times the > energy density. > Their energy density is trivial. I can't recall how many watts they are, but my guess is much less than even old celerons. I wouldn't be surprised if each of those whole machines sucks less juice than a single itanic cpu. Wasn't the G4 design spec something like 24 watts. -- -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From anandv at singnet.com.sg Sat Sep 20 01:50:26 2003 From: anandv at singnet.com.sg (Anand Vaidya) Date: Sat, 20 Sep 2003 13:50:26 +0800 Subject: Cluster distributions? In-Reply-To: References: Message-ID: <200309201350.26611.anandv@singnet.com.sg> I was amazed to read that one needs to *reinstall* all nodes just to apply one ssh update. Isn't that very wasteful? Then what happens to the custom ssh keys etc? http://www.rocksclusters.org/errata/Security/ssh-alert.html says: 7.Reinstall all compute nodes # cluster-fork /boot/kickstart/cluster-kickstart Or is there some misunderstanding on my part? Regards, Anand On Friday 19 September 2003 23:48, Robert Kane wrote: > If no one has already mentioned it, Rocks (www.rocksclusters.org) is a > nice distribution to use. It's a customized distribution of Redhat 7.x. > It's stable, runs on a widely used and supported version of Linux, and has > all sorts of cluster tools (monitors, schedulers, mpi) integrated. It's > generally quick to install, and easy to manage. > > Robert Kane > > > I've been running a very small 4 node beowulf cluster for a project. > > Right now I'm running Debian on the cluster and I'm not very happy with > > it or the tools available through the distribution. > > > > Does anyone have any suggestions of a good distribution? What about > > MSC.Linux? any good? > > I am also looking for tools to automate software installations across all > > the nodes, etc. > > I like the Debian package system, I just wish there was an easy way to > > have the process replicate across all the computers. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sun Sep 21 11:26:12 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sun, 21 Sep 2003 23:26:12 +0800 (CST) Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <1064117764.1291.4.camel@terra> Message-ID: <20030921152612.6481.qmail@web16807.mail.tpe.yahoo.com> G5s need more power than G4s for sure, as their die size is larger, but AFAIK, the typical power usage is lower than the Pentium 4s, and thus much much lower than the IA64 chips. Also, it's it a waste to have 1100 brand new ATI video cards, and another 1100 superdrives **unused** in the cluster? (I would love to have them!) Andrew. --- Dean Johnson ???: > Their energy density is trivial. I can't recall how > many watts they are, > but my guess is much less than even old celerons. I > wouldn't be > surprised if each of those whole machines sucks less > juice than a single > itanic cpu. Wasn't the G4 design spec something like > 24 watts. > > -- > > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From luken at linuxnetworx.com Fri Sep 19 21:31:10 2003 From: luken at linuxnetworx.com (Joshua Aune) Date: Fri, 19 Sep 2003 19:31:10 -0600 Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> References: <3F6B6002.2020606@cert.ucr.edu> Message-ID: <20030920013110.GU25847@earth.omner.org> On Fri, Sep 19, 2003 at 12:58:58PM -0700, Glen Kaukola wrote: > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios > from Tyan's website. I'm running the setiathome client, and a few of Do you know if the tyan BIOS has all the latest processor errata for your CPU rev implemented? We were working with 2 very large opteron systems, one with B3 rev CPUs and another with C0 rev CPUs. At the time the BIOS didn't implement all the errata for the B3 rev of CPUs so they crashed regularly with very odd oopses (random memory corruption) while the C0 based systems were solid. After the BIOS was fixed the B3s worked just as well as the C0s. > our own applications to stress test the systems. All the machines seem > to either lock up, crash with a kernel panic, or reboot out of the > blue. The lock ups, crashes, and reboots seem to happen at random about > once a day, to once every 3 days. > So, I was wondering if anyone else has had similar issues, and if you > figured out any way to overcome them? Analyzing lots and lots of oopses :) Josh _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ic02 at uow.edu.au Sat Sep 20 02:15:10 2003 From: ic02 at uow.edu.au (ivano maximus cornelius) Date: Sat, 20 Sep 2003 16:15:10 +1000 Subject: pvm: checking load of hosts Message-ID: <3F6BF06E.4030509@uow.edu.au> Hi all, I am running a pvm based monte carlo calculation consisting of a master and a number of identical spawned programs. I need a particular number of simulations performed and would like to split this number amongst the spawned programs depending upon their load at the time of spawning. Is there an easy way to do this from within my program using a routine in the pvm libraries? Thanks for your help, Iwan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rmyers1400 at comcast.net Sat Sep 20 20:52:23 2003 From: rmyers1400 at comcast.net (Robert Myers) Date: Sat, 20 Sep 2003 20:52:23 -0400 Subject: Searchable beowulf list archive In-Reply-To: <3F69BA47.4050509@inel.gov> References: <3F69322E.9060504@polyu.edu.hk> <3F69BA47.4050509@inel.gov> Message-ID: <3F6CF647.8000705@comcast.net> Andrew Shewmaker wrote: > LUK ShunTim wrote: > >> Hello, >> >> There used to be a searchable archive for this list at >> http://www.supercomputer.org/Search/ >> but when I tried to use it just now, it is no longer there. >> >> Is there other alternative location? > > > The Mailing list ARChives at AIMS are nice. > > http://marc.theaimsgroup.com/?l=beowulf&r=1&w=2v > > > -Andrew That's a good link. Google advanced search, of couse, does just fine, if you specify the site to search as beowulf.org. For me, at least, the search results are a little easier to pick through because the citations link immediately to beowulf's own indices. RM RM RM _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Sat Sep 20 09:46:02 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Sat, 20 Sep 2003 15:46:02 +0200 (CEST) Subject: opteron problems In-Reply-To: References: <3F6B6002.2020606@cert.ucr.edu> Message-ID: <2171.81.57.165.53.1064065562.squirrel@webmail.mandrakesoft.com> > We're running SuSE 8.2 beta on a cluster of dual Opterons. The > motherboards are Rioworks HDAMA. > We're not seeing the problems you describe - they generall seem stable. > Recently upgraded to the latest BIOS from the Rioworks site. I've run MandrakeClustering on this kind of hardware with a 2.4.19 (patched by the Mandrake kernel team) and the error you're talking has never been reproduced. The only problem I had was some troubles with the apic (noapic should be given to the cmdline) but the lastest stable bios (1.72) gives a lot of fixes for Linux and the noapic features is no more needed. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sja at world.std.com Sat Sep 20 15:34:40 2003 From: sja at world.std.com (Stuart J Adams) Date: Sat, 20 Sep 2003 15:34:40 -0400 (EDT) Subject: Distributing makes across the cluster Message-ID: <200309201934.PAA18306@world.std.com> Is there a way to automatically distribute makes across a cluster ?? (Similar to what gnu make -j does on an SMP machine) Thanks, Stuart _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Sun Sep 21 13:22:31 2003 From: lars at meshtechnologies.com (Lars Henriksen) Date: 21 Sep 2003 19:22:31 +0200 Subject: Distributing makes across the cluster In-Reply-To: <200309201934.PAA18306@world.std.com> References: <200309201934.PAA18306@world.std.com> Message-ID: <1064164951.2276.1.camel@fermi> On Sat, 2003-09-20 at 21:34, Stuart J Adams wrote: > Is there a way to automatically distribute makes across > a cluster ?? (Similar to what gnu make -j does on an SMP machine) I have used distcc with success. Kernel compile times drop to 40 sec :-) http://distcc.samba.org/ Hope you can use it. best regards Lars -- Lars Henriksen | MESH-Technologies ApS Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Sun Sep 21 16:17:17 2003 From: landman at scalableinformatics.com (Joe Landman) Date: Sun, 21 Sep 2003 16:17:17 -0400 Subject: Cluster distributions? In-Reply-To: <200309201350.26611.anandv@singnet.com.sg> References: <200309201350.26611.anandv@singnet.com.sg> Message-ID: <3F6E074D.2070504@scalableinformatics.com> Anand Vaidya wrote: >I was amazed to read that one needs to *reinstall* all nodes just to apply one >ssh update. Isn't that very wasteful? Then what happens to the custom ssh >keys etc? > You can simply place the RPM in a shared location and cluster-fork "rpm -Fvh /path/to/new-ssh-binary.rpm ; /etc/init.d/sshd restart" You are not forced to use the re-install method, though it is on by default. Turning it off is relatively simple: cluster-fork "/etc/init.d/rocks-grub stop; /sbin/chkconfig --level 2345 rocks-grub off" Or you can remove it from the node build tree (instructions in the documentation). As for new ssh keys, I found that I simply have to clear out the ~/.ssh/known_hosts. It is annoying to have to do this for all users though. I typically turn off the automatic re-install and default re-install for my customers. Joe > >http://www.rocksclusters.org/errata/Security/ssh-alert.html >says: > >7.Reinstall all compute nodes > # cluster-fork /boot/kickstart/cluster-kickstart > >Or is there some misunderstanding on my part? > >Regards, >Anand > > >On Friday 19 September 2003 23:48, Robert Kane wrote: > > >>If no one has already mentioned it, Rocks (www.rocksclusters.org) is a >>nice distribution to use. It's a customized distribution of Redhat 7.x. >>It's stable, runs on a widely used and supported version of Linux, and has >>all sorts of cluster tools (monitors, schedulers, mpi) integrated. It's >>generally quick to install, and easy to manage. >> >>Robert Kane >> >> >> >>>I've been running a very small 4 node beowulf cluster for a project. >>>Right now I'm running Debian on the cluster and I'm not very happy with >>>it or the tools available through the distribution. >>> >>>Does anyone have any suggestions of a good distribution? What about >>>MSC.Linux? any good? >>>I am also looking for tools to automate software installations across all >>>the nodes, etc. >>>I like the Debian package system, I just wish there was an easy way to >>>have the process replicate across all the computers. >>> >>> >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >>http://www.beowulf.org/mailman/listinfo/beowulf >> >> > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Sun Sep 21 20:54:42 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Sun, 21 Sep 2003 17:54:42 -0700 (PDT) Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> Message-ID: On Sat, 20 Sep 2003, Greg Lindahl wrote: > On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: > > > * 2+ million BTUs of cooling capacity using Liebert's extremedensity > > cooling (rack mounted cooling via liquid refrigerant) > > o traditional methods [fans] would have produced windspeeds of > > 60+ MPH > > This is kind of weird, given that their energy density isn't that > high: 2 cpus in 4U. I've seen machine rooms which have 8 times the > energy density. we have 8 opteron 240's in 4u ;) > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Tue Sep 23 09:38:31 2003 From: jcownie at etnus.com (James Cownie) Date: Tue, 23 Sep 2003 14:38:31 +0100 Subject: SC03 Tutorials Message-ID: <1A1nN9-4zk-00@etnus.com> In case folks have missed it the SC03 tutorial schedule is now available here :- http://www.sc-conference.org/sc2003/tech_tutorials.php and there are many which seem relevant to this audience (for instance) S01: Production Linux Clusters 2003 - Architecture and System Software for Serious Computing S02: A Tutorial Introduction to High Performance Data Transport S03: A practical approach to performance analysis and modeling of large-scale systems S05: An Introduction to the TotalView Debugger S09: Programming with the Partitioned Global Address Space Model M03: Lustre: A Scalable, High-Performance Distributed File System M04: Using MPI-2: Advanced Features of the Message-Passing Interface M07: PERC Tools for Performance Data Gathering and Analysis M09: Cluster Construction Utilizing Selected Packages M13: HPC and InfiniBand Architecture in Practice You can get the details from the site above. -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Tue Sep 23 08:44:08 2003 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Tue, 23 Sep 2003 09:44:08 -0300 (ART) Subject: MPICH vs LAM Message-ID: <20030923124408.91196.qmail@web12205.mail.yahoo.com> What implmentation of MPI should i use? There's the mpich and the lam implamentations, what's the best choice, and have more improviments of resources and performance? Thanks Mathias Brito Universidade Estadual de Santa Cruz Bahia-Brasil _______________________________________________________________________ Desafio AntiZona: participe do jogo de perguntas e respostas que vai dar um Renault Clio, computadores, c?meras digitais, videogames e muito mais! www.cade.com.br/antizona _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Mon Sep 22 14:34:04 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Mon, 22 Sep 2003 14:34:04 -0400 Subject: Ramdisk Size In-Reply-To: <3F6F32F0.1070400@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> Message-ID: <3F6F409C.3040101@bellsouth.net> Tony Travis wrote: > Eric R Johnson wrote: > >> I would like to create a large ramdisk to keep a database which will >> be searched often, but not changed. >> The only thing I currently know how to do to create a ramdisk is: >> mke2fs /dev/ram0 >> mount /dev/ram0 /tmp/ramdisk0 >> >> The only problem is that this creates a 4MB ramdisk, and I would like >> something on the order of 1GB. > > > Hello, Eric. > > Are you *sure* you want to do this? > > RAM disks are usually used for loading modules at boot time. That's > why they are small. Typically, the contents of a RAM disk are used as > a root filesystem for the kernel to load e.g. SCSI drivers, then > discarded when the 'real' root filesystem is mounted from SCSI disks. > People also use them to boot a 'minimal' Linux for diagnostics or for > some other reason. Check out Warewulf: warewulf-cluster.org. Very cool cluster distribution that uses a very small Ramdisk to hold the necessary parts of the OS. I've just started to use it and it's very slick. (snip) > However, I think it would be better to use 'tmpfs' for a 1Gb RAM > filesystem. The kernel will provide fast access to the most frequently > accessed areas of your database held in RAM, but it will move the less > frequently accessed areas onto the swap device freeing up RAM for your > programs and for other disk caches. What's the difference between tmpfs and a ramdisk created through the kernel? Is there a speed difference? BTW, here's a link on creating a generic ram disk: http://www.lissot.net/partition/ramdisk.html (read the bottom part). If you are happy with the speed improvement of using a ram disk to store the database, you might want to consider squashfs (http://squashfs.sourceforge.net/) It's a read-only filesystem that you can use to reduce the size of the required ram disk. You will take a hit in speed, but perhaps the ram disk will make up for it. Oh, and you will have to patch the kernel to use it. Good Luck! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bioinformaticist at mn.rr.com Mon Sep 22 02:04:11 2003 From: bioinformaticist at mn.rr.com (Eric R Johnson) Date: Mon, 22 Sep 2003 01:04:11 -0500 Subject: Ramdisk Size Message-ID: <3F6E90DB.1060505@mn.rr.com> I would like to create a large ramdisk to keep a database which will be searched often, but not changed. The only thing I currently know how to do to create a ramdisk is: mke2fs /dev/ram0 mount /dev/ram0 /tmp/ramdisk0 The only problem is that this creates a 4MB ramdisk, and I would like something on the order of 1GB. I tried running menuconfig and changing the default ramdisk size to 1GB, making the kernel, and rebooting. Now when I run menuconfig, it shows the ramdisk size as 1GB, but when I execute the mke2fs /dev/ram0, it still only creates a 4MB ramdisk. Can someone point me in the right direction? Thanks, Eric _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Mon Sep 22 06:34:39 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Mon, 22 Sep 2003 06:34:39 -0400 Subject: Virginia Tech PowerMac G5 Cluster Photos References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> Message-ID: <3F6ED03F.8E97FA32@epa.gov> Andrew Wang wrote: > > http://www.chaosmint.com/mac/techclusterphotos/ If you go through to the "supercomputercenter link" at the bottom of the page and look at the photos there, you'll get a notice that "this page has been optimised for IE, it may not show on your browser" (and it didn't, for netscape and mozilla). Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Tue Sep 23 12:49:56 2003 From: josip at lanl.gov (Josip Loncaric) Date: Tue, 23 Sep 2003 10:49:56 -0600 Subject: MPICH vs LAM In-Reply-To: <20030923124408.91196.qmail@web12205.mail.yahoo.com> References: <20030923124408.91196.qmail@web12205.mail.yahoo.com> Message-ID: <3F7079B4.4090807@lanl.gov> Mathias Brito wrote: > What implmentation of MPI should i use? There's the > mpich and the lam implamentations, what's the best > choice, and have more improviments of resources and > performance? Why not install both and give yourself a choice? MPICH is very popular and works fine, but LAM is a bit cleaner implementation which delivers a bit lower latency. Other than that, the main difference is process startup. With MPICH, "mpirun" does the job directly and accepts the usual rsh or ssh authorization overhead. With LAM, you "lamboot" LAM daemons on your nodes (w/usual overhead) but then those daemons can start your MPI processes quicker. After your MPI jobs (possibly many in sequence) are done, you use "wipe" to clean up LAM daemons. On the Coral cluster at ICASE (now absorbed by NIA), LAM was the default due to slightly lower latency. LAM is very nicely done, but having only one flavor of MPI can be limiting, so people could use MPICH, MPI/Pro (in VIA/TCP/interrupt/polling modes), or MVICH as they saw fit. For example, some codes assume MPICH-specific implementation details, which is non-standard but easy to accomodate. My suggestion is to install both LAM and MPICH, then benchmark your applications with both and use the better choice as the default. Sincerely, Josip P.S. There are other MPIs out there: various vendor MPIs, enhanced flavors of MPICH, etc. Some of these cost $/CPU/year; others are free. A commercial MPI (e.g. MPI/Pro) buys you enhanced robustness and some interesting features, e.g. interrupt-driven receive which can improve performance of certain applications. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Tue Sep 23 08:56:24 2003 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Tue, 23 Sep 2003 09:56:24 -0300 (ART) Subject: MPI Parallel algorithms Message-ID: <20030923125624.73310.qmail@web12206.mail.yahoo.com> I starting to research how to program my beowulg cluster, i already made a lot of simple programs, but i would like to see more complexs algorithms. Someone know where can I find parallel algorithms codes with MPI on the web. Thanks Mathias Brito Universidade Estadual de Santa Cruz Bahia-Brasil _______________________________________________________________________ Desafio AntiZona: participe do jogo de perguntas e respostas que vai dar um Renault Clio, computadores, c?meras digitais, videogames e muito mais! www.cade.com.br/antizona _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Tue Sep 23 04:37:45 2003 From: john.hearns at clustervision.com (John Hearns) Date: Tue, 23 Sep 2003 10:37:45 +0200 (CEST) Subject: Redhat Fedora Message-ID: Hopefully not too far off-topic, as a lot of clusters use Redhat. As most of us know, Redhat announced that they are pursuing a new community led development model for future Redhat 'consumer' releases. I'm sure I'm not the only one who was wondering what that meant for the direction of cluster computing. There's more meat on the story now with what looks like a tie-up between Redhat and the Fedora Linux project (Fedora set out to have a high quality update and addon package site). http://fedora.redhat.com And it supports YUM and apt-get, which can only be a good thing. Comments? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Mon Sep 22 15:56:20 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Mon, 22 Sep 2003 21:56:20 +0200 Subject: running mpich on multiple nodes Message-ID: <20030922195027.M59251@ug.ms.wits.ac.za> Hi i have installed mpich on a diskless network, on the server machine. I am unble to execute my programs on 2 nodes. I have enable ssh without password reiurements but i dont know what is wrong.Please help this is the error messge I am getting /usr/local/mpich-1.2.5.2/bin/mpirun -np 2 These are results /usr/local/mpich-1.2.5.2/examples/basic/cpi ug: Connection refused p0_3870: p4_error: Child process exited while making connection to remote process on ug: 0 /usr/local/mpich-1.2.5.2/bin/mpirun: line 1: 3870 Broken pipe /usr/local/mpich-1.2.5.2/examples/basic/cpi -p4pg /usr/local/mpich-1.2.5.2/examples/basic/PI3790 -p4wd /usr/local/mpich-1.2.5.2/examples/basic Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Mon Sep 22 19:53:44 2003 From: raysonlogin at yahoo.com (Rayson Ho) Date: Mon, 22 Sep 2003 16:53:44 -0700 (PDT) Subject: Fluid Dynamics Benchmark - G5 vs G4 vs Itanium2 (was: Re: Virginia Tech PowerMac G5 Cluster Photos) In-Reply-To: <200309221441.h8MEfdn01995@mycroft.ahpcrc.org> Message-ID: <20030922235344.41674.qmail@web11401.mail.yahoo.com> 1.8 GHz G5 vs 1.3 GHz Itanium2: http://www.xlr8yourmac.com/G5/G5_fluid_dynamics_bench/G5_fluid_dynamics_bench.html Rayson --- Richard Walsh wrote: > The "estimated" power consumption on for the chip is 42W at 1.8 GHz > and 1.3v, or 19W at 1.2.GHz and 1.1v according to an IBM PowerPC 970 > overview I have. > > rbw > __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Mon Sep 22 16:04:16 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Mon, 22 Sep 2003 16:04:16 -0400 Subject: Ramdisk Size In-Reply-To: <3F6F4ECC.20703@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> <3F6F4ECC.20703@rri.sari.ac.uk> Message-ID: <3F6F55C0.3010601@bellsouth.net> Tony Travis wrote: > Jeffrey B. Layton wrote: > >> [...] >> What's the difference between tmpfs and a ramdisk created through >> the kernel? Is there a speed difference? > > > The main difference is that tmpfs uses both RAM and swap. The memory > manager will therefore optimise use of RAM for programs, tmpfs, or > disk cache according to demand: Inactive pages of processes and > infrequently accessed parts of tmpfs filesystems are written to disk > to free up RAM when the system starts to run out of physical memory > (RAM). By default, tmpfs size is limited to half the available RAM > detected at boot time. Didn't know this. Thanks! Doesn't this imply that you could get swapped to disk as some point? So if you wanted to make sure everything stayed in memory you would have to use a ram disk. But if you didn't really mind getting some pages swapped to disk (not much a measurable speed hit on the application), then tmpfs is the way to go. Is this accurate? >> BTW, here's a link on creating a generic ram disk: >> >> http://www.lissot.net/partition/ramdisk.html >> >> (read the bottom part). > > > Hey! that only tells you how to format and mount a default 4Mb ramdisk. Just change the count to what you want. Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Mon Sep 22 15:34:36 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Mon, 22 Sep 2003 20:34:36 +0100 Subject: Ramdisk Size In-Reply-To: <3F6F409C.3040101@bellsouth.net> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> Message-ID: <3F6F4ECC.20703@rri.sari.ac.uk> Jeffrey B. Layton wrote: >[...] > What's the difference between tmpfs and a ramdisk created through > the kernel? Is there a speed difference? The main difference is that tmpfs uses both RAM and swap. The memory manager will therefore optimise use of RAM for programs, tmpfs, or disk cache according to demand: Inactive pages of processes and infrequently accessed parts of tmpfs filesystems are written to disk to free up RAM when the system starts to run out of physical memory (RAM). By default, tmpfs size is limited to half the available RAM detected at boot time. > BTW, here's a link on creating a generic ram disk: > > http://www.lissot.net/partition/ramdisk.html > > (read the bottom part). Hey! that only tells you how to format and mount a default 4Mb ramdisk. You don't have to boot with an "initrd=" argument. You could use e.g.: boot: vmlinuz ramdisk_size=10485760 ... mkfs -t ext3 /dev/ram0 mkdir /mnt/ramdisk mount /dev/ram0 /mnt/ramdisk cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk However, I still think you would be better off with tmpfs :-) Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rbw at ahpcrc.org Mon Sep 22 10:41:39 2003 From: rbw at ahpcrc.org (Richard Walsh) Date: Mon, 22 Sep 2003 09:41:39 -0500 Subject: Virginia Tech PowerMac G5 Cluster Photos Message-ID: <200309221441.h8MEfdn01995@mycroft.ahpcrc.org> On Sun Sep 21 00:21:03, Dean Johnson wrote: >On Sat, 2003-09-20 at 19:25, Greg Lindahl wrote: >> On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: >> >> > * 2+ million BTUs of cooling capacity using Liebert's extremedensity >> > cooling (rack mounted cooling via liquid refrigerant) >> > o traditional methods [fans] would have produced windspeeds of >> > 60+ MPH >> >> This is kind of weird, given that their energy density isn't that >> high: 2 cpus in 4U. I've seen machine rooms which have 8 times the >> energy density. >> > >Their energy density is trivial. I can't recall how many watts they are, >but my guess is much less than even old celerons. I wouldn't be >surprised if each of those whole machines sucks less juice than a single >itanic cpu. Wasn't the G4 design spec something like 24 watts. The "estimated" power consumption on for the chip is 42W at 1.8 GHz and 1.3v, or 19W at 1.2.GHz and 1.1v according to an IBM PowerPC 970 overview I have. rbw _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 22 21:48:55 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 23 Sep 2003 11:48:55 +1000 Subject: rsh In-Reply-To: <20030917024414.M10506@ug.ms.wits.ac.za> References: <20030917024414.M10506@ug.ms.wits.ac.za> Message-ID: <200309231148.56402.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > just realised that I cannot communicate with other hosts due to some rsh > setting that I need to do. To be honest I'd really suggest not using RSH and swapping over to OpenSSH instead. In that way you can use the key management stuff to authenticate users making things that little bit safer. One note of caution, make sure you get the latest update for your distribution, or build it from the latest sources. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/b6aHO2KABBYQAh8RAtJUAJ4gE6MjP6bOnqrPefVCQOAMWp51+gCdF0A6 nS5+cuFpALePLGwxIj3booo= =tPsX -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Mon Sep 22 18:32:04 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Mon, 22 Sep 2003 18:32:04 -0400 Subject: Ramdisk Size In-Reply-To: <20030922215254.GA3648@greglaptop.internal.keyresearch.com> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> <20030922215254.GA3648@greglaptop.internal.keyresearch.com> Message-ID: <3F6F7864.2000600@bellsouth.net> > > >> If you are happy with the speed improvement of using a ram disk >>to store the database, you might want to consider squashfs >> >> > >If he's doing things like gene comparisons, the work of uncompressing >the database would be a huge performance loss. > That was what I was wondering. If he's used to speed X running his DB on a disk, then moving to a ram disk may get him speed Y. If using squashfs reduces his speed to Z and Z > X, then it's still a win for him. However, if he gets used to speed Z and can afford the memory space, then skip squashfs. (flowchart anyone?). Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Mon Sep 22 17:52:54 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Mon, 22 Sep 2003 14:52:54 -0700 Subject: Ramdisk Size In-Reply-To: <3F6F409C.3040101@bellsouth.net> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> Message-ID: <20030922215254.GA3648@greglaptop.internal.keyresearch.com> On Mon, Sep 22, 2003 at 02:34:04PM -0400, Jeffrey B. Layton wrote: > Check out Warewulf: warewulf-cluster.org. Very cool cluster > distribution that uses a very small Ramdisk to hold the necessary > parts of the OS. I've just started to use it and it's very slick. That's how Scyld Beowulf works, too. > What's the difference between tmpfs and a ramdisk created through > the kernel? Is there a speed difference? I believe that tmpfs is dynamically sized, and the Linux ramdisk device is not. > If you are happy with the speed improvement of using a ram disk > to store the database, you might want to consider squashfs If he's doing things like gene comparisons, the work of uncompressing the database would be a huge performance loss. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Mon Sep 22 14:12:54 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Mon, 22 Sep 2003 19:12:54 +0100 Subject: Ramdisk Size - correction In-Reply-To: <3F6F32F0.1070400@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> Message-ID: <3F6F3BA6.10504@rri.sari.ac.uk> Tony Travis wrote: > [...] > For example, you could create a 10Mb initrd image of a RAM disk using: > > dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 > mkfs -t ext3 /tftpboot/initrd.img > mkdir /mnt/ramdisk > mount -o loop /tftpboot/initrd.img /mnt/ramdisk > cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk > umount /var/tmp/initrd.img > zcat /tftpboot/initrd.img > /tftpboot/zinitrd.img Hello, Eric. Sorry, I was careless about pathnames - Here's a correction: dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 mkfs -t ext3 /var/tmp/initrd.img mkdir /mnt/ramdisk mount -o loop /var/tmp/initrd.img /mnt/ramdisk cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk umount /var/tmp/initrd.img zcat /var/tmp/initrd.img > /tftpboot/zinitrd.img You don't need the ramdisk image in /var/tmp once you've saved the compressed image: In my case, I'm PXE booting diskless openMosix from /tftpboot which is where the kernel and ramdisk image are stored. Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Mon Sep 22 13:35:44 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Mon, 22 Sep 2003 18:35:44 +0100 Subject: Ramdisk Size In-Reply-To: <3F6E90DB.1060505@mn.rr.com> References: <3F6E90DB.1060505@mn.rr.com> Message-ID: <3F6F32F0.1070400@rri.sari.ac.uk> Eric R Johnson wrote: > I would like to create a large ramdisk to keep a database which will be > searched often, but not changed. > The only thing I currently know how to do to create a ramdisk is: > mke2fs /dev/ram0 > mount /dev/ram0 /tmp/ramdisk0 > > The only problem is that this creates a 4MB ramdisk, and I would like > something on the order of 1GB. Hello, Eric. Are you *sure* you want to do this? RAM disks are usually used for loading modules at boot time. That's why they are small. Typically, the contents of a RAM disk are used as a root filesystem for the kernel to load e.g. SCSI drivers, then discarded when the 'real' root filesystem is mounted from SCSI disks. People also use them to boot a 'minimal' Linux for diagnostics or for some other reason. For example, you could create a 10Mb initrd image of a RAM disk using: dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 mkfs -t ext3 /tftpboot/initrd.img mkdir /mnt/ramdisk mount -o loop /tftpboot/initrd.img /mnt/ramdisk cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk umount /var/tmp/initrd.img zcat /tftpboot/initrd.img > /tftpboot/zinitrd.img Then, override the 4Mb default at boot time: boot: vmlinuz ramdisk_size=10485760 initrd=zinitrd.img However, I think it would be better to use 'tmpfs' for a 1Gb RAM filesystem. The kernel will provide fast access to the most frequently accessed areas of your database held in RAM, but it will move the less frequently accessed areas onto the swap device freeing up RAM for your programs and for other disk caches. Mounting /tmp as a 'tmpfs' filesystem reserves half of your memory for the /tmp filesystem by default. Add an entry like this to /etc/fstab: tmpfs /tmp tmpfs defaults 0 0 Beware that openMosix, in particular, does not work well with tmpfs and automatically comments out tmpfs entries from your /etc/fstab when you install openMosic from the 2.4.21-openmosix1 binary rpm distribution. Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Mon Sep 22 16:49:19 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 22 Sep 2003 16:49:19 -0400 Subject: Lustre In-Reply-To: <3F67151C.A5D41000@epa.gov> References: <200309160953.42528.csamuel@vpac.org> <3F67151C.A5D41000@epa.gov> Message-ID: <1064263759.30109.39.camel@roughneck> On Tue, 2003-09-16 at 09:50, Joseph Mack wrote: > Chris Samuel wrote: > > > Anyone tried out Lustre [1] ? > > There was a talk at OLS this year > > http://www.linuxsymposium.org/2003/ > > As usual when given by a person involved in the development, > it's always presented as being easy to install and running > smoothly. > > However they do have it running on two kilonode production clusters and > have won the contract for the filesystem for some new cluster that is > currently being bid on. So it's in production in some places Just set up a 2 node setup here, it is pretty easy, and it works great as far as I can tell. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmkurtzer at lbl.gov Mon Sep 22 13:43:44 2003 From: gmkurtzer at lbl.gov (Greg Kurtzer) Date: Mon, 22 Sep 2003 10:43:44 -0700 Subject: Ramdisk Size In-Reply-To: <3F6E90DB.1060505@mn.rr.com> References: <3F6E90DB.1060505@mn.rr.com> Message-ID: <20030922174344.GA24330@tux.lbl.gov> Kernel arg: 'ramdisk=n' (ie. ramdisk=1024000 for a 1G ram disk). We typically don't go over several hundred MB's, so let me know how this works. Also, this works with standard kernels (kernel.org and the RH kernel). Greg On Mon, Sep 22, 2003 at 01:04:11AM -0500, Eric R Johnson told me: > I would like to create a large ramdisk to keep a database which will be > searched often, but not changed. > The only thing I currently know how to do to create a ramdisk is: > mke2fs /dev/ram0 > mount /dev/ram0 /tmp/ramdisk0 > > The only problem is that this creates a 4MB ramdisk, and I would like > something on the order of 1GB. > > I tried running menuconfig and changing the default ramdisk size to 1GB, > making the kernel, and rebooting. Now when I run menuconfig, it shows > the ramdisk size as 1GB, but when I execute the mke2fs /dev/ram0, it > still only creates a 4MB ramdisk. > > Can someone point me in the right direction? > > Thanks, > Eric -- Greg M. Kurtzer, CSE: Linux cluster specialist Lawrence Berkeley National Laboratory Contact: O=510.495.2307, P=510.448.4540, M=510.928.9953 1 Cyclotron Road MS:50C-3396, Berkeley, CA 94720 http://www.lbl.gov, http://scs.lbl.gov/, http://lug.lbl.gov/ Email: GMKurtzer_at_lbl.gov, Text: 5109289953_at_mobileatt.net _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Tue Sep 23 13:23:23 2003 From: josip at lanl.gov (Josip Loncaric) Date: Tue, 23 Sep 2003 11:23:23 -0600 Subject: Fluid Dynamics Benchmark - G5 vs G4 vs Itanium2 In-Reply-To: <20030922235344.41674.qmail@web11401.mail.yahoo.com> References: <20030922235344.41674.qmail@web11401.mail.yahoo.com> Message-ID: <3F70818B.4030601@lanl.gov> Rayson Ho wrote: > 1.8 GHz G5 vs 1.3 GHz Itanium2: > > http://www.xlr8yourmac.com/G5/G5_fluid_dynamics_bench/G5_fluid_dynamics_bench.html [ Summary: G5 wins the large memory case, loses small/medium cases ] Has anyone made a STREAM benchmark comparison for these systems? This CFD benchmark is probably memory bandwidth limited... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csarat1 at uic.edu Tue Sep 23 16:28:20 2003 From: csarat1 at uic.edu (Sarat C Maruvada) Date: Tue, 23 Sep 2003 15:28:20 -0500 (CDT) Subject: A question of OS!! Message-ID: Hi,I am planning on installing a 32 node beowulf cluster for high performance bio-informatics applications. As a OS I was planning on installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux has advantage in parallel computations. Having no hands-on experience in SuSe Linux, I have no idea how to verify this or do a comparision. What do you advice as the OS for the cluster? And how to determine this,as there are a bunch of unix OSes out there.Thanks. Sincerely, Sarat. ************************************************************** Sarat Chandra Maruvada 830,S.Claremont Ave., Research Assistant, CS Dept., #2, Prof. Florin Balasa, Chicago, SEO 937, IL - 60612. UIC. Ph: 312-850-9711. ************************************************************** _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wrankin at ee.duke.edu Tue Sep 23 17:01:12 2003 From: wrankin at ee.duke.edu (Bill Rankin) Date: 23 Sep 2003 17:01:12 -0400 Subject: [Fwd: HPC New Positions at UNC] Message-ID: <1064350872.2225.2.camel@rohgun.cse.duke.edu> FYI, I didn't know if this had made the list yet (I'm on digest mode) but UNC Chapel Hill is looking for cluster programmers and instructors. -bill -----Forwarded Message----- From: C. D. Poon To: hpcplanning Subject: HPC New Positions at UNC Date: 23 Sep 2003 16:45:21 -0400 We are recruiting for two new positions to further enhance support for research computing at the University of North Carolina at Chapel Hill. The focus of one position is code optimization and parallelization. The primary responsibility of the second position is to develop and teach training courses on specific computational applications, parallel code development and testing, checkpointing, and other aspects of our computing environment. The job announcements are currently posted at the UNC-CH Human Resources web site: http://www.ais.unc.edu/hr/jobs/epajobs.htm Please share this information with your colleagues. C. D. Poon High-Performance Computing Group Academic Technology & Networks University of North Carolina -------------------------------------------------------------------- Research Computing Associate for High Performance Computing EPA Non-Faculty September 17, 2003 Position Description: Research Computing Associate for High Performance Computing will provide expertise for current and future high performance computing code optimization, parallelization and porting. This position will work closely with faculty and other University researchers to determine the appropriate architecture for each application and will then assist with porting and tuning applications for optimal performance on the chosen platform. In addition to supporting the porting and optimization of researchers codes, this position will help identify and analyze codes to be used for benchmarking and research of high performance computing resources. Education Required: Position Requires a M.S. or Ph.D in computer science or related computational science discipline and demonstrated hands-on expertise and high performance programming experience with MPICH, OpenMP, Fortran and C++. Experience Required: Experience in performance evaluation of both distributed and shared parallel memory architectures is highly desired as is experience with parallel scientific simulation codes and writing applications for parallel computers. Excellent interpersonal skills, excellent oral and written communications skills and demonstrated organizational skills are required, along with strong personal motivation and a proven ability to work in a team environment. Application Deadline: October 29, 2003. Applicants should send a cover letter, resume and three references to: Diane Strong, Human Resources Mgr. Academic Technology & Networks (ATN) UNC-CH, Campus Box # 3420 Chapel Hill, NC 27599-3420 The University of North Carolina at Chapel Hill is an equal opportunity employer. -------------------------------------------------------------------- High Performance Computing Instructor EPA Non-Faculty September 17, 2003 Position Description: The High Performance Computing Instructor will determine course content, develop and teach introductory classes about the UNC-Chapel Hill high performance computing environment and how to use it, parallel programming, scientific applications, and specialized coding and performance analysis for faculty and their research teams. Within ATN, the Academic Computing Systems Division provides the infrastructure and support for the research computing, data management and storage, messaging, web and enterprise backup and recovery needs of the University community. The High Performance Computing Instructor will develop course materials for computational science techniques. These materials will be used to design on-line tutorials as well as hands-on workshops and courses for researchers with computationally intensive problems. Education Required: Position Requires a M.S. or Ph.D in computer science or related computational science discipline and demonstrated hands-on expertise and high performance programming experience with MPICH, OpenMP, Fortran and C++. Experience Required: Experience with parallel scientific codes and writing applications for parallel computers is required. Demonstrated teaching skills are highly desired. Excellent interpersonal skills, excellent oral and written communications skills and demonstrated organizational skills are required, along with strong personal motivation and proven ability to work in a team environment. Application Deadline: October 29, 2003. Applicants should send a cover letter, resume and three references to: Diane Strong, Human Resources Mgr. Academic Technology & Networks (ATN) UNC-CH, Campus Box # 3420 Chapel Hill, NC 27599-3420 The University of North Carolina at Chapel Hill is an equal opportunity employer. -------------------------------------------------------------------- -- bill rankin, ph.d. ........ director, cluster and grid technology group wrankin at ee.duke.edu .......................... center for computational duke university ...................... science engineering and medicine http://www.ee.duke.edu/~wrankin .............. http://www.csem.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jschauma at netmeister.org Mon Sep 22 22:40:26 2003 From: jschauma at netmeister.org (Jan Schaumann) Date: Mon, 22 Sep 2003 22:40:26 -0400 Subject: rsh In-Reply-To: <200309231148.56402.csamuel@vpac.org> References: <20030917024414.M10506@ug.ms.wits.ac.za> <200309231148.56402.csamuel@vpac.org> Message-ID: <20030923024026.GE4479@netmeister.org> Chris Samuel wrote: > On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > > > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > > just realised that I cannot communicate with other hosts due to some rsh > > setting that I need to do. > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > instead. Why? Most likely, ssh will introduce a non-negligible overhead in encryption and/or compression. To the OP: check your /etc/hosts.equiv and ~/.rhosts, as suggested by rsh(1) and rlogin(1). -Jan -- "I am so amazingly cool you could keep a side of meat in me for a month. I am so hip I have difficulty seeing over my pelvis." -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From glen at cert.ucr.edu Tue Sep 23 18:24:47 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Tue, 23 Sep 2003 15:24:47 -0700 Subject: A question of OS!! In-Reply-To: References: Message-ID: <3F70C82F.2070307@cert.ucr.edu> Sarat C Maruvada wrote: >Hi,I am planning on installing a 32 node beowulf cluster for high >performance bio-informatics applications. As a OS I was planning on >installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux >has advantage in parallel computations. Having no hands-on experience in >SuSe Linux, I have no idea how to verify this or do a comparision. What do >you advice as the OS for the cluster? And how to determine this,as there >are a bunch of unix OSes out there.Thanks. > One advantage I can think of that Suse might have is it's automated installation utility, autoyast I believe it's called. While it's probably a little harder to set up, I think it would be a bit better than kickstart for installing Linux on a large number of machines. Actually though, some of the functionality of autoyast seemed to be broken to me, but again it's a lot more complicated than kickstart, so maybe it was just me. You don't have to pay to use Suse's automatic update tool, but I think that this may change on Red Hat now that they've merged with Fedora. The only other thing I can think of is that the Suse distribution seems to come with about twice as many packages to choose from. Other than that, Linux is Linux, and you're not going to see much of a speed difference between either Suse or Red Hat. My two cents, Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Tue Sep 23 14:12:39 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Tue, 23 Sep 2003 19:12:39 +0100 Subject: Distributing makes across the cluster Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE164@stegosaurus.bristol.quadrics.com> I have made good success with just using the standard 'make -j' but replacing where you have something like 'CC=gcc' with 'CC="prun -n1 gcc"' in the makefile. (change above for your favourite compiler and/or method of launching a binary onto your cluster). As ever the caveats of doing software builds in parallel is that date stamps are always suspect and 'make' can sometimes take the wrong route :-( Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- > -----Original Message----- > From: Lars Henriksen [mailto:lars at meshtechnologies.com] > Sent: 21 September 2003 18:23 > To: Stuart J Adams > Cc: beowulf at beowulf.org > Subject: Re: Distributing makes across the cluster > > > On Sat, 2003-09-20 at 21:34, Stuart J Adams wrote: > > Is there a way to automatically distribute makes across > > a cluster ?? (Similar to what gnu make -j does on an SMP machine) > > I have used distcc with success. Kernel compile times drop to > 40 sec :-) > > http://distcc.samba.org/ > > Hope you can use it. > > best regards > > Lars > -- > Lars Henriksen | MESH-Technologies ApS > Systems Manager & Consultant | Forskerparken 10 > www.meshtechnologies.com | DK-5260 Odense M, Denmark > lars at meshtechnologies.com | mobile: +45 2291 2904 > direct: +45 6315 7310 | fax: +45 6315 7314 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 22 03:52:43 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 22 Sep 2003 09:52:43 +0200 Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> References: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: <1064217163.15240.6.camel@revolution.mandrakesoft.com> Ryan Burns said: > Hello, > I've been running a very small 4 node beowulf cluster for a project. > Right now I'm running Debian on the cluster and I'm not very happy with > it or the tools available through the distribution. > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? MandrakeClustering is the latest clustering distribution made by the CLIC team. It supports Pentium & the newest AMD64 architecture (Opteron & Athlon64). > I am also looking for tools to automate software installations across > all the nodes, etc. That one of our stronger point, the ka technologies from IDIMAG allow us to deploy new nodes within 3 minutes (duplication, reoboot, autoconfig). > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. This his our 2nd stronger point. For the mananging side, using the urpmi parallel command you can manage (install/remove) rpm packages with their dependencies. This tools helps you to keep your nodes up to date. Urpmi parallel uses the ka technologies so the time needed for installing 50MB on a cluster doesn't depend on the cluster size ! It takes the same time for installing on 2 or 50 nodes ! So just have an switched network (very common this days). Urpmi could be compared with apt except for the parallel side ! The autoconfiguration tool, allow people to build up a full cluster from out-of-the-box computers in a really few time (when hardware is set, of course :D). The cluster server builds everything for you just by answering few questions. It builds up automaticaly : DNS, DHCP, PXE, NIS, NFS, MPICH, LAM/MPI, TFTP, NTP,PXE, GANGLIA ... services for you. The last strong point about MDKC is this is a real linux distributions made for the cluster. This is not just a MandrakeLinux + few packages, this is a real cluster distro. Everything from installation to configuration has been thought for clustering. Everything has been build on the same base so the product is really stable, and you install everything at the same time (linux + clustering apps + configuration + tips for making the life easier.) Usual sysadmin clustering tools are included in MDKC : ka tools, gexec, pcp, mput, rshp, dsh, etc.. There is multiple ways for doing the things... If a very common tool is forgot, tell us we'll do our best to integrate it on the next release. Linuxely yours, -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Tue Sep 23 19:10:30 2003 From: csamuel at vpac.org (Chris Samuel) Date: Wed, 24 Sep 2003 09:10:30 +1000 Subject: rsh In-Reply-To: <20030923024026.GE4479@netmeister.org> References: <20030917024414.M10506@ug.ms.wits.ac.za> <200309231148.56402.csamuel@vpac.org> <20030923024026.GE4479@netmeister.org> Message-ID: <200309240910.41797.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 23 Sep 2003 12:40 pm, Jan Schaumann wrote: > Why? IMHO rsh/rlogin are inherently insecure (vulnerable to spoofing, password sniffing). We only permit interactive access to our cluster via SSH. > Most likely, ssh will introduce a non-negligible overhead in encryption > and/or compression. Certainly in our environment (Linux, PBS, MAUI, MPICH/GM) the major use of ssh internal to the cluster is as a replacement for rsh for starting MPI jobs on the various nodes. The impact of any overhead is minor given that the data will be shuffled around over Myrinet without using IP. Compared with the overhead of submitting a job into PBS and having it scheduled by MAUI, any extra latency due to SSH disappears into the noise. :-) Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/cNLuO2KABBYQAh8RAt7XAJ9l/ZKn3KTjG88D2bF494NdkXt6jQCfXrun JjnQOsLqOHvfV802lppF0I0= =qfnI -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bioinformaticist at mn.rr.com Tue Sep 23 15:49:20 2003 From: bioinformaticist at mn.rr.com (Eric R Johnson) Date: Tue, 23 Sep 2003 14:49:20 -0500 Subject: Ramdisk Size In-Reply-To: <3F6F32F0.1070400@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> Message-ID: <3F70A3C0.70601@mn.rr.com> Tony, Thanks for the advice. Let me add a little more information and get your advice about the ramdisk/tmpfs question. Without going into a lot of boring details, the goal of this project is to create a cluster of workstations whose sole purpose is to constantly search the entire contents of a single database over and over again in as short a period of time as possible. That would be it's sole purpose for existing. The database we will be searching is small enough to fit into RAM. As it is a growing database, it may break the 4 Gig barrier, but that would simply require switching to something like an Opteron (or other 64 bit) based cluster. Also, the database only changes every month, or so, and can be downloaded as a whole new database. So, in summary, this is basically a static database, which is searched, it it's entirety, every time, and must be completed in as short a period of time as possible on a system dedicated to this task. Now that you have a little more info, do you still recommend tmpfs? If so, is there a specific performance issue that leads us in that direction, because the search time is all that matters in this case. Thanks again for the advice, Eric Tony Travis wrote: > Eric R Johnson wrote: > >> I would like to create a large ramdisk to keep a database which will >> be searched often, but not changed. >> The only thing I currently know how to do to create a ramdisk is: >> mke2fs /dev/ram0 >> mount /dev/ram0 /tmp/ramdisk0 >> >> The only problem is that this creates a 4MB ramdisk, and I would like >> something on the order of 1GB. > > > Hello, Eric. > > Are you *sure* you want to do this? > > RAM disks are usually used for loading modules at boot time. That's > why they are small. Typically, the contents of a RAM disk are used as > a root filesystem for the kernel to load e.g. SCSI drivers, then > discarded when the 'real' root filesystem is mounted from SCSI disks. > People also use them to boot a 'minimal' Linux for diagnostics or for > some other reason. > > For example, you could create a 10Mb initrd image of a RAM disk using: > > dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 > mkfs -t ext3 /tftpboot/initrd.img > mkdir /mnt/ramdisk > mount -o loop /tftpboot/initrd.img /mnt/ramdisk > cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk > umount /var/tmp/initrd.img > zcat /tftpboot/initrd.img > /tftpboot/zinitrd.img > > Then, override the 4Mb default at boot time: > > boot: vmlinuz ramdisk_size=10485760 initrd=zinitrd.img > > However, I think it would be better to use 'tmpfs' for a 1Gb RAM > filesystem. The kernel will provide fast access to the most frequently > accessed areas of your database held in RAM, but it will move the less > frequently accessed areas onto the swap device freeing up RAM for your > programs and for other disk caches. > > Mounting /tmp as a 'tmpfs' filesystem reserves half of your memory for > the /tmp filesystem by default. Add an entry like this to /etc/fstab: > > tmpfs /tmp tmpfs defaults 0 0 > > Beware that openMosix, in particular, does not work well with tmpfs > and automatically comments out tmpfs entries from your /etc/fstab when > you install openMosic from the 2.4.21-openmosix1 binary rpm distribution. > > Tony. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 23 20:32:03 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 23 Sep 2003 20:32:03 -0400 (EDT) Subject: rsh In-Reply-To: <20030923024026.GE4479@netmeister.org> Message-ID: On Mon, 22 Sep 2003, Jan Schaumann wrote: > Chris Samuel wrote: > > > On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > > > > > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > > > just realised that I cannot communicate with other hosts due to some rsh > > > setting that I need to do. > > > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > > instead. > > Why? Most likely, ssh will introduce a non-negligible overhead in > encryption and/or compression. I'd actually say that it will introduce a negligible overhead in both those dimensions for most usage. As in a few seconds (extra) to launch quite a few tasks. Periodically I measure and post the numbers but can never remember what they are the next time this issue rolls around. However, they aren't horribly large in absolute terms, let alone relative ones, and are likely to be cumulatively signficant only on very large clusters for most tasks. With that said, for some usage patterns (spawning lots of very short tasks) they might impact signficantly even on small clusters (yes, the mandatory YMMV:-) , but for the more common spawning of lots of tasks intended to run >>a signficant amount of time<< (e.g. a day), the fraction of a second per task marginal cost associated with startup via ssh vs rsh is indeed negligible. Even tasks where this isn't the case can usually be engineered so that it is (and should be). If ssh isn't adequately efficient, chances are good that rsh is a (smaller but still significant) bottleneck too, since the latency differential is only about a factor of two or three IIRC, and should be scaled away by altering your task organization so that the parallel task runs a "long" time compared to startup time for any remote shell. The issue can also be avoided (as Josip notes) by using LAM or PVM, which spawn a daemon via ssh but subsequently start tasks without any shell-level overhead at all. Overall, given my sysadmin background, I tend to really dislike the idea of keeping rsh around at all and expect that one day it will be deprecated and just go away (in many environments this is true already). It is obsolete, insecure, and fairly stupid when it comes to managing the shell environment. ssh has a lot of advantages compared to rsh aside from its security. I personally wish they had left in the option to turn off encryption (AS an option) which one used to be able to do just to reduce the overhead still further on clusters while retaining its other advantages and features, but not unreasonably the ssh people decided not to so that foolish users couldn't turn it off by accident or design. In many environments, the time wasted by ONE successful password crack due to snooping is far, far greater than any number of rsh margins. rgb > > To the OP: check your /etc/hosts.equiv and ~/.rhosts, as suggested by > rsh(1) and rlogin(1). > > -Jan > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xyzzy at speakeasy.org Tue Sep 23 22:03:52 2003 From: xyzzy at speakeasy.org (Trent Piepho) Date: Tue, 23 Sep 2003 19:03:52 -0700 (PDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Robert G. Brown wrote: > quite a few tasks. Periodically I measure and post the numbers but can > never remember what they are the next time this issue rolls around. time for i in `seq 1 10` ; do rsh venturi4 echo ; done real 0m0.320s user 0m0.000s sys 0m0.050s time for i in `seq 1 10` ; do ssh venturi4 echo ; done real 0m4.014s user 0m0.820s sys 0m0.030s About one order of magnitude slower, and a great deal more cpu time. But that's not the problem I have with ssh vs rsh, it's copying large files. Just last week I needed to copy 65GB of data from an ext3 formatted IDE harddrive someone brought in. The only linux system that I could bring down to stick the drive in was my VIA-C3 based thin-client. Using scp to copy the files would have taken all day. If I could just turn off the stream encryption when I don't need it, everything would be fine. Too bad the security-nazis removed that option. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 23 22:41:44 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 23 Sep 2003 22:41:44 -0400 (EDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Trent Piepho wrote: > On Tue, 23 Sep 2003, Robert G. Brown wrote: > > quite a few tasks. Periodically I measure and post the numbers but can > > never remember what they are the next time this issue rolls around. > > time for i in `seq 1 10` ; do rsh venturi4 echo ; done > real 0m0.320s > user 0m0.000s > sys 0m0.050s > > time for i in `seq 1 10` ; do ssh venturi4 echo ; done > real 0m4.014s > user 0m0.820s > sys 0m0.030s > > About one order of magnitude slower, and a great deal more cpu time. But still pretty much negligible, on a job intended to run all day. > But that's not the problem I have with ssh vs rsh, it's copying large files. > > Just last week I needed to copy 65GB of data from an ext3 formatted IDE > harddrive someone brought in. The only linux system that I could bring down > to stick the drive in was my VIA-C3 based thin-client. Using scp to copy the > files would have taken all day. If I could just turn off the stream > encryption when I don't need it, everything would be fine. Too bad the > security-nazis removed that option. That's fair enough. Once upon a time you could, and I agree that the option should still exist. I once upon a time looked briefly at the ssh source to see if it was easy to short-circuit, but the sources are fairly involved and complicated and I couldn't see any easy place to insert a null function call (without spending a lot more time than it would save). After all, a dummy user probably won't know enough to read the man page anyway and select no encryption (especially if you make it an argument that HAS to be entered as a command line option). Anybody smart enough to read it ought to be smart enough to figure out when to use it. And they can even leave in all the encrypted handshaking and authentication required to establish the connection (which is actually a big chunk of the time you measure IIRC -- the encryption is relatively fast) and then just transfer the data quickly. I suppose in the case of the 65 GB of data you could just NFS export, mount, and copy (depending on just how thin the client was). NFS isn't lightning fast, but one can move data through it at a respectable rate (less than all day). And yeah, enabling rsh/rcp long enough to facilitate the transfer might be easier/faster. rgb > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xyzzy at speakeasy.org Wed Sep 24 00:23:50 2003 From: xyzzy at speakeasy.org (Trent Piepho) Date: Tue, 23 Sep 2003 21:23:50 -0700 (PDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Robert G. Brown wrote: > > > > About one order of magnitude slower, and a great deal more cpu time. > > But still pretty much negligible, on a job intended to run all day. True, but what about interactive tasks? Say I want to run uptime or netstat or lilo, or some other simple command on all the nodes of a 100 node cluster. Doing it via rsh would take 3 seconds, via ssh 40. That's a big difference when you are waiting for the results. > I suppose in the case of the 65 GB of data you could just NFS export, > mount, and copy (depending on just how thin the client was). NFS isn't > lightning fast, but one can move data through it at a respectable rate Exactly what I did. My thin client has a hard drive with a RH9 install, since I used it to develop the system used on the real thin clients, that only have a 32MB flash module. > (less than all day). And yeah, enabling rsh/rcp long enough to > facilitate the transfer might be easier/faster. I tried that first, but rcp barfed on the files larger than 2GB! _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jschauma at netmeister.org Tue Sep 23 22:26:16 2003 From: jschauma at netmeister.org (Jan Schaumann) Date: Tue, 23 Sep 2003 22:26:16 -0400 Subject: rsh In-Reply-To: References: <20030923024026.GE4479@netmeister.org> Message-ID: <20030924022616.GB15042@netmeister.org> "Robert G. Brown" wrote: > On Mon, 22 Sep 2003, Jan Schaumann wrote: > > > Chris Samuel wrote: > > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > > > instead. > > > > Why? Most likely, ssh will introduce a non-negligible overhead in > > encryption and/or compression. > for the more common spawning of lots of tasks intended to run >>a > signficant amount of time<< (e.g. a day), the fraction of a second per > task marginal cost associated with startup via ssh vs rsh is indeed > negligible. Fair argument. > Overall, given my sysadmin background, I tend to really dislike the idea > of keeping rsh around at all and expect that one day it will be > deprecated and just go away (in many environments this is true already). See, and given my sysadmin background, I tend to really disklike the idea of obsoleting a perfectly suitable tool just b/c under certain (different) circumstances it is not suitable. :-) -Jan -- "Ford," he said, "you're turning into a penguin. Stop it." -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From john.hearns at clustervision.com Wed Sep 24 02:45:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Wed, 24 Sep 2003 08:45:11 +0200 (CEST) Subject: A question of OS!! In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on I think this is an important question we should discuss - in the light of the Fedora project with Redhat. I've used both SuSE and Redhat, and like both. In fact, we use SuSE on Opterons, as SuSE were there first with a complete distro, which is very good. I fail to see though why SuSE would have an advantage in parallel computations. Maybe your colleague was trying distros with wildly different kernel versions? In fact, this brings up my pet peeve. As someone else has said in this thread - Linux is Linux. IMHO, we should refer to Linux distributions. The OS itself is Linux (or GNU/Linux). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Andrew.Cannon at nnc.co.uk Wed Sep 24 04:58:10 2003 From: Andrew.Cannon at nnc.co.uk (Cannon, Andrew) Date: Wed, 24 Sep 2003 09:58:10 +0100 Subject: A question of OS!! Message-ID: So which of the mainstream distros are best for Beowulfs? RH, SuSe, Mandrake etc? We use RH8, but I am unsure about whether we should still use this or go to another distro. Advice would be useful. Andrew -----Original Message----- From: John Hearns [mailto:john.hearns at clustervision.com] Sent: Wednesday, September 24, 2003 7:45 AM To: Sarat C Maruvada Cc: beowulf at beowulf.org Subject: Re: A question of OS!! On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on I think this is an important question we should discuss - in the light of the Fedora project with Redhat. I've used both SuSE and Redhat, and like both. In fact, we use SuSE on Opterons, as SuSE were there first with a complete distro, which is very good. I fail to see though why SuSE would have an advantage in parallel computations. Maybe your colleague was trying distros with wildly different kernel versions? In fact, this brings up my pet peeve. As someone else has said in this thread - Linux is Linux. IMHO, we should refer to Linux distributions. The OS itself is Linux (or GNU/Linux). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf NNC's UK Operating Companies : NNC Holdings Limited (no. 3725076), NNC Limited (no. 1120437), National Nuclear Corporation Limited (no. 2290928), STATS-NNC Limited (no. 4339062) and Technica-NNC Limited (no. 235856). The registered office of each company is at Booths Hall, Chelford Road, Knutsford, Cheshire WA16 8QZ except for Technica-NNC Limited whose registered office is at 6 Union Row, Aberdeen AB10 1DQ. This email and any files transmitted with it have been sent to you by the relevant UK operating company and are confidential and intended for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the NNC system manager by e-mail at eadm at nnc.co.uk. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Pfenniger at obs.unige.ch Wed Sep 24 04:53:54 2003 From: Daniel.Pfenniger at obs.unige.ch (Daniel.Pfenniger at obs.unige.ch) Date: Wed, 24 Sep 2003 10:53:54 +0200 Subject: rsh In-Reply-To: <200309240910.41797.csamuel@vpac.org> References: <20030917024414.M10506@ug.ms.wits.ac.za> <200309231148.56402.csamuel@vpac.org> <20030923024026.GE4479@netmeister.org> <200309240910.41797.csamuel@vpac.org> Message-ID: <1064393634.3f715ba26726d@webmail.obs.unige.ch> Selon Chris Samuel : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tue, 23 Sep 2003 12:40 pm, Jan Schaumann wrote: > > > Why? > > IMHO rsh/rlogin are inherently insecure (vulnerable to spoofing, password > sniffing). We only permit interactive access to our cluster via SSH. > What we do is (with xinetd) to block rsh from acces outside the cluster and permit rsh only within the cluster. Of course the cluster users must be trusted. ssh does introduce less interactivity for global interactive commands. > > Most likely, ssh will introduce a non-negligible overhead in encryption > > and/or compression. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lange at informatik.Uni-Koeln.DE Wed Sep 24 06:20:27 2003 From: lange at informatik.Uni-Koeln.DE (Thomas Lange) Date: Wed, 24 Sep 2003 12:20:27 +0200 Subject: Cluster distributions? In-Reply-To: <1064217163.15240.6.camel@revolution.mandrakesoft.com> References: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> <1064217163.15240.6.camel@revolution.mandrakesoft.com> Message-ID: <16241.28651.375146.363915@informatik.uni-koeln.de> > Ryan Burns said: >> Hello, I've been running a very small 4 node beowulf cluster >> for a project. Right now I'm running Debian on the cluster and >> I'm not very happy with it or the tools available through the >> distribution. Does anyone have any suggestions of a good Did you looked at FAI, the Fully Automatic Installation for Debian. I set up several Beowulf cluster using this tool. On the FAI web page, there are links to other cluster installations that uses FAI with great success. http://www.informatik.uni-koeln.de/fai/ -- Gruss Thomas _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Wed Sep 24 06:49:46 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed, 24 Sep 2003 11:49:46 +0100 Subject: Ramdisk Size In-Reply-To: <3F70A3C0.70601@mn.rr.com> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F70A3C0.70601@mn.rr.com> Message-ID: <3F7176CA.6070005@rri.sari.ac.uk> Eric R Johnson wrote: > [...] > as possible on a system dedicated to this task. Now that you have a > little more info, do you still recommend tmpfs? If so, is there a > specific performance issue that leads us in that direction, because the > search time is all that matters in this case. Hello, Eric. On the basis of what you've told us, I think you would be better off avoiding the filesystem overhead altogether and read your database directly into memory within a program. Accessing RAM directly by memory references in a program is as fast as you can get. Anything else you do will incur an OS filesystem call overhead. This may not be practical, of course, but it would give you the fastest solution. One-step (hash) lookups of whatever your database contains directly from RAM is fastest. My basic point about ramdisk/tmpfs is that I think you would be better off allowing the memory manager to keep your most recently accessed data in memory for you. However, if your entire job, including the database, fits into physical memory (RAM) then no swapping to disk will occur with tmpfs. I think you really should be looking for a solution that gives you the best performance improvement for the least effort - i.e. tmpfs. Forgive me if I'm teaching you to suck eggs, but if search time is all that matters then your search algorithm is important. There is little point accessing your database as fast as possible using a poor search algorithm: Indeed, you may find that your concern about 'fast' access to your data on ramdisk/tmpfs is not the main bottleneck after all. You could benchmark the database problem on harddisk and tmpfs to find out. Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 24 07:08:43 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 24 Sep 2003 07:08:43 -0400 (EDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Trent Piepho wrote: > On Tue, 23 Sep 2003, Robert G. Brown wrote: > > > > > > About one order of magnitude slower, and a great deal more cpu time. > > > > But still pretty much negligible, on a job intended to run all day. > > True, but what about interactive tasks? Say I want to run uptime or netstat > or lilo, or some other simple command on all the nodes of a 100 node cluster. > Doing it via rsh would take 3 seconds, via ssh 40. That's a big difference > when you are waiting for the results. Ah, but that's why I wrote xmlsysd and wulfstat (or more practically, why Erik Hendriks et al wrote bproc;-) Interactive task distribution and monitoring is NEVER very efficient or lightweight when it runs through an instantiated shell. And sure, as I said, if you use ssh to distribute lots of small tasks and want immediate results it isn't very good. Mostly because of the handshaking, I think -- I doubt that this is an encryption issue. Here is where one has to do the cost-benefit tradeoffs. Duke more or less prohibits the use of rsh on systems exposed to any sort of public network because the cost of ANY sort of cracking incident is very high, and very real. On a firewalled cluster of course one could install it, but I've seen cracking and data theft incidents at Duke WITHIN RESEARCH GROUPS (one e.g. postdoc snooping wire data and root password, stealing files, apparently exporting said files to friends back in Bulgaria in his particular case to search for value). One could argue that if it is for use on YOUR cluster, architected inside a protected network as a "true beowulf" and with only one user, you can do what you like (and who could argue:-) -- from what Erik has told me in years past, bproc isn't secure on the inside either, but the presumption there is that "nodes" have nothing of value but immediate data and the only point of access is the head node (these are guys who think rsh is -- correctly -- heaviweight:-). In any sort of environment with trust boundaries or valuable data, though, or where one might wish to export environment variables along with the shell processes, using rsh enables those boundaries to be fairly trivially violated by anyone with root/promiscuous mode access to the wire itself (via e.g. a handy laptop). ssh in that sort of environment is just a form of due diligence. sshd, too, is subjected to intense scrutiny looking for buffer overwrites and exploits, while rshd is not under the presumption perhaps that it is OBVIOUSLY not secure so why bother? At Duke we have made the decision to use ssh only, because of the security issues, and use tools like xmlsysd to do the few tasks that we might otherwise like to distribute in real time with a remote shell. At other institutions one might well choose otherwise. YMMV, and we live in a world of marvelous choices. > > (less than all day). And yeah, enabling rsh/rcp long enough to > > facilitate the transfer might be easier/faster. > > I tried that first, but rcp barfed on the files larger than 2GB! Hmm, sounds like an int counter in there somewhere...;-) Seriously, one thing that might be worth TRYING is to request that the openssh people restore a "none" encryption option on ssh/sshd, enabled by /etc/ssh/*.config options so that local admins can disable it on open networks. I can't see why this would be objectionable to them, and it would solve at least this half of the problem. rgb > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From timm at fnal.gov Wed Sep 24 07:18:56 2003 From: timm at fnal.gov (Steven Timm) Date: Wed, 24 Sep 2003 06:18:56 -0500 Subject: rsh In-Reply-To: Message-ID: Nobody has yet mentioned in this thread that there is an option to run a kerberos-enabled rsh and do kerberos authentication, and even DES encryption of your session traffic and data stream. Obviously it's a bit slower than ssh but it is quite secure and has solved our problems quite well. Steve Timm ------------------------------------------------------------------ Steven C. Timm (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division/Core Support Services Dept. Assistant Group Leader, Scientific Computing Support Group Lead of Computing Farms Team On Tue, 23 Sep 2003, Robert G. Brown wrote: > On Mon, 22 Sep 2003, Jan Schaumann wrote: > > > Chris Samuel wrote: > > > > > On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > > > > > > > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > > > > just realised that I cannot communicate with other hosts due to some rsh > > > > setting that I need to do. > > > > > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > > > instead. > > > > Why? Most likely, ssh will introduce a non-negligible overhead in > > encryption and/or compression. > > I'd actually say that it will introduce a negligible overhead in both > those dimensions for most usage. As in a few seconds (extra) to launch > quite a few tasks. Periodically I measure and post the numbers but can > never remember what they are the next time this issue rolls around. > However, they aren't horribly large in absolute terms, let alone > relative ones, and are likely to be cumulatively signficant only on very > large clusters for most tasks. > > With that said, for some usage patterns (spawning lots of very short > tasks) they might impact signficantly even on small clusters (yes, the > mandatory YMMV:-) , but for the more common spawning of lots of tasks > intended to run >>a signficant amount of time<< (e.g. a day), the > fraction of a second per task marginal cost associated with startup via > ssh vs rsh is indeed negligible. Even tasks where this isn't the case > can usually be engineered so that it is (and should be). If ssh isn't > adequately efficient, chances are good that rsh is a (smaller but still > significant) bottleneck too, since the latency differential is only > about a factor of two or three IIRC, and should be scaled away by > altering your task organization so that the parallel task runs a "long" > time compared to startup time for any remote shell. > > The issue can also be avoided (as Josip notes) by using LAM or PVM, > which spawn a daemon via ssh but subsequently start tasks without any > shell-level overhead at all. > > Overall, given my sysadmin background, I tend to really dislike the idea > of keeping rsh around at all and expect that one day it will be > deprecated and just go away (in many environments this is true already). > It is obsolete, insecure, and fairly stupid when it comes to managing > the shell environment. ssh has a lot of advantages compared to rsh > aside from its security. I personally wish they had left in the option > to turn off encryption (AS an option) which one used to be able to do > just to reduce the overhead still further on clusters while retaining > its other advantages and features, but not unreasonably the ssh people > decided not to so that foolish users couldn't turn it off by accident or > design. > > In many environments, the time wasted by ONE successful password crack > due to snooping is far, far greater than any number of rsh margins. > > rgb > > > > > To the OP: check your /etc/hosts.equiv and ~/.rhosts, as suggested by > > rsh(1) and rlogin(1). > > > > -Jan > > > > > > -- > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From r.young at irl.cri.nz Wed Sep 24 08:21:39 2003 From: r.young at irl.cri.nz (Roger Young) Date: Thu, 25 Sep 2003 00:21:39 +1200 Subject: Question about mpich libraries Message-ID: <20030925002139.2a283dc9.r.young@irl.cri.nz> I have recently installed mpich v.1.2.5.2 under Slackware Linux 9.0 with Linux kernel 2.4.22 and glibc 2.3.1 and compilers gcc 3.2.2 & lf95 v.5.5. My mpich config options were ./configure -without-romio -disable-cc++ -fc=lf95 -f90=lf95 --prefix=/usr/local/mpich However the application (geofem) will not compile because it cannot find lmpichf90nc which is meant to be in mpich/lib (but is not there). The error message is "usr/bin/ld: cannot find -lmpichf90rc" Can somebody tell me how I might generate/obtain this library? Thanks for your help, Roger Young Industrial Research Ltd PO Box 31-310 Lowe Hutt New Zealand r.young at irl.cri.nz -- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Wed Sep 24 09:24:50 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Wed, 24 Sep 2003 09:24:50 -0400 Subject: A question of OS!! In-Reply-To: References: Message-ID: <1064409890.29084.7.camel@protein.scalableinformatics.com> On Tue, 2003-09-23 at 16:28, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > has advantage in parallel computations. Having no hands-on experience in I have not heard this. If anyone has, could they provide references? > SuSe Linux, I have no idea how to verify this or do a comparision. What do > you advice as the OS for the cluster? And how to determine this,as there > are a bunch of unix OSes out there.Thanks. > > Sincerely, > Sarat. You have options. Cluster distributions (where they do lots of the hard work for you), or "normal" distributions, where you have to do the hard work. You have the cluster distributions. lcic.org has a list. I have a short list (with links) of them on http://scalableinformatics.com/rocks/, if you search down in the page for Cluster Distributions. You have normal distributions (Redhat/Fedora, Suse, Knoppix, Gentoo, Debian, etc.) Most will require significant additional work. ClusterKnoppix may alleviate some of this. I generally advise matching the distribution to your level of skill/interest in maintaining a distribution/cluster. The cluster distributions handle lots of the pain for you, though some have odd/poorly integrated management philosophies, others have great ideas with poor implementations. Some are good. -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Wed Sep 24 11:34:43 2003 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 24 Sep 2003 08:34:43 -0700 Subject: more cashing in on SCO/Linux FUD? Message-ID: <007601c382b1$614044d0$35a8a8c0@laptop152422> I've been looking at other potential operating systems for embedded clusters, and QNX is one of the options. I just received some (quasi-solicted) commercial email from them including the following interesting paragraph: In light of recent market developments around IP ownership, embedded systems manufacturers are researching OS alternatives to Linux. Developers and technical managers need to determine how software projects implemented in Linux might be transferred to other OSs - what portion of the original code can be preserved, whether similar functionality can be implemented, and how migration can be achieved. Join QNX Software Systems on September 29 for a technical web seminar, "Implementing Device Drivers - Migrating from Linux to a Microkernel OS," where David Donohoe, senior software developer, QNX Software Systems, will demonstrate how an existing Linux- developed driver can be migrated to the QNX(r) Neutrino(r) RTOS, preserving up to 80 per cent of the original software code development. While I think it's a fine thing that QNX is providing alternate solutions for hard-real-time apps, and that they're helping folks understand the difference between Linux and QNX drivers, I find the first statement a bit offensive. Jim Lux _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sherincyriac at hotmail.com Wed Sep 24 11:46:32 2003 From: sherincyriac at hotmail.com (sherin cyriac) Date: Wed, 24 Sep 2003 21:16:32 +0530 Subject: implementation details of Beowulf Message-ID: sir I am going to do a project in Beowulf cluster. I want to know some unsolved problems in this domain .I also need the details of Beowulf cluster that was practically implemented. I need the implementation details. thanking you your's faithfully sherin _________________________________________________________________ MSN Hotmail now on your Mobile phone. http://server1.msn.co.in/sp03/mobilesms/ Click here. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sherincyriac at hotmail.com Wed Sep 24 11:46:09 2003 From: sherincyriac at hotmail.com (sherin cyriac) Date: Wed, 24 Sep 2003 21:16:09 +0530 Subject: (no subject) Message-ID: sir I am going to do a project in Beowulf cluster. I want to know some unsolved problems in this domain .I also need the details of Beowulf cluster that was practically implemented. I need the implementation details. thanking you your's faithfully sherin _________________________________________________________________ Get personal loans. It's hassle-free. http://server1.msn.co.in/msnleads/citibankpersonalloan/citibankploanjuly03.asp?type=txt It's approved instantly. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csarat1 at uic.edu Wed Sep 24 11:25:24 2003 From: csarat1 at uic.edu (Sarat C Maruvada) Date: Wed, 24 Sep 2003 10:25:24 -0500 (CDT) Subject: A question of OS!! In-Reply-To: <3F70C82F.2070307@cert.ucr.edu> References: <3F70C82F.2070307@cert.ucr.edu> Message-ID: Thanks to all of you for your replies. I will definetely keep in mind the points you have mentioned before making a choice for the OS.Thank you once again... Sincerely, Sarat. ************************************************************** Sarat Chandra Maruvada 830,S.Claremont Ave., Research Assistant, CS Dept., #2, Prof. Florin Balasa, Chicago, SEO 937, IL - 60612. UIC. Ph: 312-850-9711. ************************************************************** On Tue, 23 Sep 2003, Glen Kaukola wrote: > Sarat C Maruvada wrote: > > >Hi,I am planning on installing a 32 node beowulf cluster for high > >performance bio-informatics applications. As a OS I was planning on > >installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > >has advantage in parallel computations. Having no hands-on experience in > >SuSe Linux, I have no idea how to verify this or do a comparision. What do > >you advice as the OS for the cluster? And how to determine this,as there > >are a bunch of unix OSes out there.Thanks. > > > > One advantage I can think of that Suse might have is it's automated > installation utility, autoyast I believe it's called. While it's > probably a little harder to set up, I think it would be a bit better > than kickstart for installing Linux on a large number of machines. > Actually though, some of the functionality of autoyast seemed to be > broken to me, but again it's a lot more complicated than kickstart, so > maybe it was just me. You don't have to pay to use Suse's automatic > update tool, but I think that this may change on Red Hat now that > they've merged with Fedora. The only other thing I can think of is that > the Suse distribution seems to come with about twice as many packages to > choose from. Other than that, Linux is Linux, and you're not going to > see much of a speed difference between either Suse or Red Hat. > > My two cents, > Glen > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Wed Sep 24 12:25:57 2003 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 24 Sep 2003 18:25:57 +0200 Subject: rsh In-Reply-To: References: <20030923024026.GE4479@netmeister.org> Message-ID: <20030924162557.GC2124@unthought.net> On Tue, Sep 23, 2003 at 08:32:03PM -0400, Robert G. Brown wrote: ... > The issue can also be avoided (as Josip notes) by using LAM or PVM, > which spawn a daemon via ssh but subsequently start tasks without any > shell-level overhead at all. A fair guess would be, that this connection/communication is not encrypted or strongly authenticated in any way. The resulting security benefit of SSH being null and void. There can be other good reasons for using SSH, probably ease of use if you're used to it, X forwarding, and other niceties. My point here is, that the chain is no stronger than the weakes link. ... > > In many environments, the time wasted by ONE successful password crack > due to snooping is far, far greater than any number of rsh margins. You don't use NFS? If you do, I can put anything in your .bashrc anyway. I use SSH for anything with an 'outside' connection. Typically, SSH will even be firewalled so that only a few select machines can connect to even that service. For internal systems, I am fully aware that since I run NFS, NIS, and some clustering services anyway, running SSH would buy me *zero* security. I treat the network as the computer. Any outside link is firewalled and SSH'ed as appropriate, but internally 'inside the network computer', I have just as much encryption as you have between the PCI slots in your computer. -- ................................................................ : jakob at unthought.net : And I see the elder races, : :.........................: putrid forms of man : : Jakob ?stergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csarat1 at uic.edu Wed Sep 24 12:51:58 2003 From: csarat1 at uic.edu (Sarat C Maruvada) Date: Wed, 24 Sep 2003 11:51:58 -0500 (CDT) Subject: A question of OS!! In-Reply-To: References: Message-ID: Hi, After going through the mails, I have noticed that most of SuSe "advantages" are specific to AMD Opteron processors,mainly because of 64 bit support. Actually the specification of the cluster uses an Athlon XP 2600+. Also since some of you have mentioned,I think it is also prudent to look into cluster distributions (like ROCK for example)..And clearly waiting for RHEL seems like a very good option.Once thing is clear though,no RH 9.0 :-).... Thanks once again. Sincerely, Sarat. ************************************************************** Sarat Chandra Maruvada 830,S.Claremont Ave., Research Assistant, CS Dept., #2, Prof. Florin Balasa, Chicago, SEO 937, IL - 60612. UIC. Ph: 312-850-9711. ************************************************************** On Wed, 24 Sep 2003, Rocky McGaugh wrote: > On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > > > Hi,I am planning on installing a 32 node beowulf cluster for high > > performance bio-informatics applications. As a OS I was planning on > > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > > has advantage in parallel computations. Having no hands-on experience in > > SuSe Linux, I have no idea how to verify this or do a comparision. What do > > you advice as the OS for the cluster? And how to determine this,as there > > are a bunch of unix OSes out there.Thanks. > > > > Sincerely, > > Sarat. > > > > First, I would strongly recommend against using RH9 in a production > environment. It is a short-life-cycle product meant for hobbyist desktops. > More than that, it was a way to beta test some pretty radical OS changes > on the masses before their inclusion into RHEL3. > > Secondly, if you are going to use Opteron there are very good reasons to > use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports > the Opteron while RH9 will only run in 32-bit mode. SLES also has a > 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured > NUMA support for the Opteron. > > If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 > will be out and has the same benefits as SLES. > > -- > Rocky McGaugh > Atipa Technologies > rocky at atipatechnologies.com > rmcgaugh at atipa.com > 1-785-841-9513 x3110 > http://67.8450073/ > perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 24 13:04:19 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 24 Sep 2003 13:04:19 -0400 (EDT) Subject: implementation details of Beowulf In-Reply-To: Message-ID: On Wed, 24 Sep 2003, sherin cyriac wrote: > sir > I am going to do a project in Beowulf cluster. I want to know some > unsolved problems in this domain > .I also need the details of Beowulf cluster that was practically > implemented. I need the implementation details. Start with: http://www.beowulf.org and proceed with http://www.phy.duke.edu/brahma (and links thereon). There are LOTS of web resources out there, and many of those resources are linked back to one of these sites or to sites linked to one of these sites (e.g. beowulf underground, MPI and PVM and Scyld and more). HTH, rgb > thanking you > your's faithfully > sherin > > _________________________________________________________________ > MSN Hotmail now on your Mobile phone. > http://server1.msn.co.in/sp03/mobilesms/ Click here. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Wed Sep 24 13:01:52 2003 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 24 Sep 2003 19:01:52 +0200 Subject: more cashing in on SCO/Linux FUD? In-Reply-To: <007601c382b1$614044d0$35a8a8c0@laptop152422> References: <007601c382b1$614044d0$35a8a8c0@laptop152422> Message-ID: <20030924170152.GD2124@unthought.net> On Wed, Sep 24, 2003 at 08:34:43AM -0700, Jim Lux wrote: ... > While I think it's a fine thing that QNX is providing alternate solutions > for hard-real-time apps, and that they're helping folks understand the > difference between Linux and QNX drivers, I find the first statement a bit > offensive. It's marketing. Yesterday we had a 7 hour power outage, which covered eastern Denmark and southern Sweden. The largest outage in 22 years. Today I got a mail from CA wanting to sell me backup software, to keep my data safe in case of a new power outage. Fear and greed is effective on human beings. Marketing's job is to try to be effective on human beings. And we're headed OT fast ;) -- ................................................................ : jakob at unthought.net : And I see the elder races, : :.........................: putrid forms of man : : Jakob ?stergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Sep 24 13:28:27 2003 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 24 Sep 2003 11:28:27 -0600 Subject: rsh In-Reply-To: <20030924162557.GC2124@unthought.net> References: <20030923024026.GE4479@netmeister.org> <20030924162557.GC2124@unthought.net> Message-ID: <3F71D43B.7020106@lanl.gov> Jakob Oestergaard wrote: > On Tue, Sep 23, 2003 at 08:32:03PM -0400, Robert G. Brown wrote: > ... > >>The issue can also be avoided (as Josip notes) by using LAM or PVM, >>which spawn a daemon via ssh but subsequently start tasks without any >>shell-level overhead at all. > > > A fair guess would be, that this connection/communication is not > encrypted or strongly authenticated in any way. Correct. > The resulting security benefit of SSH being null and void. > [...] > For internal systems, I am fully aware that since I run NFS, NIS, and > some clustering services anyway, running SSH would buy me *zero* > security. Yes. Inside a cluster contained in a locked computer room, you've got physical network security, so SSH overhead is not necessary. > I use SSH for anything with an 'outside' connection. Typically, SSH will > even be firewalled so that only a few select machines can connect to > even that service. A wise policy. Outside connections need SSH. However, in most cases you do not want to parallelize across outside connections, because they tend to be slow. Since nobody I know parallelizes across outside connections, SSH simply does not get involved in parallel runs. LAM damons are just fine internally. However, cluster access from the outside is usually restricted to SSH w/X forwarding. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From goutam at infovision21.com Wed Sep 24 14:47:29 2003 From: goutam at infovision21.com (goutam) Date: Wed, 24 Sep 2003 14:47:29 -0400 Subject: Comparisions,, Message-ID: Hello all, I am starting to build a cluster of 8-10 computers (PIII Redhat 2.2.x). I would like comparitive views of Scyld , OSCAR or globus.. Its mainly for Academic purposes but will run a few professional grade parallel programs.. thanks ! goutam _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctibirna at giref.ulaval.ca Wed Sep 24 14:51:52 2003 From: ctibirna at giref.ulaval.ca (Cristian Tibirna) Date: Wed, 24 Sep 2003 14:51:52 -0400 Subject: upgrading rh73 on an xCAT cluster Message-ID: <200309241451.52982.ctibirna@giref.ulaval.ca> Hello Yesterday I upgraded (first time after 7 months... I know, I know) the rh73 rpms and the kernel. Since then, I have two nasty issues: 1) The update installed a new openssh (3.1.p1-14) The auth of sshd through pam is annoyingly slower. All ssh connections (both from outside to the master and from any node to any node inside) _are_ succeeding, but a lot slower. I see this in the /var/log/messages too: Sep 24 13:16:04 n15 sshd(pam_unix)[27164]: authentication failure; logname=\ uid=0 euid=0 tty=NODEVssh ruser= rhost=n01 user=root Sep 24 13:16:04 n15 sshd(pam_unix)[24856]: session opened for user root by\ (uid=0) Both messages are for the same ssh connection attempt and the attempt succeeds, as I said. The only visible effect to the user is the slowness (the first failure is followed by a programmed delay in pam). I looked a bit around the 'net and people have already complained a lot about this problem but I found no solution. 2) I also updated the kernel to 2.4.20-20.7 (redhat rpm). Afterwards, my (and other users') SGE qmake jobs just get stuck in the middle (i.e. function correctly for a while then suddenly just sit there and do nothing for long time, without having completed). I feel it's some sort of NFS lockup problem as the master node (NFS server) gets very high loads (6.0-8.0) compared to before (2.0-3.0) the update of the kernel. The /var/log/messages says nothing useful. Did anybody already updated a rh73 cluster equipped with SGE and using ssh internally? Observed these problems? Found solutions? Thanks a lot. -- Cristian Tibirna (1-418-) 656-2131 / 4340 Laval University - Quebec, CAN ... http://www.giref.ulaval.ca/~ctibirna Research professional at GIREF ... ctibirna at giref.ulaval.ca _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 24 15:59:50 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 24 Sep 2003 12:59:50 -0700 Subject: MPICH vs LAM In-Reply-To: <3F7079B4.4090807@lanl.gov> References: <20030923124408.91196.qmail@web12205.mail.yahoo.com> <3F7079B4.4090807@lanl.gov> Message-ID: <20030924195950.GB2061@greglaptop.internal.keyresearch.com> On Tue, Sep 23, 2003 at 10:49:56AM -0600, Josip Loncaric wrote: > MPICH is very popular and works fine, but LAM is a bit cleaner > implementation which delivers a bit lower latency. Other than that, the > main difference is process startup. With MPICH, "mpirun" does the job > directly and accepts the usual rsh or ssh authorization overhead. mpich now distributes multiple startup mechanisms -- one is similar to the LAM daemon approach. This is a sign that the MPI community is doing a good job of borrowing good ideas from each other. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Wed Sep 24 03:59:19 2003 From: landman at scalableinformatics.com (Joe Landman) Date: Wed, 24 Sep 2003 03:59:19 -0400 Subject: upgrading rh73 on an xCAT cluster In-Reply-To: <200309241451.52982.ctibirna@giref.ulaval.ca> References: <200309241451.52982.ctibirna@giref.ulaval.ca> Message-ID: <1064390359.5989.3.camel@squash.scalableinformatics.com> Hi Cristian: This sounds like a mixture of a few problems, including a name service/host lookup issue for the ssh slowness. The NFS server could be a number of things. Which kernel are you using? How many machines are mounting the NFS system? What does nfsstat say, and what does vmstat indicate? How many nfsd's are running? I have found that with name services (either through NIS, DNS, etc) timing out, ssh gets quite slow. One way to try this is doing an ssh to the ip address of the compute node rather than its name. If the times are quite similar, there may be other issues at work. If the ip address method is much faster, then name resolution is not working somewhere. Joe On Wed, 2003-09-24 at 14:51, Cristian Tibirna wrote: > Hello > > Yesterday I upgraded (first time after 7 months... I know, I know) the rh73 > rpms and the kernel. Since then, I have two nasty issues: > > 1) > The update installed a new openssh (3.1.p1-14) > > The auth of sshd through pam is annoyingly slower. All ssh connections (both > from outside to the master and from any node to any node inside) _are_ > succeeding, but a lot slower. I see this in the /var/log/messages too: > > Sep 24 13:16:04 n15 sshd(pam_unix)[27164]: authentication failure; logname=\ > uid=0 euid=0 tty=NODEVssh ruser= rhost=n01 user=root > Sep 24 13:16:04 n15 sshd(pam_unix)[24856]: session opened for user root by\ > (uid=0) > > Both messages are for the same ssh connection attempt and the attempt > succeeds, as I said. The only visible effect to the user is the slowness (the > first failure is followed by a programmed delay in pam). > > I looked a bit around the 'net and people have already complained a lot about > this problem but I found no solution. > > 2) > I also updated the kernel to 2.4.20-20.7 (redhat rpm). > > Afterwards, my (and other users') SGE qmake jobs just get stuck in the middle > (i.e. function correctly for a while then suddenly just sit there and do > nothing for long time, without having completed). I feel it's some sort of > NFS lockup problem as the master node (NFS server) gets very high loads > (6.0-8.0) compared to before (2.0-3.0) the update of the kernel. The > /var/log/messages says nothing useful. > > > Did anybody already updated a rh73 cluster equipped with SGE and using ssh > internally? Observed these problems? Found solutions? > > Thanks a lot. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 24 17:15:49 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 24 Sep 2003 14:15:49 -0700 Subject: Redhat Fedora In-Reply-To: References: Message-ID: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> On Tue, Sep 23, 2003 at 10:37:45AM +0200, John Hearns wrote: > As most of us know, Redhat announced that they are pursuing a new > community led development model for future Redhat 'consumer' releases. > I'm sure I'm not the only one who was wondering what that meant for the > direction of cluster computing. It's an interesting question. RedHat's new advice on the matter is "buy a copy of Advanced Workstation for every cluster node." They are also changing how they do updates to the "consumer" OS; for example, I pointed out that xpdf was miscompiled without all the font support in RH 9, but RH did not release a fix, they closed the bug report with "fixed in RH 10." I suspect that we'll see: More use of hobbist distros like Debian People putting together freely-copyable ISOs for RedHat AW and AS More interest in Mandrake's EU-supported cluster distro The first two will be driven by the general user community, not the cluster people; "we" (the cluster community) tend to not do things like build our own distros, nor are "we" willing to pay enough for vendors (like Scyld) to build cluster distros. > And it supports YUM and apt-get, which can only be a good thing. That's a good thing -- much easier than looking around at ftp mirrors for updates. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bob at drzyzgula.org Wed Sep 24 19:44:42 2003 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Wed, 24 Sep 2003 19:44:42 -0400 Subject: Redhat Fedora In-Reply-To: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> Message-ID: <20030924194442.H3752@www2> On Wed, Sep 24, 2003 at 02:15:49PM -0700, Greg Lindahl wrote: > > On Tue, Sep 23, 2003 at 10:37:45AM +0200, John Hearns wrote: > > > As most of us know, Redhat announced that they are pursuing a new > > community led development model for future Redhat 'consumer' releases. > > I'm sure I'm not the only one who was wondering what that meant for the > > direction of cluster computing. > > It's an interesting question. RedHat's new advice on the matter is > "buy a copy of Advanced Workstation for every cluster node." They are > also changing how they do updates to the "consumer" OS; for example, I > pointed out that xpdf was miscompiled without all the font support in > RH 9, but RH did not release a fix, they closed the bug report with > "fixed in RH 10." Heck, that's nothing -- at least they fixed it. I reported a bug in the RH 6.1 implementation of kudzu in October 1999. There was no activity in bugzilla until almost three years later, in June of 2002, and that only amounted to someone asking "is this fixed yet", and getting the answer that it was not. That was the last activity on the bug; the status remains "assigned": https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=6463 > I suspect that we'll see: > > More use of hobbist distros like Debian > People putting together freely-copyable ISOs for RedHat AW and AS > More interest in Mandrake's EU-supported cluster distro I suppose there's always the posibility that the whole Fedora/Redhat merge thing will work out, and it will become a popular non-commercial alternative to Debian. Also, I'm thinking I must not know enough about Mandrake's cluster distribution. Looking at their website, it seems to cost $2320 for 1-8 processors, $4176K for 9-16 processors. Why would a cluster shop that had previously used Red Hat Linux choose to use Mandrake rather than Red Hat's Enterprise WS distribution, which costs $299 per single- or dual-processor system, except perhaps on technical grounds? > The first two will be driven by the general user community, not the > cluster people; "we" (the cluster community) tend to not do things > like build our own distros Actually, I thought that in fact "we" tended to do a lot of this -- cf. Rocks, Clustermatic, Warewulf -- it's just that these never seem to capture the market in the same way that Red Hat always was. > nor are "we" willing to pay enough for > vendors (like Scyld) to build cluster distros. This sounds right. > > And it supports YUM and apt-get, which can only be a good thing. > > That's a good thing -- much easier than looking around at ftp mirrors > for updates. No argument with that. --Bob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 24 20:54:46 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 24 Sep 2003 20:54:46 -0400 (EDT) Subject: Redhat Fedora In-Reply-To: <20030924194442.H3752@www2> Message-ID: On Wed, 24 Sep 2003, Bob Drzyzgula wrote: > > Also, I'm thinking I must not know enough about Mandrake's > cluster distribution. Looking at their website, it > seems to cost $2320 for 1-8 processors, $4176K for 9-16 > processors. Why would a cluster shop that had previously > used Red Hat Linux choose to use Mandrake rather than > Red Hat's Enterprise WS distribution, which costs $299 > per single- or dual-processor system, except perhaps on > technical grounds? The interesting question is why anybody in their right mind would pay for any sort of GPL-based distribution that scales per number of processors. Especially pay numbers like these. Hell, one might as well be running Windows. I personally think that the global internet has long since moved on to a place where this just won't fly, especially for GPL software. It's almost funny, in a very sad sort of way. People all over the world (many of them on this list) have written and contributed the actual software and continue to maintain it. Red Hat is now preparing to turn around and sell our own software back to us at an absurd markup and with little (really) added value other than the distributional packaging and testing. I personally think that what is going to happen is that the community will interpret this as control and just route around it. People on the Internet proper will just ignore the prices; corporate managers who feel a need to "buy" something may pay them. Companies seem to be trying to treat open source software as a proprietary product and moving away from the support model. The real tragedy associated with this is that linux is finally maturing to the point where it could really challenge on the desktop, not just the server/cluster/specialized market, and these absurd prices could retard that process for years. Red Hat in particular has missed the boat by charging crazy prices for box sets for some years. What, they think Joe Consumer is going to buy a box set of linux for $150 when it comes "free" and preinstalled on their PC? They should be selling box sets for $15 and trying to sell a LOT of them, not a few at crazy prices. > > nor are "we" willing to pay enough for > > vendors (like Scyld) to build cluster distros. > > This sounds right. Absolutely. It is a matter of what people need. MPI, PVM, and more are often or even generally available in rpm format and are a whole lot of what people need. The cluster users who need to go a long way beyond this exist, but there are fewer and fewer of them at each additional price notch. COTS HPC clusters were invented because they were a CHEAP road to supercomputing that supported the needs of a hell of a lot of HPC cycle consumers with only a few, relatively simple tools. The same issues exist with Myrinet and SCI -- I'll be there are dozens of clusters installed on plain old ethernet for each one of the clusters installed on high end networks (which, frankly, are worth a lot more than high end/expensive "distributions" since Red Hat cannot alter the fundamentally limiting computation/communications ratio at a given task scale, but a faster network interface can!). So pay no attention to the high road. Distribution vendors that take the high road are going to starve on a rich diet of a few high margin sales, just as a lot of the traditional Unix vendors have been doing at ever increasing rates ever since linux was invented. Look instead to the low road. Look to distributions (like Debian or freebsd) that charge nothing, or charge a very modest low margin fee. In the market of the future one will be able to grow fat on huge numbers of LOW margin sales, just as Microsoft (for the most part) has for years. If Sun Microsystems had released their perfectly good Unix for PC's for a retail price of $15/system in a box set a decade plus ago (or even $50), Linux would never have gotten off the ground, Windows NT would have been killed in its cradle, and Sun would own the universe. I think it must be the scent of blood in the water -- Microsoft is finally showing signs of distinct vulnerability, the linux desktop is finally good enough to sell to corporate customers, and the long awaited phase transition is at hand. Now all the linux distro people can think about is how fast they can get their options and their shareholder's highly depressed shares above water. Slow and steady (and low margin sales and true added value support) will win this race. Red Hat is already facing a sort of defection of a rather huge part of their installed base; how long will they stay on top once the Universities and other institutions that provide their real base of programmers turn their energies to something that they don't have to turn around and "buy back"? With the GPL, not long at all, not long at all. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From glen at cert.ucr.edu Wed Sep 24 21:22:51 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Wed, 24 Sep 2003 18:22:51 -0700 Subject: Redhat Fedora In-Reply-To: <20030924194442.H3752@www2> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> Message-ID: <3F72436B.5020609@cert.ucr.edu> Bob Drzyzgula wrote: >Also, I'm thinking I must not know enough about Mandrake's >cluster distribution. Looking at their website, it >seems to cost $2320 for 1-8 processors, $4176K for 9-16 >processors. Why would a cluster shop that had previously >used Red Hat Linux choose to use Mandrake rather than >Red Hat's Enterprise WS distribution, which costs $299 >per single- or dual-processor system, except perhaps on >technical grounds? > Mandrake puts out a completely free cluster distribution called clic: http://clic.mandrakesoft.com/index-en.html Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bob at drzyzgula.org Wed Sep 24 23:51:33 2003 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Wed, 24 Sep 2003 23:51:33 -0400 Subject: Redhat Fedora In-Reply-To: References: <20030924194442.H3752@www2> Message-ID: <20030924235133.I3752@www2> On Wed, Sep 24, 2003 at 08:54:46PM -0400, Robert G. Brown wrote: > > On Wed, 24 Sep 2003, Bob Drzyzgula wrote: > > > > > Also, I'm thinking I must not know enough about Mandrake's > > cluster distribution. Looking at their website, it > > seems to cost $2320 for 1-8 processors, $4176K for 9-16 > > processors. Why would a cluster shop that had previously > > used Red Hat Linux choose to use Mandrake rather than > > Red Hat's Enterprise WS distribution, which costs $299 > > per single- or dual-processor system, except perhaps on > > technical grounds? > > The interesting question is why anybody in their right mind would pay > for any sort of GPL-based distribution that scales per number of > processors. Especially pay numbers like these. Hell, one might as well > be running Windows. There are two reasons, actually, that I can think of, although they only apply in specific environments that probably don't include e.g. universities that are strapped for cash but have a near-inexhaustible supply of cheap labor near at hand. These reasons are: (a) The promise of support. This is generally more for managers than for the techies, since in a large proportion of cases, the techies most likely know what they are doing. But managers often don't know enough about the technology to know whether to trust that the techies know what they are doing (and perhaps have occasional real-life experiences with failure in this regard), so they at a minimum want to know that there is someone at the vendor who's *job* it is to pick up the phone and answer *their* questions. This may seem irrational, but it isn't really. (b) The promise of a patch stream. This is the thing that is of more actual use to the techies. Especially with the way that RPM works, maintaining patches for an older distribution gets to be a real burden, and it is quite clear to me why Red Hat is trying to get out from under this, especially for the "consumer" releases. To some extent, it's their own damned fault what with the way that find-requires works to build in dependencies on dot-releases of system libraries, but in any event the availability of tested, binary patches for a three-year-old system is of real value and IMHO is worth paying for. Maybe not thousands of dollars per system, but then if I had millions of dollars of income riding on a system, thousands might not seem like all that big a deal. Again, this all likely seems more attractive in a business setting than in a University setting. > I personally think that the global Internet has long since moved on to > a place where this just won't fly, especially for GPL software. I think (and certainly Red Hat is betting) that it will fly with enough customers to make them profitable in the long run. > It's almost funny, in a very sad sort of way. People all over the world > (many of them on this list) have written and contributed the actual > software and continue to maintain it. Red Hat is now preparing to turn > around and sell our own software back to us at an absurd markup and with > little (really) added value other than the distributional packaging and > testing. I'm not certain that this is what they are thinking. I'm guessing that they know full well that "we" [and I use "we" for the purposes of this discussion but acknowledge that I personally probably don't belong in this class, not having contributed much of anything] will never pay them any money for this, just like we haven't been paying them any money for it in the past. In this regard, yes, they have written us off. But I think that they have to be given credit for at least attempting to set up a structure within which we can help to maintain a usable RedHat-style distribution in the long run, without it being such a money pit for them. There is certainly lots of precedent for companies just dumping unprofitable products without such a gentle transition. > I personally think that what is going to happen is that the community > will interpret this as control and just route around it. People on the > Internet proper will just ignore the prices; corporate managers who feel > a need to "buy" something may pay them. Companies seem to be trying to > treat open source software as a proprietary product and moving away from > the support model. Well certainly I can think of one Utah-based company in particular that is taking this to absurd levels, but in Red Hat's case I think that they're just trying to find a way to make a sustainable profit on this. I don't see that it makes sense to begrudge them this -- they're not a volunteer organization or even a non-profit. I'm not sure what you mean by "moving away from the support model", because it seems to me that what Red Hat (and others) are doing is, on the contrary, to make support the *product*. All the nonsense about proprietary logos and per-system licenses is nothing more than a lever to coerce customers into buying this support. Unlike closed-source companies like Microsoft, though, Red Hat is *not* preventing you from using the software without buying this support, but they are making it very difficult for you to take advantage of all the stuff they do to make it *easy* to use their software without paying -- making it easy is what they're trying to charge for. > The real tragedy associated with this is that linux is finally maturing > to the point where it could really challenge on the desktop, not just > the server/cluster/specialized market, and these absurd prices could > retard that process for years. Red Hat in particular has missed the > boat by charging crazy prices for box sets for some years. What, they > think Joe Consumer is going to buy a box set of linux for $150 when it > comes "free" and preinstalled on their PC? They should be selling box > sets for $15 and trying to sell a LOT of them, not a few at crazy > prices. But they have been doing that, and what they have been doing has sound economic basis. They've been segmenting the market, charging each customer according to his or her propensity to pay, much for the same reason that car prices have typically been negotiated. In the past, customers have been able to buy Red Hat Linux for next to nothing -- the cost of downloading the ISOs and burning them on to CD-Rs, or even just borrowing their buddy's copy. They've been able to buy it for $10 or so by ordering white-label CDs from Cheap Bytes. They've been able to buy it for $40 or so by picking up a book on Red Hat Linux at the bookstore. They've been able to buy a basic version with a real Red Hat label for maybe $50, and a nice box with an official manual, some amount of support, and CDs containing all sorts of crap for $150. And recently they've been making it available for prices extending into the thousands of dollars, with all sorts of hand-holding thrown in. They've thus been making Red Hat Linux available at a near-continuum of prices, and letting the customers decide how much they'd like to pay. This is a pretty classic formula for maximizing your income on a product, because at the low end, you wind up getting at least a little money from people who otherwise wouldn't have given you anything. The freeloaders are of course a loss but you can at least hope that you've picked up some good will and maybe they'll think of you when they are in a mood to spend actual money. And you can wind up getting some bigger chunks of cash from the people who don't mind parting with it if it gets them something that's all dressed up nice and pretty. However, I think what we're seeing now is that their product is well-enough established in the market that they believe they can do a bit better if they knock out some of the lower prices, and make a somewhat larger distinction between the free "toy" product and the expensive version intended for the "serious" customer. It is of course possible that they've misjudged the market some and they'll have to fine tune this again, possibly by continuing to take snapshots of Fedora Linux and calling it "Red Hat", in the same way that Sun repackages some builds of Open Office as Star Office and Netscape had been using the occasional build of Mozilla for a new release of Netscape. But for now this is what they're going to try, and judging from their latest quarterly report, it seems to be working. > > > nor are "we" willing to pay enough for > > > vendors (like Scyld) to build cluster distros. > > > > This sounds right. > > Absolutely. It is a matter of what people need. MPI, PVM, and more are > often or even generally available in rpm format and are a whole lot of > what people need. The cluster users who need to go a long way beyond > this exist, but there are fewer and fewer of them at each additional > price notch. COTS HPC clusters were invented because they were a CHEAP > road to supercomputing that supported the needs of a hell of a lot of > HPC cycle consumers with only a few, relatively simple tools. The same > issues exist with Myrinet and SCI -- I'll be there are dozens of > clusters installed on plain old ethernet for each one of the clusters > installed on high end networks (which, frankly, are worth a lot more > than high end/expensive "distributions" since Red Hat cannot alter the > fundamentally limiting computation/communications ratio at a given task > scale, but a faster network interface can!). Well, I think that this is an important point, and what I'd say is that it reflects how the cluster market is different from the general-purpose business market. > So pay no attention to the high road. Distribution vendors that take > the high road are going to starve on a rich diet of a few high margin > sales, just as a lot of the traditional Unix vendors have been doing at > ever increasing rates ever since linux was invented. But the problem with traditional Unix vendors, I believe, has less to do with the price of their operating system than (a) the price of their hardware, and (b) the lack of openness in the operating system. Linux is pulling people away because it allows them to take advantage of such a wide variety of hardware, and because they have nearly unlimited choices in sources of support for their systems. Linux shops can be exactly as price- or quality-sensitive as they want or need to be and still use Linux. A little, cash-starved non-profit can run a usable file/print/web server on a box they picked up from the Salvation Army store for $50, while a Fortune 500 shop can consolidate a hundred different services onto an IBM mainframe running Linux under VM. The non-profit with the $50 server can pay the secretary's teenage son $10/hour to dig around and figure out why Samba is broken, while the Fortune 500 can pay IBM six or even seven figures a year to keep the system running like the proverbial well-oiled machine. As it stands, because of the GPL, the distribution vendors are *never* going to get any money out of that non-profit, but as long as the high-end systems work and excellent support is available, those high-margin sales are going to become more and more common; I really don't see the starvation you're expecting. > Look instead to the low road. Look to distributions (like Debian or > freebsd) that charge nothing, or charge a very modest low margin fee. > In the market of the future one will be able to grow fat on huge numbers > of LOW margin sales, just as Microsoft (for the most part) has for > years. If Sun Microsystems had released their perfectly good Unix for > PC's for a retail price of $15/system in a box set a decade plus ago (or > even $50), Linux would never have gotten off the ground, Windows NT > would have been killed in its cradle, and Sun would own the universe. Ah, but what would it have done to SPARC sales? Sun's situation is quite different from that of a Linux vendor such as Red Hat, in that Sun's business model depends on selling high-margin hardware. Solaris x86, IMHO, was always intended as a loss leader designed to get people hooked on Solaris, and draw them into buying the real thing to get broader range of applications support and greater hardware scalability. They never intended Solaris x86 to become a first-class solution platform in and of itself. Sure, a few VARs and ISPs made it work as one, but Sun was never comfortable with that and as you recall they tried a couple of years ago to kill it off altogether. Scott has always been adamant that he was never going to allow Sun to be held hostage to an externally-defined hardware architecture. One of his favorite sayings was that Sun has never lost a dime selling a PC. To a large extent this worked, but IMHO the mistake they made was when they got too greedy in the late 1990s and allowed the low end of their product line to languish; Sun hated the thought that a customer might be able to build their network around $10,000 machines when $50,000 machines could do the job just as well (yes, I meant that). As a result, in the past few years they haven't had anything affordable to offer to companies who were seeing their IT budgets drying up, and they were forced to sell x86 hardware and even -- gasp -- Linux to compensate for this mistake. > I think it must be the scent of blood in the water -- Microsoft is > finally showing signs of distinct vulnerability, the linux desktop is > finally good enough to sell to corporate customers, and the long awaited > phase transition is at hand. Now all the linux distro people can think > about is how fast they can get their options and their shareholder's > highly depressed shares above water. Another way of interpreting the behavior might be that, in slowed economic conditions they are finding it necessary to either turn a profit in short order or go under. While it may not make a lot of difference in the long term for the scientific research community -- we were getting along with Linux just fine long before it became "profitable" -- I really think that the failure of Red Hat in particular would be very bad for Linux in general. > Slow and steady (and low margin sales and true added value support) will > win this race. Red Hat is already facing a sort of defection of a > rather huge part of their installed base; how long will they stay on top > once the Universities and other institutions that provide their real > base of programmers turn their energies to something that they don't > have to turn around and "buy back"? > > With the GPL, not long at all, not long at all. Again, I think that, while they may be losing a large part of their installed base, they are not likely to be losing a large part of their *income*. This is an important distinction that goes a long way to explaining why they are doing it. And again, they are giving us an outlet for our energies that we don't have to "buy back" -- Fedora Linux. We don't have to accept it, but I think it's a better gesture than we'd get from a lot of companies. Back when Sun decided to cancel Solaris x86, I didn't see them offering to turn the product over to the community. --Bob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 25 04:01:14 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 25 Sep 2003 10:01:14 +0200 (CEST) Subject: implementation details of Beowulf In-Reply-To: Message-ID: On Wed, 24 Sep 2003, sherin cyriac wrote: > sir > I am going to do a project in Beowulf cluster. I want to know some > unsolved problems in this domain > .I also need the details of Beowulf cluster that was practically > implemented. I need the implementation details. Sherin, if you are a college or university student, I suggest a quick visit to your college library! Have a search on Amazon for books with 'Beowulf' or 'Linux Clustering' in the title, and go see if your college has them. Otherwise, start at www.beowulf.org _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Wed Sep 24 04:26:09 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 24 Sep 2003 10:26:09 +0200 Subject: A question of OS!! In-Reply-To: References: Message-ID: <1064391969.3349.17.camel@revolution.mandrakesoft.com> > I've used both SuSE and Redhat, and like both. > In fact, we use SuSE on Opterons, as SuSE were there first with > a complete distro, which is very good. This is not true, MandrakeSoft had release the Opteron version of its Linux distribution at the same time that the processor was available. So you have the choice for Opteron between MandrakeLinux and Suse. And for the clustering things, MandrakeClustering also exists for Opteron. > I fail to see though why SuSE would have an advantage in parallel > computations. I think it was initialy because suse had provided a lot of libraries. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From erwan at mandrakesoft.com Wed Sep 24 05:32:38 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 24 Sep 2003 11:32:38 +0200 Subject: A question of OS!! In-Reply-To: References: Message-ID: <1064395958.3347.90.camel@revolution.mandrakesoft.com> Le mer 24/09/2003 ? 10:58, Cannon, Andrew a ?crit : >So which of the mainstream distros are best for Beowulfs? RH, SuSe, >Mandrake etc? > We use RH8, but I am unsure about whether we should still use this or go to > another distro. For my side (MandrakeSoft of course :D), I could tell that MandrakeClustering is a real cluster oriented Linux Distribution. I mean it integrates a lot of thing in one package (a distro) "OS (MandrakeLinux) + Admin & Config tools + Deployment tools (KA) + MPICH & LAM/MPI & PVM + Usual cluster tools (rshp, gexec, ganglia, dsh, clusterit etc...)." Everything is autoconfigured and ready to run after the install process. No need to be a Linux Guru to install a full cluster. Our customers are happy about that point. Scientists (which are not all Linux Gurus) are able to install their owns cluster in a few hours ! MDKC also provides "urpmi parallel" for managing the rpm distribution across the cluster. For this points, RedHat & Suse don't do the same today. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at xnet.com Wed Sep 24 18:37:08 2003 From: djholm at xnet.com (Don Holmgren) Date: Wed, 24 Sep 2003 17:37:08 -0500 (CDT) Subject: Announcement: AMD Opteron and Athlon 64 presentation Message-ID: For those near Atlanta who might be interested in a presentation by AMD of Opteron and Athlon 64 information, see: http://www.csilabs.net/AMD64seminar/ I asked for some more details of the presentations: "As the title of the seminar suggests, the focus will be on AMD64 technology and the AMD64 portfolio of products including both Opteron and Athlon 64. We are trying to schedule some of our HQ guys from our desktop and server product marketing teams down for the event along with other local support. As part of the Opteron discussion, we will certainly discuss the benefit of Opteron processor-based systems across applications including High Performance Computing and clustering as well as the way Opteron simplifies business by providing a single platform for use across the enterprise data center." Don Holmgren _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Wed Sep 24 12:39:05 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Wed, 24 Sep 2003 11:39:05 -0500 (CDT) Subject: A question of OS!! In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > has advantage in parallel computations. Having no hands-on experience in > SuSe Linux, I have no idea how to verify this or do a comparision. What do > you advice as the OS for the cluster? And how to determine this,as there > are a bunch of unix OSes out there.Thanks. > > Sincerely, > Sarat. > First, I would strongly recommend against using RH9 in a production environment. It is a short-life-cycle product meant for hobbyist desktops. More than that, it was a way to beta test some pretty radical OS changes on the masses before their inclusion into RHEL3. Secondly, if you are going to use Opteron there are very good reasons to use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports the Opteron while RH9 will only run in 32-bit mode. SLES also has a 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured NUMA support for the Opteron. If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 will be out and has the same benefits as SLES. -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan.velu at free.fr Thu Sep 25 05:12:44 2003 From: erwan.velu at free.fr (erwan.velu at free.fr) Date: Thu, 25 Sep 2003 11:12:44 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] Message-ID: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> Drzyzgula wrote: > > >Also, I'm thinking I must not know enough about Mandrake's > >cluster distribution. Looking at their website, it > >seems to cost $2320 for 1-8 processors, $4176K for 9-16 > >processors. Why would a cluster shop that had previously > >used Red Hat Linux choose to use Mandrake rather than > >Red Hat's Enterprise WS distribution, which costs $299 > >per single- or dual-processor system, except perhaps on > >technical grounds? > > > Mandrake puts out a completely free cluster distribution called clic: > http://clic.mandrakesoft.com/index-en.html Firstable, sorry for not using my corporate email but since few days all the emails sent to beowulf ML reach the mailbox but I never seen any message (except one because I had ask for it ... I am filtred ?) All the fedora thread is really intresting... Why people should pay for such clustering oriented distro ? Why CLIC is free whereas you must pay for MDKC with a per-CPU licence ? CLIC was a project co-financed by the french government, so it MandrakeSoft and its partners were being half payed for this work. It was obvious that we are making a GPL product available for download. CLIC is a real technological success and its customers like it for this. Today we are facing the following problem : how to continue such project ? How can MandrakeSoft could make some revenues on this king of product ? What do people using CLIC gives at MandrakeSoft in return of its work ? Some buy some consulting, but some do nothing... CLIC is finishing at Decembrer 2003, so should we stop CLIC ? How the guys are working hard on such product could be payed ? We decided to launch a new product called MandrakeClustering based on the CLIC success. We have rewritten all the backend in perl, added a lot of features. We want this technology to survive. Now MDKC is a real product, not a research project as CLIC. MandrakeSoft now sells this product whereas we were just selling consulting on CLIC. This model now change the rules. Products like maui can't be resell without permissions. There is a per-cpu fee for such product, so we are now applying a per-cpu pricing on our product. The price of MDKC is really reasonable ! For 8CPUS under x86 or AMD64 it costs 2000?, until 16 -> 3600? etc.. Please make the ratio between the hardware price and the software price. What is the future of CLIC ? Is will depend of the results of MandrakeClustering. If it works fine, we will continue CLIC because the clustering team could be payed for such work. This is what we are doing today. The technology of MDKC is currently in integration in CLIC. So CLIC will contain a part of MDKC a few after the release. The last point is you can't compare RedHat Enterprise with MandrakeClustering ! MandrakeClustering is a real oriented clustering distro not just a MandrakeLinux + some few packages. MDKC is integrating the Linux Distro + configuration tools + deployment tools + clustering tools + clustering mpi stacks. Everything is auto configured, you don't need to be a Linux expert to install a cluster. My best example ? How to integrate new nodes ? CLICK on a node, CLICK on duplicate, tell the number of nodes you want to duplicate, the device you want to duplicate. Then just power on your nodes ! Nothing more to do ! You just need a few hours to setup a full cluster ! DNS, DHCP, PXE, NIS, AUTOFS, LAM/MPI, MPICH, PVM, TFTP, etc.. Everything is built automaticaly ! Does RH do it ? You can manage all the rpms on your cluster using "urpmi" (could be compared to apt) but using parallel tools ! How to install a tool on a cluster ? urpmi --parallel cluster Dependencies are automaticaly installed like apt. Does RH do it ? You can install security/product updates from the server using a simple command: urpmi --parallel cluster --auto-select --auto Updates are downloaded from Internet then parallely send to the nodes then each nodes update itslef following its own needs ! Who can do that today ? Does RH allow you to duplicate 200 nodes in 3 minutes ~500 MBits ? No, you need to integrate yourself other tools you must installed and configure. MDKC & CLIC are some real clustering oriented distro, everything has been thought for the easyness of use.. I don't know if this king of approach could match all the clustering needs, but I must say that people who did really test this kind of distribution likes it. I'm not saying that other products are poor, we just have another approach that our Linux editor postion allow us to give. I hope this mail could help everyone to understand the work of the CLIC & MDKC team. Linuxely yours, -- Erwan Velu (erwan at mandrakesoft.com) Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Wed Sep 24 05:09:32 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 24 Sep 2003 11:09:32 +0200 Subject: rsh In-Reply-To: <20030917024414.M10506@ug.ms.wits.ac.za> References: <20030917024414.M10506@ug.ms.wits.ac.za> Message-ID: <1064394572.3347.57.camel@revolution.mandrakesoft.com> > I tried running tstmachines and I got > "Errors while trying to run rsh hostname -n true > Unexpected response from hostname: > ---> Permission denied ..." > Please help with the rsh settings that I need to inorder to get the rsh to work. > Masego-otlhe Segone Please check that your the .rhost of your user account is like the following: example: final.compute.mdkc.mandrakesoft.com test compute1.compute.mdkc.mandrakesoft.com test compute3.compute.mdkc.mandrakesoft.com test compute4.compute.mdkc.mandrakesoft.com test compute5.compute.mdkc.mandrakesoft.com test compute6.compute.mdkc.mandrakesoft.com test compute7.compute.mdkc.mandrakesoft.com test compute8.compute.mdkc.mandrakesoft.com test compute9.compute.mdkc.mandrakesoft.com test compute2.compute.mdkc.mandrakesoft.com test then be sure that .rhost is 644, "vi" give usualy a group writable permission that rsh doesn't like. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From joelja at darkwing.uoregon.edu Thu Sep 25 13:07:36 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Thu, 25 Sep 2003 10:07:36 -0700 (PDT) Subject: A question of OS!! In-Reply-To: Message-ID: On Wed, 24 Sep 2003, Rocky McGaugh wrote: > On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > > > Hi,I am planning on installing a 32 node beowulf cluster for high > > performance bio-informatics applications. As a OS I was planning on > > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > > has advantage in parallel computations. Having no hands-on experience in > > SuSe Linux, I have no idea how to verify this or do a comparision. What do > > you advice as the OS for the cluster? And how to determine this,as there > > are a bunch of unix OSes out there.Thanks. > > > > Sincerely, > > Sarat. > > > > First, I would strongly recommend against using RH9 in a production > environment. It is a short-life-cycle product meant for hobbyist desktops. > More than that, it was a way to beta test some pretty radical OS changes > on the masses before their inclusion into RHEL3. nptl anyone? ;) > Secondly, if you are going to use Opteron there are very good reasons to > use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports > the Opteron while RH9 will only run in 32-bit mode. SLES also has a > 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured > NUMA support for the Opteron. > > If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 > will be out and has the same benefits as SLES. you can test the amd64 beta 2 of taroon right now if you so desire it's on ftp.redhat.com and the mirrors... > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rokrau at yahoo.com Thu Sep 25 14:08:21 2003 From: rokrau at yahoo.com (Roland Krause) Date: Thu, 25 Sep 2003 11:08:21 -0700 (PDT) Subject: MPICH, Fortran and command line arguments Message-ID: <20030925180821.42274.qmail@web40004.mail.yahoo.com> Hi all, I have a Fortran code that I am trying to run under Linux (Redhat-8) using the Intel Fortran compiler 7.1 and mpich-1.2.5.2. I start the code with mpirun -np 2 code.x input.file When the code starts it turns out that only the first instance, i.e. the process with rank 0, sees the correct command line arguments, namely input.file, the next process things the hostname I run on is the first cmdl argument. The MPI_Init manual page has this ugly little blurb about the problem: Command line arguments are not provided to Fortran programs. More precisely, non-standard Fortran routines such as getarg and iargc have undefined behavior in MPI and in this implementation. Does anyone here have experience or a recommendation how I can get the other processes to see the command line arguments? If I consider broadcasting them myself, is it at least guaranteed that the first (rank 0) process will see the command line arguments? How do you guys deal with that situation, short of rewriting the main app in C of course :-) Regards, Roland __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Thu Sep 25 20:02:30 2003 From: bill at math.ucdavis.edu (Bill Broadley) Date: Thu, 25 Sep 2003 17:02:30 -0700 Subject: A question of OS!! In-Reply-To: References: Message-ID: <20030926000229.GA7113@sphere.math.ucdavis.edu> > Secondly, if you are going to use Opteron there are very good reasons to > use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports > the Opteron while RH9 will only run in 32-bit mode. SLES also has a > 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured > NUMA support for the Opteron. Has anyone been benchmarking SLES vs RHEL v3 beta? One of my codes is twice as fast on a dual 1.4 Ghz opterons with RHEL v3 beta as compared to a dual 1.8 Ghz opteron running SLES. The application is an earthquake simulation using MPI, I used the newest MPICH with the same compiled features, and am running 2 processes in both cases. -- Bill Broadley Mathematics UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Thu Sep 25 19:50:36 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 26 Sep 2003 09:50:36 +1000 Subject: A question of OS!! In-Reply-To: References: Message-ID: <200309260950.56059.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 25 Sep 2003 02:39 am, Rocky McGaugh wrote: > If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 > will be out and has the same benefits as SLES. One of the major issues we have is that some of the commercial software our users use is only supported under RH7.3 as of yet. We'd really like to see broader support for other distributions, but it seems that the market is going the other way with people only offering support for for RH (and presumably only for RH Enterprise in the near future) or for SLES. We've had support issues already just because we're running a slightly different version of PBS to the one the vendor supports (leaving me to do a lot of bug fixing of their code, fortunately it's just Perl!). I'd hate to think what would happen if we were running a different distro altogether. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/c39MO2KABBYQAh8RAljcAKCCTaZpytPguoz+qUmF4dAbzvppfwCghObG 7X+lQvvKDJKcVD6bSj15nnY= =x1Jn -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian.dobbins at yale.edu Thu Sep 25 20:19:51 2003 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Thu, 25 Sep 2003 20:19:51 -0400 (EDT) Subject: A question of OS!! In-Reply-To: <20030926000229.GA7113@sphere.math.ucdavis.edu> Message-ID: > Has anyone been benchmarking SLES vs RHEL v3 beta? One of my codes > is twice as fast on a dual 1.4 Ghz opterons with RHEL v3 beta as compared > to a dual 1.8 Ghz opteron running SLES. Yikes.. what kernels are used on these systems by default, and how large is the code? I've been running SuSE 8.2 Pro on my nodes, and have gotten varying performance due to motherboard, BIOS level and kernel. (SuSE 8.2 Pro comes a modified 2.4.19, but I've also run 2.6.0-test5) Also, are the BIOS settings the same? And how are the RAM slots populated? That made a difference, too! (Oh, and I imagine they're both writing to a local disk, or minimal amounts over NFS? That could play a big part, too.. ) I should have some numbers at some point for how much things vary, but at the moment we've been pretty busy on our systems. Any more info on this would be great, though, since I've been looking at the faster chips, too! Cheers, -Brian _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Thu Sep 25 20:45:34 2003 From: bill at math.ucdavis.edu (Bill Broadley) Date: Thu, 25 Sep 2003 17:45:34 -0700 Subject: A question of OS!! In-Reply-To: References: <20030926000229.GA7113@sphere.math.ucdavis.edu> Message-ID: <20030926004534.GQ4151@sphere.math.ucdavis.edu> > Yikes.. what kernels are used on these systems by default, and how large > is the code? I've been running SuSE 8.2 Pro on my nodes, and have gotten Factory default in both cases AFAIK. I don't have access to the SLES system at the moment, but the redhat box is: Linux foo.math.ucdavis.edu 2.4.21-1.1931.2.349.2.2.entsmp #1 SMP Fri Jul 18 00:06:19 EDT 2003 x86_64 x86_64 x86_64 GNU/Linux What relationship that has to the original 2.4.21 I know not. > varying performance due to motherboard, BIOS level and kernel. (SuSE 8.2 > Pro comes a modified 2.4.19, but I've also run 2.6.0-test5) > Also, are the BIOS settings the same? And how are the RAM slots I don't have access to the SLES bios. > populated? That made a difference, too! I'm well aware of the RAM slot issues, and I've experimentally verified that the full bandwidth is available. Basically each cpu will see 2GB/sec or so to main memory, and both see a total of 3GB/sec if both use memory simultaneously. > (Oh, and I imagine they're both writing to a local disk, or minimal > amounts over NFS? That could play a big part, too.. ) Yeah, both local disk, and not much. I didn't notice any difference when I commented out all output. > I should have some numbers at some point for how much things vary, but > at the moment we've been pretty busy on our systems. Any more info on > this would be great, though, since I've been looking at the faster chips, > too! ACK, I never considered that the opterons might be slower in some ways at faster clock speeds. My main suspicious is that MPICH was messaging passing for local nodes in some strange way and triggering some corner case under SLES. I.e. writing an int at a time between CPUs who are fighting over the same page. None of my other MPI benchmarks for latency of bandwidth (at various message sizes) have found any sign of problems. Numerous recompiles of MPICH haven't had any effect either. -- Bill Broadley Mathematics UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Thu Sep 25 10:36:25 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Thu, 25 Sep 2003 16:36:25 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: References: Message-ID: <1064500585.18782.28.camel@revolution.mandrakesoft.com> > For many users they need nothing but the > kernel, a shell, MPI or PVM daemons or clients, and some libraries. A > relatively fancy node might add some scheduling or monitoring daemons. I agree on that point but I'm not sure that everyone wants to know how does it works, and take the time to set up everything. Scientists for example are not Linux Guru. > Who is going to pay $100 PER NODE for that? Nobody sane. Maybe some > businesses who want to build a cluster but don't want to hire somebody > that knows what they are doing to do so (which is a bit oxymoronic, > right? Likely a fairly limited, although undoubtedly high margin, > market). We've made the proof that non Linux awared people can install a cluster for their needs. I know that is not the case everywhere but some like it. [...] > If it takes me a whole day to graze the web and collect all the tool > sources, generally ready to build, directly from their primary website, > why would I (or my employer) pay thousands of dollars for them ever? > Instead of "wasting" time on this list I'd divert some of this energy > into getting and building the sources. Then (the web being what it now > is and not wanting to waste the effort) I'd put the packages in a > toplevel public repository. Suddenly anybody out there can mirror my > efforts. This is really intresting. I've met people who told the contrary ! They said : "I know how does it works, but I don't want to spend my time downloading/configuring clusters. That's your job (Linux Editors) and you do it well. Why should I spent my time doing the thing you do faster & safer". We follow the products, the security updates, the kernel stuff etc.. Some don't have the time to do it. > This is what you are facing. I'm certain that CLIC is very nice, and I > plan to try it out pretty much immediately as RH (which I have used for > years) is in a middling state of turmoil and creating THEIR OWN FUD [..] If you want to try CLIC please wait the next release (october) that will contain the newest features. Keep me in touch about this tests, I will give you a hand if needed. > What is a reasonable price per cluster then? Something in the <$100 > range. And for that money, don't sell them CLIC -- as you say, CLIC is > free! Sell them something with clear added value that costs you very > little to provide. Sell whole universities this sort of toplevel access > to a patch stream for $1000, whole governments for $100,000. If you can > sell 100 universities or 1000 clusters or one government, you can just > about pay for the one human likely to be required to run it, and > thereafter you're making money. I've missed to explain clearer that point. We are offering a price model based on the number of CPU because some software we are including do it. This model works fine with small clusters in laboratories/small companies. But we are now working on a site licence. Which allow an university/company to use the product for a while for a fixed price that we must negociate. > My own personal suggestion would be a subscription of sorts to a > private (authenticated), high bandwidth update server from which the > user could rsync a local mirror on their cluster server. Drop your > maintenance stuff in there and have it automatically propagate. Sure, > sell consulting and service on the side for market price -- some people > will need it and buy it. Keep it in the range where people can see that > they are getting their money's worth and you make a fair profit. This already exist throught the MandrakeClub. > The distribution vendors that survive in the long run will be the ones > content with EARLY Microsoft income streams -- $30 per PC per year, > maybe. Or $10. For that, they have to provide real value, but the > market is there for it. Look at (to point to an interesting metaphor) > Harry Potter. The book sells for order $10, yet it has made its author > a billionaire and its publisher likely 10 times that. No, this won't > create a huge empire of world dominating high margin arm twisted sales, > but its a living, and a comfortable one at that. Even wealth. It just > is EARNED wealth. Model is nice but I can figure how to sell such number of clusters :D > I'd put a paypal link right up on your CLIC download site, and suggest > that everyone who downloads it contribute/pay $50 "per cluster" and see > how many people do so. People are free to do it if they like our product ! They also able to do it throught the MandrakeClub.. > >Maybe nobody, but you might be surprised. Right > now I pay transgaming.com regular small money just to enable GAMES in my > household. I'll bet they are making money, and frankly it is a damn > sight harder to get Window software to run under linux than it is to > assemble a cluster package that meets a whole lot of needs. One I can > do myself in my spare time, the other I wouldn't touch with a ten foot > pole. If they charge small money, so can (and so should) you. I was one of those :D > Look at their business model. This is the linux model of the future, I > believe. GPL (and available via sourceforge) but who wants to go > through a sourceforge build when you can get a ready to run build and > updates for a few dollars? We remain GPL and available. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From erwan at mandrakesoft.com Thu Sep 25 11:30:05 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Thu, 25 Sep 2003 17:30:05 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: References: Message-ID: <1064503805.18783.70.camel@revolution.mandrakesoft.com> > > Scientists for example are not Linux Guru. > Ummm, what makes you say that? Sorry, I meant that not all the Scientists are able to manage all the configuration needed by a cluster. We give Scientists an easiest way to setup things that could take them longer. > p.s. Oh, you mean ALL scientists aren't linux bozos (gurus seems like a > pretentious word:-). Sure, but scientists who work on massively > parallel computers are fairly likely to be, or they are likely to hire > someone who is. If they are already paying that person large sums of > money, why pay you as well? You can pay someone to manage more clusters when he's working on powerfull tools rather paying one doing everything by hand. If everyone should compile/configure/understand all the technologies included in a product, some could be frighten of this ! > >If they don't have such a person, will they really be able to manage a cluster without one? Depending on what you call "manage" but for usual tasks... yes ! > >Even setting up a TCP/IP switched ethernet network requires expertise, as does adding > accounts, debugging problems. For adding accounts, you just need to know the username/password and the logical parition you want him to be. Not really complicated ! > > Plug and play solutions are indeed very expensive, because they have to be nearly flawless to work in an > environment without support. Our solution is to build autoconfigured services. It works really fine...I've integrated this product in a big architecture.. The cluster is working fine ! -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 25 06:57:20 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 25 Sep 2003 06:57:20 -0400 (EDT) Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064500585.18782.28.camel@revolution.mandrakesoft.com> Message-ID: dOn Thu, 25 Sep 2003, Erwan Velu wrote: > > For many users they need nothing but the > > kernel, a shell, MPI or PVM daemons or clients, and some libraries. A > > relatively fancy node might add some scheduling or monitoring daemons. > I agree on that point but I'm not sure that everyone wants to know how does it works, > and take the time to set up everything. Scientists for example are not > Linux Guru. Ummm, what makes you say that? rgb (Ph.D. theoretical physics, 1982) p.s. Oh, you mean ALL scientists aren't linux bozos (gurus seems like a pretentious word:-). Sure, but scientists who work on massively parallel computers are fairly likely to be, or they are likely to hire someone who is. If they are already paying that person large sums of money, why pay you as well? If they don't have such a person, will they really be able to manage a cluster without one? Even setting up a TCP/IP switched ethernet network requires expertise, as does adding accounts, debugging problems. Plug and play solutions are indeed very expensive, because they have to be nearly flawless to work in an environment without support. rgb (again, back in beowulfish guise) -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 25 09:43:04 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 25 Sep 2003 09:43:04 -0400 (EDT) Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> Message-ID: On Thu, 25 Sep 2003 erwan.velu at free.fr wrote: > All the fedora thread is really intresting... Why people should pay for > such clustering oriented distro ? Why CLIC is free whereas you must pay for > MDKC with a per-CPU licence ? > > CLIC was a project co-financed by the french government, so it MandrakeSoft and > its partners were being half payed for this work. It was obvious that we are > making a GPL product available for download. CLIC is a real technological > success and its customers like it for this. > > Today we are facing the following problem : how to continue such project ? How > can MandrakeSoft could make some revenues on this king of product ? What do > people using CLIC gives at MandrakeSoft in return of its work ? Some buy some > consulting, but some do nothing... I think this is the basic question. In part I think that the answer is for you to come up with a REASONABLE pricing model. If you follow this list (and it sounds like you do) then you know that when people are building a cluster, whether it has four nodes or four hundred nodes, prices that scale per node (period) are crazy. A cluster node has always been "a CPU" in the parallel supercomputer model, and the relative LACK of marginal cost in per node installation/maintenance of linux has been a major reason it has been used. To install and manage and operate an entire cluster, most people actually "work on" one or two server/head nodes; the rest of the cluster installs a really, really boring operating system that is little more (functionally) than a task loader. In the Scyld/bproc extreme this is literal; in clusters based on higher order distros more of a metaphor, but the point is that most of the cluster nodes don't require most of the gorp that is in a standard distro anyway. For many users they need nothing but the kernel, a shell, MPI or PVM daemons or clients, and some libraries. A relatively fancy node might add some scheduling or monitoring daemons. Who is going to pay $100 PER NODE for that? Nobody sane. Maybe some businesses who want to build a cluster but don't want to hire somebody that knows what they are doing to do so (which is a bit oxymoronic, right? Likely a fairly limited, although undoubtedly high margin, market). So prices have to reflect the scaling of the actual work, acknowledging that the very scalability of linux means that cluster managers may WELL be relatively underutilized once a cluster is installed and operating smoothly, so that they DO have opportunity cost time to do it themselves "for free" as far as marginal cost to the enterprise is concerned. If it takes me a whole day to graze the web and collect all the tool sources, generally ready to build, directly from their primary website, why would I (or my employer) pay thousands of dollars for them ever? Instead of "wasting" time on this list I'd divert some of this energy into getting and building the sources. Then (the web being what it now is and not wanting to waste the effort) I'd put the packages in a toplevel public repository. Suddenly anybody out there can mirror my efforts. This is what you are facing. I'm certain that CLIC is very nice, and I plan to try it out pretty much immediately as RH (which I have used for years) is in a middling state of turmoil and creating THEIR OWN FUD (an amazing business strategy, particularly at this time when SCO and MS and even Sun are doing their best in the FUD department without help:-). What is a reasonable price per cluster then? Something in the <$100 range. And for that money, don't sell them CLIC -- as you say, CLIC is free! Sell them something with clear added value that costs you very little to provide. Sell whole universities this sort of toplevel access to a patch stream for $1000, whole governments for $100,000. If you can sell 100 universities or 1000 clusters or one government, you can just about pay for the one human likely to be required to run it, and thereafter you're making money. My own personal suggestion would be a subscription of sorts to a private (authenticated), high bandwidth update server from which the user could rsync a local mirror on their cluster server. Drop your maintenance stuff in there and have it automatically propagate. Sure, sell consulting and service on the side for market price -- some people will need it and buy it. Keep it in the range where people can see that they are getting their money's worth and you make a fair profit. Most cluster managers, I think, value the distributions and would (as Bob said) view it as a Bad Thing if distribution vendors went away. We WANT you guys to make money. I personally pay Red Hat fifty bucks every few years just to contribute fair value back to them for what I receive (this is for my personal home systems). On the other hand, NOBODY WILL EVER MAKE MICROSOFTISH SORTS OF MONEY AGAIN! So folks need to get over it. It ain't happening. In a few years not even Microsoft will make Microsoftish money -- their empire is being forcibly ripped apart by market forces and entire governments, deliberately. The distribution vendors that survive in the long run will be the ones content with EARLY Microsoft income streams -- $30 per PC per year, maybe. Or $10. For that, they have to provide real value, but the market is there for it. Look at (to point to an interesting metaphor) Harry Potter. The book sells for order $10, yet it has made its author a billionaire and its publisher likely 10 times that. No, this won't create a huge empire of world dominating high margin arm twisted sales, but its a living, and a comfortable one at that. Even wealth. It just is EARNED wealth. I'd put a paypal link right up on your CLIC download site, and suggest that everyone who downloads it contribute/pay $50 "per cluster" and see how many people do so. Maybe nobody, but you might be surprised. Right now I pay transgaming.com regular small money just to enable GAMES in my household. I'll bet they are making money, and frankly it is a damn sight harder to get Window software to run under linux than it is to assemble a cluster package that meets a whole lot of needs. One I can do myself in my spare time, the other I wouldn't touch with a ten foot pole. If they charge small money, so can (and so should) you. Look at their business model. This is the linux model of the future, I believe. GPL (and available via sourceforge) but who wants to go through a sourceforge build when you can get a ready to run build and updates for a few dollars? rgb > > CLIC is finishing at Decembrer 2003, so should we stop CLIC ? How the guys are > working hard on such product could be payed ? We decided to launch a new > product called MandrakeClustering based on the CLIC success. > > We have rewritten all the backend in perl, added a lot of features. We want > this technology to survive. Now MDKC is a real product, not a research project > as CLIC. MandrakeSoft now sells this product whereas we were just selling > consulting on CLIC. This model now change the rules. Products like maui can't > be resell without permissions. There is a per-cpu fee for such product, so we > are now applying a per-cpu pricing on our product. > > The price of MDKC is really reasonable ! > For 8CPUS under x86 or AMD64 it costs 2000?, > until 16 -> 3600? etc.. > Please make the ratio between the hardware price and the software price. > > What is the future of CLIC ? Is will depend of the results of > MandrakeClustering. If it works fine, we will continue CLIC because the > clustering team could be payed for such work. This is what we are doing > today. The technology of MDKC is currently in integration in CLIC. So CLIC will > contain a part of MDKC a few after the release. > > The last point is you can't compare RedHat Enterprise with MandrakeClustering ! > MandrakeClustering is a real oriented clustering distro not just a > MandrakeLinux + some few packages. > MDKC is integrating the Linux Distro + configuration tools + deployment tools + > clustering tools + clustering mpi stacks. > > Everything is auto configured, you don't need to be a Linux expert to install a > cluster. > My best example ? How to integrate new nodes ? CLICK on a node, CLICK on > duplicate, tell the number of nodes you want to duplicate, the device you want > to duplicate. Then just power on your nodes ! Nothing more to do ! > > You just need a few hours to setup a full cluster ! DNS, DHCP, PXE, NIS, > AUTOFS, LAM/MPI, MPICH, PVM, TFTP, etc.. Everything is built automaticaly ! > Does RH do it ? > > You can manage all the rpms on your cluster using "urpmi" (could be compared to > apt) but using parallel tools ! How to install a tool on a cluster ? > urpmi --parallel cluster > Dependencies are automaticaly installed like apt. > Does RH do it ? > > You can install security/product updates from the server using a simple command: > urpmi --parallel cluster --auto-select --auto > Updates are downloaded from Internet then parallely send to the nodes then each > nodes update itslef following its own needs ! Who can do that today ? > > Does RH allow you to duplicate 200 nodes in 3 minutes ~500 MBits ? > No, you need to integrate yourself other tools you must installed and configure. > > MDKC & CLIC are some real clustering oriented distro, everything has been > thought for the easyness of use.. > > I don't know if this king of approach could match all the clustering needs, but > I must say that people who did really test this kind of distribution likes it. > I'm not saying that other products are poor, we just have another approach that > our Linux editor postion allow us to give. > > I hope this mail could help everyone to understand the work of the CLIC & MDKC > team. > > Linuxely yours, > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Thu Sep 25 20:57:25 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 26 Sep 2003 10:57:25 +1000 Subject: A question of OS!! In-Reply-To: <20030926004534.GQ4151@sphere.math.ucdavis.edu> References: <20030926000229.GA7113@sphere.math.ucdavis.edu> <20030926004534.GQ4151@sphere.math.ucdavis.edu> Message-ID: <200309261057.26445.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 26 Sep 2003 10:45 am, Bill Broadley wrote: > My main suspicious is that MPICH was messaging > passing for local nodes in some strange way and triggering some corner > case under SLES. I.e. writing an int at a time between CPUs who are > fighting over the same page. The RH kernel may have additional/better patches or config ? Your code benefits immensely from NPTL ? - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD4DBQE/c471O2KABBYQAh8RAh5JAJi5SEVZKG+lveIth0vifzXmNnv4AJwJ5OBv QaEdIyiMtV3iMq9ZTe8LMw== =Endv -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Thu Sep 25 11:52:00 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Thu, 25 Sep 2003 17:52:00 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <3F730945.6060606@scali.com> References: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> <3F730945.6060606@scali.com> Message-ID: <1064505120.18785.94.camel@revolution.mandrakesoft.com> > MDKC is still GPL'ed isn't it ? Atleast it contains a number of GPL products. Why isn't it possible to download MDKC and redistribute it as you can do with other distributions ? MDKC is still made by a GPL'ed product and also contains some BSD like licenses and some other licences close to but with some restriction about reselling. All our work is available trought cooker and our CVS and obsviously we send patches to the authors. > Isn't that GPL violation ? Of course not. > >Although RedHat Enterprise Linux costs $$ if you want the ISO's you can still compile your own distribution from the source they provide. You can, > Also anyone who actually paid to get the RHEL CD's can (AFAIK) make copies and give to friends without problems, correct ? Correct, > Is this true with MDKC too ? This it true but you can't sell it because some companies must be payed when you sell it. But people must understand that if they will not contribute the product will never be released anymore. That's an individual choice. > Same thing with Scyld really, what prohibits a proud owner of "Scyld Beowulf Professional Edition 28cz Series" to make a copy available for download ? Does it contain products > which isn't GPL'ed, which is what you pay $375 per processor for ? The average price is closer than 150-200? rather 375?. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sp at scali.com Thu Sep 25 11:27:01 2003 From: sp at scali.com (Steffen Persvold) Date: Thu, 25 Sep 2003 17:27:01 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> References: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> Message-ID: <3F730945.6060606@scali.com> erwan.velu at free.fr wrote: [] > > I don't know if this king of approach could match all the clustering needs, but > I must say that people who did really test this kind of distribution likes it. > I'm not saying that other products are poor, we just have another approach that > our Linux editor postion allow us to give. > > I hope this mail could help everyone to understand the work of the CLIC & MDKC > team. Erwan, MDKC is still GPL'ed isn't it ? Atleast it contains a number of GPL products. Why isn't it possible to download MDKC and redistribute it as you can do with other distributions ? Isn't that GPL violation ? Although RedHat Enterprise Linux costs $$ if you want the ISO's you can still compile your own distribution from the source they provide. Also anyone who actually paid to get the RHEL CD's can (AFAIK) make copies and give to friends without problems, correct ? Is this true with MDKC too ? Same thing with Scyld really, what prohibits a proud owner of "Scyld Beowulf Professional Edition 28cz Series" to make a copy available for download ? Does it contain products which isn't GPL'ed, which is what you pay $375 per processor for ? Regards, Steffen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 26 14:00:06 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 26 Sep 2003 11:00:06 -0700 Subject: Redhat Fedora In-Reply-To: References: <20030924194442.H3752@www2> Message-ID: <20030926180006.GB2187@greglaptop.internal.keyresearch.com> On Wed, Sep 24, 2003 at 08:54:46PM -0400, Robert G. Brown wrote: > The interesting question is why anybody in their right mind would pay > for any sort of GPL-based distribution that scales per number of > processors. Especially pay numbers like these. Hell, one might as well > be running Windows. This is a really unfair and misguided comment. You're paying for support. If you don't want to pay that much, don't. Given that most of the code is GPLed, you are free to build a GPL-only subset of any distro and give it away. Comparing it to Windozs is the unfair and misguided part. Them's fightin' words! > The same > issues exist with Myrinet and SCI -- I'll be there are dozens of > clusters installed on plain old ethernet for each one of the clusters > installed on high end networks One would hope that the choice was made with price/performance in mind, instead of the incorrect reasoning of "Myrinet is expensive." > (which, frankly, are worth a lot more > than high end/expensive "distributions" since Red Hat cannot alter the > fundamentally limiting computation/communications ratio at a given task > scale, but a faster network interface can!). There's more to a cluster than the computation. RedHat would claim that their added value is in administration. It's pointless (unfair, misguided, insert more terms here) to complain that RedHat doesn't improve computation. Really, Robert, this rant is not up to your usual quality. Sheesh. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Fri Sep 26 14:44:10 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Fri, 26 Sep 2003 20:44:10 +0200 Subject: Redhat Fedora References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> Message-ID: <3F7488FA.2000901@moene.indiv.nluug.nl> Greg Lindahl wrote: > It's an interesting question. RedHat's new advice on the matter is > "buy a copy of Advanced Workstation for every cluster node." > I suspect that we'll see: > > More use of hobbist distros like Debian Volunteer .NE. hobbyist. -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 26 13:53:12 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 26 Sep 2003 10:53:12 -0700 Subject: Redhat Fedora In-Reply-To: <20030924194442.H3752@www2> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> Message-ID: <20030926175312.GA2187@greglaptop.internal.keyresearch.com> On Wed, Sep 24, 2003 at 07:44:42PM -0400, Bob Drzyzgula wrote: > Actually, I thought that in fact "we" tended to do a lot of > this -- cf. Rocks, Clustermatic, Warewulf -- it's just that > these never seem to capture the market in the same way that > Red Hat always was. Are any of these really "distributions", or are they "a minimal set of stuff overlaid on RedHat?" It makes sense to use a free and complete conventional distro to build a cluster distro -- but now the free and complete distro is splitting into the less-reliable and less-fixed distro (consumer) and the not-exactly-up-to-date distro (enterprise), which changes what we can do cheaply. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bob at drzyzgula.org Fri Sep 26 15:43:51 2003 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Fri, 26 Sep 2003 15:43:51 -0400 Subject: Redhat Fedora In-Reply-To: <20030926175312.GA2187@greglaptop.internal.keyresearch.com> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> <20030926175312.GA2187@greglaptop.internal.keyresearch.com> Message-ID: <20030926154351.M3752@www2> On Fri, Sep 26, 2003 at 10:53:12AM -0700, Greg Lindahl wrote: > > On Wed, Sep 24, 2003 at 07:44:42PM -0400, Bob Drzyzgula wrote: > > > Actually, I thought that in fact "we" tended to do a lot of > > this -- cf. Rocks, Clustermatic, Warewulf -- it's just that > > these never seem to capture the market in the same way that > > Red Hat always was. > > Are any of these really "distributions", or are they "a minimal set of > stuff overlaid on RedHat?" It makes sense to use a free and complete > conventional distro to build a cluster distro -- but now the free and > complete distro is splitting into the less-reliable and less-fixed > distro (consumer) and the not-exactly-up-to-date distro (enterprise), > which changes what we can do cheaply. Yes, on looking closer I see that you're right about the layering thing. I saw ISOs on those sites and hadn't looked more deeply. You're also right that the way out of this problem isn't clear. I suppose that if Fedora works out better than most of us are probably expecting, then there'd be hope there. It also occurs to me that it may make sense for the cluster community to consider something like a Cluster-Fedora subproject; perhaps we could come up with something that wasn't a cluster-somethingelsestartingwithf (sorry). Possibly Debian would be a better base in the long term, not that I'm particularly fond of Debian. Possibly, given that, as Erwan said, the CLIC contract is up in December 2003, perhaps it would make more sense for the community to pick that up as a starting point. But I suppose this is a bridge we'll have to cross when we come to it, especially since the bridge doesn't even seem to be fully built as yet. --Bob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Sep 26 09:45:23 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 26 Sep 2003 09:45:23 -0400 (EDT) Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064503805.18783.70.camel@revolution.mandrakesoft.com> Message-ID: On Thu, 25 Sep 2003, Erwan Velu wrote: > > > Scientists for example are not Linux Guru. > > Ummm, what makes you say that? > Sorry, I meant that not all the Scientists are able to manage all the configuration needed by a cluster. > We give Scientists an easiest way to setup things that could take them > longer. > > > p.s. Oh, you mean ALL scientists aren't linux bozos (gurus seems like a > > pretentious word:-). Sure, but scientists who work on massively > > parallel computers are fairly likely to be, or they are likely to hire > > someone who is. If they are already paying that person large sums of > > money, why pay you as well? > You can pay someone to manage more clusters when he's working on > powerfull tools rather paying one doing everything by hand. > If everyone should compile/configure/understand all the technologies > included in a product, some could be frighten of this ! I know, I was just teasing you a bit -- note smileys (:-). However, from a purely practical point of view, we find that the thing that limits the number of systems a manager can take care of for pretty much any of the distributions these days is hardware reliability and sheer number of systems. To put it another way, give me an unlimited supply of flunkies to install and repair hardware (only) and I'll cheerily take care of a thousand node cluster all by myself, as far as the software installation and management is concerned. At this point PXE, a variety of rapid (scripted or image based) install methods, and yum for rpm post-install maintenance mean that installation and maintenance of an almost indefinite number of machines is primarily a matter of taking care of the repository(s) from which they are installed and maintained, and this has to be done the same for a cluster of eight nodes or eight hundred nodes. Once the base repository is set up and maintained, installation is turning on a node, reinstallation is rebooting a node, software maintenance is doing nothing (the node updates itself nightly) or doing a rare cluster-push to update in real time. These things do take time, but the BIGGEST time is setting up the repository and configuring a reasonable node image in the first place (neither of which are terribly hard, as one can rsync a repository mirror from many existing repositories and with a repository in hand a node image is, fundamentally, a list of rpm's that a tool like yum can be used to consistently install on top of a very minimal base image). On a successful cluster design the average non-hardware per-node FTE effort spent on the nodes themselves should be order of minutes per year. Again, this doesn't mean that your product packaging is without value -- I'm sure that it is engineered to help neophytes who are unlikely to achieve that level of efficiency initially get there quickly, and there is indeed value in providing people with a repository image to mirror including regular updates of the important core packages (as I said, this is what I think you should sell). It's just that ANY of these packagings compete with opportunity cost time available in many environments, alternative more or less prebuilt or easy to build cluster packages (oscar rocks scyld clustermatic...), and BECAUSE it is a big toolset so that one size fits all, contains a lot of tools that many cluster people will never need or use (just like a swiss army knife). Some people just want to whittle wood and clean fish and don't NEED a magnifying glass on their pocket knife! They are going to be asking themselves why they are "paying" for all these things when they don't use but three of them they could build themselves in an afternoon (or mirror prebuilt from various public repositories). This is what I think will ultimately limit your price point and needs to shape your marketing strategy. Your choice is ultimately going to be fewer high margin sales or more low margin sales (I'm pretty safe there, supply and demand law and all that:-). This is true for all the linux distributions. Or you can try some sort of mixed strategy (as was suggested earlier) and sell at all price points trying to leverage additional margin on the basis of added value and willingness to pay and service/consulting/support. In my opinion (for what it is worth) it would be a capital mistake for a distribution to seek to armtwist high margins in the linux community. As in the last mistake a lot of distributions may ever make. The problem is that the reason that there ARE distributions at all is largely because of the difficulty, once upon a time, of an ordinary human obtaining all the components required or desired to assemble a usable system. The distribution companies "sold" media where the whole shebang was right there, packaged to be "easy to install", and (as time passed) with package management systems and gradually improving installation and administrative interfaces. However, the interesting thing about software and the evolution of hardware and the internet is that the once a GPL package exists, it is available forever and eventually gets to be feature stable and readily available in a stable, functional form to pretty much anybody. New hardware (like PXE) has changed the installation equation. And the internet is now supported by new tools such as rsync, with massive amounts of storage routinely available. If I can afford a 150 GB RAID 5 system to serve my home LAN (and I just built one for about $800) then storing an utterly exhaustive linux repository isn't a big deal anymore. These things CAN change the way linux distribution operates altogether. They ARE changing the way linux distribution operates altogether. Nobody buys distribution CD's, they mirror. In professional environments, booting from floppy has all but disappeared. Packages are designed a priori to permit simple rebuilds and dependency resolution. I could >>really easily see<< an environment in five years where linux distribution is >>nothing but a network of repositories<< driven by tools like yum, augmented so that the rpm's are rebuilt as necessary in real time. Install a very primitive, pure kernel+gnu base, then yum-build from SOURCE repositories into a known final state. Right now there is little incentive for this to be built, at least not in a hurry. However, high margin price points would be a great incentive. Already half the systems people I know are wandering around moaning (and irritated), and these are the very people to do something about it if they get irritated enough. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Fri Sep 26 11:25:07 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Fri, 26 Sep 2003 17:25:07 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: References: Message-ID: <1064589906.3078.282.camel@revolution.mandrakesoft.com> [...] > Again, this doesn't mean that your product packaging is without value -- > I'm sure that it is engineered to help neophytes who are unlikely to > achieve that level of efficiency initially get there quickly, and there > is indeed value in providing people with a repository image to mirror > including regular updates of the important core packages (as I said, > this is what I think you should sell). [...] You are the kind of users which are able to setup a full cluster with a Linux From Scratch with a repository and automated building tools. Our product is aimed for people who don't have the time or don't want to setup such architecture. Obsviously we make cuts in our prices for educational research labs. Best regards, -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From adm35 at georgetown.edu Fri Sep 26 22:03:26 2003 From: adm35 at georgetown.edu (adm35 at georgetown.edu) Date: Fri, 26 Sep 2003 22:03:26 -0400 Subject: Redhat Fedora Message-ID: <24a6532445ee.2445ee24a653@georgetown.edu> It just seems to me that the entire question is less than relevant for so many of us. In many cases, we're *compelled* to use one particular distro over all others, because the researchers we support are using binary versions of applications that require a particular distro, which is frequently RedHat. I actually discovered one today that REQUIRES RedHat 8 or better!! You can all imagine that I was less than amused. We generally use OSCAR on RedHat, but not for any particular political reason. It works, and all of our applications run on it. If we had the luxury of having ready access to the source code for all of our applications, we might look elsewhere....or we might not. Some of you have the luxury of having all your programs created locally. I think many, perhaps most, of the rest of us do not. As long as I remain at the mercy of software vendors for some percentage of our computational applications, I'll probably stay with RedHat, questionable policies or not. Arnie Miles Systems Administrator: Advanced Research Computing Adjunct Faculty: Computer Science 202.687.9379 168 Reiss Science Building http://www.georgetown.edu/users/adm35 http://www.guppi.arc.georgetown.edu ----- Original Message ----- From: Bob Drzyzgula Date: Friday, September 26, 2003 3:43 pm Subject: Re: Redhat Fedora > On Fri, Sep 26, 2003 at 10:53:12AM -0700, Greg Lindahl wrote: > > > > On Wed, Sep 24, 2003 at 07:44:42PM -0400, Bob Drzyzgula wrote: > > > > > Actually, I thought that in fact "we" tended to do a lot of > > > this -- cf. Rocks, Clustermatic, Warewulf -- it's just that > > > these never seem to capture the market in the same way that > > > Red Hat always was. > > > > Are any of these really "distributions", or are they "a minimal > set of > > stuff overlaid on RedHat?" It makes sense to use a free and complete > > conventional distro to build a cluster distro -- but now the free > and> complete distro is splitting into the less-reliable and less- > fixed> distro (consumer) and the not-exactly-up-to-date distro > (enterprise),> which changes what we can do cheaply. > > > Yes, on looking closer I see that you're right about the > layering thing. I saw ISOs on those sites and hadn't looked > more deeply. > > You're also right that the way out of this problem > isn't clear. I suppose that if Fedora works out better > than most of us are probably expecting, then there'd be > hope there. It also occurs to me that it may make sense > for the cluster community to consider something like a > Cluster-Fedora subproject; perhaps we could come up with > something that wasn't a cluster-somethingelsestartingwithf > (sorry). Possibly Debian would be a better base in the > long term, not that I'm particularly fond of Debian. > > Possibly, given that, as Erwan said, the CLIC contract > is up in December 2003, perhaps it would make more sense > for the community to pick that up as a starting point. > > But I suppose this is a bridge we'll have to cross when > we come to it, especially since the bridge doesn't even > seem to be fully built as yet. > > --Bob > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Sat Sep 27 03:11:42 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Sat, 27 Sep 2003 00:11:42 -0700 Subject: Redhat Fedora In-Reply-To: <24a6532445ee.2445ee24a653@georgetown.edu> References: <24a6532445ee.2445ee24a653@georgetown.edu> Message-ID: <20030927071142.GA1838@greglaptop.greghome.keyresearch.com> On Fri, Sep 26, 2003 at 10:03:26PM -0400, adm35 at georgetown.edu wrote: > In many cases, we're *compelled* to use one particular > distro over all others, because the researchers we support are using > binary versions of applications that require a particular distro, which > is frequently RedHat. I actually discovered one today that REQUIRES > RedHat 8 or better!! You can all imagine that I was less than amused. In the future, RedHat is trying to ensure that these apps require RedHat Enterprise, not the consumer version. So "we"'re going to have to deal with that, one way or another. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel.pfenniger at obs.unige.ch Sat Sep 27 04:44:42 2003 From: daniel.pfenniger at obs.unige.ch (Daniel Pfenniger) Date: Sat, 27 Sep 2003 10:44:42 +0200 Subject: Redhat Fedora In-Reply-To: <24a6532445ee.2445ee24a653@georgetown.edu> References: <24a6532445ee.2445ee24a653@georgetown.edu> Message-ID: <3F754DFA.707@obs.unige.ch> adm35 at georgetown.edu wrote: It just seems to me that the entire question is less than relevant for so many of us. In many cases, we're *compelled* to use one particular distro over all others, because the researchers we support are using binary versions of applications that require a particular distro, which is frequently RedHat. I actually discovered one today that REQUIRES RedHat 8 or better!! You can all imagine that I was less than amused. I have seen the problem for the driver of InfiniBand cards made by Mellanox and distributed by various companies. The driver source code is of course not available, so the Infiniband companies provide binary drivers for certain RedHat and Suse kernels. Because InfiniBand is recent on the market, it is likely that new clusters aimed at high performance computing use freshly made components, so the kernels are rapidly not compatible with both the binary drivers and the recent kernels. A solution that at least one company was offering is to custom compile the driver with the particular customer kernel. The customer sends the kernel source with the custom config file and the company sends back the compatitble binary driver. Such solutions seem to me the most convenient for having coexisting open and closed source software. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Sat Sep 27 15:57:24 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Sat, 27 Sep 2003 21:57:24 +0200 Subject: Redhat Fedora References: <24a6532445ee.2445ee24a653@georgetown.edu> Message-ID: <3F75EBA4.5040001@moene.indiv.nluug.nl> adm35 at georgetown.edu wrote: > It just seems to me that the entire question is less than relevant for so > many of us. In many cases, we're *compelled* to use one particular > distro over all others, because the researchers we support are using > binary versions of applications that require a particular distro, which > is frequently RedHat. In that case, why use Free Software - you could just as well shell out the bucks and go for a proprietary solution. Note that Free in Free Software stands for freedom. If you just want to use gratis software, be my guest, but do not complain. -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amacater at galactic.demon.co.uk Sun Sep 28 16:23:57 2003 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Sun, 28 Sep 2003 20:23:57 +0000 Subject: Redhat Fedora In-Reply-To: <3F75EBA4.5040001@moene.indiv.nluug.nl> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> Message-ID: <20030928202357.GA2644@galactic.demon.co.uk> On Sat, Sep 27, 2003 at 09:57:24PM +0200, Toon Moene wrote: > adm35 at georgetown.edu wrote: > > >It just seems to me that the entire question is less than relevant for so > >many of us. In many cases, we're *compelled* to use one particular > >distro over all others, because the researchers we support are using > >binary versions of applications that require a particular distro, which > >is frequently RedHat. > > In that case, why use Free Software - you could just as well shell out > the bucks and go for a proprietary solution. > Indeed: by demanding Red Hat / SuSE you may already be using a proprietary solution (since you can't redistribute their Linux versions). If your vendor only supplies binaries which are vendor/version locked - strongly consider changing your vendor. If your vendor insists they need to supply binaries: a.) Get them to take the risk of keeping up to date with Fedora and hold them to it. b.) Get them to port the code to any LSB compliant distribution / remove kernel version specific stuff. c.) Ask them to supply you with the programming hooks necessary to use the binaries if your vendor disappears / pulls out of the Linux business. d.) If in Europe at least - advise them that you will feel free to reverse engineer their binaries to ensure onward compatibility / interoperability with other software. Remember folks: This is Linux (GNU/Linux for the pedantic :) ) - we have access to the underlying code and will continue so to do. If need be, as a community, we can kludge packages together to ensure binary compatibility for obsolete code [libc4 is still in Debian for just this reason - obsolete commercial packages depend on it] / write wrapper scripts / whatever. I have a slight residual personal interest in this: a long time ago, I wanted to port the original Extreme Linux stuff to work on a Debian system and it was hard because it was all .rpm based and predicated RH. Eventually, I managed to package LAM (before passing it off onto a much better maintainer - Hi Camm {B-) ) but gave up on several of the smaller bits because (a) Some of them were obviously RH specific / (b) Some of them appeared to be commercial software. At this stage, some years later, I can't recall exact specifics but it was enough to put me off "based on specific distribution" type hacks. Can I commend Debian to the list for serious consideration for future projects / clusters. a) It runs on virtually anything from PDA's to Alphas / Sparcs / IBM big iron / ia64 / Opteron b) Debian main is free and open and all source is available, modifiable and freely distributable. c) There are no licence costs per node. Just my $0.02 US (saving $149.98 or so per node ...) Andy _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amacater at galactic.demon.co.uk Sun Sep 28 16:23:57 2003 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Sun, 28 Sep 2003 20:23:57 +0000 Subject: Redhat Fedora In-Reply-To: <3F75EBA4.5040001@moene.indiv.nluug.nl> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> Message-ID: <20030928202357.GA2644@galactic.demon.co.uk> On Sat, Sep 27, 2003 at 09:57:24PM +0200, Toon Moene wrote: > adm35 at georgetown.edu wrote: > > >It just seems to me that the entire question is less than relevant for so > >many of us. In many cases, we're *compelled* to use one particular > >distro over all others, because the researchers we support are using > >binary versions of applications that require a particular distro, which > >is frequently RedHat. > > In that case, why use Free Software - you could just as well shell out > the bucks and go for a proprietary solution. > Indeed: by demanding Red Hat / SuSE you may already be using a proprietary solution (since you can't redistribute their Linux versions). If your vendor only supplies binaries which are vendor/version locked - strongly consider changing your vendor. If your vendor insists they need to supply binaries: a.) Get them to take the risk of keeping up to date with Fedora and hold them to it. b.) Get them to port the code to any LSB compliant distribution / remove kernel version specific stuff. c.) Ask them to supply you with the programming hooks necessary to use the binaries if your vendor disappears / pulls out of the Linux business. d.) If in Europe at least - advise them that you will feel free to reverse engineer their binaries to ensure onward compatibility / interoperability with other software. Remember folks: This is Linux (GNU/Linux for the pedantic :) ) - we have access to the underlying code and will continue so to do. If need be, as a community, we can kludge packages together to ensure binary compatibility for obsolete code [libc4 is still in Debian for just this reason - obsolete commercial packages depend on it] / write wrapper scripts / whatever. I have a slight residual personal interest in this: a long time ago, I wanted to port the original Extreme Linux stuff to work on a Debian system and it was hard because it was all .rpm based and predicated RH. Eventually, I managed to package LAM (before passing it off onto a much better maintainer - Hi Camm {B-) ) but gave up on several of the smaller bits because (a) Some of them were obviously RH specific / (b) Some of them appeared to be commercial software. At this stage, some years later, I can't recall exact specifics but it was enough to put me off "based on specific distribution" type hacks. Can I commend Debian to the list for serious consideration for future projects / clusters. a) It runs on virtually anything from PDA's to Alphas / Sparcs / IBM big iron / ia64 / Opteron b) Debian main is free and open and all source is available, modifiable and freely distributable. c) There are no licence costs per node. Just my $0.02 US (saving $149.98 or so per node ...) Andy _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From thornton at yoyoweb.com Mon Sep 29 00:07:52 2003 From: thornton at yoyoweb.com (Thornton Prime) Date: Sun, 28 Sep 2003 21:07:52 -0700 Subject: Redhat Fedora In-Reply-To: <20030928202357.GA2644@galactic.demon.co.uk> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> <20030928202357.GA2644@galactic.demon.co.uk> Message-ID: <1064808472.16662.12.camel@localhost.localdomain> > Indeed: by demanding Red Hat / SuSE you may already be using a > proprietary solution (since you can't redistribute their Linux > versions). Why do you say this? To my knowledge, the only restrictions either of these distributions have (or can have) on re-distribution are trademark restrictions involving the use of their distribution name or logo. You can't say that a re-distribution of Red Hat is Red Hat, but the code is completely open to distribution -- in both source and binary form (you are required by GPL to make the source available to whomever you distribute to). I do agree that we should press vendors to make their software vendor neutral. thornton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Sep 29 02:51:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 29 Sep 2003 08:51:11 +0200 (CEST) Subject: Redhat Fedora In-Reply-To: <20030928202357.GA2644@galactic.demon.co.uk> Message-ID: On Sun, 28 Sep 2003, Andrew M.A. Cater wrote: > a.) Get them to take the risk of keeping up to date with Fedora and hold > them to it. > > b.) Get them to port the code to any LSB compliant distribution / remove > kernel version specific stuff. > > Remember folks: This is Linux (GNU/Linux for the pedantic :) ) - we have > access to the underlying code and will continue so to do. If need be, I agree with a lot of what Andrew says - though I'm not that much of a Debian fan (*). Linux is much more than another operating system. Many scientists, engineers and users in business and the arts do not see it that way. Linux has become popular, on cheap yet high performance kit. All many people now care about is 'getting the job done' or 'running my codes'. We should realise there is much to be gained from open standards and LSB compliant systems. (*) Not ideology - I've just used other distros. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Sep 29 05:31:23 2003 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 29 Sep 2003 11:31:23 +0200 (CEST) Subject: Redhat Fedora In-Reply-To: <1064808472.16662.12.camel@localhost.localdomain> Message-ID: On Sun, 28 Sep 2003, Thornton Prime wrote: > Why do you say this? To my knowledge, the only restrictions either of > these distributions have (or can have) on re-distribution are trademark > restrictions involving the use of their distribution name or logo. Indeed - I use Pink Tie Linux at home. This is a distro supplied on CD-ROM which is equivalent to RH. The system I'm using at the moment started life as a Pink Tie 8.0 I've since used apt-get dist-upgrade to upgrade it to a 9.0 (Sorry Bob - not YUM.) I'm still a bit proud of that - dist-upgrading a RH system. There's a bit of a discussion on the Fedora list about dist-upgrade to the Severn beta. But I think I'll be safe here and run that up on another system. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Mon Sep 29 10:38:15 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Mon, 29 Sep 2003 16:38:15 +0200 (CEST) Subject: Cluster distributions? In-Reply-To: <1064217163.15240.6.camel@revolution.mandrakesoft.com> Message-ID: On Mon, 22 Sep 2003, Erwan Velu wrote: > That one of our stronger point, the ka technologies from IDIMAG allow us > to deploy new nodes within 3 minutes (duplication, reoboot, autoconfig). While 3 minutes sounds impressive, the time also depends on the amount of data that you transfer. > This tools helps you to keep your nodes up to date. Urpmi parallel uses > the ka technologies so the time needed for installing 50MB on a cluster > doesn't depend on the cluster size ! It takes the same time for installing > on 2 or 50 nodes ! So just have an switched network (very common this > days). How do you do it? With the multi-drop chain similar to Dolly? Regards, Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H16 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andy.schwierskott at sun.com Mon Sep 29 12:28:11 2003 From: andy.schwierskott at sun.com (Andy Schwierskott) Date: Mon, 29 Sep 2003 18:28:11 +0200 (MEST) Subject: AW: AW: [GE users] Scheduling In-Reply-To: <0DDE7F776936B04AA670C77F1F59A4D909D2BC@deaex001.arvinmeritor.com> References: <0DDE7F776936B04AA670C77F1F59A4D909D2BC@deaex001.arvinmeritor.com> Message-ID: Mark, job_load_adjustments are applied after a job has been dispatched. So the first job should go into the queue, but not a seconds one immediately after. >From SGE 5.3p3 you see SGE's internal bookkepping when a queue is in (virtual) alarm state with # qstat -j It shows the real load and the job_load_adjustments what the scheduler used in the last scheduling run. Andy On Mon, 29 Sep 2003, Olesen, Mark wrote: > More scheduling woes ... > > I am now completely confused about how the scheduler is supposed to work. > In accordance with sched_conf(5), I had assumed that the queues' > "load_thresholds" would be considered during scheduling, but I'm seeing > *somewhat* different behaviour. > > For all my queues, I've set > load_thresholds np_load_avg=1.2 > > As previously mentioned, I am sorting by seqno: > algorithm default > schedule_interval 0:0:15 > maxujobs 0 > queue_sort_method seqno > user_sort true > job_load_adjustments np_load_avg=0.5 > load_adjustment_decay_time 0:7:30 > load_formula np_load_avg > schedd_job_info true > > > The next queue in the sequence with any free slots has load_avg=2.05, or > np_load_avg=1.03 > > I would imagine that the scheduler predicts a new np_load_avg=1.53 (current > + joad_load_adjustments). Since this predicted load is above the queue load > threshold, I would not expect a new job to start on this queue. > Contrary to my expectations, a new job does indeed get sent to this queue. > > What other tricks do I need in order to coerce the scheduler to respect the > load thresholds? > > Cheers, > > > Dr. Mark Olesen > Thermofluid Dynamics Analyst > ArvinMeritor Light Vehicle Systems > Zeuna Staerker GmbH & Co. KG > Biberbachstr. 9 > D-86154 Augsburg, GERMANY > tel: +49 (821) 4103 - 862 > fax: +49 (821) 4103 - 7862 > Mark.Olesen at ArvinMeritor.com --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net For additional commands, e-mail: users-help at gridengine.sunsource.net From erwan at mandrakesoft.com Sun Sep 28 06:34:47 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Sun, 28 Sep 2003 12:34:47 +0200 (CEST) Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <3F760FBE.3050409@ucia.gov> References: <3F760FBE.3050409@ucia.gov> Message-ID: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> > We will stay with ROCKS for our initial deployment of 320 Opteron > system. It is due in about 60 days... Does ROCKS supports Opteron ? I never saw this announcement... So do you intend using ROCKS using an x86 OS ? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 29 04:57:31 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 29 Sep 2003 10:57:31 +0200 Subject: Redhat Fedora In-Reply-To: <20030926154351.M3752@www2> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> <20030926175312.GA2187@greglaptop.internal.keyresearch.com> <20030926154351.M3752@www2> Message-ID: <1064825850.3580.11.camel@revolution.mandrakesoft.com> > Possibly, given that, as Erwan said, the CLIC contract > is up in December 2003, perhaps it would make more sense > for the community to pick that up as a starting point. That's a very good idea, it's really make sense to continue the CLIC project. The funds for CLIC ends at December 2003 but we are ready to continue this project with people whom are intrested in. CLIC could be a really good basis for a Free clustering distribution. The new CLIC version will be available in few days and then all the basic tools & distribution will be ready for a new start. What does "Beowulf" people think about it ? -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Mon Sep 29 08:03:22 2003 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Mon, 29 Sep 2003 14:03:22 +0200 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <3F6B4D4C.8070506@moene.indiv.nluug.nl> (Toon Moene's message of "Fri, 19 Sep 2003 20:39:08 +0200") References: <3F6B4D4C.8070506@moene.indiv.nluug.nl> Message-ID: Toon Moene writes: > Robert G. Brown wrote: > >> We have some of these (including the big red switch by the door) but not >> all. Maybe soon -- the first is really a matter of sensors and scripts. > > Do you also have the "door-opening-switch" right next to the BRS ? > That can lead to very funny mistakes (especially if you're not the one > making the mistake). Couple of years ago, when my employer-at-the-time were building new premises, I walked into the computer room-to-be just as the electricians were finishing the installation of the Big Red Button. In their wisdom they had placed it by the door, between the light switch and the door lock switch. Fortunately, I was authorized to issue Change Orders, so the electricians' Orders were Changed *real* quick. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alberor at ucia.gov Sun Sep 28 09:39:56 2003 From: alberor at ucia.gov (Al R) Date: Sun, 28 Sep 2003 09:39:56 -0400 Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> References: <3F760FBE.3050409@ucia.gov> <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> Message-ID: <3F76E4AC.7070100@ucia.gov> Erwan, Our initial deployment will utilize the Opterons in 32-bit mode. Our current software baseline is 32-bit plus legacy software spanning 20+ years of development. Until we are given compiled 64-bit versions, and no vendor wants to release their source code, nor do WE want to port their code, we will simply stay at 32-bit. No problem. Our first upgrade will be an in-rack upgrade. This will be a software reload taking us to 64-bit. We perform 64-bit arithmetic, but not in a 64-bit address space. None of applications currently require this. In the next 18-36 months, yes, there may be a change. We will be ready. Let's face it, it is a tough choice to make, 32 vs 64 these days. Especially for a cluster "of size". ROCKS is so easy to use, that we have actually cut our cluster in half in support of special processing. We will be doing this with our system. The initial 120 nodes (240 processors) will be connected via Myrinet. We received additional funding for another rack of 40 nodes (+80 processors) and found the enormous cost of "Myrinet in excess of 128 nodes". We decided to let the single frontend node manage this rack/nodes, and let the PBS quename determine the need/use of these nodes. Or, we can simply make the bottom node another frontend and have it manage its own rack of 39 dual-Opteron compute nodes. It also makes a darn good "staging cluster" so software can be tested and then ported to the other side... ROCKS is simply that flexible... Take 20 nodes out of your cluster, place two on a desk for 10 people. Teach them about ROCKS and parallel processing using two nodes and a crossover cable. Put the nodes back in the rack and in 5-8 minutes reload all 20 nodes back as compute nodes... WHILE THE ORIGINAL CLUSTER IS STILL RUNNING. No impacts to operations. Just my 2 cents as a ROCKS user for 18 months. Al Erwan Velu wrote: >>We will stay with ROCKS for our initial deployment of 320 Opteron >>system. It is due in about 60 days... > > > Does ROCKS supports Opteron ? I never saw this announcement... > So do you intend using ROCKS using an x86 OS ? > > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 29 10:59:23 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 29 Sep 2003 16:59:23 +0200 Subject: Cluster distributions? In-Reply-To: References: Message-ID: <1064847562.5365.23.camel@revolution.mandrakesoft.com> > While 3 minutes sounds impressive, the time also depends on the amount > of data that you transfer. This system reach 10-11MB/sec for the harddrive duplication + ~ 1 minute for launching the minimal linux on the nodes which must be deployed and run the appropriate tools before & after the deployment. The number of nodes don't change the amount of time needed for the deployment. > How do you do it? With the multi-drop chain similar to Dolly? Yes that is the same kind of technology but it doesn't allow channel bonding like dolly. When the chain is initialised, the server sends packets to the first node which handle it and forward it to the next one. If it can't handle this packet the node is buffering its inputs and the server is buffering its outputs. Urpmi parallel is a kid of "apt" mixed with a multi-drop chain tool. Installing packages with dependencies on a full cluster is really impressive :D -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alberor at ucia.gov Sun Sep 28 11:38:48 2003 From: alberor at ucia.gov (Al R) Date: Sun, 28 Sep 2003 11:38:48 -0400 Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <1537.81.57.165.53.1064767549.squirrel@webmail.mandrakesoft.com> References: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> <3F76E4AC.7070100@ucia.gov> <1537.81.57.165.53.1064767549.squirrel@webmail.mandrakesoft.com> Message-ID: <3F770088.3060206@ucia.gov> We were in an instructional mode. Each student had to build a frontend and a compute node. It was also done in a classroom. The students were to format the drives and other crash and burn activities. :-) This was a hands-on lab, with out of body (rack) experience. Thank you for the reference in your note below. I will go take a look at this! Al Erwan Velu wrote: >>Erwan, > > [...] > >>ROCKS is simply that flexible... Take 20 nodes out of your cluster, >>place two on a desk for 10 people. Teach them about ROCKS and parallel >>processing using two nodes and a crossover cable. Put the nodes back in >>the rack and in 5-8 minutes reload all 20 nodes back as compute nodes... >>WHILE THE ORIGINAL CLUSTER IS STILL RUNNING. No impacts to operations. > > > For that kind of applications, our choice was just making a subcluster > using MAUI. You can tell this 20 nodes to be a subcluster ( a partition) > called subcluster1, then you can assign users to this partition using > DrakCluster GUI. > > After that when people submit jobs via PBS theirs jobs are sent to this > subcluster. At the end of this session, you can reintegrate this nodes to > the original cluster in the same range of time you mention. > > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Sun Sep 28 12:45:49 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Sun, 28 Sep 2003 18:45:49 +0200 (CEST) Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <3F76E4AC.7070100@ucia.gov> References: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> <3F76E4AC.7070100@ucia.gov> Message-ID: <1537.81.57.165.53.1064767549.squirrel@webmail.mandrakesoft.com> > Erwan, [...] > ROCKS is simply that flexible... Take 20 nodes out of your cluster, > place two on a desk for 10 people. Teach them about ROCKS and parallel > processing using two nodes and a crossover cable. Put the nodes back in > the rack and in 5-8 minutes reload all 20 nodes back as compute nodes... > WHILE THE ORIGINAL CLUSTER IS STILL RUNNING. No impacts to operations. For that kind of applications, our choice was just making a subcluster using MAUI. You can tell this 20 nodes to be a subcluster ( a partition) called subcluster1, then you can assign users to this partition using DrakCluster GUI. After that when people submit jobs via PBS theirs jobs are sent to this subcluster. At the end of this session, you can reintegrate this nodes to the original cluster in the same range of time you mention. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 29 17:12:44 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 29 Sep 2003 23:12:44 +0200 (CEST) Subject: Redhat Fedora In-Reply-To: <3F78773B.6020209@obs.unige.ch> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> <20030926175312.GA2187@greglaptop.internal.keyresearch.com> <20030926154351.M3752@www2> <1064825850.3580.11.camel@revolution.mandrakesoft.com> <3F78773B.6020209@obs.unige.ch> Message-ID: <1164.81.57.165.53.1064869964.squirrel@webmail.mandrakesoft.com> > I am just now installing a new cluster of dual Xeon and want to try CLIC. > However my hardware is too new and most of the chipsets are > not recognized by Mandrake 9.0, but works fine with Mandrake 9.1. > When will be the new CLIC version available? Note that I would be glad > to be a beta tester if you wish to let me download a > preliminary CLIC version. However if "a few days" mean 3-4 days > I would agree to wait. We know that CLIC is now quite obsolete due to a quite old kernel. I've just received the newest kernel (2.4.22) by my kernel team. We had to integrate it ASAP. Our schedule had planned the newest CLIC for the end of this week. So please wait for it. I'll keep you in touch when the iso is ready. You can also join our Mailing list, you can find explanations at http://clic.mandrakesoft.com In addition, you'll find the newest backend for an easiest installation & management process. Best regards, -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From michael.worsham at mci.com Mon Sep 29 14:44:34 2003 From: michael.worsham at mci.com (Michael Worsham) Date: Mon, 29 Sep 2003 14:44:34 -0400 Subject: Power Supply: Supermicro P4DL6 Board? Message-ID: <009a01c386b9$b9f55280$c96432a6@Wcomnet.com> Anyone know a good place to pick up a power supply that is supported for Supermicro's P4DL6 dual xeon motherboard? Finding normal P4 power supplies are a dime a dozen, but a P/S with both a 20pin + 8pin connector is hard to find and usually about as expensive as the case needed for the board. Thanks. -- M _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sean at asacomputers.com Mon Sep 29 19:09:54 2003 From: sean at asacomputers.com (Sean) Date: Mon, 29 Sep 2003 16:09:54 -0700 Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: <009a01c386b9$b9f55280$c96432a6@Wcomnet.com> Message-ID: <5.1.0.14.2.20030929160429.02e94ec0@pop.asacomputers.com> Michael, You need a minimum 400W ATX12 power supply. We normally recommend 460 W power supply considering the increasing number of hard drives that go with the system recently. Supermicro has their own 460W PS, Supermicro 760P4 Power supply PWS-024 you can either choose this one or go with other power supply vendors like PC Power and Cooling or Enermax. At 02:44 PM 9/29/03 -0400, Michael Worsham wrote: >Anyone know a good place to pick up a power supply that is supported for >Supermicro's P4DL6 dual xeon motherboard? Finding normal P4 power supplies >are a dime a dozen, but a P/S with both a 20pin + 8pin connector is hard to >find and usually about as expensive as the case needed for the board. > >Thanks. > >-- M > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf Thanks and Regards Sean ASA Computers Inc. 2354, Calle Del Mundo Santa Clara CA 95054 Telephone : (408) 654-2901 xtn 205 (408) 654-2900 ask for Sean (800) REAL-PCS (1-800-732-5727) Fax: (408) 654-2910 E-mail : sean at asacomputers.com URL : http://www.asacomputers.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Mon Sep 29 19:45:53 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Mon, 29 Sep 2003 16:45:53 -0700 Subject: HyperTransport note In-Reply-To: References: <20030910163913.GA3486@greglaptop.greghome.keyresearch.com> Message-ID: <20030929234553.GC2450@greglaptop.internal.keyresearch.com> We were talking about HyperTransport a while back. The HT consortium has issued a press release about HT 1.1, saying that it includes technology sufficient for reliable transport over HT. I dunno when anyone is going to implement it in HW, but they're headed in the right direction. Let's see, what else: packet handling, peer-to-peer routing, and virtual channels for streaming data. Fun stuff. http://www.hypertransport.org/pr_092903.htm -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 29 20:29:22 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 30 Sep 2003 10:29:22 +1000 Subject: Redhat Fedora In-Reply-To: <20030927071142.GA1838@greglaptop.greghome.keyresearch.com> References: <24a6532445ee.2445ee24a653@georgetown.edu> <20030927071142.GA1838@greglaptop.greghome.keyresearch.com> Message-ID: <200309301029.24376.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 27 Sep 2003 05:11 pm, Greg Lindahl wrote: > In the future, RedHat is trying to ensure that these apps require > RedHat Enterprise, not the consumer version. I suspect this will also be the case for hardware vendors as well, in that they are likely to say the supported Linux configuration will be RHEL or SLES. So there will basically be the choice of running either RHEL or SLES and getting support, or not and being on your own. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/eM5iO2KABBYQAh8RAv/wAJ9I+OFyUifmwFV18n9/8W9u6I93ZQCfQph1 l71u5hFGAAIBrtyeU9f6iKc= =INHN -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 29 20:38:47 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 30 Sep 2003 10:38:47 +1000 Subject: Redhat Fedora In-Reply-To: <1064808472.16662.12.camel@localhost.localdomain> References: <24a6532445ee.2445ee24a653@georgetown.edu> <20030928202357.GA2644@galactic.demon.co.uk> <1064808472.16662.12.camel@localhost.localdomain> Message-ID: <200309301038.49590.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 29 Sep 2003 02:07 pm, Thornton Prime wrote: > Why do you say this? To my knowledge, the only restrictions either of > these distributions have (or can have) on re-distribution are trademark > restrictions involving the use of their distribution name or logo. RHEL contains trademarked Red Hat material. You need to remove this to be able to redistribute it yourself. SLES includes non-GPL software from SuSE. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD4DBQE/eNCXO2KABBYQAh8RAq73AJY5lKm6jRB8/E7tkc8gU5u3pI5kAJ43ZfB/ 2QadBSgOfCNdiXwVDKJpnQ== =zoFi -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 29 20:36:50 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 30 Sep 2003 10:36:50 +1000 Subject: Redhat Fedora In-Reply-To: <20030928202357.GA2644@galactic.demon.co.uk> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> <20030928202357.GA2644@galactic.demon.co.uk> Message-ID: <200309301036.54005.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 29 Sep 2003 06:23 am, Andrew M.A. Cater wrote: > Indeed: by demanding Red Hat / SuSE you may already be using a > proprietary solution (since you can't redistribute their Linux > versions). For RHEL though you can build a free version from source providing you remove the RH trademarked items. I don't believe this is possible for SLES as that contains proprietary code. > If your vendor only supplies binaries which are vendor/version locked - > strongly consider changing your vendor. The problem is that this sort of thing is such a niche market there is often only one vendor. The real issue isn't an overt vendor lock in, just that if you're not running what they support then you risk getting shrugged off with "we don't support that" if (when?) something goes wrong and need some help. Personally I like open source a lot, but in our line of work we have to provide for our users requirements or they'll go elsewhere. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/eNAiO2KABBYQAh8RAqcKAJ0eBOIpk6Xoke2+hdFDvLDg/r9NYwCfUrkD 1iDVLxiGD/ru0cx6UV5Zl+s= =LoJf -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Mon Sep 29 19:43:58 2003 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Mon, 29 Sep 2003 16:43:58 -0700 (PDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: <009a01c386b9$b9f55280$c96432a6@Wcomnet.com> Message-ID: hi ya those 20pin + 8pinner is also dime a dozen ... if you're picky about reliability ... there's 2 choices of powersupplies i used in the past few years... sparklepower and zippy(aka emacs) - dont buy used power supplies or "repaired" power supplies for anything important list of power supply manufacturers http://www.Linux-1U.net/PowerSupp if you're local ( SJ or SD or LA ) we can just go pick it up ... c ya alvin On Mon, 29 Sep 2003, Michael Worsham wrote: > Anyone know a good place to pick up a power supply that is supported for > Supermicro's P4DL6 dual xeon motherboard? Finding normal P4 power supplies > are a dime a dozen, but a P/S with both a 20pin + 8pin connector is hard to > find and usually about as expensive as the case needed for the board. > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mitchel at navships.com Tue Sep 30 15:15:01 2003 From: mitchel at navships.com (Mitchel Kagawa) Date: Tue, 30 Sep 2003 09:15:01 -1000 Subject: Environment monitoring Message-ID: <005401c38787$25cad390$7001a8c0@Navatek.local> This past Friday night after everyone in the office went home our AC, for our small 64 node cluster, decided to have a compressor problem and stop producing cool air. On Sat I get a phone call saying that one of our users can't access his e-mail (server is also in cluster room). I wasn't able to get to the office until Sunday afternoon and I wasn't able to log into the head node to shut everything down remotely. By the time I was able to get to the cluster all but 20 nodes had shut themselves down because the ambient temperature of the room reached 145 deg. Thankfully all the servers, raid arrays and all but 2 nodes came back up when I hit the power button (after I got the temp back to 68). Our stupid ac doesn't have the communications package that cost an extra $5000 cause my company's too cheap to spring for it and an extra phone line. So I was wondering if anyone used one of those environment monitoring appliances that e-mail you (before stuff starts shutting off) if a temp goes out of a set range or if it detects water, high humidity, or movement in the room? I'm looking at a NetBotz/RackBots here (http://www.netbotz.com/products/rack.html) and was wondering if anyone has any experience with them or if you have any other (inexpensive <$2000) suggestions. Thanks for any help. Mitchel Kagawa _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From yudong at hsb.gsfc.nasa.gov Tue Sep 30 15:56:41 2003 From: yudong at hsb.gsfc.nasa.gov (Yudong Tian) Date: Tue, 30 Sep 2003 15:56:41 -0400 Subject: Upper bound on no. of sockets In-Reply-To: Message-ID: The number of sockets a process can open is limited by the number of file descriptors (fds). Type "ulimit -n" under bash to get this number, which usually 1024 by default. You can increase this number if you wish. Google "increase Linux file descriptors" you will find many examples, like this one: http://support.zeus.com/faq/zws/v4/entries/os/linuxfd.html If you want to be really sure, you can compile and run the following c program to get the number, which is the output plus 3 (stdout, stdin and stderr): /* # test how many fd you have $Id$ */ int main(int argc, char *argv[]) { int i = 0; while( tmpfile()){ i++; } printf("Your free fds: %d\n", i); } /*** end of program ***/ If you are running a TCP server and want to test how many clients that server can support, you can run the following perl program to test: #!/usr/bin/perl # Test the max number of tcp connections a server can support # $Id: testMaxCon.pl,v 1.1 2003/06/23 19:10:41 yudong Exp yudong $ # usage: testMaxCon.pl IP port use IO::Socket::INET; @ARGV == 2 or die "Usage: testMaxCon.pl svr_ip svr_port\n"; my ($ip, $port) = @ARGV; my $i = 0; do { $i++; $socket[$i] = IO::Socket::INET->new(PeerAddr => $ip, PeerPort => $port, Proto => "tcp", Timeout => 6, Type => SOCK_STREAM); } while ($socket[$i]); print "Max TCP connections supported for this client: ", $i-1, "\n"; ## end of program Of course for this test you have to make sure you have more fds to use than the server. Hope that helps. Yudong ------------------------------------------------------------ Falun Dafa: The Tao of Meditation (http://www.falundafa.org) ------------------------------------------------------------ Yudong Tian, Ph.D. NASA/GSFC (301) 286-2275 > -----Original Message----- > From: beowulf-admin at scyld.com [mailto:beowulf-admin at scyld.com]On Behalf > Of Balaji Rangasamy > Sent: Tuesday, September 30, 2003 12:44 AM > To: beowulf at beowulf.org > Subject: Upper bound on no. of sockets > > > Hi, > Is there an upper bound on the number of sockets that can be created by a > process? If there is one, is the limitation enforced by OS? And what other > factors does it depend on? Can you please be specific on the numbers for > different OS (RH Linux 7.2) ? > Thank you very much, > Balaji. > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Tue Sep 30 15:38:07 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 30 Sep 2003 14:38:07 -0500 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <1064950687.10771.29.camel@caffeine> Bummer about the AC. You might want to go and check your backups and backup mechanism with a renewed zeal. Based on past experience you are in for an "interesting" near future. While they came back just fine, I wouldn't expect them to last very long after experiencing that high of temp. -Dean On Tue, 2003-09-30 at 14:15, Mitchel Kagawa wrote: > This past Friday night after everyone in the office went home our AC, for > our small 64 node cluster, decided to have a compressor problem and stop > producing cool air. On Sat I get a phone call saying that one of our users > can't access his e-mail (server is also in cluster room). I wasn't able to > get to the office until Sunday afternoon and I wasn't able to log into the > head node to shut everything down remotely. By the time I was able to get to > the cluster all but 20 nodes had shut themselves down because the ambient > temperature of the room reached 145 deg. Thankfully all the servers, raid > arrays and all but 2 nodes came back up when I hit the power button (after I > got the temp back to 68). Our stupid ac doesn't have the communications > package that cost an extra $5000 cause my company's too cheap to spring for > it and an extra phone line. So I was wondering if anyone used one of those > environment monitoring appliances that e-mail you (before stuff starts > shutting off) if a temp goes out of a set range or if it detects water, high > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 30 16:00:08 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 30 Sep 2003 16:00:08 -0400 (EDT) Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: On Tue, 30 Sep 2003, Mitchel Kagawa wrote: > This past Friday night after everyone in the office went home our AC, for > our small 64 node cluster, decided to have a compressor problem and stop > producing cool air. On Sat I get a phone call saying that one of our users > can't access his e-mail (server is also in cluster room). I wasn't able to > get to the office until Sunday afternoon and I wasn't able to log into the > head node to shut everything down remotely. By the time I was able to get to > the cluster all but 20 nodes had shut themselves down because the ambient > temperature of the room reached 145 deg. Thankfully all the servers, raid > arrays and all but 2 nodes came back up when I hit the power button (after I > got the temp back to 68). Our stupid ac doesn't have the communications > package that cost an extra $5000 cause my company's too cheap to spring for > it and an extra phone line. So I was wondering if anyone used one of those > environment monitoring appliances that e-mail you (before stuff starts > shutting off) if a temp goes out of a set range or if it detects water, high > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. There are two ways to proceed. I've used netbotz, and yes, they work fine and would have saved you here. As you note, a bit expensive...;-) The cheaper alternative is to DIY. There are several companies out there that sell temperature probes that can be read via a serial port. Buy one, hook it into a serial port of the machine of your choice, and write a script to sleep/poll the device, taking any action you like at any breakpoints you like. You can even add a PC-TV card and an X10 camera (for a total of about $100-150) and monitor the room remotely to get even more of the netbotz functionality. All total should cost you no more than $250-400 to DIY with camera. But yes, you have to have some skills here and there...:-) rgb > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 30 16:00:49 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 30 Sep 2003 16:00:49 -0400 (EDT) Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: On Tue, 30 Sep 2003, Mitchel Kagawa wrote: > suggestions. Thanks for any help. Oh, I might have forgotten to say: At least some of the thermal sensors are linked to brahma: www.phy.duke.edu/brahma rgb > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dritch at hpti.com Tue Sep 30 16:20:30 2003 From: dritch at hpti.com (David B. Ritch) Date: Tue, 30 Sep 2003 16:20:30 -0400 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <1064953230.4433.65.camel@localhost> I haven't used an environmental monitoring system quite that sophisticated, but I do have an APC UPS that supports environmental monitoring. I track it using BigBrother (http://bb4.com). You can also monitor temperatures on many motherboards using lm-sensors and a tool like BigBrother. dbr On Tue, 2003-09-30 at 15:15, Mitchel Kagawa wrote: > This past Friday night after everyone in the office went home our AC, for > our small 64 node cluster, decided to have a compressor problem and stop > producing cool air. On Sat I get a phone call saying that one of our users > can't access his e-mail (server is also in cluster room). I wasn't able to > get to the office until Sunday afternoon and I wasn't able to log into the > head node to shut everything down remotely. By the time I was able to get to > the cluster all but 20 nodes had shut themselves down because the ambient > temperature of the room reached 145 deg. Thankfully all the servers, raid > arrays and all but 2 nodes came back up when I hit the power button (after I > got the temp back to 68). Our stupid ac doesn't have the communications > package that cost an extra $5000 cause my company's too cheap to spring for > it and an extra phone line. So I was wondering if anyone used one of those > environment monitoring appliances that e-mail you (before stuff starts > shutting off) if a temp goes out of a set range or if it detects water, high > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- David B. Ritch High Performance Technologies, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 30 16:05:02 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 30 Sep 2003 16:05:02 -0400 (EDT) Subject: Environment monitoring In-Reply-To: <1064950687.10771.29.camel@caffeine> Message-ID: On 30 Sep 2003, Dean Johnson wrote: > Bummer about the AC. You might want to go and check your backups and > backup mechanism with a renewed zeal. Based on past experience you are > in for an "interesting" near future. While they came back just fine, I > wouldn't expect them to last very long after experiencing that high of > temp. Geeze, I forgot to add one more comment. We had just been talking about the time required for a server room to "get hot" upon the AC shutting down. Today was AC filter changing day here. Shut down blower unit (but not cluster) for about 7 minutes. In a room with at least 30 KW sustained, roughly 4 meters x 10 meters, ambient temperature rose perhaps 10F, just barely perceptibly (to maybe 75F) by the time the AC came back on -- warmer immediately behind a rack, of course. Lots of air, not that efficient a thermal transfer, lots of thermal ballast to absorb the heat. rgb > > -Dean > > On Tue, 2003-09-30 at 14:15, Mitchel Kagawa wrote: > > This past Friday night after everyone in the office went home our AC, for > > our small 64 node cluster, decided to have a compressor problem and stop > > producing cool air. On Sat I get a phone call saying that one of our users > > can't access his e-mail (server is also in cluster room). I wasn't able to > > get to the office until Sunday afternoon and I wasn't able to log into the > > head node to shut everything down remotely. By the time I was able to get to > > the cluster all but 20 nodes had shut themselves down because the ambient > > temperature of the room reached 145 deg. Thankfully all the servers, raid > > arrays and all but 2 nodes came back up when I hit the power button (after I > > got the temp back to 68). Our stupid ac doesn't have the communications > > package that cost an extra $5000 cause my company's too cheap to spring for > > it and an extra phone line. So I was wondering if anyone used one of those > > environment monitoring appliances that e-mail you (before stuff starts > > shutting off) if a temp goes out of a set range or if it detects water, high > > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > > any experience with them or if you have any other (inexpensive <$2000) > > suggestions. Thanks for any help. > > > > Mitchel Kagawa > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Tue Sep 30 17:28:55 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Tue, 30 Sep 2003 17:28:55 -0400 (EDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: <3F79B423.AD3B5729@attglobal.net> Message-ID: > Has anyone recently analyzed how much power each of the subunits of a PC uses, i.e. how much power which > mainboard, hdd, etc. it's not hard to approximate this by reading the specs of each component. obviously, CPUs are the main consumers, probably followed by power hardware next (PS inefficiency, as well as the on-motherboard converter.) disks are much, much cooler than they used to be, probably dropping below power consumed by ram on most clusters. naturally, if you've got 6 15K RPM SCSI disks in a node with just 1G ram, that's completely different! even things like high-speed nics often amount to <10W, which simply doesn't compare to the ~80-90W per high-end CPU... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john at cusick.ws Tue Sep 30 18:54:14 2003 From: john at cusick.ws (John Cusick) Date: 30 Sep 2003 18:54:14 -0400 Subject: [tulip] 10BaseT/100BaseT Question In-Reply-To: References: Message-ID: <1064962453.5649.120.camel@sager.cusick.ws> On Tue, 2003-09-30 at 15:23, Donald Becker wrote: > On 30 Sep 2003, John Cusick wrote: > > On Mon, 2003-09-29 at 21:14, Donald Becker wrote: > > > On 23 Sep 2003, John Cusick wrote: > > > > > > > Card: Adaptec AN-8944A - 4 Port > > > > > > > > The card loads fine as a module and works, but I would like to know if > > > > its possible to force one of the ports to 10BaseT > > > What driver version are you using? > > > What is the detection message? > > > > Linux Tulip driver version 0.9.15-pre12 (Aug 9, 2002) > > Yup, that's the modified branch. It has lots of "issues" that my > version does not have. > > > tulip0: EEPROM default media type Autosense. > > tulip0: Index #0 - Media MII (#11) described by a 21140 MII PHY (1) > > block. > > tulip0: ***WARNING***: No MII transceiver found! > > That's a problem... > > > > If it's using the old driver included with the kernel, try the driver > > > from scyld.com > > > > > > mkdir /tmp/netdrivers/ > > > cd /tmp/netdrivers/ > > > ncftpget ftp://ftp.scyld.com/pub/network/netdrivers.tgz > > > tar xfvz netdrivers.tgz > > > make > > > make install > > > > > > > I've downloaded the newest version and it builds fine, so I attempted to > > use the spec file (I like to keep the rpm database up to date if > > possible, although its not critical, of course) and it also built fine > > except that it will not create the rpm's. > > That spec file is was original designed to be generic, but > as multiple incompatible RPM versions have been released and > the external module build system has changed without being fixed > we have given up and accepted that it will only work with our Linux > distribution. > > > It exits with the following error: > > > > Processing files: netdrivers-modules-3- at PACKAGE_RELEASE@.k2.4.20_20.7 > > error: File must begin with "/": %{prefix}/lib/modules/*/*/* > > You can work around this. Substitute your own release number for > @PACKAGE_RELEASE@ > Donald, I appreciate your help with this (I figured I had better say this before you decide that I'm a royal pain in the rear end). Talk about off topic... The error is cleaned up as far as @PACKAGE_RELEASE@ is concerned, however the final error still remains. I have gone through the Maximum RPM Book (which is pretty piss-poor as far as detail goes) and cannot figure out what is going on. Processing files: netdrivers-modules-3-5.2.4.20_20.7 error: File must begin with "/": %{prefix}/lib/modules/*/*/* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ RPM build errors: File must begin with "/": %{prefix}/lib/modules/*/*/* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ No rush on this, or even ignore it if you prefer since it is off-topic. I will install the drivers from the source for now to see if the previous problem is solved, and keep working on this one. If I get it done properly (tough considering the availability of documentation and my limited skills) then I will be sure to post to the list and make it available. ... snip ... > Note that this only works if > The driver implements the ioctl(), including supporting the virtual > registers with SYM transceivers. > The driver detects the transceiver (which is a problem here). Regards, John C. _______________________________________________ tulip mailing list, tulip at scyld.com To change to digest mode or unsubscribe visit http://www.scyld.com/mailman/listinfo/tulip From waitt at saic.com Tue Sep 30 18:00:20 2003 From: waitt at saic.com (Tim Wait) Date: Tue, 30 Sep 2003 18:00:20 -0400 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <3F79FCF4.9050004@saic.com> > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. > > Mitchel Kagawa http://www.apcc.com/products/family/index.cfm?id=47&tab=models You can get a standalone unit that you can plug into your network and access via telnet or http, or if you have a SmartUPS unit, get the card that plugs into the slots at the back, and use APC's powerchute software to monitor the data over a serial port. It has all the bells and whistle for custom alert/notification etc. ...and the price is right. Tim _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Tue Sep 30 16:23:37 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Tue, 30 Sep 2003 15:23:37 -0500 (CDT) Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: Dont overlook lm_sensors+cron -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 30 19:40:26 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 30 Sep 2003 19:40:26 -0400 (EDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: Message-ID: On Tue, 30 Sep 2003, Mark Hahn wrote: > > Has anyone recently analyzed how much power each of the subunits of > > a PC uses, i.e. how much power which mainboard, hdd, etc. > > it's not hard to approximate this by reading the specs of each component. > obviously, CPUs are the main consumers, probably followed by power hardware > next (PS inefficiency, as well as the on-motherboard converter.) The CPU power numbers vary. You find people quoting design limits (~80W), typical power (25W with moderate desktop usage), average power (15W averaged over a day). For the G5 I've only seen optimistic typical power guesses directly compared to worst case for x86. That's probably unfair: while it likely does use less power, it's not yet obvious that the total system uses less power to achieve the same performace as the Opteron. > disks are much, much cooler than they used to be, probably dropping > below power consumed by ram on most clusters. Note that most performance-oriented RAM types now have metal cases and heat sinks. They didn't add the metal because it _looks_ cool. > naturally, if you've > got 6 15K RPM SCSI disks in a node with just 1G ram, that's completely > different! even things like high-speed nics often amount to <10W, > which simply doesn't compare to the ~80-90W per high-end CPU... GbE NICs used to use >10W, more than half used in the transceiver chip. Today the typical GbE NIC is a single chip in the 3-5W range. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jschauma at netmeister.org Tue Sep 30 16:46:35 2003 From: jschauma at netmeister.org (Jan Schaumann) Date: Tue, 30 Sep 2003 16:46:35 -0400 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <20030930204635.GA17458@netmeister.org> Mitchel Kagawa wrote: > So I was wondering if anyone used one of those environment monitoring > appliances that e-mail you (before stuff starts shutting off) if a > temp goes out of a set range or if it detects water, high humidity, or > movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if > anyone has any experience with them or if you have any other > (inexpensive <$2000) suggestions. Thanks for any help. It just so happens that last week I installed a NetBotz (http://www.netbotz.com/products/wall.html#wallbotz400) in our machine room. It was a breeze to install/setup and looks rather nifty wrt capabilities (SNMP, email alerts, syslog, ftp- and http upload/logging, alert escalation etc.). So far (as I said, ~1 week old) I'd recommend it. I wonder what internal OS it's running. -Jan -- Wenn ich tot bin, mir soll mal Einer mit Auferstehung oder so kommen, ich hau ihm eine rein! (Anonym) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From yemi at SLAC.Stanford.EDU Tue Sep 30 17:43:17 2003 From: yemi at SLAC.Stanford.EDU (Adesanya, Adeyemi) Date: Tue, 30 Sep 2003 14:43:17 -0700 Subject: [Ganglia-general] Running as non-root on Solaris Message-ID: <4ABE591494FF5246B38C0CDA4CBFAFE8025217D6@exchange4.slac.stanford.edu> Hi Steven. Thanks for the pointer. I managed to disable/strip the built-in solaris metrics and now gmond runs in non-root!! ---- Yemi > -----Original Message----- > From: steven wagner [mailto:swagner at ilm.com] > Sent: Monday, September 29, 2003 11:05 AM > To: Adesanya, Adeyemi > Cc: 'ganglia-general at lists.sourceforge.net' > Subject: Re: [Ganglia-general] Running as non-root on Solaris > > > Sure, there's some metric stub code you can copy from in the > gmond/machines directory. I think cygwin.c is still mostly returning > dummy values for metrics ( ... anyone?), you could use that as a > template for the functions you'll be disabling. > > Sorry I can't offer you any more direct assistance, but I > don't really > have any cycles to spare for Solaris Ganglia 2.x development. > > Happy hacking :) > > Adesanya, Adeyemi wrote: > > Hi Steve. > > > > Thanks for responding. Perhaps an alternative may be switches to > > disable those metrics that require gmond to run as root? > > > > ---- > > Yemi > > > > > > > >>-----Original Message----- > >>From: steven wagner [mailto:swagner at ilm.com] > >>Sent: Monday, September 29, 2003 10:32 AM > >>To: Adesanya, Adeyemi > >>Cc: ganglia-general at lists.sourceforge.net > >>Subject: Re: [Ganglia-general] Running as non-root on Solaris > >> > >> > >>Adeyemi Adesanya wrote: > >> > >>>Hi There. > >>> > >>>I spent some time digging through the archives but I am > >> > >>unable to find > >> > >>>a way of running gmond as a non-root user on Solaris. Is > >> > >>this out of > >> > >>>the question or is there some way to patch the code? All of our > >>>critical servers run Solaris, that?s where the real action > >> > >>is and that > >> > >>>what I'd like to monitor. > >> > >>Unfortunately, not all of the information that the Solaris > monitoring > >>core collects can be gathered through a warm and fuzzy API > >>(as far as I > >>could tell, anyway). Many metrics had to be gathered by > >>peeking at the > >>kernel memory and symbol table, both of which require > >>superuser permissions. > >> > >>It's entirely possible that the kstat API has been extended > >>(or that I > >>just plain didn't RTFM thoroughly enough) and that all of > these calls > >>can be handled that way. In which case, the first person to > >>point this > >>out volunteers to rewrite all my spaghetti solaris.c code. :) > >> > >>Good luck! > >> > > ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Ganglia-general mailing list Ganglia-general at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general From lanceh at cs.unr.edu Tue Sep 30 18:08:01 2003 From: lanceh at cs.unr.edu (Lance Hutchinson) Date: Tue, 30 Sep 2003 15:08:01 -0700 Subject: [Ganglia-general] Errors starting Gmetad Message-ID: <1064959681.13298.4.camel@vrpad.cs.unr.edu> Hi all. I recently had the problem that the webfrontend of my 64 node rocks cluster did not have any graphs showing, as none of the rrds were being generated. I upgraded my gmetad, and now i get the following messages when i start gmetad with debug turned on. Can anyone tell me where i am going wrong? What do I have the wrong version of? [root at cortex lance]# /etc/init.d/gmetad start Starting GANGLIA gmetad: Going to run as user nobody Source: [Cortex, step 15] has 1 sources 127.0.0.1 xml listening on port 8651 interactive xml listening on port 8652 Data thread 7176 is monitoring [Cortex] data source 127.0.0.1 [Cortex] is an OLD version Thanks a ton. -- Lance Hutchinson ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Ganglia-general mailing list Ganglia-general at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general From becker at scyld.com Tue Sep 30 19:40:26 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 30 Sep 2003 19:40:26 -0400 (EDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: Message-ID: On Tue, 30 Sep 2003, Mark Hahn wrote: > > Has anyone recently analyzed how much power each of the subunits of > > a PC uses, i.e. how much power which mainboard, hdd, etc. > > it's not hard to approximate this by reading the specs of each component. > obviously, CPUs are the main consumers, probably followed by power hardware > next (PS inefficiency, as well as the on-motherboard converter.) The CPU power numbers vary. You find people quoting design limits (~80W), typical power (25W with moderate desktop usage), average power (15W averaged over a day). For the G5 I've only seen optimistic typical power guesses directly compared to worst case for x86. That's probably unfair: while it likely does use less power, it's not yet obvious that the total system uses less power to achieve the same performace as the Opteron. > disks are much, much cooler than they used to be, probably dropping > below power consumed by ram on most clusters. Note that most performance-oriented RAM types now have metal cases and heat sinks. They didn't add the metal because it _looks_ cool. > naturally, if you've > got 6 15K RPM SCSI disks in a node with just 1G ram, that's completely > different! even things like high-speed nics often amount to <10W, > which simply doesn't compare to the ~80-90W per high-end CPU... GbE NICs used to use >10W, more than half used in the transceiver chip. Today the typical GbE NIC is a single chip in the 3-5W range. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Sep 30 19:10:57 2003 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Wed, 01 Oct 2003 01:10:57 +0200 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> (Mitchel Kagawa's message of "Tue, 30 Sep 2003 09:15:01 -1000") References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: "Mitchel Kagawa" writes: > I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if > anyone has any experience with them or if you have any other > (inexpensive <$2000) suggestions. I'm using serial interface temperature sensors from Picotech. I've hacked a smallish Python driver to speak to them, and use rrdtool to make pretty graphs and Nagios to scream bloody murder if the temperature rises to far. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jsims at csiopen.com Tue Sep 30 21:51:06 2003 From: jsims at csiopen.com (Joey Sims) Date: Tue, 30 Sep 2003 21:51:06 -0400 Subject: Power Supply: Supermicro P4DL6 Board? Message-ID: <812B16724C38EE45A802B03DD01FD547226258@exchange.concen.com> Hello Michael, I also recommend Zippy. They make an excellent product. Below is a power supply guide that is manufacturer specific in regards to the motherboard you might have. Your particular model isn't listed here but, all the rest are that also share this same spec. If you check out the other Supermicro 400MHz series dual Xeon boards, you'll notice this. http://www.zippy.com/P_GUIDE_SEARCH.asp?lv_rfnbr=2 You're looking at ~ $125 - $140 for a decent single 460W. Regards, ================================================== Joey P. Sims 800.995.4274 - 242 Sales Manager 770.442.5896 - Fax HPC/Storage Division www.csilabs.net Concentric Systems, Inc. jsims at csiopen.com ====================================ISO9001:2000== _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mprinkey at aeolusresearch.com Mon Sep 1 16:26:17 2003 From: mprinkey at aeolusresearch.com (Michael T. Prinkey) Date: Mon, 1 Sep 2003 16:26:17 -0400 (EDT) Subject: Has Any Tested These? Message-ID: Adaptec's new gigabit ethernet card with TCP offloading: http://www.eweek.com/article2/0,3959,1228684,00.asp Any word on pricing? Thanks, Mike Prinkey Aeolus Research, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From johnt at quadrics.com Tue Sep 2 07:56:52 2003 From: johnt at quadrics.com (John Taylor) Date: Tue, 2 Sep 2003 12:56:52 +0100 Subject: Has Any Tested These? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA7CCDF25@stegosaurus.bristol.quadrics.com> The article suggests pricing at the end of about 800-900 USD about the same price as NICS for other high performance products which offer even better b/w and latency, notwithstanding switch costs of course. Be interested in performance of this. John > -----Original Message----- > From: Michael T. Prinkey [mailto:mprinkey at aeolusresearch.com] > Sent: 01 September 2003 21:26 > To: beowulf at beowulf.org > Subject: Has Any Tested These? > > > > Adaptec's new gigabit ethernet card with TCP offloading: > > http://www.eweek.com/article2/0,3959,1228684,00.asp > > Any word on pricing? > > Thanks, > > Mike Prinkey > Aeolus Research, Inc. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wade.hampton at nsc1.net Tue Sep 2 14:04:19 2003 From: wade.hampton at nsc1.net (Wade Hampton) Date: Tue, 02 Sep 2003 14:04:19 -0400 Subject: e1000 and Scyld 28cz Message-ID: <3F54DBA3.8060608@nsc1.net> G'day, I'm upgrading my first test cluster to 1G ethernet using the onboard e1000 parts on my nodes (Tyan motherboards with eepro100 and e1000). I have not gotten the nodes to boot and recognize the e1000 cards using a beoboot floppy. When I tried the latest driver from Intel (5.1.13), I could not get an e1000 module for my node boot floppy until I did the following: CFLAGS=$(KCFLAGS) make KCFLAGS="-D__BOOT_KERNEL_H_ -D__module__beoboot mv e1000.o /lib/modules/2.2.19-14.beobeoboot/net I also added the following to /etc/beowulf/config pci 0x8086 0x1008 e1000 (reference: http://www.beowulf.org/pipermail/beowulf/2002-September/004575.html) When I try to boot this floppy, it stopped booting after loading the e1000 driver and appeared to hang. I then removed the e1000.o modules and tried the Scyld RPM for 4.4.12. This seemed to hang at the same location. Any help in this matter would be appreciated. -- Wade Hampton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Tue Sep 2 22:57:47 2003 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Tue, 02 Sep 2003 21:57:47 -0500 Subject: Oh great pool of hardware and driver knowledge.... Message-ID: <3F5558AB.2010605@tamu.edu> OK, so this is off-topic again, but this group is likely the most knowledgable source of info likely to have a handle on something like this. I've got a pair of servers directly off a router/switch. There is one problemmatic server/port interface, as shown in the MRTG page at http://page4.tamu.edu/mrtg/atmres/165.91.1.44_40.html If you notice, connectivity appears to drop every 30 min. although subjectively it appears to happen more often (say, every 3 min or so...) for periods of 30-45 sec. Hardware here is a Dell PwerEdge 650 with the onboard Intel Pro/E1000 adapter and a 2nd Intel E1000 dual port copper adapter. Software is RH9 with all current patches, Apache and the Minnesota Mapserver, plus PostGres/PostGIS running. NFS services to the box die periodically and users are sayingthe web response stinks. I've taken a couple of decent shots at the problem. I'm hoping someone here might have a potential solution I can check out! Thanks! gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lehi.gracia at amd.com Wed Sep 3 16:16:07 2003 From: lehi.gracia at amd.com (lehi.gracia at amd.com) Date: Wed, 3 Sep 2003 15:16:07 -0500 Subject: Oh great pool of hardware and driver knowledge.... Message-ID: <99F2150714F93F448942F9A9F112634C07BE62FE@txexmtae.amd.com> I would try Dell drivers instead of the Intel drivers, I don't know if that is what you are using or not, but Dell does some driver development for their systems and might be different than the Intel ones. Sorry can't help any more but there is not a lot of info on your system's configuration, although it might not make a lot of difference because I haven't touch a PE650 for a few months now. Try different switch port, different switch, different network board, if you are trunking try without, try the latest DSA cd from Dell, it might have some new patches in it. Lehi -----Original Message----- From: Gerry Creager N5JXS [mailto:gerry.creager at tamu.edu] Sent: Tuesday, September 02, 2003 9:58 PM To: Beowulf Subject: Oh great pool of hardware and driver knowledge.... OK, so this is off-topic again, but this group is likely the most knowledgable source of info likely to have a handle on something like this. I've got a pair of servers directly off a router/switch. There is one problemmatic server/port interface, as shown in the MRTG page at http://page4.tamu.edu/mrtg/atmres/165.91.1.44_40.html If you notice, connectivity appears to drop every 30 min. although subjectively it appears to happen more often (say, every 3 min or so...) for periods of 30-45 sec. Hardware here is a Dell PwerEdge 650 with the onboard Intel Pro/E1000 adapter and a 2nd Intel E1000 dual port copper adapter. Software is RH9 with all current patches, Apache and the Minnesota Mapserver, plus PostGres/PostGIS running. NFS services to the box die periodically and users are sayingthe web response stinks. I've taken a couple of decent shots at the problem. I'm hoping someone here might have a potential solution I can check out! Thanks! gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joachim at ccrl-nece.de Thu Sep 4 03:57:50 2003 From: joachim at ccrl-nece.de (Joachim Worringen) Date: Thu, 4 Sep 2003 09:57:50 +0200 Subject: Intel acquiring Pallas In-Reply-To: <1062089290.4363.22.camel@localhost.localdomain> References: <1062089290.4363.22.camel@localhost.localdomain> Message-ID: <200309040957.50028.joachim@ccrl-nece.de> Vann H. Walke: > Pallas's main HPC product is Vampir/Vampirtrace. These are performance > analysis tools. As such they would only be peripherally useful for > compiler design (perhaps to measure the effects of certain changes). > Even for this purpose, Vampir/Vampirtrace doesn't provide the amount of > detail that Intel's own V-Tune product does. I agree with Vann. I assume they just want to extend V-Tune upwards, and get a fully integrated product. And maybe some additional knowledge on MPI (Pallas implemented/maintains the MPI for Fujitsu machines and maybe others). I wonder if Pallas will continue to sell PGI compilers (which are optimized for AMD/Opteron...). BTW, Intel also owns 25% of Scali. And Etnus, which produce TotalView, which Pallas sells, belongs to Dolphin, which in turn make a lot of business with Scali. Whatever that means... but it's a small world. Joachim -- Joachim Worringen - NEC C&C research lab St.Augustin fon +49-2241-9252.20 - fax .99 - http://www.ccrl-nece.de _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Thu Sep 4 11:32:22 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 04 Sep 2003 11:32:22 -0400 Subject: perl-bproc bindings In-Reply-To: References: Message-ID: <1062689542.2576.12.camel@roughneck> On Thu, 2003-08-28 at 14:35, Donald Becker wrote: > On 28 Aug 2003, Nicholas Henke wrote: > > > On Thu, 2003-08-28 at 01:49, Donald Becker wrote: > > > On Tue, 26 Aug 2003, Daniel Widyono wrote: > > > > > > > Anyone out there besides Dan Ridge from Scyld make Perl bindings for bproc, > > > > something more recent than spring of 2001? > > > > > > There are updated bindings, and a small example, at > > > ftp://www.scyld.com/pub/beowulf-components/bproc-perl/bproc-perl.tar.gz > > > > Any chance you guys have updated python bindings as well? > > 0.9-8 is the current version -- which are you using? > The last bugfix was logged in October of 2003. I am using pybproc-0.8. Where can I find the version you are referring to? > > The next planned refresh has added bindings for the Beostat statistics > library, Beomap job mapping and BBQ job scheduling systems. Awesome -- i look forward to using them. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ktaka at clustcom.com Thu Sep 4 10:49:40 2003 From: ktaka at clustcom.com (Kimitoshi Takahashi) Date: Thu, 04 Sep 2003 23:49:40 +0900 Subject: e1000 and Scyld 28cz In-Reply-To: <3F54DBA3.8060608@nsc1.net> References: <3F54DBA3.8060608@nsc1.net> Message-ID: <200309041449.AA00364@jack2.clustcom.com> Hello, >From my limited experience, I can think of the followings, 1. If your NIC chip is 82547(Intel's giga nic chip for CSA connection), the line in the /etc/beowulf/config should be, pci 0x8086 0x1019 e1000 You have to find out what your NIC's pci ID is. In the case of recent RedHat, see /etc/sysconfig/hwconfig. 2. You might have to rebuild phase 2 boot image with new e1000.o using beoboot. I hope these help. Kimitoshi Takahashi Wade Hampton ????????: >G'day, > >I'm upgrading my first test cluster to 1G ethernet >using the onboard e1000 parts on my nodes (Tyan >motherboards with eepro100 and e1000). > >I have not gotten the nodes to boot and recognize >the e1000 cards using a beoboot floppy. > >When I tried the latest driver from Intel (5.1.13), >I could not get an e1000 module for my node boot >floppy until I did the following: > > CFLAGS=$(KCFLAGS) > make KCFLAGS="-D__BOOT_KERNEL_H_ -D__module__beoboot > mv e1000.o /lib/modules/2.2.19-14.beobeoboot/net > >I also added the following to /etc/beowulf/config > pci 0x8086 0x1008 e1000 > > (reference: >http://www.beowulf.org/pipermail/beowulf/2002-September/004575.html) > >When I try to boot this floppy, it stopped booting after loading the >e1000 driver and appeared to hang. > >I then removed the e1000.o modules and tried the Scyld RPM >for 4.4.12. This seemed to hang at the same location. > >Any help in this matter would be appreciated. >-- >Wade Hampton > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tod at gust.sr.unh.edu Thu Sep 4 14:24:41 2003 From: tod at gust.sr.unh.edu (Tod Hagan) Date: 04 Sep 2003 14:24:41 -0400 Subject: Semaphore controlling access to resource in OpenPBS? SGE? Message-ID: <1062699881.452.22.camel@haze.sr.unh.edu> Hi, I'm a guest on a system running OpenPBS where I'd like to schedule a bunch of test jobs using OpenPBS. All the jobs will execute out of the same directory. Rather than duplicating the directory for each job, or using OpenPBS's support for dependency lists and rigidly specifying the execution order myself, what I really want is to declare a semaphore/mutex locking access to the directory so only one of these jobs will run at a time, and then let the batch system figure out which job is best to run given the other jobs on the cluster. Is there any way to declare this kind of semaphore or lock with OpenPBS? How about SGE? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Thu Sep 4 17:17:18 2003 From: rossini at blindglobe.net (A.J. Rossini) Date: Thu, 04 Sep 2003 14:17:18 -0700 Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? Message-ID: <85k78oi78h.fsf@blindglobe.net> I'm in the process of putting together another small compute cluster, and while I've solved the node configuration issues, I've been unable to come to any definitive solutions for networking. Because of the problems I work on (trivial or coarse parallelization, and at the most, moderate between-node communication), I've focused on copper-based gigabit. Now the hard part -- can anyone suggest NICs (Intel's e1000 is currently leading my list) and switches to uses? (Netgear GS508T/GS108, HP ProCurve 2708, etc...?). Throughput is of interest, though I've got to balance it with price... Or even better, point to comparison WWW sites? (I've been looking on Google and Tom's, but havn't really found the right query incantation yet, sigh...). best, -tony -- A.J. Rossini rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW : FAX=206-543-3461 | moving soon to a permanent office FHCRC: 206-667-7025 FAX=206-667-4812 | Voicemail is pretty sketchy/use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Johnson.Jeffre at epamail.epa.gov Thu Sep 4 16:40:53 2003 From: Johnson.Jeffre at epamail.epa.gov (Johnson.Jeffre at epamail.epa.gov) Date: Thu, 04 Sep 2003 13:40:53 -0700 Subject: Microway Opteron Cluster Message-ID: Hello; We are contemplating purchasing a small Opteron-based cluster from Microway, and I am curious as to whether anyone has any thoughts about Microway and/or Opterons. Thank you. Jeffre C. Johnson Research Chemist U.S. EPA, ORD, NERL, HEASD, EDRB (not quite the entire alphabet) Fabulous Las Vegas, Nevada _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lehi.gracia at amd.com Thu Sep 4 17:42:38 2003 From: lehi.gracia at amd.com (lehi.gracia at amd.com) Date: Thu, 4 Sep 2003 16:42:38 -0500 Subject: Microway Opteron Cluster Message-ID: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> Depending on how small is small you might want to look at Angstrom, for higher density, if realstate is a concern. http://www.angstrom.com/ Lehi -----Original Message----- From: Johnson.Jeffre at epamail.epa.gov [mailto:Johnson.Jeffre at epamail.epa.gov] Sent: Thursday, September 04, 2003 3:41 PM To: beowulf at beowulf.org Subject: Microway Opteron Cluster Hello; We are contemplating purchasing a small Opteron-based cluster from Microway, and I am curious as to whether anyone has any thoughts about Microway and/or Opterons. Thank you. Jeffre C. Johnson Research Chemist U.S. EPA, ORD, NERL, HEASD, EDRB (not quite the entire alphabet) Fabulous Las Vegas, Nevada _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctierney at hpti.com Thu Sep 4 17:57:37 2003 From: ctierney at hpti.com (Craig Tierney) Date: 04 Sep 2003 15:57:37 -0600 Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <85k78oi78h.fsf@blindglobe.net> References: <85k78oi78h.fsf@blindglobe.net> Message-ID: <1062712656.2998.12.camel@woody> On Thu, 2003-09-04 at 15:17, A.J. Rossini wrote: > I'm in the process of putting together another small compute cluster, > and while I've solved the node configuration issues, I've been unable > to come to any definitive solutions for networking. Because of the > problems I work on (trivial or coarse parallelization, and at the > most, moderate between-node communication), I've focused on > copper-based gigabit. > > Now the hard part -- can anyone suggest NICs (Intel's e1000 is > currently leading my list) and switches to uses? (Netgear > GS508T/GS108, HP ProCurve 2708, etc...?). Throughput is of interest, > though I've got to balance it with price... I haven't tried it yet, but the 12 and 24 port gigE switches from Dell look attractive. The switches provide full bandwidth and have several other good features. They are under $100 per port ($1199 and $2199 respectively). Craig > > Or even better, point to comparison WWW sites? (I've been looking on > Google and Tom's, but havn't really found the right query incantation > yet, sigh...). > > best, > -tony -- Craig Tierney _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Thu Sep 4 18:40:36 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Thu, 04 Sep 2003 18:40:36 -0400 Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <85k78oi78h.fsf@blindglobe.net> References: <85k78oi78h.fsf@blindglobe.net> Message-ID: <3F57BF64.1010008@bellsouth.net> Tony, SMC recently announced a low-cost 8 port GigE switch that can handle jumbo frames. http://www.smc.com/index.cfm?sec=About-SMC&pg=Press-Release-Details&pr_id=129&site=c They also announced GigE NICs for $30. I don't know about Linux support. If not, here's a website that some some decent information about copper GigE: http://www.cs.uni.edu/~gray/gig-over-copper/ Good Luck! Jeff >I'm in the process of putting together another small compute cluster, >and while I've solved the node configuration issues, I've been unable >to come to any definitive solutions for networking. Because of the >problems I work on (trivial or coarse parallelization, and at the >most, moderate between-node communication), I've focused on >copper-based gigabit. > >Now the hard part -- can anyone suggest NICs (Intel's e1000 is >currently leading my list) and switches to uses? (Netgear >GS508T/GS108, HP ProCurve 2708, etc...?). Throughput is of interest, >though I've got to balance it with price... > >Or even better, point to comparison WWW sites? (I've been looking on >Google and Tom's, but havn't really found the right query incantation >yet, sigh...). > >best, >-tony > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From angel at wolf.com Thu Sep 4 19:48:37 2003 From: angel at wolf.com (Angel Rivera) Date: Thu, 04 Sep 2003 23:48:37 GMT Subject: Microway Opteron Cluster In-Reply-To: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> References: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> Message-ID: <20030904234837.22385.qmail@houston.wolf.com> I love the angstrom, just be aware that if anything happens you will always be affecting two nodes. lehi.gracia at amd.com writes: > Depending on how small is small you might want to look at Angstrom, for higher density, if realstate is a concern. > http://www.angstrom.com/ > > Lehi > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sgoel at angstrom.com Thu Sep 4 20:00:20 2003 From: sgoel at angstrom.com (Sachin Goel) Date: Thu, 4 Sep 2003 20:00:20 -0400 Subject: Microway Opteron Cluster References: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> <20030904234837.22385.qmail@houston.wolf.com> Message-ID: <027a01c37340$b539a160$ae01a8c0@sachinlaptop> thanks for your appreciation Angel, but our blades are independent of each other if anything fails...so 1 blade failure affects only one blade.... cheers sachin Sachin Goel Angstrom Microsystems 27 Dry Dock Avenue, 8 Floor Boston, MA 02210 Tel: 617-695-0137 Ext 27 Cell: 617-429-2237 Fax: 617-695-0984 www.angstrom.com ----- Original Message ----- From: "Angel Rivera" To: Cc: ; Sent: Thursday, September 04, 2003 7:48 PM Subject: Re: Microway Opteron Cluster > I love the angstrom, just be aware that if anything happens you will always > be affecting two nodes. > > > lehi.gracia at amd.com writes: > > > Depending on how small is small you might want to look at Angstrom, for higher density, if realstate is a concern. > > http://www.angstrom.com/ > > > > Lehi > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From angel at wolf.com Thu Sep 4 21:31:58 2003 From: angel at wolf.com (Angel Rivera) Date: Fri, 05 Sep 2003 01:31:58 GMT Subject: Microway Opteron Cluster In-Reply-To: <027a01c37340$b539a160$ae01a8c0@sachinlaptop> References: <99F2150714F93F448942F9A9F112634C07BE630E@txexmtae.amd.com> <20030904234837.22385.qmail@houston.wolf.com> <027a01c37340$b539a160$ae01a8c0@sachinlaptop> Message-ID: <20030905013158.29603.qmail@houston.wolf.com> We run them hard and they are fast. We don't have the blades-will have to see how much money we can get out of them. :: Sachin Goel writes: > thanks for your appreciation Angel, but our blades are independent of each > other if anything fails...so 1 blade failure affects only one blade.... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Thu Sep 4 20:43:54 2003 From: becker at scyld.com (Donald Becker) Date: Thu, 4 Sep 2003 20:43:54 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <3F57BF64.1010008@bellsouth.net> Message-ID: On Thu, 4 Sep 2003, Jeffrey B. Layton wrote: > SMC recently announced a low-cost 8 port GigE switch that can handle > jumbo frames. > > http://www.smc.com/index.cfm?sec=About-SMC&pg=Press-Release-Details&pr_id=129&site=c That's very aggressive pricing for a switch with jumbo frame support. The previous lowest-cost GbE switches have a street price of $160 for 8 ports, but the per-port price goes up dramatically for larger switches. > They also announced GigE NICs for $30. I don't know about Linux > support. If not, here's a website that some some decent information > about copper GigE: My guess is that they are using Realtek rtl8169 chips. The rtl8169 is not as sophisticated as the e1000, but is still a reasonable performer. It's a much better design than the justly-maligned rtl8139. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Sep 4 22:08:12 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Fri, 5 Sep 2003 10:08:12 +0800 (CST) Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <1062699881.452.22.camel@haze.sr.unh.edu> Message-ID: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> If you use SGE, this howto will provide the info: http://gridengine.sunsource.net/project/gridengine/howto/resource.html I hope OpenPBS has similar features, but given that OpenPBS is basically dead (bugs unfixed, no new ports, very light mailing list traffic), is there a reason to keep using it? At least use Scablable PBS or SGE. Andrew. --- Tod Hagan ???? > Hi, > > I'm a guest on a system running OpenPBS where I'd > like to schedule a > bunch of test jobs using OpenPBS. All the jobs will > execute out of the > same directory. Rather than duplicating the > directory for each job, or > using OpenPBS's support for dependency lists and > rigidly specifying the > execution order myself, what I really want is to > declare a > semaphore/mutex locking access to the directory so > only one of these > jobs will run at a time, and then let the batch > system figure out which > job is best to run given the other jobs on the > cluster. > > Is there any way to declare this kind of semaphore > or lock with OpenPBS? > > How about SGE? > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Thu Sep 4 22:16:18 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 5 Sep 2003 12:16:18 +1000 Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> References: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> Message-ID: <200309051216.19692.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 5 Sep 2003 12:08 pm, Andrew Wang wrote: [OpenPBS dead] > very light mailing list traffic ...if you are lucky enough to be allowed onto it. It appears that they are ignoring attempts to subscribe these days (the list requires approval from the maintainers to join). Scalable PBS is certainly the way to go as far as we are concerned. - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/V/HyO2KABBYQAh8RAkCmAKCH1BJRYcE3P3VbEIMiZ/r/7doJtgCgg6qP V/lANRb6qLB4qVcyY5YJjDc= =VVIx -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sandeep at earl.met.fsu.edu Thu Sep 4 23:01:49 2003 From: sandeep at earl.met.fsu.edu (sandeep) Date: Thu, 04 Sep 2003 23:01:49 -0400 Subject: MMINPUT codes Message-ID: <5.2.0.9.2.20030904225959.029ddc10@earl.met.fsu.edu> Dear Beowulf, Myself in need of a code which could able to read the MMINPUT_DOMAIN files if you have the same ,could you kindly help me out . looking forward to hear from you soon. thanking you san _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 5 04:52:12 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 5 Sep 2003 10:52:12 +0200 (CEST) Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <200309051216.19692.csamuel@vpac.org> Message-ID: I'll speak up in favout of Gridengine. I'm learning more about it. The subject of Gridengine on Opteron came up recently, on a thread started by Mikhail. My experience recently was of setting up Gridengine on an Opteron cluster. Nodes run the SuSE 8.2 for x86_64. The 32 bit version of Gridengine seems to run OK. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Fri Sep 5 06:26:19 2003 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Fri, 5 Sep 2003 12:26:19 +0200 (CEST) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: <1062712656.2998.12.camel@woody> Message-ID: On 4 Sep 2003, Craig Tierney wrote: > I haven't tried it yet, but the 12 and 24 port gigE switches from > Dell look attractive. The switches provide full bandwidth and have > several other good features. Except the management interface which I find horrible. Well, comparing it with the 3Com SuperStack and BayNetworks switches that I have... Have you tested the bandwidth or is there some published test about this ? I don't have enough GigE nodes to fill one... The documentation implies that there are 2 chips inside each responsible for 12 ports. I wasn't worried about on-chip communication, but inter-chip one. -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sgaudet at wildopensource.com Fri Sep 5 09:03:33 2003 From: sgaudet at wildopensource.com (Stephen Gaudet) Date: Fri, 05 Sep 2003 09:03:33 -0400 Subject: Microway Opteron Cluster In-Reply-To: References: Message-ID: <3F5889A5.4030104@wildopensource.com> Hello Jeffre, Johnson.Jeffre at epamail.epa.gov wrote: > > > > Hello; > > We are contemplating purchasing a small Opteron-based cluster from > Microway, and I am curious as to whether anyone has any thoughts about > Microway and/or Opterons. Thank you. While Microway is a good company, you might want to check out Angstrom Microsystems. http://www.Angstrom.com I just read this article on Angstrom. http://msnbc-cnet.com.com/2100-1010_3-5071318.html?part=msnbc-cnet&tag=alert&form=feed&subj=cnetnews I have a friend that worked there for over a year and to my surprise Angstrom NEVER had one RMA. I never believed it, until I heard it from some of their customers. I might add, Lalit Jain the CEO, graduate from MIT, builds the best AMD solution of all the AMD partners. No one spends more time making sure the box is cool enough and the power is clean. Check them out, can't go wrong. Cheers, Steve Gaudet Wild Open Source (home office) ---------------------- Bedford, NH 03110 pH:603-488-1599 cell:603-498-1600 http://www.wildopensource.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Fri Sep 5 10:11:08 2003 From: deadline at plogic.com (Douglas Eadline) Date: Fri, 5 Sep 2003 10:11:08 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Thu, 4 Sep 2003, Donald Becker wrote: > On Thu, 4 Sep 2003, Jeffrey B. Layton wrote: > > > SMC recently announced a low-cost 8 port GigE switch that can handle > > jumbo frames. > > > > http://www.smc.com/index.cfm?sec=About-SMC&pg=Press-Release-Details&pr_id=129&site=c > > That's very aggressive pricing for a switch with jumbo frame support. > > The previous lowest-cost GbE switches have a street price of $160 for 8 > ports, but the per-port price goes up dramatically for larger switches. > > > They also announced GigE NICs for $30. I don't know about Linux > > support. If not, here's a website that some some decent information > > about copper GigE: > > My guess is that they are using Realtek rtl8169 chips. The rtl8169 is > not as sophisticated as the e1000, but is still a reasonable performer. > It's a much better design than the justly-maligned rtl8139. Based on what I read at: http://www.cs.uni.edu/~gray/gig-over-copper/ (note: read the paper http://www.cs.uni.edu/~gray/gig-over-copper/hsln-lcn.ps) I purchased two Netgear 302T nics for about $30 each. They are 32 bit cards that will run at 66 Mhz, have a Broadcom chipset, and use the tg3 driver. Here as some _preliminary_ results using netpipe(tcp) PCI Speed Latency Max Throughput 33 28 700 Mbits/sec (87.5 Mbytes/sec) 66 30 840 Mbits/sec (105 Mbytes/sec) The tests were done in dual PIII-1.266 Serverworks LE (Supermicro P3TDLE) using the kernel 2.40.20, 1500 byte MTU (they support Jumbos, however) While not as sexy as a PCI-X 64 bit card, at $30 a card, the performance is pretty good. Doug Doug > > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lehi.gracia at amd.com Fri Sep 5 10:48:31 2003 From: lehi.gracia at amd.com (lehi.gracia at amd.com) Date: Fri, 5 Sep 2003 09:48:31 -0500 Subject: Microway Opteron Cluster Message-ID: <99F2150714F93F448942F9A9F112634C07BE6312@txexmtae.amd.com> Hello again Jeffre, >From my prior posting I just wanted is to point out that there are options that you could fit more systems in a rack "if" you need more processors per square foot and "if" you require many systems in your cluster. Since you have been considering already a company perhaps you should contact them and ask them about their options and they may have the best solution for you and/or point you in the right direction. Lehi -----Original Message----- From: Johnson.Jeffre at epamail.epa.gov [mailto:Johnson.Jeffre at epamail.epa.gov] Sent: Thursday, September 04, 2003 3:41 PM To: beowulf at beowulf.org Subject: Microway Opteron Cluster Hello; We are contemplating purchasing a small Opteron-based cluster from Microway, and I am curious as to whether anyone has any thoughts about Microway and/or Opterons. Thank you. Jeffre C. Johnson Research Chemist U.S. EPA, ORD, NERL, HEASD, EDRB (not quite the entire alphabet) Fabulous Las Vegas, Nevada _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Fri Sep 5 11:03:05 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Fri, 05 Sep 2003 17:03:05 +0200 Subject: Semaphore controlling access to resource in OpenPBS? SGE? In-Reply-To: <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> References: <1062699881.452.22.camel@haze.sr.unh.edu> <20030905020812.27982.qmail@web16803.mail.tpe.yahoo.com> Message-ID: <20030905170305H.hanzl@unknown-domain> >> semaphore/mutex locking access to the directory > If you use SGE, this howto will provide the info: > > http://gridengine.sunsource.net/project/gridengine/howto/resource.html but to create true mutex you want slightly different configuration: - 'consumable' resource, with the overall amount to consume being just '1' - each job needs amount '1' of this resource so this resource is exhausted by the first job requesting it and available again once it finished, and this decision happens in central scheduler so it is a safe mutex. I think this howto is a closer match to this task: http://gridengine.sunsource.net/project/gridengine/howto/consumable.html (It takes some time to get used to GE's system of resources, consumables, various default values etc. but it is well worth it, they are nice building blocks for many useful things.) I would however still worry about coherency of things cached by network filesystem although some slow setups of say NFS are probably nearly safe. I personally think there is one great way of doing this: - Scheduler should know that one job needs to see filesystem changes made by another job (there are job dependencies specified via -hold_jid or there is a 'directory mutex') - Scheduler should be able to ask network filesystem "please propagate cached changes made on node A so as they are visible on node B" This would IMHO solve speed/consistency dilema for many practical purposes. It should be easy to implement the SGE part of this trick but I am not aware of any network filesystem being able to "cache a lot, propagate on demand" as described above. Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brichard at clusterworldexpo.com Fri Sep 5 12:40:35 2003 From: brichard at clusterworldexpo.com (Bryan Richard) Date: Fri, 5 Sep 2003 12:40:35 -0400 Subject: Clusters in the Finance Industry? Message-ID: <20030905164035.GJ79048@clusterworldexpo.com> Hello, I am doing research on HPC Beowulf clusters in the financial sector for the upcoming ClusterWorld and I was hoping the list might be able to help me out. I am interesting in learning more about, * How the finance industry utilizes clusters: risk management, modeling, analysis, &c. * What type of systems are being utilized * What applications are common to this industry * Who are the key individuals in HPC finance Our ultimate goal would be to include a Finance track in the ClusterWorld technical program -- something beyond Deep Green case studies -- that addresses the clustering needs of the HPC finance community. Happy to speak to anyone either on or off the list. Given the tight-lipped nature of this industry, any pointers or assistance would be greatly helpful. Thanks. Be well, Bryan Richard Conference Director QuarterPower Media +(913) 254 0592 voice +(913) 961 3601 mobile brichard at clusterworldexpo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Fri Sep 5 12:26:35 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Fri, 5 Sep 2003 09:26:35 -0700 (PDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Fri, 5 Sep 2003, Bogdan Costescu wrote: > On 4 Sep 2003, Craig Tierney wrote: > > > I haven't tried it yet, but the 12 and 24 port gigE switches from > > Dell look attractive. The switches provide full bandwidth and have > > several other good features. > > Except the management interface which I find horrible. Well, comparing it > with the 3Com SuperStack and BayNetworks switches that I have... The dells have a kind of brain-dead cisco-style cli. if you've used cat-ios you'll mostly be slightly annoyed by the things it doesn't do, if you're not a cat-ios user it'll probably just anoy the heck out of you... The switches themselves are made by accton for what it's worth. > Have you tested the bandwidth or is there some published test about this ? > I don't have enough GigE nodes to fill one... The documentation implies > that there are 2 chips inside each responsible for 12 ports. I wasn't > worried about on-chip communication, but inter-chip one. the 24 port models are two broadcom 5690's connected back to back using the 10Gb/s xaui port on each core. > -- > Bogdan Costescu > > IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen > Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY > Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 > E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tod at gust.sr.unh.edu Fri Sep 5 14:27:09 2003 From: tod at gust.sr.unh.edu (Tod Hagan) Date: 05 Sep 2003 14:27:09 -0400 Subject: MMINPUT codes In-Reply-To: <5.2.0.9.2.20030904225959.029ddc10@earl.met.fsu.edu> References: <5.2.0.9.2.20030904225959.029ddc10@earl.met.fsu.edu> Message-ID: <1062786430.2227.3.camel@haze.sr.unh.edu> On Thu, 2003-09-04 at 23:01, sandeep wrote: > Dear Beowulf, > > Myself in need of a code which could able to read the MMINPUT_DOMAIN files > if you have the same ,could you kindly help me out . Sandeep, You are more likely to get an answer for your question by asking on the mm5-users mailing list: http://mailman.ucar.edu/mailman/listinfo/mm5-users _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 5 14:43:46 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 5 Sep 2003 11:43:46 -0700 Subject: Filesystem In-Reply-To: <200308282056.54106.exa@kablonet.com.tr> References: <200308282056.54106.exa@kablonet.com.tr> Message-ID: <20030905184345.GA1378@greglaptop.internal.keyresearch.com> > I basically think ext3 and ext2 are a joke and we use XFS on the nodes with no > performance problem. Excellent reliability! Filesystem religion is fun, but keep in mind that ext2/3, XFS and Reiserfs are used in production by some pretty big communities -- Redhat, SGI, and Suse are not in the business of shipping software that fails for the common case. ext2 is somewhat annoying in that a big cluster of machines with local disks that has a power failure is likely to have at least one node that asks for a manual fsck. but reports of actal data loss are few and far between. greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mprinkey at aeolusresearch.com Fri Sep 5 13:08:50 2003 From: mprinkey at aeolusresearch.com (Michael T. Prinkey) Date: Fri, 5 Sep 2003 13:08:50 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: > (note: read the paper > http://www.cs.uni.edu/~gray/gig-over-copper/hsln-lcn.ps) > > I purchased two Netgear 302T nics for about $30 each. They are 32 bit > cards that will run at 66 Mhz, have a Broadcom chipset, and use > the tg3 driver. Here as some _preliminary_ results using netpipe(tcp) > > PCI Speed Latency Max Throughput > 33 28 700 Mbits/sec (87.5 Mbytes/sec) > 66 30 840 Mbits/sec (105 Mbytes/sec) > > The tests were done in dual PIII-1.266 Serverworks LE (Supermicro P3TDLE) > using the kernel 2.40.20, 1500 byte MTU (they support Jumbos, however) > While not as sexy as a PCI-X 64 bit card, at $30 a card, the performance > is pretty good. > > Doug > I will second this. I have been using Netgear 302Ts since late last year. I have been using them on dual Xeon boards with on-board Intel Gbit NICs. I run the on-board to a switch and use a pair of Netgears to build nearest neighbor links. It makes for a very cost effective way to speed up CFD runs that use 1D decompositions. The latency and bandwith of these cards are very impressive considering their low cost and 32-bittedness. Here is a tar file with full netpipe results for the onboard Intel thru a switch and two Netgear 302Ts back to back: http://aeolusres.homestead.com/files/np_bench.tar.gz Mike Prinkey Aeolus Research, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eccf at super.unam.mx Fri Sep 5 15:35:44 2003 From: eccf at super.unam.mx (Eduardo Cesar Cabrera Flores) Date: Fri, 5 Sep 2003 14:35:44 -0500 (CDT) Subject: Checkpoint? In-Reply-To: <200309051909.h85J9fw05503@NewBlue.Scyld.com> Message-ID: Is there any good SW to make a checkpoint in a cluster env for serial and parallel (mpi,pvm,OpenMP) application? thanks a lot cafe _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Fri Sep 5 15:53:20 2003 From: raysonlogin at yahoo.com (Rayson Ho) Date: Fri, 5 Sep 2003 12:53:20 -0700 (PDT) Subject: Checkpoint? In-Reply-To: Message-ID: <20030905195320.3145.qmail@web11404.mail.yahoo.com> http://www.checkpointing.org/ Rayson --- Eduardo Cesar Cabrera Flores wrote: > > Is there any good SW to make a checkpoint in a cluster env for serial > and > parallel (mpi,pvm,OpenMP) application? > > thanks a lot > > > cafe > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Fri Sep 5 17:31:36 2003 From: raysonlogin at yahoo.com (Rayson Ho) Date: Fri, 5 Sep 2003 14:31:36 -0700 (PDT) Subject: Mac clusters... Message-ID: <20030905213136.91079.qmail@web11406.mail.yahoo.com> 1100 dual G5s to build a supercomputer: http://www.computerweekly.com/articles/article.asp?liArticleID=124559&liFlavourID=1&sp=1 And this cluster is interesting too: --- "Hunt, Derek" wrote: > Hello all, this is in response to Charles D. Norton's post regarding > Xserve clusters. > > Site: Yale School Of Management > Yale University > Hardware Vendor: Apple Computer > Number of nodes: 45 > Number of processors: 90 > Total Memory: 90GB > Interconnect Technology: Gigabit Ethernet > OS: 10.2.6 with current patches > Comm Software: MPICH, Sun Gridengine > Application Software: GridMathematica, Matlab, C/C++, Fortran > Mail application area Scientific Computing: Financial Research, > Statistical Analysis > Year installed/upgraded: 2003 > > We are in the initial stages of deployment now. > > on a side note, I will be giving a talk on September 25th in > Rochester, > MN at the linux users group (more details at http://www.k-lug.org) if > anyone is interested in hearing the various trials we encountered > during > installation/deployment. > > - Derek __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Sat Sep 6 08:07:12 2003 From: deadline at plogic.com (Douglas Eadline) Date: Sat, 6 Sep 2003 08:07:12 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Fri, 5 Sep 2003, Michael T. Prinkey wrote: > > (note: read the paper > > http://www.cs.uni.edu/~gray/gig-over-copper/hsln-lcn.ps) > > > > I purchased two Netgear 302T nics for about $30 each. They are 32 bit > > cards that will run at 66 Mhz, have a Broadcom chipset, and use > > the tg3 driver. Here as some _preliminary_ results using netpipe(tcp) > > > > PCI Speed Latency Max Throughput > > 33 28 700 Mbits/sec (87.5 Mbytes/sec) > > 66 30 840 Mbits/sec (105 Mbytes/sec) > > > > The tests were done in dual PIII-1.266 Serverworks LE (Supermicro P3TDLE) > > using the kernel 2.40.20, 1500 byte MTU (they support Jumbos, however) > > While not as sexy as a PCI-X 64 bit card, at $30 a card, the performance > > is pretty good. > > > > Doug > > > > I will second this. I have been using Netgear 302Ts since late last year. > I have been using them on dual Xeon boards with on-board Intel Gbit NICs. > I run the on-board to a switch and use a pair of Netgears to build nearest > neighbor links. It makes for a very cost effective way to speed up CFD > runs that use 1D decompositions. The latency and bandwith of these cards > are very impressive considering their low cost and 32-bittedness. > > Here is a tar file with full netpipe results for the onboard Intel thru a > switch and two Netgear 302Ts back to back: > > http://aeolusres.homestead.com/files/np_bench.tar.gz Mike, a few questions, if you do not mind kernel and driver versions, model of switch, motherboard? Have you run the 302t's thought the switch or the intel's as xover? Just curious. Sounds like you are have built this machine to solve a specific type of problem. How is it working? Doug > > Mike Prinkey > Aeolus Research, Inc. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From iosephus at sgirmn.pluri.ucm.es Sat Sep 6 10:48:31 2003 From: iosephus at sgirmn.pluri.ucm.es (=?iso-8859-1?Q?Jos=E9_M=2E_P=E9rez_S=E1nchez?=) Date: Sat, 06 Sep 2003 16:48:31 +0200 Subject: Processor technology choice Message-ID: <20030906144831.GA1100@sgirmn.pluri.ucm.es> Hi people: We are planning to install a small cluster (around 12 processors). We mainly are going to do pattern recognition, and will run applications using artificial neural-network and genetic algorithms. We want to put around 6 dual processor machines running GNU/Linux with OpenMosix, and MPI on top of it. One server with SCSI storage and the rest of the nodes will be diskless. The main design uncertainty we have is what processor technology to use. Our choices are Intel Xeon, AMD Athlon XP/MP and Opteron. We are particularly interested in Opteron even if we have to put a couple of nodes less. My questions are: - Will we have some computing power gain using Opteron with the same budget? - What about long term maintenance cost, for how long will be 32 bit technology components generally available at normal price, now that we have Opteron and Itanium? - Most vendors say it's better to buy Intel (I personally think AMD are equivalent and cheaper), are they just trying to sell the most expensive? Any known real trouble with cooling for AMDs, or just bad case design? - We plan to use Debian, anyone with experience on it for 64 bit platforms. Thanks in advance, Jose M. P?rez. Madrid. Spain. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Sat Sep 6 12:57:17 2003 From: rossini at blindglobe.net (A.J. Rossini) Date: Sat, 06 Sep 2003 09:57:17 -0700 Subject: Processor technology choice In-Reply-To: <20030906144831.GA1100@sgirmn.pluri.ucm.es> (Jos's message of "Sat, 06 Sep 2003 16:48:31 +0200") References: <20030906144831.GA1100@sgirmn.pluri.ucm.es> Message-ID: <851xutonwy.fsf@blindglobe.net> Jos? M. P?rez S?nchez writes: > - We plan to use Debian, anyone with experience on it for 64 bit platforms. With respect to the Opteron, better review the Debian-AMD64 archives. It'll be a while (next year? longer) before it's really up and running and tested (esp for 32/64 combos). Just to make sure it'll be where you want it to be by the time you are ready to roll... best, -tony -- A.J. Rossini rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW : FAX=206-543-3461 | moving soon to a permanent office FHCRC: 206-667-7025 FAX=206-667-4812 | Voicemail is pretty sketchy/use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Sat Sep 6 12:23:25 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Sat, 6 Sep 2003 11:23:25 -0500 (CDT) Subject: Processor technology choice In-Reply-To: <20030906144831.GA1100@sgirmn.pluri.ucm.es> Message-ID: Opteron prices are on par with the Xeon's. I see no reason for anyone, anywhere to ever buy Athlon MP's ever again. The processor core is almost identical to the Opterons except that the Opterons have the 64-bit extensions. The opterons get a great benefit from the Hypertransport bus. They have great memory bandwidth and interprocess latency. Debain for amd64 is still under development. I occasionaly talk with the guys who are doing the port, and I dont think it is quite ready for prime-time yet. If you really know what you are doing, you can probably make it work, but it is not going to be trivial. Debian would be great on Xeon/Athlon/Itainium2 though. the Opteron's will give you a bit less potential raw FLOPS, but you will most likely achieve a greater effiency from those processors (versus Xeon) due to the faster memory bus. The best way is to test your code on each -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' On Sat, 6 Sep 2003, Jos? M. P?rez S?nchez wrote: > Hi people: > > We are planning to install a small cluster (around 12 processors). > We mainly are going to do pattern recognition, and will run > applications using artificial neural-network and genetic algorithms. > > We want to put around 6 dual processor machines running GNU/Linux with > OpenMosix, and MPI on top of it. One server with SCSI storage and the > rest of the nodes will be diskless. > > The main design uncertainty we have is what processor technology to use. > Our choices are Intel Xeon, AMD Athlon XP/MP and Opteron. We are > particularly interested in Opteron even if we have to put a couple of > nodes less. My questions are: > > - Will we have some computing power gain using Opteron with the same > budget? > - What about long term maintenance cost, for how long will be 32 bit > technology components generally available at normal price, now that we > have Opteron and Itanium? > - Most vendors say it's better to buy Intel (I personally think AMD are > equivalent and cheaper), are they just trying to sell the most > expensive? Any known real trouble with cooling for AMDs, or just bad > case design? > - We plan to use Debian, anyone with experience on it for 64 bit platforms. > > Thanks in advance, > > Jose M. P?rez. > Madrid. Spain. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mprinkey at aeolusresearch.com Sat Sep 6 12:30:19 2003 From: mprinkey at aeolusresearch.com (Michael T. Prinkey) Date: Sat, 6 Sep 2003 12:30:19 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Sat, 6 Sep 2003, Douglas Eadline wrote: > Mike, a few questions, if you do not mind kernel and driver versions, > model of switch, motherboard? 2.4.18 for those results, I believe. I saw about the same with 2.4.19. I haven't tested 2.4.20. I used the tg3 drivers as packaged in the kernel. I used the e1000 drivers for the onboards. I think the motherboards were INTEL SE7500CW2. CPUs were 2.4 GHz Xeons. The switch was a DLink DGS1016T. I have built 3 or 4 clusters using similar configurations. More recent ones have used the Tyan Tiger i7501 motherboard instead. > > Have you run the 302t's thought the switch or the intel's > as xover? Just curious. > No. I only had time to benchmark while I was burning the units in and didn't take the time to rewire the network. Wanted to, but didn't find time. > Sounds like you are have built this machine to solve a > specific type of problem. How is it working? > All of the clusters are running a commercial CFD package (the big one!) and are typically used for relatively small runs (2 to 16 CPUs). For these runs, principle axis decomposition is the default and about 98% of the traffic crosses the low-latency nearest neighbor links. For this application, I think such a configuration is near optimal. One day I hope to have a long enough lead time on delivery to actually benchmark with and without the xover links. For now, my only performance metric is anecdotal...I have several satisfied customers. Best, Mike Prinkey Aeolus Research, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sun Sep 7 07:57:50 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sun, 7 Sep 2003 13:57:50 +0200 (CEST) Subject: Processor technology choice In-Reply-To: Message-ID: Re. the Opteron, I'm getting experience now with installing/running them. I've been looking forward to this ever since a talk I heard by Bo Thorsen, then of SusE, a good couple of years ago. Currently running kernel 2.4.22 on them, and Sun Gridengine as the queuing system. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Sun Sep 7 14:39:13 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 07 Sep 2003 13:39:13 -0500 Subject: request for long running parallel applications. In-Reply-To: References: Message-ID: <1062959953.21346.106.camel@terra> On Sun, 2003-09-07 at 12:57, Timothy M Collins wrote: > Dear all, > I need long running parallel application(s) that I can using for testing on > my beowulf cluster. Possibly something that runs in a loop. (small or large) > Please help. Ideas well come. > regards > Collins > Try something like NAMD (http://www.ks.uiuc.edu/Research/namd/) which is a molecular dynamics code. There are benchmarks for it and you can make it run long by bumping up the timesteps of the benchmark. It will eat as much horsepower as you can throw at it. -- -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dmcollins79 at hotmail.com Sun Sep 7 13:57:02 2003 From: dmcollins79 at hotmail.com (Timothy M Collins) Date: Sun, 07 Sep 2003 18:57:02 +0100 Subject: request for long running parallel applications. Message-ID: Dear all, I need long running parallel application(s) that I can using for testing on my beowulf cluster. Possibly something that runs in a loop. (small or large) Please help. Ideas well come. regards Collins http://www.lanideas.com London. N17 _________________________________________________________________ Hotmail messages direct to your mobile phone http://www.msn.co.uk/msnmobile _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Mon Sep 8 00:02:20 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Mon, 8 Sep 2003 00:02:20 -0400 (EDT) Subject: Processor technology choice In-Reply-To: Message-ID: > Debian would be great on Xeon/Athlon/Itainium2 though. and opterons in 32b ("fast xeon") mode, I believe (no prob here with RH.) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From David_Walters at sra.com Mon Sep 8 09:04:40 2003 From: David_Walters at sra.com (Walters, David) Date: Mon, 8 Sep 2003 09:04:40 -0400 Subject: Mac clusters... Message-ID: <0EB5C81FE6FE5A4F8D1FEBF59C6C7BAA0A9FB7@durham.sra.com> Does anyone know where their funding came from? Did they choose Apple based on a cost/benefit analysis, or is Apple paying them? I am thinking back to the numbers that came out of the Cornell Theory Center here... Same scenario? Dave Walters Project Manager/Sr. Management Analyst, EPA IIASC SRA International, Inc. 919-474-4318 david_walters at sra.com -----Original Message----- From: Rayson Ho [mailto:raysonlogin at yahoo.com] Sent: Friday, September 05, 2003 5:32 PM To: beowulf Subject: Mac clusters... 1100 dual G5s to build a supercomputer: http://www.computerweekly.com/articles/article.asp?liArticleID=124559&liFlav ourID=1&sp=1 And this cluster is interesting too: --- "Hunt, Derek" wrote: > Hello all, this is in response to Charles D. Norton's post regarding > Xserve clusters. > > Site: Yale School Of Management > Yale University > Hardware Vendor: Apple Computer > Number of nodes: 45 > Number of processors: 90 > Total Memory: 90GB > Interconnect Technology: Gigabit Ethernet > OS: 10.2.6 with current patches > Comm Software: MPICH, Sun Gridengine > Application Software: GridMathematica, Matlab, C/C++, Fortran Mail > application area Scientific Computing: Financial Research, Statistical > Analysis Year installed/upgraded: 2003 > > We are in the initial stages of deployment now. > > on a side note, I will be giving a talk on September 25th in > Rochester, MN at the linux users group (more details at > http://www.k-lug.org) if anyone is interested in hearing the various > trials we encountered during > installation/deployment. > > - Derek __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Mon Sep 8 11:29:02 2003 From: deadline at plogic.com (Douglas Eadline) Date: Mon, 8 Sep 2003 11:29:02 -0400 (EDT) Subject: Guidance/suggestions for small (6-12 port) gigabit switches and gigabit NICs? In-Reply-To: Message-ID: On Sat, 6 Sep 2003, Michael T. Prinkey wrote: > On Sat, 6 Sep 2003, Douglas Eadline wrote: > > > Mike, a few questions, if you do not mind kernel and driver versions, > > model of switch, motherboard? > > 2.4.18 for those results, I believe. I saw about the same with 2.4.19. > I haven't tested 2.4.20. I used the tg3 drivers as packaged in the > kernel. I used the e1000 drivers for the onboards. I think the > motherboards were INTEL SE7500CW2. CPUs were 2.4 GHz Xeons. The > switch was a DLink DGS1016T. I have built 3 or 4 clusters using similar > configurations. More recent ones have used the Tyan Tiger i7501 > motherboard instead. > > > > > Have you run the 302t's thought the switch or the intel's > > as xover? Just curious. > > > > No. I only had time to benchmark while I was burning the units in and > didn't take the time to rewire the network. Wanted to, but didn't find > time. > > > Sounds like you are have built this machine to solve a > > specific type of problem. How is it working? > > > > All of the clusters are running a commercial CFD package (the big one!) > and are typically used for relatively small runs (2 to 16 CPUs). For > these runs, principle axis decomposition is the default and about 98% of > the traffic crosses the low-latency nearest neighbor links. For this > application, I think such a configuration is near optimal. One day I hope > to have a long enough lead time on delivery to actually benchmark with and > without the xover links. For now, my only performance metric is > anecdotal...I have several satisfied customers. That is what counts. > > Best, > > Mike Prinkey > Aeolus Research, Inc. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rkane at scs.carleton.ca Mon Sep 8 09:58:25 2003 From: rkane at scs.carleton.ca (Robert Kane) Date: Mon, 08 Sep 2003 09:58:25 -0400 Subject: CPUs for a Beowulf Message-ID: <3F5C8B01.5AD8BA4@scs.carleton.ca> Good morning, If anyone doesn't mind, may I ask a few questions. When given a specific application for which a cluster is being built it should be relatively simple to look are the requirements of the problem and the available hardware, and then determine which hardware solution is best for the problem. However, if the cluster is being built as a general purpose cluster for research, things become a bit more difficult, as (as I far as I can tell) there is no one answer. But, if anyone has any insight into the following problems it would be greatly appreciated. 1. Single versus Dual CPUs? Both of these choices have their pros and cons and are each best suited for different types of problems. Given that the cluster will be used for a variety of problems, is there one which would be a better choice? Is there a particular configuration for which the majority of problems will run better? Is there a solution that on average provides more performance per dollar? 2. CPU Type Intel and AMD's new 64-bit processors are finally beginning to become more common it appears. And from what I've seen the benchmarks are rather impressive. However, there seems to be a significant price increase going from previous generation chips (ie Xeon) to the new 64-bit chips. In general is the increased performance worth the money invested, or would a larger number of slower chips be effective cost/performance wise? Apart from the increased electricial, A/C costs of course. Thank you for any information concerning these issues, whether information be answers or links to good resources, Robert Kane _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From canon at nersc.gov Mon Sep 8 11:52:48 2003 From: canon at nersc.gov (canon at nersc.gov) Date: Mon, 08 Sep 2003 08:52:48 -0700 Subject: Processor technology choice In-Reply-To: Message from Rocky McGaugh of "Sat, 06 Sep 2003 11:23:25 CDT." Message-ID: <200309081552.h88FqmLu023230@pookie.nersc.gov> Rocky, Not to nit pick, but I think the first statement in your second paragraph and your last statement are at direct odds with each other. rocky at atipa.com said: > I see no reason for anyone, anywhere to ever buy Athlon MP's ever > again. rocky at atipa.com said: > The best way is to test your code on each When we benchmarked our codes and looked at price performance last month, we still found a slight edge to the Athlon MP's. This was too bad because we REALLY wanted an excuse to buy 70 Opterons. The reason was we made a few assumptions which are valid for our users, but would likely be untrue for others. 1. The codes would not likely be optimized for a specific platform. This would even include good choices of compiler flags. This obviously hurt the Opteron since you want to take advantage of the extra registers and some of the SSE/2 capabilities. Our users are part of a larger collaboration and they often value consistency over performance. 2. The codes don't look like they will pass the 2 GB barrier any time soon. So, while the 64 bit capability is cool, its not need...yet. Also, we are just struggling with how to move past RH 7.x, so moving to a 64 bit enabled version would be tough. This is for compatibility with the other collaborators, not because the transition itself is hard. We benchmarked on systems with dual 2200+ Athlon MP, 2.2 GHz Xeon, and Opteron 240. At the time of the benchmarks, the Opteron 240 was the only reasonably priced Opteron chip. We then assumed we would purchase 2600+, 2.6 GHz Xeon, or the Opteron 240. Even with out the clock adjustments, the Athlon and Xeon's were beating the Opteron. Of course if you turned on the right compiler flags, the Opteron fared much better. In our case the slightly higher clock speeds of the Athlons and Xeons offset the better efficiency of the Opteron. Obviously the faster Opteron's would have probably done much better, but would have cost 50% more per system. I think this will change very soon (if it hasn't already) as the Opteron CPUs drop in price, and I suspect our next big purchase will be Opteron based. So like Rocky said rocky at atipa.com said: > The best way is to test your code on each --Shane ------------------------------------------------------------------------ Shane Canon PSDF Project Lead National Energy Research Scientific Computing Center ------------------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Mon Sep 8 12:30:27 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Mon, 08 Sep 2003 09:30:27 -0700 Subject: CPUs for a Beowulf In-Reply-To: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: <5.2.0.9.2.20030908092157.018a8cf8@mailhost4.jpl.nasa.gov> RGB's book at the Duke Brahma site (I'm sure he'll post the URL) covers some of these tradeoffs.. You've posed an interesting question, because it's in the generic "what's the best way to get lowest dollars per instruction executed" way, but trickier.. It's that sticky word "performance" which is the problem. Wwe can all agree on what dollars mean and how much they are worth now, and in the future. However, performance is different things to different people. Is time worth money? (that is, is there a "wall clock time" as well as a CPU cycles aspect... Older computers are cheaper in terms of executing a particular number of instructions, but consume more support infrastructure (cooling, staff time, etc.) because, if nothing else, they have to run longer...) Is capital cost important, and, are intermediate results of interest.. There's a well known example where you have a "really big problem" that you could either spend some money now, and compute for the next two years, or save your money, wait a year, buy the (twice as fast) computers for the same price and do the computation then, finishing at the same time. Of course, you don't get any results during the first year, and for many applications, the partial results early are used to guide the later work. What's your particular labor/hardware maintenance/capital investment tradeoff.. If you have copious FREE and SKILLED labor, the trade is different... Likewise, if you have a "hard" reliability requirement and can't tolerate partial (or complete) downtime, the trade is different. Cluster computing, in some forms, also lends itself to "stealth, below-the-funding-watchdog-radar" work. You can buy, borrow, collect, etc., CPUs and gradually add them to an ever growing configuration. I notice though, that the whole cluster computing thing is a validated way to work, these days, and anyone with real work to do, and the budget to pay for it, just goes out and builds a real Beowulf. At 09:58 AM 9/8/2003 -0400, Robert Kane wrote: >Good morning, > > If anyone doesn't mind, may I ask a few questions. When given a >specific application for which a cluster is being built it should be >relatively simple to look are the requirements of the problem and the >available hardware, and then determine which hardware solution is best >for the problem. However, if the cluster is being built as a general >purpose cluster for research, things become a bit more difficult, as (as >I far as I can tell) there is no one answer. But, if anyone has any >insight into the following problems it would be greatly appreciated. > >1. Single versus Dual CPUs? > > Both of these choices have their pros and cons and are each best >suited for different types of problems. Given that the cluster will be >used for a variety of problems, is there one which would be a better >choice? Is there a particular configuration for which the majority of >problems will run better? Is there a solution that on average provides >more performance per dollar? > >2. CPU Type > > Intel and AMD's new 64-bit processors are finally beginning to become >more common it appears. And from what I've seen the benchmarks are >rather impressive. However, there seems to be a significant price >increase going from previous generation chips (ie Xeon) to the new >64-bit chips. In general is the increased performance worth the money >invested, or would a larger number of slower chips be effective >cost/performance wise? Apart from the increased electricial, A/C costs >of course. > > >Thank you for any information concerning these issues, whether >information be answers or links to good resources, > >Robert Kane >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Sep 8 13:00:59 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 8 Sep 2003 13:00:59 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <5.2.0.9.2.20030908092157.018a8cf8@mailhost4.jpl.nasa.gov> Message-ID: On Mon, 8 Sep 2003, Jim Lux wrote: > RGB's book at the Duke Brahma site (I'm sure he'll post the URL) covers > some of these tradeoffs.. http://www.phy.duke.edu/brahma But I think that Jim is optimistic that it does any better than his own stuff below. It hasn't been updated to take new architectural improvments into account, and I REALLY need to add more infrastructure stuff there as well. However, other links on the brahma site and my personal home page do address infrastructure at least some, as does an article I wrote for Linux Magazine back in June. Not so much single vs dual per se, but in general. Jim's refocussing your energy on overall infrastructure rather than CPU architecture per se is dead on the money, though. rgb > > You've posed an interesting question, because it's in the generic "what's > the best way to get lowest dollars per instruction executed" way, but > trickier.. > > It's that sticky word "performance" which is the problem. Wwe can all > agree on what dollars mean and how much they are worth now, and in the > future. However, performance is different things to different people. > > Is time worth money? (that is, is there a "wall clock time" as well as a > CPU cycles aspect... Older computers are cheaper in terms of executing a > particular number of instructions, but consume more support infrastructure > (cooling, staff time, etc.) because, if nothing else, they have to run > longer...) > > Is capital cost important, and, are intermediate results of interest.. > There's a well known example where you have a "really big problem" that you > could either spend some money now, and compute for the next two years, or > save your money, wait a year, buy the (twice as fast) computers for the > same price and do the computation then, finishing at the same time. Of > course, you don't get any results during the first year, and for many > applications, the partial results early are used to guide the later work. > > What's your particular labor/hardware maintenance/capital investment > tradeoff.. If you have copious FREE and SKILLED labor, the trade is > different... Likewise, if you have a "hard" reliability requirement and > can't tolerate partial (or complete) downtime, the trade is different. > > > Cluster computing, in some forms, also lends itself to "stealth, > below-the-funding-watchdog-radar" work. You can buy, borrow, collect, > etc., CPUs and gradually add them to an ever growing configuration. I > notice though, that the whole cluster computing thing is a validated way to > work, these days, and anyone with real work to do, and the budget to pay > for it, just goes out and builds a real Beowulf. > > > At 09:58 AM 9/8/2003 -0400, Robert Kane wrote: > >Good morning, > > > > If anyone doesn't mind, may I ask a few questions. When given a > >specific application for which a cluster is being built it should be > >relatively simple to look are the requirements of the problem and the > >available hardware, and then determine which hardware solution is best > >for the problem. However, if the cluster is being built as a general > >purpose cluster for research, things become a bit more difficult, as (as > >I far as I can tell) there is no one answer. But, if anyone has any > >insight into the following problems it would be greatly appreciated. > > > >1. Single versus Dual CPUs? > > > > Both of these choices have their pros and cons and are each best > >suited for different types of problems. Given that the cluster will be > >used for a variety of problems, is there one which would be a better > >choice? Is there a particular configuration for which the majority of > >problems will run better? Is there a solution that on average provides > >more performance per dollar? > > > >2. CPU Type > > > > Intel and AMD's new 64-bit processors are finally beginning to become > >more common it appears. And from what I've seen the benchmarks are > >rather impressive. However, there seems to be a significant price > >increase going from previous generation chips (ie Xeon) to the new > >64-bit chips. In general is the increased performance worth the money > >invested, or would a larger number of slower chips be effective > >cost/performance wise? Apart from the increased electricial, A/C costs > >of course. > > > > > >Thank you for any information concerning these issues, whether > >information be answers or links to good resources, > > > >Robert Kane > >_______________________________________________ > >Beowulf mailing list, Beowulf at beowulf.org > >To change your subscription (digest mode or unsubscribe) visit > >http://www.beowulf.org/mailman/listinfo/beowulf > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Sep 8 12:44:42 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 8 Sep 2003 12:44:42 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: On Mon, 8 Sep 2003, Robert Kane wrote: > Good morning, > > If anyone doesn't mind, may I ask a few questions. When given a > specific application for which a cluster is being built it should be > relatively simple to look are the requirements of the problem and the > available hardware, and then determine which hardware solution is best > for the problem. However, if the cluster is being built as a general > purpose cluster for research, things become a bit more difficult, as (as > I far as I can tell) there is no one answer. But, if anyone has any > insight into the following problems it would be greatly appreciated. > > 1. Single versus Dual CPUs? > > Both of these choices have their pros and cons and are each best > suited for different types of problems. Given that the cluster will be > used for a variety of problems, is there one which would be a better > choice? Is there a particular configuration for which the majority of > problems will run better? Is there a solution that on average provides > more performance per dollar? You're going to get a lot of "it depends" answers because it does. However: a) Historically dual packaging is marginally very slightly cheaper, per CPU, than single packaging. The difference is pretty much the duplicated parts -- two cases instead of one, two power supplies, two hard disks. So if your programs are expected to be purely CPU bound, you'll get a bit more CPU for your dollar in dual packaging. b) OTOH if your task(s) are likely to be memory bound, dual packagings typically oversubscribe the memory bus. That is if both CPUs are trying to read/write to memory as fast as they can, one will often be blocked waiting for the other. c) This latter problem projects down to other resources as well. For example two CPUs might end up sharing a single NIC, or two NICs might end up sharing a bus. Anywhere you have a memory/speed hierarchy that is shared by the CPUs on a dual but that has its own private resource on a single, you can end up with one CPU/task blocking while waiting for the other to free the resource. In many cases the problems described in b and c are minimal or relatively rare. In others they can significantly degrade performance. I think that's the most that can be said without a deeper knowledge of the problem space and the rest of your architecture, e.g. the network, the memory we're talking about, the CPU's, the overall architecture. With both 32 and 64 bit architectures and with a variety of FSB and memory and CPU speeds these days, something that might be a problem with one architecture might be perfectly ok with a different one, so just "single" and "dual" aren't enough data to answer your question, if they ever were. You'll probably have to do some sort of cost benefit analysis with SOME sort of idea of the boundaries of your problem space to proceed. For example, you might select duals knowing they won't scale perfectly for certain problems, because they'll outperform (dollarwise) on the others. Or you might go with singles to get the best scaling on the former, at the expense of the latter. These days the marginal cost difference isn't so great that either path is likely to be a huge win or loss, and it may be other considerations such as CPU density in a rackmount that become more important than performance per se. > 2. CPU Type > > Intel and AMD's new 64-bit processors are finally beginning to become > more common it appears. And from what I've seen the benchmarks are > rather impressive. However, there seems to be a significant price > increase going from previous generation chips (ie Xeon) to the new > 64-bit chips. In general is the increased performance worth the money > invested, or would a larger number of slower chips be effective > cost/performance wise? Apart from the increased electricial, A/C costs > of course. This is plain old impossible to answer without a knowledge of the problem space. Never one to let that stop me, let me pronounce that at the moment AMD's 64 bit chips are probably worth considering, and Intel's are not. Yet. And I could be behind the times on the yet, as well. Note that I say considering. I honestly think that the only way to answer a question like this is to get loaners of two or three candidate systems and run benchmarks. There are also additional costs outside of the raw hardware to consider, in particular a certain degree of weakness in mainline linux distribution support for the 64 bit systems. You're a bit closer to the beta level there and might well end up "paying" a bit extra in screwing-around-with-crap costs administratively to get the full benefit of the CPUs for a while yet. Again, though, the cost differentials are getting so low for AMD's that their 64 bit systems don't look TOO horrible run as 32 bit systems, and the OS situation can only improve. There is also little doubt that 64 bit support will in fact come to pass -- I know some folks that got eaten alive by Alphas when they bought them for their speed and had to deal with it when their OS support more or less evaporated, leaving them struggling and burning admin FTE with simple issues like scalable installation and mandatory security upgrades of important packages. Administrative costs can easily be THE dominant cost in a public cluster of the sort you describe, with benefits that are difficult to quantify. This tends to bias one towards conservative solutions more or less guaranteed to minimize human management time, which in turn biases one towards older "guaranteed to work" hardware, straight off the shelf commercial linux distros (or a supported cluster package like Scyld at moderate expense in dollars but presumed savings in human time), hardware from a vendor that does on site service as part of the up-front cost, and things like PXE-capable NICs, kickstart, yum. 64 bit systems would, I think, require a bit more human effort and skill in the administrative chain to make work. rgb > > > Thank you for any information concerning these issues, whether > information be answers or links to good resources, > > Robert Kane > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Sep 8 14:18:17 2003 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 8 Sep 2003 20:18:17 +0200 (CEST) Subject: CPUs for a Beowulf In-Reply-To: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: On Mon, 8 Sep 2003, Robert Kane wrote: > 2. CPU Type > > 64-bit chips. In general is the increased performance worth the money > invested, or would a larger number of slower chips be effective > cost/performance wise? Hmmmmm. Say, didn't a bunch of folks at a NASA lab do that a few years ago? Big :-) as I am slightly pulling your leg, but hopefully in a nice wayB. That is the beauty of the Beowulf idea - use cheap as chips stuff. Your point is well made. Seriously though, you might find that 64bit AMDs aren't that expensive. I think as somone pointed out recently that requiring large amounts of RAM starts to dominate the costs these days. Very much IMHO. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From iosephus at sgirmn.pluri.ucm.es Mon Sep 8 16:20:22 2003 From: iosephus at sgirmn.pluri.ucm.es (=?iso-8859-1?Q?Jos=E9_M=2E_P=E9rez_S=E1nchez?=) Date: Mon, 08 Sep 2003 22:20:22 +0200 Subject: SCSI Card Message-ID: <20030908202022.GC12233@sgirmn.pluri.ucm.es> Hi, Has anyone tested the UW-320 SCSI card Adaptec 29320-R? The strongest candidate to be purchased as cluster master node for our cluster comes with that card and 10000 RPM Fujitsu hard disks. I took a look at the Adaptec page and they say it's not RAID capable on linux. Shouldn't be hardware controlled RAID OS independent, as long as you have support for the card? What's the driver module for that card? AHA-something, AIC-something? Regards, Jose M. Perez. Madrid. Spain. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Sep 8 17:00:40 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 8 Sep 2003 17:00:40 -0400 (EDT) Subject: SCSI Card In-Reply-To: <20030908202022.GC12233@sgirmn.pluri.ucm.es> Message-ID: On Mon, 8 Sep 2003, [iso-8859-1] Jos? M. P?rez S?nchez wrote: > Hi, > > Has anyone tested the UW-320 SCSI card Adaptec 29320-R? The strongest > candidate to be purchased as cluster master node for our cluster comes with > that card and 10000 RPM Fujitsu hard disks. > > I took a look at the Adaptec page and they say it's not RAID capable on > linux. Shouldn't be hardware controlled RAID OS independent, as long as > you have support for the card? md (software) raid should work on anything, and works remarkably well on most things. We use it in large and small scale production and it performs like a champion. I've run a system six months in degraded mode at home (couldn't find time to go get a replacement disk) and still lost no data. RH's Kickstart can even kickstart install a ready-to-run md raid. As long as the card itself works under linux, building a RAID should be no problem. Hardware RAIDs often do require drivers (or that you work the RAID through a BIOS layer preboot) and leave you at the mercy of the RAID firmware designers as far as function and recovery are concerned. Adaptec (after a somewhat rocky start some years ago) has been pretty good about linux support so I'd guess drivers will be rapidly forthcoming, unless they are trying to pull the "proprietary" trick with a binary only driver (a really, really bad idea with something like a RAID controller). rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Mon Sep 8 17:51:17 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Mon, 8 Sep 2003 14:51:17 -0700 (PDT) Subject: SCSI Card In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Robert G. Brown wrote: > On Mon, 8 Sep 2003, [iso-8859-1] Jos? M. P?rez S?nchez wrote: > > > Hi, > > > > Has anyone tested the UW-320 SCSI card Adaptec 29320-R? The strongest > > candidate to be purchased as cluster master node for our cluster comes with > > that card and 10000 RPM Fujitsu hard disks. the 29320-r comes with a driver for windows that provides what they call hostraid support. something like an adaptec 2120 or 2200s is really a hardware raid controller with a full feature-set (uses aacraid rather than aic7xxx). linux will support this card but not it's partial raid functionality. linux software raid will happily work on top of all of this and probably run faster on a raid5 stripe than the 2120 to boot. the 29320-r doens't support raid-5 at all reagrdless of os. > > I took a look at the Adaptec page and they say it's not RAID capable on > > linux. Shouldn't be hardware controlled RAID OS independent, as long as > > you have support for the card? > > md (software) raid should work on anything, and works remarkably well on > most things. We use it in large and small scale production and it > performs like a champion. I've run a system six months in degraded mode > at home (couldn't find time to go get a replacement disk) and still lost > no data. RH's Kickstart can even kickstart install a ready-to-run md > raid. As long as the card itself works under linux, building a RAID > should be no problem. > > Hardware RAIDs often do require drivers (or that you work the RAID > through a BIOS layer preboot) and leave you at the mercy of the RAID > firmware designers as far as function and recovery are concerned. > Adaptec (after a somewhat rocky start some years ago) has been pretty > good about linux support so I'd guess drivers will be rapidly > forthcoming, unless they are trying to pull the "proprietary" trick with > a binary only driver (a really, really bad idea with something like a > RAID controller). > > rgb > > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 8 19:09:46 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 9 Sep 2003 09:09:46 +1000 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <200309090909.47342.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > Seriously though, you might find that 64bit AMDs aren't that expensive. My understanding is that memory bandwidth with Opterons scales with the number of CPUs due to the HyperTransport architecture, which could be a big plus to some people. I'd be interested to hear whether that turns out to be correct or not in practice! Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/XQw6O2KABBYQAh8RAo1DAJwIIoMRQEySoCGAPbjV8ZAsAR2FdQCfbUCh MSrYfESYyD7o9PEOFbsStl8= =thrl -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Mon Sep 8 21:31:30 2003 From: bill at math.ucdavis.edu (Bill Broadley) Date: Mon, 8 Sep 2003 18:31:30 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> References: <200309090909.47342.csamuel@vpac.org> Message-ID: <20030909013130.GA29417@sphere.math.ucdavis.edu> On Tue, Sep 09, 2003 at 09:09:46AM +1000, Chris Samuel wrote: > On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > > > Seriously though, you might find that 64bit AMDs aren't that expensive. > > My understanding is that memory bandwidth with Opterons scales with the number > of CPUs due to the HyperTransport architecture, which could be a big plus to > some people. This is true. > I'd be interested to hear whether that turns out to be correct or not in > practice! http://www.math.ucdavis.edu/~bill/dual-240-4xpc2700-icc.png http://www.math.ucdavis.edu/~bill/4x842-icc.png I suspect if I used the latest kernel tweaks to pin processes to a specific cpu and allocate only local memory the scaling would be even better. Oh, to compare to a P4 (or dual athlon): http://www.math.ucdavis.edu/~bill/dual-p4-2.4-icc.png -- Bill Broadley Mathematics UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Mon Sep 8 21:44:49 2003 From: becker at scyld.com (Donald Becker) Date: Mon, 8 Sep 2003 21:44:49 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> Message-ID: On Tue, 9 Sep 2003, Chris Samuel wrote: > On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > > > Seriously though, you might find that 64bit AMDs aren't that expensive. > > My understanding is that memory bandwidth with Opterons scales with > the number of CPUs due to the HyperTransport architecture, which > could be a big plus to some people. Hmmm, I would describe this a little differently. Each Opteron processor has an on-die memory controller and full bandwidth to its local memory banks. The only external connection is through Hyper Transport (HT) channels. If you want more memory banks, and more bandwidth, you have to add more memory controllers through HT. And for now, the only way to get another memory controller is to buy the one on the same die as... another Opteron. Again HyperTransport is the only way to connect the Opteron to other processors and, I/O busses Only some of the HT channels can connect CPUs -- this first type is more complex than the I/O-only HT channels. Note that the additional CPUs/memory controllers can only attach to the "better" HT channels. Now for the software issue. For a few processors, the HT-based cache coherency is fast enough to be ignored. But for larger systems the only way avoid a big latency hit from the coherency traffic is to minimize the coherency traffic by having only the applications utilize it. That means running the kernel from local memory, sharing only the locks and data structures needed for global I/O. Gee, suddenly the system starts looking like a cluster. Sure, the application can now use global shared memory, and it's now efficient to implement a single file system view. But all the approaches to making a cluster efficiently scalable can be directly applied to this type of SMP... -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 8 22:34:53 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 9 Sep 2003 12:34:53 +1000 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> References: <200309090909.47342.csamuel@vpac.org> Message-ID: <200309091234.59961.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 9 Sep 2003 09:09 am, Chris Samuel wrote: > I'd be interested to hear whether that turns out to be correct or not in > practice! Bill, Donald, Thanks very much for that useful feedback, much appreciated! Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/XTxSO2KABBYQAh8RAkgDAJ4nW2cOF4x0jKKc9M9wH84pGNSd5ACghg/i v1y+exu3FLkJd8YoblzIeNE= =ql/2 -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jim at ks.uiuc.edu Mon Sep 8 22:47:42 2003 From: jim at ks.uiuc.edu (Jim Phillips) Date: Mon, 8 Sep 2003 21:47:42 -0500 (CDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Donald Becker wrote: > For a few processors, the HT-based cache coherency is fast enough to be > ignored. But for larger systems the only way avoid a big latency hit > from the coherency traffic is to minimize the coherency traffic by > having only the applications utilize it. That means running the kernel > from local memory, sharing only the locks and data structures needed for > global I/O. > > Gee, suddenly the system starts looking like a cluster. Sure, the > application can now use global shared memory, and it's now efficient to > implement a single file system view. But all the approaches to making a > cluster efficiently scalable can be directly applied to this type of > SMP... If you could connect those HyperTransport channels into a scalable mesh then this sounds a lot like the good old Cray T3E design. Drool... It's too bad that Red Storm won't be taking that route. The T3E was so nice. -Jim _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Tue Sep 9 00:02:50 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Tue, 09 Sep 2003 00:02:50 -0400 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <1063080170.10843.108.camel@protein.scalableinformatics.com> One of the interesting aspects of the HT is that it makes it a good building block for bigger systems. Of course you have to be careful what you ask for ... The HT design reminds me of the old SGI XBOW internal fabric. Also formerly known by the (marketing/misleading) label Cray-link. Now called NUMA-link or something like that. On Mon, 2003-09-08 at 22:47, Jim Phillips wrote: > On Mon, 8 Sep 2003, Donald Becker wrote: > > > For a few processors, the HT-based cache coherency is fast enough to be > > ignored. But for larger systems the only way avoid a big latency hit > > from the coherency traffic is to minimize the coherency traffic by > > having only the applications utilize it. That means running the kernel > > from local memory, sharing only the locks and data structures needed for > > global I/O. > > > > Gee, suddenly the system starts looking like a cluster. Sure, the > > application can now use global shared memory, and it's now efficient to > > implement a single file system view. But all the approaches to making a > > cluster efficiently scalable can be directly applied to this type of > > SMP... I remember the early days of trying to get applications to scale on SGI Origins. The issues that Don alludes to here were largely present there. Some of the big issues I remember were memory hotspot identification, page migration thrashing, and a host of other fun items. Customers complained when applications took non-deterministic time to execute due to odd physical->virtual page layouts. Sure, go-ahead and malloc that 20 GB. Hope it doesnt all land on 10 processors... I remember playing with DPLACE directives on benchmarks to tweak this stuff. > If you could connect those HyperTransport channels into a scalable mesh > then this sounds a lot like the good old Cray T3E design. Drool... It's > too bad that Red Storm won't be taking that route. The T3E was so nice. You could build inexpensive Origin's or similar out of these things. You just need the OS/compiler support. That is unfortunately not a trivial thing. > > -Jim -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 9 01:06:28 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 9 Sep 2003 01:06:28 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Jim Phillips wrote: > > Gee, suddenly the system starts looking like a cluster. Sure, the > > application can now use global shared memory, and it's now efficient to > > implement a single file system view. But all the approaches to making a > > cluster efficiently scalable can be directly applied to this type of > > SMP... > > If you could connect those HyperTransport channels into a scalable mesh > then this sounds a lot like the good old Cray T3E design. Drool... It's > too bad that Red Storm won't be taking that route. The T3E was so nice. For those unfamiliar with the history and relationship The Cray T3D/T3E used Alphas, EV4 and EV5 generations HT was designed as the communication architecture for the EV7+ generation Alpha processor. AMD bought the HT technology from API, Alpha Processor Inc., later API Networks just before shutting down. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Tue Sep 9 02:37:35 2003 From: john.hearns at clustervision.com (John Hearns) Date: Tue, 9 Sep 2003 08:37:35 +0200 (CEST) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <200309090909.47342.csamuel@vpac.org> Message-ID: On Tue, 9 Sep 2003, Chris Samuel wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tue, 9 Sep 2003 04:18 am, John Hearns wrote: > > > Seriously though, you might find that 64bit AMDs aren't that expensive. > > My understanding is that memory bandwidth with Opterons scales with the number > of CPUs due to the HyperTransport architecture, which could be a big plus to > some people. > Not really an answer to your question, but I found this article on the Hammer architecture (from DEcember last year). http://www.digit-life.com/articles2/amd-hammer-family/index.html I think it is good - has discussionof of memory timings, caches, HyperTransport etc. Worth a read. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Tue Sep 9 02:43:31 2003 From: john.hearns at clustervision.com (John Hearns) Date: Tue, 9 Sep 2003 08:43:31 +0200 (CEST) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: On Mon, 8 Sep 2003, Jim Phillips wrote: > On Mon, 8 Sep 2003, Donald Becker wrote: > > If you could connect those HyperTransport channels into a scalable mesh > then this sounds a lot like the good old Cray T3E design. Drool... It's > too bad that Red Storm won't be taking that route. The T3E was so nice. The article I quoted from Digit Life floats the idea that AMD may be able to make for Cray an Opteron with 4 Hypertransports, making a 3D mesh possible. Is this just supposition? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Tue Sep 9 04:31:38 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Tue, 09 Sep 2003 10:31:38 +0200 Subject: CPUs for a Beowulf In-Reply-To: References: <3F5C8B01.5AD8BA4@scs.carleton.ca> Message-ID: <20030909103138Q.hanzl@unknown-domain> > Administrative costs can easily be THE dominant cost Yes, yes, yes. Anybody has different experience for his/her FIRST cluster? (If somebody does, he is genius and his labor has much higher value then estimated and I am right anyway :-) One year later the price of your hardware may be say half the price today. Divide this halve by 365 to get an idea how much costs each day of problems. Consider appropriate cost of human resources to counterfight this loss... If you want to get most power for the money you have, make it simple, do choices which make the system simple for you. If you want to make research how OTHERS could save, that is another story... (And that is what many people sometimes unwillingly do on this list, including myself :) Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Tue Sep 9 09:15:39 2003 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Tue, 9 Sep 2003 10:15:39 -0300 (ART) Subject: SSH is default(SuSE 8.2) Message-ID: <20030909131539.50851.qmail@web12206.mail.yahoo.com> I install the SUSE 8.2 distro with the mpich package that came with it. I made all the configurantion to use rsh, but when run a example application, it try to use ssh. What can I do to change it. _______________________________________________________________________ Desafio AntiZona: participe do jogo de perguntas e respostas que vai dar um Renault Clio, computadores, c?meras digitais, videogames e muito mais! www.cade.com.br/antizona _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Tue Sep 9 09:27:46 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Tue, 9 Sep 2003 09:27:46 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: Message-ID: > > If you could connect those HyperTransport channels into a scalable mesh > > then this sounds a lot like the good old Cray T3E design. Drool... It's > > too bad that Red Storm won't be taking that route. The T3E was so nice. > > The article I quoted from Digit Life floats the idea that AMD may be able > to make for Cray an Opteron with 4 Hypertransports, making a 3D mesh > possible. Is this just supposition? it would be astonishing if AMD hadn't prototyped this already. one sticking point is that the current architecture is based on broadcast for managing coherence. a big-scalable approach would need to be directory-based, perhaps assisted by the OS. I dimly recall that the current packet format has just three bits for node ID. I have an even more dim recollection that current HT is limited to 20cm; perhaps that would be enough for a non-torus mesh. but, as Dell points out, the market for such things is rather small. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rbw at ahpcrc.org Tue Sep 9 10:48:18 2003 From: rbw at ahpcrc.org (Richard Walsh) Date: Tue, 9 Sep 2003 09:48:18 -0500 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) Message-ID: <200309091448.h89EmIQ14373@mycroft.ahpcrc.org> On Tue Sep 9 2003, John Hearns wrote: >On Mon, 8 Sep 2003, Jim Phillips wrote: > >> On Mon, 8 Sep 2003, Donald Becker wrote: >> >> If you could connect those HyperTransport channels into a scalable mesh >> then this sounds a lot like the good old Cray T3E design. Drool... It's >> too bad that Red Storm won't be taking that route. The T3E was so nice. > >The article I quoted from Digit Life floats the idea that AMD may be able >to make for Cray an Opteron with 4 Hypertransports, making a 3D mesh >possible. Is this just supposition? Mmm ... with just 3 HT links per chip (one of which is attenutated .. IO) I don't see how you can do a full 2D torus let alone a 3D which requires 6 per node. Perhaps mesh here does not refer to the T3E's torus interconnect? The 8-way SMP Opteron layouts that I have seen have edges. Not withstanding this, Cray is custom engineering an MPP system with the Opteron as the base microprocessor and a proprietary interconnect for Sandia(?). Anyone have more detail on this machine? Regards, rbw #--------------------------------------------------- # Richard Walsh # Project Manager, Cluster Computing, Computational # Chemistry and Finance # netASPx, Inc. # 1200 Washington Ave. So. # Minneapolis, MN 55415 # VOX: 612-337-3467 # FAX: 612-337-3400 # EMAIL: rbw at networkcs.com, richard.walsh at netaspx.com # rbw at ahpcrc.org # #--------------------------------------------------- # "Why waste time learning when ignornace is # instantaneous?" -Thomas Hobbes #--------------------------------------------------- # "Life is lived between two opposing poles, the one # defines that which must be true (truism) and the # other that which is true (truth). -Max Headroom #--------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 9 12:06:40 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 9 Sep 2003 12:06:40 -0400 (EDT) Subject: Baltimore-Washington Beowulf Users Group meeting Sept. 9, 2003 Message-ID: http://BWBUG.org The September BWBUG meeting will be at the Northrop Grumman location: 7501 Greenway Center Drive, Suite 1000 Greenbelt , Maryland. on 9 Sept 2003 at 3:00 PM. Note that the fall series is continuing the spring series meeting pattern. We will be alternating venues between Greenbelt and McLean. This month's meeting is in GREENBELT. Steve Duchene Beowulf, design expert at Freddie Mac, will be our quest speaker to discuss and to demonstrate the Oscar Cluster Management Software. Steve has spoken at a number of Beowulf Cluster events such as the Large Scale Cluster Computing Workshop" at FERMILAB on May 2001. Steve was the key participate as the Atlanta Enthusiasts Representative at the 1997 Linux Expo in Atlanta. Steve was the principle design engineer at VA Linux responsible for their Beowulf Clusters. Mike Fitzmaurice http://BWBUG.org 703-205-3132 -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Tue Sep 9 13:36:34 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Tue, 9 Sep 2003 10:36:34 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: <200309090909.47342.csamuel@vpac.org> Message-ID: <20030909173634.GA2341@greglaptop.internal.keyresearch.com> While we're reinventing the T3E, it's worth noting: 1) HT's current generation doesn't have ECC, it just has a CRC-check and crashes when the check fails. That's kind-of OK for small systems, but doesn't scale. 2) AMD's cache coherence protocol is snoopy, which doesn't scale. I'd never heard the rumor about HT being the EV7+ bus (and I'd really doubt it due to (1)), but I do know that AMD only bought the API Networks technical team, not their technology. Digital/Compaq did have a long-term technical collaboration with Samsung and AMD regarding the EV6 bus. And yes, using the Linux NUMA memory placement stuff is a substantial improvement in Opteron performance for real apps. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Tue Sep 9 13:41:33 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Tue, 9 Sep 2003 10:41:33 -0700 Subject: CPUs for a Beowulf In-Reply-To: <20030909103138Q.hanzl@unknown-domain> References: <3F5C8B01.5AD8BA4@scs.carleton.ca> <20030909103138Q.hanzl@unknown-domain> Message-ID: <20030909174133.GB2341@greglaptop.internal.keyresearch.com> On Tue, Sep 09, 2003 at 10:31:38AM +0200, hanzl at noel.feld.cvut.cz wrote: > > Administrative costs can easily be THE dominant cost > > Yes, yes, yes. Anybody has different experience for his/her FIRST > cluster? (If somebody does, he is genius and his labor has much higher > value then estimated and I am right anyway :-) Clusters built by people who already have experience administering large numbers of machines tend to be pretty cost-efficient on the admin side. There isn't much crossover between the "Beowful" community and the LISA community, but there should be. My first non-small HPC cluster was 64 nodes, and it took 28 hours of labor to set up, of which only 8 hours was my time, and 20 hours was less experienced people wielding screwdrivers. Let's see, if you think $100 per node is a reasonable setup fee, and the 20 hours cost $10 each, then I was worth $775/hour. :-) (No, I'm not serious.) -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Tue Sep 9 16:53:43 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Tue, 9 Sep 2003 13:53:43 -0700 Subject: CPUs for a Beowulf In-Reply-To: <20030909174133.GB2341@greglaptop.internal.keyresearch.com> References: <3F5C8B01.5AD8BA4@scs.carleton.ca> <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> Message-ID: <20030909205343.GA3316@greglaptop.internal.keyresearch.com> On Tue, Sep 09, 2003 at 10:41:33AM -0700, Greg Lindahl wrote: > "Beowful" community Old English is weird, but not *that* weird... hm. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Tue Sep 9 19:01:26 2003 From: csamuel at vpac.org (Chris Samuel) Date: Wed, 10 Sep 2003 09:01:26 +1000 Subject: CPUs for a Beowulf In-Reply-To: <20030909205343.GA3316@greglaptop.internal.keyresearch.com> References: <3F5C8B01.5AD8BA4@scs.carleton.ca> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030909205343.GA3316@greglaptop.internal.keyresearch.com> Message-ID: <200309100901.29492.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 10 Sep 2003 06:53 am, Greg Lindahl wrote: > On Tue, Sep 09, 2003 at 10:41:33AM -0700, Greg Lindahl wrote: > > "Beowful" community > > Old English is weird, but not *that* weird... hm. I think you've just invented a new word. :-) Beowful: To use (or be full of) Beowulf clusters. Q: Do you use Linux clusters ? A: Yes, we're Beowful. :-) - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/XlvGO2KABBYQAh8RAsdTAKCYYiBqfO/qDYLu0KgFuQQ4vzlqSwCfaw0z 27RV6USmLJLESnLz6xYV9+o= =blp9 -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From timm at fnal.gov Tue Sep 9 22:34:41 2003 From: timm at fnal.gov (Steven Timm) Date: Tue, 09 Sep 2003 21:34:41 -0500 Subject: CPUs for a Beowulf In-Reply-To: <200309100901.29492.csamuel@vpac.org> Message-ID: > > Beowful: To use (or be full of) Beowulf clusters. > > Q: Do you use Linux clusters ? > A: Yes, we're Beowful. > I thought it was a cross between Beowulf + Awful. :) Steve _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 9 22:20:04 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 9 Sep 2003 22:20:04 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <20030909173634.GA2341@greglaptop.internal.keyresearch.com> Message-ID: On Tue, 9 Sep 2003, Greg Lindahl wrote: > While we're reinventing the T3E, it's worth noting: > > 1) HT's current generation doesn't have ECC, it just has a CRC-check > and crashes when the check fails. I think that it's more precisely a longitudinal check. "CRC" usually means a polynomial calculation over all of the bits, where any bit flip might change any bit in the final check word. A longitudinal-only check means that a bit flip only impacts the check word in that bit position. Longitudinal checks are much easier to implement in very high speed systems because you don't have to handle data skew combined with different length logic paths. But they catch fewer errors precisely because they are easier to implement -- they don't combine as many source bits into the result. > That's kind-of OK for small systems, > but doesn't scale. Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and correct most multibit errors, and most HT errors will be multibit. It's better to fail on corruption than to silently further corrupt. > 2) AMD's cache coherence protocol is snoopy, which doesn't scale. That's a point often missed: in order to implement a HT switch for a SMP system, you need to implement something like a cache directory. > I'd never heard the rumor about HT being the EV7+ bus (and I'd really > doubt it due to (1)), but I do know that AMD only bought the API > Networks technical team, not their technology. Digital/Compaq did have > a long-term technical collaboration with Samsung and AMD regarding the > EV6 bus. Remeber the goal of a single commodity motherboard supporting either an AMD x86 chip or an Alpha? Look where we are today... -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kdunder at sandia.gov Tue Sep 9 23:31:54 2003 From: kdunder at sandia.gov (Keith D. Underwood) Date: 09 Sep 2003 21:31:54 -0600 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <1063164714.18381.27.camel@thinkpad> > I think that it's more precisely a longitudinal check. "CRC" usually > means a polynomial calculation over all of the bits, where any bit flip > might change any bit in the final check word. A longitudinal-only check > means that a bit flip only impacts the check word in that bit position. > > Longitudinal checks are much easier to implement in very high speed > systems because you don't have to handle data skew combined with > different length logic paths. But they catch fewer errors precisely > because they are easier to implement -- they don't combine as many > source bits into the result. As I read the spec, it's actually a little weirder than that: "A 32-bit cyclic redundancy code (CRC) covers all HyperTransportTM links. The CRC is calculated on each 8-bit lane independently and covers the link as a whole, not individual packets." So, for a 16 bit link (such as on an Opteron), there are two concurrently running CRC's, one each for the two 8 bit lanes. Now, the strange part is that the CRC's cover the link and not the packets. So, a CRC is transmitted every 512 bit times (and hypertransport packets aren't that big). That means that you don't know which packets had a bad bit. > > That's kind-of OK for small systems, > > but doesn't scale. > > Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and > correct most multibit errors, and most HT errors will be multibit. > It's better to fail on corruption than to silently further corrupt. The problem is that there is no sane mechanism to know which packets are corrupted (and to therefore retransmit). At scale, that doesn't really work. e.g. if you built a Red Storm scale system using just these links, it would crash frequently because a CRC error would happen somewhere and there wouldn't be a recovery mechanism. (BTW, for those suggesting mimicking the T3E with these things, the T3E wasn't cache coherent. It just had a shared address space mechanism of sorts.) Someone asked something about Red Storm - here is a public link that includes public presentations on the topic: http://www.cs.sandia.gov/presentations/ Keith _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 10 03:01:34 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 10 Sep 2003 00:01:34 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: <20030909173634.GA2341@greglaptop.internal.keyresearch.com> Message-ID: <20030910070134.GA1731@greglaptop.greghome.keyresearch.com> On Tue, Sep 09, 2003 at 10:20:04PM -0400, Donald Becker wrote: > > That's kind-of OK for small systems, > > but doesn't scale. > > Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and > correct most multibit errors, and most HT errors will be multibit. > It's better to fail on corruption than to silently further corrupt. Well, I said "kind of OK" because you won't notice the failures on small systems until you have a bunch of them. That's not really acceptable, but then again lots of people seem willing to bet their data on systems where they haven't really thought about errors at all. I was speaking loosely when I said ECC. The CRC being used will (in theory) catch most multi-bit errors, but it's always scary to not know the pattern of the errors when you chose the CRC. The system does crash quickly enough that it's unlikely that your bad data makes it onto a network or disk. I get a kick out of asking link layer people, "So, what bad packets have you observed in the wild?" They give me the blankest looks... > Remeber the goal of a single commodity motherboard supporting either an > AMD x86 chip or an Alpha? Look where we are today... It was a great idea until the mechnical problems nuked it: Slot A vs. Socket A. I was not impressed by the reliability of Slot A. Sometimes it's the little things that cause the biggest problems. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joachim at ccrl-nece.de Wed Sep 10 03:39:26 2003 From: joachim at ccrl-nece.de (Joachim Worringen) Date: Wed, 10 Sep 2003 09:39:26 +0200 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <200309100939.26394.joachim@ccrl-nece.de> Donald Becker: > > 2) AMD's cache coherence protocol is snoopy, which doesn't scale. > > That's a point often missed: in order to implement a HT switch for a SMP > system, you need to implement something like a cache directory. Yes, but only if you really want SMP characteristics (sequential consistency). This is not mandatory for a HPC system if you can live with some kind of message passing programming model (MPI). Joachim -- Joachim Worringen - NEC C&C research lab St.Augustin fon +49-2241-9252.20 - fax .99 - http://www.ccrl-nece.de _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Wed Sep 10 04:58:48 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Wed, 10 Sep 2003 10:58:48 +0200 Subject: CPUs for a Beowulf In-Reply-To: <20030909174133.GB2341@greglaptop.internal.keyresearch.com> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> Message-ID: <20030910105848W.hanzl@unknown-domain> > > > Administrative costs can easily be THE dominant cost > > > > Yes, yes, yes. Anybody has different experience for his/her FIRST > > cluster? (If somebody does, he is genius and his labor has much higher > > value then estimated and I am right anyway :-) > > My first non-small HPC cluster was 64 nodes, and it took 28 hours of > labor to set up, of which only 8 hours was my time And the first small one? (I basically wanted to scare people who are just building the very first cluster and want to be very innovative at the same time - it is fine and a lot of fun but quite likely it is not a way to save yet. Maybe innovative soul can save a lot on the second one...) Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Wed Sep 10 09:56:57 2003 From: becker at scyld.com (Donald Becker) Date: Wed, 10 Sep 2003 09:56:57 -0400 (EDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <1063164714.18381.27.camel@thinkpad> Message-ID: On 9 Sep 2003, Keith D. Underwood wrote: > > I think that it's more precisely a longitudinal check. "CRC" usually ... > > Longitudinal checks are much easier to implement in very high speed . > As I read the spec, it's actually a little weirder than that: > > "A 32-bit cyclic redundancy code (CRC) covers all HyperTransportTM > links. The CRC is calculated on each 8-bit lane independently and > covers the link as a whole, not individual packets." Ahhh, that implies that it uses a group longitudinal check. That has much better checking than a serial single bit check, and by limiting the width the reduce the skew and logic path problem of a wider check. > So, for a 16 bit link (such as on an Opteron), there are two > concurrently running CRC's, one each for the two 8 bit lanes. Now, the > strange part is that the CRC's cover the link and not the packets. So, > a CRC is transmitted every 512 bit times (and hypertransport packets > aren't that big). That means that you don't know which packets had a > bad bit. That's an excellent design decision. Putting a check word on each packet means - the physical encoding layer need to know about packetization - a packet must be held until the check passes - the tiny packets grow - to do anything with the per-packet info, packet copies must be kept These all add complexity and latency to the highest speed path. By putting the check on fixed block boundaries you can still detect and fail an unreliable link > > > That's kind-of OK for small systems, but doesn't scale. > > Errrm, I have exactly the opposite viewpoint: ECC will fail to catch and > > correct most multibit errors, and most HT errors will be multibit. I'll repeat: ECC Bad! ECC Slow! > corrupted (and to therefore retransmit). At scale, that doesn't really > work. e.g. if you built a Red Storm scale system using just these > links, it would crash frequently because a CRC error would happen > somewhere and there wouldn't be a recovery mechanism. If you are getting CRC errors, you very likely have errors that ECC (*) would silently pass or further corrupt. * Any high-speed ECC implementation. It's possible to keep adding check bits, but anything past SECDED Single Error Correction, Double Error Detection becomes time consuming and expensive. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kdunder at sandia.gov Wed Sep 10 10:18:14 2003 From: kdunder at sandia.gov (Keith D. Underwood) Date: 10 Sep 2003 08:18:14 -0600 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: References: Message-ID: <1063203494.18382.44.camel@thinkpad> > That's an excellent design decision. > Putting a check word on each packet means > - the physical encoding layer need to know about packetization > - a packet must be held until the check passes > - the tiny packets grow > - to do anything with the per-packet info, packet copies must be kept > These all add complexity and latency to the highest speed path. > > By putting the check on fixed block boundaries you can still detect and > fail an unreliable link All very true when you have 1, 2, 4, even 8 HT links that could cause a system to crash. And I'm not suggesting that ECC would be better (that was Greg's statement), but.... if you had 10000 HT links running their maximum distance (if you used HT links to build a mesh at Red Storm scale) and any bit error on any of them causes an app to fail because you don't know which packet had an error... That would be bad. Supposedly, this is going to be fixed in HT 2.0 (I wouldn't know since the spec isn't freely available). Keith _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From molson at edgetechcomputing.com Wed Sep 10 11:19:52 2003 From: molson at edgetechcomputing.com (Myles Olson) Date: Wed, 10 Sep 2003 09:19:52 -0600 Subject: data storage location Message-ID: <000601c377ae$fc0a3c90$54aa6f90@TOSHIBER> I'm new at this so please excuse any errors on my part. I need to process datasets around 100GB. What would be the best way for the Beowulf cluster to access this data? Should the host node connect to a disk array over FC-AL? Should it be a fiber connection to a SAN? Or does all this depend on how the application breaks the data down for processing? Thanks ? Myles Olson Edgetech Computing - Technical Services Office: 403.237.5331 Mobile: 403.875.6630 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Wed Sep 10 12:15:25 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Wed, 10 Sep 2003 09:15:25 -0700 (PDT) Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <20030910070134.GA1731@greglaptop.greghome.keyresearch.com> Message-ID: On Wed, 10 Sep 2003, Greg Lindahl wrote: > > Remeber the goal of a single commodity motherboard supporting either an > > AMD x86 chip or an Alpha? Look where we are today... > > It was a great idea until the mechnical problems nuked it: Slot A > vs. Socket A. I was not impressed by the reliability of Slot A. > Sometimes it's the little things that cause the biggest problems. the processor cards for our alpha server 7000's (21164 433mhz) were something like 12"x16" the l2 cache alone is like 20 square inches. I still have one of them hanging on my wall. the larger alpha building blocks had and really continue to suffer from a serious form factor problem. The gs80 is basically stil about 3/4 of a rack for an 8cpu box. > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 13:20:52 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 10:20:52 -0700 Subject: AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf) In-Reply-To: <1063203494.18382.44.camel@thinkpad> References: Message-ID: <5.2.0.9.2.20030910101023.018a99f8@mailhost4.jpl.nasa.gov> At 08:18 AM 9/10/2003 -0600, Keith D. Underwood wrote: > > That's an excellent design decision. > > Putting a check word on each packet means > > - the physical encoding layer need to know about packetization > > - a packet must be held until the check passes > > - the tiny packets grow > > - to do anything with the per-packet info, packet copies must be kept > > These all add complexity and latency to the highest speed path. > > > > By putting the check on fixed block boundaries you can still detect and > > fail an unreliable link > >All very true when you have 1, 2, 4, even 8 HT links that could cause a >system to crash. And I'm not suggesting that ECC would be better (that >was Greg's statement), but.... if you had 10000 HT links running their >maximum distance (if you used HT links to build a mesh at Red Storm >scale) and any bit error on any of them causes an app to fail because >you don't know which packet had an error... That would be bad. Any time you're looking at large distributed systems, one needs to plan for and be able to handle failures. If a single hit causes the entire shooting match to collapse, it's never going to work. In fact, I'd claim that what "scalability" really means is that the inevitable errors have limited propagation. Otherwise, as you increase the number of "widgets" in the system, the probability of any one failing starts to get close to one. The real design decisions come in when deciding at what level to handle the error. Detecting is straightforward at the bottom level, but error handling may be best dealt with at a high level. Perhaps the overall performance of the ensemble is better if everyone goes in lockstep, rather than retrying the failed communication. Compare, for example, 11 for 8 Hamming coding at the byte level (low latency, poor rate efficiency), and CRC error detection and retries at a higher level, and then, all the sorts of block interleaving schemes which turn burst errors into isolated (and correctable on the fly) errors. Sometimes you trade determinism for performance, or latency for overall bit error rate. A lot depends on your error statistics, and such is grist for much communications theory analysis, and keeps coding specialists employed. Consider also, things like algorithm robustness to bit flips in RAM. On one system I worked on, it was faster to do the calculations three times and vote the results, with no ECC, than to do them once with ECC, because of the increased latency of the ECC logic, and the adverse interaction between ECC and cache. There is a fair amount of literature on this... The Byzantine Generals problem (unreliable communication between multiple masters) is a good example. Fault robustness/tolerance in large cluster configurations is a subject that is near and dear to my heart because I want to fly clusters in space, where errors are a given, and maintenance is impossible. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Sep 10 13:07:57 2003 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 10 Sep 2003 11:07:57 -0600 Subject: CPUs for a Beowulf In-Reply-To: <20030910105848W.hanzl@unknown-domain> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> Message-ID: <3F5F5A6D.2050201@lanl.gov> hanzl at noel.feld.cvut.cz wrote: >>>>Administrative costs can easily be THE dominant cost >> >>My first non-small HPC cluster was 64 nodes, and it took 28 hours of >>labor to set up, of which only 8 hours was my time > > And the first small one? Anyone getting started on clusters has to negotiate a learning curve, regardless of whether they build it themselves or buy a turn-key system. This part of the administrative cost is unavoidable for a new cluster user. For lots of people, learning on the system you built yourself is preferable. Manpower costs are dominant in most projects, but most manpower costs are related to using the machinery, not administering it. When administrative costs exceed the cost of equipment, people tend to buy different stuff with lower overall life cycle costs. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 10 12:39:13 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 10 Sep 2003 09:39:13 -0700 Subject: CPUs for a Beowulf In-Reply-To: <20030910105848W.hanzl@unknown-domain> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> Message-ID: <20030910163913.GA3486@greglaptop.greghome.keyresearch.com> > > My first non-small HPC cluster was 64 nodes, and it took 28 hours of > > labor to set up, of which only 8 hours was my time > > And the first small one? > > (I basically wanted to scare people who are just building the very > first cluster You shouldn't use me as an example, then: that was my first HPC cluster, but I had set several groups of systems before which had related configs, and worked on clusters up to ~ 130 systems in size while working in industry. It all depends on what your experience is. If you haven't set up any kind of cluster before, you're asking for trouble. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 10 14:21:38 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 10 Sep 2003 14:21:38 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <20030910163913.GA3486@greglaptop.greghome.keyresearch.com> Message-ID: On Wed, 10 Sep 2003, Greg Lindahl wrote: > > > My first non-small HPC cluster was 64 nodes, and it took 28 hours of > > > labor to set up, of which only 8 hours was my time > > > > And the first small one? > > > > (I basically wanted to scare people who are just building the very > > first cluster > > You shouldn't use me as an example, then: that was my first HPC > cluster, but I had set several groups of systems before which had > related configs, and worked on clusters up to ~ 130 systems in size > while working in industry. > > It all depends on what your experience is. If you haven't set up > any kind of cluster before, you're asking for trouble. Ya, which returns nicely to the original point. There are two or three elements of cluster engineering that can bite you if you've never done large scale cluster (or LAN) design or management before. Hardware, installation and maintenance, and infrastructure might be easy (but probably incomplete) categories. COTS is good, COTS is the point, but reliability becomes increasingly important as the number of systems scales up and somewhere in there it is no longer wise to get CHEAP COTS systems, as in lowest-bid vanilla boxes with no provisions for service. Hardware maintenance can eat you alive if you end up with a large batch of flaky hardware, even if you do have a service deal of some sort as it still costs you all sorts of time for every failure. Clusters eat electricity and excrete it as heat. Cluster nodes need safe homes and like to talk. An eight-node toy cluster can often be set up "anywhere" and just plugged in to a handy wall receptacle (although even 8 nodes can make a room pretty hot without enough A/C). A larger cluster often requires planning and remodeling, even engineering, of the space the nodes are to live in. This can EASILY cost more than the hardware you put into that space! It is easy to use bad methodology to install and maintain a cluster. Not as easy as it once was -- it is really pretty easy to use GOOD methodology these days it the (awesome) tools are there and are even tolerably well-documented -- but if your experience with linux is installing it off of CD's onto a few systems one at a time, you've got some learning to do. If you're really a Unix novice and don't have a decent grasp of TCP/IP, ethernet (at least), scalable administrative tools and services such as but not limited to NIS, NFS, and a whole lot more, you've got a LOT of learning to do. You, Greg are very much in the expert category -- no, I'd have to go beyond that to "trans-professional superguru" category -- with a vast experience in all of the above and then some. Besides that, you're pretty smart;-) You should be wearing a special medal or insignia -- the Royal Order of the 'Wulf or the like:-) Others (including me if it came to it) sometimes are very good at figuring out what went wrong -- after the fact -- so it helps for them/us to proceed cautiously. Even something like a cheap, non-PXE fast ethernet NIC vs a more expensive, PXE-supporting NIC may not seem important (and hey, going COTS cheap saves you MONEY, right?), until you find yourself carrying a floppy around to 64 systems to install them one at a time, or have a two-post rack collapse because it isn't adequately anchored. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Wed Sep 10 15:01:55 2003 From: gmpc at sanger.ac.uk (Guy Coates) Date: Wed, 10 Sep 2003 20:01:55 +0100 (BST) Subject: data storage location In-Reply-To: <200309101726.h8AHQId22018@NewBlue.Scyld.com> References: <200309101726.h8AHQId22018@NewBlue.Scyld.com> Message-ID: As always, it depends on what the data access pattern like, and how many nodes you have. If the data access is read-only, *and* your node number is small (<10), *and* you have a fast network then you *might* be able to get away with NFS. However, for read/write access or more than a few nodes, you need to be a bit more creative. Fibre channel + switch may look good on paper but have their problems (not least of which is cost). If you have a moderate amount of fibre clients trying to access the same disk controller you can easily saturate the fibre ISLs or disk controllers. For example, 4 clients doing 50 Mbytes/S sustained data access are easily going to saturate the fibre to your disk controller, whether switched or on FC-AL. If your data access is bursty then you might do better. Furthermore, if you want all your clients to access the same filesystem on the your fibre channel disks then you are also going to need a clustered filesystem. The alternative approach is to keep copies of the data on local disk on each node. This gives you good IO rates, but you then have a substantial data management problem; how to you copy 100Gb to each node in your cluster in a sensible amount of time, and how do you update the data and make sure it is kept consistent? The commonest approach to data distribution is to do some sort of cascading rsync/rcp which follows the topology of your network. If your dataset is larger than the amount of local disk on your nodes, you then have to partition your data up, and integrate that with your queuing system, so that jobs which need a certain bit of the data end up on a node which actually holds a copy. Recently we've had some success using multicast methods to distribute data to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of the better multicast codes, but you'll need to put some wrappers around it to make it useful. The multicast method is substantially faster than rsyncing data on large clusters. Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 10 16:58:19 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 10 Sep 2003 13:58:19 -0700 Subject: CPUs for a Beowulf In-Reply-To: <3F5F5A6D.2050201@lanl.gov> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> <3F5F5A6D.2050201@lanl.gov> Message-ID: <20030910205819.GA2023@greglaptop.internal.keyresearch.com> On Wed, Sep 10, 2003 at 11:07:57AM -0600, Josip Loncaric wrote: > hanzl at noel.feld.cvut.cz wrote: > >>>>Administrative costs can easily be THE dominant cost > >> > >>My first non-small HPC cluster was 64 nodes, and it took 28 hours of > >>labor to set up, of which only 8 hours was my time > > > >And the first small one? > > Anyone getting started on clusters has to negotiate a learning curve, > regardless of whether they build it themselves or buy a turn-key system. Josip, My point was that some admins with zero "beowulf" cluster experience have plenty of large system Unix experience. LANL used to have several such admins, and they built some large clusters for their first attempt, and it went very well. They even used cfengine, which is a tool commonly used in the large Unix system arena. Several of them probably still work for LANL; the one I know best, Susan Coghlan, is now at Argonne. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 17:00:03 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 14:00:03 -0700 Subject: cluster using handhelds Message-ID: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Has anyone tried building a cluster using handheld computers (i.e. iPaq or Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is there an implementation of MPI that will work in that environment (WinCE, PalmOS) (perhaps MPICH in some form?) Yes, I know the performance will be hideous. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Wed Sep 10 17:22:16 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Wed, 10 Sep 2003 14:22:16 -0700 (PDT) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: I could trivially do it with both my zauruses right now... the real question is why... On Wed, 10 Sep 2003, Jim Lux wrote: > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > there an implementation of MPI that will work in that environment (WinCE, > PalmOS) (perhaps MPICH in some form?) > > Yes, I know the performance will be hideous. > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 17:49:29 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 14:49:29 -0700 Subject: cluster using handhelds In-Reply-To: References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> At 02:22 PM 9/10/2003 -0700, Joel Jaeggli wrote: >I could trivially do it with both my zauruses right now... the real >question is why... Which model of Zaurus, and what sort of wireless network interface (CF or PCMCIA)? I'm looking at a distributed measurement application where each processor needs to be able to talk to some local hardware and to share measurements with other processors. I also need to distribute some computing across the processors. I'd rather not have to rewrite all the computing as platforms change, so a standardized interface like MPI is attractive. Obviously, an alternative is a battery powered single board PC with a suitable display and user interface, but, in the case of the handhelds, someone else has already dealt with the hardware integration issues, and the purchase price is generally lower than one would spend on all the pieces. I don't want to be in the PC design business, if I can avoid it. I note that the various variants of Linux at handhelds.org (formerly, and possibly still, supported by Compaq) don't support the current versions of iPaqs being sold by HP. This, in itself, is kind of a worrying trend, because I've already got enough orphan hardware and software sitting around. (I've got ISA bus machines running win95 down in the lab, because that's what's needed to run the incircuit emulators for the "no longer sold in the commercial market but still sold for spaceflight" DSPs) >On Wed, 10 Sep 2003, Jim Lux wrote: > > > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > > there an implementation of MPI that will work in that environment (WinCE, > > PalmOS) (perhaps MPICH in some form?) > > > > Yes, I know the performance will be hideous. > > > > > > -------------------------------------------------------------------------- >Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu >GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Wed Sep 10 18:22:19 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Wed, 10 Sep 2003 15:22:19 -0700 (PDT) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > At 02:22 PM 9/10/2003 -0700, Joel Jaeggli wrote: > >I could trivially do it with both my zauruses right now... the real > >question is why... > > Which model of Zaurus, and what sort of wireless network interface (CF or > PCMCIA)? 5500sl it/they have cf wireless cards... you can pretty much get the whole arm build enivironment and toolchain on a large mmc card in the sd card slot. you can run open zaurus or the debian arm distro on them in a pretty simple fashion... They also have serial port pinout on the jack on the bottom which you can get a cable for so serial consoles are feasable. also they have keyboards so failing around on a text consoles is feasable. > I'm looking at a distributed measurement application where each processor > needs to be able to talk to some local hardware and to share measurements > with other processors. I also need to distribute some computing across the > processors. I'd rather not have to rewrite all the computing as platforms > change, so a standardized interface like MPI is attractive. > > Obviously, an alternative is a battery powered single board PC with a > suitable display and user interface, but, in the case of the handhelds, > someone else has already dealt with the hardware integration issues, and > the purchase price is generally lower than one would spend on all the > pieces. I don't want to be in the PC design business, if I can avoid it. basically the things that make me say why.... are the mips/watt equation on a pda is skewed by things like the active matrix display which is still power hungry even if the backlight is off. internal batteries, particularly with a wireless card inserted don't last useful lengths of time. in the zaurus's case about and hour and 45 minutes with the card in or maybe 6 without the card and the backlight turned all the way down. then we have the under-powered cpu's which for the 200mhz make it sufficent to compare to a cca 1993 pc which isn't bad but it's not huge either... We do a substantial amount of work around some embeded pc boards made by a company called soekris engineering http://www.soekris.com/ that come with nice cases that might be appropiate for you application... in particular it makes porting stuff a non-issue since i586 or 486 code will run happily on them. I actually run a somewhat condensed version of redhat 8.0 on a couple of their boxes. a number of people use them as wireless accesspoints amount other things so they may be useful for you application. they also have useful features that's pda's don't like gpio pins. some of the models (4521) have very wide dc voltage operating ranges (11-56 volts) to accomidate things like power over ethernet. http://www.soekris.com/net4521.htm > I note that the various variants of Linux at handhelds.org (formerly, and > possibly still, supported by Compaq) don't support the current versions of > iPaqs being sold by HP. This, in itself, is kind of a worrying trend, > because I've already got enough orphan hardware and software sitting > around. the life-cyle in the pda market, especially the wince pda market is very very short. > (I've got ISA bus machines running win95 down in the lab, because > that's what's needed to run the incircuit emulators for the "no longer sold > in the commercial market but still sold for spaceflight" DSPs) > > > > > >On Wed, 10 Sep 2003, Jim Lux wrote: > > > > > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > > > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > > > there an implementation of MPI that will work in that environment (WinCE, > > > PalmOS) (perhaps MPICH in some form?) > > > > > > Yes, I know the performance will be hideous. > > > > > > > > > -------------------------------------------------------------------------- > >Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu > >GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Sep 10 18:29:36 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 10 Sep 2003 15:29:36 -0700 Subject: cluster using handhelds In-Reply-To: References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> At 04:48 PM 9/10/2003 -0500, you wrote: >On Wed, 10 Sep 2003, Jim Lux wrote: > > > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > > there an implementation of MPI that will work in that environment (WinCE, > > PalmOS) (perhaps MPICH in some form?) > > > > Yes, I know the performance will be hideous. > >Ok, I can not help but to comment. > >Once upon a time, just for S&G's, we spec'ed out a T-FLOP cluster out of >PalmPilots. Assuming no space for wiring, we figgured we could fit >about 144 of my Palms into a cubic foot. > >Crammed floor to ceiling, had our building had 1,500 stories, it would >have almost held the thing. > >Sorry for my silliness, but i still find it funny. You may laugh, but consider a structure that looks like a floppy ball some 100 meters in radius with sensing elements scattered every 10-20 cm across the surface, each with it's own processor that needs to talk to some (but not all) of the other elements to make the measurements. Perhaps not TFLOPS, but a huge pile of small processors, nonetheless. Now consider that before anybody would give you the tens of millions of dollars required to actually build such a thing, they'd want to see some sort of credible demonstration that it's feasible to do the work with processors of limited capability, but, yet, still general purpose and programmable. Further, one probably doesn't want to spend a huge amount of time designing processor nodes, but wants to buy something off the shelf, preferably consumer mass market. One would also prefer something where the internal architecture is published and understandable so that one could make credible estimates of what that fancy custom element is going to need. Hence the question about clusters of handhelds: handhelds are consumer mass market and cheap linux on a handheld is "open and visible" as far as internals go WinCE might be (depending on licensing, etc. It's been a long time since I was an "embedded windows" (forerunner to WinCE) developer) 802.11 is a wireless comm scheme with known properties. The eventual system would probably use something else (Snaggletooth, Forkbeard (Bluetooth's son), Zigbee (802.15.4), CCSDS proximity link, or whatever) MPI because I'm lazy, and I already know how to use MPI for stuff. By the way, handhelds that can talk to a cmos/ccd camera and grab frames on demand would be even better. >-- >Rocky McGaugh >Atipa Technologies >rocky at atipatechnologies.com >rmcgaugh at atipa.com >1-785-841-9513 x3110 >http://67.8450073/ >perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Wed Sep 10 17:48:26 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Wed, 10 Sep 2003 16:48:26 -0500 (CDT) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > there an implementation of MPI that will work in that environment (WinCE, > PalmOS) (perhaps MPICH in some form?) > > Yes, I know the performance will be hideous. Ok, I can not help but to comment. Once upon a time, just for S&G's, we spec'ed out a T-FLOP cluster out of PalmPilots. Assuming no space for wiring, we figgured we could fit about 144 of my Palms into a cubic foot. Crammed floor to ceiling, had our building had 1,500 stories, it would have almost held the thing. Sorry for my silliness, but i still find it funny. -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Sep 10 20:42:20 2003 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 10 Sep 2003 18:42:20 -0600 Subject: CPUs for a Beowulf In-Reply-To: <20030910205819.GA2023@greglaptop.internal.keyresearch.com> References: <20030909103138Q.hanzl@unknown-domain> <20030909174133.GB2341@greglaptop.internal.keyresearch.com> <20030910105848W.hanzl@unknown-domain> <3F5F5A6D.2050201@lanl.gov> <20030910205819.GA2023@greglaptop.internal.keyresearch.com> Message-ID: <3F5FC4EC.2090001@lanl.gov> Greg Lindahl wrote: > On Wed, Sep 10, 2003 at 11:07:57AM -0600, Josip Loncaric wrote: > >>Anyone getting started on clusters has to negotiate a learning curve, >>regardless of whether they build it themselves or buy a turn-key system. > > My point was that some admins with zero "beowulf" cluster experience > have plenty of large system Unix experience. Good point. It sure helps to start high on the learning curve... I'd only like to add that administrative costs for a functional cluster are *way* lower than for the same number of office PCs. There are many reasons for this so I do not want to enumerate them here. You probably have a better feel for this, but my guess is that a capable Linux cluster administrator can manage several hundred nodes. In other words, although support costs for an N-processor cluster scale with N, the scaling constant is fairly reasonable. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Wed Sep 10 22:04:12 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 11 Sep 2003 10:04:12 +0800 (CST) Subject: Mac clusters... In-Reply-To: <0EB5C81FE6FE5A4F8D1FEBF59C6C7BAA0A9FB7@durham.sra.com> Message-ID: <20030911020412.29314.qmail@web16808.mail.tpe.yahoo.com> Sounds to me they chose Apple due to the better cost/performance, but you can get more information from the presentation: http://www.chaosmint.com/mac/vt-supercomputer/ One thing which is not mentioned is the batch system, are they going to use PBS or SGE, anyone? Andrew. --- "Walters, David" ????> Does anyone know where their funding came from? Did > they choose Apple based > on a cost/benefit analysis, or is Apple paying them? > I am thinking back to > the numbers that came out of the Cornell Theory > Center here... Same > scenario? > > Dave Walters > Project Manager/Sr. Management Analyst, EPA IIASC > SRA International, Inc. > 919-474-4318 > david_walters at sra.com > > > -----Original Message----- > From: Rayson Ho [mailto:raysonlogin at yahoo.com] > Sent: Friday, September 05, 2003 5:32 PM > To: beowulf > Subject: Mac clusters... > > > 1100 dual G5s to build a supercomputer: > > http://www.computerweekly.com/articles/article.asp?liArticleID=124559&liFlav > ourID=1&sp=1 > > And this cluster is interesting too: > > --- "Hunt, Derek" wrote: > > Hello all, this is in response to Charles D. > Norton's post regarding > > Xserve clusters. > > > > Site: Yale School Of Management > > Yale University > > Hardware Vendor: Apple Computer > > Number of nodes: 45 > > Number of processors: 90 > > Total Memory: 90GB > > Interconnect Technology: Gigabit Ethernet > > OS: 10.2.6 with current patches > > Comm Software: MPICH, Sun Gridengine > > Application Software: GridMathematica, Matlab, > C/C++, Fortran Mail > > application area Scientific Computing: Financial > Research, Statistical > > Analysis Year installed/upgraded: 2003 > > > > We are in the initial stages of deployment now. > > > > on a side note, I will be giving a talk on > September 25th in > > Rochester, MN at the linux users group (more > details at > > http://www.k-lug.org) if anyone is interested in > hearing the various > > trials we encountered during > > installation/deployment. > > > > - Derek > > > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free, easy-to-use web site > design software > http://sitebuilder.yahoo.com > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bari at onelabs.com Wed Sep 10 23:05:34 2003 From: bari at onelabs.com (Bari Ari) Date: Wed, 10 Sep 2003 22:05:34 -0500 Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: <3F5FE67E.2090002@onelabs.com> Jim Lux wrote: > > You may laugh, but consider a structure that looks like a floppy ball > some 100 meters in radius with sensing elements scattered every 10-20 > cm across the surface, each with it's own processor that needs to talk > to some (but not all) of the other elements to make the measurements. > Perhaps not TFLOPS, but a huge pile of small processors, nonetheless. > > Now consider that before anybody would give you the tens of millions > of dollars required to actually build such a thing, they'd want to see > some sort of credible demonstration that it's feasible to do the work > with processors of limited capability, but, yet, still general purpose > and programmable. > > Further, one probably doesn't want to spend a huge amount of time > designing processor nodes, but wants to buy something off the shelf, > preferably consumer mass market. One would also prefer something where > the internal architecture is published and understandable so that one > could make credible estimates of what that fancy custom element is > going to need. Take a look at http://www.eu.renesas.com/products/mpumcu/32bit/sh-microprocessors/sh4/index.html or http://www.superh.com/products/sh5.htm The SH-4's perform at around 1GFLOP/Watt and the SH-5's at 3GFLOP/Watt. SH Linux has been around for some time. http://www.sh-linux.org/ There are a few vendors who make PC-104ish size SH based boards. You could build a TFLOP cluster using the SH-5's that would fit into a single 42-U rack and only consume around 1KW to 3KW depending on the amount of memory used. Memory would actually consume more power than the cpu's. For fixed point only apps. consider some of the boards that use the PXA or X-Scale cpu's. They cost about the same as an high end PXA based PDA. Bari _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 11 05:28:10 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 11 Sep 2003 11:28:10 +0200 (CEST) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > Has anyone tried building a cluster using handheld computers (i.e. iPaq or > Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is > there an implementation of MPI that will work in that environment (WinCE, > PalmOS) (perhaps MPICH in some form?) > If my memory serves me, I remember a presentation at the FOSDEM 2002 conference from a French lab which did something along these lines. A qucik Google doesn't find this. IT might have been CNRS, _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 11 08:11:35 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 11 Sep 2003 08:11:35 -0400 (EDT) Subject: CPUs for a Beowulf In-Reply-To: <3F5FC4EC.2090001@lanl.gov> Message-ID: On Wed, 10 Sep 2003, Josip Loncaric wrote: > I'd only like to add that administrative costs for a functional cluster > are *way* lower than for the same number of office PCs. There are many > reasons for this so I do not want to enumerate them here. > > You probably have a better feel for this, but my guess is that a capable > Linux cluster administrator can manage several hundred nodes. In other > words, although support costs for an N-processor cluster scale with N, > the scaling constant is fairly reasonable. Agreed. The scaling constant itself can even get to where it is determined primarily by hardware installation and maintenance alone, although this is also an important determinant in any LAN. With PXE/diskless operation, or PXE/kickstart installation, yum or various clone methods for keeping software up to date, a single person can "take care of" (in the sense of install and maintain OS's on) a staggering number of systems, if they never physically break. In at least some cluster environments, there aren't many users and the users themselves are pretty unix-capable and need little "support" (often a if not the major office LAN cost). You can't get out of paying for power and cooling, but once a suitable space is constructed power and cooling become a fairly predictable expense, roughly $1 per watt per year. Hardware remains the joker in the deck. Hardware support costs can be hideously variable. Bad electricity or cooling or both can cause nodes to break (done that one). Inadequate cooling fans or other node design flaws can cause nodes to break (done that one). Poor quality node components (e.g. the newer 1 yr. warranty IDE drives, a "bad batch" of nearly any component) can have a relatively high probability of failure that is low (per day) for any single system but high (per day) for 512 systems (to a certain extent unavoidable, so sure, done that). A "new" motherboard with a great clock and slick features can turn out to have a piece of s**t bios that requires two or three reflashes before it finally settles down to function, or it can just plain never stabilize and ultimately have to be replaced (times N, mind you -- and yes, done both of these). And then even really good, gold-plated name brand COTS hardware breaks with fixed probabilities and a roughly poissonian distribution (with those lovely little clusters of events) so it isn't like one hard disk fails every two weeks, its more like no disks fail for a couple of months and then four fail within a couple of days of one another. A hardware failure also can have nonlinear costs (beyond mere downtime, human time involved in fixing it, and maybe the cost of a new component) in productivity, if the computation that is proceeding at the time of failure isn't checkpointed and is tightly coupled, so a node failure brings down the whole run. At least THIS one doesn't bother me -- my own computations tend to be EP.:-) The moral of the story being -- COTS hardware, sure, but for larger clusters especially (event frequency scaling linearly with the number of nodes, disaster severity scaling NONlinearly with the number of nodes) you should PROTOTYPE and GET HIGH QUALITY NODES, and meditate carefully upon the issue of possibly onsite support contracts. Onsite support reduces the amount of time YOU (the systems managers) have to screw around with broken hardware, although a lot of people do get by with e.g. overnight or second day replacement, possibly buffering the most likely parts to fail with spares, if they have the local staff to handle it. You can trade off local human effort (opportunity cost allocation of existing staff time) with prepaid remote human effort (service contracts and so forth) depending on how it is easier to budget and how much "disposable" local FTE you wish to dispose of in this way. One way or another, node (un)reliability equals time equals money, and this "hidden" cost has to be balanced against the cost of the nodes themselves as raw hardware when computing (hands on your wallets, please:-) "Total Cost of OwnerShip" (TCOS). [As a parenthetical insertion of the sort I'm doubtless (in)famous for, has anybody noticed how TCOS is an acronym-anagram for COTS? Well, really it isn't but since I added the S to TCO it is. Spooky, huh....] I think that this is a lesson most large scale cluster managers (including those that have previously managed large scale LANs) have already learned the hard way, to the point where they are bored with the enumeration above. Cluster/LAN newbies, though, especially those who may have played with or be preparing to engineer a "toy" cluster with just a few nodes, may not have thought about all of this, so it is worth it to hammer on the point just a wee bit. After all, many of us on the list learned these lessons the hard way (go on, raise those hands, don't be shy or ashamed -- look, my hand is WAY up:-) and the least we can do is try to spare those newbies our pain. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From scheinin at crs4.it Thu Sep 11 08:33:42 2003 From: scheinin at crs4.it (Alan Scheinine) Date: Thu, 11 Sep 2003 14:33:42 +0200 Subject: Tyan 2880 memory configuration Message-ID: <200309111233.h8BCXgx05965@crs4.it> Where can I get information concerning the configuration of memory for the Tyan 2880? (I ask in this mailing list because this boards seems a likely candidate for a do-it-yourself Opteron cluster.) The documentation from Tyan does not give useful advice. Page 29 of the manual shows different ways of inserting memory but does not give advice. In particular, one reads, "memory in CPU2 DIMM1-2 is not required when running dual CPU configuration" but the diagram on page 9 indicates that access to memory would be divided between two CPUs if the DIMM sockets CPU1 and CPU2 are both used. The table on page 29 shows putting 4 memory modules in the area CPU1 as a possibility without saying whether (as I would imagine) it is better to follow the other option of putting two in CPU1 and two in CPU2. Moreover, the BIOS documentation of the 2880 manual gives various options for the use of memory but does not explain the terminology and does not explain which choice would give better performance (page 54 of the manual). I do not intend to request that someone write a detailed reply to the mailing list, it would be enough to know where to look to find the information. best regards, Alan Scheinine _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Thu Sep 11 08:59:33 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Thu, 11 Sep 2003 08:59:33 -0400 (EDT) Subject: Tyan 2880 memory configuration In-Reply-To: <200309111233.h8BCXgx05965@crs4.it> Message-ID: > inserting memory but does not give advice. In particular, > one reads, "memory in CPU2 DIMM1-2 is not required when running > dual CPU configuration" but the diagram on page 9 indicates that > access to memory would be divided between two CPUs if the DIMM > sockets CPU1 and CPU2 are both used. The table on page 29 shows > putting 4 memory modules in the area CPU1 as a possibility without > saying whether (as I would imagine) it is better to follow the > other option of putting two in CPU1 and two in CPU2. my first opteron system arrived with 4 dimms in the first CPU's bank of memory, and bank interleaving turned off in bios. turning on bank interleaving gave me around 15% speedup on stream. moving two dimms to the other CPU gave me about 25% speedup. ideally, of course, both CPUs would have two full banks (4 dimms apiece), with both node and bank interleaving turned on in bios. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Thu Sep 11 09:25:29 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Thu, 11 Sep 2003 15:25:29 +0200 (CEST) Subject: data storage location In-Reply-To: Message-ID: On Wed, 10 Sep 2003, Guy Coates wrote: [...] > Recently we've had some success using multicast methods to distribute data > to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of > the better multicast codes, but you'll need to put some wrappers around it > to make it useful. The multicast method is substantially faster than > rsyncing data on large clusters. I had two problems when testing UDPcast: - A cheaper ATI CentreCom Fast-Ethernet switch multicasted data only with the speed of the slowest connected machine, even if that machine was *not* part of the multicast group. I had to unplug all 10 Mbps links to speed up UDPcast above 1 MByte/s. - With Gigabit Ethernet and a smaller cluster of 1 GHz PentiumIII machines I had to set a rate limitation on the sender. Otherwise the sender and the receivers lost synchronization and the transmission didn't work. However, if you forgive me a short bit of advertising, we use the "Dolly" programm to replicate large amounts of data in our clusters: http://www.cs.inf.ethz.ch/CoPs/patagonia/#dolly It replicates data with a multi-drop chain over TCP and scales quite nicely. We only had performance problems on a switch with a rather limited backplane, but otherwise we use it regularly in our 16- and 128-node clusters. - Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jhalpern at howard.edu Thu Sep 11 09:34:24 2003 From: jhalpern at howard.edu (Joshua Halpern) Date: Thu, 11 Sep 2003 08:34:24 -0500 Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: <3F6079E0.1020704@howard.edu> IF you really wanted to do this, one way would be to contact a large organiztion that had a lot of the same kind of laptops with broken display screens/keyboards...much more bang for the buck at less power and they probably WANT to give them to you. Of course the power brick farm would be interesting. Josh Halpern Jim Lux wrote: > At 04:48 PM 9/10/2003 -0500, you wrote: > > >> On Wed, 10 Sep 2003, Jim Lux wrote: >> >> > Has anyone tried building a cluster using handheld computers (i.e. >> iPaq or >> > Palm type) that use wireless LAN (802.11) or IR for the >> interconnect? Is >> > there an implementation of MPI that will work in that environment >> (WinCE, >> > PalmOS) (perhaps MPICH in some form?) >> > >> > Yes, I know the performance will be hideous. >> >> Ok, I can not help but to comment. >> >> Once upon a time, just for S&G's, we spec'ed out a T-FLOP cluster out of >> PalmPilots. Assuming no space for wiring, we figgured we could fit >> about 144 of my Palms into a cubic foot. >> >> Crammed floor to ceiling, had our building had 1,500 stories, it would >> have almost held the thing. >> >> Sorry for my silliness, but i still find it funny. > > > You may laugh, but consider a structure that looks like a floppy ball > some 100 meters in radius with sensing elements scattered every 10-20 > cm across the surface, each with it's own processor that needs to talk > to some (but not all) of the other elements to make the measurements. > Perhaps not TFLOPS, but a huge pile of small processors, nonetheless. > > Now consider that before anybody would give you the tens of millions > of dollars required to actually build such a thing, they'd want to see > some sort of credible demonstration that it's feasible to do the work > with processors of limited capability, but, yet, still general purpose > and programmable. > > Further, one probably doesn't want to spend a huge amount of time > designing processor nodes, but wants to buy something off the shelf, > preferably consumer mass market. One would also prefer something where > the internal architecture is published and understandable so that one > could make credible estimates of what that fancy custom element is > going to need. > > Hence the question about clusters of handhelds: > handhelds are consumer mass market and cheap > linux on a handheld is "open and visible" as far as internals go > WinCE might be (depending on licensing, etc. It's been a long time > since I was an "embedded windows" (forerunner to WinCE) developer) > 802.11 is a wireless comm scheme with known properties. The eventual > system would probably use something else (Snaggletooth, Forkbeard > (Bluetooth's son), Zigbee (802.15.4), CCSDS proximity link, or whatever) > MPI because I'm lazy, and I already know how to use MPI for stuff. > > By the way, handhelds that can talk to a cmos/ccd camera and grab > frames on demand would be even better. > > > > >> -- >> Rocky McGaugh >> Atipa Technologies >> rocky at atipatechnologies.com >> rmcgaugh at atipa.com >> 1-785-841-9513 x3110 >> http://67.8450073/ >> perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sp at scali.com Thu Sep 11 00:34:05 2003 From: sp at scali.com (Steffen Persvold) Date: Thu, 11 Sep 2003 06:34:05 +0200 Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910143803.030e57c8@mailhost4.jpl.nasa.gov> Message-ID: <3F5FFB3D.7070208@scali.com> Jim Lux wrote: > At 02:22 PM 9/10/2003 -0700, Joel Jaeggli wrote: > >> I could trivially do it with both my zauruses right now... the real >> question is why... > > > Which model of Zaurus, and what sort of wireless network interface (CF > or PCMCIA)? > > I'm looking at a distributed measurement application where each > processor needs to be able to talk to some local hardware and to share > measurements with other processors. I also need to distribute some > computing across the processors. I'd rather not have to rewrite all the > computing as platforms change, so a standardized interface like MPI is > attractive. > > Obviously, an alternative is a battery powered single board PC with a > suitable display and user interface, but, in the case of the handhelds, > someone else has already dealt with the hardware integration issues, and > the purchase price is generally lower than one would spend on all the > pieces. I don't want to be in the PC design business, if I can avoid it. > > I note that the various variants of Linux at handhelds.org (formerly, > and possibly still, supported by Compaq) don't support the current > versions of iPaqs being sold by HP. This, in itself, is kind of a > worrying trend, because I've already got enough orphan hardware and > software sitting around. (I've got ISA bus machines running win95 down > in the lab, because that's what's needed to run the incircuit emulators > for the "no longer sold in the commercial market but still sold for > spaceflight" DSPs) > > Jim, Beeing the Linux fan I am, I recently tried to load Linux on my HP iPAQ 5450 (a relatively new one with bluetooth, 802.11b, irda and.. fingerprint scanner..). It worked ! I reached the commandline prompt with the "Familiar" development distribution. However, I did not try it out much since I needed the PDA for more productive things. It was fun though :) If you take a look at the http://www.handhelds.org/projects/h5400.html page you'll see the progress of the project. I've been thinking of contributing, but I haven't got the time at the moment... Regards, -- Steffen Persvold Senior Software Engineer mob. +47 92 48 45 11 tel. +47 22 62 89 50 fax. +47 22 62 89 51 Scali - http://www.scali.com High Performance Clustering _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Thu Sep 11 09:39:45 2003 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Thu, 11 Sep 2003 08:39:45 -0500 Subject: data storage location In-Reply-To: References: Message-ID: <3F607B21.5010904@tamu.edu> We have seen less than stellar performance with the older/cheaper ATI Centrecom switches. We were almost completely unsuccessful with multicast tests. And RFC2544 packet testing was the pits. So: small packets are bad, and multicast appears problemmatic. At least on these devices. gerry Felix Rauch wrote: > On Wed, 10 Sep 2003, Guy Coates wrote: > [...] > >>Recently we've had some success using multicast methods to distribute data >>to large numbers of nodes. udpcast http://www.udpcast.linux.lu/ is one of >>the better multicast codes, but you'll need to put some wrappers around it >>to make it useful. The multicast method is substantially faster than >>rsyncing data on large clusters. > > > I had two problems when testing UDPcast: > > - A cheaper ATI CentreCom Fast-Ethernet switch multicasted data only > with the speed of the slowest connected machine, even if that > machine was *not* part of the multicast group. I had to unplug all > 10 Mbps links to speed up UDPcast above 1 MByte/s. > > - With Gigabit Ethernet and a smaller cluster of 1 GHz PentiumIII > machines I had to set a rate limitation on the sender. Otherwise the > sender and the receivers lost synchronization and the transmission > didn't work. > > However, if you forgive me a short bit of advertising, we use the > "Dolly" programm to replicate large amounts of data in our clusters: > http://www.cs.inf.ethz.ch/CoPs/patagonia/#dolly > > It replicates data with a multi-drop chain over TCP and scales quite > nicely. We only had performance problems on a switch with a rather > limited backplane, but otherwise we use it regularly in our 16- and > 128-node clusters. > > - Felix > -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Thu Sep 11 10:00:42 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Thu, 11 Sep 2003 16:00:42 +0200 (CEST) Subject: data storage location In-Reply-To: <3F607B21.5010904@tamu.edu> Message-ID: On Thu, 11 Sep 2003, Gerry Creager N5JXS wrote: > We have seen less than stellar performance with the older/cheaper ATI > Centrecom switches. We were almost completely unsuccessful with > multicast tests. And RFC2544 packet testing was the pits. So: small > packets are bad, and multicast appears problemmatic. At least on these > devices. I forgot to mention that on the same, little switch we had perfect scalability and performance with Dolly [1] (at least for large packets, which are the interesting ones for large data replications). - Felix [1] http://www.cs.inf.ethz.ch/~rauch/tmp/FE.CentreCom742i.Agg.pdf -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 11 10:27:08 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 11 Sep 2003 16:27:08 +0200 (CEST) Subject: cluster using handhelds In-Reply-To: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 10 Sep 2003, Jim Lux wrote: > > By the way, handhelds that can talk to a cmos/ccd camera and grab frames on > demand would be even better. Your wish is my command :-) http://www.interpocket.co.uk/ Probably you would have to team this with a cheap USB camera and a dual Compact Flash jacket. Total cost is of course higher. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathog at mendel.bio.caltech.edu Thu Sep 11 10:53:58 2003 From: mathog at mendel.bio.caltech.edu (David Mathog) Date: Thu, 11 Sep 2003 07:53:58 -0700 Subject: Kill the power faster than poweroff? Message-ID: Is there a faster way to shutdown a system and kill the power than "poweroff"? This is for use in emergency overheating conditions, where something that's effectively the equivalent of: % sync [turn off the power supply] is desired. lm_sensors and mondo are running on our Athlon 2200MP cluster and I'm concerned that in the event of a catastrophic failure of the cooling system the CPUs might overheat before the poweroff sequence completes. Not necessarily flame out immediately, but at least get hot enough so that they no longer run correctly, then poweroff wouldn't be able to complete, and then it's roasted CPU time. Tyan 2466 motherboards. This is only for emergency overheating, poweroff is fine for a normal shutdown. Thanks, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From asabigue at fing.edu.uy Thu Sep 11 06:19:38 2003 From: asabigue at fing.edu.uy (Ariel Sabiguero) Date: Thu, 11 Sep 2003 13:19:38 +0300 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <3F604C3A.3060008@fing.edu.uy> How about halt -p -f or poweroff -f .... I tried it and it takes 3s instead of 20 or more :-) From the man page: NOTES Under older sysvinit releases , reboot and halt should never be called directly. From release 2.74 on halt and reboot invoke shutdown(8) if the system is not in runlevel 0 or 6. This means that if halt or reboot cannot find out the cur? rent runlevel (for example, when /var/run/utmp hasn't been initialized correctly) shutdown will be called, which might not be what you want. Use the -f flag if you want to do a hard halt or reboot. It seems that it performs the sync call BEFORE killing the system but I think you should be careful. Regards Ariel David Mathog wrote: >Is there a faster way to shutdown a system and kill >the power than "poweroff"? This is for use in >emergency overheating conditions, where something >that's effectively the equivalent of: > > % sync > [turn off the power supply] > >is desired. > >lm_sensors and mondo are running on our Athlon 2200MP >cluster and I'm concerned that in the event of a >catastrophic failure of the cooling system the CPUs >might overheat before the poweroff sequence completes. >Not necessarily flame out immediately, but at least >get hot enough so that they no longer run correctly, >then poweroff wouldn't be able to complete, and then >it's roasted CPU time. > >Tyan 2466 motherboards. > >This is only for emergency overheating, poweroff is >fine for a normal shutdown. > >Thanks, > >David Mathog >mathog at caltech.edu >Manager, Sequence Analysis Facility, Biology Division, Caltech >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From asabigue at fing.edu.uy Thu Sep 11 07:04:18 2003 From: asabigue at fing.edu.uy (Ariel Sabiguero) Date: Thu, 11 Sep 2003 14:04:18 +0300 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <3F6056B2.2050303@fing.edu.uy> David Mathog wrote: >However it doesn't do so cleanly, neither >does: > > % sync ; poweroff -f > >In both cases there were inodes to fix and the disks were >rescanned on the next boot. Which is fine - much easier >to handle that sort of problem than fried hardware. > That depends on the data on disk. Sometimes it is cheaper to waste a CPU than to restore a highly complex DB. Maybe a journaled filesystem (jfs,ext3,reiser) could help at this point to give your filesystem the coherence that might get lost due to the poweroff. Good luck! Ariel > >Thanks, > >David Mathog >mathog at caltech.edu >Manager, Sequence Analysis Facility, Biology Division, Caltech > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathog at mendel.bio.caltech.edu Thu Sep 11 12:53:43 2003 From: mathog at mendel.bio.caltech.edu (David Mathog) Date: Thu, 11 Sep 2003 09:53:43 -0700 Subject: Kill the power faster than poweroff? Message-ID: > How about > > halt -p -f > > or > > poweroff -f > .... > I tried it and it takes 3s instead of 20 or more :-) You're right, that brings it down quickly, which is what is needed here. However it doesn't do so cleanly, neither does: % sync ; poweroff -f In both cases there were inodes to fix and the disks were rescanned on the next boot. Which is fine - much easier to handle that sort of problem than fried hardware. Thanks, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alan at dasher.wustl.edu Thu Sep 11 13:15:18 2003 From: alan at dasher.wustl.edu (alan at dasher.wustl.edu) Date: Thu, 11 Sep 2003 12:15:18 -0500 Subject: Kill the power faster than poweroff? In-Reply-To: Your message of "Thu, 11 Sep 2003 14:04:18 +0300." <3F6056B2.2050303@fing.edu.uy> Message-ID: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> :>However it doesn't do so cleanly, neither :>does: :> :> % sync ; poweroff -f :> :>In both cases there were inodes to fix and the disks were :>rescanned on the next boot. Which is fine - much easier :>to handle that sort of problem than fried hardware. I've read in other places that you need to repeat the sync command a number of times (I seem to remember the article saying 3 times in particular, though I can't remember how they arrived at that number) in order to guarantee writing. Even then it seems like there ought to be race-condition like issues. Still, have you tried sync; sleep 1; sync; poweroff -f Alan Grossfield --------------------------------------------------------------------------- | Only when we are willing to accept the universe for what it is, without | | myth or fear or prejudice, can we hope to build a truly just society. | | Lawrence Krauss, New York Times, 4/22/03 | --------------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 11 13:25:18 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 11 Sep 2003 10:25:18 -0700 Subject: cluster using handhelds In-Reply-To: References: <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030911102311.018aaca0@mailhost4.jpl.nasa.gov> Most excellent.. Very intriguing, although I do note that the interpocket website doesn't mention having tested it with a camera... I would think that products like this will become more common, as the "cellphone camera" craze propagates.. At 04:27 PM 9/11/2003 +0200, John Hearns wrote: >On Wed, 10 Sep 2003, Jim Lux wrote: > > > > > By the way, handhelds that can talk to a cmos/ccd camera and grab > frames on > > demand would be even better. > >Your wish is my command :-) > >http://www.interpocket.co.uk/ > >Probably you would have to team this with a cheap USB camera and a dual >Compact Flash jacket. Total cost is of course higher. > > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 11 14:01:21 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 11 Sep 2003 11:01:21 -0700 Subject: cluster using handhelds In-Reply-To: <20030911175818.BBFF227C67@email.iwon.com> Message-ID: <5.2.0.9.2.20030911105852.030d2878@mailhost4.jpl.nasa.gov> At 01:58 PM 9/11/2003 -0400, eoster at iwon.com wrote: >James, > > > > >A good starting point is the Center for Embedded Netowrk Sensing (CENS) at >http://www.cens.ucla.edu/ . Indeed.. the paper in IEEE proceedings from CENS is what got me started thinking about iPAQs. >Depending on what you are looking for there might already be >communities. CENS has done projects on iPAQs (in particular) that >includes (but is in no way limited to) running FFTs. There's even a >development framework called Em* being offered for development on these >platforms. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Thu Sep 11 13:21:30 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 11 Sep 2003 12:21:30 -0500 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <1063300889.24932.20.camel@caffeine> On Thu, 2003-09-11 at 11:53, David Mathog wrote: > You're right, that brings it down quickly, which is what > is needed here. However it doesn't do so cleanly, neither > does: > > % sync ; poweroff -f > > In both cases there were inodes to fix and the disks were > rescanned on the next boot. Which is fine - much easier > to handle that sort of problem than fried hardware. Sounds like a good case for journalled filesystems. ;-) -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eoster at iwon.com Thu Sep 11 13:58:18 2003 From: eoster at iwon.com (eoster at iwon.com) Date: Thu, 11 Sep 2003 13:58:18 -0400 (EDT) Subject: cluster using handhelds Message-ID: <20030911175818.BBFF227C67@email.iwon.com> James, You might want to take a look at sensor networks. There's actually a lot of effort going into connecting small devices (among whom are specifically iPAQs) and doing distributed computing (in-network collaboration and processing). A good starting point is the Center for Embedded Netowrk Sensing (CENS) at http://www.cens.ucla.edu/ . Depending on what you are looking for there might already be communities. CENS has done projects on iPAQs (in particular) that includes (but is in no way limited to) running FFTs. There's even a development framework called Em* being offered for development on these platforms. Good luck, Eric =================== Has anyone tried building a cluster using handheld computers (i.e. iPaq or Palm type) that use wireless LAN (802.11) or IR for the interconnect? Is there an implementation of MPI that will work in that environment (WinCE, PalmOS) (perhaps MPICH in some form?) Yes, I know the performance will be hideous. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 11 15:53:10 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 11 Sep 2003 15:53:10 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, David Mathog wrote: > You're right, that brings it down quickly, which is what > is needed here. However it doesn't do so cleanly, neither > does: > > % sync ; poweroff -f > > In both cases there were inodes to fix and the disks were > rescanned on the next boot. Which is fine - much easier > to handle that sort of problem than fried hardware. Inodes to fix on ext3? After a sync? Inquiring minds like to know... I'd expect a forced poweroff to be no worse than (and very similar to) just punching the power switch -- the sync should get the fs consistent EXCEPT for race condition output on open files belonging to active processes. Other ways of halting try to kill all those processes politely, in principle causing them to close all files that belong to them before the final sync. I'd also expect to not infrequently end up with a bunch of those little .nfs20800200 leftovers if the systems in question have active nfs mounts. rgb > > Thanks, > > David Mathog > mathog at caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From M.Arndt at science-computing.de Thu Sep 11 10:56:04 2003 From: M.Arndt at science-computing.de (Michael Arndt) Date: Thu, 11 Sep 2003 16:56:04 +0200 Subject: cluster using handhelds In-Reply-To: <3F6079E0.1020704@howard.edu>; from jhalpern@howard.edu on Thu, Sep 11, 2003 at 08:34:24AM -0500 References: <5.2.0.9.2.20030910135416.030d5ff8@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20030910151547.030e65e8@mailhost4.jpl.nasa.gov> <3F6079E0.1020704@howard.edu> Message-ID: <20030911165604.A97778@blnsrv1.science-computing.de> someone asked related to "handheld" cluster http://www.handhelds.org/projects/skiffcluster.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Thu Sep 11 11:30:18 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Thu, 11 Sep 2003 10:30:18 -0500 (CDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: nice little snmpset to an APC Masterswitch will do the trick. -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' On Thu, 11 Sep 2003, David Mathog wrote: > Is there a faster way to shutdown a system and kill > the power than "poweroff"? This is for use in > emergency overheating conditions, where something > that's effectively the equivalent of: > > % sync > [turn off the power supply] > > is desired. > > lm_sensors and mondo are running on our Athlon 2200MP > cluster and I'm concerned that in the event of a > catastrophic failure of the cooling system the CPUs > might overheat before the poweroff sequence completes. > Not necessarily flame out immediately, but at least > get hot enough so that they no longer run correctly, > then poweroff wouldn't be able to complete, and then > it's roasted CPU time. > > Tyan 2466 motherboards. > > This is only for emergency overheating, poweroff is > fine for a normal shutdown. > > Thanks, > > David Mathog > mathog at caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ashley at pittman.co.uk Thu Sep 11 13:03:30 2003 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 11 Sep 2003 18:03:30 +0100 Subject: data storage location In-Reply-To: References: <200309101726.h8AHQId22018@NewBlue.Scyld.com> Message-ID: <1063299810.1150.287.camel@ashley> > The alternative approach is to keep copies of the data on local disk on > each node. This gives you good IO rates, but you then have a substantial > data management problem; how to you copy 100Gb to each node in your > cluster in a sensible amount of time, and how do you update the data and > make sure it is kept consistent? > > The commonest approach to data distribution is to do some sort of > cascading rsync/rcp which follows the topology of your network. I've often wondered why there isn't some kind of a 'mpicp' program to do just this. I'd imagine the command line to be something like $ mpirun -allcps mpicp node0:~/myfile.dat /tmp/ This would then use MPI_Bcast to send the data to all the nodes. The assumption here is that MPI_Bcast is fairly efficient anyway so it's best to use it rather than writing your own cascading rsync algorithm. Ashley, _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From purikk at hotmail.com Thu Sep 11 12:47:48 2003 From: purikk at hotmail.com (purushotham komaravolu) Date: Thu, 11 Sep 2003 12:47:48 -0400 Subject: Beowulf And jvm Message-ID: Hi, I am a newbie to beowulf. I have a question about running Java on a Beowulf cluster. Is it possible that I can start only one JVM on one machine and the task be run distributed on the cluster? It is a multi-threaded application. Like say, I have an application with 100 threads. Can I have 50 threads run on one machine and 50 on another by launching the application(jvm) on one machine? I dont want to use MPI or any special code. Has anyone tried something like this before? Thanks Regards, Puru -------------- next part -------------- An HTML attachment was scrubbed... URL: From becker at scyld.com Thu Sep 11 15:27:11 2003 From: becker at scyld.com (Donald Becker) Date: Thu, 11 Sep 2003 15:27:11 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> Message-ID: On Thu, 11 Sep 2003 alan at dasher.wustl.edu wrote: > I've read in other places that you need to repeat the sync command a > number of times (I seem to remember the article saying 3 times in > particular, though I can't remember how they arrived at that number) in > order to guarantee writing. Even then it seems like there ought to be > race-condition like issues. I was told over two decades ago. The principle was that the first sync queued the pages to write, and the second sync didn't start until the first write list had finished. The third was just to make sure. You know: if two is good, three must be better. I don't believe that a second 'sync' will have any beneficial effect with Linux, or any modern Unix. Somewhat over ten years ago most filesystems added an "I was unmounted cleanly" bit. What you really need to do unmount the file systems, which will have the effect of really doing what "sync" is supposed to do. > Still, have you tried > sync; sleep 1; sync; poweroff -f -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Thu Sep 11 16:03:45 2003 From: jakob at unthought.net (Jakob Oestergaard) Date: Thu, 11 Sep 2003 22:03:45 +0200 Subject: Kill the power faster than poweroff? In-Reply-To: References: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> Message-ID: <20030911200345.GB19677@unthought.net> On Thu, Sep 11, 2003 at 03:27:11PM -0400, Donald Becker wrote: > On Thu, 11 Sep 2003 alan at dasher.wustl.edu wrote: > > I've read in other places that you need to repeat the sync command a > > number of times (I seem to remember the article saying 3 times in > > particular, though I can't remember how they arrived at that number) in > > order to guarantee writing. Even then it seems like there ought to be > > race-condition like issues. > > I was told over two decades ago. The principle was that the first sync > queued the pages to write, and the second sync didn't start until the > first write list had finished. The third was just to make sure. You > know: if two is good, three must be better. The story I heard was, that sync could complete before all data had actually reached the platters (hardware write buffers or whatever...). Therefore, you had to *type* sync three times. No cheap scripting tricks - the whole idea of the two last syncs is to get some delay before you actually power off. > > I don't believe that a second 'sync' will have any beneficial effect > with Linux, or any modern Unix. And there is no state. Thus, the second sync must do exactly the same as the first sync. There is no way that it can have another effect. If you type sync three times, whatever write buffers there may be which the sync commands cannot sync, will have had time to flush. You could also type ]$ sync ]$ my ]$ disks ]$ please It just gives errors and makes you look stupid, if someone is watching over your shoulder ;) -- ................................................................ : jakob at unthought.net : And I see the elder races, : :.........................: putrid forms of man : : Jakob ?stergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Thu Sep 11 16:44:52 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Thu, 11 Sep 2003 13:44:52 -0700 Subject: Kill the power faster than poweroff? In-Reply-To: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> References: <3F6056B2.2050303@fing.edu.uy> <200309111715.h8BHFIXn000648@blitzen.wustl.edu> Message-ID: <20030911204452.GD1246@greglaptop.internal.keyresearch.com> On Thu, Sep 11, 2003 at 12:15:18PM -0500, alan at dasher.wustl.edu wrote: > I've read in other places that you need to repeat the sync command a > number of times (I seem to remember the article saying 3 times in > particular, though I can't remember how they arrived at that number) This is old advice, and it doesn't really apply to modern Linux or Unix. In Linux, the sync syscall and utility returns immediately after scheduling all the writes, instead of after completing them. In the good old days, there was probably a worry that if it took a while to write all the dirty blocks, there would be new dirty blocks, so 3 syncs in a row would pretty much have everything clean. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 11 17:13:10 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 11 Sep 2003 17:13:10 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, Donald Becker wrote: > Somewhat over ten years ago most filesystems added an "I was unmounted > cleanly" bit. What you really need to do unmount the file systems, > which will have the effect of really doing what "sync" is supposed to do. But I >>think<< that unless you kill all processes that own open files on the filesystem (or otherwise force a close) umount will complain about open files and not proceed even with e.g. the -f flag. If you do kill all processes to be sure you can run umount, then you're basically doing a full shutdown and back where you started. Or you can run lsof, filter out the files that are open for writing (and maybe reading -- don't know how umount manages files open for reading) and kill their controlling process. Quickly umount to avoid the a race on new processes opening files (which you could still lose, especially if the file owning processes are children of a process waiting to open another file owning process when its children terminate), cross your fingers, power down. Not horribly safe or elegant, but ought to get a good fraction of all open files especially on a node with a very controlled task list that owns open writeable files in the first place. I personally agree with other earlier remarks: use a journalling filesystem, do a sync to minimize at least some problems, and then just power off. You may well lose data in the write queues of open processes (as you would likely do anyway if you killed the processes) but you shouldn't end up with the system in a state that needs an fsck any more than you would if you just powered off without even the sync. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Sep 11 17:40:04 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Thu, 11 Sep 2003 17:40:04 -0400 Subject: Beowulf And jvm In-Reply-To: References: Message-ID: <1063316404.21466.7.camel@protein.scalableinformatics.com> Hi Puru: Java uses shared memory threads. It is unlikely that the beowulf architecture would be suitable for this. However, if you spawn multiple processes and create some sort of IPC mechanism (MPI is but one known functional instance of this), you might be able to get some benefit out of running on multiple CPUs across multiple nodes. If your threads are not latency sensitive, you could use a variety of IPC mechanisms. On Thu, 2003-09-11 at 12:47, purushotham komaravolu wrote: > Hi, > I am a newbie to beowulf. > I have a question about running Java on a Beowulf cluster. > Is it possible that I can start only one JVM on one machine and the > task > be run distributed on the cluster? It is a multi-threaded application. > Like say, I have an application with 100 threads. > Can I have 50 threads run on one machine and 50 on another by > launching > the application(jvm) on one machine? I dont want to use MPI or any > special code. > > Has anyone tried something like this before? > Thanks > Regards, > Puru -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Thu Sep 11 19:42:04 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Thu, 11 Sep 2003 16:42:04 -0700 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <20030911234204.GC2336@greglaptop.internal.keyresearch.com> On Thu, Sep 11, 2003 at 05:13:10PM -0400, Robert G. Brown wrote: > But I >>think<< that unless you kill all processes that own open files > on the filesystem (or otherwise force a close) umount will complain > about open files and not proceed even with e.g. the -f flag. If you do > kill all processes to be sure you can run umount, then you're basically > doing a full shutdown and back where you started. Yes, but. Linux shutdown kills everyone nicely. Kill -9 is much faster, and allows the umount. The *BSDs use death and distruction on the way down, and last I looked at it, took a couple of seconds to run their entire shutdown. No waiting for all the apps to call free() on all their data structures... swap in all their old, unused data pages... -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Fri Sep 12 00:35:28 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 12 Sep 2003 14:35:28 +1000 Subject: Kill the power faster than poweroff? In-Reply-To: <20030911200345.GB19677@unthought.net> References: <200309111715.h8BHFIXn000648@blitzen.wustl.edu> <20030911200345.GB19677@unthought.net> Message-ID: <200309121435.30380.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 12 Sep 2003 06:03 am, Jakob Oestergaard wrote: > Therefore, you had to *type* sync three times. No cheap scripting > tricks - the whole idea of the two last syncs is to get some delay > before you actually power off. This certainly used to be the case. It's just a handy habit and if it doesn't cost you anything then why not. I know my 2.4.x Linux boxes show disk activity when I type sync, so it's certainly not worthless. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/YU0RO2KABBYQAh8RAnArAJ4uWHJsim9RmqTQJNh09dW/i9ZqTgCeOKmY qJOxQPL8X33HWpWJO4O3cwk= =YnHT -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Fri Sep 12 05:13:16 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Fri, 12 Sep 2003 11:13:16 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, Robert G. Brown wrote: > But I >>think<< that unless you kill all processes that own open files > on the filesystem (or otherwise force a close) umount will complain > about open files and not proceed even with e.g. the -f flag. If you do > kill all processes to be sure you can run umount, then you're basically > doing a full shutdown and back where you started. If your processes only *read* data, you could also try mount -o ro,/path to remount the filesystem read-only before turning the power off. This should write all pending blocks to disk and set the file system's clean bit. I haven't tried it, so it might fail when there are still processes with files open for writing on that file system. - Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 12 05:53:57 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 12 Sep 2003 11:53:57 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Thu, 11 Sep 2003, David Mathog wrote: > Is there a faster way to shutdown a system and kill > the power than "poweroff"? This is for use in > emergency overheating conditions, where something > that's effectively the equivalent of: > You can use APC power distribution units. These are network-addressable and can switch on and off power to a set of outlets. We use them on our clusters. Quick look at the the Opteron cluster next to me - model number is AP9212. (I'm on customer site today) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 12 06:00:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 12 Sep 2003 12:00:11 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: ps. given all the discussion on syncing and journalled filesystems, I should say that I mention the APC units in direct response to the original question on alternative ways for fast shutdown. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Fri Sep 12 07:39:59 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Fri, 12 Sep 2003 13:39:59 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Fri, 12 Sep 2003, Juan Gallego wrote: > On 2003-09-12 11:13+0200, Felix Rauch wrote: > | mount -o ro,/path > > `mount -o ro,remount' /path maybe? Uhm, exactly, that's what I actually wanted to write... Thanks and regards, Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H18 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Fri Sep 12 06:00:57 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Fri, 12 Sep 2003 11:00:57 +0100 Subject: Kill the power faster than poweroff? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE11F@stegosaurus.bristol.quadrics.com> David, For information, both flavours of Opteron that I have here have an out-of-band BMC that I can issue remote power-on/ power-off commands to over ethernet.. Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- > -----Original Message----- > From: John Hearns [mailto:john.hearns at clustervision.com] > Sent: 12 September 2003 10:54 > To: beowulf at beowulf.org > Subject: Re: Kill the power faster than poweroff? > > > On Thu, 11 Sep 2003, David Mathog wrote: > > > Is there a faster way to shutdown a system and kill > > the power than "poweroff"? This is for use in > > emergency overheating conditions, where something > > that's effectively the equivalent of: > > > You can use APC power distribution units. > These are network-addressable and can switch on and off power to > a set of outlets. We use them on our clusters. > > > Quick look at the the Opteron cluster next to me - > model number is AP9212. (I'm on customer site today) > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Sep 12 06:55:43 2003 From: becker at scyld.com (Donald Becker) Date: Fri, 12 Sep 2003 06:55:43 -0400 (EDT) Subject: Beowulf And jvm In-Reply-To: <1063316404.21466.7.camel@protein.scalableinformatics.com> Message-ID: On Thu, 11 Sep 2003, Joseph Landman wrote: > Java uses shared memory threads. ...and it doesn't structure the sharing of information between threads. > It is unlikely that the beowulf architecture would be suitable for > this. There are Network Virtual Memory (*) systems that will emulate shared memory using virtual memory page protections. But they are very slow unless you restructure your application to avoid page bounces, and when you do that you end up with a system that is equivalent to doing message passing. (*) AKA DSM -- Distributed Shared Memory, but that name is also used for hardware-based, per-word memory coherency) > However, if you spawn multiple > processes and create some sort of IPC mechanism (MPI is but one known > functional instance of this), you might be able to get some benefit out > of running on multiple CPUs across multiple nodes. If your threads are > not latency sensitive, you could use a variety of IPC mechanisms. There are several Java-specific systems, such as JavaSpaces, that provide communication libraries. In general - You must still restructure the code - Support for low latency interconnects does not exist + They usually have a clean Java interface (vs. a MPI wrapper) My impression from others (disclaimer: not being a Java programmer myself), is that cluster Java applications are written with explicit distributed layout. Independent address spaces, which may be multithreaded, are run on the nodes and communicate without using a cluster-specific library. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Sep 12 09:54:17 2003 From: becker at scyld.com (Donald Becker) Date: Fri, 12 Sep 2003 09:54:17 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: <010C86D15E4D1247B9A5DD312B7F5AA78DE11F@stegosaurus.bristol.quadrics.com> Message-ID: On Fri, 12 Sep 2003, Daniel Kidger wrote: > For information, both flavours of Opteron that I have here have an > out-of-band BMC that I can issue remote power-on/ power-off commands to over > ethernet.. I'm a big fan of network IPMI 1.5, which is usually implemented by a Baseboard Management Controller (BMC). The BMC is a microcontroller running on stand-by power. It can control power, redirect the "serial" console to the network, monitor temp/voltage/switches, configure the BIOS and do a few other things. The cleanest IPMI implementation uses an enhanced Ethernet controller that redirects IPMI packets to the BMC instead of the OS. This allows installing a machine by plugging in only a power cord and Ethernet cable. Everything else may be done over the network. But my understanding is that the issue here is the software shutdown time. The big culprit in the slow shutdown is the daemons. We solved this problem for cluster compute nodes in the Scyld system by developing a unique system that does not require daemons. Thus we can cleanly shut down a slave node almost instantly. But we want a standard, full-featured Linux installation as a cluster master, and have the same slow shutdown issue there. So why does a 3GHz machine takes 10X more time to boot and shutdown than a slow machine with an earlier Linux installation? The biggest culprits are /etc/rc.d/init.d/functions /etc/rc.d/init.d/halt which do explicit sleeps and have very poor scaling. Fixing the fundamental problems is difficult (you can just revert to an older version), but it's easy to pick better numbers for the 'usleep' and 'sleep' calls. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Fri Sep 12 10:06:39 2003 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Fri, 12 Sep 2003 09:06:39 -0500 Subject: java and cluster Message-ID: <20030912090639.G5739@mikee.ath.cx> Accidently sent this directly to Donald Becker: On Fri, 12 Sep 2003, Donald Becker wrote: > My impression from others (disclaimer: not being a Java programmer > myself), is that cluster Java applications are written with explicit > distributed layout. Independent address spaces, which may be > multithreaded, are run on the nodes and communicate without using a > cluster-specific library. I used a JNI wrapper around PVM for my applications. Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathog at mendel.bio.caltech.edu Fri Sep 12 11:11:12 2003 From: mathog at mendel.bio.caltech.edu (David Mathog) Date: Fri, 12 Sep 2003 08:11:12 -0700 Subject: Kill the power faster than poweroff? Message-ID: On Fri, 12 Sep 2003 becker at scyld.com wrote: > The biggest culprits are > /etc/rc.d/init.d/functions > /etc/rc.d/init.d/halt > which do explicit sleeps and have very poor scaling. Replace the explicit seconds/microseconds with variables and conditionally set those variables to zero if "/fastshutdown" exists? Then a regular "poweroff" would execute without all the sleeps. I wonder though if it would work, since some of those sleeps are in there to let daemons shut down nicely before going on to the next thing. I'll have to try this at some point nd see what happens. Regards, David Mathog mathog at caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Fri Sep 12 11:34:42 2003 From: jcownie at etnus.com (James Cownie) Date: Fri, 12 Sep 2003 11:34:42 -0400 Subject: Beowulf And jvm In-Reply-To: Message from Donald Becker of "Fri, 12 Sep 2003 06:55:43 EDT." Message-ID: <200309121534.h8CFYgc32508@alien.etnus.com> If you want to rewrite code into explicitly parallel Java there's Berkeley's Titanium... http://www.cs.berkeley.edu/Research/Projects/titanium/ Titanium is an explicitly parallel dialect of Java developed at UC Berkeley to support high-performance scientific computing on large-scale multiprocessors, including massively parallel supercomputers and distributed-memory clusters with one or more processors per node. Other language goals include safety, portability, and support for building complex data structures. ... etc ... -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From abijur at craft-tech.com Fri Sep 12 12:18:29 2003 From: abijur at craft-tech.com (Ashwin Bijur) Date: Fri, 12 Sep 2003 12:18:29 -0400 Subject: channel bonding/trunking and diskless booting Message-ID: <3F61F1D5.5040504@craft-tech.com> Hi, Can anybody help me with the steps required to configure a linux cluster node with channel bonding (or trunking) and diskless booting? We use Redhat Linux 8.0, Pentium III dual processor PCs. Any help is greatly appreciated. Thanks, Ashwin Bijur Asst. Systems Administrator. Combustion Research and Flow Technology, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Fri Sep 12 12:43:27 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Fri, 12 Sep 2003 18:43:27 +0200 Subject: data storage location In-Reply-To: References: <200309101726.h8AHQId22018@NewBlue.Scyld.com> Message-ID: <20030912184327Q.hanzl@unknown-domain> > The alternative approach is to keep copies of the data on local disk on > each node. This gives you good IO rates, but you then have a substantial > data management problem; how to you copy 100Gb to each node in your > cluster in a sensible amount of time, and how do you update the data and > make sure it is kept consistent? > > ... > > If your dataset is larger than the amount of local disk on your nodes, you > then have to partition your data up, and integrate that with your queuing > system, so that jobs which need a certain bit of the data end up on a node > which actually holds a copy. This is exactly what we do. But moving the right data at right place and doing good job scheduling at the same time is not easy. Ideally we would like to automate it via huge caches on local disks: - central server has some 400GB of read-only data - nodes cache them on their harddisks as needed - queing system preferes some regular patterns in job/node assignment to make cache hits likely Cache-like behavior would save a lot of manual work but unfortunately I am not aware of any working solution for linux, I want something like cachefs (nonexistent for linux) or caching ability of AFS/Coda (too cumbersome for cluster) or theoretical features of Intermezzo (still and maybe forever unfinished). At the moment we work on small kernel hack to solve this, unfortunately repeating for the n-th time what many others once did and never maintained. Maybe genome research will generate more need for this data access pattern and more chance for re-usable software solution? Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Fri Sep 12 13:04:14 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Fri, 12 Sep 2003 13:04:14 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: > I'm a big fan of network IPMI 1.5, which is usually implemented by a > Baseboard Management Controller (BMC). The BMC is a microcontroller > running on stand-by power. It can control power, redirect the "serial" IPMI and BMC are attractive, but as far as I can tell, completely non-viable for moderately-priced nodes. I mean, if my 1-u dual costs $US 2k, adding a BMC probably means a 35-50% boost in price. or are there cheap BMC options? otherwise, it seems like a holdover from the world of "enterprise" gold-plated servers, where $1k on a $50K server is in the noise... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Fri Sep 12 13:20:34 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Fri, 12 Sep 2003 10:20:34 -0700 (PDT) Subject: data storage location In-Reply-To: <20030912184327Q.hanzl@unknown-domain> Message-ID: On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > The alternative approach is to keep copies of the data on local disk on > > each node. This gives you good IO rates, but you then have a substantial > > data management problem; how to you copy 100Gb to each node in your > > cluster in a sensible amount of time, and how do you update the data and > > make sure it is kept consistent? > > > > ... > > > > If your dataset is larger than the amount of local disk on your nodes, you > > then have to partition your data up, and integrate that with your queuing > > system, so that jobs which need a certain bit of the data end up on a node > > which actually holds a copy. > > This is exactly what we do. But moving the right data at right place > and doing good job scheduling at the same time is not easy. Ideally we > would like to automate it via huge caches on local disks: > > - central server has some 400GB of read-only data > - nodes cache them on their harddisks as needed > - queing system preferes some regular patterns in job/node assignment > to make cache hits likely > > Cache-like behavior would save a lot of manual work but unfortunately > I am not aware of any working solution for linux, I want something > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > (too cumbersome for cluster) or theoretical features of > Intermezzo (still and maybe forever unfinished). it's not clear to me that coda is too cumbersome for clusters... but the other approach to scale the storage infrastructure to the point where it's performance approaches that of locally attached disk. there's a lot of work that's been done on san and iscsi architecture for storage clusters that seems relevant to the issues of compute clusters as well... > At the moment we work on small kernel hack to solve this, > unfortunately repeating for the n-th time what many others once did > and never maintained. > > Maybe genome research will generate more need for this data access > pattern and more chance for re-usable software solution? > > Regards > > Vaclav Hanzl > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeff at aslab.com Fri Sep 12 16:03:59 2003 From: jeff at aslab.com (Jeff Nguyen) Date: Fri, 12 Sep 2003 13:03:59 -0700 Subject: Kill the power faster than poweroff? References: Message-ID: <20a001c37969$01b3bf70$6502a8c0@jeff> > > I'm a big fan of network IPMI 1.5, which is usually implemented by a > > Baseboard Management Controller (BMC). The BMC is a microcontroller > > running on stand-by power. It can control power, redirect the "serial" > > IPMI and BMC are attractive, but as far as I can tell, completely > non-viable for moderately-priced nodes. I mean, if my 1-u dual costs > $US 2k, adding a BMC probably means a 35-50% boost in price. > > or are there cheap BMC options? otherwise, it seems like a holdover > from the world of "enterprise" gold-plated servers, where $1k on a > $50K server is in the noise... There is low cost IPMI/BMC interface at a fraction of the system cost. Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Fri Sep 12 13:23:43 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Fri, 12 Sep 2003 18:23:43 +0100 Subject: Kill the power faster than poweroff? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE129@stegosaurus.bristol.quadrics.com> > > I'm a big fan of network IPMI 1.5, which is usually implemented by a > > Baseboard Management Controller (BMC). The BMC is a microcontroller > > running on stand-by power. It can control power, redirect > the "serial" > > IPMI and BMC are attractive, but as far as I can tell, completely > non-viable for moderately-priced nodes. I mean, if my 1-u dual costs > $US 2k, adding a BMC probably means a 35-50% boost in price. > > or are there cheap BMC options? otherwise, it seems like a holdover > from the world of "enterprise" gold-plated servers, where $1k on a > $50K server is in the noise... I would suggest that the majority of new dual CPU rack mounted nodes have BMCs on the MoBo. Certainly all the 'big band-name' ones seem to. Intel ones tend to have the BMC hijacking the main eth0. Others (Serverworks, Khepri, AMD, ZX1 et al) tend to use a seperate ethernet socket or hijack the serial port. Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Sep 12 17:50:35 2003 From: becker at scyld.com (Donald Becker) Date: Fri, 12 Sep 2003 17:50:35 -0400 (EDT) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Fri, 12 Sep 2003, Mark Hahn wrote: > > I'm a big fan of network IPMI 1.5, which is usually implemented by a > > Baseboard Management Controller (BMC). The BMC is a microcontroller > > running on stand-by power. It can control power, redirect the "serial" > > IPMI and BMC are attractive, but as far as I can tell, completely > non-viable for moderately-priced nodes. I mean, if my 1-u dual costs > $US 2k, adding a BMC probably means a 35-50% boost in price. I wouldn't be a big fan if either it was too expensive to be common it was proprietary I've seen quotes for around $30 extra when bundled with the system board. Of course finding a BMC daughterboard later is difficult and likely expensive, so you have to specify it with the system. > or are there cheap BMC options? otherwise, it seems like a holdover > from the world of "enterprise" gold-plated servers, where $1k on a > $50K server is in the noise... And you thought the option -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 13 05:34:12 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 13 Sep 2003 11:34:12 +0200 (CEST) Subject: data storage location In-Reply-To: <20030912184327Q.hanzl@unknown-domain> Message-ID: On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > Cache-like behavior would save a lot of manual work but unfortunately > I am not aware of any working solution for linux, I want something > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > (too cumbersome for cluster) or theoretical features of Why do you say AFS is too cumbersome? I'm just saying that to provoke a debate - I've never actually set up an AFS infrastructure from scratch (kerberos, kernel patches etc...) but I know its not an afternoon's work. I have had call to work closely with AFS, doing a bit of performance tuning for caches etc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 13 05:39:02 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 13 Sep 2003 11:39:02 +0200 (CEST) Subject: Kill the power faster than poweroff? In-Reply-To: Message-ID: On Fri, 12 Sep 2003, Donald Becker wrote: HAs anyone got specifics for BMCs for HDAMA motherboards? Are they now available, cost etc. In fact, a nice summary of BMC capabilities, and which motherboards have or support them would be excellent. I guess it may be a little early in the game for that, though pointers welcome. Also, if anyoen has got lm_sensors working on HDAMA with a 2.4.22 kernel I'd be interested to chat off-list. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Sat Sep 13 07:56:38 2003 From: becker at scyld.com (Donald Becker) Date: Sat, 13 Sep 2003 07:56:38 -0400 (EDT) Subject: data storage location In-Reply-To: <20030912184327Q.hanzl@unknown-domain> Message-ID: On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > The alternative approach is to keep copies of the data on local disk on > > each node. This gives you good IO rates, but you then have a substantial > > data management problem; how to you copy 100Gb to each node in your > > cluster in a sensible amount of time, and how do you update the data and > > make sure it is kept consistent? One significant question is the access pattern and requirement. Is there a single or multiple application access patterns? Does the application read all or only a subset of the data? Is the subset predictable? Externally by a scheduler? Does the application step linearly through the data, or randomly access? If linear stepping, is the state space of the application small? If small, as with a best-match search, the processing time per file byte tends to be small. We recommend a static split of the data across machines and migrating the process instead. In the case of a single file read we can often do this without modifying the application, or localize the changes to a few lines around the read loop. If large, e.g. building a tree in memory from the data, what is the per-byte processing time? across machines and migrate the process. How is the data set updated? A read-only data set allows many file handling options. If files are updated as a whole, you may use a caching versioned file system. That is specialized, but provides many opportunities for optimization Handling arbitrary writes in the middle of files requires a consistent file system, and the cost for consistency is very high. > Cache-like behavior would save a lot of manual work but unfortunately > I am not aware of any working solution for linux, Several exist. It depends on the semantics you need, as doing this efficiently requires making assumptions about the access patterns and desired semantics. > or caching ability of AFS/Coda > (too cumbersome for cluster) or theoretical features of > Intermezzo (still and maybe forever unfinished). ...Declare it a success and move on -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Sat Sep 13 09:16:28 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 13 Sep 2003 09:16:28 -0400 Subject: data storage location In-Reply-To: References: Message-ID: <1063458985.31562.9.camel@QUIGLEY.LINIAC.UPENN.EDU> On Sat, 2003-09-13 at 05:34, John Hearns wrote: > On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > I am not aware of any working solution for linux, I want something > > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > > (too cumbersome for cluster) or theoretical features of > Why do you say AFS is too cumbersome? > I'm just saying that to provoke a debate - I've never actually set up > an AFS infrastructure from scratch (kerberos, kernel patches etc...) > but I know its not an afternoon's work. > I have had call to work closely with AFS, doing a bit of performance > tuning for caches etc. Ok, I'll bite :) We have been exploring filesystem options here, trying to get away from NFS. Our 'architecture' is 128 nodes with FastE, and 4 IO servers, each with ~600 GB of RAID5 and GigE. At first glance, AFS, really OpenAFS in this case, seemed like a dream come true. There was the ability to have mutliple file servers in a global namespace, support for doing amanda backups, could let users export their own data to any machine that is an AFS client -- even across the country/world. Now... for the practical side. When I went to implement it, I was unaware how much a pain it could be to setup the kerberos stuff. At Penn we have a campus wide krb5 setup, and getting users to use krb5 logins was not a problem. However, openAFS needs a krb4 ticket to work, and Penn will not add the krb524 translator. To get around this, it was necessary to do a bunch of hacks to store the krb4 ticket on the machine itself, and tell krb5 that localhost was a valid krb524 server. This may not sound too bad, but to figure it out, and then to implement correctly is such a major PITA, I almost dumped the project there. So, now that it is talkint to the kerberos setup just fine, it was very apparent that having your ticket timeout was a very annoying problem. -- Basically you have to do some really ugly script/hack to have a program remember all of the user's passwords, either in memory or in a gpg protected file, and periodically check for a ticket that was about to expire, and re klogin for that user. Not pretty. The second issue, and the real deal killer is the lack of support for >2GB files. Absolutely unacceptable. We do natural language processing on one cluster, and genomics on another, with files that regularily exceed 2GB. So... in conclusion. IF openAFS could support >2GB files, and the krb5 mess could be cleanedup, it might work. For now, we will be installing & playing with Lustre. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 13 10:13:54 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 13 Sep 2003 10:13:54 -0400 (EDT) Subject: data storage location In-Reply-To: Message-ID: On Sat, 13 Sep 2003, John Hearns wrote: > On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > I am not aware of any working solution for linux, I want something > > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > > (too cumbersome for cluster) or theoretical features of > Why do you say AFS is too cumbersome? > I'm just saying that to provoke a debate - I've never actually set up > an AFS infrastructure from scratch (kerberos, kernel patches etc...) > but I know its not an afternoon's work. > I have had call to work closely with AFS, doing a bit of performance > tuning for caches etc. AFS is in wide use at Duke (it is the basis of the students' globally shared home directory system on campus). It is doable administratively, although as you note it isn't an afternoon's work. However, it isn't likely to be a really good cluster fs. Reasons: a) It is even less efficient and more resource intensive than NFS (unsurprising, as it does more) b) It can (unless you take care with e.g. kerberos tickets) produce "odd" and undesirable behavior such as losing the ability to write to a file you are holding open after x hours unless/until you re-authenticate, blocking the process that owns the open file c) Its caching behavior is insane and (in my opinion) unreliable, at least as of the last time I tried it. By this I mean that I have directly experienced the following AFS bug/feature: System A opens file output in AFS directory System A writes lots of stuff into output System A fflushes and keeps running, output is still open System B wants to graze the current contents of output, so it opens output for reading. What does system B find? According to standard/reliable programming conventions, the use of fflush should force a write of all cached data back to the file. AFS, however, only flushes the data (apparently) back to its LOCAL cache/image of the file on system A, and only resync's with the global image when the file is closed. So System B will read nothing of what System A has written until A closes the file. So sure, system A could close and open the file repeatedly, but this adds a lot of overhead as open/close is expensive (open requires a full stat, checking permissions, etc.). Most of this is manageable and one can learn how to work with the system in a cluster environment for at least certain classes of task, but it is (as the man says:-) "cumbersome", as to a certain extent is administration (file acls are more powerful but also require more thought and human energy, etc). Too cumbersome is up to personal judgement. Many clusters use a user filesystem only to launch tasks and permit a single results file to eventually be written back -- "anything" can be made to work there, and AFS would work as well as NFS or whatever. Other cluster tasks might well be unsuitable as in my system A/B example, although with foreknowledge this can be coped with. Hope this helps. I personally no longer use AFS so perhaps the fflush "bug" has been fixed, and we have/do/are still meditating upon using AFS (since we have it running) in at least some capacity on a public cluster we're engineering, but it isn't a knee-jerk thing to do EVEN if it is already there, and will definitely require a bit of consideration if you have to set it up from scratch to use it at all. rgb > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rodmur at maybe.org Sat Sep 13 10:48:14 2003 From: rodmur at maybe.org (Dale Harris) Date: Sat, 13 Sep 2003 07:48:14 -0700 Subject: data storage location In-Reply-To: <1063458985.31562.9.camel@QUIGLEY.LINIAC.UPENN.EDU> References: <1063458985.31562.9.camel@QUIGLEY.LINIAC.UPENN.EDU> Message-ID: <20030913144814.GB6831@maybe.org> On Sat, Sep 13, 2003 at 09:16:28AM -0400, Nicholas Henke elucidated: > > Ok, I'll bite :) We have been exploring filesystem options here, trying > to get away from NFS. Our 'architecture' is 128 nodes with FastE, and 4 > IO servers, each with ~600 GB of RAID5 and GigE. > Any impressions on NFS4? I notice it's included, at least experimentally, in the 2.6 kernel. -- Dale Harris rodmur at maybe.org /.-) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From epaulson at cs.wisc.edu Sat Sep 13 11:10:11 2003 From: epaulson at cs.wisc.edu (Erik Paulson) Date: Sat, 13 Sep 2003 10:10:11 -0500 Subject: data storage location In-Reply-To: ; from rgb@phy.duke.edu on Sat, Sep 13, 2003 at 10:13:54AM -0400 References: Message-ID: <20030913101008.A3238@chopin.cs.wisc.edu> On Sat, Sep 13, 2003 at 10:13:54AM -0400, Robert G. Brown wrote: > On Sat, 13 Sep 2003, John Hearns wrote: > > > On Fri, 12 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > > > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > > I am not aware of any working solution for linux, I want something > > > like cachefs (nonexistent for linux) or caching ability of AFS/Coda > > > (too cumbersome for cluster) or theoretical features of > > Why do you say AFS is too cumbersome? > > I'm just saying that to provoke a debate - I've never actually set up > > an AFS infrastructure from scratch (kerberos, kernel patches etc...) > > but I know its not an afternoon's work. > > I have had call to work closely with AFS, doing a bit of performance > > tuning for caches etc. > > AFS is in wide use at Duke (it is the basis of the students' globally > shared home directory system on campus). It is doable administratively, > although as you note it isn't an afternoon's work. However, it isn't > likely to be a really good cluster fs. Reasons: > > a) It is even less efficient and more resource intensive than NFS > (unsurprising, as it does more) > > b) It can (unless you take care with e.g. kerberos tickets) produce > "odd" and undesirable behavior such as losing the ability to write to a > file you are holding open after x hours unless/until you > re-authenticate, blocking the process that owns the open file > First of all, you should be relaying on your batch system to manage the credentials for your job for you. Then, you really should be prepared to deal with that anyway - an NFS server can go away at any time too. > c) Its caching behavior is insane and (in my opinion) unreliable, at > least as of the last time I tried it. By this I mean that I have > directly experienced the following AFS bug/feature: > > System A opens file output in AFS directory > System A writes lots of stuff into output > System A fflushes and keeps running, output is still open > System B wants to graze the current contents of output, so it opens > output for reading. > > What does system B find? According to standard/reliable programming > conventions, the use of fflush should force a write of all cached data > back to the file. AFS, however, only flushes the data (apparently) back > to its LOCAL cache/image of the file on system A, and only resync's with > the global image when the file is closed. So System B will read nothing > of what System A has written until A closes the file. > > So sure, system A could close and open the file repeatedly, but this > adds a lot of overhead as open/close is expensive (open requires a full > stat, checking permissions, etc.). > > Most of this is manageable and one can learn how to work with the system > in a cluster environment for at least certain classes of task, but it is > (as the man says:-) "cumbersome", as to a certain extent is > administration (file acls are more powerful but also require more > thought and human energy, etc). Too cumbersome is up to personal > judgement. Many clusters use a user filesystem only to launch tasks and > permit a single results file to eventually be written back -- "anything" > can be made to work there, and AFS would work as well as NFS or > whatever. Other cluster tasks might well be unsuitable as in my system > A/B example, although with foreknowledge this can be coped with. > > Hope this helps. I personally no longer use AFS so perhaps the fflush > "bug" has been fixed, None of the AFS people will tell you it's a bug. AFS is a naturally more file-oriented system - AFS caches whole files, not subblocks of the file, so it makes sense that changes are propgated back to the server only at close() (and hey, watch out - close can fail - how many people actually check the return code from close?). AFS is probably a big win for regular users, where there's not much active sharing between processes. However, if you're using the filesystem for coordination between multiple processes, AFS is not for you. > and we have/do/are still meditating upon using AFS > (since we have it running) in at least some capacity on a public cluster > we're engineering, but it isn't a knee-jerk thing to do EVEN if it is > already there, and will definitely require a bit of consideration if you > have to set it up from scratch to use it at all. > For a shared global namespace filesystem that actually does some sort of authentication, AFS is really the only game in town. I can't imagine doing a campus-wide NFS or PVFS setup... -Erik _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bernd-schubert at web.de Sat Sep 13 11:43:26 2003 From: bernd-schubert at web.de (Bernd Schubert) Date: Sat, 13 Sep 2003 17:43:26 +0200 Subject: Kill the power faster than poweroff? In-Reply-To: References: Message-ID: <200309131743.26320.bernd-schubert@web.de> On Thursday 11 September 2003 16:53, David Mathog wrote: > Is there a faster way to shutdown a system and kill > the power than "poweroff"? This is for use in > emergency overheating conditions, where something > that's effectively the equivalent of: > > % sync > [turn off the power supply] > Hi, since linux-2.4.21 the magic-sysrq's can be trigger via keyboard. echo s > /proc/sysrq-trigger echo u >/proc/sysrq-trigger echo o /proc/sysrq-trigger takes less then a second to turn my system off. Cheers, Bernd _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 13 17:34:13 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 13 Sep 2003 17:34:13 -0400 (EDT) Subject: data storage location In-Reply-To: <20030913101008.A3238@chopin.cs.wisc.edu> Message-ID: On Sat, 13 Sep 2003, Erik Paulson wrote: > None of the AFS people will tell you it's a bug. AFS is a naturally > more file-oriented system - AFS caches whole files, not subblocks of the > file, so it makes sense that changes are propgated back to the server only > at close() (and hey, watch out - close can fail - how many people actually > check the return code from close?). AFS is probably a big win for regular > users, where there's not much active sharing between processes. However, > if you're using the filesystem for coordination between multiple processes, > AFS is not for you. Well, I did put bug/feature together, or used quotes, because I know they don't think it is a bug, but I still get/got somewhat annoyed when standard systems calls behave(d) in unexpected ways. Technically, of course, fflush only flushes user level buffers to the kernel, but leaves it to the kernel to flush them back to the actual file (which it does according to the dictates of the filesystem and so forth. So technically it isn't a real bug, just an arguable/debateable design decision. Some stuff like this made more sense a decade ago than it does now, as well, as overall capacities of a cheap >>client<< now exceed the capacity of most of the most expensive >>servers<< in use back then by a Moore's Law Mile. The feature can bite the hell out of you anyway, giving a whole new meaning to the term "race condition" if you're not aware of the possibility. As in one usually thinks of a race condition as being "short", and NFS is very good, really, at ensuring that any cached writebacks are written physically to disk (or serviced out of its cache) if you write on A and read on B. It is "difficult" to catch the system in the window in between a write and the point where the read will read the correct data. With AFS the race isn't just tortoise and hare, it can actually be against a creature that is virtually sessile... It's fixed now anyway, or there is a workaround (I just logged into two systems that share my campus AFS home directory to try it out). If you do int fd; fd = open("dummy",O_WRONLY); printf("Writing out to dummy.\n"); write(fd,"This is written out.",21); fsync(fd); fprintf(stdout,"Done.\n"); sleep(30); on system A now, the fsync actually forces a kernel level write through. fflush on files opened with fopen still doesn't work. My recollection of the last time I tried this (quite a few years ago) is that neither of them would work, which I would have (and did) call a bug. So now you can force a writeback at the expense of using low level file I/O. Unless somebody knows how to get the low level file descriptor associated with a high level file pointer (if there is any such beast to get) -- I've never tried to do this. > > and we have/do/are still meditating upon using AFS > > (since we have it running) in at least some capacity on a public cluster > > we're engineering, but it isn't a knee-jerk thing to do EVEN if it is > > already there, and will definitely require a bit of consideration if you > > have to set it up from scratch to use it at all. > > > > For a shared global namespace filesystem that actually does some sort of > authentication, AFS is really the only game in town. I can't imagine doing a > campus-wide NFS or PVFS setup... No, we agree, which is why we're thinking seriously about it. We may end up doing both -- permit AFS access to a "head node" so people can connect their own account space to the cluster across the campus WAN, but use something else on the "inside" of the head node/firewall. Or something else entirely. I'd welcome suggestions -- we're having a fairly major meeting on the project in a week or two. Real Life experiences of people who use AFS across a cluster would be lovely. Especially experiences or tests/benchmarks regarding things like efficiency and system overhead/load/requirements, and any caveats or gotchas to be overcome (given by hypothesis a longstanding, well managed campus-wide AFS setup -- at least seven or eight years old, but I don't really remember when they started using AFS and it might be a fair bit longer). rgb > > -Erik > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 13 19:17:27 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sun, 14 Sep 2003 01:17:27 +0200 (CEST) Subject: data storage location In-Reply-To: Message-ID: On Sat, 13 Sep 2003, Robert G. Brown wrote: > On Sat, 13 Sep 2003, Erik Paulson wrote: > > Or something else entirely. I'd welcome suggestions -- we're having a > fairly major meeting on the project in a week or two. Real Life > experiences of people who use AFS across a cluster would be lovely. > Especially experiences or tests/benchmarks regarding things like > efficiency and system overhead/load/requirements, and any caveats or > gotchas to be overcome (given by hypothesis a longstanding, well managed AFS is heavily used at CERN. They have requirements for access to an international base of users, and have a kerberos infrastructure in place. IIRC, they still use V4 for the reasons mentioned earlier in the thread. AFS is used across the computational cluster, which is where I met it. The cluster is batch oriented - not a parallel Beowulf type. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From iosephus at sgirmn.pluri.ucm.es Sun Sep 14 05:40:36 2003 From: iosephus at sgirmn.pluri.ucm.es (=?iso-8859-1?Q?Jos=E9_M=2E_P=E9rez_S=E1nchez?=) Date: Sun, 14 Sep 2003 11:40:36 +0200 Subject: Beowulf And jvm In-Reply-To: References: Message-ID: <20030914094036.GA19325@sgirmn.pluri.ucm.es> On Thu, Sep 11, 2003 at 12:47:48PM -0400, purushotham komaravolu wrote: > Hi, > I am a newbie to beowulf. > I have a question about running Java on a Beowulf cluster. > Is it possible that I can start only one JVM on one machine and the task > be run distributed on the cluster? It is a multi-threaded application. > Like say, I have an application with 100 threads. > Can I have 50 threads run on one machine and 50 on another by launching > the application(jvm) on one machine? I dont want to use MPI or any special code. > > Has anyone tried something like this before? > Thanks > Regards, > Puru A colleague of mine who makes parallel Java programming uses the 'Colt Distribution', maybe it can help you: http://hoschek.home.cern.ch/hoschek/colt/ Regards, Jose M. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bvds at bvds.geneva.edu Sat Sep 13 17:43:46 2003 From: bvds at bvds.geneva.edu (bvds at bvds.geneva.edu) Date: Sat, 13 Sep 2003 17:43:46 -0400 (EDT) Subject: channel bonding/trunking and diskless booting Message-ID: <20030913214346.61A67E2468@bvds.geneva.edu> I looked into this and decided that channel bonding could not be done in a system with diskless booting unless there was a separate network to handle the booting. >Can anybody help me with the steps required to configure a linux cluster >node with channel bonding (or trunking) and diskless booting? Brett van de Sande _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Sun Sep 14 16:54:14 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Sun, 14 Sep 2003 22:54:14 +0200 Subject: Diskless Beowulf Message-ID: <20030914205154.M46076@ug.ms.wits.ac.za> I know this may seem as a trivial question but can anyone please give me a guideline on where to start in setting up a diskless Beowulf cluster.I need something that will assist a Linux dummy. Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel.pfenniger at obs.unige.ch Mon Sep 15 03:51:29 2003 From: daniel.pfenniger at obs.unige.ch (Daniel Pfenniger) Date: Mon, 15 Sep 2003 09:51:29 +0200 Subject: channel bonding/trunking and diskless booting In-Reply-To: <20030913214346.61A67E2468@bvds.geneva.edu> References: <20030913214346.61A67E2468@bvds.geneva.edu> Message-ID: <3F656F81.8070508@obs.unige.ch> bvds at bvds.geneva.edu wrote: > I looked into this and decided that channel bonding could > not be done in a system with diskless booting unless there was a > separate network to handle the booting. Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 (say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be used just for the booting network? What is the obstacle for such a configuration? Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Florent.Calvayrac at univ-lemans.fr Mon Sep 15 06:05:59 2003 From: Florent.Calvayrac at univ-lemans.fr (Florent Calvayrac) Date: Mon, 15 Sep 2003 12:05:59 +0200 Subject: channel bonding/trunking and diskless booting In-Reply-To: <3F656F81.8070508@obs.unige.ch> References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> Message-ID: <3F658F07.2030101@univ-lemans.fr> Daniel Pfenniger wrote: > > > bvds at bvds.geneva.edu wrote: > >> I looked into this and decided that channel bonding could >> not be done in a system with diskless booting unless there was a >> separate network to handle the booting. > > > Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 > (say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be > used just > for the booting network? What is the obstacle for such a configuration? > > Dan > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > then you should rewrite the PXE rom to support channel bonding... why not have the last machine of your cluster with a disk to use as a BOOTP server, then once all the cluster is booted boot it as a node ? Or use an old Pentium 75 laying around as a BOOTP server downloading a channel bonding enabled kernel ? -- Florent Calvayrac | Tel : 02 43 83 26 26 Laboratoire de Physique de l'Etat Condense | Fax : 02 43 83 35 18 UMR-CNRS 6087 | http://www.univ-lemans.fr/~fcalvay Universite du Maine-Faculte des Sciences | 72085 Le Mans Cedex 9 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Mon Sep 15 07:19:10 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Mon, 15 Sep 2003 07:19:10 -0400 Subject: howto calculate pi to large (eg 10^6) places Message-ID: <3F65A02E.58937413@epa.gov> I was reading Petr Beckmann's book on the history of pi, which includes a chapter on the digit hunters (people who calculated pi to large numbers of digits). He says that (repeatedly) calculating pi to large numbers of places is a good way of testing whether a machine is functioning accurately. I just looked with google for such a program but couldn't find anything that went beyond the bit width (32 or 64) of the host machine. Does anyone know where I can get the code? How do you do a calculation to 10^6 places on a 32 bit machine? Thanks Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Mon Sep 15 07:58:32 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Mon, 15 Sep 2003 07:58:32 -0400 Subject: howto calculate pi to large (eg 10^6) places In-Reply-To: <3F65A02E.58937413@epa.gov> References: <3F65A02E.58937413@epa.gov> Message-ID: <1063627112.20882.83.camel@protein.scalableinformatics.com> Hi Joe: There are many methods available. You can exploit various of the expansions which yield pi or 1/pi. You will need an extended precision library, such as David Bailey's MP, or similar. As for algorithms, you have several choices. The Bailey-Borwein-Plouffe method will give you hexadecimal digits, though it might not be as compute intensive as you wish. See http://www.cecm.sfu.ca/organics/papers/borwein/paper/html/paper.html for some nice details on some of this. Note: some of the good folks at Cray used to do this with their machines. One of the "burn-in" tests was generating a few hundred-million digits. Joe On Mon, 2003-09-15 at 07:19, Joseph Mack wrote: > I was reading Petr Beckmann's book on the history of pi, which includes a chapter > on the digit hunters (people who calculated pi to large numbers of digits). > He says that (repeatedly) calculating pi to large numbers of places is a good way > of testing whether a machine is functioning accurately. > > I just looked with google for such a program but couldn't find anything that went > beyond the bit width (32 or 64) of the host machine. Does anyone know where > I can get the code? How do you do a calculation to 10^6 places on a 32 bit > machine? > > Thanks Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Mon Sep 15 08:53:50 2003 From: gmpc at sanger.ac.uk (Guy Coates) Date: Mon, 15 Sep 2003 13:53:50 +0100 (BST) Subject: data storage location In-Reply-To: References: Message-ID: > Does the application read all or only a subset of the data? > Is the subset predictable? Externally by a scheduler? Automatic data replication tied into the schedular would be really nice; data-grid for beowulf-clusters. (Buzzword overload?) The schedular would examine the jobs waiting in the queue, and then populate the cluster with the required data. Frequently accessed datasets would get replicated to more nodes than less popular ones. Of course, the devil is always in the details. One can imagine all sorts of nasty scenarios, such as someone submitting a mass of short running jobs, which kicks off a mass of replication events, which then crushes your network. Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Mon Sep 15 11:02:57 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 15 Sep 2003 11:02:57 -0400 Subject: data storage location In-Reply-To: References: Message-ID: <1063638177.26176.26.camel@roughneck> On Sat, 2003-09-13 at 19:17, John Hearns wrote: > On Sat, 13 Sep 2003, Robert G. Brown wrote: > > > On Sat, 13 Sep 2003, Erik Paulson wrote: > > > > Or something else entirely. I'd welcome suggestions -- we're having a > > fairly major meeting on the project in a week or two. Real Life > > experiences of people who use AFS across a cluster would be lovely. > > Especially experiences or tests/benchmarks regarding things like > > efficiency and system overhead/load/requirements, and any caveats or > > gotchas to be overcome (given by hypothesis a longstanding, well managed > > AFS is heavily used at CERN. They have requirements for access to an > international base of users, and have a kerberos infrastructure in place. > IIRC, they still use V4 for the reasons mentioned earlier in the thread. > AFS is used across the computational cluster, which is where I met it. > The cluster is batch oriented - not a parallel Beowulf type. Does anyone have any suggestions on how a batch system can handle the ticket timeout problem gracefully? Nic BTW -- Does the 2GB file limit matter to anybody but me ? -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ilopez at millicom.com.ar Sun Sep 14 17:59:58 2003 From: ilopez at millicom.com.ar (Ignacio =?iso-8859-1?q?L=F3pez?=) Date: Sun, 14 Sep 2003 23:59:58 +0200 Subject: Diskless Beowulf In-Reply-To: <20030914205154.M46076@ug.ms.wits.ac.za> References: <20030914205154.M46076@ug.ms.wits.ac.za> Message-ID: <200309142359.58948.ilopez@millicom.com.ar> El Dom 14 Sep 2003 22:54, masego-otlhe_segone escribi?: > I know this may seem as a trivial question but can anyone please give me a > guideline on where to start in setting up a diskless Beowulf cluster.I need > something that will assist a Linux dummy. > > Masego-otlhe Segone Well...you can start by taking a look at Linux Terminal Server Project (www.ltsp.org)...in my opinion it's a great way of setting diskless linux clients...of course that you have to do much more to boild a beowulf, but that'll get you started ;-) > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ic02 at uow.edu.au Sun Sep 14 23:31:45 2003 From: ic02 at uow.edu.au (Iwan Maximus Cornelius) Date: Mon, 15 Sep 2003 13:31:45 +1000 Subject: pvm crashing Message-ID: Hi, I am having some problems running a Monte Carlo physics simulation on a 9 machine (all linux redhat 9.0) cluster using pvm (version 3.4.4). The application I am running is composed of a parent application and 9 identical daughter applications. The parent essential does house keeping work, starting the pvm daemon, adding machine etc, sets catchout and spawns the daughters. It then sits in a loop, listening for messages from the daughters. Being a Monte Carlo simulation this can result in 10,000,000 or so messages being passed between daughters and the parent. Well it should but the pvmd crashes mysteriously after about 3,000,000 with the following error message: libpvm [t40001]: mxfer() mxinput bad return on pvmd sock For some reason the pvmd is crashing. Could anybody suggest if something here is obvious, or if not, could suggest a way of debugging such a problem? Thanks for your time, Iwan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Mon Sep 15 11:40:05 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Mon, 15 Sep 2003 17:40:05 +0200 Subject: data storage location In-Reply-To: References: <20030912184327Q.hanzl@unknown-domain> Message-ID: <20030915174005Q.hanzl@unknown-domain> > Why do you say AFS is too cumbersome? - cannot export normal filesystem like ext2/3 (which existed before) - tickets which expire - something like five daemons on server - semantics different from UNIX filesystem - I am not sure about big partitions (say 100-200GB) - on this list it was often discouraged for beowulf Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Mon Sep 15 11:44:58 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Mon, 15 Sep 2003 17:44:58 +0200 Subject: data storage location In-Reply-To: References: Message-ID: <20030915174458A.hanzl@unknown-domain> > AFS is heavily used at CERN. They have requirements for access to an > international base of users, and have a kerberos infrastructure in place. > IIRC, they still use V4 for the reasons mentioned earlier in the thread. > AFS is used across the computational cluster, which is where I met it. > The cluster is batch oriented - not a parallel Beowulf type. Are there setup details anywhere on the www? I guess this experience is rare... Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tegner at nada.kth.se Mon Sep 15 03:02:22 2003 From: tegner at nada.kth.se (tegner at nada.kth.se) Date: Mon, 15 Sep 2003 09:02:22 +0200 (MEST) Subject: Diskless Beowulf In-Reply-To: <20030914205154.M46076@ug.ms.wits.ac.za> References: <20030914205154.M46076@ug.ms.wits.ac.za> Message-ID: <64835.150.227.16.253.1063609342.squirrel@webmail.nada.kth.se> Check http://www.linuxnetmag.com/en/issue5/m5diskless1.html > I know this may seem as a trivial question but can anyone please give me > a guideline on where to start in setting up a diskless Beowulf cluster.I > need something that will assist a Linux dummy. > > Masego-otlhe Segone > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Mon Sep 15 11:29:12 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Mon, 15 Sep 2003 17:29:12 +0200 Subject: data storage location In-Reply-To: References: <20030912184327Q.hanzl@unknown-domain> Message-ID: <20030915172912S.hanzl@unknown-domain> > A read-only data set allows many file handling options. > If files are updated as a whole, you may use a caching versioned file > system. That is specialized, but provides many opportunities for > optimization > ... > > > Cache-like behavior would save a lot of manual work but unfortunately > > I am not aware of any working solution for linux, > > Several exist. It depends on the semantics you need, as doing this > efficiently requires making assumptions about the access patterns and > desired semantics. Please what do you include in these 'several'? Am I right that none of them is opensource AND freely downloadable ? I try hard to find out what does this list in your head include, please correct my thoughts: - you certainly did not qualify InterMezzo and Lustre as working - old dead projects like ClusterNFS or amd hacks are also out of question - I don't quite believe you meant AFS or Coda either - you could consider some non-free software - you could consider something you use at Scyld, being even opensource but not freely downloadable anywhere (if I got right one of your older posts) Basically I just want confirmation for this one bit of information: "There is no reliable opensource freely downloadable filesystem (or addition to filesystem) for (contemporary) linux which would use local harddisk as a cache, even for the most relaxed semantics one can imagine (whole file caching, read-only, nearly no synchronisation)." Or better yet, tell me where it is, I'll be happy to be wrong. Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Mon Sep 15 14:12:03 2003 From: becker at scyld.com (Donald Becker) Date: Mon, 15 Sep 2003 14:12:03 -0400 (EDT) Subject: data storage location In-Reply-To: <20030915172912S.hanzl@unknown-domain> Message-ID: On Mon, 15 Sep 2003 hanzl at noel.feld.cvut.cz wrote: > > A read-only data set allows many file handling options. > > If files are updated as a whole, you may use a caching versioned file > > system. That is specialized, but provides many opportunities for > > optimization > > ... > > > > > Cache-like behavior would save a lot of manual work but unfortunately > > > I am not aware of any working solution for linux, > > > > Several exist. It depends on the semantics you need, as doing this > > efficiently requires making assumptions about the access patterns and > > desired semantics. > > Please what do you include in these 'several'? Am I right that none of > them is opensource AND freely downloadable ? You've instantly moved from a technical agenda to a political one. "Open Source" is a technical attribute, while "free" and "no cost online access" is different. The approaches I am talking about are a mix of - the Transparent File System, originally intended for caching in front of mounted CDs, but improved to sit in front of NFS. It was first implemented in Linux around 1993, but I haven't seen any updates - several academic projects, two of which I've seen at recent LinuxWorlds. There was an interesting related paper at ClusterWorld, which focused on co-servers -- using peers instead of the main file server. - and, of course, the implementation that we have. > I try hard to find out what does this list in your head include, > please correct my thoughts: The challenge here is that the kind of file server I'm talking about does not cleanly fit with Unix end-user semantics. Thus they are often hidden or not considered a file system. Specifically, the following conflict with Unix expectations Using versions while allowing multiple open versions with the same name. Prohibiting anything but whole-file updates Not providing consistent readdir() and access() results, as there is no fixed list of files. > - you certainly did not qualify InterMezzo and Lustre as working That's a political minefield. > - old dead projects like ClusterNFS or amd hacks are also out... Those are new rules. Caching file systems are very ad hoc, and thus used for only one purpose. There is not the broad user base motivation to make them general or keep them current. That doesn't mean they don't exist. And yes, some of the systems I'm referring to are related to automount daemons (AMD). I'm especially familar with those, as many are implemented with NFS servers. I wrote one of the first user-level NFS servers in the late '80s, and that NFS code became the basis for implementing several quirky, ad hoc filesystems. (For those that don't know how AMDs work, they mount a server or create a file in a local directory and return a symbolic link pointing to it.) One specifically was a FTP filesystem. When a file block was requested, the whole file was moved locally. It had the advantage over most caching systems of also being able to support readdir(). Why use the NFS / automount approach? Because it's the only user-level interface to the filesystem NFS is already quirky > Basically I just want confirmation for this one bit of information: > > "There is no reliable opensource freely downloadable filesystem (or > addition to filesystem) for (contemporary) linux which would use local > harddisk as a cache, even for the most relaxed semantics one can > imagine (whole file caching, read-only, nearly no synchronisation)." -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bvds at bvds.geneva.edu Mon Sep 15 12:11:48 2003 From: bvds at bvds.geneva.edu (bvds at bvds.geneva.edu) Date: Mon, 15 Sep 2003 12:11:48 -0400 (EDT) Subject: channel bonding/trunking and diskless booting In-Reply-To: <3F656F81.8070508@obs.unige.ch> (message from Daniel Pfenniger on Mon, 15 Sep 2003 09:51:29 +0200) References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> Message-ID: <20030915161148.87ECBE2468@bvds.geneva.edu> > >Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 >(say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be used just >for the booting network? What is the obstacle for such a configuration? > I am *really* hoping that someone would tell me how to do diskless nodes plus channel bonding. Adding an additional machine to act as DHCP server and tftp server is out (for me) because I don't don't have any free ports left on my switch. Can you really set up eth0 so that it is both bonded and unbonded? How do you do this? What is this eth0:0 thing? (I display my ignorance) Brett van de Sande _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bvds at bvds.geneva.edu Mon Sep 15 14:16:46 2003 From: bvds at bvds.geneva.edu (bvds at bvds.geneva.edu) Date: Mon, 15 Sep 2003 14:16:46 -0400 (EDT) Subject: channel bonding/trunking and diskless booting In-Reply-To: <3F656F81.8070508@obs.unige.ch> (message from Daniel Pfenniger on Mon, 15 Sep 2003 09:51:29 +0200) References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> Message-ID: <20030915181646.B6C6CE246B@bvds.geneva.edu> > >Perhaps a naive suggestion, but what about bond0 bonding eth0 and eth1 >(say with network 192.168.1.0) and eth0:0 (with network 192.168.2.0) be used just >for the booting network? What is the obstacle for such a configuration? > I just found an old post from Rob Latham asking the same question: http://www.beowulf.org/pipermail/beowulf/2001-June/000391.html He tried this and he said it didn't work: > shows what i know :>. it looks to me like the bonding tricks hold on > pretty tight to the slave interfaces and won't let me trick it into > sending trafic over just one nic when i want to. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Mon Sep 15 15:38:04 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Mon, 15 Sep 2003 21:38:04 +0200 Subject: howto calculate pi to large (eg 10^6) places References: <3F65A02E.58937413@epa.gov> <1063627112.20882.83.camel@protein.scalableinformatics.com> Message-ID: <3F66151C.8060604@moene.indiv.nluug.nl> Joseph Landman wrote: > Note: some of the good folks at Cray used to do this with their > machines. One of the "burn-in" tests was generating a few > hundred-million digits. Heh, the way we (previous employer) burnt in our new Cray was to have CWI, Amsterdam (www.cwi.nl) run one of their award-winning crypto programs "for free" before going operational ... -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From serguei.patchkovskii at sympatico.ca Mon Sep 15 15:32:55 2003 From: serguei.patchkovskii at sympatico.ca (serguei.patchkovskii at sympatico.ca) Date: Mon, 15 Sep 2003 15:32:55 -0400 Subject: channel bonding/trunking and diskless booting Message-ID: <20030915193255.KDRQ26607.tomts24-srv.bellnexxia.net@[209.226.175.20]> ----- Original Message ----- From: To: Cc: "Ashwin Bijur" Sent: Saturday, September 13, 2003 5:43 PM Subject: channel bonding/trunking and diskless booting > I looked into this and decided that channel bonding > could not be done in a system with diskless booting > unless there was a separate network to handle the > booting. Not true. I configured a 90-node cluster at the University of Calgary last fall to do exactly that. The system has two 100Mbit networks between the nodes, with only one of those attached to the front node. This "front" network is used for remote boot, then gets bonded with the the "back" network for the internode traffic. Getting this to work requires a bit of trickery with the network startup scripts. You have to create a temporary ram filesystem, copy -statically linked- ifenslave, ifconfig, route, shell, and the setup script to that filesystem, then activate channel bonding and set up the routing. The key with routing in such network is that it is perfectly possible to direct outgoing packets to systems, which are not fully attached to both networks, to the appropriate slaved interface. For a case where both networks are attached to the front node, there is an additional complication, as you do not necessarily know which interface PXE will try first. This can be solved by assigning unique subnets to both interfaces, which are -different- from the bonded subnet. Having done this, you can use set up IP routing on the front node such that it will respond through the correct interface while PXE is active. Debugging such setup is a bitch - but once it works, it works really well! Serguei _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Mon Sep 15 16:19:13 2003 From: lathama at yahoo.com (Andrew Latham) Date: Mon, 15 Sep 2003 13:19:13 -0700 (PDT) Subject: Diskless Cluster Message-ID: <20030915201913.20695.qmail@web80510.mail.yahoo.com> diskless is my preference. simply speaking you need some good hardware before you worry about software. The software is easy simply us PXE and TFTP to get a kernel to your node. Then Have that kernel load a NFS root on a file server or down load one to RAM. Start your system. ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 15 19:35:03 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 16 Sep 2003 09:35:03 +1000 Subject: data storage location In-Reply-To: <1063638177.26176.26.camel@roughneck> References: <1063638177.26176.26.camel@roughneck> Message-ID: <200309160935.05024.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 16 Sep 2003 01:02 am, Nicholas Henke wrote: > BTW -- Does the 2GB file limit matter to anybody but me ? It would if we used AFS, but we don't, we use NFS. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/ZkynO2KABBYQAh8RAhbpAKCO//vP6dn6Qos7Y+zR9WfoRMxSSACfZLBZ Rv6CONuRhmVRGbnM7MaeOGk= =6W96 -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 15 19:57:59 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 16 Sep 2003 09:57:59 +1000 Subject: channel bonding/trunking and diskless booting In-Reply-To: <20030915161148.87ECBE2468@bvds.geneva.edu> References: <20030913214346.61A67E2468@bvds.geneva.edu> <3F656F81.8070508@obs.unige.ch> <20030915161148.87ECBE2468@bvds.geneva.edu> Message-ID: <200309160958.00864.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 16 Sep 2003 02:11 am, bvds at bvds.geneva.edu wrote: > I am *really* hoping that someone would tell me how to do > diskless nodes plus channel bonding. The only idea I have would be whether doing a LinuxBIOS on the system would help - but then of course there's an even worse bootstrapping problem of installing LinuxBIOS on all your systems, not to mention small things like invalidating warranties, risk of blowing your systems up, etc.. :-) - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD4DBQE/ZlIHO2KABBYQAh8RAlDHAJdGn8u15uRzQI+yutR1Y0bTy+t1AJwJS/TC QMkuLi66IoW7dD3/JF9RDg== =7bnL -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From egan at sense.net Mon Sep 15 20:41:00 2003 From: egan at sense.net (egan at sense.net) Date: Mon, 15 Sep 2003 18:41:00 -0600 (MDT) Subject: howto calculate pi to large (eg 10^6) places In-Reply-To: <3F65A02E.58937413@epa.gov> References: <3F65A02E.58937413@epa.gov> Message-ID: <1063672860.3f665c1c22c2c@www.sense.net> Quoting Joseph Mack : > I can get the code? How do you do a calculation to 10^6 places on a 32 > bit > machine? echo "scale=1000000;a(1)*4" | bc -l _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 15 19:53:41 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 16 Sep 2003 09:53:41 +1000 Subject: Lustre Message-ID: <200309160953.42528.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Anyone tried out Lustre [1] ? What did you think ? - From what I've read it looks interesting, but I've not heard good things regarding stability. Chris [1] - http://www.lustre.org/ - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/ZlEFO2KABBYQAh8RAvySAJ9wJD62ogP/2WvADgqXI5qV+5z5mACaApZZ 8vvqRbBSeQ7b5LB0/xkqvUE= =h4et -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rouds at servihoo.com Tue Sep 16 06:06:42 2003 From: rouds at servihoo.com (RoUdY) Date: Tue, 16 Sep 2003 14:06:42 +0400 Subject: mpich2 In-Reply-To: <200309151829.h8FITmd25993@NewBlue.Scyld.com> Message-ID: Is their an expert in MPICH2-0.93??? Please mail me, I need some help in some issues Thanks Roudy -------------------------------------------------- Get your free email address from Servihoo.com! http://www.servihoo.com The Portal of Mauritius _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nican at nsc.liu.se Tue Sep 16 07:40:28 2003 From: nican at nsc.liu.se (Niclas Andersson) Date: Tue, 16 Sep 2003 13:40:28 +0200 (CEST) Subject: ANNOUNCEMENT: Workshop on Linux Clusters for Super Computing Message-ID: ============================================================== # 4th Annual Workshop on LINUX CLUSTERS FOR SUPER COMPUTING # Clusters for High Performance Computing and Grid Solutions # # October 22-24, 2003 # Hosted by # National Supercomputer Centre (NSC) # Link?ping University, Sweden ============================================================== The fast development of the processing power of high-end PCs and workstations, together with the availability of open source operating systems such as Linux have made it possible to build very cost effective clusters of commodity hardware, also known as beowulfs. With the addition of high bandwidth networks with low latency, open source based beowulfs has become the most common platform for high performance computing. Assembling a beowulf is, however, a non-trivial task due to the fast moving hardware market as well as the rapid development of Linux and available tools for clustered computing. The aim of the workshop is to gather together both people with experience of building and maintaining clusters and people using or interested in using cluster resources. The workshop will address the hardware and software issues encountered during design as well as deployed applications' efficiency, portability and utilization. We intend to cover the following topics: o Storage solutions for clusters and Grids. o Distributed computing and Grid technologies o Applications adapted for using parallel and distributed resources o Software and tools for using and managing clusters o Scalability issues, benchmarking and tuning o Job scheduling strategies o Key application areas o Update on the latest hardware developments In addition to invited talks, there will be tutorials (Oct. 22), exhibits and vendor presentations. The complete list of speakers and programme will be available at http://www.nsc.liu.se/lcsc/. If you wish to present your own cluster, Grid project, software development or give a talk on a topic related to clusters or Grid computing you are most welcome to contact Niclas Andersson E-mail: Phone: +46 13 281464 Fax: +46 13 282535 If you want to present your products in a vendor session or in our exhibit outside the auditorium, please contact Torgny Fax?n E-mail: Phone: +46 13 285798 Fax: +46 13 282535 CONFERENCE DETAILS Date October 22-24, 2003 Location The conference will take place at Collegium, Link?ping close to Link?ping University. Tutorials will be held at the University. Registration On-line registration at http://www.nsc.liu.se/lcsc Last date: 10 October. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gropp at mcs.anl.gov Tue Sep 16 08:23:37 2003 From: gropp at mcs.anl.gov (William Gropp) Date: Tue, 16 Sep 2003 07:23:37 -0500 Subject: mpich2 In-Reply-To: References: <200309151829.h8FITmd25993@NewBlue.Scyld.com> Message-ID: <5.1.1.6.2.20030916072231.02e51d88@localhost> At 02:06 PM 9/16/2003 +0400, RoUdY wrote: >Is their an expert in MPICH2-0.93??? >Please mail me, I need some help in some issues >Thanks Bug reports and questions on mpich2 should be sent to mpich2-maint at mcs.anl.gov . You may also wish to try the current version, 0.94 . Bill _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lusk at mcs.anl.gov Tue Sep 16 08:40:47 2003 From: lusk at mcs.anl.gov (Rusty Lusk) Date: Tue, 16 Sep 2003 07:40:47 -0500 Subject: mpich2 In-Reply-To: Message from "RoUdY" of "Tue, 16 Sep 2003 14:06:42 +0400." Message-ID: <200309161240.h8GCemL23134@shakey.mcs.anl.gov> | Is their an expert in MPICH2-0.93??? | Please mail me, I need some help in some issues | Thanks | Roudy The current release of MPICH2 is 0.94. Please send your problem, inlcuding whether you are using the Windows or Unix version of MPICH2, and the exact nature of the problem, to mpich2-maint at mcs.anl.gov. Regards, Rusty Lusk _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeffrey.b.layton at lmco.com Tue Sep 16 08:38:16 2003 From: jeffrey.b.layton at lmco.com (Jeff Layton) Date: Tue, 16 Sep 2003 08:38:16 -0400 Subject: Desired to Test on Opterons Message-ID: <3F670438.5050309@lmco.com> Hello, I'd like to test our codes on a small Opteron cluster, preferrably with Infiniband. We're trying to test with a vendor but we're having lots of problems with them, so we'd like to find a reliable vendor who can put together a small (4-8 node) dual Opteron cluster with at least GigE and if at all possible IB. Each node should have at least 2 Gigs of memory. The vendor needs to have the cluster up and running within 10 days of signing a Non-Disclosure Agreement (NDA) with Lockheed. The pot of gold at the end of the rainbow is that we're looking for several clusters next year. The first one we hope to order in November for a Jan. delivery. No guarantees though :) If you are interested, please contact me ASAP. Thanks! Jeff Layton Lockheed-Martin Aeronautics Company _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Tue Sep 16 09:50:20 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Tue, 16 Sep 2003 09:50:20 -0400 Subject: Lustre References: <200309160953.42528.csamuel@vpac.org> Message-ID: <3F67151C.A5D41000@epa.gov> Chris Samuel wrote: > Anyone tried out Lustre [1] ? There was a talk at OLS this year http://www.linuxsymposium.org/2003/ As usual when given by a person involved in the development, it's always presented as being easy to install and running smoothly. However they do have it running on two kilonode production clusters and have won the contract for the filesystem for some new cluster that is currently being bid on. So it's in production in some places Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Tue Sep 16 09:34:24 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Tue, 16 Sep 2003 09:34:24 -0400 Subject: howto calculate pi to large (eg 10^6) places References: <3F65A02E.58937413@epa.gov> <1063672860.3f665c1c22c2c@www.sense.net> Message-ID: <3F671160.D543756A@epa.gov> egan at sense.net wrote: > > > How do you do a calculation to 10^6 places on a 32 bit machine? > echo "scale=1000000;a(1)*4" | bc -l :-) I didn't know you could do this (still waiting for the prompt to return). Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Tue Sep 16 11:32:38 2003 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Tue, 16 Sep 2003 17:32:38 +0200 Subject: data storage location In-Reply-To: References: <20030915172912S.hanzl@unknown-domain> Message-ID: <20030916173238X.hanzl@unknown-domain> > Caching file systems are very ad hoc, and thus used for only one > purpose. There is not the broad user base motivation to make them > general or keep them current. I believe cachefs for linux would meet enough demand to be viable. Or little kernel module intercepting open(2) and friends and letting userland to decide when these calls should go stright through and when they should ask for userland action (like fetching out-of-cache file) - an autofs-style variant of old amd-hacks game. Yes, you are right, nothing simple/universal managed to catch up so far. > You've instantly moved from a technical agenda to a political one. > "Open Source" is a technical attribute, while "free" and "no cost online > access" is different. For me "Open Source & free & no cost online access" means big user base and good chance of fixing problems via community collaboration - strong technical reason (with political side-effect). (I however understand and appreciate that your work extends technical possibilities even beyond the above limits, i.e. that one can even buy good solutions :) Best Regards Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 16 12:40:31 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 16 Sep 2003 12:40:31 -0400 (EDT) Subject: Next Beowulf training class, September 24-25 in San Francisco Message-ID: Scyld is offering detailed training in the installation, configuration, administration and programming of Beowulf clusters. I will be the instructor for several of the upcoming courses including the next session, to be held in San Francisco on September 24th and 25th. If you'd like more information or want to register please visit http://www.scyld.com/training.html We typically alternate training between the east and west coast (U.S.), but other locations may be added to schedule based on demand. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Tue Sep 16 12:47:09 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Tue, 16 Sep 2003 09:47:09 -0700 (PDT) Subject: howto calculate pi to large (eg 10^6) places In-Reply-To: <3F671160.D543756A@epa.gov> Message-ID: gnu bc is a useful tool to have. it may take a while for the prompt to come back... ;) 1000 places takes around 3 seconds on a 900mhz pIII joelja On Tue, 16 Sep 2003, Joseph Mack wrote: > egan at sense.net wrote: > > > > > How do you do a calculation to 10^6 places on a 32 bit machine? > > > echo "scale=1000000;a(1)*4" | bc -l > > :-) I didn't know you could do this (still waiting for the prompt to return). > > Joe > > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Tue Sep 16 17:27:29 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Tue, 16 Sep 2003 23:27:29 +0200 Subject: Diskless Beowulf Message-ID: <20030916212730.M21242@ug.ms.wits.ac.za> Actually I alredy have a working diskless cluster, I need details on how to setup the beowulf and to configurations I need to do to get MPI working and to start running parallel programs. Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian.dobbins at yale.edu Tue Sep 16 20:16:09 2003 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Tue, 16 Sep 2003 20:16:09 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. Message-ID: Hi guys, Has anyone on this list been involved in the construction of a small 'machine room' for their clusters? We're beginning to look at creating an enclosure with an attached AC unit and, possibly, a raised floor. I've started poking around the 'net a bit, but was interested in hearing any experiences or tips from the people here. Anything about materials to use, people to talk to, hidden costs to watch out for, etc. The overall goal is to have a cool, clean, local systems 'room' for our computational facilities. As for specifics for size, that's unknown at the moment... perhaps, just for the sake of discussion, say 15' by 25' (by however tall we need, at least 8.5'). We'd like to plan for a capacity of up to 4 racks (w/ 32 nodes each), plus a few miscellaneous desksides lying around. (While we do have a central machine room on campus, our network is isolated for security reasons, and additionally we tend to run GigE between our newest machines -even some desktops- and we certainly can't get that sort of throughput from the central machine room! Thus, even though this might be costly, it may be worth it in the end.) Thanks for any pointers whatsoever! It's much appreciated! Cheers, - Brian Brian Dobbins Yale Mechanical Engineering ------------------------------------------------------------------ "Be nice to other people.. they outnumber you six billion to one." _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 16 21:57:11 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 16 Sep 2003 21:57:11 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: Message-ID: On Tue, 16 Sep 2003, Brian Dobbins wrote: > > Hi guys, > > Has anyone on this list been involved in the construction of a small > 'machine room' for their clusters? We're beginning to look at creating an > enclosure with an attached AC unit and, possibly, a raised floor. > > I've started poking around the 'net a bit, but was interested in hearing > any experiences or tips from the people here. Anything about materials to > use, people to talk to, hidden costs to watch out for, etc. The overall > goal is to have a cool, clean, local systems 'room' for our computational > facilities. There are a variety things you'll want to consider, especially: a) power; There have been some very extensive discussions on the list about power gotchas associated with driving a large number of non-power-factor corrected switching power supplies. There are vendors out there who sell server-room sized main transformers that compensate for the 180 Hz harmonic line distortion that can occur. You can also consider line conditioning and UPS at the same time, depending on your needs and budget. You'll need to locate power poles or other receptacle sets convenient to the racks. b) capacity and airflow patterns of your air conditioning compared to your power supply and expected loading; What goes in must come out, basically, with enough spare capacity to allow for future increases in rackmount power density and to keep the room COOL, not just "not hot". c) structural integrity; Loaded racks are >>heavy<< -- they can weigh 800 lbs or even more if they contain UPS batteries, on less than a square meter of floor space. Two post racks also are often under considerable torque when loaded, and need to be fastened carefully to the floor and/or equipped with cases with center mounts that balance. d) sound and light; Four full racks plus AC will sound like a small jet taxiing. All the time. Tolerable for short stretches but not a good place to work. Besides it's cold, right? Having enough light, effectively located, to be able to work with is also good if you are an Old Guy with eyes that get stressed by reading little bitty etched print upside down in the dark. e) network; Cable trays, rackspace or wall racks for network POPs and switches. WAN (organizational level) connections as well as the actual cluster LAN. f) security; Lessee, a room with 160 or so $2000 or so boxes = $320K. Locating the space in a room with a convenient door out to your dark and unattended loading dock, however convenient it is for delivery, is ill-advised. Locks, booby traps, x10 video cams, alarms, and armed guards optional. Find a comfort level here. g) comfort/convenience; We find things like a communal jacket on the back of the cluster room door, a workbench with tools (rechargable screwdriver), portable KVM, maybe a laptop for serial port access (for certain motherboards), a comfy workchair or two. Sound excluding/noise reducing stereo headphones can be nifty (and probably OSHA approved). > As for specifics for size, that's unknown at the moment... perhaps, just > for the sake of discussion, say 15' by 25' (by however tall we need, at > least 8.5'). We'd like to plan for a capacity of up to 4 racks (w/ 32 > nodes each), plus a few miscellaneous desksides lying around. If you mean a few tower-mounted servers, that's great, but see notes above about suitability as a real workspace. It is typically loud, cold, locked (or should be), and a bad place to take cups of coffee or food. So it's not a great place to locate humans for more than the times required to work on node installation and maintenance. > (While we do have a central machine room on campus, our network is > isolated for security reasons, and additionally we tend to run GigE > between our newest machines -even some desktops- and we certainly can't > get that sort of throughput from the central machine room! Thus, even > though this might be costly, it may be worth it in the end.) > > Thanks for any pointers whatsoever! It's much appreciated! Only two. Other folks will probably add some to bigger/better locations, but you can take a "walking tour in pictures" of our brahma cluster on http://www.phy.duke.edu/brahma. You can also find a book on engineering clusters and a few other documents which address infrastructure. Hope this helps. Oh, and I'd strongly suggest getting a professional architect (one with experience doing computer machine rooms) to do your plans. And avoid bozo electricians who try e.g. wiring three phase circuits with a single common neutral. rgb > > Cheers, > - Brian > > > Brian Dobbins > Yale Mechanical Engineering > ------------------------------------------------------------------ > "Be nice to other people.. they outnumber you six billion to one." > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Tue Sep 16 22:51:44 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Wed, 17 Sep 2003 04:51:44 +0200 Subject: rsh Message-ID: <20030917024414.M10506@ug.ms.wits.ac.za> I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I just realised that I cannot communicate with other hosts due to some rsh setting that I need to do. I tried running tstmachines and I got "Errors while trying to run rsh hostname -n true Unexpected response from hostname: ---> Permission denied ..." Please help with the rsh settings that I need to inorder to get the rsh to work. Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amazonia at myrealbox.com Wed Sep 17 05:01:18 2003 From: amazonia at myrealbox.com (amazonia) Date: Wed, 17 Sep 2003 03:01:18 -0600 Subject: Need Rocks cluster feedback Message-ID: <1063789278.82f649c0amazonia@myrealbox.com> Hi, We will be soon implementing a new 32 node gigabit cluster based on x86 arch. The bosses are looking at rockscluster or RH9+scripts or RH7.x+oscar. (My favourite is FAI, I will be probably part time admining the cluster) I would like to know the experience of listmembers and pros/cons of Rocks Cluster distr v3.0 for x86. I heard that rocks people advocate a quick reinstall of faulty node without any analysis. Is it true? I wouldn't prefer that. What is the best cluster mgmt tool? (Oscar etc). Which is the most popular? Regards, Matt _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rkane at scs.carleton.ca Wed Sep 17 09:32:03 2003 From: rkane at scs.carleton.ca (Robert Kane) Date: Wed, 17 Sep 2003 09:32:03 -0400 Subject: Need Rocks cluster feedback References: <1063789278.82f649c0amazonia@myrealbox.com> Message-ID: <3F686253.F7ABBF70@scs.carleton.ca> The experience I've had with Rocks has been fairly positive. For what it's worth though we're still using v2.3.2 and v3.0. The pros for Rocks are generally the ease of use. Everything (OS and tools [excluding lam-mpi]) are included in one very straightforward package. Getting a cluster up and running is fairly quick. A quick linux install on the frontend and then a couple minutes for each node to install. Assuming no problems, getting our 64 node cluster from scratch to up and running shouldn't take more than half-day. The cons. The above time estimate assumed that no problems occurred. When upgrading recently we did have some form of inexplicable glitch that caused the post-install configuration on the frontend to fail and have to be configured manually. I should point out that this error was the only time something like that has been known to happen. I should also point out that another pro to using Rocks is the helpful advice of the Rocks developers. When we had the above mentioned glitch, one of the developers graciously spent a fair amount of time making sure everything was working fine. Robert Kane _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jac67 at georgetown.edu Wed Sep 17 09:21:50 2003 From: jac67 at georgetown.edu (Jess Cannata) Date: Wed, 17 Sep 2003 09:21:50 -0400 Subject: Need Rocks cluster feedback In-Reply-To: <1063789278.82f649c0amazonia@myrealbox.com> References: <1063789278.82f649c0amazonia@myrealbox.com> Message-ID: <3F685FEE.7040107@georgetown.edu> We've been using Oscar on all of our clusters for a year now and have been satisfied with it. Our only complaint was Oscar's lack of support for RedHat 8.0/9.0, and that has since been remedied with Oscar 2.3. It is easy to install, and it has a good set of management tools. Jess Georgetown University amazonia wrote: >Hi, > >We will be soon implementing a new 32 node gigabit cluster based on x86 arch. > >The bosses are looking at rockscluster or RH9+scripts or RH7.x+oscar. (My favourite is FAI, I will be probably part time admining the cluster) > >I would like to know the experience of listmembers and pros/cons of Rocks Cluster distr v3.0 for x86. > >I heard that rocks people advocate a quick reinstall of faulty node without any analysis. Is it true? I wouldn't prefer that. > >What is the best cluster mgmt tool? (Oscar etc). Which is the most popular? > >Regards, >Matt > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From pu at ku.ac.th Wed Sep 17 10:37:48 2003 From: pu at ku.ac.th (Putchong Uthayopas) Date: Wed, 17 Sep 2003 21:37:48 +0700 Subject: Diskless Cluster References: <20030915201913.20695.qmail@web80510.mail.yahoo.com> Message-ID: <003f01c37d29$47d7ec10$0e01010a@hpcncd.cpe.ku.ac.th> Hi, That seems to be quite simple. Anyway, we found that: 1. This eat up a lot of RAM. So, it will not work well if your node does not have large memory. 2. NFS root create a lot of overhead and does not seems to scale well. Putchong ----- Original Message ----- From: "Andrew Latham" To: Cc: "beowulf" Sent: Tuesday, September 16, 2003 3:19 AM Subject: RE: Diskless Cluster > diskless is my preference. simply speaking you need some good hardware before > you worry about software. The software is easy simply us PXE and TFTP to get a > kernel to your node. Then Have that kernel load a NFS root on a file server or > down load one to RAM. Start your system. > > ===== > Andrew Latham > > Penguin loving, moralist agnostic. > > LathamA.com - (lay-th-ham-eh) > lathama at lathama.com - lathama at yahoo.com > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sean at asacomputers.com Wed Sep 17 11:28:45 2003 From: sean at asacomputers.com (Sean) Date: Wed, 17 Sep 2003 08:28:45 -0700 Subject: Need Rocks cluster feedback In-Reply-To: <1063789278.82f649c0amazonia@myrealbox.com> Message-ID: <5.1.0.14.2.20030917082343.02e5aca0@pop.asacomputers.com> Matt, We have done several Rocks cluster installation and it is very easy and straight forward. Linux comes embedded in it and this is an added advantage. C-3 is a widely accepted cluster management tool. At 03:01 AM 9/17/03 -0600, amazonia wrote: >Hi, > >We will be soon implementing a new 32 node gigabit cluster based on x86 arch. > >The bosses are looking at rockscluster or RH9+scripts or RH7.x+oscar. (My >favourite is FAI, I will be probably part time admining the cluster) > >I would like to know the experience of listmembers and pros/cons of Rocks >Cluster distr v3.0 for x86. > >I heard that rocks people advocate a quick reinstall of faulty node >without any analysis. Is it true? I wouldn't prefer that. > >What is the best cluster mgmt tool? (Oscar etc). Which is the most popular? > >Regards, >Matt > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf Thanks and Regards Sean ASA Computers Inc. 2354, Calle Del Mundo Santa Clara CA 95054 Telephone : (408) 654-2901 xtn 205 (408) 654-2900 ask for Sean (800) REAL-PCS (1-800-732-5727) Fax: (408) 654-2910 E-mail : sean at asacomputers.com URL : http://www.asacomputers.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jim at ks.uiuc.edu Wed Sep 17 11:52:51 2003 From: jim at ks.uiuc.edu (Jim Phillips) Date: Wed, 17 Sep 2003 10:52:51 -0500 (CDT) Subject: Diskless Cluster In-Reply-To: <003f01c37d29$47d7ec10$0e01010a@hpcncd.cpe.ku.ac.th> Message-ID: Another option is a bproc-based system like Scyld or Clustermatic. You don't get a full Linux install on the slaves, but if you can live with that limitation administration is pretty easy. We're currently booting our diskless slaves off of floppies, which is OK with bproc because the actual runtime kernel is loaded off of the network during the boot process. -Jim On Wed, 17 Sep 2003, Putchong Uthayopas wrote: > Hi, > > That seems to be quite simple. Anyway, we found that: > > 1. This eat up a lot of RAM. So, it will not work well if your node does not > have large memory. > 2. NFS root create a lot of overhead and does not seems to scale well. > > Putchong > ----- Original Message ----- > From: "Andrew Latham" > To: > Cc: "beowulf" > Sent: Tuesday, September 16, 2003 3:19 AM > Subject: RE: Diskless Cluster > > > > diskless is my preference. simply speaking you need some good hardware > before > > you worry about software. The software is easy simply us PXE and TFTP to > get a > > kernel to your node. Then Have that kernel load a NFS root on a file > server or > > down load one to RAM. Start your system. > > > > ===== > > Andrew Latham > > > > Penguin loving, moralist agnostic. > > > > LathamA.com - (lay-th-ham-eh) > > lathama at lathama.com - lathama at yahoo.com > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mas at ucla.edu Wed Sep 17 12:09:01 2003 From: mas at ucla.edu (Michael Stein) Date: Wed, 17 Sep 2003 09:09:01 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: ; from rgb@phy.duke.edu on Tue, Sep 16, 2003 at 09:57:11PM -0400 References: Message-ID: <20030917090901.B4711@mas1.ats.ucla.edu> Just some additions, good coverage.. - - - Access to both front and back of racks... > a) power; > There have been some very extensive discussions on the list about power > gotchas associated with driving a large number of non-power-factor > corrected switching power supplies. There are vendors out there who > sell server-room sized main transformers that compensate for the 180 Hz > harmonic line distortion that can occur. The 1U dual Xeon machines we've seen have power factor correcting power supplies so the harmonic problems might not exist. At 250 W/machine and 42 in a rack you do need an impressive amount of power, over 10 KW/rack. Watch out, future machines may need more... Watch the head/air flow. Typical rack mount machines want cool air in the front and exhaust hot air out the back. A rack full will heat a rack size volume of air all at once (something like 75 F in 99 F out all at once!). Don't run this hot air into the front of the next row of racks... Watch out -- the power usages varies depending on the CPU loading. A quick measurement of an idle machine will be very misleading: 1U Dual Xeon 2.66 Ghz 2 GB ram (& Kill-A-Watt power meter) idle: 109 W 1 CPU 174 W (1 x burnP6) 2 CPU 254 W (4 x burnP6) Make sure the electricians know the power you intend to use -- they need to size the circuits *larger* than this by some factor (from code?) -- you can't put a 20 A load on a 20 A breaker. > c) structural integrity; > > Loaded racks are >>heavy<< -- they can weigh 800 lbs or even more if > they contain UPS batteries 30 lb 1U machine * 42 in rack -> 1260 lbs or were they 40 lb machines? 40 lb 1U machine * 42 in rack -> 1680 lbs This does NOT include the weight of the rack, cables and any other stuff you manage to hang on the rack (rack rails?). > Oh, and I'd strongly suggest getting a professional architect (one with > experience doing computer machine rooms) to do your plans. And avoid > bozo electricians who try e.g. wiring three phase circuits with a single > common neutral. I'd strongly suggest that (in addition) you also run the numbers yourself: - power - heat flow (and power for it?) just to make sure (these numbers are large for more than a small single digit size cluster and get absolutely huge for 1000 nodes). Discovering a miscalculation here early is no big deal, late it's a disaster... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 17 14:06:16 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 17 Sep 2003 14:06:16 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <20030917090901.B4711@mas1.ats.ucla.edu> Message-ID: On Wed, 17 Sep 2003, Michael Stein wrote: > Just some additions, good coverage.. Some good additions to my additions, and they reminded me of still more:-) > Make sure the electricians know the power you intend to use -- they > need to size the circuits *larger* than this by some factor (from > code?) -- you can't put a 20 A load on a 20 A breaker. Also make sure that they put a distribution panel (or two) IN THE SPACE. Long runs between panel and receptacle carrying near-peak currents eat voltage, which in turn requires still more current at the lower voltages. Overwiring (using a higher gauge wire than strictly necessary for the run length) is better than marginal or underwiring. > > Oh, and I'd strongly suggest getting a professional architect (one with > > experience doing computer machine rooms) to do your plans. And avoid > > bozo electricians who try e.g. wiring three phase circuits with a single > > common neutral. > > I'd strongly suggest that (in addition) you also run the numbers yourself: > > - power > - heat flow (and power for it?) > > just to make sure (these numbers are large for more than a small > single digit size cluster and get absolutely huge for 1000 nodes). > > Discovering a miscalculation here early is no big deal, late it's > a disaster... Very good advice. I also forgot to mention two other important, even critical issues and a Useful Rule of Thumb. a) Thermal kill switch. You may well want the room to be equipped with one. This is a thermostatted switch that kills all room power if the ambient temperature exceeds some threshold, e.g. 90F. The idea is that if AC fails and you don't get down there to shut down nodes fast enough, the kill switch kills power all at once rather than permit the nodes to overheat to where they physically break (as they will if they get hot enough). Remember, at 10 KW/rack, four racks is 40 KW all being released in your itty bitty 15x25' room. The room will go from ambient cool to meltdown in a matter of minutes if AC fails, and (Murphy's law being what it is) it WILL FAIL sooner or later. b) When working out the AC, remember that in many locations, especially ones where it gets cold in the winter, physical plant people often engage in the unhealthy practice of turning off their chillers or putting them in a winter standby mode. After all, its bloody cold outside! Who needs a chiller! You do. Believe me. If the chilled water (or whatever) entering your machine room stops being (say) 42F and starts being the standby temperature of (say) 70F, the ambient air going into the front of your nodes will rapidly go up to (say) 85F or even 90F, trip your kill switch or break your nodes because it ALMOST trips and stays hot for days, and make your life a living hell as you try to convince the physical plant people to turn the air conditioning back on and they tell you that they aren't budgeted for year round operation and besides it will ice up their coils. Been there, done that. So be sure to get your physical plant people (or whoever is to run your actual air conditioning chiller/blower system) to sign a contract written in blood and promising you their firstborn child if they do a silly little thing like turn the air conditioning off the first time the outdoor temperature drops to ten below zero or something. Then be prepared to firebomb their houses when they do it anyway (just kidding, just kidding...:-). c) Don't forget to budget in the COST OF OPERATION for all of this infrastructure when you go with your hat in your hands looking for money. The remodelling will be a lot more than you think it will be by the time you get the 16 or so tons of AC you need, the huge blower, the chiller lines, and all that harmonically mitigated, line conditioned power. But THEN you get to buy the electricity and cooling. A reasonable rule of thumb estimate for electrical power is: $1 per watt per year of 24x7 operation. This includes both the power itself and the cost of the AC required to remove that power continuously. Note that it is only an estimate, and reality could be lower by a bit or higher by a factor of two, depending on local utility costs. For your presumed 40 KW cluster, though, you're looking at $40K/year just to keep the thing plugged in and running, AFTER paying off the renovation costs and buying all the hardware. I didn't address the raised floor issue here or before, by the way, but yes there are some good things about a raised floor design. For example, it is relatively easy to deliver AC and power directly to the racks from underneath. On the other hand, the raised floor has to support the racks and they are (as noted) likely to be HEAVY and to TORQUE on their floor support as well as just press down on it. Also, the hidden wiring is an advantage when it doesn't need to be worked on, then it becomes a small disadvantage relative to more open and accessible trays. I suspect it costs more too. Somebody who has tried both kinds of floor and prefers raised may be able to do better than this. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laurenceliew at yahoo.com.sg Wed Sep 17 14:31:14 2003 From: laurenceliew at yahoo.com.sg (Laurence Liew) Date: 18 Sep 2003 02:31:14 +0800 Subject: Need Rocks cluster feedback In-Reply-To: <3F686253.F7ABBF70@scs.carleton.ca> References: <1063789278.82f649c0amazonia@myrealbox.com> <3F686253.F7ABBF70@scs.carleton.ca> Message-ID: <1063823473.2962.408.camel@scalable> Hi, You can get lam-mpi for Rocks from www.x2ca.com download section... Cheers! Laurence On Wed, 2003-09-17 at 21:32, Robert Kane wrote: > The experience I've had with Rocks has been fairly positive. For what > it's worth though we're still using v2.3.2 and v3.0. > > The pros for Rocks are generally the ease of use. Everything (OS and > tools [excluding lam-mpi]) are included in one very straightforward > package. Getting a cluster up and running is fairly quick. A quick linux > install on the frontend and then a couple minutes for each node to > install. Assuming no problems, getting our 64 node cluster from scratch > to up and running shouldn't take more than half-day. > > The cons. The above time estimate assumed that no problems occurred. > When upgrading recently we did have some form of inexplicable glitch > that caused the post-install configuration on the frontend to fail and > have to be configured manually. I should point out that this error was > the only time something like that has been known to happen. I should > also point out that another pro to using Rocks is the helpful advice of > the Rocks developers. When we had the above mentioned glitch, one of the > developers graciously spent a fair amount of time making sure everything > was working fine. > > Robert Kane > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mas at ucla.edu Wed Sep 17 14:53:42 2003 From: mas at ucla.edu (Michael Stein) Date: Wed, 17 Sep 2003 11:53:42 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: ; from rgb@phy.duke.edu on Wed, Sep 17, 2003 at 02:06:16PM -0400 References: <20030917090901.B4711@mas1.ats.ucla.edu> Message-ID: <20030917115342.A5030@mas1.ats.ucla.edu> > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). > > Remember, at 10 KW/rack, four racks is 40 KW all being released in your > itty bitty 15x25' room. The room will go from ambient cool to meltdown > in a matter of minutes if AC fails, and (Murphy's law being what it is) > it WILL FAIL sooner or later. At that size/power a lot less than minutes. 1 1U machine might output about 30 CFM of 99 F air. 4 racks full of them (42 * 4) would be about 5000 CFM of 99 F air. You have 15 x 25 x 8? so about 3000 cubic feet of 75 F air to start. At 5000 CFM through the machines the whole room will be 99 F in about 36 seconds. After that it gets interesting (the machines are now taking in 99 F air)... And this assumes uniform/perfect air flow (no hot/cold spots). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 17 16:00:09 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 17 Sep 2003 16:00:09 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <20030917115342.A5030@mas1.ats.ucla.edu> Message-ID: On Wed, 17 Sep 2003, Michael Stein wrote: > > a) Thermal kill switch. You may well want the room to be equipped > > with one. This is a thermostatted switch that kills all room power if > > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > > that if AC fails and you don't get down there to shut down nodes fast > > enough, the kill switch kills power all at once rather than permit the > > nodes to overheat to where they physically break (as they will if they > > get hot enough). > > > > Remember, at 10 KW/rack, four racks is 40 KW all being released in your > > itty bitty 15x25' room. The room will go from ambient cool to meltdown > > in a matter of minutes if AC fails, and (Murphy's law being what it is) > > it WILL FAIL sooner or later. > > At that size/power a lot less than minutes. > > 1 1U machine might output about 30 CFM of 99 F air. 4 racks full of > them (42 * 4) would be about 5000 CFM of 99 F air. > > You have 15 x 25 x 8? so about 3000 cubic feet of 75 F air to start. > At 5000 CFM through the machines the whole room will be 99 F in about > 36 seconds. After that it gets interesting (the machines are now taking > in 99 F air)... > > And this assumes uniform/perfect air flow (no hot/cold spots). Ahh, you theorists. My claim of "minutes" was experimental. Alas. (Just kidding, I'm really a theorist myself;-) The point being that minutes or seconds, a thermal kill switch is very much a good idea. So are things like netbotz, other machine readable thermal monitors, or the use of lmsensors and so forth of you've got 'em. An automated kill SCRIPT that triggers before the kill SWITCH permits clean shutdown instead of instant loss of power (good even though with e.g. ext3 the latter is no longer quite so worrisome). Anyway, in our direct experience it isn't QUITE the "less than a minute" a pessimistic estimate yields, probably because there is a certain amount of thermal ballast in the room itself and the cases and the air (I think you're overestimating the perfection of the circulation and mixing process, for example), the walls and floor and ceiling do let some heat out, especially if they start good and cold (concrete has a fairly high specific heat), and if it is "just" the chiller that goes out but the blower keeps running there is a bit more "stored cold" (I know, stored absence of heat:-) in the AC coils and ductwork. We've found semi-empirically (the hard way) that it takes between five and fifteen minutes for the room space to really get over 90 on a full AC kill, admittedly starting an easy 10F colder than 75 -- barely enough time to run a distributed shutdown that heads off some of the meltdown/thermal kill process, or run down and open the door to the hall and throw a big fan in if one happens to be paying close attention. In fact, I'd also generally recommend holding the room ambient temperature well under 70, (as low as 60F or lower if you can stand it) partly to increase this window of where you can intervene less destructively than with a thermal kill, partly because computer components tend to give up roughly a year of projected lifetime for every 10F increase in ambient operating temperature. So a cluster with ambient air at 75F will have a LOT more hardware problems and a shorter lifetime than one at 65F, and one with ambient 85F air will just plain break all the time, including bitlevel errors that don't actually permanently break the hardware but cause a node to freeze up. Our cluster operated for a fairly extended period (several weeks) with ambient air in the high 70's - low 80's when they first turned off the chiller last winter before we convinced them that several hundred thousand dollars worth of equipment was at risk and that they'd PROMISED to keep the room cold all year long back in our original design meeting for the space. We got to see a whole lot of these problems firsthand -- I still have broken nodes lying around that are good only for spare parts, and everybody experienced systems crashes and lost work. We keep the incoming air pretty cold, just about as cold as we possibly can. After mixing, the air ends up cold but not unbearably cold ambient, a bit colder in front of racks (where the air vents are directed). In front it is actively uncomfortable due to a cold draft (low grade wind) you can catch cold of it. Behind a rack the air is a balmy 75F or so after mixing (not right in front of the vents but a few inches away). Warm but not hot. This is probably what extends our meltdown time out to minutes, and is also why we keep a jacket down there for general use -- in the summertime, shorts, sandles, and a hawaiian shirt outside don't translate well into cluster room garb;-) rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Wed Sep 17 17:23:09 2003 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 17 Sep 2003 14:23:09 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. References: Message-ID: <000401c37d62$02b773b0$35a8a8c0@laptop152422> > > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). Don't have the kill turn off ALL the power.. Only turn off the power to the computers, but leave the lights and some receptacles alive. More than once, I had to stumble around in a computer room that was totally dark after someone hit the Emergency OFF accidentally. Bear in mind that if you have all that fancy UPS gear, the kill switch needs to turn off the output of the UPS, not the line side. There are actually code requirements for some installations (mostly aimed at helping out firement, who don't want to hit the Emergency OFF, and have the equipment still energized). This will typically be done by things like "shunt trips".... your competent A&E person should know what's supposed to be done. You also don't want the thermal kill to shut down the power to the blowers, now, do you? There are books on computer room design around, and while not everything is applicable, a lot is. It might be worth finding one at the library or technical bookstore and browsing through it so that you can be an "educated consumer". _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Wed Sep 17 14:24:50 2003 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Wed, 17 Sep 2003 14:24:50 -0400 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: <20030917090901.B4711@mas1.ats.ucla.edu> Message-ID: <20030917182450.GA15185@central.cis.upenn.edu> Raised floor vs. overhead cable trays: raised floor requires careful routing underneath -- you don't want to keep lifting tiles and snaking things back and forth when changing cabling; also, it can get cramped in there unless you're talking 6' raised floors (ahhhh, to work for the government) overhead cable trays are a pain to route as well, but end routing is much easier to manipulate. depending on AC with raised floor means you must keep the apertures to a minimum (both size and number) else you lose cooling air flow everywhere raised floor? make sure you leave enough room between rack rows to be able to pull at least two tiles in between (back to careful routing of cables) dell's 42" racks fit nicely in 1x2 tile areas, fwiw. torque on raised floor? we use 4-post racks exclusively, no connecting to the tiles/floor, just screw the extensions down to floor level I'm not sure of the advantage of hidden wiring except if you're comparing raised floor to running cables on the floor (the latter being a really bad idea regardless of comparison). BTW we always have at least two A/C's for each room, one semi-redundant. Regards, Dan W. > I didn't address the raised floor issue here or before, by the way, but > yes there are some good things about a raised floor design. For > example, it is relatively easy to deliver AC and power directly to the > racks from underneath. On the other hand, the raised floor has to > support the racks and they are (as noted) likely to be HEAVY and to > TORQUE on their floor support as well as just press down on it. Also, > the hidden wiring is an advantage when it doesn't need to be worked on, > then it becomes a small disadvantage relative to more open and > accessible trays. I suspect it costs more too. > > Somebody who has tried both kinds of floor and prefers raised may be > able to do better than this. > > rgb -- -- Daniel Widyono http://www.cis.upenn.edu/~widyono -- Liniac Project, CIS Dept., SEAS, University of Pennsylvania -- Mail: CIS Dept, 302 Levine 3330 Walnut St Philadelphia, PA 19104 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Wed Sep 17 16:28:09 2003 From: lathama at yahoo.com (Andrew Latham) Date: Wed, 17 Sep 2003 13:28:09 -0700 (PDT) Subject: Building a small machine room? Message-ID: <20030917202809.1339.qmail@web80501.mail.yahoo.com> A book and Ideas. first get the National Electric Code. it applies to everything and is what the fire code is biased off of. second contact a Halon installation company (chemical fire stopper - sprinklers are bad) look at Hoffman enclosures and B-line ladders (possible to use these to get away from dedicated room and stay cool) ask the physics department for help on heat dispersal there are much more efficient methods putting up expanded metal under or between sheet-rock can reduce RF. If a new structure. try buried hoses for a geothermal effect. ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shuntim.luk at polyu.edu.hk Thu Sep 18 00:18:54 2003 From: shuntim.luk at polyu.edu.hk (LUK ShunTim) Date: Thu, 18 Sep 2003 12:18:54 +0800 Subject: Searchable beowulf list archive Message-ID: <3F69322E.9060504@polyu.edu.hk> Hello, There used to be a searchable archive for this list at http://www.supercomputer.org/Search/ but when I tried to use it just now, it is no longer there. Is there other alternative location? Regards, ST -- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From fluffie at gmx.net Thu Sep 18 07:13:45 2003 From: fluffie at gmx.net (Fluffie) Date: Thu, 18 Sep 2003 13:13:45 +0200 (MEST) Subject: PVM: problems with 'add host' Message-ID: <21108.1063883625@www28.gmx.net> Hi to all, I'm running PVM 3.4.3 on a Linux 6.4 machine and on a Windows 2000 machine. On the Linux machine I tried to add the Windows machine like this: pvm> add XYZ.XYZ.XYZ.XYZ add XYZ.XYZ.XYZ.XYZ 0 successful HOST DTID XYZ.XYZ.XYZ.XYZ Can't start pvmd Auto-Diagnosing Failed Hosts... XYZ.XYZ.XYZ.XYZ... Verifying Local Path to "rsh"... Rsh found in /usr/bin/rsh - O.K. Testing Rsh/Rhosts Access to Host "XYZ.XYZ.XYZ.XYZ"... Rsh/Rhosts Access is O.K. Checking O.S. Type (Unix test) on Host "XYZ.XYZ.XYZ.XYZ"... Checking O.S. Type (Win test) on Host "XYZ.XYZ.XYZ.XYZ"... Host XYZ.XYZ.XYZ.XYZ is Windows-based. Checking %PVM_ROOT% on Host "XYZ.XYZ.XYZ.XYZ"... %PVM_ROOT% on XYZ.XYZ.XYZ.XYZ Appears O.K. ("C:\Programme\PVM3.4 ") Verifying Location of PVM Daemon Script on Host "XYZ.XYZ.XYZ.XYZ"... PVM Daemon Script Found ("pvmd.bat") Determining PVM Architecture on Host "XYZ.XYZ.XYZ.XYZ"... %PVM_ARCH% on XYZ.XYZ.XYZ.XYZ set to WIN32 Verifying Existence of PVM Daemon Executable on Host "XYZ.XYZ.XYZ.XYZ"... PVM Daemon Executable Found ("pvmd3.exe") Determining PVM Temporary Directory on Host "XYZ.XYZ.XYZ.XYZ"... %PVM_TMP% on XYZ.XYZ.XYZ.XYZ set to c:\tmp Checking for Leftover PVM Daemon Files on Host "XYZ.XYZ.XYZ.XYZ"... No PVM Daemon Files Found. Host XYZ.XYZ.XYZ.XYZ Appears to Be Correctly Configured. Please check your local /tmp/pvml.501 log file for error messages, or else email "pvm at msr.epm.ornl.gov" for further assistance. The "pvmd3.exe" process has been started on the win2K machine, but the Linux machine does not recognize it. On the contrary, the auto diagnose starts after a certain time as seen above. The content of the pvml.501 log file is: [t80040000] 09/03 11:14:05 (127.0.0.2:1028) LINUX 3.4.3 [t80040000] 09/03 11:14:05 ready Wed Sep 3 11:14:05 2003 [t80000000] 09/03 11:14:21 stderr at XYZ.XYZ.XYZ.XYZ: The command "$PVM_ROOT" [t80000000] 09/03 11:14:21 stderr at XYZ.XYZ.XYZ.XYZ: could not be found. [t80000000] 09/03 11:14:21 stdout at XYZ.XYZ.XYZ.XYZ: EOF [t80000000] 09/03 11:15:21 pl_startup() giving up on host XYZ.XYZ.XYZ.XYZ after 60 secs [t80040000] 09/03 11:15:21 startack() host XYZ.XYZ.XYZ.XYZ expected version, got "PvmCantStart" What else can be wrong??? Stefan -- +++ GMX - die erste Adresse f?r Mail, Message, More! +++ Getestet von Stiftung Warentest: GMX FreeMail (GUT), GMX ProMail (GUT) (Heft 9/03 - 23 e-mail-Tarife: 6 gut, 12 befriedigend, 5 ausreichend) Jetzt selbst kostenlos testen: http://www.gmx.net _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmarch at prodigy.net Thu Sep 18 02:57:22 2003 From: jmarch at prodigy.net (Jim March) Date: Wed, 17 Sep 2003 23:57:22 -0700 Subject: Need the help of a Beowulf rig for a political problem... References: <20030917090901.B4711@mas1.ats.ucla.edu> <20030917115342.A5030@mas1.ats.ucla.edu> Message-ID: <00b801c37db2$1e98ffe0$dcc74fd1@me> Folks, I realize this will probably be about the weirdest thing to hit this mailing list. Short form: we have a real-life encyption problem with serious implications on our hands. We need to break a ZIP password that we suspect has up to 15 characters in the password, and only one file. More or less a worst-case scenario. There is a ZIP password breaker application at http://zipcracker.bengburken.net/ - with versions compatible with BeoWulf. The password-protected ZIP file in question is at: http://www.equalccw.com/ATL-TSRepair.zip We *think* it was zipped and encrypted with WinRAR: http://www.rarlab.com/download.htm Right, so what's going on here? This file was one of 40,000 pulled off of an open FTP site mis-managed by incompetents at Diebold Elections Systems. DES sells the computers and software to run elections; among their customers are the entire state of Georgia, at least 12 counties in California and scads of others. The software is proprietary, the source code is held as tightly as MS holds their stuff, and in the case of their new "touchscreen" product, there's no paper trail at all. All of the security measures are electronic. Only one company, a testing lab chosen by the Federal Elections Commission ("Metamor", now known as "Ciber Inc") got access to the source code; state and county-level elections officials and Secretaries of State were supposed to approve voting products while able to see how compiled and running systems work, but with no access to the source code themselves. The president of Diebold corporation is a Republican party activist and fundraiser who has been quoted as saying he's "going to deliver Ohio to George Bush", while at the same time submitting bids on voting systems in that state. http://www.portclintonnewsherald.com/news/stories/20030827/localnews/140871.html Any alarms going off yet? It gets worse. WAY worse. http://www.scoop.co.nz/mason/stories/HL0307/S00065.htm - this is the results of testing of the actual Diebold compiled Windows code. This sucker is rigged for vote fraud six ways from Sunday. Guys, they used MS-Access as a database engine, THEN deliberately crippled the security in at least two different ways. It's utter madness. Wanna test it for yourself? Here's your kit - download the executables and live voting data which Diebold conveniently left up on their site back in January: http://www.equalccw.com/dieboldtestnotes.html Since we (a pretty "diverse" bunch of activists) started showing people how the Deibold code works, Diebold insiders have been leaking internal memos and manuals. Y'all literally won't believe some of this stuff - they KNEW exactly what they were doing the whole time: http://workersrighttovote.org/bbv/diebold-memos-1.htm - we have a HUGE archive of Diebold Elections division EMail traffic - this file is a "best of" selection of that. Horrifying stuff, including deception of the Federally-approved testing lab ("Metamor", now called "Ciber, Inc"). http://www.scoop.co.nz/mason/stories/HL0309/S00150.htm - New Zealand mirror to the above... http://www.equalccw.com/smokinggun.pdf - if you want to see what the most damning of these internal EMails looked like as they originally appeared on Diebold's internal system... http://www.scoop.co.nz/mason/stories/HL0309/S00157.htm - Full copy plus the comical highlights of a Diebold internal manual on how to run elections-day procedures, written for the hapless Diebold (Canadian) tech staff. This material was NEVER intended for the public, and in places is drop-dead funny. http://www.equalccw.com/ElectionSupportGuide.pdf - same internal manual in it's original, unmodified form complete with corporate logos, fancy formatting and such. If you're going to show a copy to reporters/politicians/etc, use this one as it's clearly genuine. OK, so what's with the ATL-TSRepair.zip file? It's a Microsoft Access database (.MDB internally) dated just four days AFTER the November of 2002 general elections - elections in which a huge number of Republican "surprise victories" broke out. We don't know what the hell is on it, but it's fishy as hell - "ATL" probably refers to Atlanta. Why would they even have a small voting data file created after the election, unless it was to hide something nasty? Another problem: the filename is a lie. "TS" means "TouchScreen", the voting terminals. But those systems don't USE .MDB files - data is transmitted to the central box running GEMS ("Global Elections Management Software") right after the polls close, but not in Access format. So no possible "TouchScreen Repair File" could involve MS-Access data. Prior to the elections, a huge number of poorly-tested patches for the touchscreen terminals (running Windows CE!) were being passed around - it appears this file was designed to mimic those, but with ZIP encryption, you can still load the file enough to see the full filename and original date inside the compressed ZIP file. We want to know what's in there. If you have a beefy BeoWulf cluster PVM 3.4.2 or greater, and this project is of interest, drop me a line. VIA PGP :) unless you like getting Diebold cease'n'desist orders :). My real name is Jim March, email is jmarch at prodigy.net and my public key is registered at the default PGP keyserver at ldap://certserver.pgp.com - if you see two entries, use the SECOND (or later-date one, they're a week apart, only the 2nd one works). Sorry for the intrusion and this'll be my last posting to your list. But y'all gotta admit, it's one hell of a cool real-world encryption problem :). I'm BCCing a small number of activists involved in this, including Bev Harris, the lady who did the original documentation on the matter and is basically the leader (as much as we have one...). Jim March _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 18 08:50:37 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 18 Sep 2003 08:50:37 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <000401c37d62$02b773b0$35a8a8c0@laptop152422> Message-ID: On Wed, 17 Sep 2003, Jim Lux wrote: > You also don't want the thermal kill to shut down the power to the blowers, > now, do you? You don't but ours does, I'm pretty sure. I'm not certain what the precise motivation was -- I believe it was the issue of those firemen, not knowing what circuits in the room were on and what were off. > There are books on computer room design around, and while not everything is > applicable, a lot is. It might be worth finding one at the library or > technical bookstore and browsing through it so that you can be an "educated > consumer". Do you have any specific ones to recommend? Amazon is around the metaphorical corner, and it seems like this would be a useful addition to my bookshelf. Although at this point I've learned a lot on my own the very hardest of ways;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 18 09:25:56 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 18 Sep 2003 15:25:56 +0200 (CEST) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: Message-ID: On Wed, 17 Sep 2003, Robert G. Brown wrote: > > This is probably what extends our meltdown time out to minutes, and is > also why we keep a jacket down there for general use -- in the > summertime, shorts, sandles, and a hawaiian shirt outside don't > translate well into cluster room garb;-) Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. What would it consist of then? As you say a fleece for warmth. Lots of pockets for tools and screws. Ideas? Then again, we would have to get systems engineers to model it on the catwalk... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Thu Sep 18 09:49:59 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 18 Sep 2003 08:49:59 -0500 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: Message-ID: <1063892998.1106.86.camel@caffeine> On Thu, 2003-09-18 at 08:25, John Hearns wrote: > On Wed, 17 Sep 2003, Robert G. Brown wrote: > > > > This is probably what extends our meltdown time out to minutes, and is > > also why we keep a jacket down there for general use -- in the > > summertime, shorts, sandles, and a hawaiian shirt outside don't > > translate well into cluster room garb;-) > > Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. > What would it consist of then? > As you say a fleece for warmth. Lots of pockets for tools and screws. > Ideas? > > Then again, we would have to get systems engineers to model it on the > catwalk... > What next, a computer room edition of "Trading Spaces"? -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at lathama.com Thu Sep 18 09:03:10 2003 From: lathama at lathama.com (Andrew Latham) Date: Thu, 18 Sep 2003 06:03:10 -0700 (PDT) Subject: Building a small machine room? In-Reply-To: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <20030918130310.99878.qmail@web80507.mail.yahoo.com> Halon is used in situations where water is a threat. If a sprinkle head starts dumping water on a system pluged into a UPS it will get ugly. Halon is not banned from telcom rooms so just sell it to the fire marshal as a telcom room. I was under the impression that the room would have no people permanently stationed in it. Halon does not kill people. It replaces Oxygen. It can injure or kill a person after large and lengthy exposure but not likely. --- Daniel Kidger wrote: > > > > second contact a Halon installation company > > (chemical fire stopper - sprinklers are bad) > > Unfortunately in many places, halon is now banned from new machine room > installations - upon the mistaken belief that human lives are worth more > than the computers. > > > Daniel. > > -------------------------------------------------------------- > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > ----------------------- www.quadrics.com -------------------- > ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Thu Sep 18 09:59:35 2003 From: shewa at inel.gov (Andrew Shewmaker) Date: Thu, 18 Sep 2003 07:59:35 -0600 Subject: Searchable beowulf list archive In-Reply-To: <3F69322E.9060504@polyu.edu.hk> References: <3F69322E.9060504@polyu.edu.hk> Message-ID: <3F69BA47.4050509@inel.gov> LUK ShunTim wrote: > Hello, > > There used to be a searchable archive for this list at > http://www.supercomputer.org/Search/ > but when I tried to use it just now, it is no longer there. > > Is there other alternative location? The Mailing list ARChives at AIMS are nice. http://marc.theaimsgroup.com/?l=beowulf&r=1&w=2v -Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 18 10:43:05 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 18 Sep 2003 10:43:05 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <1063892998.1106.86.camel@caffeine> Message-ID: On 18 Sep 2003, Dean Johnson wrote: > On Thu, 2003-09-18 at 08:25, John Hearns wrote: > > On Wed, 17 Sep 2003, Robert G. Brown wrote: > > > > > > This is probably what extends our meltdown time out to minutes, and is > > > also why we keep a jacket down there for general use -- in the > > > summertime, shorts, sandles, and a hawaiian shirt outside don't > > > translate well into cluster room garb;-) > > > > Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. > > What would it consist of then? > > As you say a fleece for warmth. Lots of pockets for tools and screws. > > Ideas? > > > > Then again, we would have to get systems engineers to model it on the > > catwalk... > > > > What next, a computer room edition of "Trading Spaces"? Well, we have the makings of a great plot in the form of helping to crack an encrypted file that could contain evidence of political crimes that would make watergate pale in comparison. I doubt that it needs a cluster -- anybody that would leave incriminating information around (encrypted or not) in a publically accessible place is bound to be stupid enough to choose a crypt key like "elephant_man" or "republicans_rule". Or the name of their collie. crack is overkill. Fifteen minutes of hunt and peck and they'll have it. The only question is -- comedy or tragedy? I wanna be played by John Malkovich. rgb > > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Thu Sep 18 11:13:00 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Thu, 18 Sep 2003 08:13:00 -0700 (PDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <1063892998.1106.86.camel@caffeine> Message-ID: On 18 Sep 2003, Dean Johnson wrote: > On Thu, 2003-09-18 at 08:25, John Hearns wrote: > > On Wed, 17 Sep 2003, Robert G. Brown wrote: > > > > > > This is probably what extends our meltdown time out to minutes, and is > > > also why we keep a jacket down there for general use -- in the > > > summertime, shorts, sandles, and a hawaiian shirt outside don't > > > translate well into cluster room garb;-) > > > > Now THERE'S a good idea. 'Cluster room couture' by Bob Brown. > > What would it consist of then? > > As you say a fleece for warmth. Lots of pockets for tools and screws. > > Ideas? > > > > Then again, we would have to get systems engineers to model it on the > > catwalk... > > > > What next, a computer room edition of "Trading Spaces"? can't get much for $1000 > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sth at ceh.ac.uk Thu Sep 18 11:57:11 2003 From: sth at ceh.ac.uk (Steve Hindmarsh) Date: Thu, 18 Sep 2003 16:57:11 +0100 Subject: Building a small machine room? Message-ID: Halon replacements are now available here in the UK at least - we have recently replaced our server room halon system with FM200 which is supposed to be as effective but safer for humans (and the environment). I don't think you are allowed to install new Halon systems in the UK now... Steve >>> Andrew Latham 18/09/2003 14:03:10 >>> Halon is used in situations where water is a threat. If a sprinkle head starts dumping water on a system pluged into a UPS it will get ugly. Halon is not banned from telcom rooms so just sell it to the fire marshal as a telcom room. I was under the impression that the room would have no people permanently stationed in it. Halon does not kill people. It replaces Oxygen. It can injure or kill a person after large and lengthy exposure but not likely. --- Daniel Kidger wrote: > > > > second contact a Halon installation company > > (chemical fire stopper - sprinklers are bad) > > Unfortunately in many places, halon is now banned from new machine room > installations - upon the mistaken belief that human lives are worth more > than the computers. > > > Daniel. > > -------------------------------------------------------------- > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > ----------------------- www.quadrics.com -------------------- > ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Sep 18 11:20:01 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Thu, 18 Sep 2003 11:20:01 -0400 Subject: more NCBI BLAST testing on opterons Message-ID: <1063898401.4635.179.camel@protein.scalableinformatics.com> Folks: We have been doing more testing on Opterons. This time a 246 unit. For the particular NCBI BLAST job we are using (blastx of 74 sequences vs nr from July, will happily supply both the database and the test scripts to anyone in exchange for their results and machine details), the dual Opteron is showing an execution time with 2 threads giving the best results, and the Xeon 3 GHz system showing best results with 4 threads (odd, HT must be working). What we are seeing is that comparing the best efforts of these two platforms, using the binary suitable with gcc on both platforms, that the 246 completes the run in 586 seconds, while the Xeon completes it in 1006 seconds. These are wall clock seconds, as measured by the start and end times of the script. Note: We are trying to work with the Intel compiler as well to see if it helps. We are using sse2 on gcc, and m64 mode on the Opteron. We are also seeing some interesting affinity bits. It looks like we need a way to pin memories and CPUs together for scheduling for best performance. Has anyone successfully built the NCBI toolkit using the ICC? If so, would you be willing to share your linux.ncbi.mk file? Thanks! Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Thu Sep 18 11:47:38 2003 From: jcownie at etnus.com (James Cownie) Date: Thu, 18 Sep 2003 16:47:38 +0100 Subject: Building a small machine room? In-Reply-To: Message from Andrew Latham of "Thu, 18 Sep 2003 06:03:10 PDT." <20030918130310.99878.qmail@web80507.mail.yahoo.com> Message-ID: <1A010M-7bE-00@etnus.com> > Halon is not banned from telcom rooms so just sell it to the fire marshal as a > telcom room. Maybe not in the US, but some of us are in Europe (at least Dan and I)... e.g. http://www.groundworkwales.org.uk/business/oct2002.htm#Halon Fire suppression using Halon will be banned under EU Regulation 2037/2000 from 31st December 2002. From 31st December 2003, all existing systems and extinguishers must be decommissioned, except for those industries that are designated as critical users e.g. aircraft compartments, the Channel Tunnel, and military vessels. The full regulation is here http://europa.eu.int/smartapi/cgi/sga_doc?smartapi!celexapi!prod!CELEXnumdoc&lg=EN&numdoc=32000R2037&model=guichett -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Thu Sep 18 11:14:20 2003 From: lathama at yahoo.com (Andrew Latham) Date: Thu, 18 Sep 2003 08:14:20 -0700 (PDT) Subject: Q: Building a small machine room? Message-ID: <20030918151420.59284.qmail@web80504.mail.yahoo.com> Keep in mind that the below mentioned Disconnects are required by the National Electric Code. The NEC has an ever growing section on computers (low voltage telecommunication devices & rooms) I friend who taught me the NEC is also a writer for it. He being an old electrician himself stated that the code is for saving lives and should be read as such. Yes he has witnessed larger buildings burn because of faulty wiring of only low voltage systems (it was a small office telephone company that messed up). LathamA Subject: Re: Q: Building a small machine room? Materials/costs/etc. Date: Wed, 17 Sep 2003 14:23:09 -0700 > > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). Don't have the kill turn off ALL the power.. Only turn off the power to the computers, but leave the lights and some receptacles alive. More than once, I had to stumble around in a computer room that was totally dark after someone hit the Emergency OFF accidentally. Bear in mind that if you have all that fancy UPS gear, the kill switch needs to turn off the output of the UPS, not the line side. There are actually code requirements for some installations (mostly aimed at helping out firement, who don't want to hit the Emergency OFF, and have the equipment still energized). This will typically be done by things like "shunt trips".... your competent A&E person should know what's supposed to be done. You also don't want the thermal kill to shut down the power to the blowers, now, do you? There are books on computer room design around, and while not everything is applicable, a lot is. It might be worth finding one at the library or technical bookstore and browsing through it so that you can be an "educated consumer". ===== Andrew Latham Penguin loving, moralist agnostic. LathamA.com - (lay-th-ham-eh) lathama at lathama.com - lathama at yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 18 12:18:37 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 18 Sep 2003 09:18:37 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: <000401c37d62$02b773b0$35a8a8c0@laptop152422> Message-ID: <5.2.0.9.2.20030918091503.02f397b8@mailhost4.jpl.nasa.gov> At 08:50 AM 9/18/2003 -0400, Robert G. Brown wrote: >On Wed, 17 Sep 2003, Jim Lux wrote: > > > You also don't want the thermal kill to shut down the power to the blowers, > > now, do you? > >You don't but ours does, I'm pretty sure. I'm not certain what the >precise motivation was -- I believe it was the issue of those firemen, >not knowing what circuits in the room were on and what were off. > I suppose this is why you want someone who has done this sort of design before... There's no real electrical reason why you can't have a thermal kill separate from the Emergency OFF kill. All a matter of relays, subpanels, etc. There IS a reason to shut down blowers when overtemp hits.. Most fire alarm systems (in big installations) are tied to the forced air ventilation system to prevent a fire from being pushed by the HVAC. This was generally motivated by the experience in the MGM fire disaster a few decades back (which is also responsible for things like "plenum rated" cabling, etc.) So, what you probably want is a sort of staged setup.. moderate overtemp shuts down computer power, bigger overtemp (i.e. a fire) shuts down the blowers, pressing the big red button next to the door shuts down all the power. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From johnb at quadrics.com Thu Sep 18 09:39:22 2003 From: johnb at quadrics.com (John Brookes) Date: Thu, 18 Sep 2003 14:39:22 +0100 Subject: Q: Building a small machine room? Materials/costs/etc. Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA7E5E2E7@stegosaurus.bristol.quadrics.com> Just a couple of quick points on raised versus, erm, not-raised floors. A raised floor is far and away the better idea for a nice, stable system, IMHO. It's easier to dress off your cables nicely and neatly if you've got a nice bit of under-floor space to store any excess. If the machine room will house only clusters that form single rows, then you'd be wasting cash getting a raised floor, to an extent. If you're going to have multiple, interconnected rows, though, then the cables'd have to go overhead without that space. Overheads are at least as difficult to sort out as under-floor (YMMV - I'm a bit of a shorta**e :) So you're left with one major difference: PRICE. A raised floor to support that kind of weight is not inexpensive, I would imagine. Just my tuppence. John Brookes Quadrics > -----Original Message----- > From: Robert G. Brown [mailto:rgb at phy.duke.edu] > Sent: 17 September 2003 19:06 > To: Michael Stein > Cc: Brian Dobbins; beowulf at beowulf.org > Subject: Re: Q: Building a small machine room? Materials/costs/etc. > > > On Wed, 17 Sep 2003, Michael Stein wrote: > > > Just some additions, good coverage.. > > Some good additions to my additions, and they reminded me of still > more:-) > > > Make sure the electricians know the power you intend to use -- they > > need to size the circuits *larger* than this by some factor (from > > code?) -- you can't put a 20 A load on a 20 A breaker. > > Also make sure that they put a distribution panel (or two) IN > THE SPACE. > Long runs between panel and receptacle carrying near-peak currents eat > voltage, which in turn requires still more current at the lower > voltages. Overwiring (using a higher gauge wire than > strictly necessary > for the run length) is better than marginal or underwiring. > > > > Oh, and I'd strongly suggest getting a professional > architect (one with > > > experience doing computer machine rooms) to do your > plans. And avoid > > > bozo electricians who try e.g. wiring three phase > circuits with a single > > > common neutral. > > > > I'd strongly suggest that (in addition) you also run the > numbers yourself: > > > > - power > > - heat flow (and power for it?) > > > > just to make sure (these numbers are large for more than a small > > single digit size cluster and get absolutely huge for 1000 nodes). > > > > Discovering a miscalculation here early is no big deal, late it's > > a disaster... > > Very good advice. > > I also forgot to mention two other important, even critical > issues and a > Useful Rule of Thumb. > > a) Thermal kill switch. You may well want the room to be equipped > with one. This is a thermostatted switch that kills all room power if > the ambient temperature exceeds some threshold, e.g. 90F. The idea is > that if AC fails and you don't get down there to shut down nodes fast > enough, the kill switch kills power all at once rather than permit the > nodes to overheat to where they physically break (as they will if they > get hot enough). > > Remember, at 10 KW/rack, four racks is 40 KW all being > released in your > itty bitty 15x25' room. The room will go from ambient cool > to meltdown > in a matter of minutes if AC fails, and (Murphy's law being > what it is) > it WILL FAIL sooner or later. > > b) When working out the AC, remember that in many locations, > especially ones where it gets cold in the winter, physical > plant people > often engage in the unhealthy practice of turning off their > chillers or > putting them in a winter standby mode. After all, its bloody cold > outside! Who needs a chiller! > > You do. Believe me. If the chilled water (or whatever) entering your > machine room stops being (say) 42F and starts being the standby > temperature of (say) 70F, the ambient air going into the front of your > nodes will rapidly go up to (say) 85F or even 90F, trip your kill > switch or break your nodes because it ALMOST trips and stays hot for > days, and make your life a living hell as you try to convince the > physical plant people to turn the air conditioning back on > and they tell > you that they aren't budgeted for year round operation and besides it > will ice up their coils. Been there, done that. > > So be sure to get your physical plant people (or whoever is > to run your > actual air conditioning chiller/blower system) to sign a contract > written in blood and promising you their firstborn child if they do a > silly little thing like turn the air conditioning off the > first time the > outdoor temperature drops to ten below zero or something. Then be > prepared to firebomb their houses when they do it anyway > (just kidding, > just kidding...:-). > > c) Don't forget to budget in the COST OF OPERATION for all of this > infrastructure when you go with your hat in your hands looking for > money. The remodelling will be a lot more than you think it > will be by > the time you get the 16 or so tons of AC you need, the huge > blower, the > chiller lines, and all that harmonically mitigated, line conditioned > power. But THEN you get to buy the electricity and cooling. A > reasonable rule of thumb estimate for electrical power is: > > $1 per watt per year of 24x7 operation. > > This includes both the power itself and the cost of the AC required to > remove that power continuously. Note that it is only an estimate, and > reality could be lower by a bit or higher by a factor of two, > depending > on local utility costs. For your presumed 40 KW cluster, > though, you're > looking at $40K/year just to keep the thing plugged in and running, > AFTER paying off the renovation costs and buying all the hardware. > > I didn't address the raised floor issue here or before, by > the way, but > yes there are some good things about a raised floor design. For > example, it is relatively easy to deliver AC and power directly to the > racks from underneath. On the other hand, the raised floor has to > support the racks and they are (as noted) likely to be HEAVY and to > TORQUE on their floor support as well as just press down on it. Also, > the hidden wiring is an advantage when it doesn't need to be > worked on, > then it becomes a small disadvantage relative to more open and > accessible trays. I suspect it costs more too. > > Somebody who has tried both kinds of floor and prefers raised may be > able to do better than this. > > rgb > > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Thu Sep 18 12:08:31 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Thu, 18 Sep 2003 09:08:31 -0700 (PDT) Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> Message-ID: well actually halon is banned most places for new installations because it's a cfc... http://www.epa.gov/ozone/defns.html#halon It is possible to be grandfathered or get an exclusion but there's really no reason any more. In some installations osha rules requires the availablity of scba gear (self contained breathing aparatus) in the facility. our telco switchroom is setup in such a fashion... There are a number of acceptable replacemnts for total flood dry chemical fire suppression systems... which would include thing like intergen ig-541, hcfc-22, ammonium phosphate evirogel, halotron, or even good old CO2 On Thu, 18 Sep 2003, Andrew Latham wrote: > Halon is used in situations where water is a threat. If a sprinkle head starts > dumping water on a system pluged into a UPS it will get ugly. > > Halon is not banned from telcom rooms so just sell it to the fire marshal as a > telcom room. I was under the impression that the room would have no people > permanently stationed in it. > > Halon does not kill people. It replaces Oxygen. It can injure or kill a person > after large and lengthy exposure but not likely. > > --- Daniel Kidger wrote: > > > > > > > second contact a Halon installation company > > > (chemical fire stopper - sprinklers are bad) > > > > Unfortunately in many places, halon is now banned from new machine room > > installations - upon the mistaken belief that human lives are worth more > > than the computers. > > > > > > Daniel. > > > > -------------------------------------------------------------- > > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > > ----------------------- www.quadrics.com -------------------- > > > > ===== > Andrew Latham > > Penguin loving, moralist agnostic. > > LathamA.com - (lay-th-ham-eh) > lathama at lathama.com - lathama at yahoo.com > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 18 12:22:36 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 18 Sep 2003 09:22:36 -0700 Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> References: < <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <5.2.0.9.2.20030918091902.018acd08@mailhost4.jpl.nasa.gov> At 06:03 AM 9/18/2003 -0700, Andrew Latham wrote: >Halon is used in situations where water is a threat. If a sprinkle head >starts >dumping water on a system pluged into a UPS it will get ugly. Precisely why they require the emergency off to kill the UPS load side power! >Halon is not banned from telcom rooms so just sell it to the fire marshal as a >telcom room. I was under the impression that the room would have no people >permanently stationed in it. > >Halon does not kill people. It replaces Oxygen. It can injure or kill a person >after large and lengthy exposure but not likely. Halon is now incredibly expensive and an "ozone depleter", so that first "proof of performance" test after installation will set you back a pretty penny. Back in 1974, I saw a demo at the National Computer Conference from the mfrs of Halon systems where the sales guy stood in a big phone booth like tube, poured solvent all over his clothes, lit it on fire, and had the Halon discharge and put the fire out, all while he was talking through the presentation. One of the selling points was that it was much safer than CO2, which, til then, was the popular "non-water" fire extinguishing agent. >--- Daniel Kidger wrote: > > > > > > > second contact a Halon installation company > > > (chemical fire stopper - sprinklers are bad) > > > > Unfortunately in many places, halon is now banned from new machine room > > installations - upon the mistaken belief that human lives are worth more > > than the computers. > > > > > > Daniel. > > James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Thu Sep 18 08:18:23 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Thu, 18 Sep 2003 13:18:23 +0100 Subject: Building a small machine room? Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> > second contact a Halon installation company > (chemical fire stopper - sprinklers are bad) Unfortunately in many places, halon is now banned from new machine room installations - upon the mistaken belief that human lives are worth more than the computers. Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Thu Sep 18 12:32:43 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 18 Sep 2003 11:32:43 -0500 Subject: Building a small machine room? In-Reply-To: References: Message-ID: <1063902762.1120.96.camel@caffeine> I have an alternative (albeit silly) solution, but mind you it doesn't make sense in quite a few cases. Take the money you were going to spend on clever (and expensive) fire suppression systems in the bank and use the ordinary building sprinkler systems. In the highly unlikely event that you have a serious fire (you have an offsite data backup, right?), you take the money you banked and buy much better gear for cheaper. For department level COTS clusters, this might be the most cost effective method. -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Wally-Edmondson at utc.edu Thu Sep 18 12:47:22 2003 From: Wally-Edmondson at utc.edu (Wally Edmondson) Date: Thu, 18 Sep 2003 12:47:22 -0400 Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> References: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <5.1.1.6.0.20030918115942.00b4b280@imap.utc.edu> An alternative to Halon is FM-200 gas. It is safer for humans and won't harm any equipment. The worst that it can do is give you frostbite if you are standing directly under one of the nozzles. Of course, there are flashing lights and beeping noises before it dumps, giving everyone a chance to evacuate (or cancel the dump with a switch near the door). It is installed in my server room. We have never had the misfortune to use it, but it lets me sleep peacefully at night. Here is a link comparing it to Halon: http://www.fireturnkey.com/sys_halonalt.htm. At 06:03 AM 9/18/2003 -0700, Andrew Latham wrote: >Halon is used in situations where water is a threat. If a sprinkle head >starts >dumping water on a system pluged into a UPS it will get ugly. > >Halon is not banned from telcom rooms so just sell it to the fire marshal as a >telcom room. I was under the impression that the room would have no people >permanently stationed in it. > >Halon does not kill people. It replaces Oxygen. It can injure or kill a person >after large and lengthy exposure but not likely. > >--- Daniel Kidger wrote: > > > > > > > second contact a Halon installation company > > > (chemical fire stopper - sprinklers are bad) > > > > Unfortunately in many places, halon is now banned from new machine room > > installations - upon the mistaken belief that human lives are worth more > > than the computers. > > > > > > Daniel. > > > > -------------------------------------------------------------- > > Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com > > One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 > > ----------------------- www.quadrics.com -------------------- > > > >===== >Andrew Latham > >Penguin loving, moralist agnostic. > >LathamA.com - (lay-th-ham-eh) >lathama at lathama.com - lathama at yahoo.com >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 18 12:25:21 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 18 Sep 2003 12:25:21 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <5.2.0.9.2.20030918091503.02f397b8@mailhost4.jpl.nasa.gov> Message-ID: On Thu, 18 Sep 2003, Jim Lux wrote: > So, what you probably want is a sort of staged setup.. moderate overtemp > shuts down computer power, bigger overtemp (i.e. a fire) shuts down the > blowers, pressing the big red button next to the door shuts down all the power. We have some of these (including the big red switch by the door) but not all. Maybe soon -- the first is really a matter of sensors and scripts. rgb > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Thu Sep 18 09:24:54 2003 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Thu, 18 Sep 2003 09:24:54 -0400 Subject: Building a small machine room? In-Reply-To: <20030918130310.99878.qmail@web80507.mail.yahoo.com> References: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> <20030918130310.99878.qmail@web80507.mail.yahoo.com> Message-ID: <20030918132454.GA10234@central.cis.upenn.edu> > Halon does not kill people. It replaces Oxygen. It can injure or kill a > person after large and lengthy exposure but not likely. http://www.umich.edu/~oseh/guidhalo.pdf OSEH Health Guideline for Halogenated Fire Extinguishing Systems Dan W. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From frank at joerdens.de Thu Sep 18 12:28:31 2003 From: frank at joerdens.de (Frank Joerdens) Date: Thu, 18 Sep 2003 18:28:31 +0200 Subject: Beowulfery under CYGWIN Message-ID: <20030918182831.B26638@superfly.archi-me-des.de> I've got a wee little 10-node Athlon 1800+ cluster here in the office which is sitting idle a lot of the time. It's designated purpose is Cinema4D rendering, which only runs under Windoze, unfortunately (I *really* tried to get it to work under WINE, it's no-go: some arcane bug relating to wsock32, it seems, where I'm totally out of depth as to fixing it, and the WINE people weren't interested enough in the problem), so I can't run Linux on it (dual-boot I looked into also; it's impractical in our scenario here). Hence my thinking, seeing that I'd like to learn some Beowulfery, that it might be possible to do it under CYGWIN. Has anyone tried yet? Would it be worth having a go at it, or is it clear from the start that you'd have to hack the sources quite extensively? Regards, Frank _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Thu Sep 18 14:43:30 2003 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Thu, 18 Sep 2003 13:43:30 -0500 Subject: Beowulfery under CYGWIN In-Reply-To: <20030918182831.B26638@superfly.archi-me-des.de>; from frank@joerdens.de on Thu, Sep 18, 2003 at 06:28:31PM +0200 References: <20030918182831.B26638@superfly.archi-me-des.de> Message-ID: <20030918134330.F26572@mikee.ath.cx> On Thu, 18 Sep 2003, Frank Joerdens wrote: > I've got a wee little 10-node Athlon 1800+ cluster here in the office > which is sitting idle a lot of the time. It's designated purpose is > Cinema4D rendering, which only runs under Windoze, unfortunately (I > *really* tried to get it to work under WINE, it's no-go: some arcane bug > relating to wsock32, it seems, where I'm totally out of depth as to > fixing it, and the WINE people weren't interested enough in the > problem), so I can't run Linux on it (dual-boot I looked into also; it's > impractical in our scenario here). > > Hence my thinking, seeing that I'd like to learn some Beowulfery, that > it might be possible to do it under CYGWIN. > > Has anyone tried yet? Would it be worth having a go at it, or is it > clear from the start that you'd have to hack the sources quite > extensively? I don't know about the full architecture, but I do know that sshd runs just fine as a service under cygwin. Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Thu Sep 18 15:29:51 2003 From: deadline at plogic.com (Douglas Eadline) Date: Thu, 18 Sep 2003 15:29:51 -0400 (EDT) Subject: Informal Survey In-Reply-To: Message-ID: Back in July, I asked people about how they find information about clusters (see below). I finally got around to collecting the answers. You may view them on http://www.cluster-rant.com/ as part of the "New Poll" story. While you are there, take the cluster size poll. I think the results will be interesting. Thanks, Doug On Wed, 9 Jul 2003, Douglas Eadline wrote: > > I am curious where everyone gets information on clusters. > Obviously this list is one source, but what about other > sources. In addition, what kind of information do people most > want/need about clusters. Please comment on the following > questions if you have the time. You can respond to me directly > and I will summarize the results for the list. > > 1. Where do you find "howto" information on clusters > (besides this list) > > a) Google > b) Vendor > c) Trade Show > d) News Sites (what news sites are there?) > e) Other > > 2. If there were a subscription print/web magazine on clusters, what kind > of coverage would you want? > > a) howto information > b) new products > c) case studies > d) benchmarks > e) other > > > Thanks, > > Doug > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Thu Sep 18 16:13:01 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Thu, 18 Sep 2003 15:13:01 -0500 (CDT) Subject: Informal Survey In-Reply-To: Message-ID: On Thu, 18 Sep 2003, Douglas Eadline wrote: > Back in July, I asked people about how they find information about > clusters (see below). I finally got around to collecting the answers. You > may view them on http://www.cluster-rant.com/ as part of the "New Poll" > story. > > While you are there, take the cluster size poll. I think the results > will be interesting. > > Thanks, > > Doug It's too bad the size of the results was so small. Lets try to guess who the 5 people were that knew everything..:) -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at plogic.com Thu Sep 18 16:25:58 2003 From: deadline at plogic.com (Douglas Eadline) Date: Thu, 18 Sep 2003 16:25:58 -0400 (EDT) Subject: Informal Survey In-Reply-To: Message-ID: On Thu, 18 Sep 2003, Rocky McGaugh wrote: > On Thu, 18 Sep 2003, Douglas Eadline wrote: > > > Back in July, I asked people about how they find information about > > clusters (see below). I finally got around to collecting the answers. You > > may view them on http://www.cluster-rant.com/ as part of the "New Poll" > > story. > > > > While you are there, take the cluster size poll. I think the results > > will be interesting. > > > > Thanks, > > > > Doug > > > It's too bad the size of the results was so small. > > Lets try to guess who the 5 people were that knew everything..:) Or the same person who voted 5 times. I would be great to get more input. The readership of cluster rant is growing, so maybe the polls with start to have some slight significance. They should be taken with a grain of salt however, I like to think of them as ball park view of what is going on. Doug > > -- ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.814.2800 130 Webster Street | PARALLEL | Fax:+610.814.5844 Bethlehem, PA 18015 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Thu Sep 18 16:57:24 2003 From: lars at meshtechnologies.com (Lars Henriksen) Date: 18 Sep 2003 22:57:24 +0200 Subject: Beowulfery under CYGWIN In-Reply-To: <20030918182831.B26638@superfly.archi-me-des.de> References: <20030918182831.B26638@superfly.archi-me-des.de> Message-ID: <1063918644.2273.11.camel@fermi> On Thu, 2003-09-18 at 18:28, Frank Joerdens wrote: > I've got a wee little 10-node Athlon 1800+ cluster here in the office > which is sitting idle a lot of the time. It's designated purpose is > Cinema4D rendering, which only runs under Windoze, unfortunately (I > *really* tried to get it to work under WINE, it's no-go: some arcane bug > relating to wsock32, it seems, where I'm totally out of depth as to > fixing it, and the WINE people weren't interested enough in the > problem), so I can't run Linux on it (dual-boot I looked into also; it's > impractical in our scenario here). > > Hence my thinking, seeing that I'd like to learn some Beowulfery, that > it might be possible to do it under CYGWIN. You might wanna consider doing it the other way round. Running Linux on the nodes, and on top of that VMWare with the appropriate Windows and your application. Gather the virtual windows desktops in a single VNC desktop under linux eases the pain of running multiple windows machines :-) I've done this with some success in a 12 node setup. Best regards Lars -- Lars Henriksen | MESH-Technologies ApS Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rburns at cs.ucsb.edu Thu Sep 18 19:40:21 2003 From: rburns at cs.ucsb.edu (Ryan Burns) Date: Thu, 18 Sep 2003 16:40:21 -0700 Subject: Cluster distributions? Message-ID: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Hello, I've been running a very small 4 node beowulf cluster for a project. Right now I'm running Debian on the cluster and I'm not very happy with it or the tools available through the distribution. Does anyone have any suggestions of a good distribution? What about MSC.Linux? any good? I am also looking for tools to automate software installations across all the nodes, etc. I like the Debian package system, I just wish there was an easy way to have the process replicate across all the computers. Thanks, Ryan Burns _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Thu Sep 18 20:13:43 2003 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Thu, 18 Sep 2003 17:13:43 -0700 (PDT) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: hi ya ryan fun stuff.... On Thu, 18 Sep 2003, Ryan Burns wrote: > Hello, > > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? other cluster distro and installers http://Www.Linux-Consulting.com/Cluster/ > I am also looking for tools to automate software installations across all > the nodes, etc. installing -- cdrom, network install, rsync updating -- create your-local*.rpm and each node downloads new "required" updates that has been tested on other "test clusters" - or rsync or *.tgz or ?? - and a package/scripting status files to say which machined applied which patches to itself > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. there is .. lots of easy ways ... - "easy" is all relative to the user's ideas ?? personally, i don't like debians installer or *.deb packages or any other *.rpm packages but use the same equivalent "my-apt-get.pl *.tgz" and i am happy as a clam c ya alvin _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Sep 18 19:58:21 2003 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 18 Sep 2003 16:58:21 -0700 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: References: <5.2.0.9.2.20030918091503.02f397b8@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20030918164810.018b2108@mailhost4.jpl.nasa.gov> At 12:25 PM 9/18/2003 -0400, Robert G. Brown wrote: >On Thu, 18 Sep 2003, Jim Lux wrote: > > > So, what you probably want is a sort of staged setup.. moderate overtemp > > shuts down computer power, bigger overtemp (i.e. a fire) shuts down the > > blowers, pressing the big red button next to the door shuts down all > the power. > >We have some of these (including the big red switch by the door) but not >all. Maybe soon -- the first is really a matter of sensors and scripts. > > rgb Personally, I prefer not relying on a computer (that's doing anything else) to shut down the computer for a serious overtemp. I'd rather see a sensor and a hardwired connection to some sort of relay. OK, I might go for a dedicated PLC (programmable logic controller), but I'd want it to be rock solid and robust in a variety of environments, something that no PC is ever going to be (PCs are just too complex with too many logic gates internally). Sometimes simple is good. The sensors and scripts would be good for the graceful shutdown as you get close to the "yellow limit". Anyone who is enamored of the total software control of things that can cause physical damage should remember one word: "THERAC-25"... Nancy Leveson at MIT has a site at http://sunnyday.mit.edu/ where there is much about software safety. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Luc.Vereecken at chem.kuleuven.ac.be Fri Sep 19 06:57:18 2003 From: Luc.Vereecken at chem.kuleuven.ac.be (Luc Vereecken) Date: Fri, 19 Sep 2003 12:57:18 +0200 Subject: Building a small machine room? In-Reply-To: <5.2.0.9.2.20030918091902.018acd08@mailhost4.jpl.nasa.gov> References: <20030918130310.99878.qmail@web80507.mail.yahoo.com> < <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: <3.0.6.32.20030919125718.00930560@arrhenius.chem.kuleuven.ac.be> >At 06:03 AM 9/18/2003 -0700, Andrew Latham wrote: >>Halon is not banned from telcom rooms so just sell it to the fire marshal as a >>telcom room. I was under the impression that the room would have no people >>permanently stationed in it. >> >>Halon does not kill people. It replaces Oxygen. It can injure or kill a person >>after large and lengthy exposure but not likely. > Actually, Halon does not replace oxygen. CO2 extinguishers work by replacing oxygen, depriving the flames of their oxidant. Halon molecules (which contain halogen atoms, primarily Bromine) decompose in the flame, and the Bromine atoms catalyse the recombination of radicals in the flame, causing the flame to die from a lack of active radicals. This is why you need clouds of CO2, but only a small percentage of halons (< 0.5% ? ) in the air to stop a fire. The O2-concentration is hardly affected by this small amount of halon. At 09:22 AM 9/18/03 -0700, Jim Lux wrote: >Halon is now incredibly expensive and an "ozone depleter", . Because the bromine atoms, released after decomposition in the stratosphere by hard radiation, are efficient radical quenchers, they efficiently start to quench our ozone (O3 radicals). Bummer. Which is why you should never use Halon, even if it would have been legal to do so, and certainly not use any loopholes in the law to use it anyway. Putting the entire world population at risk because it is convenient for yourself is not acceptable. A number of alternatives have already been mentioned in other posts. :-) Luc Vereecken (who uses his cluster to work on combustion chemistry (-: ) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 19 10:34:43 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 19 Sep 2003 16:34:43 +0200 (CEST) Subject: Building a small machine room? In-Reply-To: <010C86D15E4D1247B9A5DD312B7F5AA78DE13F@stegosaurus.bristol.quadrics.com> Message-ID: On Thu, 18 Sep 2003, Daniel Kidger wrote: > > Unfortunately in many places, halon is now banned from new machine room > installations - upon the mistaken belief that human lives are worth more > than the computers. That's sad. Where will BOFH be without his Halon hold-off button to torture unsuspecting service engineers and IT managers? http://www.theregister.co.uk/content/30/30779.html "No, I mean, lift a couple of floor tiles, tie some cat-5 cable at ankle height between racks, unscrew the Halon Hold-off knobs, remove the doorknobs from the inside of the computer room doors and hide the oxygen masks." _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Fri Sep 19 10:44:20 2003 From: john.hearns at clustervision.com (John Hearns) Date: Fri, 19 Sep 2003 16:44:20 +0200 (CEST) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: On Thu, 18 Sep 2003, Ryan Burns wrote: > Hello, > > I am also looking for tools to automate software installations across all > the nodes, etc. If you like apt-get, check out apt-get for RPM at www.freshrpms.net And Yum at http://linux.duke.edu/projects/yum/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Sep 19 12:12:54 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 19 Sep 2003 12:12:54 -0400 (EDT) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: On Thu, 18 Sep 2003, Ryan Burns wrote: > Hello, > > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? > I am also looking for tools to automate software installations across all > the nodes, etc. > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. I think that your best choices are likely to be one of the following: Scyld ("free" version of commercial open source, probably, for such a small cluster) or Clustermatic (fully GPL). Red Hat with kickstart and yum (kickstart is precisely an easy way to install boilerplate scripted systems, although yes you have to learn to use it). If your systems have PXE-capable NICs you can get it to where you can (re)install a system over the network by just turning it on, or turning it on and selecting kickstart at a boot prompt. Yum then keeps the systems up to date, provided that you mirror your base repository from a site that mirrors a site that mirrors Red Hat that keeps the repository up to date with regard to at least security fixes and major bugfixes. Oscar, Rocks or other cluster toolsets (as discussed on the list just last week, check list archives) have tools to facilitate this kind of thing as well, often on top of one or more of the major distributions. Don't be too hasty to drop Debian -- you may just not be using it right. Check out the Debian FAI (Fully Automated Installation) site. All the links you might need for this stuff (in addition to being readily available via google) are collected on: http://www.phy.duke.edu/brahma/Resources/links.php along with a lot of other stuff, of course. Some of which you might also find interesting, but which isn't directly related to cluster installation methodologies. rgb > > > Thanks, > Ryan Burns > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rkane at scs.carleton.ca Fri Sep 19 11:48:38 2003 From: rkane at scs.carleton.ca (Robert Kane) Date: Fri, 19 Sep 2003 11:48:38 -0400 (EDT) Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: If no one has already mentioned it, Rocks (www.rocksclusters.org) is a nice distribution to use. It's a customized distribution of Redhat 7.x. It's stable, runs on a widely used and supported version of Linux, and has all sorts of cluster tools (monitors, schedulers, mpi) integrated. It's generally quick to install, and easy to manage. Robert Kane > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? > I am also looking for tools to automate software installations across all > the nodes, etc. > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Sep 19 12:29:50 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 19 Sep 2003 12:29:50 -0400 (EDT) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <5.2.0.9.2.20030918164810.018b2108@mailhost4.jpl.nasa.gov> Message-ID: On Thu, 18 Sep 2003, Jim Lux wrote: > At 12:25 PM 9/18/2003 -0400, Robert G. Brown wrote: > >On Thu, 18 Sep 2003, Jim Lux wrote: > > > > > So, what you probably want is a sort of staged setup.. moderate overtemp > > > shuts down computer power, bigger overtemp (i.e. a fire) shuts down the > > > blowers, pressing the big red button next to the door shuts down all > > the power. > > > >We have some of these (including the big red switch by the door) but not > >all. Maybe soon -- the first is really a matter of sensors and scripts. > > > > rgb > > Personally, I prefer not relying on a computer (that's doing anything else) > to shut down the computer for a serious overtemp. I'd rather see a sensor > and a hardwired connection to some sort of relay. OK, I might go for a > dedicated PLC (programmable logic controller), but I'd want it to be rock > solid and robust in a variety of environments, something that no PC is ever > going to be (PCs are just too complex with too many logic gates > internally). Sometimes simple is good. I had in mind something like one or more "server" class hosts with sensors connected that they poll every minute or five minutes or so (depending on how small your space is and how paranoid you are). This just gives you access to the room temperature at the sensor locations inside e.g. scripts. At that point, you can do anything you like -- mail out a warning at threshold A, initiate automated powerdowns by banks or all together at temperaure B (saving servers for last so all writebacks can occur) -- whatever. netbotz are expensive, but they provide such an interface (and more) completely canned and SNMP or web-accessible and even have internal mailers to do the mailing at an alarm threshold. And expensive is cheap for people who don't want to DIY and do want to protect a few hundred thousand dollars of hardware. Beyond that it isn't horribly easy to find attachable (e.g.) RS232 readable thermal sensors, but it can be done. Some are linked to brahma/Resources/vendors.php): the ibutton, sensorsoft, pico tech. There are likely more if one is good at googling. One of these days I'm going to get this set up for the cluster/server room. It actually already has lots of built in monitoring and automated emergency action, but it is all at the University level and the cluster control is all local, so I think it will be good to have the intermediate layer under our own immediate automagic control. > The sensors and scripts would be good for the graceful shutdown as you get > close to the "yellow limit". Precisely. > Anyone who is enamored of the total software control of things that can > cause physical damage should remember one word: "THERAC-25"... This is what I'm concerned about the other way, actually. We're already plugged into a University level IT sensor structure beyond our immediate control AND that provides us with no immediate information at the systems level in the room itself. By the time even their 24 hour service responds to a thermal problem, it is likely to be WAY too late because whether the meltdown (or rather, thermal kill) time is less than a minute or less then fifteen minutes, it is >>short<< once AC is really interrupted. We just got grazed by a potentially interrupting event (and yes, damn it, I've been Franned again and my house has no electricity and probably won't for several days, if the past is any guide). I'd really prefer the yellow limit scripts, sorted out so that nodes shut down first and maybe even only. If the room is large enough, it can sometimes manage to handle the heat output of only a few servers via thermal ballast and losses through walls and floor and so forth, especially if one can provide even an ordinary rotating fan to move the air. rgb > > Nancy Leveson at MIT has a site at http://sunnyday.mit.edu/ where there is > much about software safety. > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jsims at csiopen.com Fri Sep 19 13:24:33 2003 From: jsims at csiopen.com (Joey Sims) Date: Fri, 19 Sep 2003 13:24:33 -0400 Subject: Gigabit Switch Recommendation Message-ID: <812B16724C38EE45A802B03DD01FD5472605E7@exchange.concen.com> http://www.dninetworks.com/prod_lan_feature.asp?id=43 12port - GbE Jumbo Frames Support ---------------------------------------------------- |Joey P. Sims 800.995.4274 x 242 |Sales Manager 770.442.5896 - Fax |HPC/Storage Division jsims at csiopen.com |Concentric Systems,Inc. www.csilabs.net ---------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From julien.leduc at imag.fr Fri Sep 19 15:05:21 2003 From: julien.leduc at imag.fr (Julien Leduc) Date: Fri, 19 Sep 2003 19:05:21 +0000 Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> References: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: <20030919190521.718f8407.julien.leduc@imag.fr> Hi, > I've been running a very small 4 node beowulf cluster for a project. Right > now I'm running Debian on the cluster and I'm not very happy with it or > the tools available through the distribution. > > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? You should have a try with CLIC distribution or mandrake clustering, it automates deployment of the nodes and setup everything (NIS, dhcp, pxe ...) you need to run the cluster. Deployment is very fast (I am working on it, intensively use it), and reliable (for example, it takes me 3 minutes to clone 15 nodes, including the reboot). > I am also looking for tools to automate software installations across all > the nodes, etc. There is also urpmi parallel, that allow you to install packages over the cluster (you can even install a single machine with compiled sources and deploy it on the cluster). > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. If you want, you can only use the ka-tools, with a "correctly" configured environement (ie ssh to any node from a server without pass authentification, relying on keys for ex), you can execute your aptget on all your nodes (the problem will be the bandwidth throughput here, but I don't think it is an issue for a 4 nodes cluster...). You can try to use tools to spread your .deb packages efficiently over the network (mput for example, from the ka-tools, so that you won't have to compile anything but these tools to optimize your already existing installation). These tools are developped in my lab, and are very performant (and scalable) and also opensource. So you should have a closer look at http://ka-tools.sourceforge.net/ the are in deepth explanations about the method used. Regards, Julien Leduc Developper & Cluster admin Informatique et distribution Web: http://www-id.imag.fr _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Fri Sep 19 14:39:08 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Fri, 19 Sep 2003 20:39:08 +0200 Subject: Q: Building a small machine room? Materials/costs/etc. References: Message-ID: <3F6B4D4C.8070506@moene.indiv.nluug.nl> Robert G. Brown wrote: > We have some of these (including the big red switch by the door) but not > all. Maybe soon -- the first is really a matter of sensors and scripts. Do you also have the "door-opening-switch" right next to the BRS ? That can lead to very funny mistakes (especially if you're not the one making the mistake). "I'll see you out !". Wham ! SSSSsssssssshhhhhhhh :-) -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From glen at cert.ucr.edu Fri Sep 19 15:58:58 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Fri, 19 Sep 2003 12:58:58 -0700 Subject: opteron problems Message-ID: <3F6B6002.2020606@cert.ucr.edu> Hi, We've purchased a few Opteron's to play with, and they're turning out to not be very stable. The motherboard in the systems is the Tyan Thunder K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 beta on the other two. Each machine has the latest version of the bios from Tyan's website. I'm running the setiathome client, and a few of our own applications to stress test the systems. All the machines seem to either lock up, crash with a kernel panic, or reboot out of the blue. The lock ups, crashes, and reboots seem to happen at random about once a day, to once every 3 days. So, I was wondering if anyone else has had similar issues, and if you figured out any way to overcome them? Thanks for your time, Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian at chpc.utah.edu Fri Sep 19 16:43:07 2003 From: brian at chpc.utah.edu (Brian D. Haymore) Date: Fri, 19 Sep 2003 14:43:07 -0600 (MDT) Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> Message-ID: I have a couple dozen opterons in house on the arima board and have had no such issues as you are reporting. We did have a tyan in house for a couple days but did not give it a thorough test to see what your are seeing. We have been running Suse 8.1 and 8.2 beta on them as well as the beta of RedHat WS (Taroon) for x86-64 on those that we have. On Fri, 19 Sep 2003, Glen Kaukola wrote: > Hi, > > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios > from Tyan's website. I'm running the setiathome client, and a few of > our own applications to stress test the systems. All the machines seem > to either lock up, crash with a kernel panic, or reboot out of the > blue. The lock ups, crashes, and reboots seem to happen at random about > once a day, to once every 3 days. > > So, I was wondering if anyone else has had similar issues, and if you > figured out any way to overcome them? > > Thanks for your time, > Glen > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112-0190 Email: brian at chpc.utah.edu - Phone: (801) 585-1755 - Fax: (801) 585-5366 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Fri Sep 19 16:39:09 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Fri, 19 Sep 2003 13:39:09 -0700 (PDT) Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> Message-ID: On Fri, 19 Sep 2003, Glen Kaukola wrote: > Hi, > > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios > from Tyan's website. I'm running the setiathome client, and a few of > our own applications to stress test the systems. All the machines seem > to either lock up, crash with a kernel panic, or reboot out of the > blue. The lock ups, crashes, and reboots seem to happen at random about > once a day, to once every 3 days. > > So, I was wondering if anyone else has had similar issues, and if you > figured out any way to overcome them? we have 4 rioworks hdama boards with two 240s ea and two 1GB pc 2110 dimms interleaved in the primary cpu's bank 0 and 2. They've been pretty stable so far (about 6weeks). > Thanks for your time, > Glen > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 19 18:51:21 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 19 Sep 2003 15:51:21 -0700 Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> References: <3F6B6002.2020606@cert.ucr.edu> Message-ID: <20030919225121.GA5148@greglaptop.internal.keyresearch.com> On Fri, Sep 19, 2003 at 12:58:58PM -0700, Glen Kaukola wrote: > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. You didn't comment on what kernel you're running. 2.4.22 has some patches for the ia32 mode on the opteron, and these are probably the same patches already in 2.6. We have found apps that reliably crash our Opterons without this patch, but moving to 2.6 was much more stable. We haven't tried 2.4.22. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From glen at cert.ucr.edu Fri Sep 19 21:46:18 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Fri, 19 Sep 2003 18:46:18 -0700 Subject: opteron problems In-Reply-To: <20030919225121.GA5148@greglaptop.internal.keyresearch.com> References: <3F6B6002.2020606@cert.ucr.edu> <20030919225121.GA5148@greglaptop.internal.keyresearch.com> Message-ID: <3F6BB16A.4000409@cert.ucr.edu> Greg Lindahl wrote: >You didn't comment on what kernel you're running. 2.4.22 has some >patches for the ia32 mode on the opteron, and these are probably the >same patches already in 2.6. We have found apps that reliably crash >our Opterons without this patch, but moving to 2.6 was much more >stable. We haven't tried 2.4.22. > > On the redhat systems, I'm just going with the lastest kernel rpm package, kernel-2.4.20-20.9. And on Suse, it's k_smp-2.4.19-237. I hate to not use rpm, but I guess I'll try 2.4.22 and see what that does for me. Thanks for the advice. Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 20 05:39:05 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 20 Sep 2003 11:39:05 +0200 (CEST) Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: Message-ID: On Fri, 19 Sep 2003, Robert G. Brown wrote: > netbotz are expensive, but they provide such an interface (and more) > completely canned and SNMP or web-accessible and even have internal > mailers to do the mailing at an alarm threshold. And expensive is cheap > for people who don't want to DIY and do want to protect a few hundred You could add these to your links page: http://web6.scan.co.uk/Products/Info.asp?WPID=42160 Maybe a good alternative for Europeans. Haven't ever used them - I just saw them at RAL last week. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 20 05:31:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 20 Sep 2003 11:31:11 +0200 (CEST) Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> Message-ID: On Fri, 19 Sep 2003, Glen Kaukola wrote: > Hi, > > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios We're running SuSE 8.2 beta on a cluster of dual Opterons. The motherboards are Rioworks HDAMA. We're not seeing the problems you describe - they generall seem stable. Recently upgraded to the latest BIOS from the Rioworks site. The main problem we are seeing is occasinally nodes halting with the message: hda: dma_timer_expiry: dma status == 0x21 I'm sure this is not a ahrdware problem with the disks. Have run kernels 2.4.22 and the SuSE patched 2.4.21. Anyone else seen this? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Sep 20 05:59:36 2003 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 20 Sep 2003 11:59:36 +0200 (CEST) Subject: opteron problems In-Reply-To: <3F6BB16A.4000409@cert.ucr.edu> Message-ID: On Fri, 19 Sep 2003, Glen Kaukola wrote: > On the redhat systems, I'm just going with the lastest kernel rpm > package, kernel-2.4.20-20.9. And on Suse, it's k_smp-2.4.19-237. I > hate to not use rpm, but I guess I'll try 2.4.22 and see what that does > for me. You can build a new kernel as an RPM. In the kernel source directory, do a 'make rpm'. (Or something like that - you get the idea). Also if you are trying 2.4.22, apply the ACPI hotfix patch from ftp://ftp.x86-64.org/pub/linux-x86_64/v2.4 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Sat Sep 20 13:10:12 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Sat, 20 Sep 2003 13:10:12 -0400 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> Message-ID: <3F6C89F4.303@bellsouth.net> Wow! It looks like they're only getting 12 nodes per rack. That's only 24 CPUs per rack (or for a 42U rack it's like 1.75U per CPU). That's like a 4U to hold a dual CPU node (welcome to the good-old days!). Yes, I know this is the only way to get dual G5's right now, but wow (hey look around this website. There's a seminar in the UK on 23 Sept. to discuss the G5 Xserve). So, for 1,100 G5 nodes, it will take at least 92 racks! I hope they have the floor space, power, and AC ready. Cool fans on the top of the racks though. Hey! If you poke around the website, you'll find some notes from the VT presentation. Here are some comments about the facilities: * 3 MW power, double redundant with backups - UPS and diesel o 1.5 MW reserved for the TCF * 2+ million BTUs of cooling capacity using Liebert's extremedensity cooling (rack mounted cooling via liquid refrigerant) o traditional methods [fans] would have produced windspeeds of 60+ MPH * Racks were custom designed for this system Yep, inexpensive, _custom_ racks (as much an oxymoron as military intelligence as I've heard). If they charge $1000 a rack (pretty cheap IMHO), then there's almost $1 million for the racks out of a stated $5 million. OK, should we have a pool or a poll on who's bankrolling this thing? Jeff P.S. Andrew - thanks for the link. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sat Sep 20 12:38:51 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sun, 21 Sep 2003 00:38:51 +0800 (CST) Subject: Virginia Tech PowerMac G5 Cluster Photos Message-ID: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> http://www.chaosmint.com/mac/techclusterphotos/ Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 20 15:23:04 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 20 Sep 2003 15:23:04 -0400 (EDT) Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <3F6C89F4.303@bellsouth.net> Message-ID: On Sat, 20 Sep 2003, Jeffrey B. Layton wrote: > take at least 92 racks! ... > Yep, inexpensive, _custom_ racks (as much an oxymoron > as military intelligence as I've heard). If they charge $1000 > a rack (pretty cheap IMHO), then there's almost $1 million Ummm, 92 x 1000 = ? Don't feel bad. I do the same thing all the time. > for the racks out of a stated $5 million. OK, should we have > a pool or a poll on who's bankrolling this thing? Somebody with very deep pockets. Liquid cooled racks are probably a lot more expensive than $1000 each, but one really should likely lump the power, cooling, racks, and room all together as "base infrastructure" and sure, $1M isn't a crazy estimate for that. That leaves $4M for systems and (possibly) for humans, service agreements, electricity, and more. The power capacity also does not compute wrt their existing configuration. 1000 300W nodes is only 300 KW. Even if the nodes run at 500W each (which would be insanely hot IMHO at 250 W/processor -- about as much as a loaded dual athlon PER PROCESSOR, and enough to make even an Alpha seem cool) it is only 500 KW. Add half again for AC and miscellaneous, and they are still only up to 750 KW (which costs them roughly $500,000/year as an ongoing operating expense. They've got a factor of four surplus power capacity (if not more like six). They must be planning ahead to upgrade those racks with real rackmounts at quadruple the power/compute density during the lifetime of the facility. Assuming a three year grant cycle, $2M for infrastructure and power and (probably) salaries, they've got $3M left for nodes, which works out order of $3K/node. This doesn't sound too insane, but I don't really know the prices of these nodes. Apple has been pushing these nodes for doing bioinformatics and gene cataloguing, so my dollar is on NIH. I doubt NSA or DOE or NSF -- its a bit rich for NSF, DOE tends to be conservative, NSA tends to run their own stuff. DOD is always a possibility, but any of the latter agencies I think would be less inclined to fund an Apple cluster with such a ticklish (liquid cooled) base design. Whoever it is, they expect to go back to the well, I betcha, when rackmounts become available, or else their architect's brother sells big generators and overkill facilities plans. rgb > > Jeff > > P.S. Andrew - thanks for the link. > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Sat Sep 20 17:48:52 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Sat, 20 Sep 2003 17:48:52 -0400 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: References: Message-ID: <3F6CCB44.2020409@bellsouth.net> Robert G. Brown wrote: >On Sat, 20 Sep 2003, Jeffrey B. Layton wrote: > > > >>take at least 92 racks! >> >> >... > > >>Yep, inexpensive, _custom_ racks (as much an oxymoron >>as military intelligence as I've heard). If they charge $1000 >>a rack (pretty cheap IMHO), then there's almost $1 million >> >> > >Ummm, 92 x 1000 = ? > >Don't feel bad. I do the same thing all the time. > Sorry. Homer Simpson moment. >>for the racks out of a stated $5 million. OK, should we have >>a pool or a poll on who's bankrolling this thing? >> >> > >Somebody with very deep pockets. Liquid cooled racks are probably a lot >more expensive than $1000 each, but one really should likely lump the >power, cooling, racks, and room all together as "base infrastructure" >and sure, $1M isn't a crazy estimate for that. > >That leaves $4M for systems and (possibly) for humans, service >agreements, electricity, and more. > >The power capacity also does not compute wrt their existing >configuration. 1000 300W nodes is only 300 KW. Even if the nodes run >at 500W each (which would be insanely hot IMHO at 250 W/processor -- >about as much as a loaded dual athlon PER PROCESSOR, and enough to make >even an Alpha seem cool) it is only 500 KW. Add half again for AC and >miscellaneous, and they are still only up to 750 KW (which costs them >roughly $500,000/year as an ongoing operating expense. They've got a >factor of four surplus power capacity (if not more like six). They must >be planning ahead to upgrade those racks with real rackmounts at >quadruple the power/compute density during the lifetime of the facility. > >Assuming a three year grant cycle, $2M for infrastructure and power and >(probably) salaries, they've got $3M left for nodes, which works out >order of $3K/node. This doesn't sound too insane, but I don't really >know the prices of these nodes. > >Apple has been pushing these nodes for doing bioinformatics and gene >cataloguing, so my dollar is on NIH. I doubt NSA or DOE or NSF -- its a >bit rich for NSF, DOE tends to be conservative, NSA tends to run their >own stuff. DOD is always a possibility, but any of the latter agencies >I think would be less inclined to fund an Apple cluster with such a >ticklish (liquid cooled) base design. Whoever it is, they expect to go >back to the well, I betcha, when rackmounts become available, or else >their architect's brother sells big generators and overkill facilities >plans. > I wasn't thinking so much about NIH, but rather an individual running a company. Perhaps a couple of companies? The notes I saw said it was a collection from several schools within the university. Anyway, it's always fun to speculate (heck, those old retired guys get _paid_ to do it on TV everytime we have a war or a crisis). When RGB starts his new clothing line, perhaps I'll take up beowulf fashion commentary :) Thanks! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Sat Sep 20 20:25:13 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Sat, 20 Sep 2003 17:25:13 -0700 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <3F6C89F4.303@bellsouth.net> References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> <3F6C89F4.303@bellsouth.net> Message-ID: <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: > * 2+ million BTUs of cooling capacity using Liebert's extremedensity > cooling (rack mounted cooling via liquid refrigerant) > o traditional methods [fans] would have produced windspeeds of > 60+ MPH This is kind of weird, given that their energy density isn't that high: 2 cpus in 4U. I've seen machine rooms which have 8 times the energy density. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Sun Sep 21 00:16:04 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 20 Sep 2003 23:16:04 -0500 Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> <3F6C89F4.303@bellsouth.net> <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> Message-ID: <1064117764.1291.4.camel@terra> On Sat, 2003-09-20 at 19:25, Greg Lindahl wrote: > On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: > > > * 2+ million BTUs of cooling capacity using Liebert's extremedensity > > cooling (rack mounted cooling via liquid refrigerant) > > o traditional methods [fans] would have produced windspeeds of > > 60+ MPH > > This is kind of weird, given that their energy density isn't that > high: 2 cpus in 4U. I've seen machine rooms which have 8 times the > energy density. > Their energy density is trivial. I can't recall how many watts they are, but my guess is much less than even old celerons. I wouldn't be surprised if each of those whole machines sucks less juice than a single itanic cpu. Wasn't the G4 design spec something like 24 watts. -- -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From anandv at singnet.com.sg Sat Sep 20 01:50:26 2003 From: anandv at singnet.com.sg (Anand Vaidya) Date: Sat, 20 Sep 2003 13:50:26 +0800 Subject: Cluster distributions? In-Reply-To: References: Message-ID: <200309201350.26611.anandv@singnet.com.sg> I was amazed to read that one needs to *reinstall* all nodes just to apply one ssh update. Isn't that very wasteful? Then what happens to the custom ssh keys etc? http://www.rocksclusters.org/errata/Security/ssh-alert.html says: 7.Reinstall all compute nodes # cluster-fork /boot/kickstart/cluster-kickstart Or is there some misunderstanding on my part? Regards, Anand On Friday 19 September 2003 23:48, Robert Kane wrote: > If no one has already mentioned it, Rocks (www.rocksclusters.org) is a > nice distribution to use. It's a customized distribution of Redhat 7.x. > It's stable, runs on a widely used and supported version of Linux, and has > all sorts of cluster tools (monitors, schedulers, mpi) integrated. It's > generally quick to install, and easy to manage. > > Robert Kane > > > I've been running a very small 4 node beowulf cluster for a project. > > Right now I'm running Debian on the cluster and I'm not very happy with > > it or the tools available through the distribution. > > > > Does anyone have any suggestions of a good distribution? What about > > MSC.Linux? any good? > > I am also looking for tools to automate software installations across all > > the nodes, etc. > > I like the Debian package system, I just wish there was an easy way to > > have the process replicate across all the computers. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sun Sep 21 11:26:12 2003 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sun, 21 Sep 2003 23:26:12 +0800 (CST) Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <1064117764.1291.4.camel@terra> Message-ID: <20030921152612.6481.qmail@web16807.mail.tpe.yahoo.com> G5s need more power than G4s for sure, as their die size is larger, but AFAIK, the typical power usage is lower than the Pentium 4s, and thus much much lower than the IA64 chips. Also, it's it a waste to have 1100 brand new ATI video cards, and another 1100 superdrives **unused** in the cluster? (I would love to have them!) Andrew. --- Dean Johnson ???: > Their energy density is trivial. I can't recall how > many watts they are, > but my guess is much less than even old celerons. I > wouldn't be > surprised if each of those whole machines sucks less > juice than a single > itanic cpu. Wasn't the G4 design spec something like > 24 watts. > > -- > > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From luken at linuxnetworx.com Fri Sep 19 21:31:10 2003 From: luken at linuxnetworx.com (Joshua Aune) Date: Fri, 19 Sep 2003 19:31:10 -0600 Subject: opteron problems In-Reply-To: <3F6B6002.2020606@cert.ucr.edu> References: <3F6B6002.2020606@cert.ucr.edu> Message-ID: <20030920013110.GU25847@earth.omner.org> On Fri, Sep 19, 2003 at 12:58:58PM -0700, Glen Kaukola wrote: > We've purchased a few Opteron's to play with, and they're turning out to > not be very stable. The motherboard in the systems is the Tyan Thunder > K8S S2280, and I'm running Red Hat 9 on two of them and Suse 8.2 x86_64 > beta on the other two. Each machine has the latest version of the bios > from Tyan's website. I'm running the setiathome client, and a few of Do you know if the tyan BIOS has all the latest processor errata for your CPU rev implemented? We were working with 2 very large opteron systems, one with B3 rev CPUs and another with C0 rev CPUs. At the time the BIOS didn't implement all the errata for the B3 rev of CPUs so they crashed regularly with very odd oopses (random memory corruption) while the C0 based systems were solid. After the BIOS was fixed the B3s worked just as well as the C0s. > our own applications to stress test the systems. All the machines seem > to either lock up, crash with a kernel panic, or reboot out of the > blue. The lock ups, crashes, and reboots seem to happen at random about > once a day, to once every 3 days. > So, I was wondering if anyone else has had similar issues, and if you > figured out any way to overcome them? Analyzing lots and lots of oopses :) Josh _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ic02 at uow.edu.au Sat Sep 20 02:15:10 2003 From: ic02 at uow.edu.au (ivano maximus cornelius) Date: Sat, 20 Sep 2003 16:15:10 +1000 Subject: pvm: checking load of hosts Message-ID: <3F6BF06E.4030509@uow.edu.au> Hi all, I am running a pvm based monte carlo calculation consisting of a master and a number of identical spawned programs. I need a particular number of simulations performed and would like to split this number amongst the spawned programs depending upon their load at the time of spawning. Is there an easy way to do this from within my program using a routine in the pvm libraries? Thanks for your help, Iwan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rmyers1400 at comcast.net Sat Sep 20 20:52:23 2003 From: rmyers1400 at comcast.net (Robert Myers) Date: Sat, 20 Sep 2003 20:52:23 -0400 Subject: Searchable beowulf list archive In-Reply-To: <3F69BA47.4050509@inel.gov> References: <3F69322E.9060504@polyu.edu.hk> <3F69BA47.4050509@inel.gov> Message-ID: <3F6CF647.8000705@comcast.net> Andrew Shewmaker wrote: > LUK ShunTim wrote: > >> Hello, >> >> There used to be a searchable archive for this list at >> http://www.supercomputer.org/Search/ >> but when I tried to use it just now, it is no longer there. >> >> Is there other alternative location? > > > The Mailing list ARChives at AIMS are nice. > > http://marc.theaimsgroup.com/?l=beowulf&r=1&w=2v > > > -Andrew That's a good link. Google advanced search, of couse, does just fine, if you specify the site to search as beowulf.org. For me, at least, the search results are a little easier to pick through because the citations link immediately to beowulf's own indices. RM RM RM _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Sat Sep 20 09:46:02 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Sat, 20 Sep 2003 15:46:02 +0200 (CEST) Subject: opteron problems In-Reply-To: References: <3F6B6002.2020606@cert.ucr.edu> Message-ID: <2171.81.57.165.53.1064065562.squirrel@webmail.mandrakesoft.com> > We're running SuSE 8.2 beta on a cluster of dual Opterons. The > motherboards are Rioworks HDAMA. > We're not seeing the problems you describe - they generall seem stable. > Recently upgraded to the latest BIOS from the Rioworks site. I've run MandrakeClustering on this kind of hardware with a 2.4.19 (patched by the Mandrake kernel team) and the error you're talking has never been reproduced. The only problem I had was some troubles with the apic (noapic should be given to the cmdline) but the lastest stable bios (1.72) gives a lot of fixes for Linux and the noapic features is no more needed. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sja at world.std.com Sat Sep 20 15:34:40 2003 From: sja at world.std.com (Stuart J Adams) Date: Sat, 20 Sep 2003 15:34:40 -0400 (EDT) Subject: Distributing makes across the cluster Message-ID: <200309201934.PAA18306@world.std.com> Is there a way to automatically distribute makes across a cluster ?? (Similar to what gnu make -j does on an SMP machine) Thanks, Stuart _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Sun Sep 21 13:22:31 2003 From: lars at meshtechnologies.com (Lars Henriksen) Date: 21 Sep 2003 19:22:31 +0200 Subject: Distributing makes across the cluster In-Reply-To: <200309201934.PAA18306@world.std.com> References: <200309201934.PAA18306@world.std.com> Message-ID: <1064164951.2276.1.camel@fermi> On Sat, 2003-09-20 at 21:34, Stuart J Adams wrote: > Is there a way to automatically distribute makes across > a cluster ?? (Similar to what gnu make -j does on an SMP machine) I have used distcc with success. Kernel compile times drop to 40 sec :-) http://distcc.samba.org/ Hope you can use it. best regards Lars -- Lars Henriksen | MESH-Technologies ApS Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Sun Sep 21 16:17:17 2003 From: landman at scalableinformatics.com (Joe Landman) Date: Sun, 21 Sep 2003 16:17:17 -0400 Subject: Cluster distributions? In-Reply-To: <200309201350.26611.anandv@singnet.com.sg> References: <200309201350.26611.anandv@singnet.com.sg> Message-ID: <3F6E074D.2070504@scalableinformatics.com> Anand Vaidya wrote: >I was amazed to read that one needs to *reinstall* all nodes just to apply one >ssh update. Isn't that very wasteful? Then what happens to the custom ssh >keys etc? > You can simply place the RPM in a shared location and cluster-fork "rpm -Fvh /path/to/new-ssh-binary.rpm ; /etc/init.d/sshd restart" You are not forced to use the re-install method, though it is on by default. Turning it off is relatively simple: cluster-fork "/etc/init.d/rocks-grub stop; /sbin/chkconfig --level 2345 rocks-grub off" Or you can remove it from the node build tree (instructions in the documentation). As for new ssh keys, I found that I simply have to clear out the ~/.ssh/known_hosts. It is annoying to have to do this for all users though. I typically turn off the automatic re-install and default re-install for my customers. Joe > >http://www.rocksclusters.org/errata/Security/ssh-alert.html >says: > >7.Reinstall all compute nodes > # cluster-fork /boot/kickstart/cluster-kickstart > >Or is there some misunderstanding on my part? > >Regards, >Anand > > >On Friday 19 September 2003 23:48, Robert Kane wrote: > > >>If no one has already mentioned it, Rocks (www.rocksclusters.org) is a >>nice distribution to use. It's a customized distribution of Redhat 7.x. >>It's stable, runs on a widely used and supported version of Linux, and has >>all sorts of cluster tools (monitors, schedulers, mpi) integrated. It's >>generally quick to install, and easy to manage. >> >>Robert Kane >> >> >> >>>I've been running a very small 4 node beowulf cluster for a project. >>>Right now I'm running Debian on the cluster and I'm not very happy with >>>it or the tools available through the distribution. >>> >>>Does anyone have any suggestions of a good distribution? What about >>>MSC.Linux? any good? >>>I am also looking for tools to automate software installations across all >>>the nodes, etc. >>>I like the Debian package system, I just wish there was an easy way to >>>have the process replicate across all the computers. >>> >>> >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >>http://www.beowulf.org/mailman/listinfo/beowulf >> >> > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Sun Sep 21 20:54:42 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Sun, 21 Sep 2003 17:54:42 -0700 (PDT) Subject: Virginia Tech PowerMac G5 Cluster Photos In-Reply-To: <20030921002513.GA4350@greglaptop.greghome.keyresearch.com> Message-ID: On Sat, 20 Sep 2003, Greg Lindahl wrote: > On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: > > > * 2+ million BTUs of cooling capacity using Liebert's extremedensity > > cooling (rack mounted cooling via liquid refrigerant) > > o traditional methods [fans] would have produced windspeeds of > > 60+ MPH > > This is kind of weird, given that their energy density isn't that > high: 2 cpus in 4U. I've seen machine rooms which have 8 times the > energy density. we have 8 opteron 240's in 4u ;) > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Tue Sep 23 09:38:31 2003 From: jcownie at etnus.com (James Cownie) Date: Tue, 23 Sep 2003 14:38:31 +0100 Subject: SC03 Tutorials Message-ID: <1A1nN9-4zk-00@etnus.com> In case folks have missed it the SC03 tutorial schedule is now available here :- http://www.sc-conference.org/sc2003/tech_tutorials.php and there are many which seem relevant to this audience (for instance) S01: Production Linux Clusters 2003 - Architecture and System Software for Serious Computing S02: A Tutorial Introduction to High Performance Data Transport S03: A practical approach to performance analysis and modeling of large-scale systems S05: An Introduction to the TotalView Debugger S09: Programming with the Partitioned Global Address Space Model M03: Lustre: A Scalable, High-Performance Distributed File System M04: Using MPI-2: Advanced Features of the Message-Passing Interface M07: PERC Tools for Performance Data Gathering and Analysis M09: Cluster Construction Utilizing Selected Packages M13: HPC and InfiniBand Architecture in Practice You can get the details from the site above. -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Tue Sep 23 08:44:08 2003 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Tue, 23 Sep 2003 09:44:08 -0300 (ART) Subject: MPICH vs LAM Message-ID: <20030923124408.91196.qmail@web12205.mail.yahoo.com> What implmentation of MPI should i use? There's the mpich and the lam implamentations, what's the best choice, and have more improviments of resources and performance? Thanks Mathias Brito Universidade Estadual de Santa Cruz Bahia-Brasil _______________________________________________________________________ Desafio AntiZona: participe do jogo de perguntas e respostas que vai dar um Renault Clio, computadores, c?meras digitais, videogames e muito mais! www.cade.com.br/antizona _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Mon Sep 22 14:34:04 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Mon, 22 Sep 2003 14:34:04 -0400 Subject: Ramdisk Size In-Reply-To: <3F6F32F0.1070400@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> Message-ID: <3F6F409C.3040101@bellsouth.net> Tony Travis wrote: > Eric R Johnson wrote: > >> I would like to create a large ramdisk to keep a database which will >> be searched often, but not changed. >> The only thing I currently know how to do to create a ramdisk is: >> mke2fs /dev/ram0 >> mount /dev/ram0 /tmp/ramdisk0 >> >> The only problem is that this creates a 4MB ramdisk, and I would like >> something on the order of 1GB. > > > Hello, Eric. > > Are you *sure* you want to do this? > > RAM disks are usually used for loading modules at boot time. That's > why they are small. Typically, the contents of a RAM disk are used as > a root filesystem for the kernel to load e.g. SCSI drivers, then > discarded when the 'real' root filesystem is mounted from SCSI disks. > People also use them to boot a 'minimal' Linux for diagnostics or for > some other reason. Check out Warewulf: warewulf-cluster.org. Very cool cluster distribution that uses a very small Ramdisk to hold the necessary parts of the OS. I've just started to use it and it's very slick. (snip) > However, I think it would be better to use 'tmpfs' for a 1Gb RAM > filesystem. The kernel will provide fast access to the most frequently > accessed areas of your database held in RAM, but it will move the less > frequently accessed areas onto the swap device freeing up RAM for your > programs and for other disk caches. What's the difference between tmpfs and a ramdisk created through the kernel? Is there a speed difference? BTW, here's a link on creating a generic ram disk: http://www.lissot.net/partition/ramdisk.html (read the bottom part). If you are happy with the speed improvement of using a ram disk to store the database, you might want to consider squashfs (http://squashfs.sourceforge.net/) It's a read-only filesystem that you can use to reduce the size of the required ram disk. You will take a hit in speed, but perhaps the ram disk will make up for it. Oh, and you will have to patch the kernel to use it. Good Luck! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bioinformaticist at mn.rr.com Mon Sep 22 02:04:11 2003 From: bioinformaticist at mn.rr.com (Eric R Johnson) Date: Mon, 22 Sep 2003 01:04:11 -0500 Subject: Ramdisk Size Message-ID: <3F6E90DB.1060505@mn.rr.com> I would like to create a large ramdisk to keep a database which will be searched often, but not changed. The only thing I currently know how to do to create a ramdisk is: mke2fs /dev/ram0 mount /dev/ram0 /tmp/ramdisk0 The only problem is that this creates a 4MB ramdisk, and I would like something on the order of 1GB. I tried running menuconfig and changing the default ramdisk size to 1GB, making the kernel, and rebooting. Now when I run menuconfig, it shows the ramdisk size as 1GB, but when I execute the mke2fs /dev/ram0, it still only creates a 4MB ramdisk. Can someone point me in the right direction? Thanks, Eric _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mack.joseph at epa.gov Mon Sep 22 06:34:39 2003 From: mack.joseph at epa.gov (Joseph Mack) Date: Mon, 22 Sep 2003 06:34:39 -0400 Subject: Virginia Tech PowerMac G5 Cluster Photos References: <20030920163851.96473.qmail@web16809.mail.tpe.yahoo.com> Message-ID: <3F6ED03F.8E97FA32@epa.gov> Andrew Wang wrote: > > http://www.chaosmint.com/mac/techclusterphotos/ If you go through to the "supercomputercenter link" at the bottom of the page and look at the photos there, you'll get a notice that "this page has been optimised for IE, it may not show on your browser" (and it didn't, for netscape and mozilla). Joe -- Joseph Mack PhD, High Performance Computing & Scientific Visualization SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal Contact - John B. Smith 919-541-1087 - smith.johnb at epa.gov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Tue Sep 23 12:49:56 2003 From: josip at lanl.gov (Josip Loncaric) Date: Tue, 23 Sep 2003 10:49:56 -0600 Subject: MPICH vs LAM In-Reply-To: <20030923124408.91196.qmail@web12205.mail.yahoo.com> References: <20030923124408.91196.qmail@web12205.mail.yahoo.com> Message-ID: <3F7079B4.4090807@lanl.gov> Mathias Brito wrote: > What implmentation of MPI should i use? There's the > mpich and the lam implamentations, what's the best > choice, and have more improviments of resources and > performance? Why not install both and give yourself a choice? MPICH is very popular and works fine, but LAM is a bit cleaner implementation which delivers a bit lower latency. Other than that, the main difference is process startup. With MPICH, "mpirun" does the job directly and accepts the usual rsh or ssh authorization overhead. With LAM, you "lamboot" LAM daemons on your nodes (w/usual overhead) but then those daemons can start your MPI processes quicker. After your MPI jobs (possibly many in sequence) are done, you use "wipe" to clean up LAM daemons. On the Coral cluster at ICASE (now absorbed by NIA), LAM was the default due to slightly lower latency. LAM is very nicely done, but having only one flavor of MPI can be limiting, so people could use MPICH, MPI/Pro (in VIA/TCP/interrupt/polling modes), or MVICH as they saw fit. For example, some codes assume MPICH-specific implementation details, which is non-standard but easy to accomodate. My suggestion is to install both LAM and MPICH, then benchmark your applications with both and use the better choice as the default. Sincerely, Josip P.S. There are other MPIs out there: various vendor MPIs, enhanced flavors of MPICH, etc. Some of these cost $/CPU/year; others are free. A commercial MPI (e.g. MPI/Pro) buys you enhanced robustness and some interesting features, e.g. interrupt-driven receive which can improve performance of certain applications. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Tue Sep 23 08:56:24 2003 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Tue, 23 Sep 2003 09:56:24 -0300 (ART) Subject: MPI Parallel algorithms Message-ID: <20030923125624.73310.qmail@web12206.mail.yahoo.com> I starting to research how to program my beowulg cluster, i already made a lot of simple programs, but i would like to see more complexs algorithms. Someone know where can I find parallel algorithms codes with MPI on the web. Thanks Mathias Brito Universidade Estadual de Santa Cruz Bahia-Brasil _______________________________________________________________________ Desafio AntiZona: participe do jogo de perguntas e respostas que vai dar um Renault Clio, computadores, c?meras digitais, videogames e muito mais! www.cade.com.br/antizona _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Tue Sep 23 04:37:45 2003 From: john.hearns at clustervision.com (John Hearns) Date: Tue, 23 Sep 2003 10:37:45 +0200 (CEST) Subject: Redhat Fedora Message-ID: Hopefully not too far off-topic, as a lot of clusters use Redhat. As most of us know, Redhat announced that they are pursuing a new community led development model for future Redhat 'consumer' releases. I'm sure I'm not the only one who was wondering what that meant for the direction of cluster computing. There's more meat on the story now with what looks like a tie-up between Redhat and the Fedora Linux project (Fedora set out to have a high quality update and addon package site). http://fedora.redhat.com And it supports YUM and apt-get, which can only be a good thing. Comments? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From msegone at ug.ms.wits.ac.za Mon Sep 22 15:56:20 2003 From: msegone at ug.ms.wits.ac.za (masego-otlhe_segone) Date: Mon, 22 Sep 2003 21:56:20 +0200 Subject: running mpich on multiple nodes Message-ID: <20030922195027.M59251@ug.ms.wits.ac.za> Hi i have installed mpich on a diskless network, on the server machine. I am unble to execute my programs on 2 nodes. I have enable ssh without password reiurements but i dont know what is wrong.Please help this is the error messge I am getting /usr/local/mpich-1.2.5.2/bin/mpirun -np 2 These are results /usr/local/mpich-1.2.5.2/examples/basic/cpi ug: Connection refused p0_3870: p4_error: Child process exited while making connection to remote process on ug: 0 /usr/local/mpich-1.2.5.2/bin/mpirun: line 1: 3870 Broken pipe /usr/local/mpich-1.2.5.2/examples/basic/cpi -p4pg /usr/local/mpich-1.2.5.2/examples/basic/PI3790 -p4wd /usr/local/mpich-1.2.5.2/examples/basic Masego-otlhe Segone _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Mon Sep 22 19:53:44 2003 From: raysonlogin at yahoo.com (Rayson Ho) Date: Mon, 22 Sep 2003 16:53:44 -0700 (PDT) Subject: Fluid Dynamics Benchmark - G5 vs G4 vs Itanium2 (was: Re: Virginia Tech PowerMac G5 Cluster Photos) In-Reply-To: <200309221441.h8MEfdn01995@mycroft.ahpcrc.org> Message-ID: <20030922235344.41674.qmail@web11401.mail.yahoo.com> 1.8 GHz G5 vs 1.3 GHz Itanium2: http://www.xlr8yourmac.com/G5/G5_fluid_dynamics_bench/G5_fluid_dynamics_bench.html Rayson --- Richard Walsh wrote: > The "estimated" power consumption on for the chip is 42W at 1.8 GHz > and 1.3v, or 19W at 1.2.GHz and 1.1v according to an IBM PowerPC 970 > overview I have. > > rbw > __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Mon Sep 22 16:04:16 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Mon, 22 Sep 2003 16:04:16 -0400 Subject: Ramdisk Size In-Reply-To: <3F6F4ECC.20703@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> <3F6F4ECC.20703@rri.sari.ac.uk> Message-ID: <3F6F55C0.3010601@bellsouth.net> Tony Travis wrote: > Jeffrey B. Layton wrote: > >> [...] >> What's the difference between tmpfs and a ramdisk created through >> the kernel? Is there a speed difference? > > > The main difference is that tmpfs uses both RAM and swap. The memory > manager will therefore optimise use of RAM for programs, tmpfs, or > disk cache according to demand: Inactive pages of processes and > infrequently accessed parts of tmpfs filesystems are written to disk > to free up RAM when the system starts to run out of physical memory > (RAM). By default, tmpfs size is limited to half the available RAM > detected at boot time. Didn't know this. Thanks! Doesn't this imply that you could get swapped to disk as some point? So if you wanted to make sure everything stayed in memory you would have to use a ram disk. But if you didn't really mind getting some pages swapped to disk (not much a measurable speed hit on the application), then tmpfs is the way to go. Is this accurate? >> BTW, here's a link on creating a generic ram disk: >> >> http://www.lissot.net/partition/ramdisk.html >> >> (read the bottom part). > > > Hey! that only tells you how to format and mount a default 4Mb ramdisk. Just change the count to what you want. Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Mon Sep 22 15:34:36 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Mon, 22 Sep 2003 20:34:36 +0100 Subject: Ramdisk Size In-Reply-To: <3F6F409C.3040101@bellsouth.net> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> Message-ID: <3F6F4ECC.20703@rri.sari.ac.uk> Jeffrey B. Layton wrote: >[...] > What's the difference between tmpfs and a ramdisk created through > the kernel? Is there a speed difference? The main difference is that tmpfs uses both RAM and swap. The memory manager will therefore optimise use of RAM for programs, tmpfs, or disk cache according to demand: Inactive pages of processes and infrequently accessed parts of tmpfs filesystems are written to disk to free up RAM when the system starts to run out of physical memory (RAM). By default, tmpfs size is limited to half the available RAM detected at boot time. > BTW, here's a link on creating a generic ram disk: > > http://www.lissot.net/partition/ramdisk.html > > (read the bottom part). Hey! that only tells you how to format and mount a default 4Mb ramdisk. You don't have to boot with an "initrd=" argument. You could use e.g.: boot: vmlinuz ramdisk_size=10485760 ... mkfs -t ext3 /dev/ram0 mkdir /mnt/ramdisk mount /dev/ram0 /mnt/ramdisk cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk However, I still think you would be better off with tmpfs :-) Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rbw at ahpcrc.org Mon Sep 22 10:41:39 2003 From: rbw at ahpcrc.org (Richard Walsh) Date: Mon, 22 Sep 2003 09:41:39 -0500 Subject: Virginia Tech PowerMac G5 Cluster Photos Message-ID: <200309221441.h8MEfdn01995@mycroft.ahpcrc.org> On Sun Sep 21 00:21:03, Dean Johnson wrote: >On Sat, 2003-09-20 at 19:25, Greg Lindahl wrote: >> On Sat, Sep 20, 2003 at 01:10:12PM -0400, Jeffrey B. Layton wrote: >> >> > * 2+ million BTUs of cooling capacity using Liebert's extremedensity >> > cooling (rack mounted cooling via liquid refrigerant) >> > o traditional methods [fans] would have produced windspeeds of >> > 60+ MPH >> >> This is kind of weird, given that their energy density isn't that >> high: 2 cpus in 4U. I've seen machine rooms which have 8 times the >> energy density. >> > >Their energy density is trivial. I can't recall how many watts they are, >but my guess is much less than even old celerons. I wouldn't be >surprised if each of those whole machines sucks less juice than a single >itanic cpu. Wasn't the G4 design spec something like 24 watts. The "estimated" power consumption on for the chip is 42W at 1.8 GHz and 1.3v, or 19W at 1.2.GHz and 1.1v according to an IBM PowerPC 970 overview I have. rbw _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 22 21:48:55 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 23 Sep 2003 11:48:55 +1000 Subject: rsh In-Reply-To: <20030917024414.M10506@ug.ms.wits.ac.za> References: <20030917024414.M10506@ug.ms.wits.ac.za> Message-ID: <200309231148.56402.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > just realised that I cannot communicate with other hosts due to some rsh > setting that I need to do. To be honest I'd really suggest not using RSH and swapping over to OpenSSH instead. In that way you can use the key management stuff to authenticate users making things that little bit safer. One note of caution, make sure you get the latest update for your distribution, or build it from the latest sources. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/b6aHO2KABBYQAh8RAtJUAJ4gE6MjP6bOnqrPefVCQOAMWp51+gCdF0A6 nS5+cuFpALePLGwxIj3booo= =tPsX -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at bellsouth.net Mon Sep 22 18:32:04 2003 From: laytonjb at bellsouth.net (Jeffrey B. Layton) Date: Mon, 22 Sep 2003 18:32:04 -0400 Subject: Ramdisk Size In-Reply-To: <20030922215254.GA3648@greglaptop.internal.keyresearch.com> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> <20030922215254.GA3648@greglaptop.internal.keyresearch.com> Message-ID: <3F6F7864.2000600@bellsouth.net> > > >> If you are happy with the speed improvement of using a ram disk >>to store the database, you might want to consider squashfs >> >> > >If he's doing things like gene comparisons, the work of uncompressing >the database would be a huge performance loss. > That was what I was wondering. If he's used to speed X running his DB on a disk, then moving to a ram disk may get him speed Y. If using squashfs reduces his speed to Z and Z > X, then it's still a win for him. However, if he gets used to speed Z and can afford the memory space, then skip squashfs. (flowchart anyone?). Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Mon Sep 22 17:52:54 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Mon, 22 Sep 2003 14:52:54 -0700 Subject: Ramdisk Size In-Reply-To: <3F6F409C.3040101@bellsouth.net> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F6F409C.3040101@bellsouth.net> Message-ID: <20030922215254.GA3648@greglaptop.internal.keyresearch.com> On Mon, Sep 22, 2003 at 02:34:04PM -0400, Jeffrey B. Layton wrote: > Check out Warewulf: warewulf-cluster.org. Very cool cluster > distribution that uses a very small Ramdisk to hold the necessary > parts of the OS. I've just started to use it and it's very slick. That's how Scyld Beowulf works, too. > What's the difference between tmpfs and a ramdisk created through > the kernel? Is there a speed difference? I believe that tmpfs is dynamically sized, and the Linux ramdisk device is not. > If you are happy with the speed improvement of using a ram disk > to store the database, you might want to consider squashfs If he's doing things like gene comparisons, the work of uncompressing the database would be a huge performance loss. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Mon Sep 22 14:12:54 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Mon, 22 Sep 2003 19:12:54 +0100 Subject: Ramdisk Size - correction In-Reply-To: <3F6F32F0.1070400@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> Message-ID: <3F6F3BA6.10504@rri.sari.ac.uk> Tony Travis wrote: > [...] > For example, you could create a 10Mb initrd image of a RAM disk using: > > dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 > mkfs -t ext3 /tftpboot/initrd.img > mkdir /mnt/ramdisk > mount -o loop /tftpboot/initrd.img /mnt/ramdisk > cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk > umount /var/tmp/initrd.img > zcat /tftpboot/initrd.img > /tftpboot/zinitrd.img Hello, Eric. Sorry, I was careless about pathnames - Here's a correction: dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 mkfs -t ext3 /var/tmp/initrd.img mkdir /mnt/ramdisk mount -o loop /var/tmp/initrd.img /mnt/ramdisk cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk umount /var/tmp/initrd.img zcat /var/tmp/initrd.img > /tftpboot/zinitrd.img You don't need the ramdisk image in /var/tmp once you've saved the compressed image: In my case, I'm PXE booting diskless openMosix from /tftpboot which is where the kernel and ramdisk image are stored. Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Mon Sep 22 13:35:44 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Mon, 22 Sep 2003 18:35:44 +0100 Subject: Ramdisk Size In-Reply-To: <3F6E90DB.1060505@mn.rr.com> References: <3F6E90DB.1060505@mn.rr.com> Message-ID: <3F6F32F0.1070400@rri.sari.ac.uk> Eric R Johnson wrote: > I would like to create a large ramdisk to keep a database which will be > searched often, but not changed. > The only thing I currently know how to do to create a ramdisk is: > mke2fs /dev/ram0 > mount /dev/ram0 /tmp/ramdisk0 > > The only problem is that this creates a 4MB ramdisk, and I would like > something on the order of 1GB. Hello, Eric. Are you *sure* you want to do this? RAM disks are usually used for loading modules at boot time. That's why they are small. Typically, the contents of a RAM disk are used as a root filesystem for the kernel to load e.g. SCSI drivers, then discarded when the 'real' root filesystem is mounted from SCSI disks. People also use them to boot a 'minimal' Linux for diagnostics or for some other reason. For example, you could create a 10Mb initrd image of a RAM disk using: dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 mkfs -t ext3 /tftpboot/initrd.img mkdir /mnt/ramdisk mount -o loop /tftpboot/initrd.img /mnt/ramdisk cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk umount /var/tmp/initrd.img zcat /tftpboot/initrd.img > /tftpboot/zinitrd.img Then, override the 4Mb default at boot time: boot: vmlinuz ramdisk_size=10485760 initrd=zinitrd.img However, I think it would be better to use 'tmpfs' for a 1Gb RAM filesystem. The kernel will provide fast access to the most frequently accessed areas of your database held in RAM, but it will move the less frequently accessed areas onto the swap device freeing up RAM for your programs and for other disk caches. Mounting /tmp as a 'tmpfs' filesystem reserves half of your memory for the /tmp filesystem by default. Add an entry like this to /etc/fstab: tmpfs /tmp tmpfs defaults 0 0 Beware that openMosix, in particular, does not work well with tmpfs and automatically comments out tmpfs entries from your /etc/fstab when you install openMosic from the 2.4.21-openmosix1 binary rpm distribution. Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From henken at seas.upenn.edu Mon Sep 22 16:49:19 2003 From: henken at seas.upenn.edu (Nicholas Henke) Date: 22 Sep 2003 16:49:19 -0400 Subject: Lustre In-Reply-To: <3F67151C.A5D41000@epa.gov> References: <200309160953.42528.csamuel@vpac.org> <3F67151C.A5D41000@epa.gov> Message-ID: <1064263759.30109.39.camel@roughneck> On Tue, 2003-09-16 at 09:50, Joseph Mack wrote: > Chris Samuel wrote: > > > Anyone tried out Lustre [1] ? > > There was a talk at OLS this year > > http://www.linuxsymposium.org/2003/ > > As usual when given by a person involved in the development, > it's always presented as being easy to install and running > smoothly. > > However they do have it running on two kilonode production clusters and > have won the contract for the filesystem for some new cluster that is > currently being bid on. So it's in production in some places Just set up a 2 node setup here, it is pretty easy, and it works great as far as I can tell. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmkurtzer at lbl.gov Mon Sep 22 13:43:44 2003 From: gmkurtzer at lbl.gov (Greg Kurtzer) Date: Mon, 22 Sep 2003 10:43:44 -0700 Subject: Ramdisk Size In-Reply-To: <3F6E90DB.1060505@mn.rr.com> References: <3F6E90DB.1060505@mn.rr.com> Message-ID: <20030922174344.GA24330@tux.lbl.gov> Kernel arg: 'ramdisk=n' (ie. ramdisk=1024000 for a 1G ram disk). We typically don't go over several hundred MB's, so let me know how this works. Also, this works with standard kernels (kernel.org and the RH kernel). Greg On Mon, Sep 22, 2003 at 01:04:11AM -0500, Eric R Johnson told me: > I would like to create a large ramdisk to keep a database which will be > searched often, but not changed. > The only thing I currently know how to do to create a ramdisk is: > mke2fs /dev/ram0 > mount /dev/ram0 /tmp/ramdisk0 > > The only problem is that this creates a 4MB ramdisk, and I would like > something on the order of 1GB. > > I tried running menuconfig and changing the default ramdisk size to 1GB, > making the kernel, and rebooting. Now when I run menuconfig, it shows > the ramdisk size as 1GB, but when I execute the mke2fs /dev/ram0, it > still only creates a 4MB ramdisk. > > Can someone point me in the right direction? > > Thanks, > Eric -- Greg M. Kurtzer, CSE: Linux cluster specialist Lawrence Berkeley National Laboratory Contact: O=510.495.2307, P=510.448.4540, M=510.928.9953 1 Cyclotron Road MS:50C-3396, Berkeley, CA 94720 http://www.lbl.gov, http://scs.lbl.gov/, http://lug.lbl.gov/ Email: GMKurtzer_at_lbl.gov, Text: 5109289953_at_mobileatt.net _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Tue Sep 23 13:23:23 2003 From: josip at lanl.gov (Josip Loncaric) Date: Tue, 23 Sep 2003 11:23:23 -0600 Subject: Fluid Dynamics Benchmark - G5 vs G4 vs Itanium2 In-Reply-To: <20030922235344.41674.qmail@web11401.mail.yahoo.com> References: <20030922235344.41674.qmail@web11401.mail.yahoo.com> Message-ID: <3F70818B.4030601@lanl.gov> Rayson Ho wrote: > 1.8 GHz G5 vs 1.3 GHz Itanium2: > > http://www.xlr8yourmac.com/G5/G5_fluid_dynamics_bench/G5_fluid_dynamics_bench.html [ Summary: G5 wins the large memory case, loses small/medium cases ] Has anyone made a STREAM benchmark comparison for these systems? This CFD benchmark is probably memory bandwidth limited... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csarat1 at uic.edu Tue Sep 23 16:28:20 2003 From: csarat1 at uic.edu (Sarat C Maruvada) Date: Tue, 23 Sep 2003 15:28:20 -0500 (CDT) Subject: A question of OS!! Message-ID: Hi,I am planning on installing a 32 node beowulf cluster for high performance bio-informatics applications. As a OS I was planning on installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux has advantage in parallel computations. Having no hands-on experience in SuSe Linux, I have no idea how to verify this or do a comparision. What do you advice as the OS for the cluster? And how to determine this,as there are a bunch of unix OSes out there.Thanks. Sincerely, Sarat. ************************************************************** Sarat Chandra Maruvada 830,S.Claremont Ave., Research Assistant, CS Dept., #2, Prof. Florin Balasa, Chicago, SEO 937, IL - 60612. UIC. Ph: 312-850-9711. ************************************************************** _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wrankin at ee.duke.edu Tue Sep 23 17:01:12 2003 From: wrankin at ee.duke.edu (Bill Rankin) Date: 23 Sep 2003 17:01:12 -0400 Subject: [Fwd: HPC New Positions at UNC] Message-ID: <1064350872.2225.2.camel@rohgun.cse.duke.edu> FYI, I didn't know if this had made the list yet (I'm on digest mode) but UNC Chapel Hill is looking for cluster programmers and instructors. -bill -----Forwarded Message----- From: C. D. Poon To: hpcplanning Subject: HPC New Positions at UNC Date: 23 Sep 2003 16:45:21 -0400 We are recruiting for two new positions to further enhance support for research computing at the University of North Carolina at Chapel Hill. The focus of one position is code optimization and parallelization. The primary responsibility of the second position is to develop and teach training courses on specific computational applications, parallel code development and testing, checkpointing, and other aspects of our computing environment. The job announcements are currently posted at the UNC-CH Human Resources web site: http://www.ais.unc.edu/hr/jobs/epajobs.htm Please share this information with your colleagues. C. D. Poon High-Performance Computing Group Academic Technology & Networks University of North Carolina -------------------------------------------------------------------- Research Computing Associate for High Performance Computing EPA Non-Faculty September 17, 2003 Position Description: Research Computing Associate for High Performance Computing will provide expertise for current and future high performance computing code optimization, parallelization and porting. This position will work closely with faculty and other University researchers to determine the appropriate architecture for each application and will then assist with porting and tuning applications for optimal performance on the chosen platform. In addition to supporting the porting and optimization of researchers codes, this position will help identify and analyze codes to be used for benchmarking and research of high performance computing resources. Education Required: Position Requires a M.S. or Ph.D in computer science or related computational science discipline and demonstrated hands-on expertise and high performance programming experience with MPICH, OpenMP, Fortran and C++. Experience Required: Experience in performance evaluation of both distributed and shared parallel memory architectures is highly desired as is experience with parallel scientific simulation codes and writing applications for parallel computers. Excellent interpersonal skills, excellent oral and written communications skills and demonstrated organizational skills are required, along with strong personal motivation and a proven ability to work in a team environment. Application Deadline: October 29, 2003. Applicants should send a cover letter, resume and three references to: Diane Strong, Human Resources Mgr. Academic Technology & Networks (ATN) UNC-CH, Campus Box # 3420 Chapel Hill, NC 27599-3420 The University of North Carolina at Chapel Hill is an equal opportunity employer. -------------------------------------------------------------------- High Performance Computing Instructor EPA Non-Faculty September 17, 2003 Position Description: The High Performance Computing Instructor will determine course content, develop and teach introductory classes about the UNC-Chapel Hill high performance computing environment and how to use it, parallel programming, scientific applications, and specialized coding and performance analysis for faculty and their research teams. Within ATN, the Academic Computing Systems Division provides the infrastructure and support for the research computing, data management and storage, messaging, web and enterprise backup and recovery needs of the University community. The High Performance Computing Instructor will develop course materials for computational science techniques. These materials will be used to design on-line tutorials as well as hands-on workshops and courses for researchers with computationally intensive problems. Education Required: Position Requires a M.S. or Ph.D in computer science or related computational science discipline and demonstrated hands-on expertise and high performance programming experience with MPICH, OpenMP, Fortran and C++. Experience Required: Experience with parallel scientific codes and writing applications for parallel computers is required. Demonstrated teaching skills are highly desired. Excellent interpersonal skills, excellent oral and written communications skills and demonstrated organizational skills are required, along with strong personal motivation and proven ability to work in a team environment. Application Deadline: October 29, 2003. Applicants should send a cover letter, resume and three references to: Diane Strong, Human Resources Mgr. Academic Technology & Networks (ATN) UNC-CH, Campus Box # 3420 Chapel Hill, NC 27599-3420 The University of North Carolina at Chapel Hill is an equal opportunity employer. -------------------------------------------------------------------- -- bill rankin, ph.d. ........ director, cluster and grid technology group wrankin at ee.duke.edu .......................... center for computational duke university ...................... science engineering and medicine http://www.ee.duke.edu/~wrankin .............. http://www.csem.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jschauma at netmeister.org Mon Sep 22 22:40:26 2003 From: jschauma at netmeister.org (Jan Schaumann) Date: Mon, 22 Sep 2003 22:40:26 -0400 Subject: rsh In-Reply-To: <200309231148.56402.csamuel@vpac.org> References: <20030917024414.M10506@ug.ms.wits.ac.za> <200309231148.56402.csamuel@vpac.org> Message-ID: <20030923024026.GE4479@netmeister.org> Chris Samuel wrote: > On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > > > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > > just realised that I cannot communicate with other hosts due to some rsh > > setting that I need to do. > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > instead. Why? Most likely, ssh will introduce a non-negligible overhead in encryption and/or compression. To the OP: check your /etc/hosts.equiv and ~/.rhosts, as suggested by rsh(1) and rlogin(1). -Jan -- "I am so amazingly cool you could keep a side of meat in me for a month. I am so hip I have difficulty seeing over my pelvis." -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From glen at cert.ucr.edu Tue Sep 23 18:24:47 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Tue, 23 Sep 2003 15:24:47 -0700 Subject: A question of OS!! In-Reply-To: References: Message-ID: <3F70C82F.2070307@cert.ucr.edu> Sarat C Maruvada wrote: >Hi,I am planning on installing a 32 node beowulf cluster for high >performance bio-informatics applications. As a OS I was planning on >installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux >has advantage in parallel computations. Having no hands-on experience in >SuSe Linux, I have no idea how to verify this or do a comparision. What do >you advice as the OS for the cluster? And how to determine this,as there >are a bunch of unix OSes out there.Thanks. > One advantage I can think of that Suse might have is it's automated installation utility, autoyast I believe it's called. While it's probably a little harder to set up, I think it would be a bit better than kickstart for installing Linux on a large number of machines. Actually though, some of the functionality of autoyast seemed to be broken to me, but again it's a lot more complicated than kickstart, so maybe it was just me. You don't have to pay to use Suse's automatic update tool, but I think that this may change on Red Hat now that they've merged with Fedora. The only other thing I can think of is that the Suse distribution seems to come with about twice as many packages to choose from. Other than that, Linux is Linux, and you're not going to see much of a speed difference between either Suse or Red Hat. My two cents, Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Kidger at quadrics.com Tue Sep 23 14:12:39 2003 From: Daniel.Kidger at quadrics.com (Daniel Kidger) Date: Tue, 23 Sep 2003 19:12:39 +0100 Subject: Distributing makes across the cluster Message-ID: <010C86D15E4D1247B9A5DD312B7F5AA78DE164@stegosaurus.bristol.quadrics.com> I have made good success with just using the standard 'make -j' but replacing where you have something like 'CC=gcc' with 'CC="prun -n1 gcc"' in the makefile. (change above for your favourite compiler and/or method of launching a binary onto your cluster). As ever the caveats of doing software builds in parallel is that date stamps are always suspect and 'make' can sometimes take the wrong route :-( Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger at quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- > -----Original Message----- > From: Lars Henriksen [mailto:lars at meshtechnologies.com] > Sent: 21 September 2003 18:23 > To: Stuart J Adams > Cc: beowulf at beowulf.org > Subject: Re: Distributing makes across the cluster > > > On Sat, 2003-09-20 at 21:34, Stuart J Adams wrote: > > Is there a way to automatically distribute makes across > > a cluster ?? (Similar to what gnu make -j does on an SMP machine) > > I have used distcc with success. Kernel compile times drop to > 40 sec :-) > > http://distcc.samba.org/ > > Hope you can use it. > > best regards > > Lars > -- > Lars Henriksen | MESH-Technologies ApS > Systems Manager & Consultant | Forskerparken 10 > www.meshtechnologies.com | DK-5260 Odense M, Denmark > lars at meshtechnologies.com | mobile: +45 2291 2904 > direct: +45 6315 7310 | fax: +45 6315 7314 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 22 03:52:43 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 22 Sep 2003 09:52:43 +0200 Subject: Cluster distributions? In-Reply-To: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> References: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> Message-ID: <1064217163.15240.6.camel@revolution.mandrakesoft.com> Ryan Burns said: > Hello, > I've been running a very small 4 node beowulf cluster for a project. > Right now I'm running Debian on the cluster and I'm not very happy with > it or the tools available through the distribution. > Does anyone have any suggestions of a good distribution? What about > MSC.Linux? any good? MandrakeClustering is the latest clustering distribution made by the CLIC team. It supports Pentium & the newest AMD64 architecture (Opteron & Athlon64). > I am also looking for tools to automate software installations across > all the nodes, etc. That one of our stronger point, the ka technologies from IDIMAG allow us to deploy new nodes within 3 minutes (duplication, reoboot, autoconfig). > I like the Debian package system, I just wish there was an easy way to > have the process replicate across all the computers. This his our 2nd stronger point. For the mananging side, using the urpmi parallel command you can manage (install/remove) rpm packages with their dependencies. This tools helps you to keep your nodes up to date. Urpmi parallel uses the ka technologies so the time needed for installing 50MB on a cluster doesn't depend on the cluster size ! It takes the same time for installing on 2 or 50 nodes ! So just have an switched network (very common this days). Urpmi could be compared with apt except for the parallel side ! The autoconfiguration tool, allow people to build up a full cluster from out-of-the-box computers in a really few time (when hardware is set, of course :D). The cluster server builds everything for you just by answering few questions. It builds up automaticaly : DNS, DHCP, PXE, NIS, NFS, MPICH, LAM/MPI, TFTP, NTP,PXE, GANGLIA ... services for you. The last strong point about MDKC is this is a real linux distributions made for the cluster. This is not just a MandrakeLinux + few packages, this is a real cluster distro. Everything from installation to configuration has been thought for clustering. Everything has been build on the same base so the product is really stable, and you install everything at the same time (linux + clustering apps + configuration + tips for making the life easier.) Usual sysadmin clustering tools are included in MDKC : ka tools, gexec, pcp, mput, rshp, dsh, etc.. There is multiple ways for doing the things... If a very common tool is forgot, tell us we'll do our best to integrate it on the next release. Linuxely yours, -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Tue Sep 23 19:10:30 2003 From: csamuel at vpac.org (Chris Samuel) Date: Wed, 24 Sep 2003 09:10:30 +1000 Subject: rsh In-Reply-To: <20030923024026.GE4479@netmeister.org> References: <20030917024414.M10506@ug.ms.wits.ac.za> <200309231148.56402.csamuel@vpac.org> <20030923024026.GE4479@netmeister.org> Message-ID: <200309240910.41797.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 23 Sep 2003 12:40 pm, Jan Schaumann wrote: > Why? IMHO rsh/rlogin are inherently insecure (vulnerable to spoofing, password sniffing). We only permit interactive access to our cluster via SSH. > Most likely, ssh will introduce a non-negligible overhead in encryption > and/or compression. Certainly in our environment (Linux, PBS, MAUI, MPICH/GM) the major use of ssh internal to the cluster is as a replacement for rsh for starting MPI jobs on the various nodes. The impact of any overhead is minor given that the data will be shuffled around over Myrinet without using IP. Compared with the overhead of submitting a job into PBS and having it scheduled by MAUI, any extra latency due to SSH disappears into the noise. :-) Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/cNLuO2KABBYQAh8RAt7XAJ9l/ZKn3KTjG88D2bF494NdkXt6jQCfXrun JjnQOsLqOHvfV802lppF0I0= =qfnI -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bioinformaticist at mn.rr.com Tue Sep 23 15:49:20 2003 From: bioinformaticist at mn.rr.com (Eric R Johnson) Date: Tue, 23 Sep 2003 14:49:20 -0500 Subject: Ramdisk Size In-Reply-To: <3F6F32F0.1070400@rri.sari.ac.uk> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> Message-ID: <3F70A3C0.70601@mn.rr.com> Tony, Thanks for the advice. Let me add a little more information and get your advice about the ramdisk/tmpfs question. Without going into a lot of boring details, the goal of this project is to create a cluster of workstations whose sole purpose is to constantly search the entire contents of a single database over and over again in as short a period of time as possible. That would be it's sole purpose for existing. The database we will be searching is small enough to fit into RAM. As it is a growing database, it may break the 4 Gig barrier, but that would simply require switching to something like an Opteron (or other 64 bit) based cluster. Also, the database only changes every month, or so, and can be downloaded as a whole new database. So, in summary, this is basically a static database, which is searched, it it's entirety, every time, and must be completed in as short a period of time as possible on a system dedicated to this task. Now that you have a little more info, do you still recommend tmpfs? If so, is there a specific performance issue that leads us in that direction, because the search time is all that matters in this case. Thanks again for the advice, Eric Tony Travis wrote: > Eric R Johnson wrote: > >> I would like to create a large ramdisk to keep a database which will >> be searched often, but not changed. >> The only thing I currently know how to do to create a ramdisk is: >> mke2fs /dev/ram0 >> mount /dev/ram0 /tmp/ramdisk0 >> >> The only problem is that this creates a 4MB ramdisk, and I would like >> something on the order of 1GB. > > > Hello, Eric. > > Are you *sure* you want to do this? > > RAM disks are usually used for loading modules at boot time. That's > why they are small. Typically, the contents of a RAM disk are used as > a root filesystem for the kernel to load e.g. SCSI drivers, then > discarded when the 'real' root filesystem is mounted from SCSI disks. > People also use them to boot a 'minimal' Linux for diagnostics or for > some other reason. > > For example, you could create a 10Mb initrd image of a RAM disk using: > > dd if=/dev/zero of=/var/tmp/initrd.img bs=1024 count=10240 > mkfs -t ext3 /tftpboot/initrd.img > mkdir /mnt/ramdisk > mount -o loop /tftpboot/initrd.img /mnt/ramdisk > cp -a /stuff /mnt/ramdisk # stuff you want on RAM disk > umount /var/tmp/initrd.img > zcat /tftpboot/initrd.img > /tftpboot/zinitrd.img > > Then, override the 4Mb default at boot time: > > boot: vmlinuz ramdisk_size=10485760 initrd=zinitrd.img > > However, I think it would be better to use 'tmpfs' for a 1Gb RAM > filesystem. The kernel will provide fast access to the most frequently > accessed areas of your database held in RAM, but it will move the less > frequently accessed areas onto the swap device freeing up RAM for your > programs and for other disk caches. > > Mounting /tmp as a 'tmpfs' filesystem reserves half of your memory for > the /tmp filesystem by default. Add an entry like this to /etc/fstab: > > tmpfs /tmp tmpfs defaults 0 0 > > Beware that openMosix, in particular, does not work well with tmpfs > and automatically comments out tmpfs entries from your /etc/fstab when > you install openMosic from the 2.4.21-openmosix1 binary rpm distribution. > > Tony. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 23 20:32:03 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 23 Sep 2003 20:32:03 -0400 (EDT) Subject: rsh In-Reply-To: <20030923024026.GE4479@netmeister.org> Message-ID: On Mon, 22 Sep 2003, Jan Schaumann wrote: > Chris Samuel wrote: > > > On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > > > > > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > > > just realised that I cannot communicate with other hosts due to some rsh > > > setting that I need to do. > > > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > > instead. > > Why? Most likely, ssh will introduce a non-negligible overhead in > encryption and/or compression. I'd actually say that it will introduce a negligible overhead in both those dimensions for most usage. As in a few seconds (extra) to launch quite a few tasks. Periodically I measure and post the numbers but can never remember what they are the next time this issue rolls around. However, they aren't horribly large in absolute terms, let alone relative ones, and are likely to be cumulatively signficant only on very large clusters for most tasks. With that said, for some usage patterns (spawning lots of very short tasks) they might impact signficantly even on small clusters (yes, the mandatory YMMV:-) , but for the more common spawning of lots of tasks intended to run >>a signficant amount of time<< (e.g. a day), the fraction of a second per task marginal cost associated with startup via ssh vs rsh is indeed negligible. Even tasks where this isn't the case can usually be engineered so that it is (and should be). If ssh isn't adequately efficient, chances are good that rsh is a (smaller but still significant) bottleneck too, since the latency differential is only about a factor of two or three IIRC, and should be scaled away by altering your task organization so that the parallel task runs a "long" time compared to startup time for any remote shell. The issue can also be avoided (as Josip notes) by using LAM or PVM, which spawn a daemon via ssh but subsequently start tasks without any shell-level overhead at all. Overall, given my sysadmin background, I tend to really dislike the idea of keeping rsh around at all and expect that one day it will be deprecated and just go away (in many environments this is true already). It is obsolete, insecure, and fairly stupid when it comes to managing the shell environment. ssh has a lot of advantages compared to rsh aside from its security. I personally wish they had left in the option to turn off encryption (AS an option) which one used to be able to do just to reduce the overhead still further on clusters while retaining its other advantages and features, but not unreasonably the ssh people decided not to so that foolish users couldn't turn it off by accident or design. In many environments, the time wasted by ONE successful password crack due to snooping is far, far greater than any number of rsh margins. rgb > > To the OP: check your /etc/hosts.equiv and ~/.rhosts, as suggested by > rsh(1) and rlogin(1). > > -Jan > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xyzzy at speakeasy.org Tue Sep 23 22:03:52 2003 From: xyzzy at speakeasy.org (Trent Piepho) Date: Tue, 23 Sep 2003 19:03:52 -0700 (PDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Robert G. Brown wrote: > quite a few tasks. Periodically I measure and post the numbers but can > never remember what they are the next time this issue rolls around. time for i in `seq 1 10` ; do rsh venturi4 echo ; done real 0m0.320s user 0m0.000s sys 0m0.050s time for i in `seq 1 10` ; do ssh venturi4 echo ; done real 0m4.014s user 0m0.820s sys 0m0.030s About one order of magnitude slower, and a great deal more cpu time. But that's not the problem I have with ssh vs rsh, it's copying large files. Just last week I needed to copy 65GB of data from an ext3 formatted IDE harddrive someone brought in. The only linux system that I could bring down to stick the drive in was my VIA-C3 based thin-client. Using scp to copy the files would have taken all day. If I could just turn off the stream encryption when I don't need it, everything would be fine. Too bad the security-nazis removed that option. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 23 22:41:44 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 23 Sep 2003 22:41:44 -0400 (EDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Trent Piepho wrote: > On Tue, 23 Sep 2003, Robert G. Brown wrote: > > quite a few tasks. Periodically I measure and post the numbers but can > > never remember what they are the next time this issue rolls around. > > time for i in `seq 1 10` ; do rsh venturi4 echo ; done > real 0m0.320s > user 0m0.000s > sys 0m0.050s > > time for i in `seq 1 10` ; do ssh venturi4 echo ; done > real 0m4.014s > user 0m0.820s > sys 0m0.030s > > About one order of magnitude slower, and a great deal more cpu time. But still pretty much negligible, on a job intended to run all day. > But that's not the problem I have with ssh vs rsh, it's copying large files. > > Just last week I needed to copy 65GB of data from an ext3 formatted IDE > harddrive someone brought in. The only linux system that I could bring down > to stick the drive in was my VIA-C3 based thin-client. Using scp to copy the > files would have taken all day. If I could just turn off the stream > encryption when I don't need it, everything would be fine. Too bad the > security-nazis removed that option. That's fair enough. Once upon a time you could, and I agree that the option should still exist. I once upon a time looked briefly at the ssh source to see if it was easy to short-circuit, but the sources are fairly involved and complicated and I couldn't see any easy place to insert a null function call (without spending a lot more time than it would save). After all, a dummy user probably won't know enough to read the man page anyway and select no encryption (especially if you make it an argument that HAS to be entered as a command line option). Anybody smart enough to read it ought to be smart enough to figure out when to use it. And they can even leave in all the encrypted handshaking and authentication required to establish the connection (which is actually a big chunk of the time you measure IIRC -- the encryption is relatively fast) and then just transfer the data quickly. I suppose in the case of the 65 GB of data you could just NFS export, mount, and copy (depending on just how thin the client was). NFS isn't lightning fast, but one can move data through it at a respectable rate (less than all day). And yeah, enabling rsh/rcp long enough to facilitate the transfer might be easier/faster. rgb > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xyzzy at speakeasy.org Wed Sep 24 00:23:50 2003 From: xyzzy at speakeasy.org (Trent Piepho) Date: Tue, 23 Sep 2003 21:23:50 -0700 (PDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Robert G. Brown wrote: > > > > About one order of magnitude slower, and a great deal more cpu time. > > But still pretty much negligible, on a job intended to run all day. True, but what about interactive tasks? Say I want to run uptime or netstat or lilo, or some other simple command on all the nodes of a 100 node cluster. Doing it via rsh would take 3 seconds, via ssh 40. That's a big difference when you are waiting for the results. > I suppose in the case of the 65 GB of data you could just NFS export, > mount, and copy (depending on just how thin the client was). NFS isn't > lightning fast, but one can move data through it at a respectable rate Exactly what I did. My thin client has a hard drive with a RH9 install, since I used it to develop the system used on the real thin clients, that only have a 32MB flash module. > (less than all day). And yeah, enabling rsh/rcp long enough to > facilitate the transfer might be easier/faster. I tried that first, but rcp barfed on the files larger than 2GB! _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jschauma at netmeister.org Tue Sep 23 22:26:16 2003 From: jschauma at netmeister.org (Jan Schaumann) Date: Tue, 23 Sep 2003 22:26:16 -0400 Subject: rsh In-Reply-To: References: <20030923024026.GE4479@netmeister.org> Message-ID: <20030924022616.GB15042@netmeister.org> "Robert G. Brown" wrote: > On Mon, 22 Sep 2003, Jan Schaumann wrote: > > > Chris Samuel wrote: > > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > > > instead. > > > > Why? Most likely, ssh will introduce a non-negligible overhead in > > encryption and/or compression. > for the more common spawning of lots of tasks intended to run >>a > signficant amount of time<< (e.g. a day), the fraction of a second per > task marginal cost associated with startup via ssh vs rsh is indeed > negligible. Fair argument. > Overall, given my sysadmin background, I tend to really dislike the idea > of keeping rsh around at all and expect that one day it will be > deprecated and just go away (in many environments this is true already). See, and given my sysadmin background, I tend to really disklike the idea of obsoleting a perfectly suitable tool just b/c under certain (different) circumstances it is not suitable. :-) -Jan -- "Ford," he said, "you're turning into a penguin. Stop it." -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From john.hearns at clustervision.com Wed Sep 24 02:45:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Wed, 24 Sep 2003 08:45:11 +0200 (CEST) Subject: A question of OS!! In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on I think this is an important question we should discuss - in the light of the Fedora project with Redhat. I've used both SuSE and Redhat, and like both. In fact, we use SuSE on Opterons, as SuSE were there first with a complete distro, which is very good. I fail to see though why SuSE would have an advantage in parallel computations. Maybe your colleague was trying distros with wildly different kernel versions? In fact, this brings up my pet peeve. As someone else has said in this thread - Linux is Linux. IMHO, we should refer to Linux distributions. The OS itself is Linux (or GNU/Linux). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Andrew.Cannon at nnc.co.uk Wed Sep 24 04:58:10 2003 From: Andrew.Cannon at nnc.co.uk (Cannon, Andrew) Date: Wed, 24 Sep 2003 09:58:10 +0100 Subject: A question of OS!! Message-ID: So which of the mainstream distros are best for Beowulfs? RH, SuSe, Mandrake etc? We use RH8, but I am unsure about whether we should still use this or go to another distro. Advice would be useful. Andrew -----Original Message----- From: John Hearns [mailto:john.hearns at clustervision.com] Sent: Wednesday, September 24, 2003 7:45 AM To: Sarat C Maruvada Cc: beowulf at beowulf.org Subject: Re: A question of OS!! On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on I think this is an important question we should discuss - in the light of the Fedora project with Redhat. I've used both SuSE and Redhat, and like both. In fact, we use SuSE on Opterons, as SuSE were there first with a complete distro, which is very good. I fail to see though why SuSE would have an advantage in parallel computations. Maybe your colleague was trying distros with wildly different kernel versions? In fact, this brings up my pet peeve. As someone else has said in this thread - Linux is Linux. IMHO, we should refer to Linux distributions. The OS itself is Linux (or GNU/Linux). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf NNC's UK Operating Companies : NNC Holdings Limited (no. 3725076), NNC Limited (no. 1120437), National Nuclear Corporation Limited (no. 2290928), STATS-NNC Limited (no. 4339062) and Technica-NNC Limited (no. 235856). The registered office of each company is at Booths Hall, Chelford Road, Knutsford, Cheshire WA16 8QZ except for Technica-NNC Limited whose registered office is at 6 Union Row, Aberdeen AB10 1DQ. This email and any files transmitted with it have been sent to you by the relevant UK operating company and are confidential and intended for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the NNC system manager by e-mail at eadm at nnc.co.uk. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Daniel.Pfenniger at obs.unige.ch Wed Sep 24 04:53:54 2003 From: Daniel.Pfenniger at obs.unige.ch (Daniel.Pfenniger at obs.unige.ch) Date: Wed, 24 Sep 2003 10:53:54 +0200 Subject: rsh In-Reply-To: <200309240910.41797.csamuel@vpac.org> References: <20030917024414.M10506@ug.ms.wits.ac.za> <200309231148.56402.csamuel@vpac.org> <20030923024026.GE4479@netmeister.org> <200309240910.41797.csamuel@vpac.org> Message-ID: <1064393634.3f715ba26726d@webmail.obs.unige.ch> Selon Chris Samuel : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tue, 23 Sep 2003 12:40 pm, Jan Schaumann wrote: > > > Why? > > IMHO rsh/rlogin are inherently insecure (vulnerable to spoofing, password > sniffing). We only permit interactive access to our cluster via SSH. > What we do is (with xinetd) to block rsh from acces outside the cluster and permit rsh only within the cluster. Of course the cluster users must be trusted. ssh does introduce less interactivity for global interactive commands. > > Most likely, ssh will introduce a non-negligible overhead in encryption > > and/or compression. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lange at informatik.Uni-Koeln.DE Wed Sep 24 06:20:27 2003 From: lange at informatik.Uni-Koeln.DE (Thomas Lange) Date: Wed, 24 Sep 2003 12:20:27 +0200 Subject: Cluster distributions? In-Reply-To: <1064217163.15240.6.camel@revolution.mandrakesoft.com> References: <20030918164021.1d2168f3.rburns@cs.ucsb.edu> <1064217163.15240.6.camel@revolution.mandrakesoft.com> Message-ID: <16241.28651.375146.363915@informatik.uni-koeln.de> > Ryan Burns said: >> Hello, I've been running a very small 4 node beowulf cluster >> for a project. Right now I'm running Debian on the cluster and >> I'm not very happy with it or the tools available through the >> distribution. Does anyone have any suggestions of a good Did you looked at FAI, the Fully Automatic Installation for Debian. I set up several Beowulf cluster using this tool. On the FAI web page, there are links to other cluster installations that uses FAI with great success. http://www.informatik.uni-koeln.de/fai/ -- Gruss Thomas _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ajt at rri.sari.ac.uk Wed Sep 24 06:49:46 2003 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed, 24 Sep 2003 11:49:46 +0100 Subject: Ramdisk Size In-Reply-To: <3F70A3C0.70601@mn.rr.com> References: <3F6E90DB.1060505@mn.rr.com> <3F6F32F0.1070400@rri.sari.ac.uk> <3F70A3C0.70601@mn.rr.com> Message-ID: <3F7176CA.6070005@rri.sari.ac.uk> Eric R Johnson wrote: > [...] > as possible on a system dedicated to this task. Now that you have a > little more info, do you still recommend tmpfs? If so, is there a > specific performance issue that leads us in that direction, because the > search time is all that matters in this case. Hello, Eric. On the basis of what you've told us, I think you would be better off avoiding the filesystem overhead altogether and read your database directly into memory within a program. Accessing RAM directly by memory references in a program is as fast as you can get. Anything else you do will incur an OS filesystem call overhead. This may not be practical, of course, but it would give you the fastest solution. One-step (hash) lookups of whatever your database contains directly from RAM is fastest. My basic point about ramdisk/tmpfs is that I think you would be better off allowing the memory manager to keep your most recently accessed data in memory for you. However, if your entire job, including the database, fits into physical memory (RAM) then no swapping to disk will occur with tmpfs. I think you really should be looking for a solution that gives you the best performance improvement for the least effort - i.e. tmpfs. Forgive me if I'm teaching you to suck eggs, but if search time is all that matters then your search algorithm is important. There is little point accessing your database as fast as possible using a poor search algorithm: Indeed, you may find that your concern about 'fast' access to your data on ramdisk/tmpfs is not the main bottleneck after all. You could benchmark the database problem on harddisk and tmpfs to find out. Tony. -- Dr. A.J.Travis, | mailto:ajt at rri.sari.ac.uk Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 24 07:08:43 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 24 Sep 2003 07:08:43 -0400 (EDT) Subject: rsh In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Trent Piepho wrote: > On Tue, 23 Sep 2003, Robert G. Brown wrote: > > > > > > About one order of magnitude slower, and a great deal more cpu time. > > > > But still pretty much negligible, on a job intended to run all day. > > True, but what about interactive tasks? Say I want to run uptime or netstat > or lilo, or some other simple command on all the nodes of a 100 node cluster. > Doing it via rsh would take 3 seconds, via ssh 40. That's a big difference > when you are waiting for the results. Ah, but that's why I wrote xmlsysd and wulfstat (or more practically, why Erik Hendriks et al wrote bproc;-) Interactive task distribution and monitoring is NEVER very efficient or lightweight when it runs through an instantiated shell. And sure, as I said, if you use ssh to distribute lots of small tasks and want immediate results it isn't very good. Mostly because of the handshaking, I think -- I doubt that this is an encryption issue. Here is where one has to do the cost-benefit tradeoffs. Duke more or less prohibits the use of rsh on systems exposed to any sort of public network because the cost of ANY sort of cracking incident is very high, and very real. On a firewalled cluster of course one could install it, but I've seen cracking and data theft incidents at Duke WITHIN RESEARCH GROUPS (one e.g. postdoc snooping wire data and root password, stealing files, apparently exporting said files to friends back in Bulgaria in his particular case to search for value). One could argue that if it is for use on YOUR cluster, architected inside a protected network as a "true beowulf" and with only one user, you can do what you like (and who could argue:-) -- from what Erik has told me in years past, bproc isn't secure on the inside either, but the presumption there is that "nodes" have nothing of value but immediate data and the only point of access is the head node (these are guys who think rsh is -- correctly -- heaviweight:-). In any sort of environment with trust boundaries or valuable data, though, or where one might wish to export environment variables along with the shell processes, using rsh enables those boundaries to be fairly trivially violated by anyone with root/promiscuous mode access to the wire itself (via e.g. a handy laptop). ssh in that sort of environment is just a form of due diligence. sshd, too, is subjected to intense scrutiny looking for buffer overwrites and exploits, while rshd is not under the presumption perhaps that it is OBVIOUSLY not secure so why bother? At Duke we have made the decision to use ssh only, because of the security issues, and use tools like xmlsysd to do the few tasks that we might otherwise like to distribute in real time with a remote shell. At other institutions one might well choose otherwise. YMMV, and we live in a world of marvelous choices. > > (less than all day). And yeah, enabling rsh/rcp long enough to > > facilitate the transfer might be easier/faster. > > I tried that first, but rcp barfed on the files larger than 2GB! Hmm, sounds like an int counter in there somewhere...;-) Seriously, one thing that might be worth TRYING is to request that the openssh people restore a "none" encryption option on ssh/sshd, enabled by /etc/ssh/*.config options so that local admins can disable it on open networks. I can't see why this would be objectionable to them, and it would solve at least this half of the problem. rgb > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From timm at fnal.gov Wed Sep 24 07:18:56 2003 From: timm at fnal.gov (Steven Timm) Date: Wed, 24 Sep 2003 06:18:56 -0500 Subject: rsh In-Reply-To: Message-ID: Nobody has yet mentioned in this thread that there is an option to run a kerberos-enabled rsh and do kerberos authentication, and even DES encryption of your session traffic and data stream. Obviously it's a bit slower than ssh but it is quite secure and has solved our problems quite well. Steve Timm ------------------------------------------------------------------ Steven C. Timm (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division/Core Support Services Dept. Assistant Group Leader, Scientific Computing Support Group Lead of Computing Farms Team On Tue, 23 Sep 2003, Robert G. Brown wrote: > On Mon, 22 Sep 2003, Jan Schaumann wrote: > > > Chris Samuel wrote: > > > > > On Wed, 17 Sep 2003 12:51 pm, masego-otlhe_segone wrote: > > > > > > > I have managed to install mpich v1.2.5.2 on my cluster of two nodes, but I > > > > just realised that I cannot communicate with other hosts due to some rsh > > > > setting that I need to do. > > > > > > To be honest I'd really suggest not using RSH and swapping over to OpenSSH > > > instead. > > > > Why? Most likely, ssh will introduce a non-negligible overhead in > > encryption and/or compression. > > I'd actually say that it will introduce a negligible overhead in both > those dimensions for most usage. As in a few seconds (extra) to launch > quite a few tasks. Periodically I measure and post the numbers but can > never remember what they are the next time this issue rolls around. > However, they aren't horribly large in absolute terms, let alone > relative ones, and are likely to be cumulatively signficant only on very > large clusters for most tasks. > > With that said, for some usage patterns (spawning lots of very short > tasks) they might impact signficantly even on small clusters (yes, the > mandatory YMMV:-) , but for the more common spawning of lots of tasks > intended to run >>a signficant amount of time<< (e.g. a day), the > fraction of a second per task marginal cost associated with startup via > ssh vs rsh is indeed negligible. Even tasks where this isn't the case > can usually be engineered so that it is (and should be). If ssh isn't > adequately efficient, chances are good that rsh is a (smaller but still > significant) bottleneck too, since the latency differential is only > about a factor of two or three IIRC, and should be scaled away by > altering your task organization so that the parallel task runs a "long" > time compared to startup time for any remote shell. > > The issue can also be avoided (as Josip notes) by using LAM or PVM, > which spawn a daemon via ssh but subsequently start tasks without any > shell-level overhead at all. > > Overall, given my sysadmin background, I tend to really dislike the idea > of keeping rsh around at all and expect that one day it will be > deprecated and just go away (in many environments this is true already). > It is obsolete, insecure, and fairly stupid when it comes to managing > the shell environment. ssh has a lot of advantages compared to rsh > aside from its security. I personally wish they had left in the option > to turn off encryption (AS an option) which one used to be able to do > just to reduce the overhead still further on clusters while retaining > its other advantages and features, but not unreasonably the ssh people > decided not to so that foolish users couldn't turn it off by accident or > design. > > In many environments, the time wasted by ONE successful password crack > due to snooping is far, far greater than any number of rsh margins. > > rgb > > > > > To the OP: check your /etc/hosts.equiv and ~/.rhosts, as suggested by > > rsh(1) and rlogin(1). > > > > -Jan > > > > > > -- > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From r.young at irl.cri.nz Wed Sep 24 08:21:39 2003 From: r.young at irl.cri.nz (Roger Young) Date: Thu, 25 Sep 2003 00:21:39 +1200 Subject: Question about mpich libraries Message-ID: <20030925002139.2a283dc9.r.young@irl.cri.nz> I have recently installed mpich v.1.2.5.2 under Slackware Linux 9.0 with Linux kernel 2.4.22 and glibc 2.3.1 and compilers gcc 3.2.2 & lf95 v.5.5. My mpich config options were ./configure -without-romio -disable-cc++ -fc=lf95 -f90=lf95 --prefix=/usr/local/mpich However the application (geofem) will not compile because it cannot find lmpichf90nc which is meant to be in mpich/lib (but is not there). The error message is "usr/bin/ld: cannot find -lmpichf90rc" Can somebody tell me how I might generate/obtain this library? Thanks for your help, Roger Young Industrial Research Ltd PO Box 31-310 Lowe Hutt New Zealand r.young at irl.cri.nz -- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Wed Sep 24 09:24:50 2003 From: landman at scalableinformatics.com (Joseph Landman) Date: Wed, 24 Sep 2003 09:24:50 -0400 Subject: A question of OS!! In-Reply-To: References: Message-ID: <1064409890.29084.7.camel@protein.scalableinformatics.com> On Tue, 2003-09-23 at 16:28, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > has advantage in parallel computations. Having no hands-on experience in I have not heard this. If anyone has, could they provide references? > SuSe Linux, I have no idea how to verify this or do a comparision. What do > you advice as the OS for the cluster? And how to determine this,as there > are a bunch of unix OSes out there.Thanks. > > Sincerely, > Sarat. You have options. Cluster distributions (where they do lots of the hard work for you), or "normal" distributions, where you have to do the hard work. You have the cluster distributions. lcic.org has a list. I have a short list (with links) of them on http://scalableinformatics.com/rocks/, if you search down in the page for Cluster Distributions. You have normal distributions (Redhat/Fedora, Suse, Knoppix, Gentoo, Debian, etc.) Most will require significant additional work. ClusterKnoppix may alleviate some of this. I generally advise matching the distribution to your level of skill/interest in maintaining a distribution/cluster. The cluster distributions handle lots of the pain for you, though some have odd/poorly integrated management philosophies, others have great ideas with poor implementations. Some are good. -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Wed Sep 24 11:34:43 2003 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 24 Sep 2003 08:34:43 -0700 Subject: more cashing in on SCO/Linux FUD? Message-ID: <007601c382b1$614044d0$35a8a8c0@laptop152422> I've been looking at other potential operating systems for embedded clusters, and QNX is one of the options. I just received some (quasi-solicted) commercial email from them including the following interesting paragraph: In light of recent market developments around IP ownership, embedded systems manufacturers are researching OS alternatives to Linux. Developers and technical managers need to determine how software projects implemented in Linux might be transferred to other OSs - what portion of the original code can be preserved, whether similar functionality can be implemented, and how migration can be achieved. Join QNX Software Systems on September 29 for a technical web seminar, "Implementing Device Drivers - Migrating from Linux to a Microkernel OS," where David Donohoe, senior software developer, QNX Software Systems, will demonstrate how an existing Linux- developed driver can be migrated to the QNX(r) Neutrino(r) RTOS, preserving up to 80 per cent of the original software code development. While I think it's a fine thing that QNX is providing alternate solutions for hard-real-time apps, and that they're helping folks understand the difference between Linux and QNX drivers, I find the first statement a bit offensive. Jim Lux _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sherincyriac at hotmail.com Wed Sep 24 11:46:32 2003 From: sherincyriac at hotmail.com (sherin cyriac) Date: Wed, 24 Sep 2003 21:16:32 +0530 Subject: implementation details of Beowulf Message-ID: sir I am going to do a project in Beowulf cluster. I want to know some unsolved problems in this domain .I also need the details of Beowulf cluster that was practically implemented. I need the implementation details. thanking you your's faithfully sherin _________________________________________________________________ MSN Hotmail now on your Mobile phone. http://server1.msn.co.in/sp03/mobilesms/ Click here. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sherincyriac at hotmail.com Wed Sep 24 11:46:09 2003 From: sherincyriac at hotmail.com (sherin cyriac) Date: Wed, 24 Sep 2003 21:16:09 +0530 Subject: (no subject) Message-ID: sir I am going to do a project in Beowulf cluster. I want to know some unsolved problems in this domain .I also need the details of Beowulf cluster that was practically implemented. I need the implementation details. thanking you your's faithfully sherin _________________________________________________________________ Get personal loans. It's hassle-free. http://server1.msn.co.in/msnleads/citibankpersonalloan/citibankploanjuly03.asp?type=txt It's approved instantly. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csarat1 at uic.edu Wed Sep 24 11:25:24 2003 From: csarat1 at uic.edu (Sarat C Maruvada) Date: Wed, 24 Sep 2003 10:25:24 -0500 (CDT) Subject: A question of OS!! In-Reply-To: <3F70C82F.2070307@cert.ucr.edu> References: <3F70C82F.2070307@cert.ucr.edu> Message-ID: Thanks to all of you for your replies. I will definetely keep in mind the points you have mentioned before making a choice for the OS.Thank you once again... Sincerely, Sarat. ************************************************************** Sarat Chandra Maruvada 830,S.Claremont Ave., Research Assistant, CS Dept., #2, Prof. Florin Balasa, Chicago, SEO 937, IL - 60612. UIC. Ph: 312-850-9711. ************************************************************** On Tue, 23 Sep 2003, Glen Kaukola wrote: > Sarat C Maruvada wrote: > > >Hi,I am planning on installing a 32 node beowulf cluster for high > >performance bio-informatics applications. As a OS I was planning on > >installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > >has advantage in parallel computations. Having no hands-on experience in > >SuSe Linux, I have no idea how to verify this or do a comparision. What do > >you advice as the OS for the cluster? And how to determine this,as there > >are a bunch of unix OSes out there.Thanks. > > > > One advantage I can think of that Suse might have is it's automated > installation utility, autoyast I believe it's called. While it's > probably a little harder to set up, I think it would be a bit better > than kickstart for installing Linux on a large number of machines. > Actually though, some of the functionality of autoyast seemed to be > broken to me, but again it's a lot more complicated than kickstart, so > maybe it was just me. You don't have to pay to use Suse's automatic > update tool, but I think that this may change on Red Hat now that > they've merged with Fedora. The only other thing I can think of is that > the Suse distribution seems to come with about twice as many packages to > choose from. Other than that, Linux is Linux, and you're not going to > see much of a speed difference between either Suse or Red Hat. > > My two cents, > Glen > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Wed Sep 24 12:25:57 2003 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 24 Sep 2003 18:25:57 +0200 Subject: rsh In-Reply-To: References: <20030923024026.GE4479@netmeister.org> Message-ID: <20030924162557.GC2124@unthought.net> On Tue, Sep 23, 2003 at 08:32:03PM -0400, Robert G. Brown wrote: ... > The issue can also be avoided (as Josip notes) by using LAM or PVM, > which spawn a daemon via ssh but subsequently start tasks without any > shell-level overhead at all. A fair guess would be, that this connection/communication is not encrypted or strongly authenticated in any way. The resulting security benefit of SSH being null and void. There can be other good reasons for using SSH, probably ease of use if you're used to it, X forwarding, and other niceties. My point here is, that the chain is no stronger than the weakes link. ... > > In many environments, the time wasted by ONE successful password crack > due to snooping is far, far greater than any number of rsh margins. You don't use NFS? If you do, I can put anything in your .bashrc anyway. I use SSH for anything with an 'outside' connection. Typically, SSH will even be firewalled so that only a few select machines can connect to even that service. For internal systems, I am fully aware that since I run NFS, NIS, and some clustering services anyway, running SSH would buy me *zero* security. I treat the network as the computer. Any outside link is firewalled and SSH'ed as appropriate, but internally 'inside the network computer', I have just as much encryption as you have between the PCI slots in your computer. -- ................................................................ : jakob at unthought.net : And I see the elder races, : :.........................: putrid forms of man : : Jakob ?stergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csarat1 at uic.edu Wed Sep 24 12:51:58 2003 From: csarat1 at uic.edu (Sarat C Maruvada) Date: Wed, 24 Sep 2003 11:51:58 -0500 (CDT) Subject: A question of OS!! In-Reply-To: References: Message-ID: Hi, After going through the mails, I have noticed that most of SuSe "advantages" are specific to AMD Opteron processors,mainly because of 64 bit support. Actually the specification of the cluster uses an Athlon XP 2600+. Also since some of you have mentioned,I think it is also prudent to look into cluster distributions (like ROCK for example)..And clearly waiting for RHEL seems like a very good option.Once thing is clear though,no RH 9.0 :-).... Thanks once again. Sincerely, Sarat. ************************************************************** Sarat Chandra Maruvada 830,S.Claremont Ave., Research Assistant, CS Dept., #2, Prof. Florin Balasa, Chicago, SEO 937, IL - 60612. UIC. Ph: 312-850-9711. ************************************************************** On Wed, 24 Sep 2003, Rocky McGaugh wrote: > On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > > > Hi,I am planning on installing a 32 node beowulf cluster for high > > performance bio-informatics applications. As a OS I was planning on > > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > > has advantage in parallel computations. Having no hands-on experience in > > SuSe Linux, I have no idea how to verify this or do a comparision. What do > > you advice as the OS for the cluster? And how to determine this,as there > > are a bunch of unix OSes out there.Thanks. > > > > Sincerely, > > Sarat. > > > > First, I would strongly recommend against using RH9 in a production > environment. It is a short-life-cycle product meant for hobbyist desktops. > More than that, it was a way to beta test some pretty radical OS changes > on the masses before their inclusion into RHEL3. > > Secondly, if you are going to use Opteron there are very good reasons to > use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports > the Opteron while RH9 will only run in 32-bit mode. SLES also has a > 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured > NUMA support for the Opteron. > > If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 > will be out and has the same benefits as SLES. > > -- > Rocky McGaugh > Atipa Technologies > rocky at atipatechnologies.com > rmcgaugh at atipa.com > 1-785-841-9513 x3110 > http://67.8450073/ > perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 24 13:04:19 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 24 Sep 2003 13:04:19 -0400 (EDT) Subject: implementation details of Beowulf In-Reply-To: Message-ID: On Wed, 24 Sep 2003, sherin cyriac wrote: > sir > I am going to do a project in Beowulf cluster. I want to know some > unsolved problems in this domain > .I also need the details of Beowulf cluster that was practically > implemented. I need the implementation details. Start with: http://www.beowulf.org and proceed with http://www.phy.duke.edu/brahma (and links thereon). There are LOTS of web resources out there, and many of those resources are linked back to one of these sites or to sites linked to one of these sites (e.g. beowulf underground, MPI and PVM and Scyld and more). HTH, rgb > thanking you > your's faithfully > sherin > > _________________________________________________________________ > MSN Hotmail now on your Mobile phone. > http://server1.msn.co.in/sp03/mobilesms/ Click here. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Wed Sep 24 13:01:52 2003 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 24 Sep 2003 19:01:52 +0200 Subject: more cashing in on SCO/Linux FUD? In-Reply-To: <007601c382b1$614044d0$35a8a8c0@laptop152422> References: <007601c382b1$614044d0$35a8a8c0@laptop152422> Message-ID: <20030924170152.GD2124@unthought.net> On Wed, Sep 24, 2003 at 08:34:43AM -0700, Jim Lux wrote: ... > While I think it's a fine thing that QNX is providing alternate solutions > for hard-real-time apps, and that they're helping folks understand the > difference between Linux and QNX drivers, I find the first statement a bit > offensive. It's marketing. Yesterday we had a 7 hour power outage, which covered eastern Denmark and southern Sweden. The largest outage in 22 years. Today I got a mail from CA wanting to sell me backup software, to keep my data safe in case of a new power outage. Fear and greed is effective on human beings. Marketing's job is to try to be effective on human beings. And we're headed OT fast ;) -- ................................................................ : jakob at unthought.net : And I see the elder races, : :.........................: putrid forms of man : : Jakob ?stergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Sep 24 13:28:27 2003 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 24 Sep 2003 11:28:27 -0600 Subject: rsh In-Reply-To: <20030924162557.GC2124@unthought.net> References: <20030923024026.GE4479@netmeister.org> <20030924162557.GC2124@unthought.net> Message-ID: <3F71D43B.7020106@lanl.gov> Jakob Oestergaard wrote: > On Tue, Sep 23, 2003 at 08:32:03PM -0400, Robert G. Brown wrote: > ... > >>The issue can also be avoided (as Josip notes) by using LAM or PVM, >>which spawn a daemon via ssh but subsequently start tasks without any >>shell-level overhead at all. > > > A fair guess would be, that this connection/communication is not > encrypted or strongly authenticated in any way. Correct. > The resulting security benefit of SSH being null and void. > [...] > For internal systems, I am fully aware that since I run NFS, NIS, and > some clustering services anyway, running SSH would buy me *zero* > security. Yes. Inside a cluster contained in a locked computer room, you've got physical network security, so SSH overhead is not necessary. > I use SSH for anything with an 'outside' connection. Typically, SSH will > even be firewalled so that only a few select machines can connect to > even that service. A wise policy. Outside connections need SSH. However, in most cases you do not want to parallelize across outside connections, because they tend to be slow. Since nobody I know parallelizes across outside connections, SSH simply does not get involved in parallel runs. LAM damons are just fine internally. However, cluster access from the outside is usually restricted to SSH w/X forwarding. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From goutam at infovision21.com Wed Sep 24 14:47:29 2003 From: goutam at infovision21.com (goutam) Date: Wed, 24 Sep 2003 14:47:29 -0400 Subject: Comparisions,, Message-ID: Hello all, I am starting to build a cluster of 8-10 computers (PIII Redhat 2.2.x). I would like comparitive views of Scyld , OSCAR or globus.. Its mainly for Academic purposes but will run a few professional grade parallel programs.. thanks ! goutam _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctibirna at giref.ulaval.ca Wed Sep 24 14:51:52 2003 From: ctibirna at giref.ulaval.ca (Cristian Tibirna) Date: Wed, 24 Sep 2003 14:51:52 -0400 Subject: upgrading rh73 on an xCAT cluster Message-ID: <200309241451.52982.ctibirna@giref.ulaval.ca> Hello Yesterday I upgraded (first time after 7 months... I know, I know) the rh73 rpms and the kernel. Since then, I have two nasty issues: 1) The update installed a new openssh (3.1.p1-14) The auth of sshd through pam is annoyingly slower. All ssh connections (both from outside to the master and from any node to any node inside) _are_ succeeding, but a lot slower. I see this in the /var/log/messages too: Sep 24 13:16:04 n15 sshd(pam_unix)[27164]: authentication failure; logname=\ uid=0 euid=0 tty=NODEVssh ruser= rhost=n01 user=root Sep 24 13:16:04 n15 sshd(pam_unix)[24856]: session opened for user root by\ (uid=0) Both messages are for the same ssh connection attempt and the attempt succeeds, as I said. The only visible effect to the user is the slowness (the first failure is followed by a programmed delay in pam). I looked a bit around the 'net and people have already complained a lot about this problem but I found no solution. 2) I also updated the kernel to 2.4.20-20.7 (redhat rpm). Afterwards, my (and other users') SGE qmake jobs just get stuck in the middle (i.e. function correctly for a while then suddenly just sit there and do nothing for long time, without having completed). I feel it's some sort of NFS lockup problem as the master node (NFS server) gets very high loads (6.0-8.0) compared to before (2.0-3.0) the update of the kernel. The /var/log/messages says nothing useful. Did anybody already updated a rh73 cluster equipped with SGE and using ssh internally? Observed these problems? Found solutions? Thanks a lot. -- Cristian Tibirna (1-418-) 656-2131 / 4340 Laval University - Quebec, CAN ... http://www.giref.ulaval.ca/~ctibirna Research professional at GIREF ... ctibirna at giref.ulaval.ca _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 24 15:59:50 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 24 Sep 2003 12:59:50 -0700 Subject: MPICH vs LAM In-Reply-To: <3F7079B4.4090807@lanl.gov> References: <20030923124408.91196.qmail@web12205.mail.yahoo.com> <3F7079B4.4090807@lanl.gov> Message-ID: <20030924195950.GB2061@greglaptop.internal.keyresearch.com> On Tue, Sep 23, 2003 at 10:49:56AM -0600, Josip Loncaric wrote: > MPICH is very popular and works fine, but LAM is a bit cleaner > implementation which delivers a bit lower latency. Other than that, the > main difference is process startup. With MPICH, "mpirun" does the job > directly and accepts the usual rsh or ssh authorization overhead. mpich now distributes multiple startup mechanisms -- one is similar to the LAM daemon approach. This is a sign that the MPI community is doing a good job of borrowing good ideas from each other. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Wed Sep 24 03:59:19 2003 From: landman at scalableinformatics.com (Joe Landman) Date: Wed, 24 Sep 2003 03:59:19 -0400 Subject: upgrading rh73 on an xCAT cluster In-Reply-To: <200309241451.52982.ctibirna@giref.ulaval.ca> References: <200309241451.52982.ctibirna@giref.ulaval.ca> Message-ID: <1064390359.5989.3.camel@squash.scalableinformatics.com> Hi Cristian: This sounds like a mixture of a few problems, including a name service/host lookup issue for the ssh slowness. The NFS server could be a number of things. Which kernel are you using? How many machines are mounting the NFS system? What does nfsstat say, and what does vmstat indicate? How many nfsd's are running? I have found that with name services (either through NIS, DNS, etc) timing out, ssh gets quite slow. One way to try this is doing an ssh to the ip address of the compute node rather than its name. If the times are quite similar, there may be other issues at work. If the ip address method is much faster, then name resolution is not working somewhere. Joe On Wed, 2003-09-24 at 14:51, Cristian Tibirna wrote: > Hello > > Yesterday I upgraded (first time after 7 months... I know, I know) the rh73 > rpms and the kernel. Since then, I have two nasty issues: > > 1) > The update installed a new openssh (3.1.p1-14) > > The auth of sshd through pam is annoyingly slower. All ssh connections (both > from outside to the master and from any node to any node inside) _are_ > succeeding, but a lot slower. I see this in the /var/log/messages too: > > Sep 24 13:16:04 n15 sshd(pam_unix)[27164]: authentication failure; logname=\ > uid=0 euid=0 tty=NODEVssh ruser= rhost=n01 user=root > Sep 24 13:16:04 n15 sshd(pam_unix)[24856]: session opened for user root by\ > (uid=0) > > Both messages are for the same ssh connection attempt and the attempt > succeeds, as I said. The only visible effect to the user is the slowness (the > first failure is followed by a programmed delay in pam). > > I looked a bit around the 'net and people have already complained a lot about > this problem but I found no solution. > > 2) > I also updated the kernel to 2.4.20-20.7 (redhat rpm). > > Afterwards, my (and other users') SGE qmake jobs just get stuck in the middle > (i.e. function correctly for a while then suddenly just sit there and do > nothing for long time, without having completed). I feel it's some sort of > NFS lockup problem as the master node (NFS server) gets very high loads > (6.0-8.0) compared to before (2.0-3.0) the update of the kernel. The > /var/log/messages says nothing useful. > > > Did anybody already updated a rh73 cluster equipped with SGE and using ssh > internally? Observed these problems? Found solutions? > > Thanks a lot. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Wed Sep 24 17:15:49 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Wed, 24 Sep 2003 14:15:49 -0700 Subject: Redhat Fedora In-Reply-To: References: Message-ID: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> On Tue, Sep 23, 2003 at 10:37:45AM +0200, John Hearns wrote: > As most of us know, Redhat announced that they are pursuing a new > community led development model for future Redhat 'consumer' releases. > I'm sure I'm not the only one who was wondering what that meant for the > direction of cluster computing. It's an interesting question. RedHat's new advice on the matter is "buy a copy of Advanced Workstation for every cluster node." They are also changing how they do updates to the "consumer" OS; for example, I pointed out that xpdf was miscompiled without all the font support in RH 9, but RH did not release a fix, they closed the bug report with "fixed in RH 10." I suspect that we'll see: More use of hobbist distros like Debian People putting together freely-copyable ISOs for RedHat AW and AS More interest in Mandrake's EU-supported cluster distro The first two will be driven by the general user community, not the cluster people; "we" (the cluster community) tend to not do things like build our own distros, nor are "we" willing to pay enough for vendors (like Scyld) to build cluster distros. > And it supports YUM and apt-get, which can only be a good thing. That's a good thing -- much easier than looking around at ftp mirrors for updates. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bob at drzyzgula.org Wed Sep 24 19:44:42 2003 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Wed, 24 Sep 2003 19:44:42 -0400 Subject: Redhat Fedora In-Reply-To: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> Message-ID: <20030924194442.H3752@www2> On Wed, Sep 24, 2003 at 02:15:49PM -0700, Greg Lindahl wrote: > > On Tue, Sep 23, 2003 at 10:37:45AM +0200, John Hearns wrote: > > > As most of us know, Redhat announced that they are pursuing a new > > community led development model for future Redhat 'consumer' releases. > > I'm sure I'm not the only one who was wondering what that meant for the > > direction of cluster computing. > > It's an interesting question. RedHat's new advice on the matter is > "buy a copy of Advanced Workstation for every cluster node." They are > also changing how they do updates to the "consumer" OS; for example, I > pointed out that xpdf was miscompiled without all the font support in > RH 9, but RH did not release a fix, they closed the bug report with > "fixed in RH 10." Heck, that's nothing -- at least they fixed it. I reported a bug in the RH 6.1 implementation of kudzu in October 1999. There was no activity in bugzilla until almost three years later, in June of 2002, and that only amounted to someone asking "is this fixed yet", and getting the answer that it was not. That was the last activity on the bug; the status remains "assigned": https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=6463 > I suspect that we'll see: > > More use of hobbist distros like Debian > People putting together freely-copyable ISOs for RedHat AW and AS > More interest in Mandrake's EU-supported cluster distro I suppose there's always the posibility that the whole Fedora/Redhat merge thing will work out, and it will become a popular non-commercial alternative to Debian. Also, I'm thinking I must not know enough about Mandrake's cluster distribution. Looking at their website, it seems to cost $2320 for 1-8 processors, $4176K for 9-16 processors. Why would a cluster shop that had previously used Red Hat Linux choose to use Mandrake rather than Red Hat's Enterprise WS distribution, which costs $299 per single- or dual-processor system, except perhaps on technical grounds? > The first two will be driven by the general user community, not the > cluster people; "we" (the cluster community) tend to not do things > like build our own distros Actually, I thought that in fact "we" tended to do a lot of this -- cf. Rocks, Clustermatic, Warewulf -- it's just that these never seem to capture the market in the same way that Red Hat always was. > nor are "we" willing to pay enough for > vendors (like Scyld) to build cluster distros. This sounds right. > > And it supports YUM and apt-get, which can only be a good thing. > > That's a good thing -- much easier than looking around at ftp mirrors > for updates. No argument with that. --Bob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Sep 24 20:54:46 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 24 Sep 2003 20:54:46 -0400 (EDT) Subject: Redhat Fedora In-Reply-To: <20030924194442.H3752@www2> Message-ID: On Wed, 24 Sep 2003, Bob Drzyzgula wrote: > > Also, I'm thinking I must not know enough about Mandrake's > cluster distribution. Looking at their website, it > seems to cost $2320 for 1-8 processors, $4176K for 9-16 > processors. Why would a cluster shop that had previously > used Red Hat Linux choose to use Mandrake rather than > Red Hat's Enterprise WS distribution, which costs $299 > per single- or dual-processor system, except perhaps on > technical grounds? The interesting question is why anybody in their right mind would pay for any sort of GPL-based distribution that scales per number of processors. Especially pay numbers like these. Hell, one might as well be running Windows. I personally think that the global internet has long since moved on to a place where this just won't fly, especially for GPL software. It's almost funny, in a very sad sort of way. People all over the world (many of them on this list) have written and contributed the actual software and continue to maintain it. Red Hat is now preparing to turn around and sell our own software back to us at an absurd markup and with little (really) added value other than the distributional packaging and testing. I personally think that what is going to happen is that the community will interpret this as control and just route around it. People on the Internet proper will just ignore the prices; corporate managers who feel a need to "buy" something may pay them. Companies seem to be trying to treat open source software as a proprietary product and moving away from the support model. The real tragedy associated with this is that linux is finally maturing to the point where it could really challenge on the desktop, not just the server/cluster/specialized market, and these absurd prices could retard that process for years. Red Hat in particular has missed the boat by charging crazy prices for box sets for some years. What, they think Joe Consumer is going to buy a box set of linux for $150 when it comes "free" and preinstalled on their PC? They should be selling box sets for $15 and trying to sell a LOT of them, not a few at crazy prices. > > nor are "we" willing to pay enough for > > vendors (like Scyld) to build cluster distros. > > This sounds right. Absolutely. It is a matter of what people need. MPI, PVM, and more are often or even generally available in rpm format and are a whole lot of what people need. The cluster users who need to go a long way beyond this exist, but there are fewer and fewer of them at each additional price notch. COTS HPC clusters were invented because they were a CHEAP road to supercomputing that supported the needs of a hell of a lot of HPC cycle consumers with only a few, relatively simple tools. The same issues exist with Myrinet and SCI -- I'll be there are dozens of clusters installed on plain old ethernet for each one of the clusters installed on high end networks (which, frankly, are worth a lot more than high end/expensive "distributions" since Red Hat cannot alter the fundamentally limiting computation/communications ratio at a given task scale, but a faster network interface can!). So pay no attention to the high road. Distribution vendors that take the high road are going to starve on a rich diet of a few high margin sales, just as a lot of the traditional Unix vendors have been doing at ever increasing rates ever since linux was invented. Look instead to the low road. Look to distributions (like Debian or freebsd) that charge nothing, or charge a very modest low margin fee. In the market of the future one will be able to grow fat on huge numbers of LOW margin sales, just as Microsoft (for the most part) has for years. If Sun Microsystems had released their perfectly good Unix for PC's for a retail price of $15/system in a box set a decade plus ago (or even $50), Linux would never have gotten off the ground, Windows NT would have been killed in its cradle, and Sun would own the universe. I think it must be the scent of blood in the water -- Microsoft is finally showing signs of distinct vulnerability, the linux desktop is finally good enough to sell to corporate customers, and the long awaited phase transition is at hand. Now all the linux distro people can think about is how fast they can get their options and their shareholder's highly depressed shares above water. Slow and steady (and low margin sales and true added value support) will win this race. Red Hat is already facing a sort of defection of a rather huge part of their installed base; how long will they stay on top once the Universities and other institutions that provide their real base of programmers turn their energies to something that they don't have to turn around and "buy back"? With the GPL, not long at all, not long at all. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From glen at cert.ucr.edu Wed Sep 24 21:22:51 2003 From: glen at cert.ucr.edu (Glen Kaukola) Date: Wed, 24 Sep 2003 18:22:51 -0700 Subject: Redhat Fedora In-Reply-To: <20030924194442.H3752@www2> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> Message-ID: <3F72436B.5020609@cert.ucr.edu> Bob Drzyzgula wrote: >Also, I'm thinking I must not know enough about Mandrake's >cluster distribution. Looking at their website, it >seems to cost $2320 for 1-8 processors, $4176K for 9-16 >processors. Why would a cluster shop that had previously >used Red Hat Linux choose to use Mandrake rather than >Red Hat's Enterprise WS distribution, which costs $299 >per single- or dual-processor system, except perhaps on >technical grounds? > Mandrake puts out a completely free cluster distribution called clic: http://clic.mandrakesoft.com/index-en.html Glen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bob at drzyzgula.org Wed Sep 24 23:51:33 2003 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Wed, 24 Sep 2003 23:51:33 -0400 Subject: Redhat Fedora In-Reply-To: References: <20030924194442.H3752@www2> Message-ID: <20030924235133.I3752@www2> On Wed, Sep 24, 2003 at 08:54:46PM -0400, Robert G. Brown wrote: > > On Wed, 24 Sep 2003, Bob Drzyzgula wrote: > > > > > Also, I'm thinking I must not know enough about Mandrake's > > cluster distribution. Looking at their website, it > > seems to cost $2320 for 1-8 processors, $4176K for 9-16 > > processors. Why would a cluster shop that had previously > > used Red Hat Linux choose to use Mandrake rather than > > Red Hat's Enterprise WS distribution, which costs $299 > > per single- or dual-processor system, except perhaps on > > technical grounds? > > The interesting question is why anybody in their right mind would pay > for any sort of GPL-based distribution that scales per number of > processors. Especially pay numbers like these. Hell, one might as well > be running Windows. There are two reasons, actually, that I can think of, although they only apply in specific environments that probably don't include e.g. universities that are strapped for cash but have a near-inexhaustible supply of cheap labor near at hand. These reasons are: (a) The promise of support. This is generally more for managers than for the techies, since in a large proportion of cases, the techies most likely know what they are doing. But managers often don't know enough about the technology to know whether to trust that the techies know what they are doing (and perhaps have occasional real-life experiences with failure in this regard), so they at a minimum want to know that there is someone at the vendor who's *job* it is to pick up the phone and answer *their* questions. This may seem irrational, but it isn't really. (b) The promise of a patch stream. This is the thing that is of more actual use to the techies. Especially with the way that RPM works, maintaining patches for an older distribution gets to be a real burden, and it is quite clear to me why Red Hat is trying to get out from under this, especially for the "consumer" releases. To some extent, it's their own damned fault what with the way that find-requires works to build in dependencies on dot-releases of system libraries, but in any event the availability of tested, binary patches for a three-year-old system is of real value and IMHO is worth paying for. Maybe not thousands of dollars per system, but then if I had millions of dollars of income riding on a system, thousands might not seem like all that big a deal. Again, this all likely seems more attractive in a business setting than in a University setting. > I personally think that the global Internet has long since moved on to > a place where this just won't fly, especially for GPL software. I think (and certainly Red Hat is betting) that it will fly with enough customers to make them profitable in the long run. > It's almost funny, in a very sad sort of way. People all over the world > (many of them on this list) have written and contributed the actual > software and continue to maintain it. Red Hat is now preparing to turn > around and sell our own software back to us at an absurd markup and with > little (really) added value other than the distributional packaging and > testing. I'm not certain that this is what they are thinking. I'm guessing that they know full well that "we" [and I use "we" for the purposes of this discussion but acknowledge that I personally probably don't belong in this class, not having contributed much of anything] will never pay them any money for this, just like we haven't been paying them any money for it in the past. In this regard, yes, they have written us off. But I think that they have to be given credit for at least attempting to set up a structure within which we can help to maintain a usable RedHat-style distribution in the long run, without it being such a money pit for them. There is certainly lots of precedent for companies just dumping unprofitable products without such a gentle transition. > I personally think that what is going to happen is that the community > will interpret this as control and just route around it. People on the > Internet proper will just ignore the prices; corporate managers who feel > a need to "buy" something may pay them. Companies seem to be trying to > treat open source software as a proprietary product and moving away from > the support model. Well certainly I can think of one Utah-based company in particular that is taking this to absurd levels, but in Red Hat's case I think that they're just trying to find a way to make a sustainable profit on this. I don't see that it makes sense to begrudge them this -- they're not a volunteer organization or even a non-profit. I'm not sure what you mean by "moving away from the support model", because it seems to me that what Red Hat (and others) are doing is, on the contrary, to make support the *product*. All the nonsense about proprietary logos and per-system licenses is nothing more than a lever to coerce customers into buying this support. Unlike closed-source companies like Microsoft, though, Red Hat is *not* preventing you from using the software without buying this support, but they are making it very difficult for you to take advantage of all the stuff they do to make it *easy* to use their software without paying -- making it easy is what they're trying to charge for. > The real tragedy associated with this is that linux is finally maturing > to the point where it could really challenge on the desktop, not just > the server/cluster/specialized market, and these absurd prices could > retard that process for years. Red Hat in particular has missed the > boat by charging crazy prices for box sets for some years. What, they > think Joe Consumer is going to buy a box set of linux for $150 when it > comes "free" and preinstalled on their PC? They should be selling box > sets for $15 and trying to sell a LOT of them, not a few at crazy > prices. But they have been doing that, and what they have been doing has sound economic basis. They've been segmenting the market, charging each customer according to his or her propensity to pay, much for the same reason that car prices have typically been negotiated. In the past, customers have been able to buy Red Hat Linux for next to nothing -- the cost of downloading the ISOs and burning them on to CD-Rs, or even just borrowing their buddy's copy. They've been able to buy it for $10 or so by ordering white-label CDs from Cheap Bytes. They've been able to buy it for $40 or so by picking up a book on Red Hat Linux at the bookstore. They've been able to buy a basic version with a real Red Hat label for maybe $50, and a nice box with an official manual, some amount of support, and CDs containing all sorts of crap for $150. And recently they've been making it available for prices extending into the thousands of dollars, with all sorts of hand-holding thrown in. They've thus been making Red Hat Linux available at a near-continuum of prices, and letting the customers decide how much they'd like to pay. This is a pretty classic formula for maximizing your income on a product, because at the low end, you wind up getting at least a little money from people who otherwise wouldn't have given you anything. The freeloaders are of course a loss but you can at least hope that you've picked up some good will and maybe they'll think of you when they are in a mood to spend actual money. And you can wind up getting some bigger chunks of cash from the people who don't mind parting with it if it gets them something that's all dressed up nice and pretty. However, I think what we're seeing now is that their product is well-enough established in the market that they believe they can do a bit better if they knock out some of the lower prices, and make a somewhat larger distinction between the free "toy" product and the expensive version intended for the "serious" customer. It is of course possible that they've misjudged the market some and they'll have to fine tune this again, possibly by continuing to take snapshots of Fedora Linux and calling it "Red Hat", in the same way that Sun repackages some builds of Open Office as Star Office and Netscape had been using the occasional build of Mozilla for a new release of Netscape. But for now this is what they're going to try, and judging from their latest quarterly report, it seems to be working. > > > nor are "we" willing to pay enough for > > > vendors (like Scyld) to build cluster distros. > > > > This sounds right. > > Absolutely. It is a matter of what people need. MPI, PVM, and more are > often or even generally available in rpm format and are a whole lot of > what people need. The cluster users who need to go a long way beyond > this exist, but there are fewer and fewer of them at each additional > price notch. COTS HPC clusters were invented because they were a CHEAP > road to supercomputing that supported the needs of a hell of a lot of > HPC cycle consumers with only a few, relatively simple tools. The same > issues exist with Myrinet and SCI -- I'll be there are dozens of > clusters installed on plain old ethernet for each one of the clusters > installed on high end networks (which, frankly, are worth a lot more > than high end/expensive "distributions" since Red Hat cannot alter the > fundamentally limiting computation/communications ratio at a given task > scale, but a faster network interface can!). Well, I think that this is an important point, and what I'd say is that it reflects how the cluster market is different from the general-purpose business market. > So pay no attention to the high road. Distribution vendors that take > the high road are going to starve on a rich diet of a few high margin > sales, just as a lot of the traditional Unix vendors have been doing at > ever increasing rates ever since linux was invented. But the problem with traditional Unix vendors, I believe, has less to do with the price of their operating system than (a) the price of their hardware, and (b) the lack of openness in the operating system. Linux is pulling people away because it allows them to take advantage of such a wide variety of hardware, and because they have nearly unlimited choices in sources of support for their systems. Linux shops can be exactly as price- or quality-sensitive as they want or need to be and still use Linux. A little, cash-starved non-profit can run a usable file/print/web server on a box they picked up from the Salvation Army store for $50, while a Fortune 500 shop can consolidate a hundred different services onto an IBM mainframe running Linux under VM. The non-profit with the $50 server can pay the secretary's teenage son $10/hour to dig around and figure out why Samba is broken, while the Fortune 500 can pay IBM six or even seven figures a year to keep the system running like the proverbial well-oiled machine. As it stands, because of the GPL, the distribution vendors are *never* going to get any money out of that non-profit, but as long as the high-end systems work and excellent support is available, those high-margin sales are going to become more and more common; I really don't see the starvation you're expecting. > Look instead to the low road. Look to distributions (like Debian or > freebsd) that charge nothing, or charge a very modest low margin fee. > In the market of the future one will be able to grow fat on huge numbers > of LOW margin sales, just as Microsoft (for the most part) has for > years. If Sun Microsystems had released their perfectly good Unix for > PC's for a retail price of $15/system in a box set a decade plus ago (or > even $50), Linux would never have gotten off the ground, Windows NT > would have been killed in its cradle, and Sun would own the universe. Ah, but what would it have done to SPARC sales? Sun's situation is quite different from that of a Linux vendor such as Red Hat, in that Sun's business model depends on selling high-margin hardware. Solaris x86, IMHO, was always intended as a loss leader designed to get people hooked on Solaris, and draw them into buying the real thing to get broader range of applications support and greater hardware scalability. They never intended Solaris x86 to become a first-class solution platform in and of itself. Sure, a few VARs and ISPs made it work as one, but Sun was never comfortable with that and as you recall they tried a couple of years ago to kill it off altogether. Scott has always been adamant that he was never going to allow Sun to be held hostage to an externally-defined hardware architecture. One of his favorite sayings was that Sun has never lost a dime selling a PC. To a large extent this worked, but IMHO the mistake they made was when they got too greedy in the late 1990s and allowed the low end of their product line to languish; Sun hated the thought that a customer might be able to build their network around $10,000 machines when $50,000 machines could do the job just as well (yes, I meant that). As a result, in the past few years they haven't had anything affordable to offer to companies who were seeing their IT budgets drying up, and they were forced to sell x86 hardware and even -- gasp -- Linux to compensate for this mistake. > I think it must be the scent of blood in the water -- Microsoft is > finally showing signs of distinct vulnerability, the linux desktop is > finally good enough to sell to corporate customers, and the long awaited > phase transition is at hand. Now all the linux distro people can think > about is how fast they can get their options and their shareholder's > highly depressed shares above water. Another way of interpreting the behavior might be that, in slowed economic conditions they are finding it necessary to either turn a profit in short order or go under. While it may not make a lot of difference in the long term for the scientific research community -- we were getting along with Linux just fine long before it became "profitable" -- I really think that the failure of Red Hat in particular would be very bad for Linux in general. > Slow and steady (and low margin sales and true added value support) will > win this race. Red Hat is already facing a sort of defection of a > rather huge part of their installed base; how long will they stay on top > once the Universities and other institutions that provide their real > base of programmers turn their energies to something that they don't > have to turn around and "buy back"? > > With the GPL, not long at all, not long at all. Again, I think that, while they may be losing a large part of their installed base, they are not likely to be losing a large part of their *income*. This is an important distinction that goes a long way to explaining why they are doing it. And again, they are giving us an outlet for our energies that we don't have to "buy back" -- Fedora Linux. We don't have to accept it, but I think it's a better gesture than we'd get from a lot of companies. Back when Sun decided to cancel Solaris x86, I didn't see them offering to turn the product over to the community. --Bob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Sep 25 04:01:14 2003 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 25 Sep 2003 10:01:14 +0200 (CEST) Subject: implementation details of Beowulf In-Reply-To: Message-ID: On Wed, 24 Sep 2003, sherin cyriac wrote: > sir > I am going to do a project in Beowulf cluster. I want to know some > unsolved problems in this domain > .I also need the details of Beowulf cluster that was practically > implemented. I need the implementation details. Sherin, if you are a college or university student, I suggest a quick visit to your college library! Have a search on Amazon for books with 'Beowulf' or 'Linux Clustering' in the title, and go see if your college has them. Otherwise, start at www.beowulf.org _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Wed Sep 24 04:26:09 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 24 Sep 2003 10:26:09 +0200 Subject: A question of OS!! In-Reply-To: References: Message-ID: <1064391969.3349.17.camel@revolution.mandrakesoft.com> > I've used both SuSE and Redhat, and like both. > In fact, we use SuSE on Opterons, as SuSE were there first with > a complete distro, which is very good. This is not true, MandrakeSoft had release the Opteron version of its Linux distribution at the same time that the processor was available. So you have the choice for Opteron between MandrakeLinux and Suse. And for the clustering things, MandrakeClustering also exists for Opteron. > I fail to see though why SuSE would have an advantage in parallel > computations. I think it was initialy because suse had provided a lot of libraries. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From erwan at mandrakesoft.com Wed Sep 24 05:32:38 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 24 Sep 2003 11:32:38 +0200 Subject: A question of OS!! In-Reply-To: References: Message-ID: <1064395958.3347.90.camel@revolution.mandrakesoft.com> Le mer 24/09/2003 ? 10:58, Cannon, Andrew a ?crit : >So which of the mainstream distros are best for Beowulfs? RH, SuSe, >Mandrake etc? > We use RH8, but I am unsure about whether we should still use this or go to > another distro. For my side (MandrakeSoft of course :D), I could tell that MandrakeClustering is a real cluster oriented Linux Distribution. I mean it integrates a lot of thing in one package (a distro) "OS (MandrakeLinux) + Admin & Config tools + Deployment tools (KA) + MPICH & LAM/MPI & PVM + Usual cluster tools (rshp, gexec, ganglia, dsh, clusterit etc...)." Everything is autoconfigured and ready to run after the install process. No need to be a Linux Guru to install a full cluster. Our customers are happy about that point. Scientists (which are not all Linux Gurus) are able to install their owns cluster in a few hours ! MDKC also provides "urpmi parallel" for managing the rpm distribution across the cluster. For this points, RedHat & Suse don't do the same today. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at xnet.com Wed Sep 24 18:37:08 2003 From: djholm at xnet.com (Don Holmgren) Date: Wed, 24 Sep 2003 17:37:08 -0500 (CDT) Subject: Announcement: AMD Opteron and Athlon 64 presentation Message-ID: For those near Atlanta who might be interested in a presentation by AMD of Opteron and Athlon 64 information, see: http://www.csilabs.net/AMD64seminar/ I asked for some more details of the presentations: "As the title of the seminar suggests, the focus will be on AMD64 technology and the AMD64 portfolio of products including both Opteron and Athlon 64. We are trying to schedule some of our HQ guys from our desktop and server product marketing teams down for the event along with other local support. As part of the Opteron discussion, we will certainly discuss the benefit of Opteron processor-based systems across applications including High Performance Computing and clustering as well as the way Opteron simplifies business by providing a single platform for use across the enterprise data center." Don Holmgren _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Wed Sep 24 12:39:05 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Wed, 24 Sep 2003 11:39:05 -0500 (CDT) Subject: A question of OS!! In-Reply-To: Message-ID: On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > Hi,I am planning on installing a 32 node beowulf cluster for high > performance bio-informatics applications. As a OS I was planning on > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > has advantage in parallel computations. Having no hands-on experience in > SuSe Linux, I have no idea how to verify this or do a comparision. What do > you advice as the OS for the cluster? And how to determine this,as there > are a bunch of unix OSes out there.Thanks. > > Sincerely, > Sarat. > First, I would strongly recommend against using RH9 in a production environment. It is a short-life-cycle product meant for hobbyist desktops. More than that, it was a way to beta test some pretty radical OS changes on the masses before their inclusion into RHEL3. Secondly, if you are going to use Opteron there are very good reasons to use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports the Opteron while RH9 will only run in 32-bit mode. SLES also has a 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured NUMA support for the Opteron. If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 will be out and has the same benefits as SLES. -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan.velu at free.fr Thu Sep 25 05:12:44 2003 From: erwan.velu at free.fr (erwan.velu at free.fr) Date: Thu, 25 Sep 2003 11:12:44 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] Message-ID: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> Drzyzgula wrote: > > >Also, I'm thinking I must not know enough about Mandrake's > >cluster distribution. Looking at their website, it > >seems to cost $2320 for 1-8 processors, $4176K for 9-16 > >processors. Why would a cluster shop that had previously > >used Red Hat Linux choose to use Mandrake rather than > >Red Hat's Enterprise WS distribution, which costs $299 > >per single- or dual-processor system, except perhaps on > >technical grounds? > > > Mandrake puts out a completely free cluster distribution called clic: > http://clic.mandrakesoft.com/index-en.html Firstable, sorry for not using my corporate email but since few days all the emails sent to beowulf ML reach the mailbox but I never seen any message (except one because I had ask for it ... I am filtred ?) All the fedora thread is really intresting... Why people should pay for such clustering oriented distro ? Why CLIC is free whereas you must pay for MDKC with a per-CPU licence ? CLIC was a project co-financed by the french government, so it MandrakeSoft and its partners were being half payed for this work. It was obvious that we are making a GPL product available for download. CLIC is a real technological success and its customers like it for this. Today we are facing the following problem : how to continue such project ? How can MandrakeSoft could make some revenues on this king of product ? What do people using CLIC gives at MandrakeSoft in return of its work ? Some buy some consulting, but some do nothing... CLIC is finishing at Decembrer 2003, so should we stop CLIC ? How the guys are working hard on such product could be payed ? We decided to launch a new product called MandrakeClustering based on the CLIC success. We have rewritten all the backend in perl, added a lot of features. We want this technology to survive. Now MDKC is a real product, not a research project as CLIC. MandrakeSoft now sells this product whereas we were just selling consulting on CLIC. This model now change the rules. Products like maui can't be resell without permissions. There is a per-cpu fee for such product, so we are now applying a per-cpu pricing on our product. The price of MDKC is really reasonable ! For 8CPUS under x86 or AMD64 it costs 2000?, until 16 -> 3600? etc.. Please make the ratio between the hardware price and the software price. What is the future of CLIC ? Is will depend of the results of MandrakeClustering. If it works fine, we will continue CLIC because the clustering team could be payed for such work. This is what we are doing today. The technology of MDKC is currently in integration in CLIC. So CLIC will contain a part of MDKC a few after the release. The last point is you can't compare RedHat Enterprise with MandrakeClustering ! MandrakeClustering is a real oriented clustering distro not just a MandrakeLinux + some few packages. MDKC is integrating the Linux Distro + configuration tools + deployment tools + clustering tools + clustering mpi stacks. Everything is auto configured, you don't need to be a Linux expert to install a cluster. My best example ? How to integrate new nodes ? CLICK on a node, CLICK on duplicate, tell the number of nodes you want to duplicate, the device you want to duplicate. Then just power on your nodes ! Nothing more to do ! You just need a few hours to setup a full cluster ! DNS, DHCP, PXE, NIS, AUTOFS, LAM/MPI, MPICH, PVM, TFTP, etc.. Everything is built automaticaly ! Does RH do it ? You can manage all the rpms on your cluster using "urpmi" (could be compared to apt) but using parallel tools ! How to install a tool on a cluster ? urpmi --parallel cluster Dependencies are automaticaly installed like apt. Does RH do it ? You can install security/product updates from the server using a simple command: urpmi --parallel cluster --auto-select --auto Updates are downloaded from Internet then parallely send to the nodes then each nodes update itslef following its own needs ! Who can do that today ? Does RH allow you to duplicate 200 nodes in 3 minutes ~500 MBits ? No, you need to integrate yourself other tools you must installed and configure. MDKC & CLIC are some real clustering oriented distro, everything has been thought for the easyness of use.. I don't know if this king of approach could match all the clustering needs, but I must say that people who did really test this kind of distribution likes it. I'm not saying that other products are poor, we just have another approach that our Linux editor postion allow us to give. I hope this mail could help everyone to understand the work of the CLIC & MDKC team. Linuxely yours, -- Erwan Velu (erwan at mandrakesoft.com) Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Wed Sep 24 05:09:32 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 24 Sep 2003 11:09:32 +0200 Subject: rsh In-Reply-To: <20030917024414.M10506@ug.ms.wits.ac.za> References: <20030917024414.M10506@ug.ms.wits.ac.za> Message-ID: <1064394572.3347.57.camel@revolution.mandrakesoft.com> > I tried running tstmachines and I got > "Errors while trying to run rsh hostname -n true > Unexpected response from hostname: > ---> Permission denied ..." > Please help with the rsh settings that I need to inorder to get the rsh to work. > Masego-otlhe Segone Please check that your the .rhost of your user account is like the following: example: final.compute.mdkc.mandrakesoft.com test compute1.compute.mdkc.mandrakesoft.com test compute3.compute.mdkc.mandrakesoft.com test compute4.compute.mdkc.mandrakesoft.com test compute5.compute.mdkc.mandrakesoft.com test compute6.compute.mdkc.mandrakesoft.com test compute7.compute.mdkc.mandrakesoft.com test compute8.compute.mdkc.mandrakesoft.com test compute9.compute.mdkc.mandrakesoft.com test compute2.compute.mdkc.mandrakesoft.com test then be sure that .rhost is 644, "vi" give usualy a group writable permission that rsh doesn't like. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From joelja at darkwing.uoregon.edu Thu Sep 25 13:07:36 2003 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Thu, 25 Sep 2003 10:07:36 -0700 (PDT) Subject: A question of OS!! In-Reply-To: Message-ID: On Wed, 24 Sep 2003, Rocky McGaugh wrote: > On Tue, 23 Sep 2003, Sarat C Maruvada wrote: > > > Hi,I am planning on installing a 32 node beowulf cluster for high > > performance bio-informatics applications. As a OS I was planning on > > installing Red Hat Linux 9.0.But I had heard from somebody that SuSe Linux > > has advantage in parallel computations. Having no hands-on experience in > > SuSe Linux, I have no idea how to verify this or do a comparision. What do > > you advice as the OS for the cluster? And how to determine this,as there > > are a bunch of unix OSes out there.Thanks. > > > > Sincerely, > > Sarat. > > > > First, I would strongly recommend against using RH9 in a production > environment. It is a short-life-cycle product meant for hobbyist desktops. > More than that, it was a way to beta test some pretty radical OS changes > on the masses before their inclusion into RHEL3. nptl anyone? ;) > Secondly, if you are going to use Opteron there are very good reasons to > use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports > the Opteron while RH9 will only run in 32-bit mode. SLES also has a > 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured > NUMA support for the Opteron. > > If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 > will be out and has the same benefits as SLES. you can test the amd64 beta 2 of taroon right now if you so desire it's on ftp.redhat.com and the mirrors... > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rokrau at yahoo.com Thu Sep 25 14:08:21 2003 From: rokrau at yahoo.com (Roland Krause) Date: Thu, 25 Sep 2003 11:08:21 -0700 (PDT) Subject: MPICH, Fortran and command line arguments Message-ID: <20030925180821.42274.qmail@web40004.mail.yahoo.com> Hi all, I have a Fortran code that I am trying to run under Linux (Redhat-8) using the Intel Fortran compiler 7.1 and mpich-1.2.5.2. I start the code with mpirun -np 2 code.x input.file When the code starts it turns out that only the first instance, i.e. the process with rank 0, sees the correct command line arguments, namely input.file, the next process things the hostname I run on is the first cmdl argument. The MPI_Init manual page has this ugly little blurb about the problem: Command line arguments are not provided to Fortran programs. More precisely, non-standard Fortran routines such as getarg and iargc have undefined behavior in MPI and in this implementation. Does anyone here have experience or a recommendation how I can get the other processes to see the command line arguments? If I consider broadcasting them myself, is it at least guaranteed that the first (rank 0) process will see the command line arguments? How do you guys deal with that situation, short of rewriting the main app in C of course :-) Regards, Roland __________________________________ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Thu Sep 25 20:02:30 2003 From: bill at math.ucdavis.edu (Bill Broadley) Date: Thu, 25 Sep 2003 17:02:30 -0700 Subject: A question of OS!! In-Reply-To: References: Message-ID: <20030926000229.GA7113@sphere.math.ucdavis.edu> > Secondly, if you are going to use Opteron there are very good reasons to > use SuSE Enterprise Linux (SLES) instead of RH9. SLES natively supports > the Opteron while RH9 will only run in 32-bit mode. SLES also has a > 5 or so year lifespan, versus the 1 year on RH9. SLES has preconfigured > NUMA support for the Opteron. Has anyone been benchmarking SLES vs RHEL v3 beta? One of my codes is twice as fast on a dual 1.4 Ghz opterons with RHEL v3 beta as compared to a dual 1.8 Ghz opteron running SLES. The application is an earthquake simulation using MPI, I used the newest MPICH with the same compiled features, and am running 2 processes in both cases. -- Bill Broadley Mathematics UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Thu Sep 25 19:50:36 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 26 Sep 2003 09:50:36 +1000 Subject: A question of OS!! In-Reply-To: References: Message-ID: <200309260950.56059.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 25 Sep 2003 02:39 am, Rocky McGaugh wrote: > If you wait another month, the RedHat Enterprise Linux (RHEL) version 3 > will be out and has the same benefits as SLES. One of the major issues we have is that some of the commercial software our users use is only supported under RH7.3 as of yet. We'd really like to see broader support for other distributions, but it seems that the market is going the other way with people only offering support for for RH (and presumably only for RH Enterprise in the near future) or for SLES. We've had support issues already just because we're running a slightly different version of PBS to the one the vendor supports (leaving me to do a lot of bug fixing of their code, fortunately it's just Perl!). I'd hate to think what would happen if we were running a different distro altogether. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/c39MO2KABBYQAh8RAljcAKCCTaZpytPguoz+qUmF4dAbzvppfwCghObG 7X+lQvvKDJKcVD6bSj15nnY= =x1Jn -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian.dobbins at yale.edu Thu Sep 25 20:19:51 2003 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Thu, 25 Sep 2003 20:19:51 -0400 (EDT) Subject: A question of OS!! In-Reply-To: <20030926000229.GA7113@sphere.math.ucdavis.edu> Message-ID: > Has anyone been benchmarking SLES vs RHEL v3 beta? One of my codes > is twice as fast on a dual 1.4 Ghz opterons with RHEL v3 beta as compared > to a dual 1.8 Ghz opteron running SLES. Yikes.. what kernels are used on these systems by default, and how large is the code? I've been running SuSE 8.2 Pro on my nodes, and have gotten varying performance due to motherboard, BIOS level and kernel. (SuSE 8.2 Pro comes a modified 2.4.19, but I've also run 2.6.0-test5) Also, are the BIOS settings the same? And how are the RAM slots populated? That made a difference, too! (Oh, and I imagine they're both writing to a local disk, or minimal amounts over NFS? That could play a big part, too.. ) I should have some numbers at some point for how much things vary, but at the moment we've been pretty busy on our systems. Any more info on this would be great, though, since I've been looking at the faster chips, too! Cheers, -Brian _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Thu Sep 25 20:45:34 2003 From: bill at math.ucdavis.edu (Bill Broadley) Date: Thu, 25 Sep 2003 17:45:34 -0700 Subject: A question of OS!! In-Reply-To: References: <20030926000229.GA7113@sphere.math.ucdavis.edu> Message-ID: <20030926004534.GQ4151@sphere.math.ucdavis.edu> > Yikes.. what kernels are used on these systems by default, and how large > is the code? I've been running SuSE 8.2 Pro on my nodes, and have gotten Factory default in both cases AFAIK. I don't have access to the SLES system at the moment, but the redhat box is: Linux foo.math.ucdavis.edu 2.4.21-1.1931.2.349.2.2.entsmp #1 SMP Fri Jul 18 00:06:19 EDT 2003 x86_64 x86_64 x86_64 GNU/Linux What relationship that has to the original 2.4.21 I know not. > varying performance due to motherboard, BIOS level and kernel. (SuSE 8.2 > Pro comes a modified 2.4.19, but I've also run 2.6.0-test5) > Also, are the BIOS settings the same? And how are the RAM slots I don't have access to the SLES bios. > populated? That made a difference, too! I'm well aware of the RAM slot issues, and I've experimentally verified that the full bandwidth is available. Basically each cpu will see 2GB/sec or so to main memory, and both see a total of 3GB/sec if both use memory simultaneously. > (Oh, and I imagine they're both writing to a local disk, or minimal > amounts over NFS? That could play a big part, too.. ) Yeah, both local disk, and not much. I didn't notice any difference when I commented out all output. > I should have some numbers at some point for how much things vary, but > at the moment we've been pretty busy on our systems. Any more info on > this would be great, though, since I've been looking at the faster chips, > too! ACK, I never considered that the opterons might be slower in some ways at faster clock speeds. My main suspicious is that MPICH was messaging passing for local nodes in some strange way and triggering some corner case under SLES. I.e. writing an int at a time between CPUs who are fighting over the same page. None of my other MPI benchmarks for latency of bandwidth (at various message sizes) have found any sign of problems. Numerous recompiles of MPICH haven't had any effect either. -- Bill Broadley Mathematics UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Thu Sep 25 10:36:25 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Thu, 25 Sep 2003 16:36:25 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: References: Message-ID: <1064500585.18782.28.camel@revolution.mandrakesoft.com> > For many users they need nothing but the > kernel, a shell, MPI or PVM daemons or clients, and some libraries. A > relatively fancy node might add some scheduling or monitoring daemons. I agree on that point but I'm not sure that everyone wants to know how does it works, and take the time to set up everything. Scientists for example are not Linux Guru. > Who is going to pay $100 PER NODE for that? Nobody sane. Maybe some > businesses who want to build a cluster but don't want to hire somebody > that knows what they are doing to do so (which is a bit oxymoronic, > right? Likely a fairly limited, although undoubtedly high margin, > market). We've made the proof that non Linux awared people can install a cluster for their needs. I know that is not the case everywhere but some like it. [...] > If it takes me a whole day to graze the web and collect all the tool > sources, generally ready to build, directly from their primary website, > why would I (or my employer) pay thousands of dollars for them ever? > Instead of "wasting" time on this list I'd divert some of this energy > into getting and building the sources. Then (the web being what it now > is and not wanting to waste the effort) I'd put the packages in a > toplevel public repository. Suddenly anybody out there can mirror my > efforts. This is really intresting. I've met people who told the contrary ! They said : "I know how does it works, but I don't want to spend my time downloading/configuring clusters. That's your job (Linux Editors) and you do it well. Why should I spent my time doing the thing you do faster & safer". We follow the products, the security updates, the kernel stuff etc.. Some don't have the time to do it. > This is what you are facing. I'm certain that CLIC is very nice, and I > plan to try it out pretty much immediately as RH (which I have used for > years) is in a middling state of turmoil and creating THEIR OWN FUD [..] If you want to try CLIC please wait the next release (october) that will contain the newest features. Keep me in touch about this tests, I will give you a hand if needed. > What is a reasonable price per cluster then? Something in the <$100 > range. And for that money, don't sell them CLIC -- as you say, CLIC is > free! Sell them something with clear added value that costs you very > little to provide. Sell whole universities this sort of toplevel access > to a patch stream for $1000, whole governments for $100,000. If you can > sell 100 universities or 1000 clusters or one government, you can just > about pay for the one human likely to be required to run it, and > thereafter you're making money. I've missed to explain clearer that point. We are offering a price model based on the number of CPU because some software we are including do it. This model works fine with small clusters in laboratories/small companies. But we are now working on a site licence. Which allow an university/company to use the product for a while for a fixed price that we must negociate. > My own personal suggestion would be a subscription of sorts to a > private (authenticated), high bandwidth update server from which the > user could rsync a local mirror on their cluster server. Drop your > maintenance stuff in there and have it automatically propagate. Sure, > sell consulting and service on the side for market price -- some people > will need it and buy it. Keep it in the range where people can see that > they are getting their money's worth and you make a fair profit. This already exist throught the MandrakeClub. > The distribution vendors that survive in the long run will be the ones > content with EARLY Microsoft income streams -- $30 per PC per year, > maybe. Or $10. For that, they have to provide real value, but the > market is there for it. Look at (to point to an interesting metaphor) > Harry Potter. The book sells for order $10, yet it has made its author > a billionaire and its publisher likely 10 times that. No, this won't > create a huge empire of world dominating high margin arm twisted sales, > but its a living, and a comfortable one at that. Even wealth. It just > is EARNED wealth. Model is nice but I can figure how to sell such number of clusters :D > I'd put a paypal link right up on your CLIC download site, and suggest > that everyone who downloads it contribute/pay $50 "per cluster" and see > how many people do so. People are free to do it if they like our product ! They also able to do it throught the MandrakeClub.. > >Maybe nobody, but you might be surprised. Right > now I pay transgaming.com regular small money just to enable GAMES in my > household. I'll bet they are making money, and frankly it is a damn > sight harder to get Window software to run under linux than it is to > assemble a cluster package that meets a whole lot of needs. One I can > do myself in my spare time, the other I wouldn't touch with a ten foot > pole. If they charge small money, so can (and so should) you. I was one of those :D > Look at their business model. This is the linux model of the future, I > believe. GPL (and available via sourceforge) but who wants to go > through a sourceforge build when you can get a ready to run build and > updates for a few dollars? We remain GPL and available. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From erwan at mandrakesoft.com Thu Sep 25 11:30:05 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Thu, 25 Sep 2003 17:30:05 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: References: Message-ID: <1064503805.18783.70.camel@revolution.mandrakesoft.com> > > Scientists for example are not Linux Guru. > Ummm, what makes you say that? Sorry, I meant that not all the Scientists are able to manage all the configuration needed by a cluster. We give Scientists an easiest way to setup things that could take them longer. > p.s. Oh, you mean ALL scientists aren't linux bozos (gurus seems like a > pretentious word:-). Sure, but scientists who work on massively > parallel computers are fairly likely to be, or they are likely to hire > someone who is. If they are already paying that person large sums of > money, why pay you as well? You can pay someone to manage more clusters when he's working on powerfull tools rather paying one doing everything by hand. If everyone should compile/configure/understand all the technologies included in a product, some could be frighten of this ! > >If they don't have such a person, will they really be able to manage a cluster without one? Depending on what you call "manage" but for usual tasks... yes ! > >Even setting up a TCP/IP switched ethernet network requires expertise, as does adding > accounts, debugging problems. For adding accounts, you just need to know the username/password and the logical parition you want him to be. Not really complicated ! > > Plug and play solutions are indeed very expensive, because they have to be nearly flawless to work in an > environment without support. Our solution is to build autoconfigured services. It works really fine...I've integrated this product in a big architecture.. The cluster is working fine ! -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 25 06:57:20 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 25 Sep 2003 06:57:20 -0400 (EDT) Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064500585.18782.28.camel@revolution.mandrakesoft.com> Message-ID: dOn Thu, 25 Sep 2003, Erwan Velu wrote: > > For many users they need nothing but the > > kernel, a shell, MPI or PVM daemons or clients, and some libraries. A > > relatively fancy node might add some scheduling or monitoring daemons. > I agree on that point but I'm not sure that everyone wants to know how does it works, > and take the time to set up everything. Scientists for example are not > Linux Guru. Ummm, what makes you say that? rgb (Ph.D. theoretical physics, 1982) p.s. Oh, you mean ALL scientists aren't linux bozos (gurus seems like a pretentious word:-). Sure, but scientists who work on massively parallel computers are fairly likely to be, or they are likely to hire someone who is. If they are already paying that person large sums of money, why pay you as well? If they don't have such a person, will they really be able to manage a cluster without one? Even setting up a TCP/IP switched ethernet network requires expertise, as does adding accounts, debugging problems. Plug and play solutions are indeed very expensive, because they have to be nearly flawless to work in an environment without support. rgb (again, back in beowulfish guise) -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Sep 25 09:43:04 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 25 Sep 2003 09:43:04 -0400 (EDT) Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> Message-ID: On Thu, 25 Sep 2003 erwan.velu at free.fr wrote: > All the fedora thread is really intresting... Why people should pay for > such clustering oriented distro ? Why CLIC is free whereas you must pay for > MDKC with a per-CPU licence ? > > CLIC was a project co-financed by the french government, so it MandrakeSoft and > its partners were being half payed for this work. It was obvious that we are > making a GPL product available for download. CLIC is a real technological > success and its customers like it for this. > > Today we are facing the following problem : how to continue such project ? How > can MandrakeSoft could make some revenues on this king of product ? What do > people using CLIC gives at MandrakeSoft in return of its work ? Some buy some > consulting, but some do nothing... I think this is the basic question. In part I think that the answer is for you to come up with a REASONABLE pricing model. If you follow this list (and it sounds like you do) then you know that when people are building a cluster, whether it has four nodes or four hundred nodes, prices that scale per node (period) are crazy. A cluster node has always been "a CPU" in the parallel supercomputer model, and the relative LACK of marginal cost in per node installation/maintenance of linux has been a major reason it has been used. To install and manage and operate an entire cluster, most people actually "work on" one or two server/head nodes; the rest of the cluster installs a really, really boring operating system that is little more (functionally) than a task loader. In the Scyld/bproc extreme this is literal; in clusters based on higher order distros more of a metaphor, but the point is that most of the cluster nodes don't require most of the gorp that is in a standard distro anyway. For many users they need nothing but the kernel, a shell, MPI or PVM daemons or clients, and some libraries. A relatively fancy node might add some scheduling or monitoring daemons. Who is going to pay $100 PER NODE for that? Nobody sane. Maybe some businesses who want to build a cluster but don't want to hire somebody that knows what they are doing to do so (which is a bit oxymoronic, right? Likely a fairly limited, although undoubtedly high margin, market). So prices have to reflect the scaling of the actual work, acknowledging that the very scalability of linux means that cluster managers may WELL be relatively underutilized once a cluster is installed and operating smoothly, so that they DO have opportunity cost time to do it themselves "for free" as far as marginal cost to the enterprise is concerned. If it takes me a whole day to graze the web and collect all the tool sources, generally ready to build, directly from their primary website, why would I (or my employer) pay thousands of dollars for them ever? Instead of "wasting" time on this list I'd divert some of this energy into getting and building the sources. Then (the web being what it now is and not wanting to waste the effort) I'd put the packages in a toplevel public repository. Suddenly anybody out there can mirror my efforts. This is what you are facing. I'm certain that CLIC is very nice, and I plan to try it out pretty much immediately as RH (which I have used for years) is in a middling state of turmoil and creating THEIR OWN FUD (an amazing business strategy, particularly at this time when SCO and MS and even Sun are doing their best in the FUD department without help:-). What is a reasonable price per cluster then? Something in the <$100 range. And for that money, don't sell them CLIC -- as you say, CLIC is free! Sell them something with clear added value that costs you very little to provide. Sell whole universities this sort of toplevel access to a patch stream for $1000, whole governments for $100,000. If you can sell 100 universities or 1000 clusters or one government, you can just about pay for the one human likely to be required to run it, and thereafter you're making money. My own personal suggestion would be a subscription of sorts to a private (authenticated), high bandwidth update server from which the user could rsync a local mirror on their cluster server. Drop your maintenance stuff in there and have it automatically propagate. Sure, sell consulting and service on the side for market price -- some people will need it and buy it. Keep it in the range where people can see that they are getting their money's worth and you make a fair profit. Most cluster managers, I think, value the distributions and would (as Bob said) view it as a Bad Thing if distribution vendors went away. We WANT you guys to make money. I personally pay Red Hat fifty bucks every few years just to contribute fair value back to them for what I receive (this is for my personal home systems). On the other hand, NOBODY WILL EVER MAKE MICROSOFTISH SORTS OF MONEY AGAIN! So folks need to get over it. It ain't happening. In a few years not even Microsoft will make Microsoftish money -- their empire is being forcibly ripped apart by market forces and entire governments, deliberately. The distribution vendors that survive in the long run will be the ones content with EARLY Microsoft income streams -- $30 per PC per year, maybe. Or $10. For that, they have to provide real value, but the market is there for it. Look at (to point to an interesting metaphor) Harry Potter. The book sells for order $10, yet it has made its author a billionaire and its publisher likely 10 times that. No, this won't create a huge empire of world dominating high margin arm twisted sales, but its a living, and a comfortable one at that. Even wealth. It just is EARNED wealth. I'd put a paypal link right up on your CLIC download site, and suggest that everyone who downloads it contribute/pay $50 "per cluster" and see how many people do so. Maybe nobody, but you might be surprised. Right now I pay transgaming.com regular small money just to enable GAMES in my household. I'll bet they are making money, and frankly it is a damn sight harder to get Window software to run under linux than it is to assemble a cluster package that meets a whole lot of needs. One I can do myself in my spare time, the other I wouldn't touch with a ten foot pole. If they charge small money, so can (and so should) you. Look at their business model. This is the linux model of the future, I believe. GPL (and available via sourceforge) but who wants to go through a sourceforge build when you can get a ready to run build and updates for a few dollars? rgb > > CLIC is finishing at Decembrer 2003, so should we stop CLIC ? How the guys are > working hard on such product could be payed ? We decided to launch a new > product called MandrakeClustering based on the CLIC success. > > We have rewritten all the backend in perl, added a lot of features. We want > this technology to survive. Now MDKC is a real product, not a research project > as CLIC. MandrakeSoft now sells this product whereas we were just selling > consulting on CLIC. This model now change the rules. Products like maui can't > be resell without permissions. There is a per-cpu fee for such product, so we > are now applying a per-cpu pricing on our product. > > The price of MDKC is really reasonable ! > For 8CPUS under x86 or AMD64 it costs 2000?, > until 16 -> 3600? etc.. > Please make the ratio between the hardware price and the software price. > > What is the future of CLIC ? Is will depend of the results of > MandrakeClustering. If it works fine, we will continue CLIC because the > clustering team could be payed for such work. This is what we are doing > today. The technology of MDKC is currently in integration in CLIC. So CLIC will > contain a part of MDKC a few after the release. > > The last point is you can't compare RedHat Enterprise with MandrakeClustering ! > MandrakeClustering is a real oriented clustering distro not just a > MandrakeLinux + some few packages. > MDKC is integrating the Linux Distro + configuration tools + deployment tools + > clustering tools + clustering mpi stacks. > > Everything is auto configured, you don't need to be a Linux expert to install a > cluster. > My best example ? How to integrate new nodes ? CLICK on a node, CLICK on > duplicate, tell the number of nodes you want to duplicate, the device you want > to duplicate. Then just power on your nodes ! Nothing more to do ! > > You just need a few hours to setup a full cluster ! DNS, DHCP, PXE, NIS, > AUTOFS, LAM/MPI, MPICH, PVM, TFTP, etc.. Everything is built automaticaly ! > Does RH do it ? > > You can manage all the rpms on your cluster using "urpmi" (could be compared to > apt) but using parallel tools ! How to install a tool on a cluster ? > urpmi --parallel cluster > Dependencies are automaticaly installed like apt. > Does RH do it ? > > You can install security/product updates from the server using a simple command: > urpmi --parallel cluster --auto-select --auto > Updates are downloaded from Internet then parallely send to the nodes then each > nodes update itslef following its own needs ! Who can do that today ? > > Does RH allow you to duplicate 200 nodes in 3 minutes ~500 MBits ? > No, you need to integrate yourself other tools you must installed and configure. > > MDKC & CLIC are some real clustering oriented distro, everything has been > thought for the easyness of use.. > > I don't know if this king of approach could match all the clustering needs, but > I must say that people who did really test this kind of distribution likes it. > I'm not saying that other products are poor, we just have another approach that > our Linux editor postion allow us to give. > > I hope this mail could help everyone to understand the work of the CLIC & MDKC > team. > > Linuxely yours, > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Thu Sep 25 20:57:25 2003 From: csamuel at vpac.org (Chris Samuel) Date: Fri, 26 Sep 2003 10:57:25 +1000 Subject: A question of OS!! In-Reply-To: <20030926004534.GQ4151@sphere.math.ucdavis.edu> References: <20030926000229.GA7113@sphere.math.ucdavis.edu> <20030926004534.GQ4151@sphere.math.ucdavis.edu> Message-ID: <200309261057.26445.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 26 Sep 2003 10:45 am, Bill Broadley wrote: > My main suspicious is that MPICH was messaging > passing for local nodes in some strange way and triggering some corner > case under SLES. I.e. writing an int at a time between CPUs who are > fighting over the same page. The RH kernel may have additional/better patches or config ? Your code benefits immensely from NPTL ? - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD4DBQE/c471O2KABBYQAh8RAh5JAJi5SEVZKG+lveIth0vifzXmNnv4AJwJ5OBv QaEdIyiMtV3iMq9ZTe8LMw== =Endv -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Thu Sep 25 11:52:00 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Thu, 25 Sep 2003 17:52:00 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <3F730945.6060606@scali.com> References: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> <3F730945.6060606@scali.com> Message-ID: <1064505120.18785.94.camel@revolution.mandrakesoft.com> > MDKC is still GPL'ed isn't it ? Atleast it contains a number of GPL products. Why isn't it possible to download MDKC and redistribute it as you can do with other distributions ? MDKC is still made by a GPL'ed product and also contains some BSD like licenses and some other licences close to but with some restriction about reselling. All our work is available trought cooker and our CVS and obsviously we send patches to the authors. > Isn't that GPL violation ? Of course not. > >Although RedHat Enterprise Linux costs $$ if you want the ISO's you can still compile your own distribution from the source they provide. You can, > Also anyone who actually paid to get the RHEL CD's can (AFAIK) make copies and give to friends without problems, correct ? Correct, > Is this true with MDKC too ? This it true but you can't sell it because some companies must be payed when you sell it. But people must understand that if they will not contribute the product will never be released anymore. That's an individual choice. > Same thing with Scyld really, what prohibits a proud owner of "Scyld Beowulf Professional Edition 28cz Series" to make a copy available for download ? Does it contain products > which isn't GPL'ed, which is what you pay $375 per processor for ? The average price is closer than 150-200? rather 375?. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sp at scali.com Thu Sep 25 11:27:01 2003 From: sp at scali.com (Steffen Persvold) Date: Thu, 25 Sep 2003 17:27:01 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> References: <1064481164.3f72b18ce1fa3@imp3-a.free.fr> Message-ID: <3F730945.6060606@scali.com> erwan.velu at free.fr wrote: [] > > I don't know if this king of approach could match all the clustering needs, but > I must say that people who did really test this kind of distribution likes it. > I'm not saying that other products are poor, we just have another approach that > our Linux editor postion allow us to give. > > I hope this mail could help everyone to understand the work of the CLIC & MDKC > team. Erwan, MDKC is still GPL'ed isn't it ? Atleast it contains a number of GPL products. Why isn't it possible to download MDKC and redistribute it as you can do with other distributions ? Isn't that GPL violation ? Although RedHat Enterprise Linux costs $$ if you want the ISO's you can still compile your own distribution from the source they provide. Also anyone who actually paid to get the RHEL CD's can (AFAIK) make copies and give to friends without problems, correct ? Is this true with MDKC too ? Same thing with Scyld really, what prohibits a proud owner of "Scyld Beowulf Professional Edition 28cz Series" to make a copy available for download ? Does it contain products which isn't GPL'ed, which is what you pay $375 per processor for ? Regards, Steffen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 26 14:00:06 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 26 Sep 2003 11:00:06 -0700 Subject: Redhat Fedora In-Reply-To: References: <20030924194442.H3752@www2> Message-ID: <20030926180006.GB2187@greglaptop.internal.keyresearch.com> On Wed, Sep 24, 2003 at 08:54:46PM -0400, Robert G. Brown wrote: > The interesting question is why anybody in their right mind would pay > for any sort of GPL-based distribution that scales per number of > processors. Especially pay numbers like these. Hell, one might as well > be running Windows. This is a really unfair and misguided comment. You're paying for support. If you don't want to pay that much, don't. Given that most of the code is GPLed, you are free to build a GPL-only subset of any distro and give it away. Comparing it to Windozs is the unfair and misguided part. Them's fightin' words! > The same > issues exist with Myrinet and SCI -- I'll be there are dozens of > clusters installed on plain old ethernet for each one of the clusters > installed on high end networks One would hope that the choice was made with price/performance in mind, instead of the incorrect reasoning of "Myrinet is expensive." > (which, frankly, are worth a lot more > than high end/expensive "distributions" since Red Hat cannot alter the > fundamentally limiting computation/communications ratio at a given task > scale, but a faster network interface can!). There's more to a cluster than the computation. RedHat would claim that their added value is in administration. It's pointless (unfair, misguided, insert more terms here) to complain that RedHat doesn't improve computation. Really, Robert, this rant is not up to your usual quality. Sheesh. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Fri Sep 26 14:44:10 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Fri, 26 Sep 2003 20:44:10 +0200 Subject: Redhat Fedora References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> Message-ID: <3F7488FA.2000901@moene.indiv.nluug.nl> Greg Lindahl wrote: > It's an interesting question. RedHat's new advice on the matter is > "buy a copy of Advanced Workstation for every cluster node." > I suspect that we'll see: > > More use of hobbist distros like Debian Volunteer .NE. hobbyist. -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Fri Sep 26 13:53:12 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Fri, 26 Sep 2003 10:53:12 -0700 Subject: Redhat Fedora In-Reply-To: <20030924194442.H3752@www2> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> Message-ID: <20030926175312.GA2187@greglaptop.internal.keyresearch.com> On Wed, Sep 24, 2003 at 07:44:42PM -0400, Bob Drzyzgula wrote: > Actually, I thought that in fact "we" tended to do a lot of > this -- cf. Rocks, Clustermatic, Warewulf -- it's just that > these never seem to capture the market in the same way that > Red Hat always was. Are any of these really "distributions", or are they "a minimal set of stuff overlaid on RedHat?" It makes sense to use a free and complete conventional distro to build a cluster distro -- but now the free and complete distro is splitting into the less-reliable and less-fixed distro (consumer) and the not-exactly-up-to-date distro (enterprise), which changes what we can do cheaply. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bob at drzyzgula.org Fri Sep 26 15:43:51 2003 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Fri, 26 Sep 2003 15:43:51 -0400 Subject: Redhat Fedora In-Reply-To: <20030926175312.GA2187@greglaptop.internal.keyresearch.com> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> <20030926175312.GA2187@greglaptop.internal.keyresearch.com> Message-ID: <20030926154351.M3752@www2> On Fri, Sep 26, 2003 at 10:53:12AM -0700, Greg Lindahl wrote: > > On Wed, Sep 24, 2003 at 07:44:42PM -0400, Bob Drzyzgula wrote: > > > Actually, I thought that in fact "we" tended to do a lot of > > this -- cf. Rocks, Clustermatic, Warewulf -- it's just that > > these never seem to capture the market in the same way that > > Red Hat always was. > > Are any of these really "distributions", or are they "a minimal set of > stuff overlaid on RedHat?" It makes sense to use a free and complete > conventional distro to build a cluster distro -- but now the free and > complete distro is splitting into the less-reliable and less-fixed > distro (consumer) and the not-exactly-up-to-date distro (enterprise), > which changes what we can do cheaply. Yes, on looking closer I see that you're right about the layering thing. I saw ISOs on those sites and hadn't looked more deeply. You're also right that the way out of this problem isn't clear. I suppose that if Fedora works out better than most of us are probably expecting, then there'd be hope there. It also occurs to me that it may make sense for the cluster community to consider something like a Cluster-Fedora subproject; perhaps we could come up with something that wasn't a cluster-somethingelsestartingwithf (sorry). Possibly Debian would be a better base in the long term, not that I'm particularly fond of Debian. Possibly, given that, as Erwan said, the CLIC contract is up in December 2003, perhaps it would make more sense for the community to pick that up as a starting point. But I suppose this is a bridge we'll have to cross when we come to it, especially since the bridge doesn't even seem to be fully built as yet. --Bob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Sep 26 09:45:23 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 26 Sep 2003 09:45:23 -0400 (EDT) Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: <1064503805.18783.70.camel@revolution.mandrakesoft.com> Message-ID: On Thu, 25 Sep 2003, Erwan Velu wrote: > > > Scientists for example are not Linux Guru. > > Ummm, what makes you say that? > Sorry, I meant that not all the Scientists are able to manage all the configuration needed by a cluster. > We give Scientists an easiest way to setup things that could take them > longer. > > > p.s. Oh, you mean ALL scientists aren't linux bozos (gurus seems like a > > pretentious word:-). Sure, but scientists who work on massively > > parallel computers are fairly likely to be, or they are likely to hire > > someone who is. If they are already paying that person large sums of > > money, why pay you as well? > You can pay someone to manage more clusters when he's working on > powerfull tools rather paying one doing everything by hand. > If everyone should compile/configure/understand all the technologies > included in a product, some could be frighten of this ! I know, I was just teasing you a bit -- note smileys (:-). However, from a purely practical point of view, we find that the thing that limits the number of systems a manager can take care of for pretty much any of the distributions these days is hardware reliability and sheer number of systems. To put it another way, give me an unlimited supply of flunkies to install and repair hardware (only) and I'll cheerily take care of a thousand node cluster all by myself, as far as the software installation and management is concerned. At this point PXE, a variety of rapid (scripted or image based) install methods, and yum for rpm post-install maintenance mean that installation and maintenance of an almost indefinite number of machines is primarily a matter of taking care of the repository(s) from which they are installed and maintained, and this has to be done the same for a cluster of eight nodes or eight hundred nodes. Once the base repository is set up and maintained, installation is turning on a node, reinstallation is rebooting a node, software maintenance is doing nothing (the node updates itself nightly) or doing a rare cluster-push to update in real time. These things do take time, but the BIGGEST time is setting up the repository and configuring a reasonable node image in the first place (neither of which are terribly hard, as one can rsync a repository mirror from many existing repositories and with a repository in hand a node image is, fundamentally, a list of rpm's that a tool like yum can be used to consistently install on top of a very minimal base image). On a successful cluster design the average non-hardware per-node FTE effort spent on the nodes themselves should be order of minutes per year. Again, this doesn't mean that your product packaging is without value -- I'm sure that it is engineered to help neophytes who are unlikely to achieve that level of efficiency initially get there quickly, and there is indeed value in providing people with a repository image to mirror including regular updates of the important core packages (as I said, this is what I think you should sell). It's just that ANY of these packagings compete with opportunity cost time available in many environments, alternative more or less prebuilt or easy to build cluster packages (oscar rocks scyld clustermatic...), and BECAUSE it is a big toolset so that one size fits all, contains a lot of tools that many cluster people will never need or use (just like a swiss army knife). Some people just want to whittle wood and clean fish and don't NEED a magnifying glass on their pocket knife! They are going to be asking themselves why they are "paying" for all these things when they don't use but three of them they could build themselves in an afternoon (or mirror prebuilt from various public repositories). This is what I think will ultimately limit your price point and needs to shape your marketing strategy. Your choice is ultimately going to be fewer high margin sales or more low margin sales (I'm pretty safe there, supply and demand law and all that:-). This is true for all the linux distributions. Or you can try some sort of mixed strategy (as was suggested earlier) and sell at all price points trying to leverage additional margin on the basis of added value and willingness to pay and service/consulting/support. In my opinion (for what it is worth) it would be a capital mistake for a distribution to seek to armtwist high margins in the linux community. As in the last mistake a lot of distributions may ever make. The problem is that the reason that there ARE distributions at all is largely because of the difficulty, once upon a time, of an ordinary human obtaining all the components required or desired to assemble a usable system. The distribution companies "sold" media where the whole shebang was right there, packaged to be "easy to install", and (as time passed) with package management systems and gradually improving installation and administrative interfaces. However, the interesting thing about software and the evolution of hardware and the internet is that the once a GPL package exists, it is available forever and eventually gets to be feature stable and readily available in a stable, functional form to pretty much anybody. New hardware (like PXE) has changed the installation equation. And the internet is now supported by new tools such as rsync, with massive amounts of storage routinely available. If I can afford a 150 GB RAID 5 system to serve my home LAN (and I just built one for about $800) then storing an utterly exhaustive linux repository isn't a big deal anymore. These things CAN change the way linux distribution operates altogether. They ARE changing the way linux distribution operates altogether. Nobody buys distribution CD's, they mirror. In professional environments, booting from floppy has all but disappeared. Packages are designed a priori to permit simple rebuilds and dependency resolution. I could >>really easily see<< an environment in five years where linux distribution is >>nothing but a network of repositories<< driven by tools like yum, augmented so that the rpm's are rebuilt as necessary in real time. Install a very primitive, pure kernel+gnu base, then yum-build from SOURCE repositories into a known final state. Right now there is little incentive for this to be built, at least not in a hurry. However, high margin price points would be a great incentive. Already half the systems people I know are wandering around moaning (and irritated), and these are the very people to do something about it if they get irritated enough. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Fri Sep 26 11:25:07 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Fri, 26 Sep 2003 17:25:07 +0200 Subject: Why MDKC & CLIC are not comparable to RH Advanced [Was Re: Redhat Fedora] In-Reply-To: References: Message-ID: <1064589906.3078.282.camel@revolution.mandrakesoft.com> [...] > Again, this doesn't mean that your product packaging is without value -- > I'm sure that it is engineered to help neophytes who are unlikely to > achieve that level of efficiency initially get there quickly, and there > is indeed value in providing people with a repository image to mirror > including regular updates of the important core packages (as I said, > this is what I think you should sell). [...] You are the kind of users which are able to setup a full cluster with a Linux From Scratch with a repository and automated building tools. Our product is aimed for people who don't have the time or don't want to setup such architecture. Obsviously we make cuts in our prices for educational research labs. Best regards, -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From adm35 at georgetown.edu Fri Sep 26 22:03:26 2003 From: adm35 at georgetown.edu (adm35 at georgetown.edu) Date: Fri, 26 Sep 2003 22:03:26 -0400 Subject: Redhat Fedora Message-ID: <24a6532445ee.2445ee24a653@georgetown.edu> It just seems to me that the entire question is less than relevant for so many of us. In many cases, we're *compelled* to use one particular distro over all others, because the researchers we support are using binary versions of applications that require a particular distro, which is frequently RedHat. I actually discovered one today that REQUIRES RedHat 8 or better!! You can all imagine that I was less than amused. We generally use OSCAR on RedHat, but not for any particular political reason. It works, and all of our applications run on it. If we had the luxury of having ready access to the source code for all of our applications, we might look elsewhere....or we might not. Some of you have the luxury of having all your programs created locally. I think many, perhaps most, of the rest of us do not. As long as I remain at the mercy of software vendors for some percentage of our computational applications, I'll probably stay with RedHat, questionable policies or not. Arnie Miles Systems Administrator: Advanced Research Computing Adjunct Faculty: Computer Science 202.687.9379 168 Reiss Science Building http://www.georgetown.edu/users/adm35 http://www.guppi.arc.georgetown.edu ----- Original Message ----- From: Bob Drzyzgula Date: Friday, September 26, 2003 3:43 pm Subject: Re: Redhat Fedora > On Fri, Sep 26, 2003 at 10:53:12AM -0700, Greg Lindahl wrote: > > > > On Wed, Sep 24, 2003 at 07:44:42PM -0400, Bob Drzyzgula wrote: > > > > > Actually, I thought that in fact "we" tended to do a lot of > > > this -- cf. Rocks, Clustermatic, Warewulf -- it's just that > > > these never seem to capture the market in the same way that > > > Red Hat always was. > > > > Are any of these really "distributions", or are they "a minimal > set of > > stuff overlaid on RedHat?" It makes sense to use a free and complete > > conventional distro to build a cluster distro -- but now the free > and> complete distro is splitting into the less-reliable and less- > fixed> distro (consumer) and the not-exactly-up-to-date distro > (enterprise),> which changes what we can do cheaply. > > > Yes, on looking closer I see that you're right about the > layering thing. I saw ISOs on those sites and hadn't looked > more deeply. > > You're also right that the way out of this problem > isn't clear. I suppose that if Fedora works out better > than most of us are probably expecting, then there'd be > hope there. It also occurs to me that it may make sense > for the cluster community to consider something like a > Cluster-Fedora subproject; perhaps we could come up with > something that wasn't a cluster-somethingelsestartingwithf > (sorry). Possibly Debian would be a better base in the > long term, not that I'm particularly fond of Debian. > > Possibly, given that, as Erwan said, the CLIC contract > is up in December 2003, perhaps it would make more sense > for the community to pick that up as a starting point. > > But I suppose this is a bridge we'll have to cross when > we come to it, especially since the bridge doesn't even > seem to be fully built as yet. > > --Bob > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Sat Sep 27 03:11:42 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Sat, 27 Sep 2003 00:11:42 -0700 Subject: Redhat Fedora In-Reply-To: <24a6532445ee.2445ee24a653@georgetown.edu> References: <24a6532445ee.2445ee24a653@georgetown.edu> Message-ID: <20030927071142.GA1838@greglaptop.greghome.keyresearch.com> On Fri, Sep 26, 2003 at 10:03:26PM -0400, adm35 at georgetown.edu wrote: > In many cases, we're *compelled* to use one particular > distro over all others, because the researchers we support are using > binary versions of applications that require a particular distro, which > is frequently RedHat. I actually discovered one today that REQUIRES > RedHat 8 or better!! You can all imagine that I was less than amused. In the future, RedHat is trying to ensure that these apps require RedHat Enterprise, not the consumer version. So "we"'re going to have to deal with that, one way or another. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel.pfenniger at obs.unige.ch Sat Sep 27 04:44:42 2003 From: daniel.pfenniger at obs.unige.ch (Daniel Pfenniger) Date: Sat, 27 Sep 2003 10:44:42 +0200 Subject: Redhat Fedora In-Reply-To: <24a6532445ee.2445ee24a653@georgetown.edu> References: <24a6532445ee.2445ee24a653@georgetown.edu> Message-ID: <3F754DFA.707@obs.unige.ch> adm35 at georgetown.edu wrote: It just seems to me that the entire question is less than relevant for so many of us. In many cases, we're *compelled* to use one particular distro over all others, because the researchers we support are using binary versions of applications that require a particular distro, which is frequently RedHat. I actually discovered one today that REQUIRES RedHat 8 or better!! You can all imagine that I was less than amused. I have seen the problem for the driver of InfiniBand cards made by Mellanox and distributed by various companies. The driver source code is of course not available, so the Infiniband companies provide binary drivers for certain RedHat and Suse kernels. Because InfiniBand is recent on the market, it is likely that new clusters aimed at high performance computing use freshly made components, so the kernels are rapidly not compatible with both the binary drivers and the recent kernels. A solution that at least one company was offering is to custom compile the driver with the particular customer kernel. The customer sends the kernel source with the custom config file and the company sends back the compatitble binary driver. Such solutions seem to me the most convenient for having coexisting open and closed source software. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From toon at moene.indiv.nluug.nl Sat Sep 27 15:57:24 2003 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Sat, 27 Sep 2003 21:57:24 +0200 Subject: Redhat Fedora References: <24a6532445ee.2445ee24a653@georgetown.edu> Message-ID: <3F75EBA4.5040001@moene.indiv.nluug.nl> adm35 at georgetown.edu wrote: > It just seems to me that the entire question is less than relevant for so > many of us. In many cases, we're *compelled* to use one particular > distro over all others, because the researchers we support are using > binary versions of applications that require a particular distro, which > is frequently RedHat. In that case, why use Free Software - you could just as well shell out the bucks and go for a proprietary solution. Note that Free in Free Software stands for freedom. If you just want to use gratis software, be my guest, but do not complain. -- Toon Moene - mailto:toon at moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc-g95.sourceforge.net/ (under construction) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amacater at galactic.demon.co.uk Sun Sep 28 16:23:57 2003 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Sun, 28 Sep 2003 20:23:57 +0000 Subject: Redhat Fedora In-Reply-To: <3F75EBA4.5040001@moene.indiv.nluug.nl> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> Message-ID: <20030928202357.GA2644@galactic.demon.co.uk> On Sat, Sep 27, 2003 at 09:57:24PM +0200, Toon Moene wrote: > adm35 at georgetown.edu wrote: > > >It just seems to me that the entire question is less than relevant for so > >many of us. In many cases, we're *compelled* to use one particular > >distro over all others, because the researchers we support are using > >binary versions of applications that require a particular distro, which > >is frequently RedHat. > > In that case, why use Free Software - you could just as well shell out > the bucks and go for a proprietary solution. > Indeed: by demanding Red Hat / SuSE you may already be using a proprietary solution (since you can't redistribute their Linux versions). If your vendor only supplies binaries which are vendor/version locked - strongly consider changing your vendor. If your vendor insists they need to supply binaries: a.) Get them to take the risk of keeping up to date with Fedora and hold them to it. b.) Get them to port the code to any LSB compliant distribution / remove kernel version specific stuff. c.) Ask them to supply you with the programming hooks necessary to use the binaries if your vendor disappears / pulls out of the Linux business. d.) If in Europe at least - advise them that you will feel free to reverse engineer their binaries to ensure onward compatibility / interoperability with other software. Remember folks: This is Linux (GNU/Linux for the pedantic :) ) - we have access to the underlying code and will continue so to do. If need be, as a community, we can kludge packages together to ensure binary compatibility for obsolete code [libc4 is still in Debian for just this reason - obsolete commercial packages depend on it] / write wrapper scripts / whatever. I have a slight residual personal interest in this: a long time ago, I wanted to port the original Extreme Linux stuff to work on a Debian system and it was hard because it was all .rpm based and predicated RH. Eventually, I managed to package LAM (before passing it off onto a much better maintainer - Hi Camm {B-) ) but gave up on several of the smaller bits because (a) Some of them were obviously RH specific / (b) Some of them appeared to be commercial software. At this stage, some years later, I can't recall exact specifics but it was enough to put me off "based on specific distribution" type hacks. Can I commend Debian to the list for serious consideration for future projects / clusters. a) It runs on virtually anything from PDA's to Alphas / Sparcs / IBM big iron / ia64 / Opteron b) Debian main is free and open and all source is available, modifiable and freely distributable. c) There are no licence costs per node. Just my $0.02 US (saving $149.98 or so per node ...) Andy _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amacater at galactic.demon.co.uk Sun Sep 28 16:23:57 2003 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Sun, 28 Sep 2003 20:23:57 +0000 Subject: Redhat Fedora In-Reply-To: <3F75EBA4.5040001@moene.indiv.nluug.nl> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> Message-ID: <20030928202357.GA2644@galactic.demon.co.uk> On Sat, Sep 27, 2003 at 09:57:24PM +0200, Toon Moene wrote: > adm35 at georgetown.edu wrote: > > >It just seems to me that the entire question is less than relevant for so > >many of us. In many cases, we're *compelled* to use one particular > >distro over all others, because the researchers we support are using > >binary versions of applications that require a particular distro, which > >is frequently RedHat. > > In that case, why use Free Software - you could just as well shell out > the bucks and go for a proprietary solution. > Indeed: by demanding Red Hat / SuSE you may already be using a proprietary solution (since you can't redistribute their Linux versions). If your vendor only supplies binaries which are vendor/version locked - strongly consider changing your vendor. If your vendor insists they need to supply binaries: a.) Get them to take the risk of keeping up to date with Fedora and hold them to it. b.) Get them to port the code to any LSB compliant distribution / remove kernel version specific stuff. c.) Ask them to supply you with the programming hooks necessary to use the binaries if your vendor disappears / pulls out of the Linux business. d.) If in Europe at least - advise them that you will feel free to reverse engineer their binaries to ensure onward compatibility / interoperability with other software. Remember folks: This is Linux (GNU/Linux for the pedantic :) ) - we have access to the underlying code and will continue so to do. If need be, as a community, we can kludge packages together to ensure binary compatibility for obsolete code [libc4 is still in Debian for just this reason - obsolete commercial packages depend on it] / write wrapper scripts / whatever. I have a slight residual personal interest in this: a long time ago, I wanted to port the original Extreme Linux stuff to work on a Debian system and it was hard because it was all .rpm based and predicated RH. Eventually, I managed to package LAM (before passing it off onto a much better maintainer - Hi Camm {B-) ) but gave up on several of the smaller bits because (a) Some of them were obviously RH specific / (b) Some of them appeared to be commercial software. At this stage, some years later, I can't recall exact specifics but it was enough to put me off "based on specific distribution" type hacks. Can I commend Debian to the list for serious consideration for future projects / clusters. a) It runs on virtually anything from PDA's to Alphas / Sparcs / IBM big iron / ia64 / Opteron b) Debian main is free and open and all source is available, modifiable and freely distributable. c) There are no licence costs per node. Just my $0.02 US (saving $149.98 or so per node ...) Andy _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From thornton at yoyoweb.com Mon Sep 29 00:07:52 2003 From: thornton at yoyoweb.com (Thornton Prime) Date: Sun, 28 Sep 2003 21:07:52 -0700 Subject: Redhat Fedora In-Reply-To: <20030928202357.GA2644@galactic.demon.co.uk> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> <20030928202357.GA2644@galactic.demon.co.uk> Message-ID: <1064808472.16662.12.camel@localhost.localdomain> > Indeed: by demanding Red Hat / SuSE you may already be using a > proprietary solution (since you can't redistribute their Linux > versions). Why do you say this? To my knowledge, the only restrictions either of these distributions have (or can have) on re-distribution are trademark restrictions involving the use of their distribution name or logo. You can't say that a re-distribution of Red Hat is Red Hat, but the code is completely open to distribution -- in both source and binary form (you are required by GPL to make the source available to whomever you distribute to). I do agree that we should press vendors to make their software vendor neutral. thornton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Sep 29 02:51:11 2003 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 29 Sep 2003 08:51:11 +0200 (CEST) Subject: Redhat Fedora In-Reply-To: <20030928202357.GA2644@galactic.demon.co.uk> Message-ID: On Sun, 28 Sep 2003, Andrew M.A. Cater wrote: > a.) Get them to take the risk of keeping up to date with Fedora and hold > them to it. > > b.) Get them to port the code to any LSB compliant distribution / remove > kernel version specific stuff. > > Remember folks: This is Linux (GNU/Linux for the pedantic :) ) - we have > access to the underlying code and will continue so to do. If need be, I agree with a lot of what Andrew says - though I'm not that much of a Debian fan (*). Linux is much more than another operating system. Many scientists, engineers and users in business and the arts do not see it that way. Linux has become popular, on cheap yet high performance kit. All many people now care about is 'getting the job done' or 'running my codes'. We should realise there is much to be gained from open standards and LSB compliant systems. (*) Not ideology - I've just used other distros. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Sep 29 05:31:23 2003 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 29 Sep 2003 11:31:23 +0200 (CEST) Subject: Redhat Fedora In-Reply-To: <1064808472.16662.12.camel@localhost.localdomain> Message-ID: On Sun, 28 Sep 2003, Thornton Prime wrote: > Why do you say this? To my knowledge, the only restrictions either of > these distributions have (or can have) on re-distribution are trademark > restrictions involving the use of their distribution name or logo. Indeed - I use Pink Tie Linux at home. This is a distro supplied on CD-ROM which is equivalent to RH. The system I'm using at the moment started life as a Pink Tie 8.0 I've since used apt-get dist-upgrade to upgrade it to a 9.0 (Sorry Bob - not YUM.) I'm still a bit proud of that - dist-upgrading a RH system. There's a bit of a discussion on the Fedora list about dist-upgrade to the Severn beta. But I think I'll be safe here and run that up on another system. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rauch at inf.ethz.ch Mon Sep 29 10:38:15 2003 From: rauch at inf.ethz.ch (Felix Rauch) Date: Mon, 29 Sep 2003 16:38:15 +0200 (CEST) Subject: Cluster distributions? In-Reply-To: <1064217163.15240.6.camel@revolution.mandrakesoft.com> Message-ID: On Mon, 22 Sep 2003, Erwan Velu wrote: > That one of our stronger point, the ka technologies from IDIMAG allow us > to deploy new nodes within 3 minutes (duplication, reoboot, autoconfig). While 3 minutes sounds impressive, the time also depends on the amount of data that you transfer. > This tools helps you to keep your nodes up to date. Urpmi parallel uses > the ka technologies so the time needed for installing 50MB on a cluster > doesn't depend on the cluster size ! It takes the same time for installing > on 2 or 50 nodes ! So just have an switched network (very common this > days). How do you do it? With the multi-drop chain similar to Dolly? Regards, Felix -- Felix Rauch | Email: rauch at inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H16 | Phone: +41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: +41 1 632 1307 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andy.schwierskott at sun.com Mon Sep 29 12:28:11 2003 From: andy.schwierskott at sun.com (Andy Schwierskott) Date: Mon, 29 Sep 2003 18:28:11 +0200 (MEST) Subject: AW: AW: [GE users] Scheduling In-Reply-To: <0DDE7F776936B04AA670C77F1F59A4D909D2BC@deaex001.arvinmeritor.com> References: <0DDE7F776936B04AA670C77F1F59A4D909D2BC@deaex001.arvinmeritor.com> Message-ID: Mark, job_load_adjustments are applied after a job has been dispatched. So the first job should go into the queue, but not a seconds one immediately after. >From SGE 5.3p3 you see SGE's internal bookkepping when a queue is in (virtual) alarm state with # qstat -j It shows the real load and the job_load_adjustments what the scheduler used in the last scheduling run. Andy On Mon, 29 Sep 2003, Olesen, Mark wrote: > More scheduling woes ... > > I am now completely confused about how the scheduler is supposed to work. > In accordance with sched_conf(5), I had assumed that the queues' > "load_thresholds" would be considered during scheduling, but I'm seeing > *somewhat* different behaviour. > > For all my queues, I've set > load_thresholds np_load_avg=1.2 > > As previously mentioned, I am sorting by seqno: > algorithm default > schedule_interval 0:0:15 > maxujobs 0 > queue_sort_method seqno > user_sort true > job_load_adjustments np_load_avg=0.5 > load_adjustment_decay_time 0:7:30 > load_formula np_load_avg > schedd_job_info true > > > The next queue in the sequence with any free slots has load_avg=2.05, or > np_load_avg=1.03 > > I would imagine that the scheduler predicts a new np_load_avg=1.53 (current > + joad_load_adjustments). Since this predicted load is above the queue load > threshold, I would not expect a new job to start on this queue. > Contrary to my expectations, a new job does indeed get sent to this queue. > > What other tricks do I need in order to coerce the scheduler to respect the > load thresholds? > > Cheers, > > > Dr. Mark Olesen > Thermofluid Dynamics Analyst > ArvinMeritor Light Vehicle Systems > Zeuna Staerker GmbH & Co. KG > Biberbachstr. 9 > D-86154 Augsburg, GERMANY > tel: +49 (821) 4103 - 862 > fax: +49 (821) 4103 - 7862 > Mark.Olesen at ArvinMeritor.com --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net For additional commands, e-mail: users-help at gridengine.sunsource.net From erwan at mandrakesoft.com Sun Sep 28 06:34:47 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Sun, 28 Sep 2003 12:34:47 +0200 (CEST) Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <3F760FBE.3050409@ucia.gov> References: <3F760FBE.3050409@ucia.gov> Message-ID: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> > We will stay with ROCKS for our initial deployment of 320 Opteron > system. It is due in about 60 days... Does ROCKS supports Opteron ? I never saw this announcement... So do you intend using ROCKS using an x86 OS ? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 29 04:57:31 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 29 Sep 2003 10:57:31 +0200 Subject: Redhat Fedora In-Reply-To: <20030926154351.M3752@www2> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> <20030926175312.GA2187@greglaptop.internal.keyresearch.com> <20030926154351.M3752@www2> Message-ID: <1064825850.3580.11.camel@revolution.mandrakesoft.com> > Possibly, given that, as Erwan said, the CLIC contract > is up in December 2003, perhaps it would make more sense > for the community to pick that up as a starting point. That's a very good idea, it's really make sense to continue the CLIC project. The funds for CLIC ends at December 2003 but we are ready to continue this project with people whom are intrested in. CLIC could be a really good basis for a Free clustering distribution. The new CLIC version will be available in few days and then all the basic tools & distribution will be ready for a new start. What does "Beowulf" people think about it ? -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Mon Sep 29 08:03:22 2003 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Mon, 29 Sep 2003 14:03:22 +0200 Subject: Q: Building a small machine room? Materials/costs/etc. In-Reply-To: <3F6B4D4C.8070506@moene.indiv.nluug.nl> (Toon Moene's message of "Fri, 19 Sep 2003 20:39:08 +0200") References: <3F6B4D4C.8070506@moene.indiv.nluug.nl> Message-ID: Toon Moene writes: > Robert G. Brown wrote: > >> We have some of these (including the big red switch by the door) but not >> all. Maybe soon -- the first is really a matter of sensors and scripts. > > Do you also have the "door-opening-switch" right next to the BRS ? > That can lead to very funny mistakes (especially if you're not the one > making the mistake). Couple of years ago, when my employer-at-the-time were building new premises, I walked into the computer room-to-be just as the electricians were finishing the installation of the Big Red Button. In their wisdom they had placed it by the door, between the light switch and the door lock switch. Fortunately, I was authorized to issue Change Orders, so the electricians' Orders were Changed *real* quick. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alberor at ucia.gov Sun Sep 28 09:39:56 2003 From: alberor at ucia.gov (Al R) Date: Sun, 28 Sep 2003 09:39:56 -0400 Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> References: <3F760FBE.3050409@ucia.gov> <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> Message-ID: <3F76E4AC.7070100@ucia.gov> Erwan, Our initial deployment will utilize the Opterons in 32-bit mode. Our current software baseline is 32-bit plus legacy software spanning 20+ years of development. Until we are given compiled 64-bit versions, and no vendor wants to release their source code, nor do WE want to port their code, we will simply stay at 32-bit. No problem. Our first upgrade will be an in-rack upgrade. This will be a software reload taking us to 64-bit. We perform 64-bit arithmetic, but not in a 64-bit address space. None of applications currently require this. In the next 18-36 months, yes, there may be a change. We will be ready. Let's face it, it is a tough choice to make, 32 vs 64 these days. Especially for a cluster "of size". ROCKS is so easy to use, that we have actually cut our cluster in half in support of special processing. We will be doing this with our system. The initial 120 nodes (240 processors) will be connected via Myrinet. We received additional funding for another rack of 40 nodes (+80 processors) and found the enormous cost of "Myrinet in excess of 128 nodes". We decided to let the single frontend node manage this rack/nodes, and let the PBS quename determine the need/use of these nodes. Or, we can simply make the bottom node another frontend and have it manage its own rack of 39 dual-Opteron compute nodes. It also makes a darn good "staging cluster" so software can be tested and then ported to the other side... ROCKS is simply that flexible... Take 20 nodes out of your cluster, place two on a desk for 10 people. Teach them about ROCKS and parallel processing using two nodes and a crossover cable. Put the nodes back in the rack and in 5-8 minutes reload all 20 nodes back as compute nodes... WHILE THE ORIGINAL CLUSTER IS STILL RUNNING. No impacts to operations. Just my 2 cents as a ROCKS user for 18 months. Al Erwan Velu wrote: >>We will stay with ROCKS for our initial deployment of 320 Opteron >>system. It is due in about 60 days... > > > Does ROCKS supports Opteron ? I never saw this announcement... > So do you intend using ROCKS using an x86 OS ? > > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 29 10:59:23 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 29 Sep 2003 16:59:23 +0200 Subject: Cluster distributions? In-Reply-To: References: Message-ID: <1064847562.5365.23.camel@revolution.mandrakesoft.com> > While 3 minutes sounds impressive, the time also depends on the amount > of data that you transfer. This system reach 10-11MB/sec for the harddrive duplication + ~ 1 minute for launching the minimal linux on the nodes which must be deployed and run the appropriate tools before & after the deployment. The number of nodes don't change the amount of time needed for the deployment. > How do you do it? With the multi-drop chain similar to Dolly? Yes that is the same kind of technology but it doesn't allow channel bonding like dolly. When the chain is initialised, the server sends packets to the first node which handle it and forward it to the next one. If it can't handle this packet the node is buffering its inputs and the server is buffering its outputs. Urpmi parallel is a kid of "apt" mixed with a multi-drop chain tool. Installing packages with dependencies on a full cluster is really impressive :D -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alberor at ucia.gov Sun Sep 28 11:38:48 2003 From: alberor at ucia.gov (Al R) Date: Sun, 28 Sep 2003 11:38:48 -0400 Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <1537.81.57.165.53.1064767549.squirrel@webmail.mandrakesoft.com> References: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> <3F76E4AC.7070100@ucia.gov> <1537.81.57.165.53.1064767549.squirrel@webmail.mandrakesoft.com> Message-ID: <3F770088.3060206@ucia.gov> We were in an instructional mode. Each student had to build a frontend and a compute node. It was also done in a classroom. The students were to format the drives and other crash and burn activities. :-) This was a hands-on lab, with out of body (rack) experience. Thank you for the reference in your note below. I will go take a look at this! Al Erwan Velu wrote: >>Erwan, > > [...] > >>ROCKS is simply that flexible... Take 20 nodes out of your cluster, >>place two on a desk for 10 people. Teach them about ROCKS and parallel >>processing using two nodes and a crossover cable. Put the nodes back in >>the rack and in 5-8 minutes reload all 20 nodes back as compute nodes... >>WHILE THE ORIGINAL CLUSTER IS STILL RUNNING. No impacts to operations. > > > For that kind of applications, our choice was just making a subcluster > using MAUI. You can tell this 20 nodes to be a subcluster ( a partition) > called subcluster1, then you can assign users to this partition using > DrakCluster GUI. > > After that when people submit jobs via PBS theirs jobs are sent to this > subcluster. At the end of this session, you can reintegrate this nodes to > the original cluster in the same range of time you mention. > > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Sun Sep 28 12:45:49 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Sun, 28 Sep 2003 18:45:49 +0200 (CEST) Subject: Comparisions,, - use ROCKS 2.3.2 In-Reply-To: <3F76E4AC.7070100@ucia.gov> References: <32889.81.57.165.53.1064745287.squirrel@webmail.mandrakesoft.com> <3F76E4AC.7070100@ucia.gov> Message-ID: <1537.81.57.165.53.1064767549.squirrel@webmail.mandrakesoft.com> > Erwan, [...] > ROCKS is simply that flexible... Take 20 nodes out of your cluster, > place two on a desk for 10 people. Teach them about ROCKS and parallel > processing using two nodes and a crossover cable. Put the nodes back in > the rack and in 5-8 minutes reload all 20 nodes back as compute nodes... > WHILE THE ORIGINAL CLUSTER IS STILL RUNNING. No impacts to operations. For that kind of applications, our choice was just making a subcluster using MAUI. You can tell this 20 nodes to be a subcluster ( a partition) called subcluster1, then you can assign users to this partition using DrakCluster GUI. After that when people submit jobs via PBS theirs jobs are sent to this subcluster. At the end of this session, you can reintegrate this nodes to the original cluster in the same range of time you mention. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Mon Sep 29 17:12:44 2003 From: erwan at mandrakesoft.com (Erwan Velu) Date: Mon, 29 Sep 2003 23:12:44 +0200 (CEST) Subject: Redhat Fedora In-Reply-To: <3F78773B.6020209@obs.unige.ch> References: <20030924211549.GC2061@greglaptop.internal.keyresearch.com> <20030924194442.H3752@www2> <20030926175312.GA2187@greglaptop.internal.keyresearch.com> <20030926154351.M3752@www2> <1064825850.3580.11.camel@revolution.mandrakesoft.com> <3F78773B.6020209@obs.unige.ch> Message-ID: <1164.81.57.165.53.1064869964.squirrel@webmail.mandrakesoft.com> > I am just now installing a new cluster of dual Xeon and want to try CLIC. > However my hardware is too new and most of the chipsets are > not recognized by Mandrake 9.0, but works fine with Mandrake 9.1. > When will be the new CLIC version available? Note that I would be glad > to be a beta tester if you wish to let me download a > preliminary CLIC version. However if "a few days" mean 3-4 days > I would agree to wait. We know that CLIC is now quite obsolete due to a quite old kernel. I've just received the newest kernel (2.4.22) by my kernel team. We had to integrate it ASAP. Our schedule had planned the newest CLIC for the end of this week. So please wait for it. I'll keep you in touch when the iso is ready. You can also join our Mailing list, you can find explanations at http://clic.mandrakesoft.com In addition, you'll find the newest backend for an easiest installation & management process. Best regards, -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From michael.worsham at mci.com Mon Sep 29 14:44:34 2003 From: michael.worsham at mci.com (Michael Worsham) Date: Mon, 29 Sep 2003 14:44:34 -0400 Subject: Power Supply: Supermicro P4DL6 Board? Message-ID: <009a01c386b9$b9f55280$c96432a6@Wcomnet.com> Anyone know a good place to pick up a power supply that is supported for Supermicro's P4DL6 dual xeon motherboard? Finding normal P4 power supplies are a dime a dozen, but a P/S with both a 20pin + 8pin connector is hard to find and usually about as expensive as the case needed for the board. Thanks. -- M _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From sean at asacomputers.com Mon Sep 29 19:09:54 2003 From: sean at asacomputers.com (Sean) Date: Mon, 29 Sep 2003 16:09:54 -0700 Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: <009a01c386b9$b9f55280$c96432a6@Wcomnet.com> Message-ID: <5.1.0.14.2.20030929160429.02e94ec0@pop.asacomputers.com> Michael, You need a minimum 400W ATX12 power supply. We normally recommend 460 W power supply considering the increasing number of hard drives that go with the system recently. Supermicro has their own 460W PS, Supermicro 760P4 Power supply PWS-024 you can either choose this one or go with other power supply vendors like PC Power and Cooling or Enermax. At 02:44 PM 9/29/03 -0400, Michael Worsham wrote: >Anyone know a good place to pick up a power supply that is supported for >Supermicro's P4DL6 dual xeon motherboard? Finding normal P4 power supplies >are a dime a dozen, but a P/S with both a 20pin + 8pin connector is hard to >find and usually about as expensive as the case needed for the board. > >Thanks. > >-- M > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf Thanks and Regards Sean ASA Computers Inc. 2354, Calle Del Mundo Santa Clara CA 95054 Telephone : (408) 654-2901 xtn 205 (408) 654-2900 ask for Sean (800) REAL-PCS (1-800-732-5727) Fax: (408) 654-2910 E-mail : sean at asacomputers.com URL : http://www.asacomputers.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at keyresearch.com Mon Sep 29 19:45:53 2003 From: lindahl at keyresearch.com (Greg Lindahl) Date: Mon, 29 Sep 2003 16:45:53 -0700 Subject: HyperTransport note In-Reply-To: References: <20030910163913.GA3486@greglaptop.greghome.keyresearch.com> Message-ID: <20030929234553.GC2450@greglaptop.internal.keyresearch.com> We were talking about HyperTransport a while back. The HT consortium has issued a press release about HT 1.1, saying that it includes technology sufficient for reliable transport over HT. I dunno when anyone is going to implement it in HW, but they're headed in the right direction. Let's see, what else: packet handling, peer-to-peer routing, and virtual channels for streaming data. Fun stuff. http://www.hypertransport.org/pr_092903.htm -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 29 20:29:22 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 30 Sep 2003 10:29:22 +1000 Subject: Redhat Fedora In-Reply-To: <20030927071142.GA1838@greglaptop.greghome.keyresearch.com> References: <24a6532445ee.2445ee24a653@georgetown.edu> <20030927071142.GA1838@greglaptop.greghome.keyresearch.com> Message-ID: <200309301029.24376.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 27 Sep 2003 05:11 pm, Greg Lindahl wrote: > In the future, RedHat is trying to ensure that these apps require > RedHat Enterprise, not the consumer version. I suspect this will also be the case for hardware vendors as well, in that they are likely to say the supported Linux configuration will be RHEL or SLES. So there will basically be the choice of running either RHEL or SLES and getting support, or not and being on your own. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/eM5iO2KABBYQAh8RAv/wAJ9I+OFyUifmwFV18n9/8W9u6I93ZQCfQph1 l71u5hFGAAIBrtyeU9f6iKc= =INHN -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 29 20:38:47 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 30 Sep 2003 10:38:47 +1000 Subject: Redhat Fedora In-Reply-To: <1064808472.16662.12.camel@localhost.localdomain> References: <24a6532445ee.2445ee24a653@georgetown.edu> <20030928202357.GA2644@galactic.demon.co.uk> <1064808472.16662.12.camel@localhost.localdomain> Message-ID: <200309301038.49590.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 29 Sep 2003 02:07 pm, Thornton Prime wrote: > Why do you say this? To my knowledge, the only restrictions either of > these distributions have (or can have) on re-distribution are trademark > restrictions involving the use of their distribution name or logo. RHEL contains trademarked Red Hat material. You need to remove this to be able to redistribute it yourself. SLES includes non-GPL software from SuSE. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD4DBQE/eNCXO2KABBYQAh8RAq73AJY5lKm6jRB8/E7tkc8gU5u3pI5kAJ43ZfB/ 2QadBSgOfCNdiXwVDKJpnQ== =zoFi -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Mon Sep 29 20:36:50 2003 From: csamuel at vpac.org (Chris Samuel) Date: Tue, 30 Sep 2003 10:36:50 +1000 Subject: Redhat Fedora In-Reply-To: <20030928202357.GA2644@galactic.demon.co.uk> References: <24a6532445ee.2445ee24a653@georgetown.edu> <3F75EBA4.5040001@moene.indiv.nluug.nl> <20030928202357.GA2644@galactic.demon.co.uk> Message-ID: <200309301036.54005.csamuel@vpac.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, 29 Sep 2003 06:23 am, Andrew M.A. Cater wrote: > Indeed: by demanding Red Hat / SuSE you may already be using a > proprietary solution (since you can't redistribute their Linux > versions). For RHEL though you can build a free version from source providing you remove the RH trademarked items. I don't believe this is possible for SLES as that contains proprietary code. > If your vendor only supplies binaries which are vendor/version locked - > strongly consider changing your vendor. The problem is that this sort of thing is such a niche market there is often only one vendor. The real issue isn't an overt vendor lock in, just that if you're not running what they support then you risk getting shrugged off with "we don't support that" if (when?) something goes wrong and need some help. Personally I like open source a lot, but in our line of work we have to provide for our users requirements or they'll go elsewhere. Chris - -- Christopher Samuel - (03)9925 4751 - VPAC Systems & Network Admin Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/eNAiO2KABBYQAh8RAqcKAJ0eBOIpk6Xoke2+hdFDvLDg/r9NYwCfUrkD 1iDVLxiGD/ru0cx6UV5Zl+s= =LoJf -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Mon Sep 29 19:43:58 2003 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Mon, 29 Sep 2003 16:43:58 -0700 (PDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: <009a01c386b9$b9f55280$c96432a6@Wcomnet.com> Message-ID: hi ya those 20pin + 8pinner is also dime a dozen ... if you're picky about reliability ... there's 2 choices of powersupplies i used in the past few years... sparklepower and zippy(aka emacs) - dont buy used power supplies or "repaired" power supplies for anything important list of power supply manufacturers http://www.Linux-1U.net/PowerSupp if you're local ( SJ or SD or LA ) we can just go pick it up ... c ya alvin On Mon, 29 Sep 2003, Michael Worsham wrote: > Anyone know a good place to pick up a power supply that is supported for > Supermicro's P4DL6 dual xeon motherboard? Finding normal P4 power supplies > are a dime a dozen, but a P/S with both a 20pin + 8pin connector is hard to > find and usually about as expensive as the case needed for the board. > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mitchel at navships.com Tue Sep 30 15:15:01 2003 From: mitchel at navships.com (Mitchel Kagawa) Date: Tue, 30 Sep 2003 09:15:01 -1000 Subject: Environment monitoring Message-ID: <005401c38787$25cad390$7001a8c0@Navatek.local> This past Friday night after everyone in the office went home our AC, for our small 64 node cluster, decided to have a compressor problem and stop producing cool air. On Sat I get a phone call saying that one of our users can't access his e-mail (server is also in cluster room). I wasn't able to get to the office until Sunday afternoon and I wasn't able to log into the head node to shut everything down remotely. By the time I was able to get to the cluster all but 20 nodes had shut themselves down because the ambient temperature of the room reached 145 deg. Thankfully all the servers, raid arrays and all but 2 nodes came back up when I hit the power button (after I got the temp back to 68). Our stupid ac doesn't have the communications package that cost an extra $5000 cause my company's too cheap to spring for it and an extra phone line. So I was wondering if anyone used one of those environment monitoring appliances that e-mail you (before stuff starts shutting off) if a temp goes out of a set range or if it detects water, high humidity, or movement in the room? I'm looking at a NetBotz/RackBots here (http://www.netbotz.com/products/rack.html) and was wondering if anyone has any experience with them or if you have any other (inexpensive <$2000) suggestions. Thanks for any help. Mitchel Kagawa _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From yudong at hsb.gsfc.nasa.gov Tue Sep 30 15:56:41 2003 From: yudong at hsb.gsfc.nasa.gov (Yudong Tian) Date: Tue, 30 Sep 2003 15:56:41 -0400 Subject: Upper bound on no. of sockets In-Reply-To: Message-ID: The number of sockets a process can open is limited by the number of file descriptors (fds). Type "ulimit -n" under bash to get this number, which usually 1024 by default. You can increase this number if you wish. Google "increase Linux file descriptors" you will find many examples, like this one: http://support.zeus.com/faq/zws/v4/entries/os/linuxfd.html If you want to be really sure, you can compile and run the following c program to get the number, which is the output plus 3 (stdout, stdin and stderr): /* # test how many fd you have $Id$ */ int main(int argc, char *argv[]) { int i = 0; while( tmpfile()){ i++; } printf("Your free fds: %d\n", i); } /*** end of program ***/ If you are running a TCP server and want to test how many clients that server can support, you can run the following perl program to test: #!/usr/bin/perl # Test the max number of tcp connections a server can support # $Id: testMaxCon.pl,v 1.1 2003/06/23 19:10:41 yudong Exp yudong $ # usage: testMaxCon.pl IP port use IO::Socket::INET; @ARGV == 2 or die "Usage: testMaxCon.pl svr_ip svr_port\n"; my ($ip, $port) = @ARGV; my $i = 0; do { $i++; $socket[$i] = IO::Socket::INET->new(PeerAddr => $ip, PeerPort => $port, Proto => "tcp", Timeout => 6, Type => SOCK_STREAM); } while ($socket[$i]); print "Max TCP connections supported for this client: ", $i-1, "\n"; ## end of program Of course for this test you have to make sure you have more fds to use than the server. Hope that helps. Yudong ------------------------------------------------------------ Falun Dafa: The Tao of Meditation (http://www.falundafa.org) ------------------------------------------------------------ Yudong Tian, Ph.D. NASA/GSFC (301) 286-2275 > -----Original Message----- > From: beowulf-admin at scyld.com [mailto:beowulf-admin at scyld.com]On Behalf > Of Balaji Rangasamy > Sent: Tuesday, September 30, 2003 12:44 AM > To: beowulf at beowulf.org > Subject: Upper bound on no. of sockets > > > Hi, > Is there an upper bound on the number of sockets that can be created by a > process? If there is one, is the limitation enforced by OS? And what other > factors does it depend on? Can you please be specific on the numbers for > different OS (RH Linux 7.2) ? > Thank you very much, > Balaji. > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Tue Sep 30 15:38:07 2003 From: dtj at uberh4x0r.org (Dean Johnson) Date: 30 Sep 2003 14:38:07 -0500 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <1064950687.10771.29.camel@caffeine> Bummer about the AC. You might want to go and check your backups and backup mechanism with a renewed zeal. Based on past experience you are in for an "interesting" near future. While they came back just fine, I wouldn't expect them to last very long after experiencing that high of temp. -Dean On Tue, 2003-09-30 at 14:15, Mitchel Kagawa wrote: > This past Friday night after everyone in the office went home our AC, for > our small 64 node cluster, decided to have a compressor problem and stop > producing cool air. On Sat I get a phone call saying that one of our users > can't access his e-mail (server is also in cluster room). I wasn't able to > get to the office until Sunday afternoon and I wasn't able to log into the > head node to shut everything down remotely. By the time I was able to get to > the cluster all but 20 nodes had shut themselves down because the ambient > temperature of the room reached 145 deg. Thankfully all the servers, raid > arrays and all but 2 nodes came back up when I hit the power button (after I > got the temp back to 68). Our stupid ac doesn't have the communications > package that cost an extra $5000 cause my company's too cheap to spring for > it and an extra phone line. So I was wondering if anyone used one of those > environment monitoring appliances that e-mail you (before stuff starts > shutting off) if a temp goes out of a set range or if it detects water, high > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 30 16:00:08 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 30 Sep 2003 16:00:08 -0400 (EDT) Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: On Tue, 30 Sep 2003, Mitchel Kagawa wrote: > This past Friday night after everyone in the office went home our AC, for > our small 64 node cluster, decided to have a compressor problem and stop > producing cool air. On Sat I get a phone call saying that one of our users > can't access his e-mail (server is also in cluster room). I wasn't able to > get to the office until Sunday afternoon and I wasn't able to log into the > head node to shut everything down remotely. By the time I was able to get to > the cluster all but 20 nodes had shut themselves down because the ambient > temperature of the room reached 145 deg. Thankfully all the servers, raid > arrays and all but 2 nodes came back up when I hit the power button (after I > got the temp back to 68). Our stupid ac doesn't have the communications > package that cost an extra $5000 cause my company's too cheap to spring for > it and an extra phone line. So I was wondering if anyone used one of those > environment monitoring appliances that e-mail you (before stuff starts > shutting off) if a temp goes out of a set range or if it detects water, high > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. There are two ways to proceed. I've used netbotz, and yes, they work fine and would have saved you here. As you note, a bit expensive...;-) The cheaper alternative is to DIY. There are several companies out there that sell temperature probes that can be read via a serial port. Buy one, hook it into a serial port of the machine of your choice, and write a script to sleep/poll the device, taking any action you like at any breakpoints you like. You can even add a PC-TV card and an X10 camera (for a total of about $100-150) and monitor the room remotely to get even more of the netbotz functionality. All total should cost you no more than $250-400 to DIY with camera. But yes, you have to have some skills here and there...:-) rgb > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 30 16:00:49 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 30 Sep 2003 16:00:49 -0400 (EDT) Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: On Tue, 30 Sep 2003, Mitchel Kagawa wrote: > suggestions. Thanks for any help. Oh, I might have forgotten to say: At least some of the thermal sensors are linked to brahma: www.phy.duke.edu/brahma rgb > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dritch at hpti.com Tue Sep 30 16:20:30 2003 From: dritch at hpti.com (David B. Ritch) Date: Tue, 30 Sep 2003 16:20:30 -0400 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <1064953230.4433.65.camel@localhost> I haven't used an environmental monitoring system quite that sophisticated, but I do have an APC UPS that supports environmental monitoring. I track it using BigBrother (http://bb4.com). You can also monitor temperatures on many motherboards using lm-sensors and a tool like BigBrother. dbr On Tue, 2003-09-30 at 15:15, Mitchel Kagawa wrote: > This past Friday night after everyone in the office went home our AC, for > our small 64 node cluster, decided to have a compressor problem and stop > producing cool air. On Sat I get a phone call saying that one of our users > can't access his e-mail (server is also in cluster room). I wasn't able to > get to the office until Sunday afternoon and I wasn't able to log into the > head node to shut everything down remotely. By the time I was able to get to > the cluster all but 20 nodes had shut themselves down because the ambient > temperature of the room reached 145 deg. Thankfully all the servers, raid > arrays and all but 2 nodes came back up when I hit the power button (after I > got the temp back to 68). Our stupid ac doesn't have the communications > package that cost an extra $5000 cause my company's too cheap to spring for > it and an extra phone line. So I was wondering if anyone used one of those > environment monitoring appliances that e-mail you (before stuff starts > shutting off) if a temp goes out of a set range or if it detects water, high > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. > > Mitchel Kagawa > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- David B. Ritch High Performance Technologies, Inc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Sep 30 16:05:02 2003 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 30 Sep 2003 16:05:02 -0400 (EDT) Subject: Environment monitoring In-Reply-To: <1064950687.10771.29.camel@caffeine> Message-ID: On 30 Sep 2003, Dean Johnson wrote: > Bummer about the AC. You might want to go and check your backups and > backup mechanism with a renewed zeal. Based on past experience you are > in for an "interesting" near future. While they came back just fine, I > wouldn't expect them to last very long after experiencing that high of > temp. Geeze, I forgot to add one more comment. We had just been talking about the time required for a server room to "get hot" upon the AC shutting down. Today was AC filter changing day here. Shut down blower unit (but not cluster) for about 7 minutes. In a room with at least 30 KW sustained, roughly 4 meters x 10 meters, ambient temperature rose perhaps 10F, just barely perceptibly (to maybe 75F) by the time the AC came back on -- warmer immediately behind a rack, of course. Lots of air, not that efficient a thermal transfer, lots of thermal ballast to absorb the heat. rgb > > -Dean > > On Tue, 2003-09-30 at 14:15, Mitchel Kagawa wrote: > > This past Friday night after everyone in the office went home our AC, for > > our small 64 node cluster, decided to have a compressor problem and stop > > producing cool air. On Sat I get a phone call saying that one of our users > > can't access his e-mail (server is also in cluster room). I wasn't able to > > get to the office until Sunday afternoon and I wasn't able to log into the > > head node to shut everything down remotely. By the time I was able to get to > > the cluster all but 20 nodes had shut themselves down because the ambient > > temperature of the room reached 145 deg. Thankfully all the servers, raid > > arrays and all but 2 nodes came back up when I hit the power button (after I > > got the temp back to 68). Our stupid ac doesn't have the communications > > package that cost an extra $5000 cause my company's too cheap to spring for > > it and an extra phone line. So I was wondering if anyone used one of those > > environment monitoring appliances that e-mail you (before stuff starts > > shutting off) if a temp goes out of a set range or if it detects water, high > > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > > any experience with them or if you have any other (inexpensive <$2000) > > suggestions. Thanks for any help. > > > > Mitchel Kagawa > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Tue Sep 30 17:28:55 2003 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Tue, 30 Sep 2003 17:28:55 -0400 (EDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: <3F79B423.AD3B5729@attglobal.net> Message-ID: > Has anyone recently analyzed how much power each of the subunits of a PC uses, i.e. how much power which > mainboard, hdd, etc. it's not hard to approximate this by reading the specs of each component. obviously, CPUs are the main consumers, probably followed by power hardware next (PS inefficiency, as well as the on-motherboard converter.) disks are much, much cooler than they used to be, probably dropping below power consumed by ram on most clusters. naturally, if you've got 6 15K RPM SCSI disks in a node with just 1G ram, that's completely different! even things like high-speed nics often amount to <10W, which simply doesn't compare to the ~80-90W per high-end CPU... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john at cusick.ws Tue Sep 30 18:54:14 2003 From: john at cusick.ws (John Cusick) Date: 30 Sep 2003 18:54:14 -0400 Subject: [tulip] 10BaseT/100BaseT Question In-Reply-To: References: Message-ID: <1064962453.5649.120.camel@sager.cusick.ws> On Tue, 2003-09-30 at 15:23, Donald Becker wrote: > On 30 Sep 2003, John Cusick wrote: > > On Mon, 2003-09-29 at 21:14, Donald Becker wrote: > > > On 23 Sep 2003, John Cusick wrote: > > > > > > > Card: Adaptec AN-8944A - 4 Port > > > > > > > > The card loads fine as a module and works, but I would like to know if > > > > its possible to force one of the ports to 10BaseT > > > What driver version are you using? > > > What is the detection message? > > > > Linux Tulip driver version 0.9.15-pre12 (Aug 9, 2002) > > Yup, that's the modified branch. It has lots of "issues" that my > version does not have. > > > tulip0: EEPROM default media type Autosense. > > tulip0: Index #0 - Media MII (#11) described by a 21140 MII PHY (1) > > block. > > tulip0: ***WARNING***: No MII transceiver found! > > That's a problem... > > > > If it's using the old driver included with the kernel, try the driver > > > from scyld.com > > > > > > mkdir /tmp/netdrivers/ > > > cd /tmp/netdrivers/ > > > ncftpget ftp://ftp.scyld.com/pub/network/netdrivers.tgz > > > tar xfvz netdrivers.tgz > > > make > > > make install > > > > > > > I've downloaded the newest version and it builds fine, so I attempted to > > use the spec file (I like to keep the rpm database up to date if > > possible, although its not critical, of course) and it also built fine > > except that it will not create the rpm's. > > That spec file is was original designed to be generic, but > as multiple incompatible RPM versions have been released and > the external module build system has changed without being fixed > we have given up and accepted that it will only work with our Linux > distribution. > > > It exits with the following error: > > > > Processing files: netdrivers-modules-3- at PACKAGE_RELEASE@.k2.4.20_20.7 > > error: File must begin with "/": %{prefix}/lib/modules/*/*/* > > You can work around this. Substitute your own release number for > @PACKAGE_RELEASE@ > Donald, I appreciate your help with this (I figured I had better say this before you decide that I'm a royal pain in the rear end). Talk about off topic... The error is cleaned up as far as @PACKAGE_RELEASE@ is concerned, however the final error still remains. I have gone through the Maximum RPM Book (which is pretty piss-poor as far as detail goes) and cannot figure out what is going on. Processing files: netdrivers-modules-3-5.2.4.20_20.7 error: File must begin with "/": %{prefix}/lib/modules/*/*/* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ RPM build errors: File must begin with "/": %{prefix}/lib/modules/*/*/* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ No rush on this, or even ignore it if you prefer since it is off-topic. I will install the drivers from the source for now to see if the previous problem is solved, and keep working on this one. If I get it done properly (tough considering the availability of documentation and my limited skills) then I will be sure to post to the list and make it available. ... snip ... > Note that this only works if > The driver implements the ioctl(), including supporting the virtual > registers with SYM transceivers. > The driver detects the transceiver (which is a problem here). Regards, John C. _______________________________________________ tulip mailing list, tulip at scyld.com To change to digest mode or unsubscribe visit http://www.scyld.com/mailman/listinfo/tulip From waitt at saic.com Tue Sep 30 18:00:20 2003 From: waitt at saic.com (Tim Wait) Date: Tue, 30 Sep 2003 18:00:20 -0400 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <3F79FCF4.9050004@saic.com> > humidity, or movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if anyone has > any experience with them or if you have any other (inexpensive <$2000) > suggestions. Thanks for any help. > > Mitchel Kagawa http://www.apcc.com/products/family/index.cfm?id=47&tab=models You can get a standalone unit that you can plug into your network and access via telnet or http, or if you have a SmartUPS unit, get the card that plugs into the slots at the back, and use APC's powerchute software to monitor the data over a serial port. It has all the bells and whistle for custom alert/notification etc. ...and the price is right. Tim _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rocky at atipa.com Tue Sep 30 16:23:37 2003 From: rocky at atipa.com (Rocky McGaugh) Date: Tue, 30 Sep 2003 15:23:37 -0500 (CDT) Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: Dont overlook lm_sensors+cron -- Rocky McGaugh Atipa Technologies rocky at atipatechnologies.com rmcgaugh at atipa.com 1-785-841-9513 x3110 http://67.8450073/ perl -e 'print unpack(u, ".=W=W+F%T:7\!A+F-O;0H`");' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Sep 30 19:40:26 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 30 Sep 2003 19:40:26 -0400 (EDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: Message-ID: On Tue, 30 Sep 2003, Mark Hahn wrote: > > Has anyone recently analyzed how much power each of the subunits of > > a PC uses, i.e. how much power which mainboard, hdd, etc. > > it's not hard to approximate this by reading the specs of each component. > obviously, CPUs are the main consumers, probably followed by power hardware > next (PS inefficiency, as well as the on-motherboard converter.) The CPU power numbers vary. You find people quoting design limits (~80W), typical power (25W with moderate desktop usage), average power (15W averaged over a day). For the G5 I've only seen optimistic typical power guesses directly compared to worst case for x86. That's probably unfair: while it likely does use less power, it's not yet obvious that the total system uses less power to achieve the same performace as the Opteron. > disks are much, much cooler than they used to be, probably dropping > below power consumed by ram on most clusters. Note that most performance-oriented RAM types now have metal cases and heat sinks. They didn't add the metal because it _looks_ cool. > naturally, if you've > got 6 15K RPM SCSI disks in a node with just 1G ram, that's completely > different! even things like high-speed nics often amount to <10W, > which simply doesn't compare to the ~80-90W per high-end CPU... GbE NICs used to use >10W, more than half used in the transceiver chip. Today the typical GbE NIC is a single chip in the 3-5W range. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jschauma at netmeister.org Tue Sep 30 16:46:35 2003 From: jschauma at netmeister.org (Jan Schaumann) Date: Tue, 30 Sep 2003 16:46:35 -0400 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: <20030930204635.GA17458@netmeister.org> Mitchel Kagawa wrote: > So I was wondering if anyone used one of those environment monitoring > appliances that e-mail you (before stuff starts shutting off) if a > temp goes out of a set range or if it detects water, high humidity, or > movement in the room? I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if > anyone has any experience with them or if you have any other > (inexpensive <$2000) suggestions. Thanks for any help. It just so happens that last week I installed a NetBotz (http://www.netbotz.com/products/wall.html#wallbotz400) in our machine room. It was a breeze to install/setup and looks rather nifty wrt capabilities (SNMP, email alerts, syslog, ftp- and http upload/logging, alert escalation etc.). So far (as I said, ~1 week old) I'd recommend it. I wonder what internal OS it's running. -Jan -- Wenn ich tot bin, mir soll mal Einer mit Auferstehung oder so kommen, ich hau ihm eine rein! (Anonym) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available URL: From yemi at SLAC.Stanford.EDU Tue Sep 30 17:43:17 2003 From: yemi at SLAC.Stanford.EDU (Adesanya, Adeyemi) Date: Tue, 30 Sep 2003 14:43:17 -0700 Subject: [Ganglia-general] Running as non-root on Solaris Message-ID: <4ABE591494FF5246B38C0CDA4CBFAFE8025217D6@exchange4.slac.stanford.edu> Hi Steven. Thanks for the pointer. I managed to disable/strip the built-in solaris metrics and now gmond runs in non-root!! ---- Yemi > -----Original Message----- > From: steven wagner [mailto:swagner at ilm.com] > Sent: Monday, September 29, 2003 11:05 AM > To: Adesanya, Adeyemi > Cc: 'ganglia-general at lists.sourceforge.net' > Subject: Re: [Ganglia-general] Running as non-root on Solaris > > > Sure, there's some metric stub code you can copy from in the > gmond/machines directory. I think cygwin.c is still mostly returning > dummy values for metrics ( ... anyone?), you could use that as a > template for the functions you'll be disabling. > > Sorry I can't offer you any more direct assistance, but I > don't really > have any cycles to spare for Solaris Ganglia 2.x development. > > Happy hacking :) > > Adesanya, Adeyemi wrote: > > Hi Steve. > > > > Thanks for responding. Perhaps an alternative may be switches to > > disable those metrics that require gmond to run as root? > > > > ---- > > Yemi > > > > > > > >>-----Original Message----- > >>From: steven wagner [mailto:swagner at ilm.com] > >>Sent: Monday, September 29, 2003 10:32 AM > >>To: Adesanya, Adeyemi > >>Cc: ganglia-general at lists.sourceforge.net > >>Subject: Re: [Ganglia-general] Running as non-root on Solaris > >> > >> > >>Adeyemi Adesanya wrote: > >> > >>>Hi There. > >>> > >>>I spent some time digging through the archives but I am > >> > >>unable to find > >> > >>>a way of running gmond as a non-root user on Solaris. Is > >> > >>this out of > >> > >>>the question or is there some way to patch the code? All of our > >>>critical servers run Solaris, that?s where the real action > >> > >>is and that > >> > >>>what I'd like to monitor. > >> > >>Unfortunately, not all of the information that the Solaris > monitoring > >>core collects can be gathered through a warm and fuzzy API > >>(as far as I > >>could tell, anyway). Many metrics had to be gathered by > >>peeking at the > >>kernel memory and symbol table, both of which require > >>superuser permissions. > >> > >>It's entirely possible that the kstat API has been extended > >>(or that I > >>just plain didn't RTFM thoroughly enough) and that all of > these calls > >>can be handled that way. In which case, the first person to > >>point this > >>out volunteers to rewrite all my spaghetti solaris.c code. :) > >> > >>Good luck! > >> > > ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Ganglia-general mailing list Ganglia-general at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general From lanceh at cs.unr.edu Tue Sep 30 18:08:01 2003 From: lanceh at cs.unr.edu (Lance Hutchinson) Date: Tue, 30 Sep 2003 15:08:01 -0700 Subject: [Ganglia-general] Errors starting Gmetad Message-ID: <1064959681.13298.4.camel@vrpad.cs.unr.edu> Hi all. I recently had the problem that the webfrontend of my 64 node rocks cluster did not have any graphs showing, as none of the rrds were being generated. I upgraded my gmetad, and now i get the following messages when i start gmetad with debug turned on. Can anyone tell me where i am going wrong? What do I have the wrong version of? [root at cortex lance]# /etc/init.d/gmetad start Starting GANGLIA gmetad: Going to run as user nobody Source: [Cortex, step 15] has 1 sources 127.0.0.1 xml listening on port 8651 interactive xml listening on port 8652 Data thread 7176 is monitoring [Cortex] data source 127.0.0.1 [Cortex] is an OLD version Thanks a ton. -- Lance Hutchinson ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Ganglia-general mailing list Ganglia-general at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-general From becker at scyld.com Tue Sep 30 19:40:26 2003 From: becker at scyld.com (Donald Becker) Date: Tue, 30 Sep 2003 19:40:26 -0400 (EDT) Subject: Power Supply: Supermicro P4DL6 Board? In-Reply-To: Message-ID: On Tue, 30 Sep 2003, Mark Hahn wrote: > > Has anyone recently analyzed how much power each of the subunits of > > a PC uses, i.e. how much power which mainboard, hdd, etc. > > it's not hard to approximate this by reading the specs of each component. > obviously, CPUs are the main consumers, probably followed by power hardware > next (PS inefficiency, as well as the on-motherboard converter.) The CPU power numbers vary. You find people quoting design limits (~80W), typical power (25W with moderate desktop usage), average power (15W averaged over a day). For the G5 I've only seen optimistic typical power guesses directly compared to worst case for x86. That's probably unfair: while it likely does use less power, it's not yet obvious that the total system uses less power to achieve the same performace as the Opteron. > disks are much, much cooler than they used to be, probably dropping > below power consumed by ram on most clusters. Note that most performance-oriented RAM types now have metal cases and heat sinks. They didn't add the metal because it _looks_ cool. > naturally, if you've > got 6 15K RPM SCSI disks in a node with just 1G ram, that's completely > different! even things like high-speed nics often amount to <10W, > which simply doesn't compare to the ~80-90W per high-end CPU... GbE NICs used to use >10W, more than half used in the transceiver chip. Today the typical GbE NIC is a single chip in the 3-5W range. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Sep 30 19:10:57 2003 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Wed, 01 Oct 2003 01:10:57 +0200 Subject: Environment monitoring In-Reply-To: <005401c38787$25cad390$7001a8c0@Navatek.local> (Mitchel Kagawa's message of "Tue, 30 Sep 2003 09:15:01 -1000") References: <005401c38787$25cad390$7001a8c0@Navatek.local> Message-ID: "Mitchel Kagawa" writes: > I'm looking at a NetBotz/RackBots here > (http://www.netbotz.com/products/rack.html) and was wondering if > anyone has any experience with them or if you have any other > (inexpensive <$2000) suggestions. I'm using serial interface temperature sensors from Picotech. I've hacked a smallish Python driver to speak to them, and use rrdtool to make pretty graphs and Nagios to scream bloody murder if the temperature rises to far. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jsims at csiopen.com Tue Sep 30 21:51:06 2003 From: jsims at csiopen.com (Joey Sims) Date: Tue, 30 Sep 2003 21:51:06 -0400 Subject: Power Supply: Supermicro P4DL6 Board? Message-ID: <812B16724C38EE45A802B03DD01FD547226258@exchange.concen.com> Hello Michael, I also recommend Zippy. They make an excellent product. Below is a power supply guide that is manufacturer specific in regards to the motherboard you might have. Your particular model isn't listed here but, all the rest are that also share this same spec. If you check out the other Supermicro 400MHz series dual Xeon boards, you'll notice this. http://www.zippy.com/P_GUIDE_SEARCH.asp?lv_rfnbr=2 You're looking at ~ $125 - $140 for a decent single 460W. Regards, ================================================== Joey P. Sims 800.995.4274 - 242 Sales Manager 770.442.5896 - Fax HPC/Storage Division www.csilabs.net Concentric Systems, Inc. jsims at csiopen.com ====================================ISO9001:2000== _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf