From proksy_me at yahoo.com Fri Jan 2 03:27:20 2004 From: proksy_me at yahoo.com (prakash borade) Date: Fri, 2 Jan 2004 00:27:20 -0800 (PST) Subject: [Beowulf] i have problem in my beowulf Message-ID: <20040102082720.81089.qmail@web40104.mail.yahoo.com> hello everybody i am very new to clustering environ ment and also to beowulf i need some guid in how to build a beowul cluster i m trying it with mpich 1.2.5.2 is it neccesary to initialize the lam environmnet what i know is that lam/mpi and mpich are to diffrent implementtations of mpi i get following error for the standard cpilog program after i have initialized lam environment using lamboot [mpich at tung mpich]$ mpirun -np 2 cpilog Process 0 running on tung Process 0 running on dyp pi is approximately 3.1415926535897634, Error is 0.0000000000000298 wall clock time = 0.768179 ----------------------------------------------------------------------------- It seems that [at least] one of processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Sat Jan 3 01:42:21 2004 From: proksy_me at yahoo.com (prakash borade) Date: Fri, 2 Jan 2004 22:42:21 -0800 (PST) Subject: [Beowulf] i have problem in my beowulf Message-ID: <20040103064221.4142.qmail@web40112.mail.yahoo.com> > hi glen > first of all thanks a lot for replying > me. > as mentioned that mpirun command would not require > lam > > > but at my machines seems that unless and until i > enable lam environment i cannot run mpirun > > also > > the output i.e > process 0 of 0 at node00 > process 0 of 0 at node01 > > is incorrect > and it should be as > > process 0 of 0 at node00 > process 1 of 0 at node01 > > also all my paths are well set > > i have also added all my hosts to the file > /etc/lam/lam-bhosts.def > as > > node00 > node01 > > is thist thinf necessary > > because it has not been mentioned > in any of the beowulf building manual > > > > > > __________________________________ > Do you Yahoo!? > Find out what made the Top Yahoo! Searches of 2003 > http://search.yahoo.com/top2003 __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Kris.Boutilier at scrd.bc.ca Sun Jan 4 23:50:44 2004 From: Kris.Boutilier at scrd.bc.ca (Kris Boutilier) Date: Sun, 4 Jan 2004 20:50:44 -0800 Subject: [Beowulf] Offtopic: Emulex cLan 1000 support under kernels 2.4.x or 2.6.x? Message-ID: <1174450A1968D111AAAF00805FC162AEBE1344@deep_thought.secure.scrd.bc.ca> This seems like the most probable place to find someone who will know first hand - Is anyone aware of step-by-step instructions, updated code or any other means of getting the cLan 1000 VIA hardware running under anything higher than a 2.2 kernel? Emulex still offer binaries for Redhat 6.x on their site, and they offer source code for v1.3.1 of the driver, but I understand it to now be considered very obsolete... certainly it doesn't compile out of the box against 2.4.x Suggestions anyone? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tony at mpi-softtech.com Mon Jan 5 10:32:28 2004 From: tony at mpi-softtech.com (Anthony Skjellum) Date: Mon, 5 Jan 2004 09:32:28 -0600 Subject: [Beowulf] Offtopic: Emulex cLan 1000 support under kernels 2.4.x or 2.6.x? References: <1174450A1968D111AAAF00805FC162AEBE1344@deep_thought.secure.scrd.bc.ca> Message-ID: <00c901c3d3a1$211a8f50$b600a8c0@ciswinnet.cis.uab.edu> Hi, we have a Yahoo! support group for cLan/Emulex VIA hardware... please see http://groups.yahoo.com/group/giganet-network-users/ There are 57 members, and people have successfully run on recent RH kernels, and you are likely to get feedback from those folks. Tony ----- Original Message ----- From: "Kris Boutilier" To: Sent: Sunday, January 04, 2004 10:50 PM Subject: [Beowulf] Offtopic: Emulex cLan 1000 support under kernels 2.4.x or 2.6.x? > This seems like the most probable place to find someone who will know first > hand - Is anyone aware of step-by-step instructions, updated code or any > other means of getting the cLan 1000 VIA hardware running under anything > higher than a 2.2 kernel? Emulex still offer binaries for Redhat 6.x on > their site, and they offer source code for v1.3.1 of the driver, but I > understand it to now be considered very obsolete... certainly it doesn't > compile out of the box against 2.4.x > > Suggestions anyone? > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at comcast.net Mon Jan 5 18:30:07 2004 From: laytonjb at comcast.net (Jeffrey B. Layton) Date: Mon, 05 Jan 2004 18:30:07 -0500 Subject: [Beowulf] Article on First Clusters from Around the World Message-ID: <3FF9F37F.2050207@comcast.net> Hello, I'm working on an article for ClusterWorld magazine about people's first clusters from around the World. The intent of the article is to find out why people started using clusters, what problems they are trying to solve, how they learned about clusters, and what their experiences have been. If you have built a small cluster recently, I'd like to interview you for the article. Just email me at the return address on this posting. Thanks! Jeff Layton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rene.storm at emplics.com Tue Jan 6 08:20:19 2004 From: rene.storm at emplics.com (Rene Storm) Date: Tue, 6 Jan 2004 14:20:19 +0100 Subject: [Beowulf] Benchmark results In-Reply-To: <200312301523.SAA06085@nocserv.free.net> References: <200312301523.SAA06085@nocserv.free.net> Message-ID: <20040106142019.48d39cee.rene.storm@emplics.com> Dear cluster folks, I saw some weird results of different benchmarks on my systems. My problem is to verify these results. I've played with hpl benchmark on our clusters (8 CPU Intel, 8 CPU Opteron) and wasn't able to get two times the same result. Of course no configuration changes. Difference round 5-10%. So I went down to one machine, but the same behavior. Standart mpich, per-complied lapack, two processor on an smp machine. 10 times the same benchmark gives back 10 different results. Same trouble with stream memory benchmark, dgemv matrix calculation and others. There was no network, but it was a big installation, so it could be, that there are some running jobs (eg cron-jobs) which disturb my benchmarks results. Next Step: I created a self-booting cdrom from the scratch - added litte bit of X and a gui my benchmarks. 1) cpi - calculating pi with mpich on smp (ch_p4 on loopback) 2+3) dgemv - 5kx5k matrix calculation , single and smp 4) crafty chess benchmark - 2 threads 5) /bin/true - calling true via fork(), taking the time 6) PMB-MPI1 - Pallas benchmark There is only sshd for mpich running, everything is loaded from the cdrom into a ramdisk. BUT same result for the results. 5-10% unsteady. Please let me know, if you could see same behavior on your machines. Does someone know a reason for that? If you would like to check this cd out -> http://212.12.58.172/bench/bench_on_cd.iso.gz 40 MB download. You will need at least 512MB and no usb mice, keyboard. Thanks in advance Rene Storm _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Tue Jan 6 09:10:04 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Tue, 06 Jan 2004 09:10:04 -0500 Subject: [Beowulf] Benchmark results In-Reply-To: <20040106142019.48d39cee.rene.storm@emplics.com> References: <200312301523.SAA06085@nocserv.free.net> <20040106142019.48d39cee.rene.storm@emplics.com> Message-ID: <3FFAC1BC.6020201@scalableinformatics.com> Hi Rene: How long were the runs in comparison to the timer resolution? What other processes (if any) were running? Did various cron-jobs light off during any portion of this? For opterons, there is an a processor affinity issue whereby you can get different performance if the memory allocated and used is not associated with that processor. I have heard that there are patches which deal with this, though I haven't had time to try them. I do seem to remember an alignment issue for Xeon as well. I would not expect absolutely identical timing data each time the benchmark were run, but I would expect a tight grouping around a mean value. The relative size of the error would be related to a number of factors, including timer resolution, machine state, etc. For runs of reasonable length (longer than 30 minutes), the timer resolution effects should be minimal. If you have other processes lighting off, consuming memory, processor cycles, cache, bandwidth, interrupts, it is likely that your distribution will reflect this. Joe Rene Storm wrote: >Dear cluster folks, > >I saw some weird results of different benchmarks on my systems. >My problem is to verify these results. > >I've played with hpl benchmark on our clusters (8 CPU Intel, 8 CPU Opteron) and wasn't able to get two times the same result. Of course no configuration changes. Difference round 5-10%. > >So I went down to one machine, but the same behavior. >Standart mpich, per-complied lapack, two processor on an smp machine. >10 times the same benchmark gives back 10 different results. >Same trouble with stream memory benchmark, dgemv matrix calculation and others. > >There was no network, but it was a big installation, so it could be, that there are some running jobs (eg cron-jobs) which disturb my benchmarks results. > >Next Step: I created a self-booting cdrom from the scratch - added litte bit of X and a gui my benchmarks. > >1) cpi - calculating pi with mpich on smp (ch_p4 on loopback) >2+3) dgemv - 5kx5k matrix calculation , single and smp >4) crafty chess benchmark - 2 threads >5) /bin/true - calling true via fork(), taking the time >6) PMB-MPI1 - Pallas benchmark > >There is only sshd for mpich running, everything is loaded from the cdrom into a ramdisk. >BUT same result for the results. 5-10% unsteady. > >Please let me know, if you could see same behavior on your machines. >Does someone know a reason for that? > > >If you would like to check this cd out -> >http://212.12.58.172/bench/bench_on_cd.iso.gz >40 MB download. You will need at least 512MB and no usb mice, keyboard. > >Thanks in advance >Rene Storm > > > > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 6 09:51:25 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 6 Jan 2004 09:51:25 -0500 (EST) Subject: [Beowulf] Benchmark results In-Reply-To: <3FFAC1BC.6020201@scalableinformatics.com> Message-ID: On Tue, 6 Jan 2004, Joe Landman wrote: > Hi Rene: > > How long were the runs in comparison to the timer resolution? What > other processes (if any) were running? Did various cron-jobs light off > during any portion of this? > > For opterons, there is an a processor affinity issue whereby you can > get different performance if the memory allocated and used is not > associated with that processor. I have heard that there are patches > which deal with this, though I haven't had time to try them. I do seem > to remember an alignment issue for Xeon as well. > > I would not expect absolutely identical timing data each time the > benchmark were run, but I would expect a tight grouping around a mean > value. The relative size of the error would be related to a number of > factors, including timer resolution, machine state, etc. For runs of > reasonable length (longer than 30 minutes), the timer resolution effects > should be minimal. If you have other processes lighting off, consuming > memory, processor cycles, cache, bandwidth, interrupts, it is likely > that your distribution will reflect this. > > Joe Ya. This is one of the things motivating the bproc beowulf design -- the ability to build a "stripped" machine with much more controlled and predictable state and with an absolutely minimal set of competing "system" tasks running on an otherwise quiescent system. You can check out Scyld (open source commercial) or Clustermatic (non-commercial/GPL) if you want to pursue this route. The run length issue is pure central limit theorem. If you do short runs (or internally time short loops), expect large fluctuations and a relatively broad distribution of times as e.g. a task swap or context switch in the middle of a timing loop will lead to wildly different times on a per-case basis. If you run "long" times you can average more effectively over the fluctuating/uncontrolled part of the system state and your mean times SHOULD home in their true mean value, with a distribution of means that narrows roughly with the inverse square root of the run time. Exceptions to this are, as Joe noted, persistent state differences between runs. These should be investigated and understood, as 5% runtime difference is like getting a free node in twenty and worth pursuing. ps and top are your friend. They'll eat some cycles themselves (good old Heisenberg, sort of:-), but they'll also give you a realtime picture of the tasks that are competing. xmlsysd/wulfstat might help as well (sort of like vmstat and/or netstat run across the whole cluster at once). The other thing I can think of that might result in a large persistent variance (besides memory/motherboard issues) is substantial difference in network communication pattern. If your communications aren't carefully organized, they may self-organize depending on a nucleation condition (who talks to who and and gets through first). I can easily imagine distinct communication patterns emerging with a large variance. Actually, if I were a REAL computer scientist, this would be a fairly interesting topic for a research project -- spontaneous emergence of persistent patterns with very different efficiencies in complex networks. If you look (and upon examination it looks like this might be the problem) there are probably research papers out there on it already. Can't imagine that this hasn't been studied. rgb > > Rene Storm wrote: > > >Dear cluster folks, > > > >I saw some weird results of different benchmarks on my systems. > >My problem is to verify these results. > > > >I've played with hpl benchmark on our clusters (8 CPU Intel, 8 CPU Opteron) and wasn't able to get two times the same result. Of course no configuration changes. Difference round 5-10%. > > > >So I went down to one machine, but the same behavior. > >Standart mpich, per-complied lapack, two processor on an smp machine. > >10 times the same benchmark gives back 10 different results. > >Same trouble with stream memory benchmark, dgemv matrix calculation and others. > > > >There was no network, but it was a big installation, so it could be, that there are some running jobs (eg cron-jobs) which disturb my benchmarks results. > > > >Next Step: I created a self-booting cdrom from the scratch - added litte bit of X and a gui my benchmarks. > > > >1) cpi - calculating pi with mpich on smp (ch_p4 on loopback) > >2+3) dgemv - 5kx5k matrix calculation , single and smp > >4) crafty chess benchmark - 2 threads > >5) /bin/true - calling true via fork(), taking the time > >6) PMB-MPI1 - Pallas benchmark > > > >There is only sshd for mpich running, everything is loaded from the cdrom into a ramdisk. > >BUT same result for the results. 5-10% unsteady. > > > >Please let me know, if you could see same behavior on your machines. > >Does someone know a reason for that? > > > > > >If you would like to check this cd out -> > >http://212.12.58.172/bench/bench_on_cd.iso.gz > >40 MB download. You will need at least 512MB and no usb mice, keyboard. > > > >Thanks in advance > >Rene Storm > > > > > > > > > >_______________________________________________ > >Beowulf mailing list, Beowulf at beowulf.org > >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > -- > Joseph Landman, Ph.D > Scalable Informatics LLC, > email: landman at scalableinformatics.com > web : http://scalableinformatics.com > phone: +1 734 612 4615 > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Per.Lindstrom at me.chalmers.se Tue Jan 6 14:54:49 2004 From: Per.Lindstrom at me.chalmers.se (=?ISO-8859-1?Q?Per_Lindstr=F6m?=) Date: Tue, 06 Jan 2004 20:54:49 +0100 Subject: [Beowulf] Website for Linux workstations and small clusters Message-ID: <3FFB1289.90800@me.chalmers.se> Hi I wish you and your near and dear a Happy New Year. We, Department of Naval Architecture and Ocean Engineering, Chalmers University of Technology will set up a Web site with instructions how to build, install, configure and operate Linux workstations and small Linux clusters as well as the installation and the basic use of calculation and simulation softwares. The site's structure will be very much influenced by the Materials Algorithms Project, http://www.msm.cam.ac.uk/map/mapmain.html It will take some weeks until we are up and running. Please contact me if you want to contribute with your experience and/or have some ideas how this should be done Best regards Per Lindstr?m, Industrial Doctoral Student Per.Lindstrom at me.chalmers.se www.na.chalmers.se Per Lindstr?m M.Sc, CEng, SenMWeldI, IWE, Welding Engineer to Kvaerner Power AB +46 (0)31 50 10 14 Per.Lindstrom at akerkvaerner.com http://akerkvaerner.com/Internet/IndustriesAndServices/Power/default.htm Jim Lux wrote: >I fully agree... I suspect that most readers of this list start with a small >cluster, and a historical record of what it took to get it up and running is >quite useful, especially the hiccups and problems that you inevitably >encounter. (e.g. what do you mean the circuit breaker just tripped on the >plug strip when we plugged all those things into it?) > >----- Original Message ----- >From: "Per Lindstr?m" >To: "D. Scott" ; "Anand TNC" > >Cc: "Beowulf" ; "Josh Moore" ; "Per" > >Sent: Monday, December 29, 2003 11:59 AM >Subject: [Beowulf] Websites for small clusters > > > > >>Hi Dan, >> >>It should be great if you publish your cluster work instructions on a >>website. I have found that there is need for a such place. >> >>The site http://www.msm.cam.ac.uk/map/mapmain.html is a good example on >>how a website sharing scientific and/or professional experience can be >>aranged. >> >>If it not allready exist, shall we arrange something similar for few >>node clusters? (Few node clusters 2 - 30 nodes?) >> >>Best regards >>Per Lindstr?m >> >> >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >> >> >http://www.beowulf.org/mailman/listinfo/beowulf > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mwill at penguincomputing.com Tue Jan 6 12:53:13 2004 From: mwill at penguincomputing.com (Michael Will) Date: Tue, 6 Jan 2004 09:53:13 -0800 (PST) Subject: [Beowulf] Re: Benchmark results Message-ID: <40318.64.243.132.186.1073411593.squirrel@imap.penguincomputing.com> Rene, Do you see that fluctuation only on the opterons or also on the intel cpus (I assume Xeon?) If it was only on the opteron, then you will have node memory interleave turned off in your bios, and sometimes the data you work with is close to your CPU and sometimes far. (each CPU has a bank of RAM and accesses the other CPUs bank through hypertransport, which is slower) When running a memory and floatpoint intensive benchmark (neuronal network training program like SNNS), we got two consistent results: They were alternating, depending on which CPU the code ran, and they were 20% off of each other for that setup. Once we put in the same amount of memory in both banks, and turned on 'node memory interleave' in the bios, they were consistently between the two, a bit closer to the faster value. I will take a closer look at your CD later, too. Michael Will PS: Did I mention the opterons blew away the xeons by far? -- Michael Will, Linux Sales Engineer Tel: 415.358.2673 Toll Free: 888-PENGUIN PENGUIN COMPUTING - The World's Most Reliable Linux System www.penguincomputing.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rene.storm at emplics.com Tue Jan 6 18:00:18 2004 From: rene.storm at emplics.com (Rene Storm) Date: Wed, 7 Jan 2004 00:00:18 +0100 Subject: [Beowulf] ...Re: Benchmark results Message-ID: <29B376A04977B944A3D87D22C495FB230127BB@vertrieb.emplics.com> HI, first of all, thanks for the responses.... It seems that there are some more points I have to observe. For example the hpl benchmark: The job was running on two processors (smp) on loopback. Used time round about 20min. There was no cronjob, no X, only a niced top. The machines environments are almost constant (temperature->lm_sensors). A reboot after every single run. And yes, the differences on the opterons are greater as on i386. I will take a look at interleaving. But why should the memory allocation change between two runs? So it seems, I will to go a little bit deeper into that and try to get some more results. What I do expect is some sort of Gaussian distribution on the results. I will take the hpl and throw the results into gnuplot. One question about the walltime. Could this be a problem? Does someone know how this really works? I read something about HZ and jiffies and so, but it was a bit confusing. Maybe this could be the point were old, unfair heisenberg could have had a chance. Rene Storm -----Urspr?ngliche Nachricht----- Von: Michael Will [mailto:mwill at penguincomputing.com] Gesendet: Dienstag, 6. Januar 2004 18:53 An: beowulf at beowulf.org Betreff: [Beowulf] Re: Benchmark results Rene, Do you see that fluctuation only on the opterons or also on the intel cpus (I assume Xeon?) If it was only on the opteron, then you will have node memory interleave turned off in your bios, and sometimes the data you work with is close to your CPU and sometimes far. (each CPU has a bank of RAM and accesses the other CPUs bank through hypertransport, which is slower) When running a memory and floatpoint intensive benchmark (neuronal network training program like SNNS), we got two consistent results: They were alternating, depending on which CPU the code ran, and they were 20% off of each other for that setup. Once we put in the same amount of memory in both banks, and turned on 'node memory interleave' in the bios, they were consistently between the two, a bit closer to the faster value. I will take a closer look at your CD later, too. Michael Will PS: Did I mention the opterons blew away the xeons by far? -- Michael Will, Linux Sales Engineer Tel: 415.358.2673 Toll Free: 888-PENGUIN PENGUIN COMPUTING - The World's Most Reliable Linux System www.penguincomputing.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Tue Jan 6 19:44:31 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Tue, 06 Jan 2004 19:44:31 -0500 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <29B376A04977B944A3D87D22C495FB230127BB@vertrieb.emplics.com> References: <29B376A04977B944A3D87D22C495FB230127BB@vertrieb.emplics.com> Message-ID: <1073436271.15823.30.camel@protein.scalableinformatics.com> On Tue, 2004-01-06 at 18:00, Rene Storm wrote: > HI, > first of all, thanks for the responses.... > > It seems that there are some more points I have to observe. > For example the hpl benchmark: > The job was running on two processors (smp) on loopback. Used time round about 20min. > There was no cronjob, no X, only a niced top. > The machines environments are almost constant (temperature->lm_sensors). > A reboot after every single run. One of the odder bits about benchmarking... Most folks I know don't reboot machines between real end user runs. Or umount/mount after a heavy IO run. If you are really aiming for the maximal in limited interference, you have probably turned everything off anyway, and are running in single user mode (or just above it with networking enabled). Doesn't guarantee much more than some default services are not run. You might want to turn of atd and disable stuff in /etc/cron.d . Not that this really makes much sense from a "realism" point of view, as end users are not (normally) allowed to do that. > And yes, the differences on the opterons are greater as on i386. > I will take a look at interleaving. But why should the memory allocation change between two runs? In short, when a process launches the NUMA nature of its memory could mean that the memory system where the thread gets its physical allocations differs from the memory system which is tied to the CPU running the code. For a NUMA system, this gives you an extra "hop" for memory accesses. There is the concept of process affinity whereby the thread is tied to a particular CPU it has been running on, which tends to prevent the cache from being rendered less effective by having the process hop from CPU to CPU. Tying an allocation to a CPU exists in various Unixen (IRIX in particular with the old dplace and similar directives), and I have heard rumors of a "runon" command (applying both a process/thread affinity, and pegging the memory allocated by the thread to the CPU running the thread). > So it seems, I will to go a little bit deeper into that and try to get some more results. > What I do expect is some sort of Gaussian distribution on the results. You should get some distribution. I would be reluctant to discuss the shape without at least a reasonable empirical theory. It does seem that the allocation locality would cause at least a bimodal shape (rough guess), something like 2 overlapping distributions. Other effects are going to spread this a bit. > I will take the hpl and throw the results into gnuplot. > > One question about the walltime. Could this be a problem? > Does someone know how this really works? I read something about HZ and jiffies and so, but it was a bit confusing. > Maybe this could be the point were old, unfair heisenberg could have had a chance. Physics 101: Take your precision of your clock. Your uncertainty in using the clock as a measuring device is +/- 1/2 of the clocks precision (call this delta_t), or 1/2 of the smallest interval that this clock can measure. As you are really interested in time intervals, you may need to appeal to basic error analysis. That said, if you look at a really rough estimate of the contribution of the random error of the clock for your runs, have a look at the ratio of delta_t to the total run time. For a run of 30 minutes, t ~ 1800 seconds, while using 0.01 seconds as the clocks precision gives you about 3 x 10**(-6) (or 1 part in 360000). That is, the contribution to the error due to the clock is small here. Even for a run of 60 seconds it is still in the 0.01% regime. To get a 5% error out of that, you would need delta_t/t ~ 0.05, or t about 0.1 second for your total benchmark run time. You wouldn't even need another process taking up time, the kernel will happily do this for you with some of its housekeeping chores (bdflush, kswapd, ...). Servicing lots of interrupts is a good way to slow things down, as are doing many small page allocations where one large one would suffice, etc. > > Rene Storm > > > -----Urspr?ngliche Nachricht----- > Von: Michael Will [mailto:mwill at penguincomputing.com] > Gesendet: Dienstag, 6. Januar 2004 18:53 > An: beowulf at beowulf.org > Betreff: [Beowulf] Re: Benchmark results > > Rene, > > Do you see that fluctuation only on the opterons or also on the intel cpus > (I assume Xeon?) > > If it was only on the opteron, then you will have node memory interleave > turned off in your bios, and sometimes the data you work with is close to > your CPU and sometimes far. (each CPU has a bank of RAM and accesses the > other CPUs bank through hypertransport, which is slower) > > When running a memory and floatpoint intensive benchmark (neuronal network > training program like SNNS), we got two consistent results: They were > alternating, depending on which CPU the code ran, and they were 20% off of > each other for that setup. > > Once we put in the same amount of memory in both banks, and turned on > 'node memory interleave' in the bios, they were consistently between the > two, a bit closer to the faster value. > > I will take a closer look at your CD later, too. > > Michael Will > PS: Did I mention the opterons blew away the xeons by far? -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From maikel_punie at hotmail.com Wed Jan 7 10:16:07 2004 From: maikel_punie at hotmail.com (Maikel Punie) Date: Wed, 07 Jan 2004 16:16:07 +0100 Subject: [Beowulf] Need help Message-ID: hallo everyone, well for my final project at school i need to build a beowulf, well i'm this far that all the node (diskless system) boot from the main node (ntfs) and there config and stuff from there, the main node can see all off his nodes. Now my question is whats the next step? How do i make the nodes do the calculations and stuff for the main node, how do i realy integrate the parrallel processing? Thanks a lot Maikel _________________________________________________________________ Al aan je vakantie gedacht? http://www.msn.be/reizen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 7 11:18:33 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 7 Jan 2004 11:18:33 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <1073436271.15823.30.camel@protein.scalableinformatics.com> Message-ID: On Tue, 6 Jan 2004, Joe Landman wrote: > Physics 101: > > Take your precision of your clock. Your uncertainty in using the clock > as a measuring device is +/- 1/2 of the clocks precision (call this > delta_t), or 1/2 of the smallest interval that this clock can measure. > As you are really interested in time intervals, you may need to appeal > to basic error analysis. That said, if you look at a really rough > estimate of the contribution of the random error of the clock for your > runs, have a look at the ratio of delta_t to the total run time. For a > run of 30 minutes, t ~ 1800 seconds, while using 0.01 seconds as the > clocks precision gives you about 3 x 10**(-6) (or 1 part in 360000). > That is, the contribution to the error due to the clock is small here. > > Even for a run of 60 seconds it is still in the 0.01% regime. To get a > 5% error out of that, you would need delta_t/t ~ 0.05, or t about 0.1 > second for your total benchmark run time. A Brief Digression on Benchmark Timing (Yes, I lie like a rug. It is not brief. If you already know about timing, or don't care, hit 'd' now...:-) To add a bit of pure CPS 1 empiricism to Joe's remarks, I've played quite a bit with timers while writing cpu_rate and using lmbench and so forth. There are two reasonably accurate "simple" timing mechanisms available in linux, one simpler (but less accurate) than the other. gettimeofday returns a timeval struct with seconds and microseconds since the Epoch. Typical use in a timing loop saves the timeval at an interval start, runs the loop, saves the timeval at the stop, and computes the delta. This generates wall clock delta and EMPIRICALLY is accurate to about two microseconds, no more. This makes it nearly useless for timing short loops -- say a few hundred iterations of a multiply -- because multiplies take order of a nanosecond (depending on locality and clock) and one has to execute thousands and thousands of them to use up more time than the resolution of your clock. A timing of 2 usec +/- 2 usec isn't very useful. If you execute a few billion of them (so the internal loop takes order of seconds to complete) things are better -- 2 sec +/- 2 usec isn't too bad, precision wise. However now there is another tradeoff. A timed interval that occupies only a hundred microseconds or so has an excellent chance of not being interrupted (depending, of course, on system environment -- I'm assuming a basically quiescent network and no other toplevel userspace tasks running). One that occupies two seconds will be interrupted hundreds of times, because if nothing else the system timer will issue interrupts that the kernel has to handle. If you run "vmstat 1" on your box, you'll note that there are roughly 100 interrupts per second steady state on a quiet system. Associated with these are 100+ context switches. These interrupts and context switches are quite "expensive", and one is tempted to get off into a discussion of the kernel design and upper/lower half handlers and ways the system tries to bundle up its work to minimize impact on running processes, but it would be a moderate digression. Suffice it to say that if you time LOTS of 100 microsecond loops separately in a continuous process and look at the results, you're likely to see the bimodal distribution Joe refers to (or even a fully multimodal distribution) because SOME of those intervals will be interrupted and the interrupt/context switch will be "timed" along with the swapped out loop by the wall-clock timer. This can be somewhat ameliorated by adding a usleep right before each timing loop iteration, as the kernel will often take advantage of a task pause to complete pending interrupts. However, it alas isn't a magic bullet because not all interrupts are "pendable". gettimeofday is actually a mediocre timer for nearly all microbenchmarking purposes as it is so much slower than the processes that it times that getting good precision between these competing limits is not at all easy. It is just fine for macroscopic, large scale completion times that average over all sorts of system state changes (e.g. all those int/css). A much better timer is available on nearly all intel/amd cpus. You can actually read the CPU's onboard cycle counter with a suitable bit of assembler magic. By also reading the CPU's clock (from /proc/cpuinfo) and doing a bit of division, you can obtain a "time" accurate (empirically, again) to roughly 10-60 nanoseconds. Very roughly, as this depends strongly on the clock and other systems architecture differences that can vary a bit from system to system and whether or not the clock is wrapped up quasi-portably in a subroutine call (adding a bit of overhead) or inlined as raw assembler on both sides of the timing loop -- this is just what I measure. This nanotimer is FAR more suitable for use in microbenchmarking -- relatively small loops where one devoutly hopes to avoid getting nailed by a context switch as one is interested in "how many floating point multiplies can the system perform on numbers in its registers/L1 cache/L2 cache/memory per second". A loop with a few microseconds worth of operations, fairly likely not to be interrupted if you start immediately after a timer click, can yield < 1% accuracy. Multimodality in a distribution of such short loop timings yields "interesting" information about the system's CPU, cache, memory architecture, and their interaction with the kernel. I've wrapped both of the timers up in a single file in a somewhat portable form. Because there can be a LOT of clocks in a second (several billion, obviously) the nanotimer is a bit complex -- on the first call it saves the base clock registers as a baseline that it subtracts from subsequent calls BEFORE converting the results to a double precision nanosecond count. Done in the wrong order, you can end up with a significant precision loss subtracting two large numbers to get a very small number, hence the runaround. I'm pretty sure it is done right at this point. I attach the timer source below for anyone who cares. rgb > You wouldn't even need another process taking up time, the kernel will > happily do this for you with some of its housekeeping chores (bdflush, > kswapd, ...). Servicing lots of interrupts is a good way to slow things > down, as are doing many small page allocations where one large one would > suffice, etc. Absolutely. In fact, one of the "fun" things about microbenchmarking is that a good microbenchmarking suite and/or a set of good vmstat-like tools can really help you understand and "see" the actual costs of stuff like this. Josip Loncaric (on this list) worked very hard some years ago to run down a moderately hienous intermittant latency hit in the (IIRC) TCP networking stack, for example. Every now and then instead of a predictable relatively short packet latency a really long (again IIRC, 2 msec or the like compared to timings on the order of tens of usec) hit would appear. Sending lots of small packets is "bad". Doing lots of malloc is "bad". Doing any sort of interrupt is bad, in part because the system has a tendency to swap in waiting/competing tasks (if any) when the interrupt returns and of course because of the relatively high cost of the interrupt itself. Lots of system calls can have the same effect -- select, for example, has about the same cost as gettimeofday, and AFAICT nanosleep is really usleep -- the ONLY way I've found to "pause" for a nanoscale-specified time is to use a nanotimer-based polling loop. FORTUNATELY, most of this isn't horribly difficult to understand or to avoid in actual code, and FORTUNATELY most people really are interested in macroscopic wall-clock completion times of large scale code, where "date" at the shell is accurate enough to get decent timings. Where it makes a difference is in optimizing core loops or understanding how you spent all this money on a 3 GHz CPU system that IN PRINCIPLE has a 6 GFLOPS peak but in practice you are lucky to get 1 GFLOPS on a good day with a tail wind. [Parenthetical Aside: Youngsters don't know HOW lucky you are to get that 1 GFLOPS in a $1000 system, of course. Suffice it to say that as of a few years ago, your desktop would have been classified as a weapon for purposes of export, and sending one outside of the US would have taken a special license, and a decade ago a GFLOPS system (even a cluster) would have cost order of $1 million. Gotta love Moore's Law (and the fact that Our Government, helped enormously by the beowulf movement, finally wised up to Moore's Law and moved any remaining weapons boundaries out so far that they are invisible).] rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu nanotimer: COPYING is GPL v2b, so play on through. You can get the whole package in context in cpu_rate from my personal website and I think on brahma. %< snip snip snip ======================================================= /* *======================================================================== * $Id: nanotimer.h,v 1.1 2003/06/10 15:41:27 rgb Exp $ * * See copyright in copyright.h and the accompanying file COPYING *======================================================================== */ #include "copyright.h" #define K 1024 /* * The following are the struct components of the timer(s). They * should let us write one bit of code and use it with either * timer transparently. */ typedef struct { double (*timer)(); double start; double stop; double delta; double nsec_granularity; char name[K]; } Nanotimer; /* * Built in cycle counter timer (Intel or AMD) resolution * roughly 1 CPU clock (sub-nanosecond on modern CPUs). */ static double nsec_per_cycle; static unsigned long ax_first,dx_first; static unsigned long long count_first; void nanotimer_rdtsc_init(Nanotimer *new_nanotimer); double nanotimer_rdtsc(); /* * Fallback timer using gettimeofday. Resolution of roughly * 2 usec on a good day, but the way we scale test intervals * it doesn't usually matter. */ static struct timeval tv_first; void nanotimer_gttod_init(Nanotimer *nanotimer); double nanotimer_gttod(); %< snip snip snip ======================================================= /* * $Id: nanotimer.c,v 1.9 2003/02/28 21:18:04 rgb Exp $ * * See copyright in copyright.h and the accompanying file COPYING * */ /* *======================================================================== * timing and utility sources. tv_start and tv_stop are globals. *======================================================================== */ #include "cpu_rate.h" /* * The following to programs constitute a NONportable, but very accurate, * intel/amd timer for pentia and athlons (tsc flag in /proc/cpuinfo). * The init call basically extracts the CPU clock from /proc/cpuinfo * and converts it into nanoseconds/cpu clock cycle. The second reads * the cpu cycle counter directly and converts it into nanoseconds. * * To avoid potential problems with loss of precision on the interval * subtraction, this program stores the register contents on the * first call and only counts the nanoseconds from the first call, not * since the last boot. * * It is quite fast (order of 40-50 nsec). */ void nanotimer_rdtsc_init(Nanotimer *new_nanotimer) { int i,imax,l,numfields; char statbuf[K]; char delim[2],*nextval; FILE *cpuinfo_fd; double loop_delta_1000,loop_delta_1000000,loop_ratio; snprintf(new_nanotimer->name,K,"cpu cycle counter nanotimer"); if(verbose == V_TIMER){ printf("#========================================================================\n"); printf("# Initializing timer %s\n",new_nanotimer->name); } new_nanotimer->timer = nanotimer_rdtsc; /* * First we extract the number of nanoseconds per cycle (required!) * from /proc/cpuinfo. AFAICT, this is the easiest/only way to get * it, although I don't know about its precision. This is fairly * standard parsing code (ripped from xmlsysd). */ delim[0] = ':'; /* separator */ delim[1] = (char) NULL; /* string terminator */ cpuinfo_fd = fopen("/proc/cpuinfo","r"); while(-1){ /* Normal EOF causes break from while loop */ if((fgets(statbuf,K,cpuinfo_fd) == NULL)) break; if(strncmp(statbuf,"cpu MHz",7) == 0) { nextval = strtok(statbuf,delim); /* first field skip */ nextval = strtok((char *)NULL,delim); /* second field is it */ nsec_per_cycle = 1000.0/atof(nextval); break; } } fclose(cpuinfo_fd); if(nsec_per_cycle == 0.0){ fprintf(stderr,"Error: Cannot parse out the cpu MHz from /proc/cpuinfo.\n"); fprintf(stderr,"Cannot use built-in CPU cycle counter as timer.\n"); fprintf(stderr,"Try the -g flag to use gettimeofday instead.\n"); exit(0); } /* * Now we set the key elements of the nanotimer struct and TEST its * call granularity. * * We subtract off the time base to ensure that times of * order seconds (and then some) have resolvable differences * in double precision. This is therefore the initial call to * the cpu cycle timer to extract the initial time base and * MUST precede the use of the timer. We do NOT CHECK to keep * the timer fast! */ asm volatile("rdtsc" : "=a" (ax_first), "=d" (dx_first)); count_first = dx_first; count_first = count_first<<32; count_first += ax_first; /* * To determine the granularity, we just start calling it in a loop until * we've accumlated a few zillion calls, and see how long they took on * average. We also crudely time the loop itself, and make sure that * its timing SCALES WITH SIZE. */ new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Empty loop. The printf(""); seems more reliable than anything * including a nanosleep etc. at ensuring that the test start on a * clock cycle break, yielding the most consistent results. */ printf(""); new_nanotimer->start = nanotimer_rdtsc(); for(i=0;i<1000;i++){ } new_nanotimer->stop = nanotimer_rdtsc(); loop_delta_1000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000 = %f\n",loop_delta_1000); printf(""); new_nanotimer->start = nanotimer_rdtsc(); for(i=0;i<1000000;i++){ } new_nanotimer->stop = nanotimer_rdtsc(); loop_delta_1000000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000000 = %f\n",loop_delta_1000000); loop_ratio = loop_delta_1000000/loop_delta_1000; if(verbose == V_TIMER) printf("# loop_ratio = %f (should be near 1000)\n",loop_ratio); /* * It should get bigger. I don't much care how MUCH bigger */ if(loop_ratio < 10){ fprintf(stderr,"Warning: Empty loop measurement not scaling in nanotimer_rdtsc_init()\n"); fprintf(stderr," The compiler may optimize away empty loops!\n"); } new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Full loop. We start right after a clock tick, we hope, and * we subtract out the loop_delta from above. */ printf(""); new_nanotimer->start = nanotimer_rdtsc(); for(i=0;i<1000000;i++){ new_nanotimer->stop = nanotimer_rdtsc(); } new_nanotimer->delta = new_nanotimer->stop - new_nanotimer->start - loop_delta_1000000; if(verbose == V_TIMER) printf("# nanotimer delta = %f for 1000000 iterations\n",new_nanotimer->delta); new_nanotimer->nsec_granularity = new_nanotimer->delta/1000000.0; if(verbose == V_TIMER){ printf("# %s: granularity = %f nanoseconds\n", new_nanotimer->name,new_nanotimer->nsec_granularity); } } double nanotimer_rdtsc() { unsigned long ax, dx; unsigned long long count; double nanotime; asm volatile("rdtsc" : "=a" (ax), "=d" (dx)); count = dx; count = count<<32; count += ax; count -= count_first; nanotime = (double) count; nanotime *= nsec_per_cycle; /* printf("nanotime = %f\n",nanotime); */ return(nanotime); } /* * This is a portable nanosecond timer. It uses gettimeofday (wall clock * time) with the time of the first call subtracted off to keep intervals * from horribly overflowing the double with irrelevant numbers (causing * a loss of precision). Note that my direct measurements show that * gettimeofday() itself takes about 2 usec to complete. */ void nanotimer_gttod_init(Nanotimer *new_nanotimer) { int i; double loop_delta_1000,loop_delta_1000000,loop_ratio; snprintf(new_nanotimer->name,K,"gettimeofday nanotimer"); if(verbose == V_TIMER){ printf("#========================================================================\n"); printf("# Initializing timer %s\n",new_nanotimer->name); } new_nanotimer->timer = nanotimer_gttod; /* * Now we set the key elements of the nanotimer struct and TEST its * call granularity. * * To determine the granularity, we just start calling it in a loop until * we've accumlated a few zillion calls, and see how long they took on * average. We also crudely time the loop itself, and make sure that * its timing SCALES WITH SIZE. */ new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Empty loop. The printf(""); seems more reliable than anything * including a nanosleep etc. at ensuring that the test start on a * clock cycle break, yielding the most consistent results. */ printf(""); new_nanotimer->start = nanotimer_gttod(); for(i=0;i<1000;i++){ } new_nanotimer->stop = nanotimer_gttod(); loop_delta_1000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000 = %f\n",loop_delta_1000); printf(""); new_nanotimer->start = nanotimer_gttod(); for(i=0;i<1000000;i++){ } new_nanotimer->stop = nanotimer_gttod(); loop_delta_1000000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000000 = %f\n",loop_delta_1000000); loop_ratio = loop_delta_1000000/loop_delta_1000; if(verbose == V_TIMER) printf("# loop_ratio = %f (should be near 1000)\n",loop_ratio); /* * It should get bigger. I don't much care how MUCH bigger */ if(loop_ratio < 10){ fprintf(stderr,"Warning: Empty loop measurement not scaling in nanotimer_gttod_init()\n"); fprintf(stderr," The compiler may optimize away empty loops!\n"); } new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Full loop. We start right after a clock tick, we hope, and * we subtract out the loop_delta from above. */ printf(""); new_nanotimer->start = nanotimer_gttod(); for(i=0;i<1000000;i++){ new_nanotimer->stop = nanotimer_gttod(); } new_nanotimer->delta = new_nanotimer->stop - new_nanotimer->start - loop_delta_1000000; if(verbose == V_TIMER) printf("# nanotimer delta = %f for 1000000 iterations\n",new_nanotimer->delta); new_nanotimer->nsec_granularity = new_nanotimer->delta/1000000.0; if(verbose == V_TIMER){ printf("# %s: granularity = %f nanoseconds\n", new_nanotimer->name,new_nanotimer->nsec_granularity); } } double nanotimer_gttod() { struct timeval tv_now; double nanotime; /* * This avoids potential precision problems by computing the starting * time as of the first call, and measuring all subsequent times * relative to this. Gets rid of a LOT of seconds. */ if((tv_first.tv_sec == 0) && (tv_first.tv_usec == 0)){ gettimeofday(&tv_first, (struct timezone *) NULL); } gettimeofday(&tv_now, (struct timezone *) NULL); nanotime = (double)(tv_now.tv_sec - tv_first.tv_sec); nanotime += 1.0e-6*(double)(tv_now.tv_usec - tv_first.tv_usec); /* return nanoseconds */ nanotime *= 1.e+9; return(nanotime); } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 7 11:41:12 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 7 Jan 2004 11:41:12 -0500 (EST) Subject: [Beowulf] Need help In-Reply-To: Message-ID: On Wed, 7 Jan 2004, Maikel Punie wrote: > > hallo everyone, > > well for my final project at school i need to build a beowulf, well i'm this > far that all the node (diskless system) boot from the main node (ntfs) and > there config and stuff from there, the main node can see all off his nodes. > > Now my question is whats the next step? How do i make the nodes do the > calculations and stuff for the main node, how do i realy integrate the > parrallel processing? > > Thanks a lot Visit http://www.phy.duke.edu/~rgb and grab the PVM project template under General. Install PVM on all the nodes and configure it to run. Get a PVM book (there is a good one from MIT press). Run the PVM example programs. OR do the same thing for MPI. OR learn to write raw socket code for doing IPC's. AND consider buying e.g. Ian Foster's book on parallel programming (available online for free in the short run), linked to http://www.phy.duke.edu/brahma under one of the resources links. Two final resources: Subscribe to Cluster World Magazine for more getting started code and tips and so forth. You can get three free trial issues if you live in the US by visiting its website. Subscribe to Linux Magazine ditto -- Forrest Hoffman has been writing an excellent column on cluster computing there for years now, and recent issues (within the last year) have had complete getting started in MPI instructions, for example, with sample code. HTH, rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ken_i_m at fatair.net Wed Jan 7 12:12:54 2004 From: ken_i_m at fatair.net (ken_i_m at fatair.net) Date: Wed, 7 Jan 2004 10:12:54 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <1073436271.15823.30.camel@protein.scalableinformatics.com> Message-ID: <20040107171254.GB4848@hyper> On Wed, Jan 07, 2004 at 11:18:33AM -0500, Robert G. Brown (rgb at phy.duke.edu) wrote:These interrupts and context switches are quite "expensive", > and one is tempted to get off into a discussion of the kernel design and > upper/lower half handlers and ways the system tries to bundle up its > work to minimize impact on running processes, but it would be a moderate > digression. Whereas this digression is merely brief. :-) -- Ken Dyke "Bits at the speed of light" _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Wed Jan 7 11:57:30 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed, 7 Jan 2004 08:57:30 -0800 Subject: [Beowulf] Need help Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C012@orsmsx402.jf.intel.com> From: Maikel Punie; Wednesday, January 07, 2004 7:16 AM > > well for my final project at school i need to build a beowulf, well i'm > this > far that all the node (diskless system) boot from the main node (ntfs) and > there config and stuff from there, the main node can see all off his > nodes. Congratulations! > Now my question is whats the next step? How do i make the nodes do the > calculations and stuff for the main node, how do i realy integrate the > parrallel processing? Try running an MPI or PVM-based program. There are many such codes out there, the biggest problem, though is "how accessible (understandable) are the concepts to those you will present these results too?" While somewhat mundane in terms of parallel processing, try running POV-Ray in serial and parallel modes. For standard POV-Ray, try http://www.povray.org/ and for a parallel version, try http://pvmpov.sourceforge.net/ Alternatively, look at http://chromium.sourceforge.net/ -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From maikel_punie at hotmail.com Wed Jan 7 12:02:59 2004 From: maikel_punie at hotmail.com (Maikel Punie) Date: Wed, 07 Jan 2004 18:02:59 +0100 Subject: [Beowulf] Need help Message-ID: >Try running an MPI or PVM-based program. There are many such codes out >there, the biggest problem, though is "how accessible (understandable) >are the concepts to those you will present these results too?" well thats not my problem :) nah i know it will be hard to explain it, but its a fun project. Maikel _________________________________________________________________ MSN Messenger je favoriete software van 2003? Stem op http://pcmagazine.zdnet.be/pvhj.cfm _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Jan 7 13:41:58 2004 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 07 Jan 2004 11:41:58 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: References: Message-ID: <3FFC52F6.6080503@lanl.gov> Robert G. Brown wrote: > A Brief Digression on Benchmark Timing > > (Yes, I lie like a rug. It is not brief. If you already know about > timing, or don't care, hit 'd' now...:-) Thank you for providing this excellent article. I'd just like to add a few details on fine grained timings, which are indeed useful in tracking down some performance quirks... > gettimeofday is actually a mediocre timer for nearly all > microbenchmarking purposes While gettimeofday() is quite portable as a function call, its timing accuracy is not. On some machines (e.g. Linux PCs) the times are reliable to within a few microseconds. On other systems, true granularity can be 1000 microseconds, apparently related to the scheduler frequency. This seriously undermines the portability of fine grained timings done with gettimeofday(). > A much better timer is available on nearly > all intel/amd cpus. Indeed -- and thank you for your code examples. One minor simplifying comment (I learned this recently) is that GNU C has the ability to read the full 64-bit time stamp counter as follows: unsigned long long tsc; asm volatile ("rdtsc": "=A" (tsc)); where the capitalized "=A" tells gcc that the operand constraints of RDTSC involve both a and d registers, as in: "A -- Specifies the a or d registers. This is primarily useful for 64-bit integer values (when in 32-bit mode) intended to be returned with the d register holding the most significant bits and the a register holding the least significant bits." --http://gcc.gnu.org/onlinedocs/gcc-3.3.2/gcc/Machine-Constraints.html Regarding the RDTSC instruction, those interested in more detail can consult http://cedar.intel.com/software/idap/media/pdf/rdtscpm1.pdf but I'd like to add a few observations: (1) Pentiums execute instructions out-of-order. Although RDTSC instruction will read the 64-bit Time Stamp Counter which advances at the CPU's clock frequency (~3GHz these days), the meaning of the count might not be exactly what you expected. For maximum reliability, one can issue a serializing instruction such as CPUID before RDTSC, at the cost of about 300 clock cycles (CPUID can take variable amount of time to execute). BTW, just doing the RDTSC and moving its result to other registers takes about 86 cycles on a Pentium III. Intel's document suggests some compensation techniques, but in many cases 300 cycle resolution (about 100 nanoseconds) is already OK, so CPUID is optional. Also, older Pentiums (plain or MMX) do not need CPUID. (2) CPU time stamp counters exist on other processors, e.g. PowerPC and Alpha. While 64-bit counters won't overflow for centuries, machines with 32-bit counters (Alpha) experience counter rollovers every few seconds. This must be accounted for in interpreting the results. (3) Thermal management on recent Pentiums duty cycles the clock whenever the processor is in danger of overheating. The clock frequency does not slow down, but the CPU core sees normal/stopped clock with a thermally controlled duty cycle. This begs the question: Is the time stamp counter affected? I hope not, but I do not know for sure. There are reasons for both approaches: Counting either only the cycles seen by the core, or counting all clock cycles; I do not know which way Intel's engineers went. (4) SMP processors present a special challenge in using CPU time stamp counters for precision timings, because the timing process can migrate from one CPU to another, where the counter might have a different offset. Fortunately, Linux checks this on startup, and synchronizes CPU time stamp counters when booted. Provided that all CPUs count the same clock cycles, they should remain synchronized, so process migration need not affect timings. However, this can be a serious problem if the operating system does not ensure that CPU counters are closely synchronized, or if the individual CPU counters drift apart due to some reason (e.g. counting thermally duty-cycled clock, as in (3) above). (5) Precision timings are sometimes available from other sources, e.g. some high end network cards (Quadrics elan_clock() returns nanoseconds). Also, MPI specifies portable MPI_Wtime() and MPI_Wtick() calls, but again, their actual performance varies by MPI implementation and platform (e.g. some MPI implementations use gettimeofday() to return MPI_Wtime() in seconds). (6) All of the above applies to measurements of *local* time only. The notion of *global* time across a cluster is much fuzzier than it ought to be, even though it should be technically possible to keep node clocks synchronized to microseconds across a cluster. NTP helps, but even better synchronization is needed. One approach to collecting fine grained global timings is to work with local timers, then correct their offsets in a post-processing step (this can be tricky, since estimating individual offsets assumes that there were enough common timed events during the run). Here is my challenge to the Beowulf and MPI communities: Achieve microsecond-level *global* synchronization and tracking of all system clocks within a cluster, for application use via portable calls gettimeofday() and/or MPI_Wtime(). > Absolutely. In fact, one of the "fun" things about microbenchmarking is > that a good microbenchmarking suite and/or a set of good vmstat-like > tools can really help you understand and "see" the actual costs of stuff > like this. Josip Loncaric (on this list) worked very hard some years > ago to run down a moderately hienous intermittant latency hit in the > (IIRC) TCP networking stack, for example. Every now and then instead of > a predictable relatively short packet latency a really long (again IIRC, > 2 msec or the like compared to timings on the order of tens of usec) hit > would appear. In Linux kernels 2.0.x, this periodic TCP performance hit was 20 ms (2000:1), in 2.2.x it was 10 ms (1000:1), but Linux 2.4.x finally seems to have fixed this problem without my patches. While my old web page at ICASE went away, history of this Linux TCP quirk can be found at: http://jl-icase.home.comcast.net/LinuxTCP.html http://jl-icase.home.comcast.net/LinuxTCP2.html http://jl-icase.home.comcast.net/LinuxTCP-patches.html Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gropp at mcs.anl.gov Wed Jan 7 13:00:04 2004 From: gropp at mcs.anl.gov (William Gropp) Date: Wed, 07 Jan 2004 12:00:04 -0600 Subject: [Beowulf] Need help In-Reply-To: References: Message-ID: <6.0.0.22.2.20040107115455.0256dd30@localhost> At 09:16 AM 1/7/2004, Maikel Punie wrote: >hallo everyone, > >well for my final project at school i need to build a beowulf, well i'm >this far that all the node (diskless system) boot from the main node >(ntfs) and there config and stuff from there, the main node can see all >off his nodes. > >Now my question is whats the next step? How do i make the nodes do the >calculations and stuff for the main node, how do i realy integrate the >parrallel processing? At the risk of sounding a tiny bit commercial, you should have a look at the second edition of the Beowulf Cluster Computing with Linux (http://mitpress.mit.edu/catalog/item/default.asp?sid=DD198EEC-B4B8-48BD-B728-C28DF093FBD4&ttype=2&tid=9947). The second edition has a great deal of new material on managing and using clusters, including using other approaches to parallelism besides MPI and PVM and using existing software libraries instead of writing your own. Royalties go to a fund to support student travel to IEEE Cluster. Bill _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 7 13:10:45 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 07 Jan 2004 10:10:45 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <1073436271.15823.30.camel@protein.scalableinformatics.com> Message-ID: <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> At 11:18 AM 1/7/2004 -0500, Robert G. Brown wrote: >A Brief Digression on Benchmark Timing >Suffice it to say that if you time LOTS of 100 microsecond loops >separately in a continuous process and look at the results, you're >likely to see the bimodal distribution Joe refers to (or even a fully >multimodal distribution) because SOME of those intervals will be >interrupted and the interrupt/context switch will be "timed" along with >the swapped out loop by the wall-clock timer. This can be somewhat >ameliorated by adding a usleep right before each timing loop iteration, >as the kernel will often take advantage of a task pause to complete >pending interrupts. However, it alas isn't a magic bullet because not >all interrupts are "pendable". That very distribution might be useful as a diagnostic tool, by the way... >This nanotimer is FAR more suitable for use in microbenchmarking -- >relatively small loops where one devoutly hopes to avoid getting nailed >by a context switch as one is interested in "how many floating point >multiplies can the system perform on numbers in its registers/L1 >cache/L2 cache/memory per second". A loop with a few microseconds worth >of operations, fairly likely not to be interrupted if you start >immediately after a timer click, can yield < 1% accuracy. Multimodality >in a distribution of such short loop timings yields "interesting" >information about the system's CPU, cache, memory architecture, and >their interaction with the kernel. This is along the lines of what that nifty benchmark (the name of which eludes me right now) that produces a graph of speed vs computation size does.. You can "see" the effects of cache etc as the problems space gets bigger. (Edward Tufte would be pleased with the results) >I've wrapped both of the timers up in a single file in a somewhat >portable form. Because there can be a LOT of clocks in a second >(several billion, obviously) the nanotimer is a bit complex -- >on the first call it saves the base clock registers as a baseline that >it subtracts from subsequent calls BEFORE converting the results to a >double precision nanosecond count. Done in the wrong order, you can end >up with a significant precision loss subtracting two large numbers to >get a very small number, hence the runaround. I'm pretty sure it is >done right at this point. I attach the timer source below for anyone >who cares. But, except for diagnostic purposes, code tuning, research into the performance of CPU architectures, and the like, are such nanoscale measurements really useful. The real figure of merit is "how long does it take to run my problem" and that's a wall clock time issue. Given a suitably scalable problem, is it worth spending much time making the code 10% faster, or are you better off just adding 10% more CPU resources? > rgb > > > You wouldn't even need another process taking up time, the kernel will > > happily do this for you with some of its housekeeping chores (bdflush, > > kswapd, ...). Servicing lots of interrupts is a good way to slow things > > down, as are doing many small page allocations where one large one would > > suffice, etc. > >Absolutely. In fact, one of the "fun" things about microbenchmarking is >that a good microbenchmarking suite and/or a set of good vmstat-like >tools can really help you understand and "see" the actual costs of stuff >like this. Josip Loncaric (on this list) worked very hard some years >ago to run down a moderately hienous intermittant latency hit in the >(IIRC) TCP networking stack, for example. Every now and then instead of >a predictable relatively short packet latency a really long (again IIRC, >2 msec or the like compared to timings on the order of tens of usec) hit >would appear. This sort of nondeterministic latency is the bane of high performance computing, particularly for those of us with real-time or embedded applications. In these, you typically need to have an absolute bound on maximum time. >Sending lots of small packets is "bad". Doing lots of malloc is "bad". >Doing any sort of interrupt is bad, in part because the system has a >tendency to swap in waiting/competing tasks (if any) when the interrupt >returns and of course because of the relatively high cost of the >interrupt itself. Lots of system calls can have the same effect -- >select, for example, has about the same cost as gettimeofday, and AFAICT >nanosleep is really usleep -- the ONLY way I've found to "pause" for a >nanoscale-specified time is to use a nanotimer-based polling loop. It might be "bad" in the overall performance sense, but it might be "good" in the reduction of latency variability standpoint. If you had a very tightly coupled parallel stream with lots of barrier synchronization, the overall problem might actually run faster with a longer, but more consistent, or more fine grained, latencies, rather than something that usually runs real fast, but occasionally takes a lot longer. Things like heaps and garbage collection are notorious for this kind of thing. >FORTUNATELY, most of this isn't horribly difficult to understand or to >avoid in actual code, and FORTUNATELY most people really are interested >in macroscopic wall-clock completion times of large scale code, where >"date" at the shell is accurate enough to get decent timings. Where it >makes a difference is in optimizing core loops or understanding how you >spent all this money on a 3 GHz CPU system that IN PRINCIPLE has a 6 >GFLOPS peak but in practice you are lucky to get 1 GFLOPS on a good day >with a tail wind. Folks who are squeezing out the last operation per clock cycle typically use external measurement equipment (i.e. a logic analyzer or oscilloscope), because you wouldn't want to waste any precious cycles on internally instrumenting the code. We did a substantial amount of debugging on a loosely coupled cluster of DSPs just looking at the supply current to the processors on a scope, since instantaneous power consumption is very tightly tied to processor utilization. You can see the end of each pass of butterflies in the FFT (very high current) as you set up the next pass(low current), and we could see interrupt handling (low current) at inopportune times causing context switching in the middle of a dense pipelined code section (because we hadn't masked the interrupt). >[Parenthetical Aside: Youngsters don't know HOW lucky you are to get >that 1 GFLOPS in a $1000 system, of course. Suffice it to say that as >of a few years ago, your desktop would have been classified as a weapon >for purposes of export, and sending one outside of the US would have >taken a special license, and a decade ago a GFLOPS system (even a >cluster) would have cost order of $1 million. Gotta love Moore's Law >(and the fact that Our Government, helped enormously by the beowulf >movement, finally wised up to Moore's Law and moved any remaining >weapons boundaries out so far that they are invisible).] Those non-exportable video games with Motorola 68000's were "arms" or "munitions", or even "dual use technologies" , not "weapons"... Such distinctions are important to the folks at Commerce and State. Besides, being an "international arms dealer" has a much better sound to it than "gun runner". Then too, such formerly silly rules helped create news stories about "bad guys" building immense clusters from networked Playstation 2s. I suspect nobody's ever successfully done this; at least to a practical state, running any sort of useful parallel processing. There was a lot of interest in it for a while, and if you go back through the archives, you can probably find it. >James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ken_i_m at fatair.net Wed Jan 7 14:02:07 2004 From: ken_i_m at fatair.net (ken_i_m at fatair.net) Date: Wed, 7 Jan 2004 12:02:07 -0700 Subject: [Beowulf] Need help In-Reply-To: <6.0.0.22.2.20040107115455.0256dd30@localhost> References: <6.0.0.22.2.20040107115455.0256dd30@localhost> Message-ID: <20040107190207.GC5273@hyper> On Wed, Jan 07, 2004 at 12:00:04PM -0600, William Gropp (gropp at mcs.anl.gov) wrote: > the second edition of the Beowulf Cluster Computing with Linux Dang, didn't know the 2nd ed was coming out. Bought the 1st not 2 months ago... -- Ken Dyke "Bits at the speed of light" _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From twhitcomb at apl.washington.edu Wed Jan 7 15:34:36 2004 From: twhitcomb at apl.washington.edu (Timothy R. Whitcomb) Date: Wed, 7 Jan 2004 12:34:36 -0800 (PST) Subject: [Beowulf] beostatus not showing master node Message-ID: Any ideas why the "beostatus" program would show only slave nodes for a while (several months) then display the master node (node -1) for a while (another several months) and then go back to displaying only the slave nodes again with no obvious changes in the software? TRW _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From twhitcomb at apl.washington.edu Wed Jan 7 15:32:12 2004 From: twhitcomb at apl.washington.edu (Timothy R. Whitcomb) Date: Wed, 7 Jan 2004 12:32:12 -0800 (PST) Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) Message-ID: First of all, thanks to everyone who responded to my first message. I received many helpful suggestions to test, which I have documented here. As a quick review of my (still unsolved) situation, we are running a weather model on a 5-node, 10-processor Scyld Beowulf cluster. As long as we use a single processor per machine, performance is fine - but when we use both processors on a single node then the node freezes after a variable length of time (sometimes minutes, sometimes hours). The suggestions I received are summarized as I have tested them: -Make sure kernel is compiled for athlon architecture > Kernel is indeed compiled for Athlon and SMP -Try 1 IDE disk/cable > There is only 1 disk/cable in the slave nodes -Manually ventilate cases > no change in results when cases were opened up and fanned -Replace NIC cables > replaced all NIC cables with no change in results -Tyan 2466 motherboards have stability issues > This is not a 2466 motherboard -Check with the Navy and see if this is run with SMP > The Naval Research Lab runs the model using SMP machines and a Linux cluster (though not necessarily Scyld) -Check thermal problems > no change -Flash update the BIOS > no change, BIOS updated to most recent version -Run CPUBURN for several days > ran over the weekend on all nodes, system seemed stable -Try new network drivers > Updated network drivers to Becker's most recent 3c59x for the dual 3Com 3c982 Server Tornado built-in Ethernet, but with no apparent change The suggestions that I have not been able to try as yet: -Try upgrading the kernel and bproc package Since this is a Scyld system purchased from a retailer, the kernel has all sorts of patches in it. Can anyone point me to anywhere detailing how to do a kernel upgrade on one of these systems? I've built kernels before but usually only using the kernel.org sources. I'd also like to know if it is possible for the cluster interconnects to be "too slow" for the software - there is a 100BaseT ethernet between the nodes but there is a _lot_ of data being passed around. Something else I have noticed is that if I run a 2-processor run on 2 nodes, it goes fairly quickly. However, if I switch it to 2 processors on 1 machine, it stalls for very long periods of time. >From the memory usage, it looks like there's quite a bit in the swap memory (there's 512MB/node of RAM and 1GB of swap) but there is little to no CPU access on either processor for very long periods of time (on the order of several minutes and up) whereas in the 2-node case there are brief pauses, but CPU usage jumps up to 100% fairly quickly. On the 1-node case, one processor will go up a little but then fall back down quickly. Are there any other ideas as to what would be causing these nodes to freeze up? Thanks again for all the help I've received. Tim Whitcomb _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 7 16:04:43 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 07 Jan 2004 16:04:43 -0500 Subject: [Beowulf] beostatus not showing master node In-Reply-To: References: Message-ID: <1073509483.4697.8.camel@pel> On Wed, 2004-01-07 at 15:34, Timothy R. Whitcomb wrote: > Any ideas why the "beostatus" program would show only slave nodes for a > while (several months) then display the master node (node -1) for a while > (another several months) and then go back to displaying only the slave > nodes again with no obvious changes in the software? What version of Scyld are you running? You may want to contact Scyld and see if they have a newer version. I recall this problem and also recall fixing it when I was still at Scyld. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From twhitcomb at apl.washington.edu Wed Jan 7 16:11:23 2004 From: twhitcomb at apl.washington.edu (Timothy R. Whitcomb) Date: Wed, 7 Jan 2004 13:11:23 -0800 (PST) Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: <1073509387.4697.5.camel@pel> References: <1073509387.4697.5.camel@pel> Message-ID: No, I'm not using myrinet. It's a Scyld 28cz4 with 100BaseT interconnects, dual-processor Tyan AMD boards. TRW On Wed, 7 Jan 2004, Jag wrote: > On Wed, 2004-01-07 at 15:32, Timothy R. Whitcomb wrote: > > First of all, thanks to everyone who responded to my first message. I > > received many helpful suggestions to test, which I have documented > > here. > > > > As a quick review of my (still unsolved) situation, we are running a > > weather model on a 5-node, 10-processor Scyld Beowulf cluster. As > > long as we use a single processor per machine, performance is fine - > > but when we use both processors on a single node then the node freezes > > after a variable length of time (sometimes minutes, sometimes hours). > > Unfortunately I missed your first post. Are you using myrinet? I seem > to recall some past problems with two MPI jobs on one host running myri > on Scyld systems. > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 7 16:03:07 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 07 Jan 2004 16:03:07 -0500 Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: References: Message-ID: <1073509387.4697.5.camel@pel> On Wed, 2004-01-07 at 15:32, Timothy R. Whitcomb wrote: > First of all, thanks to everyone who responded to my first message. I > received many helpful suggestions to test, which I have documented > here. > > As a quick review of my (still unsolved) situation, we are running a > weather model on a 5-node, 10-processor Scyld Beowulf cluster. As > long as we use a single processor per machine, performance is fine - > but when we use both processors on a single node then the node freezes > after a variable length of time (sometimes minutes, sometimes hours). Unfortunately I missed your first post. Are you using myrinet? I seem to recall some past problems with two MPI jobs on one host running myri on Scyld systems. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wade.hampton at nsc1.net Wed Jan 7 16:36:23 2004 From: wade.hampton at nsc1.net (Wade Hampton) Date: Wed, 07 Jan 2004 16:36:23 -0500 Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: References: <1073509387.4697.5.camel@pel> Message-ID: <3FFC7BD7.3020402@nsc1.net> Timothy R. Whitcomb wrote: >No, I'm not using myrinet. It's a Scyld 28cz4 with 100BaseT >interconnects, dual-processor Tyan AMD boards. > I too missed your first post, but this thread reminded me of problems I had 6 or so months ago. We had some node hangs with Scyld our cluster, however we were using Tyan dual XEON boards, not AMD. We also had 28cz4. We were using the 100T interface (not the GB). A reboot of the node fixed the problem. We've had problems with this cluster recently, but it is now using the GB interfaces and I have not seen this problem. (Note: getting GB working in this config was a PITA.) Is the node locking up or is it just losing networking? Can you attach monitors to the nodes and see if it is still running when it "locks up"? My head node (same Tyan dual MB as my nodes, hyperthread enabled), would sometimes lose the eepro100 100T connection (outside world), but the e1000 (cluster) interface kept working. However, I would still have to reboot the head node. This would happen every few months initially, but then happened almost every day. I had to install a PCI NIC and disable the onboard 100T interface. It worked fine for months after that (until I lost my RAID -- another story). -- Wade Hampton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 7 16:53:21 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 07 Jan 2004 13:53:21 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFC52F6.6080503@lanl.gov> References: Message-ID: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> A >Here is my challenge to the Beowulf and MPI communities: Achieve >microsecond-level *global* synchronization and tracking of all system >clocks within a cluster, for application use via portable calls >gettimeofday() and/or MPI_Wtime(). Is added hardware on each node legal along with distributing some periodic time mark (like a 1 pulse per second from a GPS receiver)? All you need is a free running counter which can be latched in an atomic operation, driven from a clock of suitable speed (for submicrosecond timing, the clock must be >1MHz), derived from the clock you wish to track. There must be something suitable on the run of the mill PC these days. I haven't looked in detail at this recently, but, on an original PC which used an 8253 timer chip for a number of purposes, one could do this fairly easily. Perhaps the generally unused "game port" on the mobo might work for this. One could certainly build a very inexpensive unit with a USB interface to do what's needed. One of the inexpensive USB data acquisition pods could probably do it, although Linux support seems to be a bit spotty on these, and you'd need to cobble up an interface from the processor board clock to the pod. I'm looking into seeing if the IrDA interface available on many mobos these days might provide a convenient way to distribute the reference time marks (optical, diffuse coupling) With hardware that happened to support this kind of thing (i.e. a DSP with a specialized peripheral interface chip that had UARTs, timers, etc. on it), we were able to synchronize multiple processors and their associated analog to digital converters by this technique to substantially better than 100 nanoseconds absolute. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 7 16:29:59 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 07 Jan 2004 16:29:59 -0500 Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: References: <1073509387.4697.5.camel@pel> Message-ID: <1073510998.4697.19.camel@pel> On Wed, 2004-01-07 at 16:11, Timothy R. Whitcomb wrote: > No, I'm not using myrinet. It's a Scyld 28cz4 with 100BaseT > interconnects, dual-processor Tyan AMD boards. Hm.. I believe they have 29cz out now. If nothing else, that should solve the beostatus problem. (The beostatus problem is actually a bug that's potentially triggered based on the date/time) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rodmur at maybe.org Wed Jan 7 18:25:56 2004 From: rodmur at maybe.org (Dale Harris) Date: Wed, 7 Jan 2004 15:25:56 -0800 Subject: [Beowulf] pvm and bproc 4.0 Message-ID: <20040107232556.GR11808@maybe.org> Thought I would try this list to see if anyone has some possible insights. Basically I'm trying to get PVM to work with bproc 4.0, but having some problems. Initially I was a having some little problems with some node naming conventions changing with bproc 4.0, but I've worked around those problems (instead of identifying nodes like .1 for gethostbyname() they use n1). However, the problem I've ran into now is that I can get PVM to work with bproc 4.0 if I run PVM as root, but it doesn't work for normal users. Which is kind of a problem. It appears that pvm isn't able to access a socket, that it should be able to do. So I see errors like: [t80040000] 01/07 18:30:28 n-1 (192.168.2.1:33095) BEOSCYLD 3.4.4 [t80040000] 01/07 18:30:28 ready Wed Jan 7 18:30:28 2004 [t80000000] 01/07 18:30:32 stderr at n40: /proc/self/fd/3: /proc/self/fd/3: Permission denied [t80000000] 01/07 18:30:32 stdout at n40: EOF fd3 points to a Unix socket. Anyone have any insights? PVM isn't supported like it used to be, unfortunately. -- Dale Harris rodmur at maybe.org /.-) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Jan 7 19:18:46 2004 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 07 Jan 2004 17:18:46 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> Message-ID: <3FFCA1E6.9070905@lanl.gov> Jim Lux wrote: > Josip Loncaric wrote: > >> Here is my challenge to the Beowulf and MPI communities: Achieve >> microsecond-level *global* synchronization and tracking of all system >> clocks within a cluster, for application use via portable calls >> gettimeofday() and/or MPI_Wtime(). > > > Is added hardware on each node legal along with distributing some > periodic time mark (like a 1 pulse per second from a GPS receiver)? I do not think this would be required, especially on clusters which already have a high performance network. These days, 1/2 round trip time can be as low as 2.4 microseconds (Quadrics). Even Fast Ethernet may prove sufficient (over time) if the delays are not too noisy. BTW, the objective here is to support parallel timings within a cluster, so "global time" is still "local to this cluster only", although one may want to run NTP on the head node to stay close to the official time. Back when VXIbus was introduced, I really liked its extra trigger lines which could synchronize multiple instruments within a few nanoseconds, but it is unlikely that people would distribute clock that way in a cluster. Despite various software/hardware network layers and longer distances involved, it should be possible to get microsecond synchronization through the network by gradually tweaking local clock offset and drift... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 7 19:35:36 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 07 Jan 2004 16:35:36 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFCA1E6.9070905@lanl.gov> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: >>Josip Loncaric wrote: >> >>>Here is my challenge to the Beowulf and MPI communities: Achieve >>>microsecond-level *global* synchronization and tracking of all system >>>clocks within a cluster, for application use via portable calls >>>gettimeofday() and/or MPI_Wtime(). >> >>Is added hardware on each node legal along with distributing some >>periodic time mark (like a 1 pulse per second from a GPS receiver)? > >I do not think this would be required, especially on clusters which >already have a high performance network. These days, 1/2 round trip time >can be as low as 2.4 microseconds (Quadrics). Even Fast Ethernet may >prove sufficient (over time) if the delays are not too noisy. BTW, the >objective here is to support parallel timings within a cluster, so "global >time" is still "local to this cluster only", although one may want to run >NTP on the head node to stay close to the official time. > >Back when VXIbus was introduced, I really liked its extra trigger lines >which could synchronize multiple instruments within a few nanoseconds, but >it is unlikely that people would distribute clock that way in a >cluster. Despite various software/hardware network layers and longer >distances involved, it should be possible to get microsecond >synchronization through the network by gradually tweaking local clock >offset and drift... Most of the more recent implementations of NTP clients can do this sort of thing, sending messages both ways across the link to measure the latency (and latency distribution) There's a fairly big literature on synchronizing clocks over noisy, unreliable, non-deterministic communication links. (Some of the basic theory is tied into what's called the "Byzantine Generals" problem.) I wasn't so much advocating synchronization in an absolute sense, but that one can accurately synchronize a cluster to much better than a microsecond, if one can distribute a common time mark to all processors in some way. A really crummy communications network, plus a single time hack, can do it. In some ways, I guess this could be considered a classic barrier synchronization scheme. Conceptually, the time synchronization process in all threads gets to the barrier and waits for the pulse to release them. In my own implementation, part of the work was done by moderately specialized hardware. While reasonably deterministic in their execution time, even DSPs weren't good enough. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 7 19:46:46 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 7 Jan 2004 19:46:46 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 7 Jan 2004, Jim Lux wrote: > This is along the lines of what that nifty benchmark (the name of which > eludes me right now) that produces a graph of speed vs computation size > does.. You can "see" the effects of cache etc as the problems space gets > bigger. (Edward Tufte would be pleased with the results) Well, that's one of the things cpu_rate does (the package from which these timers were lifted. It run one of a variety of stream-like tests but uses malloc instead of a hard data allocation so that one can vary the vector size for vector tests. In fact, all the stream tests are embedded in cpu_rate, even though they are more correctly viewable as memory tests, or maybe integrated architecture (CPU and memory together) tests. cpu_rate also contains the old savage benchmark and a stream extension that includes division (division is typically MUCH SLOWER than multiplication and addition, making it "unsexy" to include in a benchmark designed to show how fast your system is, although I find that my own code has plenty of divisions in it:-). It doesn't provide a graph per se -- it provides a bogomegarate. However, there is a script included that will usually generate a file of rates at a range of vector sizes that can then easily be graphed. All available on brahma or my personal website. lmbench is also a very good microbenchmark, by the way -- arguably better than cpu_rate. Larry has been in the benchmark business longer and Carl Staelin also is very smart. For a while, though, Larry had some "odd" rules on the benchmark (designed not to close the source but rather to keep vendors from corrupting its application to their systems). I just prefer a straight GPL, so I developed a VERY old "FLOPS" benchmark suite of mine (dating back to the late 80's or early 90's -- I think I originally wrote it using DeSmet C if any old PC users recognize it:-) into a microbenchmark I could use and publish without care. > But, except for diagnostic purposes, code tuning, research into the > performance of CPU architectures, and the like, are such nanoscale > measurements really useful. The real figure of merit is "how long does it > take to run my problem" and that's a wall clock time issue. Given a > suitably scalable problem, is it worth spending much time making the code > 10% faster, or are you better off just adding 10% more CPU resources? Agreed. Although ATLAS is a pretty clear demonstration that sometimes it's a matter of 200-300% faster. Algorithm and tuning CAN matter a lot, and it is worth learning "enough" to know whether that is likely to be the case for your code. Just something like inverting a value so you can multiply it many times inside a core loop instead of divide it many times inside a core loop can make a BIG difference in core loop timing... > Folks who are squeezing out the last operation per clock cycle typically > use external measurement equipment (i.e. a logic analyzer or oscilloscope), > because you wouldn't want to waste any precious cycles on internally > instrumenting the code. We did a substantial amount of debugging on a > loosely coupled cluster of DSPs just looking at the supply current to the > processors on a scope, since instantaneous power consumption is very > tightly tied to processor utilization. You can see the end of each pass of > butterflies in the FFT (very high current) as you set up the next pass(low > current), and we could see interrupt handling (low current) at inopportune > times causing context switching in the middle of a dense pipelined code > section (because we hadn't masked the interrupt). Now THAT'S serious performance optimization. > Those non-exportable video games with Motorola 68000's were "arms" or > "munitions", or even "dual use technologies" , not "weapons"... Such > distinctions are important to the folks at Commerce and State. Besides, > being an "international arms dealer" has a much better sound to it than > "gun runner". Then too, such formerly silly rules helped create news > stories about "bad guys" building immense clusters from networked > Playstation 2s. I suspect nobody's ever successfully done this; at least to > a practical state, running any sort of useful parallel processing. There > was a lot of interest in it for a while, and if you go back through the > archives, you can probably find it. They were just worried that bad persons/countries would use the systems to simulate critical nuclear cores so that they could build small/efficient devices without having to actually test them. Correctly worried -- I imagine that my laptop would work just fine to simulate at least some aspects of a nuclear chain reaction, given the right code. I personally never worried much about this. The nuclear bomb is 1940's technology. Give me access to U235 or plutonium and I'll build one. Moderately efficient non-implosion uranium bombs are trivial to build -- literally garage stuff. Implosion/plutonium bombs are more difficult -- explosive lenses and so forth -- but c'mon, this is 2004. Saddam had functional test pits at the end of the FIRST gulf war -- he just didn't have the nuclear grade material (from what was reported at the time). Conventional explosives are pretty well known by now. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 08:01:16 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 08:01:16 -0500 (EST) Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: Message-ID: On Wed, 7 Jan 2004, Timothy R. Whitcomb wrote: > Something else I have noticed is that if I run a 2-processor run on 2 > nodes, it goes fairly quickly. However, if I switch it to 2 > processors on 1 machine, it stalls for very long periods of time. > >From the memory usage, it looks like there's quite a bit in the swap > memory (there's 512MB/node of RAM and 1GB of swap) but there is little > to no CPU access on either processor for very long periods of time (on > the order of several minutes and up) whereas in the 2-node case there > are brief pauses, but CPU usage jumps up to 100% fairly quickly. On > the 1-node case, one processor will go up a little but then fall back > down quickly. > > Are there any other ideas as to what would be causing these nodes to > freeze up? Thanks again for all the help I've received. I'm assuming that you've looked at and for memory leaks or just plain running out of memory. It is difficult for me to think offhand why a dual CPU job would leak and a single CPU job wouldn't (except leak faster and maybe crash sooner. It is pretty easy to believe that one job uses (at some point) 60% of available memory and the second job (at the same point) uses 60% of available memory -- or whatever -- enough to push it over a critical threshold. IF your system ever really runs out of memory it will definitely die shortly after the kernel requests a page and can't get it. Often die horribly. If this were 1997 or 1998, I'd also suspect an SMP kernel deadlock on some interrupt controlled resource. In 2003 that's a bit tougher to imagine, but it still isn't impossible to imagine either a memory management issue with your particular kernel snapshot (the one in the Scyld version you bought) or something LIKE a deadlock associated with your particular hardware combination. Over the years, I have very definitely seen certain motherboard/memory combinations that were just poison to linux kernels -- sometimes in common enough usage that they'd get fixed; in a few cases they took their instability with them to their early graves. SMP kernels are generally marginally less stable than UP kernels, although honestly it has been maybe 2-3 years since I last had an issue with any linux kernel at all (and I run lots of SMP systems). Still, two things to try are: a) Try a different version of Scyld, either earlier or later (one with a different kernel). See if problems magically vanish. Yes, even earlier kernels can be more stable -- some problems don't exist, then appear in a particular snapshot but make it through the "stable" selection process to make it out into the world as a plague, then vanish with a later kernel as somebody puts the offending variable or struct or algorithm back in order. Oh, and the issue isn't just kernel -- there have been significant variations in e.g. glibc over the last year plus -- any linked library could be the offender, although some bugs would cause more mundane/recoverable crashes in userspace. b) Try a non-Scyld linux on one or two boxes, e.g. a vanilla RH 9. Run your job to see if it crashes. This gives you the ability to play a LOT more with kernels, including building new stable or unstable kernels, and gives you a base that is in active use with patch feedback on a LOT more platforms and hence a lot less likely to contain unpatched critical bugs. Editorial comment: The ONE THING that I think really hurts packages like Scyld and Mosix is the fact that they tend to significantly lag commercial linux kernel update streams, and the kernel update streams are one of the best of the various reasons to consider commercial linuces in the first place. Kernel updates in the commercial streams tend to fairly rapidly address critical instabilities that sneak into a released kernel snapshot, and they also address the rare but real kernel-level security hole that sneak through ditto. Over the years I've played the roll-your-own kernel game (quite extensively) and although it is fairly straightforward on a per-system basis, it isn't terribly easy to come up with a configuration schema for a good general purpose/portable/installable kernel that can be built "identically" for smp, amd, i386, i686, etc and installed across a LAN AND to keep it regularly rebuilt and updated. In fact it is a lot of work -- work it is worth it to pay somebody else to do for you. It is perhaps a silly vision, but I still dream of the day when linux comes prewrapped to be automagically built, top to bottom, by fully automated tools so that updating a particular package is a matter of dropping in a new e.g. src rpm and doing nothing. I'm hoping that future revisions of yum and/or caosity eventually move it in this direction. Of course, this still begs the question of testing and validation for commercial application and won't remove the raison d'etre of the big commercial linuces, but it will mean that perhaps the smaller commercial linuces like Scyld can lock their offerings more tightly to the current patch levels of the larger linuces. I mean, these are COMPUTERS we're talking about, for God's sake. They run PROGRAMS. PROGRAMS are these nifty things that do work for you in an automated and moderately reproducible way. So why does building stuff still require so much handiwork? rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 09:10:24 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 09:10:24 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 7 Jan 2004, Jim Lux wrote: > >Here is my challenge to the Beowulf and MPI communities: Achieve > >microsecond-level *global* synchronization and tracking of all system > >clocks within a cluster, for application use via portable calls > >gettimeofday() and/or MPI_Wtime(). > > Is added hardware on each node legal along with distributing some periodic > time mark (like a 1 pulse per second from a GPS receiver)? IIRC, we discussed this on list within the last couple of years. There were only two approaches that seemed worth considering -- a network based schema that used ntp or an ntp-like daemon to gradually refine the system local clock to be accurate within a microsecond across a network. This isn't horribly easy on a network with a latency on the order of hundreds of microseconds, but it is doable -- it just takes a lot of packets and complete control of the network. This synchronization won't be accurate to absolute time unless the master clock is, but it would be synchronization. It seems reasonable to do this as a grant funded project. I've got a couple or three student hackers handy that might tackle it if I could get it funded (with a very small grant). The nice thing about this approach is that it could be a fully GPL, fully portable solution that could be built right into e.g. a bproc cluster. Note also that all that is really required in terms of counters is to be able to match the distributed/refined time base to a particular value of the CPU cycle counter. Then a trivial revision of the non-portable CPU-based nanoscale gettimeofday I posted (or an architecture-aware replacement of the archaic gettimeofday library functions) would make this effectively transparent. The second approach is hardware, which seems perfectly "legal" but which requires an additional investment and which pushes one a bit away from the "commodity" ideal and into the "hardware hacker" ideal which is not everybody's cup of tea. GPS is (as was discussed) a good way to get a very accurate time base, with the caveat that GPS signals are very weak (my handheld GPS won't work indoors, or even inside a car, let alone in a basement server room underneat a few hundred tons of steel and concrete). GPS is getting to be VERY cheap -- almost all cell phones now apparently have built in GPS, and for $45 or so I bought a GPS-sync'd clock from Wal-Mart (which won't work inside the house because GPS signals are so weak, sigh). GPS thus seems like something that would actually be fairly tricky to get working inside a lot of server room environments unless an external antenna were used and piped down to the device, which would add to the expense and hassle and be moderately inefficient on a per-system basis. I have my own reasons for liking the design you indicate below, and would like it very much if two PCs separated by tens of kilometers and not even ON a common network could synchronize an internal clock to less than a microsecond, better yet to less than 100 nanoseconds. For this to happen, the whole thing REALLY needs to move onto the motherboard and be tightly integrated with the system design, although a PCI card or USB device both are alternatives that would likely "work", especially if one wrote kernel drivers for them that could do the entire job of reading and setting the internal cycle-counter-to-absolute-time variable as an atomic operation AND actually correct the result for the time it took to do the set. If one could build a hardware device that kept nanosecond-accurate time, it should be possible to write a gettimeofday or inline assembler fragment that is accurate to within a few nanoseconds (the minimal overhead associated with reading the counter and saving the result as a timestamp for later conversion). Perhaps Intel and/or AMD would consider (or are considering) adding a GPS receiver to their future mobo designs. Add a single port on the back for an external antenna, or figure out how to use some of the considerable quantity of metal attached to the system as a default antenna. Sell amplified GPS antennae as an inexpensive add-on. Or ditto on a PCI card, but again one has to find a manufacturer who will make and sell it for $25, or maybe $35, or it won't be worth it to most people and the price then has to rise. Personally I can think of all sorts of benefits to having each and every PC clock in the universe set and locked to real, absolute time within a few nanoseconds, and similarly located in space to within a few meters (those same nanoseconds times c). The little radiotelescope project I've got a student (still, sigh) working on being one of them, but far from the only one. I'd really like to be able to sit at my PC and have it go "happy new year" at PRECISELY the new year, for example. While we are at it sailing around in the blue sky, can we also put an entropy generator on the add-on card? Something that can generate random bits (in parallel) at oh, 32 GHz, buffer them internally, and deliver the entire buffer to memory via DMA on a single interrupt? My systems get SO tired of generating pseudorands...:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 10:44:42 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 08:44:42 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> Message-ID: <3FFD7AEA.4060709@lanl.gov> Jim Lux wrote: > > At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: > >> Despite various software/hardware network >> layers and longer distances involved, it should be possible to get >> microsecond synchronization through the network by gradually tweaking >> local clock offset and drift... > > > Most of the more recent implementations of NTP clients can do this sort > of thing, sending messages both ways across the link to measure the > latency (and latency distribution) I love NTP, but in its current state it does not deliver microsecond level synchronization. I just checked my old cluster (which runs NTP) and the compute nodes are off by 500-25000 microseconds. Actually, most are within 1000 microseconds, but about 20% are off by up to 25000 microseconds. This happens over Fast Ethernet where ntpdate usually reports timing dispersion of 0-20 microseconds (based on 7 ping-pong measurements). In other words, network timing is quite precise, but we'd need about 1000 times better clock synchronization. NTP may need to be streamlined and tuned for clusters... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kalyanigoli at yahoo.com Thu Jan 8 03:35:32 2004 From: kalyanigoli at yahoo.com (kalyani goli) Date: Thu, 8 Jan 2004 00:35:32 -0800 (PST) Subject: [Beowulf] have u encountered this problem Message-ID: <20040108083532.89213.qmail@web20203.mail.yahoo.com> Hi all, Iam new to this mailing list. Iam doing a project related to MPI. when running the testcase programs in mpich, ch_p4 device ,example cpi program iam getting the following error. could u plz tell me how to solve this problem. [kalyani at mpi1 basic]$ mpirun -np 5 systest Process 3 is alive on mpi1 Process 1 is alive on mpi2 Process 0 is alive on mpi1 Process 2 is alive on mpi3 Process 4 is alive on mpi2 p3_2940: p4_error: Timeout in establishing connection to remote process: 0 bm_list_2926: (303.941183) wakeup_slave: unable to interrupt slave 0 pid 2925 bm_list_2926: (303.941283) wakeup_slave: unable to interrupt slave 0 pid 2925 p1_1579: p4_error: net_recv read: probable EOF on socket: 1 p4_1593: p4_error: net_recv read: probable EOF on socket: 1 Iam running this program on three machines. mpi,mpi2,mpi3.Iam able to do ssh on all the machines. Thank you all in advance __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 8 11:29:46 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 8 Jan 2004 08:29:46 -0800 Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results References: Message-ID: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> ----- Original Message ----- Subject: Re: [Beowulf] ...Re: Benchmark results > On Wed, 7 Jan 2004, Jim Lux wrote: > > > >Here is my challenge to the Beowulf and MPI communities: Achieve > > >microsecond-level *global* synchronization and tracking of all system > > >clocks within a cluster, for application use via portable calls > > >gettimeofday() and/or MPI_Wtime(). > > > > Is added hardware on each node legal along with distributing some periodic > > time mark (like a 1 pulse per second from a GPS receiver)? > > IIRC, we discussed this on list within the last couple of years. There > were only two approaches that seemed worth considering -- a network > based schema that used ntp or an ntp-like daemon to gradually refine the > system local clock to be accurate within a microsecond across a network. > This isn't horribly easy on a network with a latency on the order of > hundreds of microseconds, but it is doable -- it just takes a lot of > packets and complete control of the network. This synchronization won't > be accurate to absolute time unless the master clock is, but it would be > synchronization. > > It seems reasonable to do this as a grant funded project. I've got a > couple or three student hackers handy that might tackle it if I could > get it funded (with a very small grant). The nice thing about this > approach is that it could be a fully GPL, fully portable solution that > could be built right into e.g. a bproc cluster. Note also that all that > is really required in terms of counters is to be able to match the > distributed/refined time base to a particular value of the CPU cycle > counter. Then a trivial revision of the non-portable CPU-based > nanoscale gettimeofday I posted (or an architecture-aware replacement of > the archaic gettimeofday library functions) would make this effectively > transparent. But, does the CPU cycle counter run at constant speed? Or does it ratchet up and down with power management, for instance. This was a curse for us in the DSP world, where we did alter CPU clock rate(as in stopping the processor) to reduce power during "idle time", which is why we went to using a counter that happened to be in a peripheral chip. > > The second approach is hardware, which seems perfectly "legal" but which > requires an additional investment and which pushes one a bit away from > the "commodity" ideal and into the "hardware hacker" ideal which is not > everybody's cup of tea. GPS is (as was discussed) a good way to get a > very accurate time base, with the caveat that GPS signals are very weak > (my handheld GPS won't work indoors, or even inside a car, let alone in > a basement server room underneat a few hundred tons of steel and > concrete). I think that one can probably do it with no mods to the motherboard or outside hardware. For instance, the NTP servers can bring in the 1pps on one of the serial port lines. And, "legacy free" notwithstanding, most motherboards still have a parallel printer port. Perhaps something along the lines of PAPERS? > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > have built in GPS, In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest widget, a GPS receiver that's about 10x15 mm, and can do what's called "assisted" fixes, where the cell site does a lot of the work in terms of keeping track of satellite ephemerides, and so forth. (Greatly reducing the processing, and power, in the phone). It's all part of the E911 mandate which requires cellphone providers to give locations within a few meters when someone makes a 911 call on their cellphone. (Such position information is also quite commercially valuable.. they could send you customized SMS text messages: "Looking for Starbucks? 2 blocks ahead on the left?") and for $45 or so I bought a GPS-sync'd clock from > Wal-Mart (which won't work inside the house because GPS signals are so > weak, sigh). GPS thus seems like something that would actually be > fairly tricky to get working inside a lot of server room environments > unless an external antenna were used and piped down to the device, which > would add to the expense and hassle and be moderately inefficient on a > per-system basis. That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least most of the widely advertised "atomic clocks" do that...) You don't need to have GPS on each computer (in fact, that would be an egregious waste of money), just the sync signal piped to them all. A single wire, pulsing at 1pps, is sufficient. Put the GPS somewhere convenient. You can send the 1pps down cat5 wire, or on a fiber optic link, or almost anything. Even better (if you're into this kind of thing), is a GPS timing receiver like the surplus Z3801, which use GPS to "discipline" a very high quality quartz oscillator, which then generates a very clean and precise 10MHz, as well as the 1 pps. You can also generate precision IRIG (or AES/EBU or SMPTE) time code, and distribute that. > > I have my own reasons for liking the design you indicate below, and > would like it very much if two PCs separated by tens of kilometers and > not even ON a common network could synchronize an internal clock to less > than a microsecond, better yet to less than 100 nanoseconds. Such things are available off the shelf now, just not as a commodity. I would think, though, that a Z3801 type receiver ($250 surplus) hooked to each PC can do it to microsecond precision. Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all these sorts of things, with external trigger inputs and so forth. For this > to happen, the whole thing REALLY needs to move onto the motherboard and > be tightly integrated with the system design, although a PCI card or USB > device both are alternatives that would likely "work", especially if one > wrote kernel drivers for them that could do the entire job of reading > and setting the internal cycle-counter-to-absolute-time variable as an > atomic operation AND actually correct the result for the time it took to > do the set. If one could build a hardware device that kept > nanosecond-accurate time, it should be possible to write a gettimeofday > or inline assembler fragment that is accurate to within a few > nanoseconds (the minimal overhead associated with reading the counter > and saving the result as a timestamp for later conversion). This might already exist on the motherboard.. one just needs to find something that latches the counter deterministically. I'd look at the game port interface. > > Perhaps Intel and/or AMD would consider (or are considering) adding a > GPS receiver to their future mobo designs. Add a single port on the > back for an external antenna, or figure out how to use some of the > considerable quantity of metal attached to the system as a default > antenna. Sell amplified GPS antennae as an inexpensive add-on. Nahh.. have the user buy a GPS receiver pod that is connected via a power/data cable and keep all the RF stuff at the antenna. Like the DeLorme EarthMate or TripMate widgets. The key is getting the 1pps tick into the box in a "nice" fashion. > > Personally I can think of all sorts of benefits to having each and every > PC clock in the universe set and locked to real, absolute time within a > few nanoseconds, and similarly located in space to within a few meters > (those same nanoseconds times c). The little radiotelescope project > I've got a student (still, sigh) working on being one of them, but far > from the only one. I'd really like to be able to sit at my PC and have > it go "happy new year" at PRECISELY the new year, for example. This is what LOFAR and the SKA are all about... I think the Allen Telescope Array up in Hat Creek is also using GPS disciplined clocks to sync everything. > > While we are at it sailing around in the blue sky, can we also put an > entropy generator on the add-on card? Something that can generate > random bits (in parallel) at oh, 32 GHz, buffer them internally, and > deliver the entire buffer to memory via DMA on a single interrupt? My > systems get SO tired of generating pseudorands...:-) How about a free running counter, latched by the output from a geiger counter instead of the 1pps.. (Tough to get numbers at 32 Gnumbers/second though.. but rates in the 10s of kHz would be easy to get with a few microcuries of Am241 and a suitable detector tube/amplifier. ) Or, wide band noise fed to a video A/D and digitized. (you'd have some spectral whitening issues and A/D nonlinearities to deal with) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 8 11:46:21 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 8 Jan 2004 08:46:21 -0800 Subject: [Beowulf] ...Re: Benchmark results References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> Message-ID: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Josip Loncaric" To: "Jim Lux" Cc: Sent: Thursday, January 08, 2004 7:44 AM Subject: Re: [Beowulf] ...Re: Benchmark results > Jim Lux wrote: > > > > At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: > > > >> Despite various software/hardware network > >> layers and longer distances involved, it should be possible to get > >> microsecond synchronization through the network by gradually tweaking > >> local clock offset and drift... > > > > > > Most of the more recent implementations of NTP clients can do this sort > > of thing, sending messages both ways across the link to measure the > > latency (and latency distribution) > > I love NTP, but in its current state it does not deliver microsecond > level synchronization. > > I just checked my old cluster (which runs NTP) and the compute nodes are > off by 500-25000 microseconds. Actually, most are within 1000 > microseconds, but about 20% are off by up to 25000 microseconds. This > happens over Fast Ethernet where ntpdate usually reports timing > dispersion of 0-20 microseconds (based on 7 ping-pong measurements). In > other words, network timing is quite precise, but we'd need about 1000 > times better clock synchronization. > > NTP may need to be streamlined and tuned for clusters... Longer integration times in the smoothing algorithm? The limiting thing might be the instability of the clock oscillator on the motherboard. They're not necessarily designed for low phase noise, and may have terrible short term statistics. The other thing that will bite you with the NTP approach is the asynchronous nature of the Ethernet interface against the CPU. The Ethernet card has it's own clock, it's own logic, etc., so the time from "receving a message" over the wire to the CPU knowing about it could vary quite widely, and the statistics might be non-stationary, which would kill any sort of statistical averaging scheme. You're probably right.. NTP, using just ethernet, will probably never get to sub microsecond without external help, or cleverly designed specialized ethernet hardware (use the bit clock in the PHY layer as the time transfer medium) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 11:34:26 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 11:34:26 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFD7AEA.4060709@lanl.gov> Message-ID: On Thu, 8 Jan 2004, Josip Loncaric wrote: > Jim Lux wrote: > > > > At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: > > > >> Despite various software/hardware network > >> layers and longer distances involved, it should be possible to get > >> microsecond synchronization through the network by gradually tweaking > >> local clock offset and drift... > > > > > > Most of the more recent implementations of NTP clients can do this sort > > of thing, sending messages both ways across the link to measure the > > latency (and latency distribution) > > I love NTP, but in its current state it does not deliver microsecond > level synchronization. > > I just checked my old cluster (which runs NTP) and the compute nodes are > off by 500-25000 microseconds. Actually, most are within 1000 > microseconds, but about 20% are off by up to 25000 microseconds. This > happens over Fast Ethernet where ntpdate usually reports timing > dispersion of 0-20 microseconds (based on 7 ping-pong measurements). In > other words, network timing is quite precise, but we'd need about 1000 > times better clock synchronization. > > NTP may need to be streamlined and tuned for clusters... I'm not sure that the problem is NTP itself -- from what I recall reading in its documentation it is CAPABLE of achieving 1 usec synchronization, over time. However, in order to do that it needs better tools than the current gettimeofday (and other time calls) which seem to deliver no better than perhaps millisecond resolution. It also very likely needs to be done not by a userspace daemon but by the kernel, as only the kernel can decide to do something in a true "single user mode" and for specified brief periods pretty much ignore interrupts and pending tasks and internal sources of delay and noise. If one used the internal hardware cycle counter as the timer (with sub-nanosecond hardware resolution) and worked solely on iteratively refining a single clock set/sync variable (the offset of the CPU cycle counter relative to the epoch, basically) then one could IN PRINCIPLE get below a microsecond even with fast ethernet with enough handshaking. It might take a few tens of thousands of packets per node on an otherwise quiet network, but eventually both ends would agree on packet timings, reject exchanges that fell outside the empirically determined minimum/optimum packet timing for the particular hardware, and settle down to let the central limit theorem and enough exchanges reduce the error bars to the desired level of precision. That's why I think it is a reasonable project for a bright undergraduate. It doesn't seem to me that it would be that difficult, at least not at the cluster/LAN/one-hop level where the networking hardware can be at least moderately deterministic. ntp is really a WAN tool and is robust and portable at the expense of potentially achievable precision. rgb > > Sincerely, > Josip > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Jan 8 12:08:36 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Thu, 08 Jan 2004 12:08:36 -0500 Subject: [Beowulf] tracking down NaN's in an mpi job Message-ID: <1073581716.5027.83.camel@protein.scalableinformatics.com> Hi folks: I am trying to help a customer track down the spontaneous appearance of NaN's in a StarCD job. When submitted in parallel (using mpich 1.2.4 as supplied by Adapco) from their head node on their cluster, they get instability in later iterations, and eventually NaN's start cropping up. When run on one CPU, it appears to be stable (on the head node and compute nodes). They are using the p4 mechanism with rsh. I don't know if others have seen anything like this. Is it possible that messages are being missed somehow in the parallel run? The head node and compute nodes are running two different versions of the Linux kernel (same distribution). Suggestions/hints welcome. Joe -- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 13:13:54 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 11:13:54 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> Message-ID: <3FFD9DE2.10303@lanl.gov> Jim Lux wrote: > From: "Josip Loncaric" >> >>NTP may need to be streamlined and tuned for clusters... > > Longer integration times in the smoothing algorithm? Better yet, extended Kalman filtering (given my control theory background). BTW, there was a highly relevant paper 9 years ago: "On Efficiently Implementing Global Time for Performance Evaluation on Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of Parallel and Distributed Computing, No. 28, pp. 84-93 (1995). Their experimental analysis of the relationship between the system clock and the reference clock over a 50-minute period showed a good fit to a linear model, within about 50-100 microseconds, with slow 16-minute periodicity apparently related to thermal swings in the computer room (A/C cycling). These measurements were done using their T-Node "Meganode" system (128 T800 transputers) on the Transputer network, older than 1995 vintage, probably with clock frequency of about 3.8 MHz. Current networks are much faster, although clock drift behavior may still be similar. Presumably, clock drift coefficients could be correlated to motherboard temperature measurements. Over time, one should be able to construct a linearized model of clock dynamics, linking the local clock drift, motherboard temperature, and possibly even power supply voltages. This should overcome the main sources of timing uncertainty. Sincerely, Josip P.S. Internally, clusters often have a single broadcast domain. Even if the head node broadcast an augmented NTP packet every second, this would still not impact the network much -- but may generate enough timing data so that each node's filter would have at least some good measurements to work with... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 13:25:53 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 10:25:53 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFD9DE2.10303@lanl.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> Message-ID: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: > >>From: "Josip Loncaric" >>> >>>NTP may need to be streamlined and tuned for clusters... >>Longer integration times in the smoothing algorithm? > >Better yet, extended Kalman filtering (given my control theory background). > >BTW, there was a highly relevant paper 9 years ago: > >"On Efficiently Implementing Global Time for Performance Evaluation on >Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of Parallel >and Distributed Computing, No. 28, pp. 84-93 (1995). I'll get a copy and take a look. >Their experimental analysis of the relationship between the system clock >and the reference clock over a 50-minute period showed a good fit to a >linear model, within about 50-100 microseconds, with slow 16-minute >periodicity apparently related to thermal swings in the computer room (A/C >cycling). > >These measurements were done using their T-Node "Meganode" system (128 >T800 transputers) on the Transputer network, older than 1995 vintage, >probably with clock frequency of about 3.8 MHz. Current networks are much >faster, although clock drift behavior may still be similar. What was the measurement/integration interval for their frequency measurements? Robert (and I, and others) are interested in stability over fairly short time scales (milliseconds?). >Presumably, clock drift coefficients could be correlated to motherboard >temperature measurements. Over time, one should be able to construct a >linearized model of clock dynamics, linking the local clock drift, >motherboard temperature, and possibly even power supply voltages. This >should overcome the main sources of timing uncertainty. I wouldn't be too sure that you could build a suitable model. One could certainly make a very accurate predictor in the long term sense: it would very accurately model things like how many cycles of the processor clock there are in every 10 or 60 second interval. But at shorter time scales, the clocks MIGHT be pretty bad, with random jumps and stuff. I don't know, but since it's relevant to my current research, I'm going to find out, starting today.. The run of the mill clock oscillators you see on a motherboard do not have very good short term characteristics. For instance, you can't use them for timing digital communications interfaces like SONET or OC-3 or ATM, where they use much higher quality oscillators, with better jitter specs. A typical processor isn't going to much care if there's, say, 10% jitter (which is really hideously bad) in the clock supplied to it , providing the clock multiplier loops on the chip can stay locked. Likewise, since the processor is, generally, synchronously clocked logic (except for those of you running PDP-8 based clusters!), instantaneous variation in clock rates don't do much, other than set a bound on timing margins (more jitter means you need more timing margin in the propagation paths on chip). If you plot the frequency versus time for an oscillator, a "nice" oscillator will show a fairly smooth monotonic drift with aging (usually they drift up with time) and smooth variations with temperature and things like gravitational force or acceleration(if you change the orientation of the oscillator). A "bad" oscillator will have spontaneous jumps up and down. We're talking parts per million here, though, just to keep things in perspective. >Sincerely, >Josip > > >P.S. Internally, clusters often have a single broadcast domain. Even if >the head node broadcast an augmented NTP packet every second, this would >still not impact the network much -- but may generate enough timing data >so that each node's filter would have at least some good measurements to >work with... James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 14:05:57 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 14:05:57 -0500 (EST) Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> Message-ID: On Thu, 8 Jan 2004, Jim Lux wrote: > But, does the CPU cycle counter run at constant speed? Or does it ratchet > up and down with power management, for instance. This was a curse for us in > the DSP world, where we did alter CPU clock rate(as in stopping the > processor) to reduce power during "idle time", which is why we went to > using a counter that happened to be in a peripheral chip. One would have to check the Intel or AMD specs to be sure, but I recall that it is a hardware counter of the stabilized baseline clock crystal for the system and is hence very, very stable and quite accurate. However, I admit that I'm not certain -- clock drift in gettimeofday on some systems USED to be a really annoying problem and a load dependent one at that. Also, I have no idea what happens to it on a CPU that does cycle throttling when it gets hot (e.g. P4). The problem with putting it in a peripheral chip is that it inevitably increases the overhead associated with reading it, probably to hundreds of nanoseconds or worse. Probably adequate to achieve usec resolution, but not adequate to get nsec resolution. > > The second approach is hardware, which seems perfectly "legal" but which > > requires an additional investment and which pushes one a bit away from > > the "commodity" ideal and into the "hardware hacker" ideal which is not > > everybody's cup of tea. GPS is (as was discussed) a good way to get a > > very accurate time base, with the caveat that GPS signals are very weak > > (my handheld GPS won't work indoors, or even inside a car, let alone in > > a basement server room underneat a few hundred tons of steel and > > concrete). > > I think that one can probably do it with no mods to the motherboard or > outside hardware. For instance, the NTP servers can bring in the 1pps on one > of the serial port lines. And, "legacy free" notwithstanding, most > motherboards still have a parallel printer port. Perhaps something along > the lines of PAPERS? Again, one has to navigate the device driver for the peripheral and likely hack the kernel or suffer random delays. If you're going this route one might as well just use the network and work harder, as either way you'll need the same techniques to overcome the various sources of random latency and refine an average to the desired resolution. > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > > have built in GPS, > > In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest > widget, a GPS receiver that's about 10x15 mm, and can do what's called > "assisted" fixes, where the cell site does a lot of the work in terms of > keeping track of satellite ephemerides, and so forth. (Greatly reducing the > processing, and power, in the phone). > It's all part of the E911 mandate which requires cellphone providers to give > locations within a few meters when someone makes a 911 call on their > cellphone. (Such position information is also quite commercially valuable.. > they could send you customized SMS text messages: "Looking for Starbucks? 2 > blocks ahead on the left?") I was pretty surprised, but then my Christmas present in 2002 was a cell-phone sized GPS that cost about $100, complete with nifty screen, built in maps, leedle controls -- and it is still mostly battery, as I take it the GPS signals from the various satellites are quite weak and require a lot of amplification. > and for $45 or so I bought a GPS-sync'd clock from > > Wal-Mart (which won't work inside the house because GPS signals are so > > weak, sigh). GPS thus seems like something that would actually be > > fairly tricky to get working inside a lot of server room environments > > unless an external antenna were used and piped down to the device, which > > would add to the expense and hassle and be moderately inefficient on a > > per-system basis. > > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least > most of the widely advertised "atomic clocks" do that...) Hmmm, checking it out on the web it does indeed. I'm probably getting my wife's new cell phone and the various new PDA's scrambled with the thermometer/clock (Acurite Wireless Weather and Atomic Clock Station, $22.44 at Wal-Mart:-). Silly me -- I assume(d) that it just meant that somebody in China had cloned a single-chip GPS receiver and was selling it for $0.50 to the clock's maker. I didn't realize that the cell phone devices were also indirectly being run through the cell towers instead of directly from satellites, although it makes sense since cellular signals work indoors but GPS's don't. At $22.44 I'm tempted to buy one and hack it up and see if one can leech the clock register contents out in a machine readable form. Might not get microsecond resolution that way, but it would be good enough for an NTP server for my house and beat the hell out of the $400 cards sold for that purpose over the counter...;-). I've been TEMPTED to see if my $100 GPS will do the same thing -- somewhere in there it must be reading the time base(s) to do the location computation. > You don't need to have GPS on each computer (in fact, that would be an > egregious waste of money), just the sync signal piped to them all. A single Egregious at current prices, yes. > wire, pulsing at 1pps, is sufficient. Put the GPS somewhere convenient. > You can send the 1pps down cat5 wire, or on a fiber optic link, or almost > anything. > > Even better (if you're into this kind of thing), is a GPS timing receiver > like the surplus Z3801, which use GPS to "discipline" a very high quality > quartz oscillator, which then generates a very clean and precise 10MHz, as > well as the 1 pps. You can also generate precision IRIG (or AES/EBU or > SMPTE) time code, and distribute that. And read it with a $400 card? Alas, I'm not into this kind of thing at that level. What I REALLY want is an over the counter device that sells for $22.95, attaches to a USB port or a PCI slot, works in a server room deep in the bowels of the world, and magically keeps the best clock available to linux on a given architecture accurate to a few nsec. Not too much, really. Couple of years, mass market GPS, prices drop, or somebody will figure out a way to use cell phone towers to achieve the same thing. > > I have my own reasons for liking the design you indicate below, and > > would like it very much if two PCs separated by tens of kilometers and > > not even ON a common network could synchronize an internal clock to less > > than a microsecond, better yet to less than 100 nanoseconds. > > Such things are available off the shelf now, just not as a commodity. I > would think, though, that a Z3801 type receiver ($250 surplus) hooked to > each PC can do it to microsecond precision. > > Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and > reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all > these sorts of things, with external trigger inputs and so forth. Ya, for prices that seem to exceed $500 a pop by a goodly amount. Even a card to read a time reference signal from a provided standard interface is $400 from e.g. masterclock, even a WWVB card is $400 from e.g. beagle. This is nuts (especially for the WWVB card). If Wal-Mart can sell a WWVB clock thermometer with remote wireless thermometer included for less than $25 full retail, and a PCI card base costs a few dollars, I can't believe that one couldn't build and sell a WWVB PCI time base card (and hell, throw temperature in too with a remote sensor probe that plugs into the back or onto the card internally) for considerably less than $100. I suppose there is a limited market but that in part is CREATED by the high prices. I might buy a card for my home network for $100. I might buy a card for every system in my home network for $25. Prices of $1000 (ballpark of a lot of the GPS-based offerings) might be ok for a departmental LAN or an institution, but they are not by any means "consumer" level offerings. Odd at a time that radio clocks are over the counter available and even becoming "common" at less than $50 full retail. > > to happen, the whole thing REALLY needs to move onto the motherboard and > > be tightly integrated with the system design, although a PCI card or USB > > device both are alternatives that would likely "work", especially if one > > wrote kernel drivers for them that could do the entire job of reading > > and setting the internal cycle-counter-to-absolute-time variable as an > > atomic operation AND actually correct the result for the time it took to > > do the set. If one could build a hardware device that kept > > nanosecond-accurate time, it should be possible to write a gettimeofday > > or inline assembler fragment that is accurate to within a few > > nanoseconds (the minimal overhead associated with reading the counter > > and saving the result as a timestamp for later conversion). > > This might already exist on the motherboard.. one just needs to find > something that latches the counter deterministically. I'd look at the game > port interface. Wouldn't one need something that could manage pulses with sub-microsecond rise times and trigger reliably? I'd expect a game interface to be way too slow. I'd even worry about RS232. As I said, a lot of the $300-500 cards sold appear to be nothing but an interface that can reliably manage a time pulse accurate to within a usec or so. > > Personally I can think of all sorts of benefits to having each and every > > PC clock in the universe set and locked to real, absolute time within a > > few nanoseconds, and similarly located in space to within a few meters > > (those same nanoseconds times c). The little radiotelescope project > > I've got a student (still, sigh) working on being one of them, but far > > from the only one. I'd really like to be able to sit at my PC and have > > it go "happy new year" at PRECISELY the new year, for example. > > This is what LOFAR and the SKA are all about... I think the Allen Telescope > Array up in Hat Creek is also using GPS disciplined clocks to sync > everything. nsec syncronization on a global scale has lots of benefits, some of which could enable projects that would dwarf SETI at home. The heck with turning every cellular tower in the country into a giant radiotelescope array -- turn every suitably equipped PC in the country into a receiver in a really enormous radiotelescope array. Now there is a project for a GRID! Frankly, it would probably work better for SETI as well. Maybe even tabletop relativity experiments as well. > > While we are at it sailing around in the blue sky, can we also put an > > entropy generator on the add-on card? Something that can generate > > random bits (in parallel) at oh, 32 GHz, buffer them internally, and > > deliver the entire buffer to memory via DMA on a single interrupt? My > > systems get SO tired of generating pseudorands...:-) > > How about a free running counter, latched by the output from a geiger > counter instead of the 1pps.. > (Tough to get numbers at 32 Gnumbers/second though.. but rates in the 10s of > kHz would be easy to get with a few microcuries of Am241 and a suitable > detector tube/amplifier. ) > Or, wide band noise fed to a video A/D and digitized. (you'd have some > spectral whitening issues and A/D nonlinearities to deal with) Oh, there are people looking and and even implementing a lot of these sorts of things. Unfortunately, a lot of "random" physical sources turn out not be random at all, just to have a relatively short autocorrelation function and to have distributions that are not uniform as well. A lot of the noise-based generators fall into this category, but quantum generators can as well (one has to look at second and higher order correlation functions, for example -- even resonance fluorescence, a textbook example of a "random quantum process" turns out to be photon antibunched, and there is also the venerable work of Hanbury Brown and Twiss. It turns out not to be horribly easy to build a hardware generator that passes diehard or the NIST suite, let alone one that works fast. Lots of effort is actually expended here in corporate america as random numbers are a key element of cryptography, and some devices are on the market. They're just expensive and slow and in some cases not terribly high quality -- "unpredictable" but not necessarily "random". Unpredictable can be made random, sort of, sometimes -- but only via transformations that slow the generator down. Right now I don't know of any process or device that can deliver uniform deviates as fast or faster (or of any better quality) than the faster/better pseudorandom generators, although there may be some under development or out there as the world keeps changing. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 8 15:08:07 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 8 Jan 2004 12:08:07 -0800 Subject: [Beowulf] tracking down NaN's in an mpi job In-Reply-To: <1073581716.5027.83.camel@protein.scalableinformatics.com> References: <1073581716.5027.83.camel@protein.scalableinformatics.com> Message-ID: <20040108200807.GB3000@greglaptop.internal.keyresearch.com> On Thu, Jan 08, 2004 at 12:08:36PM -0500, Joe Landman wrote: > I am trying to help a customer track down the spontaneous appearance > of NaN's in a StarCD job. The usual fix is to compile with floating point options set such that the application faults instead of generating NaNs. This sounds like a turnkey commercial app, though, so why not just file a bug report? -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 14:38:03 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 11:38:03 -0800 Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> Message-ID: <5.2.0.9.2.20040108111528.031bde38@mailhost4.jpl.nasa.gov> At 02:05 PM 1/8/2004 -0500, Robert G. Brown wrote: >On Thu, 8 Jan 2004, Jim Lux wrote: > > > > > I think that one can probably do it with no mods to the motherboard or > > outside hardware. For instance, the NTP servers can bring in the 1pps > on one > > of the serial port lines. And, "legacy free" notwithstanding, most > > motherboards still have a parallel printer port. Perhaps something along > > the lines of PAPERS? > >Again, one has to navigate the device driver for the peripheral and >likely hack the kernel or suffer random delays. If you're going this >route one might as well just use the network and work harder, as either >way you'll need the same techniques to overcome the various sources of >random latency and refine an average to the desired resolution. hacking the kernel is not necessarily evil. I suppose it's a tradeoff: hack the kernel and use commodity hardware; use stock kernel and use hacked hardware. > > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > > > have built in GPS, > > > > In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest > > widget, a GPS receiver that's about 10x15 mm, and can do what's called >I was pretty surprised, but then my Christmas present in 2002 was a >cell-phone sized GPS that cost about $100, complete with nifty screen, >built in maps, leedle controls -- and it is still mostly battery, as I >take it the GPS signals from the various satellites are quite weak and >require a lot of amplification. yep.. as a standalone device, there are lots of $50 (retail) GPS units available in one form or another. The basic module, with antenna, runs about $30-$50 in single quantities. But, there aren't a whole lot of phones with GPS in them yet. No real customer demand yet (although, I can certainly see a "On-Star" type service being attractive to people.. I'm lost, how do I get to the freeway) > > > > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at > least > > most of the widely advertised "atomic clocks" do that...) > >Hmmm, checking it out on the web it does indeed. I'm probably getting >my wife's new cell phone and the various new PDA's scrambled with the >thermometer/clock (Acurite Wireless Weather and Atomic Clock Station, >$22.44 at Wal-Mart:-). Silly me -- I assume(d) that it just meant that >somebody in China had cloned a single-chip GPS receiver and was selling >it for $0.50 to the clock's maker. I didn't realize that the cell phone >devices were also indirectly being run through the cell towers instead >of directly from satellites, although it makes sense since cellular >signals work indoors but GPS's don't. Sort of a combination of the two.. Rather than have to do the entire fix in the phone, they can just receive the GPS signals and make some raw measurements of code and carrier phase, then send it to the cell site, where it can get turned into position. >At $22.44 I'm tempted to buy one and hack it up and see if one can leech >the clock register contents out in a machine readable form. Might not >get microsecond resolution that way, but it would be good enough for an >NTP server for my house and beat the hell out of the $400 cards sold >for that purpose over the counter...;-). I've been TEMPTED to see if my >$100 GPS will do the same thing -- somewhere in there it must be reading >the time base(s) to do the location computation. Check out the documentation for the various Time Servers software.. Almost all GPS receivers have a 1pps output, and many synchronize the NMEA serial output string to the 1pps (that is, you can use the start bit of the first character in the sentence for sync). The highly integrated GPS chips do NOT make it easy to read the internal registers, for a variety of reasons. http://www.synergy-gps.com/ is a big dealer of Motorola Oncore modules and has a lot of documentation. Googling for the ST Microdevices ST20GP6 will find a data sheet for a typical GPS receiver/processor chip. (several years old design, though.. ) Search for "Totally Accurate Clock" and you'll find stuff about using WWVB and/or GPS for this sort of thing. > > You don't need to have GPS on each computer (in fact, that would be an > > egregious waste of money), just the sync signal piped to them all. A > single > >Egregious at current prices, yes. > > > wire, pulsing at 1pps, is sufficient. Put the GPS somewhere convenient. > > You can send the 1pps down cat5 wire, or on a fiber optic link, or almost > > anything. > > > > Even better (if you're into this kind of thing), is a GPS timing receiver > > like the surplus Z3801, which use GPS to "discipline" a very high quality > > quartz oscillator, which then generates a very clean and precise 10MHz, as > > well as the 1 pps. You can also generate precision IRIG (or AES/EBU or > > SMPTE) time code, and distribute that. > >And read it with a $400 card? Read it with software and a single bit input. If you can latch that register on the edges, you're all set, because the time code itself is quite slow, and because time marches forward in a fairly regular fashion, you can take a lot of time decoding the time code sequence (you could do 1 bit per second, and after a few minutes, have the complete sequence locked up). You can certainly decode most audio timecode signals (i.e. IRIG) (but not synchronize to microseconds) with a sound card. >Alas, I'm not into this kind of thing at that level. What I REALLY want >is an over the counter device that sells for $22.95, attaches to a USB >port or a PCI slot, works in a server room deep in the bowels of the >world, and magically keeps the best clock available to linux on a given >architecture accurate to a few nsec. Not too much, really. > >Couple of years, mass market GPS, prices drop, or somebody will figure >out a way to use cell phone towers to achieve the same thing. You could do it today for <$100, and a free serial port, and enough wire to get the GPS receiver to a view of the sky. > > Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and > > reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all > > these sorts of things, with external trigger inputs and so forth. > >Ya, for prices that seem to exceed $500 a pop by a goodly amount. Even >a card to read a time reference signal from a provided standard >interface is $400 from e.g. masterclock, even a WWVB card is $400 from >e.g. beagle. This is nuts (especially for the WWVB card). If Wal-Mart >can sell a WWVB clock thermometer with remote wireless thermometer >included for less than $25 full retail, and a PCI card base costs a few >dollars, I can't believe that one couldn't build and sell a WWVB PCI >time base card (and hell, throw temperature in too with a remote sensor >probe that plugs into the back or onto the card internally) for >considerably less than $100. You could, if you could convince Wal-Mart that they could sell millions of them. >Odd at a time that radio clocks are over the counter available and even >becoming "common" at less than $50 full retail. Wireless LAN interfaces used to be $200, and are now <$30... it might happen. > > > > This might already exist on the motherboard.. one just needs to find > > something that latches the counter deterministically. I'd look at the game > > port interface. > >Wouldn't one need something that could manage pulses with >sub-microsecond rise times and trigger reliably? I'd expect a game >interface to be way too slow. The original game interface latched a counter driven off the processor clock. Yes, overkill to measure a potentiometer position to 1 part in 100, but that's the hardware available. The question would be how slavishly have the chipset designers held to the original design concept. > I'd even worry about RS232. As I said, a >lot of the $300-500 cards sold appear to be nothing but an interface >that can reliably manage a time pulse accurate to within a usec or so. For RS232, it's the interrupt latency on the RS232 status line that determines the performance, which can be quite impressive. Essentially a kernel hack, but someone's already done it for the NTB clock drivers. Sub microsecond is fairly easy. All the ISR has to do is latch the processor time counter and set a flag, so it can run unmasked, etc. Likewise the IRQ7 line coming in from the printer port can be used. Although it's low priority, you don't need perfect accuracy.. you just throw out the interrupts that are obviously late, having been overridden by a higher priority interrupt. For those of you working with Win NT, take a look at AboutTime, which has a kernel driver doing just this sort of thing. I don't know how deterministic the interrupt latency is on the modern x-ium and x-lon processors though. It kind of depends on if the designer wanted to flush the pipeline before servicing the interrupt, or if they stall it. It might be a bit tricky to even find out from the x-ium documentation exactly what it does, since for most users, the few nanoseconds difference doesn't make any difference. There's probably also some latching pipeline logic running at fairly low clock rates between the IRQ line and the actually processor. Theinterrupt controller could introduce some random latency depending on the relative phases of the clock latching the interrupt request and the processor seeing the IRQ asserted on the chip. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dag at sonsorol.org Thu Jan 8 14:12:20 2004 From: dag at sonsorol.org (Chris Dagdigian) Date: Thu, 08 Jan 2004 14:12:20 -0500 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <3FFDAB94.2070805@sonsorol.org> {Forwarded to this list on behalf of a friend with some email troubles...} > I am in the process of trying to get a stopgap perfomance monitoring > system going on a 64 CPU Linux cluster with LSF. Ultimately, I hope to > use PCP for data collection, but since nobody seems to be doing this yet, > we are going to be rolling our own solution. To meet some of their needs, > management has asked for an interim solution that gives them a web page > with aggregate usage statistics and such. > > Unfortunately, ganglia is a non-starter because our networking group can't > enable multicast for the private network the cluster lives on (it would > break almost everything else). > > Does anyone have any suggestions for an alternative that can be quickly > implemented, doesn't rely on multicast, and that can generate graphs of aggregate > statistics on the fly? Regards, Chris BioTeam.net -- Chris Dagdigian, Independent life science IT & research computing consulting Office: 617-666-6454, Mobile: 617-877-5498, Fax: 425-699-0193 Work: http://BioTeam.net PGP KeyID: 83D4310E Yahoo IM: craffi _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Thu Jan 8 15:03:00 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Thu, 8 Jan 2004 12:03:00 -0800 (PST) Subject: [Beowulf] MPI noob prog, needs help Message-ID: <20040108200300.60882.qmail@web12705.mail.yahoo.com> hello all, Can I implement MPI usage in common C programs directly by MPI_Init and MPI_Finalize and run it on slave nodes. Also is MPI:Complete Reference available in pdf format. I stay in India and its not printed locally here. Online prices are too high and the shipping is too much , total cost ends up at around 60 $. will be glad if someone can optimize the following C program to run on multiple nodes in LAM/MPI. Its for a project. thanks in advance, toufeeq ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus -------------- next part -------------- A non-text attachment was scrubbed... Name: prime_sieve.c Type: application/octet-stream Size: 1344 bytes Desc: prime_sieve.c URL: From josip at lanl.gov Thu Jan 8 16:43:30 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 14:43:30 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <3FFDCF02.3000501@lanl.gov> Jim Lux wrote: > At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote: > >> "On Efficiently Implementing Global Time for Performance Evaluation on >> Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of >> Parallel and Distributed Computing, No. 28, pp. 84-93 (1995). > > What was the measurement/integration interval for their frequency > measurements? Robert (and I, and others) are interested in stability > over fairly short time scales (milliseconds?). Alas, their sampling interval was about 6 seconds. They had 500 message exchanges in 50 minutes. However, drift due to independent random events should grow like sqrt(time), so in absolute terms it should be lower at smaller time scales (even though it may be larger in relative terms). In other words, if you had 6 microseconds of uncompensated drift in 6 seconds (1ppm), and you assume that the uncompensated part was due to a random walk process, you'd see about 0.06 microseconds of drift in 0.6 milliseconds. This is not bad in absolute terms, even though in relative terms it looks bad (100ppm). >> Presumably, clock drift coefficients could be correlated to >> motherboard temperature measurements. Over time, one should be able >> to construct a linearized model of clock dynamics, linking the local >> clock drift, motherboard temperature, and possibly even power supply >> voltages. This should overcome the main sources of timing uncertainty. > > > I wouldn't be too sure that you could build a suitable model. One could > certainly make a very accurate predictor in the long term sense: it > would very accurately model things like how many cycles of the processor > clock there are in every 10 or 60 second interval. But at shorter time > scales, the clocks MIGHT be pretty bad, with random jumps and stuff. I > don't know, but since it's relevant to my current research, I'm going to > find out, starting today.. Hmmm... I'm no expert, but I believe that typical clock oscillators become quite good when temperature compensation is used. I am not sure if there is any significant supply voltage dependence, particularly since there may be some voltage stabilization in the clock circuit. I assume we're talking about a crystal oscillator plus clock multipliers; there may be significant phase jitter but ultimately one cares only about cycle counting. These days, there can be over 3000 cycles in a microsecond. If good quality synchronization pulses can be provided every second, then microsecond requirement means that one can afford 1 ppm of uncompensated clock drift. Over a year, the uncompensated drift could be 31.5 seconds. Clock in my car does almost as well, despite being subjected to 60 deg. C temperature swings and variable accelerations. Perhaps I'm overly optimistic, but writing a kernel module to achieve cluster-wide microsecond-level clock synchronization just might be feasible... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amacater at galactic.demon.co.uk Thu Jan 8 17:53:52 2004 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Thu, 8 Jan 2004 22:53:52 +0000 Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> Message-ID: <20040108225352.GA29682@galactic.demon.co.uk> On Thu, Jan 08, 2004 at 02:05:57PM -0500, Robert G. Brown wrote: > On Thu, 8 Jan 2004, Jim Lux wrote: > > > > The second approach is hardware, which seems perfectly "legal" but which > > > requires an additional investment and which pushes one a bit away from > > > the "commodity" ideal and into the "hardware hacker" ideal which is not > > > everybody's cup of tea. GPS is (as was discussed) a good way to get a > > > very accurate time base, with the caveat that GPS signals are very weak > > > (my handheld GPS won't work indoors, or even inside a car, let alone in > > > a basement server room underneat a few hundred tons of steel and > > > concrete). > > > > I think that one can probably do it with no mods to the motherboard or > > outside hardware. For instance, the NTP servers can bring in the 1pps on one > > of the serial port lines. And, "legacy free" notwithstanding, most > > motherboards still have a parallel printer port. Perhaps something along > > the lines of PAPERS? > > Again, one has to navigate the device driver for the peripheral and > likely hack the kernel or suffer random delays. If you're going this > route one might as well just use the network and work harder, as either > way you'll need the same techniques to overcome the various sources of > random latency and refine an average to the desired resolution. Maplin electronics did a kit - cheaply - to run off the time code from 60kHz clocks and put it into the serial/parallel port. Know any radio hams - a lot are taking a temperature controlled crystal oscillator and phase locking to WWWV or equivalent. > > > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > > > have built in GPS, > > > > In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest > > widget, a GPS receiver that's about 10x15 mm, and can do what's called > > "assisted" fixes, where the cell site does a lot of the work in terms of > > keeping track of satellite ephemerides, and so forth. (Greatly reducing the > > processing, and power, in the phone). > > It's all part of the E911 mandate which requires cellphone providers to give > > locations within a few meters when someone makes a 911 call on their > > cellphone. (Such position information is also quite commercially valuable.. > > they could send you customized SMS text messages: "Looking for Starbucks? 2 > > blocks ahead on the left?") > > I was pretty surprised, but then my Christmas present in 2002 was a > cell-phone sized GPS that cost about $100, complete with nifty screen, > built in maps, leedle controls -- and it is still mostly battery, as I > take it the GPS signals from the various satellites are quite weak and > require a lot of amplification. Correlation of the satellite constellation is what takes the processing power. > > > and for $45 or so I bought a GPS-sync'd clock from > > > Wal-Mart (which won't work inside the house because GPS signals are so > > > weak, sigh). GPS thus seems like something that would actually be > > > fairly tricky to get working inside a lot of server room environments > > > unless an external antenna were used and piped down to the device, which > > > would add to the expense and hassle and be moderately inefficient on a > > > per-system basis. > > > > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least > > most of the widely advertised "atomic clocks" do that...) At 60kHz, the clocks you get in the States have warnings on - not necessarily reliable on the Eastern seaboard because of co-channel interference from Rugby, UK. Here in Europe, we also have/had DCF77 on approx 77kHz from Germany. > > nsec syncronization on a global scale has lots of benefits, some of > which could enable projects that would dwarf SETI at home. The heck with > turning every cellular tower in the country into a giant radiotelescope > array -- turn every suitably equipped PC in the country into a receiver > in a really enormous radiotelescope array. Now there is a project for a > GRID! Haven't you heard - the Govt. is using them to track individuals by their individual radar fingerprint. They can watch me wherever I walk and they're using the system to mess with my mind ... :) > > Frankly, it would probably work better for SETI as well. > > Maybe even tabletop relativity experiments as well. > > > > > While we are at it sailing around in the blue sky, can we also put an > > > entropy generator on the add-on card? Something that can generate > > > random bits (in parallel) at oh, 32 GHz, buffer them internally, and > > > deliver the entire buffer to memory via DMA on a single interrupt? My > > > systems get SO tired of generating pseudorands...:-) > > > Unfortunately, a lot of "random" physical sources turn > out not be random at all, just to have a relatively short > autocorrelation function and to have distributions that are not uniform > as well. A lot of the noise-based generators fall into this category, > but quantum generators can as well (one has to look at second and higher > order correlation functions, for example -- even resonance fluorescence, > a textbook example of a "random quantum process" turns out to be photon > antibunched, and there is also the venerable work of Hanbury Brown and > Twiss. It turns out not to be horribly easy to build a hardware > generator that passes diehard or the NIST suite, let alone one that > works fast. > The Knuth random number generator that converged to a value - the first example in Seminumerical Algorithms springs to mind :) > Lots of effort is actually expended here in corporate america as random > numbers are a key element of cryptography, and some devices are on the > market. They're just expensive and slow and in some cases not terribly > high quality -- "unpredictable" but not necessarily "random". > Unpredictable can be made random, sort of, sometimes -- but only via > transformations that slow the generator down. Right now I don't know of > any process or device that can deliver uniform deviates as fast or > faster (or of any better quality) than the faster/better pseudorandom > generators, although there may be some under development or out there as > the world keeps changing. > > rgb > > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From atp at piskorski.com Thu Jan 8 16:48:35 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Thu, 8 Jan 2004 16:48:35 -0500 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <200401081951.i08JpDh12979@NewBlue.scyld.com> References: <200401081951.i08JpDh12979@NewBlue.scyld.com> Message-ID: <20040108214835.GA34465@piskorski.com> On Thu, Jan 08, 2004 at 11:34:26AM -0500, Robert G. Brown wrote: > On Thu, 8 Jan 2004, Josip Loncaric wrote: > > I love NTP, but in its current state it does not deliver microsecond > > level synchronization. > > NTP may need to be streamlined and tuned for clusters... > > I'm not sure that the problem is NTP itself -- from what I recall > reading in its documentation it is CAPABLE of achieving 1 usec > synchronization, over time. However, in order to do that it needs > better tools than the current gettimeofday (and other time calls) which > seem to deliver no better than perhaps millisecond resolution. It also > very likely needs to be done not by a userspace daemon but by the > kernel, as only the kernel can decide to do something in a true "single Would DJB's clockspeed, taiclock, etc. stuff be any better than ntp for that? His website mentions using CPU hardware tick or nanosecond counters, but doesn't define "precise" when it says that taiclock is "... suitable for precise time synchronization over a local area network ...": http://cr.yp.to/clockspeed.html http://cr.yp.to/libtai.html -- Andrew Piskorski http://www.piskorski.com/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 8 18:20:00 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 8 Jan 2004 15:20:00 -0800 Subject: [Beowulf] MPI noob prog, needs help In-Reply-To: <20040108200300.60882.qmail@web12705.mail.yahoo.com> References: <20040108200300.60882.qmail@web12705.mail.yahoo.com> Message-ID: <20040108232000.GA5350@greglaptop.internal.keyresearch.com> On Thu, Jan 08, 2004 at 12:03:00PM -0800, 70uf33q Hu5541n wrote: > Also is MPI:Complete Reference available in pdf > format. It looks like the complete contents are: http://www.netlib.org/utk/papers/mpi-book/mpi-book.html Is this just the Standard? -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 17:37:19 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 15:37:19 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <3FFDDB9F.3050201@lanl.gov> Jim Lux wrote: > > We're talking parts per million here, though, just to keep things in > perspective. I just checked some quartz crystal oscillator specs, and they range all over the place. The cheap ones recommended for Gigabit Ethernet devices claim 100 ppm stability, but good quality ones can be ordered with 0.1 ppm specs. HP application node 200-2 "Fundamentals of Quartz Oscillators" (see http://www.measurement.tm.agilent.com/main.cgi?PSP_NEXT=Redirect&Document:CONTENT_ID=6980) says that temperature effects are the dominant influence (2.5 ppm within 0-50 deg. C). This is followed by aging (<0.3ppm/month), line voltage change (<0.1ppm/10% change), and various other smaller effects. A good crystal may deliver <2.5 ppm frequency variation over a 0 to 50 deg. C range, but the temperature dependence may be quite nonlinear. Temperature compensated crystals can deliver 0.5 ppm stability over the same range. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 18:33:28 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 16:33:28 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <20040108214835.GA34465@piskorski.com> References: <200401081951.i08JpDh12979@NewBlue.scyld.com> <20040108214835.GA34465@piskorski.com> Message-ID: <3FFDE8C8.40700@lanl.gov> Andrew Piskorski wrote: > Would DJB's clockspeed, taiclock, etc. stuff be any better than ntp > for that? Interestingly enough, DJB says that after he applied clockspeed compensation, his machine drifted only 0.21 seconds in three months. This would imply long term clock stability of about 0.027 ppm, but as Jim already pointed out, short term stability usually isn't as good. Using the random walk assumption, DJB's clock could drift 75 microseconds in either direction each second (at random), and still meet these long term stability results... While full NTP may be too much, some NTP-like approach is probably needed... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 19:14:38 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 16:14:38 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFDCF02.3000501@lanl.gov> References: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20040108160818.031eb830@mailhost4.jpl.nasa.gov> At 02:43 PM 1/8/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: > >>At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote: >> >>>"On Efficiently Implementing Global Time for Performance Evaluation on >>>Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of >>>Parallel and Distributed Computing, No. 28, pp. 84-93 (1995). >>What was the measurement/integration interval for their frequency >>measurements? Robert (and I, and others) are interested in stability >>over fairly short time scales (milliseconds?). > >Alas, their sampling interval was about 6 seconds. They had 500 message >exchanges in 50 minutes. > >However, drift due to independent random events should grow like >sqrt(time), so in absolute terms it should be lower at smaller time scales >(even though it may be larger in relative terms). In other words, if you >had 6 microseconds of uncompensated drift in 6 seconds (1ppm), and you >assume that the uncompensated part was due to a random walk process, you'd >see about 0.06 microseconds of drift in 0.6 milliseconds. This is not bad >in absolute terms, even though in relative terms it looks bad (100ppm). Unfortunately, the errors aren't nice additive white gaussian processes... You could have substantial short term variations that average out in the long run. >>>Presumably, clock drift coefficients could be correlated to motherboard >>>temperature measurements. Over time, one should be able to construct a >>>linearized model of clock dynamics, linking the local clock drift, >>>motherboard temperature, and possibly even power supply voltages. This >>>should overcome the main sources of timing uncertainty. >> >>I wouldn't be too sure that you could build a suitable model. One could >>certainly make a very accurate predictor in the long term sense: it would >>very accurately model things like how many cycles of the processor clock >>there are in every 10 or 60 second interval. But at shorter time scales, >>the clocks MIGHT be pretty bad, with random jumps and stuff. I don't >>know, but since it's relevant to my current research, I'm going to find >>out, starting today.. > >Hmmm... I'm no expert, but I believe that typical clock oscillators >become quite good when temperature compensation is used. I am not sure if >there is any significant supply voltage dependence, particularly since >there may be some voltage stabilization in the clock circuit. I assume >we're talking about a crystal oscillator plus clock multipliers; there may >be significant phase jitter but ultimately one cares only about cycle >counting. These days, there can be over 3000 cycles in a microsecond. Yes, they're reasonably stable in terms of frequency, measured over a long time, but over short time spans (tau in the timing literature) they can be much worse. This kind of thing is what separates run of the mill GPS receivers (which have 1pps jitters of around 100ns) from so-called timing grade receivers (which have 1pps jitter of <10-20 ns). >If good quality synchronization pulses can be provided every second, then >microsecond requirement means that one can afford 1 ppm of uncompensated >clock drift. Over a year, the uncompensated drift could be 31.5 >seconds. Clock in my car does almost as well, despite being subjected to >60 deg. C temperature swings and variable accelerations. The 32.768 kHz crystal in your car clock is a good example of an oscillator that's not too hot in the short run (seconds) but is very good in the long run. Of course, they run at really, really low powers, which doesn't help. >Perhaps I'm overly optimistic, but writing a kernel module to achieve >cluster-wide microsecond-level clock synchronization just might be feasible... I am entirely certain that this can be done, and may have already been done. It's such a useful thing to "know, exactly, what time it is".... James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Thu Jan 8 19:12:06 2004 From: shewa at inel.gov (Andrew Shewmaker) Date: Thu, 08 Jan 2004 17:12:06 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> References: <1073436271.15823.30.camel@protein.scalableinformatics.com> <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> Message-ID: <3FFDF1D6.4040309@inel.gov> Jim Lux wrote: > This is along the lines of what that nifty benchmark (the name of which > eludes me right now) that produces a graph of speed vs computation size > does.. You can "see" the effects of cache etc as the problems space gets > bigger. (Edward Tufte would be pleased with the results) Are you referring to HINT? The original Ameslab home page is gone, but it looks like there is a mirror at http://hint.byu.edu An example of the memory regime graph is at http://hint.byu.edu/tutorials/graphs/regime/regime.html I'm sure Mr. Tufte would have some very good suggestions to improve the graph on the above page, though. Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 19:07:34 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 16:07:34 -0800 Subject: [Beowulf] more about crystal oscillators than you ever wanted to know, was Re: Benchmark results In-Reply-To: <3FFDDB9F.3050201@lanl.gov> References: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20040108153952.031c8ba0@mailhost4.jpl.nasa.gov> Fascinating as this is, it is straying pretty far afield from Beowulfery though. I've put some comments on crystal oscillators below Josip's comments.. To keep it somewhat list related, I've looked over some documentation on early PCs, and it appears that the 8253(PC)/8254 (AT and later) didn't have any of the gate inputs brought out conveniently, so you can't use it for timing. And, the game port is a latched input port, with some 555 type timers, triggered by another port output, so you can't jam anything in there either. Looks like IRQs off the serial or parallel port is your best bet. I haven't been able to fully determine how the usual chipsets these days implement the 8259 interrupt controller logic, and how the average P4 handles the interrupt in the first place. So, what one really needs is something that will field the IRQ, grab the CPU counter, and stuff it in a known location somewhere. I should think that this would have very much sub microsecond scatter, even if the entire operation takes several microseconds to do. (and, on modern processors, that's a bunch of ops you're giving up to do the timing...) At 03:37 PM 1/8/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: >>We're talking parts per million here, though, just to keep things in >>perspective. > >I just checked some quartz crystal oscillator specs, and they range all >over the place. The cheap ones recommended for Gigabit Ethernet devices >claim 100 ppm stability, but good quality ones can be ordered with 0.1 ppm >specs. > >HP application node 200-2 "Fundamentals of Quartz Oscillators" (see >http://www.measurement.tm.agilent.com/main.cgi?PSP_NEXT=Redirect&Document:CONTENT_ID=6980) >says that temperature effects are the dominant influence (2.5 ppm within >0-50 deg. C). This is followed by aging (<0.3ppm/month), line voltage >change (<0.1ppm/10% change), and various other smaller effects. > >A good crystal may deliver <2.5 ppm frequency variation over a 0 to 50 >deg. C range, but the temperature dependence may be quite nonlinear. >Temperature compensated crystals can deliver 0.5 ppm stability over the >same range. There's also the issue of phase noise.. you could have a fairly unstable (in the long term sense), but very clean, oscillator. A cesium beam standard is exactly the opposite.. very, very stable in the long term, really noisy in the short term, so the usual standard source uses a very clean quartz oscillator that is locked to the cesium reference (or to GPS, nowadays.. GPS uses Cesium and Rubidium sources on board). It kind of depends on what time scale you want to measure things over. If it's down in the microseconds for the sort of nanobenchmarking that Robert was talking about, then you need a good low jitter source. In a typical Ethernet receiver, there's a tracking loop that acquires the frequency (bit timing) of the incoming signal during the preamble, and it can tolerate a fairly wide offset. However, it really depends on the bit to bit timing being very stable, so it can make the decisions. The internal oscillator in a lab time interval counter might be pretty poor, noise wise,but very stable, since its function is to measure frequency. An oscillator in a receiver, transmitter, or spectrum analyzer may have poorer frequency stability, but be "cleaner" because noise in the oscillator output shows up in the signal being transmitted or received because of reciprocal mixing. There's a tradeoff between aging and temperature dependence as well. A temperature compensated oscillator uses a temperature dependent capacitor in combination with the crystal to pull the crystal slightly to compensate the variations in its resonant frequency. However, the fact that the crystal can be pulled means that it has to be lower Q, and hence, poorer close in noise. VCXOs (Voltage controlled oscillators) have the same general problem.. if you can adjust it, it implies that "other things" can also adjust it, causing random variations. Aging is more a matter of drive power and mechanical stress. Driving the crystal harder gives you a cleaner output (because your farther above the thermal noise floor), but also heats the crystal, causing it to age faster. The ideal scheme is to have a system that does not depend on the crystal frequency being at a particular value, but, rather, can adapt to changing frequencies. Then, you can have a very high quality (stiff) crystal (so the environmental and loading effects don't affect it, and so it has low phase noise), drive the heck out of it (causing it to drift and age rapidly), and measure it periodically against an accurate clock (i.e. Cs, Rb, or GPS). This is basically the NTP approach... measure your computer's clock against a higher stratum reference, and change the "ticks per second" number to adjust so that the clock runs true. You then apply a calibration factor, or retune the radio, or whatever, to account for the known frequency of the oscillator. By the way, this is what they do in GPS receivers.. one of the variables the receiver solves for (in addition to pseudo range and pseudo range rate) is local clock offset and offset rate. It also means you need more satellites to get a fix, though. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 19:48:16 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 16:48:16 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <20040108214835.GA34465@piskorski.com> References: <200401081951.i08JpDh12979@NewBlue.scyld.com> <200401081951.i08JpDh12979@NewBlue.scyld.com> Message-ID: <5.2.0.9.2.20040108164704.031c4aa8@mailhost4.jpl.nasa.gov> At 04:48 PM 1/8/2004 -0500, Andrew Piskorski wrote: >On Thu, Jan 08, 2004 at 11:34:26AM -0500, Robert G. Brown wrote: > > On Thu, 8 Jan 2004, Josip Loncaric wrote: > > > > I love NTP, but in its current state it does not deliver microsecond > > > level synchronization. > > > > NTP may need to be streamlined and tuned for clusters... > > > > I'm not sure that the problem is NTP itself -- from what I recall > > reading in its documentation it is CAPABLE of achieving 1 usec > > synchronization, over time. However, in order to do that it needs > > better tools than the current gettimeofday (and other time calls) which > > seem to deliver no better than perhaps millisecond resolution. It also > > very likely needs to be done not by a userspace daemon but by the > > kernel, as only the kernel can decide to do something in a true "single > >Would DJB's clockspeed, taiclock, etc. stuff be any better than ntp >for that? His website mentions using CPU hardware tick or nanosecond >counters, but doesn't define "precise" when it says that taiclock is >"... suitable for precise time synchronization over a local area >network ...": > > http://cr.yp.to/clockspeed.html > http://cr.yp.to/libtai.html Nope.. from what I can see in the source code, DJB doesn't have a hardware "tick" interface. I suppose, over a long enough time, the uncertainty introduced by grabbing the time from NTP, etc., would average out, but there's still a sync problem. >-- >Andrew Piskorski >http://www.piskorski.com/ >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 8 20:52:51 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 8 Jan 2004 17:52:51 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040108160818.031eb830@mailhost4.jpl.nasa.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040108160818.031eb830@mailhost4.jpl.nasa.gov> Message-ID: <20040109015251.GA6567@greglaptop.internal.keyresearch.com> On Thu, Jan 08, 2004 at 04:14:38PM -0800, Jim Lux wrote: > >Perhaps I'm overly optimistic, but writing a kernel module to achieve > >cluster-wide microsecond-level clock synchronization just might be > >feasible... > > I am entirely certain that this can be done, and may have already been > done. It's such a useful thing to "know, exactly, what time it is".... Octiga Bay claimed to have done it at SC. However, they made it sound like they did a lot of work to get it going, as what they were doing was enabling MPI programs to run in lock-step on microsecond timescales. That's much harder than just getting global times. No doubt SGI has done something clever on their big Altix boxes. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Fri Jan 9 04:44:48 2004 From: jcownie at etnus.com (James Cownie) Date: Fri, 09 Jan 2004 09:44:48 +0000 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: Message from Greg Lindahl of "Thu, 08 Jan 2004 17:52:51 PST." <20040109015251.GA6567@greglaptop.internal.keyresearch.com> Message-ID: <1AetCC-1YI-00@etnus.com> Greg Lindahl wrote :- > On Thu, Jan 08, 2004 at 04:14:38PM -0800, Jim Lux wrote: > > > >Perhaps I'm overly optimistic, but writing a kernel module to achieve > > >cluster-wide microsecond-level clock synchronization just might be > > >feasible... > > > > I am entirely certain that this can be done, and may have already been > > done. It's such a useful thing to "know, exactly, what time it is".... > > Octiga Bay claimed to have done it at SC. However, they made it sound > like they did a lot of work to get it going, as what they were doing > was enabling MPI programs to run in lock-step on microsecond > timescales. That's much harder than just getting global times. But the OctigaBay system has a single network clock from which all the machine clocks are derived, so all of the processors are already synchronised. Given than architecture they can then easily ensure that timer interrupts are also synchronised on all nodes, and it's no big leap from there to co-scheduling the MPI processes. That's rather different from the problem being discussed here, which is how to get clock synchonisation in a Beowulf where you do _not_ have any common clock. (Indeed many of the solutions to that problem which are being suggested are basically to provide a common timing signal (clock input) to all the nodes...) -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Fri Jan 9 04:54:43 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Fri, 9 Jan 2004 09:54:43 +0000 (GMT) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <200401090009.i0909Vh10376@NewBlue.scyld.com> References: <200401090009.i0909Vh10376@NewBlue.scyld.com> Message-ID: > --__--__-- > > > I am in the process of trying to get a stopgap perfomance monitoring > > system going on a 64 CPU Linux cluster with LSF. Just use LSF's own built in data monitoring. "lsload -l" will give you the load, memory, swap usage etc of every node in the cluster. You can extend the data collection with your own scripts (elim) to measure whatever else you want. For example, we monitor temperature and some database load information as well as the usual machine statistics. Data collection is done via a very simple perl script which collects lsload info at regular intervals, and then splats the data into rrdtool (www.rrdtool.com) for storing the historical data and producing graphs. Cheers, Guy Coates. -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 08:29:23 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 08:29:23 -0500 (EST) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <3FFDAB94.2070805@sonsorol.org> Message-ID: On Thu, 8 Jan 2004, Chris Dagdigian wrote: > > {Forwarded to this list on behalf of a friend with some email troubles...} > > > I am in the process of trying to get a stopgap perfomance monitoring > > system going on a 64 CPU Linux cluster with LSF. Ultimately, I hope to > > use PCP for data collection, but since nobody seems to be doing this yet, > > we are going to be rolling our own solution. To meet some of their needs, > > management has asked for an interim solution that gives them a web page > > with aggregate usage statistics and such. > > > > Unfortunately, ganglia is a non-starter because our networking group can't > > enable multicast for the private network the cluster lives on (it would > > break almost everything else). > > > > Does anyone have any suggestions for an alternative that can be quickly > > implemented, doesn't rely on multicast, and that can generate graphs of aggregate > > statistics on the fly? > > Regards, > Chris > BioTeam.net > > > http://www.phy.duke.edu/brahma/Resources/xmlsysd.php http://www.phy.duke.edu/brahma/Resources/wulfstat.php xmlsysd is a daemon that runs on each client. It parses /proc and runs certain systems commands on demand, extracts the requested data, wraps it up in an XML tagging that is fairly self-explanatory and easy/trivial to parse with any of a bunch of XML toolsets and libraries in a variety of scripting and compiler languages, and sends it back via a standard point-to-point TCP connection (no multicast) to the connecting host. It is throttleable -- it only returns the information you want to monitor, in fairly coarse-grained blocks (e.g. it tends to deliver all the information from a given /proc file all at once as the primary overhead is rewinding and reading the file at all, not parsing any given line once it is read, and the burden of sending a packet of 300 bytes and 750 bytes is similarly nearly identical). wulfstat is a provided tty/ncurses monitoring program that reads in an XML-based cluster descriptor/config file (it is very easy to define a cluster using what amount to scanf/printf wildcards and numerical ranges), connects to xmlsysd on all hosts specified in the config file, and builds a display that updates every five seconds or so of the entire cluster in a tty window that can be scrolled or paged. The default display currently presents a vmstat-like set of data, but one can also monitor only 5/10/15 load average, network stats, memory stats, cpu and time info (including duty cycle) and userspace running jobs (which can additionally be masked in the config file with -- you guessed it -- an XML-based set of tags). Wulfstat runs doesn't generate graphs of statistics on the fly, but because xmlsysd is trivial to connect to and control (you can telnet to the port and type in its control commands and see exactly what it returns and how to control/throttle configure it online) and because it returns xml-wrapped data, it would be would be easy, I think, to make it generate graphs and it will definitely scale to 64 nodes. In fact I think one could probably write a simple perl or python script that connects to the cluster xmlsysds, polls the cluster every (say) minute, extracts the quantities you wish to monitor, and write them out on a per-node basis to a file where any graphing utility you like could plot them. There are also almost certainly web/php widgets for plotting -- I just haven't had the need for them and so haven't written this myself (my nodes tend to go to load 1 per CPU and stay there for a month or so, which is pretty boring -- the "interesting" exceptions are easy to spot in real time in wulfstat). If you go this route and implement the fully GPL (v2b, where the b means you owe me a beer if and when we ever meet if you use it:-) toolset, please consider sending me any also GPL cool tools you develop that extend the usefulness of the basic xmlsysd and I'll wrap them up in the package for others. In fact, GPL v2b them and I'll have to buy YOU a beer back if/when we ever meet in human person with a bar handy...;-) rgb P.S. - I don't know exactly how many sites are using xmlsysd/wulfstat, but there are a few -- the tools have been around for a while and I do actively maintain them and have been known to add displays to wulfstat on request just to make a given user happy. One day I'll have LOTS of time on my hands I'm sure and will even port wulfstat to a gtk "real" GUI form, but the nice thing about a tty based display is that it works on a basic tty console if no X is running at all (which may well happen) and everybody has a vast range of xterm choices under X, so it is definitely the common denominator of monitor interfaces. -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 09:06:52 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 09:06:52 -0500 (EST) Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: <20040108225352.GA29682@galactic.demon.co.uk> Message-ID: On Thu, 8 Jan 2004, Andrew M.A. Cater wrote: > Maplin electronics did a kit - cheaply - to run off the time code from > 60kHz clocks and put it into the serial/parallel port. Know any > radio hams - a lot are taking a temperature controlled crystal > oscillator and phase locking to WWWV or equivalent. This is useful to know. I'd really like to set up a local ntp server in my house and it wouldn't hurt to have one in the department at Duke, but not for hundreds of dollars in either case. A 60 KHz signal really sounds like one should be able to tune it and amplify it and deliver it as a pulse accurate within an easy millisecond with a five dollars worth of electronics (coil, cap, rectifier, amplifier and maybe a serial interface chip). I just don't have the time to do a lot of hobby electronics -- I have a really nifty thermal sensor kit upstairs that I've had for six months now without having had time to build it, for example. > > nsec syncronization on a global scale has lots of benefits, some of > > which could enable projects that would dwarf SETI at home. The heck with > > turning every cellular tower in the country into a giant radiotelescope > > array -- turn every suitably equipped PC in the country into a receiver > > in a really enormous radiotelescope array. Now there is a project for a > > GRID! > > Haven't you heard - the Govt. is using them to track individuals by > their individual radar fingerprint. They can watch me wherever I walk > and they're using the system to mess with my mind ... :) Time to take your medication, there, there, it will be all right...;-) > > Unfortunately, a lot of "random" physical sources turn > > out not be random at all, just to have a relatively short > > autocorrelation function and to have distributions that are not uniform > > as well. A lot of the noise-based generators fall into this category, > > but quantum generators can as well (one has to look at second and higher > > order correlation functions, for example -- even resonance fluorescence, > > a textbook example of a "random quantum process" turns out to be photon > > antibunched, and there is also the venerable work of Hanbury Brown and > > Twiss. It turns out not to be horribly easy to build a hardware > > generator that passes diehard or the NIST suite, let alone one that > > works fast. > > > > The Knuth random number generator that converged to a value - the first > example in Seminumerical Algorithms springs to mind :) It is certainly worth remembering that in a deterministic universe (and we DO live in a deterministic universe according to physics, regardless of what one is tempted to conclude by looking at a quantum system that is "inside" a generalized master equation and is analyzed with non-relativistic quantum as well) there are no random number generators or random events, period. There is only entropy, which the way I learned it is the log of the missing information. Missing information is an expression of our degree of ignorance of true state given a limited/projective view of a large, multivariate space, not a statement that the information is in any sense not there. I do wish that more quantum classes taught the GME and quantum statistics properly, I do I do... Marsaglia apparently shares this view. IIRC it is him that I'm semiquoting when I say that hardware RNG's often aren't, at least according to diehard. Even with entropy generators and thermal noise, the RNG has to be BOTH unpredictable AND uniform at the bit level to all orders, and the latter is actually not terribly easy to accomplish without a pure random binary process at hand. I've written this nifty random number generator rating program (as always, alas, functional but unfinished in terms of original design goals) that when finished will encapsulate all the diehard tests and all the NIST STS/FIPS tests in an shell that permits them to be automagically applied to any GSL-wrapped RNG. It also permits the RNGs to be timed for benchmarking purposes (using the cpu_rate timer I published on list a few days ago). Leaving ppm effects aside :-) one discovers that the linux /dev/random entropy generator is pretty good but slow as molasses (not surprising). The Mersenne twister rocks. GSL rocks -- there are over 50 RNG's pre-wrapped and ready to go, including lots of the old/famous failures. It's kind of fun to test some of the original rand()'s and watch them fail all over the place... rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 10:52:26 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 07:52:26 -0800 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C01E@orsmsx402.jf.intel.com> From: Robert G. Brown; Friday, January 09, 2004 5:29 AM > > On Thu, 8 Jan 2004, Chris Dagdigian wrote: > > > > > {Forwarded to this list on behalf of a friend with some email > troubles...} > > > > > I am in the process of trying to get a stopgap perfomance monitoring > > > system going on a 64 CPU Linux cluster with LSF. Ultimately, I hope > to > > > use PCP for data collection, but since nobody seems to be doing this > yet, > > > we are going to be rolling our own solution. To meet some of their > needs, > > > management has asked for an interim solution that gives them a web > page > > > with aggregate usage statistics and such. I'm not sure what is meant by "nobody seems to be doing this yet". Is "this" read as "using pcp" or "using pcp with LSF"? At any rate, CluMon http://clumon.ncsa.uiuc.edu/ uses PCP to collect data, along with Apache, PHP, and MySQL to manage the data. The only issue with the specific request is that it's tied to PBS as the queuing system. That shouldn't be too hard to eliminate or port to LSF. Alternatively, just use the data collection facilities of LSF directly, and while you're at it, use the GUI facilities of LSF to display the data. Alternatively, if the LSF GUI doesn't work for you, just build the GUI you want using the LSF data. Is there a compelling reason to collect the data twice? > > > Unfortunately, ganglia is a non-starter because our networking group > can't > > > enable multicast for the private network the cluster lives on (it > would > > > break almost everything else). I guess the network isn't that private to the cluster then. Otherwise, how would activities on a private network, at best accessible only via the routing on the headnode, impact the rest of the network? But, I digress, challenging IT policy ;^) > One day I'll have LOTS of > time on my hands I'm sure and will even port wulfstat to a gtk "real" > GUI form, but the nice thing about a tty based display is that it works > on a basic tty console if no X is running at all (which may well happen) > and everybody has a vast range of xterm choices under X, so it is > definitely the common denominator of monitor interfaces. I'd argue the web interface is the common denominator. Viewable via text browser, plus you get out of the OS wars... Yes, I know that real sys admins *never* use a web interface... -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From M.Arndt at science-computing.de Fri Jan 9 04:54:51 2004 From: M.Arndt at science-computing.de (Michael Arndt) Date: Fri, 9 Jan 2004 10:54:51 +0100 Subject: [Beowulf] profiling memory bandwidth of a job Message-ID: <20040109105451.A32360@blnsrv1.science-computing.de> Hello, Problem: to get the characteristics of a commercial numerical CFD code on a Cluster with respect to memory bandwidth. The Application is running single cpu jobs on SMP Nodes, no parallel application. The runtimes show symptomes of concurring about memory bandwith. I.e., if two One-CPU Jobs run on the dual CPU Node, the total run time is far more than twice the time of one job + e.g 30 % penalty for ressource competition of both jobs on a SMP Node. Question: Is it possible, to show with standard linux profiling commands that really memory bandwidth is the issue ? (Memory requirements and disk IO rate of the jobs are moderate, as far as iostat / top shows) thx Micha _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 11:57:41 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 11:57:41 -0500 (EST) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <187D3A7CAB42A54DB61F1D05F012572201D4C01E@orsmsx402.jf.intel.com> Message-ID: On Fri, 9 Jan 2004, Lombard, David N wrote: > I'd argue the web interface is the common denominator. Viewable via > text browser, plus you get out of the OS wars... For users, David, I agree, although we tend to be linux-centric and wulfstat is popular enough. procstatd had a perl-cgi web interface written by a friend here at Duke and I'm sure it would be easy enough to port. > Yes, I know that real sys admins *never* use a web interface... It isn't never, but I did write this as a pretty high end administrative/monitoring interface that would be useful to sysadmins running lans and not just clustering people running clusters. I also wrote/designed it so that it would be easy to write things like web interfaces for it. I just haven't got the time (or TOO much motivation) to do so myself, as I don't have users clamoring for it. Somebody (I can't offhand recall who) was looking at it to possibly wrap it up to work within ganglia -- one way to get a web interface. From what I've seen of the ganglia interface, though, wulfstat is MUCH more practically useful for most cluster users or administrators as it stands, xterm or not. Icons and nifty bar charts are pretty, but take up a lot of room. A grid of regularly updated number cells is ugly, but oh, so informative and compact. The current default display is something like (sorry for the ncurses cut/paste ugliness, but hey, it does cut and paste and the column/boxes do actually align:-): name |st|Load 1,5,15|eth0 rx|eth0 tx| si/so | pi/po |ctxt|intr|users ======================================================================== ganesh |up|0.0,0.0,0.0| 8415 | 882 | 0/0 | 0/2 | 156| 160| x c00 |up|2.0,2.0,2.0| 330 | 676 | 0/0 | 0/2 | 9| 105| x c01 |up|2.0,2.0,2.0| 330 | 675 | 0/0 | 0/2 | 9| 105| x ... status, load, network traffic (ganesh's is relatively large because it is receiving all of the xmlsysd data from the many monitored nodes), swapin/swapout, pagein/pageout, context switches, interrupts (users doesn't do anything yet but is intended to somehow indicate what and whose jobs are running...already available in detail on another display). At a glance and without a mouse or even a keyboard used for paging, one has a very good chance of identifying cluster problems on as many hosts as will fit in a tty window on as many tty windows as one can fit on a screen -- hundreds, at least. Usually problems show up as downed nodes, anomalous load averages, unusual network traffic, lots of swapping or paging (indicating memory problems) or lots of ctxt/intr activity (indicating a possible thrashing kernel task, I/O where there shouldn't be, etc.). I do have a mental vision of an even more efficient "panel of lights" display that would run under X as a GTK process that used "temperature" metaphor colors to encode the number values, green to red and still presented the number values if the mouse pointer hovered over a light -- such a display could probably allow a really big cluster (1024 nodes plus) to be monitored at a glance with detail readily available as problems emerged. I just don't have that many nodes that wulfstat doesn't work locally, and writing the app and debugging it is maybe a week's work. With such a display and lights that were order of 10-20 pixels square, one could put a LOT of them onscreen in a neat little array... I don't know if this would be possible in a web browser, at least on a timescale you could live with. Binary can have portability issues, but it is as fast as it gets. rgb > > -- > David N. Lombard > > My comments represent my opinions, not those of Intel Corporation. > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 11:16:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 11:16:08 -0500 (EST) Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results (fwd) Message-ID: ---------- Forwarded message ---------- Date: Fri, 09 Jan 2004 09:22:44 -0600 From: Gerry Creager N5JXS To: Robert G. Brown Subject: Re: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results Check http://www.synergy-gps.com/ and see what they have in their used and refurbed sales pile for an OnCore. There's plenty of info on NTP using an OcCore receiver with 1pps output. Alternately, look for Tom Clark's TAC-2 (Totally Accurate Clock) and an OnCore board or Garmin (but the Garmin's more expensive). Tom's reported some stellar examples of short- and long-term accuracy. If this is interesting enough, please feel free to repost it for me. If not, flush it... I'm off to Seattle to act like a meteorologist for the next week... TTFN, gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 Seems worth it to me, rgb _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 14:15:48 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 11:15:48 -0800 Subject: [Beowulf] profiling memory bandwidth of a job Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C024@orsmsx402.jf.intel.com> From: Michael Arndt; Friday, January 09, 2004 1:55 AM > > Hello, > > Problem: to get the characteristics of a commercial numerical > CFD code on a Cluster with respect to memory bandwidth. > > The Application is running single cpu jobs on SMP Nodes, no > parallel application. > > The runtimes show symptomes of concurring about memory bandwith. > > I.e., if two One-CPU Jobs run on the dual CPU Node, the total run > time is far more than twice the time of one job + e.g 30 % penalty for > ressource competition of both jobs on a SMP Node. > > Question: Is it possible, to show with standard linux profiling commands > that really memory bandwidth is the issue ? > > (Memory requirements and disk IO rate of the jobs are moderate, as far > as iostat / top shows) I'm not sure about profiling tools, but running the job on systems with differing memory configurations (and nothing else) can certainly quantify the impact of memory bandwidth. You also state that disk I/O is a potentially significant factor. If it is, which is a little odd for a CFD app, you also need to consider that. Also look at memory size as an issue potentially affecting multiple jobs. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeffrey.baker at lmco.com Fri Jan 9 14:47:02 2004 From: jeffrey.baker at lmco.com (Baker, Jeffrey) Date: Fri, 09 Jan 2004 13:47:02 -0600 Subject: [Beowulf] Upcoming Beowulf Presentation - Help! Message-ID: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Howdy, all... I've just been selected to give a presentation on 1/27 to the entire IT department about Beowulf's -- of which I know nothing about! I'm interested in finding out what good books or websites are out there that can get me up to "presentation speed" in short order ("Beowulfs for Dummies") ... So far, I've found the Beowulf history < http://beowulf.gsfc.nasa.gov/overview.html > ... Beowulf FAQ < http://www.canonical.org/~kragen/beowulf-faq.txt > ... and Engineering a Beowulf-style Compute Cluster < http://www.phy.duke.edu/brahma/beowulf_book/ > ... all which should help tremendously. Any other sites I should know about? Many thanks! Jeff in Dallas _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 14:58:01 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 14:58:01 -0500 (EST) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <187D3A7CAB42A54DB61F1D05F012572201D4C020@orsmsx402.jf.intel.com> Message-ID: On Fri, 9 Jan 2004, Lombard, David N wrote: > By timescale, I assume you mean display time? That's where the > abstractions come in to play, and, for the all-at-once method, better > output preparation, i.e., a single bit image over multiple images. So, > I'd suggest that the only thing you lose is live-update, which can be an > issue, to be sure. I don't think that we disagree, really. I freely admit there are times where a web view of a cluster is adequate and they certainly tend to be very easy to understand. It is really the quintessential Macintoad tradeoff, epitomized in the epigrammatic "You can learn to use a Mac in a day, then pay for that knowledge for the rest of your life" (anon). Really, this applies to GUIs in general, not just Macs -- there are certain things for which a GUI abstraction are really good for, but often at the expense of speed, or power, or complexity. A command line interface and Unix itself are described as being "expert friendly" -- difficult to learn, complex, but SO powerful once you do learn them. A GUI browser is very lovely for certain kids of browsing, but if you want to move all the files in a directory with the glob foo*.bar.?arian-[0-9]* into the trash, gimme a command line every time, especially if there are a few thousand of them. Still, try EXPLAINING how to do that to a thirteen year old compared to saying "pick up the file with the mouse and drag it into the trash, silly" and the virtue of GUIs as well becomes clear. Each has its place. xmlsysd is designed to support both -- it is divorced from the display mechanism entirely. Connect to it, talk to it, ask it nicely, it spits out xml-tagged anonymous data you might be interested in monitoring. Do whatever you like with this data -- present it via TTY, pipe it to a file, graph it, whatever. I invested my time in the tty interface THIS time because I made the mistake of building a GUI interface first LAST time with procstatd (xmlsysd's predecessor) and spent a year or two later kicking myself for a variety of reasons. The tty interface is fast and information-dense, but has flaws as well (it is a pain to rearrange and customize, for example). If I ever have time, as I said, I'll write a better interface (and maybe even two or three). In the meantime, anyone who wants them sooner can play through -- it is really pretty simple;-) rgb > > -- > David N. Lombard Sr. Staff Software Engineer 1.949.567.4843 > Parallel & Distributed Processing Division, Intel Corporation > > My comments represent my opinions, not those of Intel Corporation. > > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 15:57:12 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 12:57:12 -0800 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C027@orsmsx402.jf.intel.com> From: Robert G. ; Friday, January 09, 2004 11:58 AM > > On Fri, 9 Jan 2004, Lombard, David N wrote: > > > By timescale, I assume you mean display time? That's where the > > abstractions come in to play, and, for the all-at-once method, better > > output preparation, i.e., a single bit image over multiple images. So, > > I'd suggest that the only thing you lose is live-update, which can be an > > issue, to be sure. > > I don't think that we disagree, really. Actually, you're preaching to the choir. > Really, this applies to GUIs in general, not just Macs -- there are > certain things for which a GUI abstraction are really good for, but > often at the expense of speed, or power, or complexity. I agree fully. A GUI is the best way to display massive amounts of info, especially where you want to be able to quickly vary the level of detail, like in a comprehensive view of a cluster with the ability to drill down at many points. Nothing new -- use text when you want relatively small amounts of detail, and graphics when you want trends and large amounts of information. > A command line interface and Unix itself are described as being "expert > friendly" -- difficult to learn, complex, but SO powerful once you do > learn them. Amen brother. I very much prefer Linux, and have a long and pleasent history in UNIX. > A GUI browser is very lovely for certain kids of browsing, > but if you want to move all the files in a directory with the glob > foo*.bar.?arian-[0-9]* into the trash, gimme a command line every time, > especially if there are a few thousand of them. Still, try EXPLAINING > how to do that to a thirteen year old compared to saying "pick up the > file with the mouse and drag it into the trash, silly" and the virtue of > GUIs as well becomes clear. Each has its place. Violently agreed, both points. > xmlsysd is designed to support both -- it is divorced from the display > mechanism entirely. Connect to it, talk to it, ask it nicely, it spits > out xml-tagged anonymous data you might be interested in monitoring. Do > whatever you like with this data -- present it via TTY, pipe it to a > file, graph it, whatever. I invested my time in the tty interface THIS > time because I made the mistake of building a GUI interface first LAST > time with procstatd (xmlsysd's predecessor) and spent a year or two > later kicking myself for a variety of reasons. The tty interface is > fast and information-dense, but has flaws as well (it is a pain to > rearrange and customize, for example). That was exactly my motivation for the web comment. I spent a fair amount of time putting together web-based interfaces for command-line driven tools, and I've found that's the ideal combination for both system administration and users. Develop the tool in an environment that favors utility and (most importantly) composability, and then _optionally_ interact with the tool in an environment that favors usability and accessibility. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 13:47:51 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 10:47:51 -0800 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C020@orsmsx402.jf.intel.com> From: Robert G. Brown [mailto:rgb at phy.duke.edu]; Friday, January 09, 2004 > > I do have a mental vision of an even more efficient "panel of lights" > display that would run under X as a GTK process that used "temperature" > metaphor colors to encode the number values, green to red and still > presented the number values if the mouse pointer hovered over a light -- > such a display could probably allow a really big cluster (1024 nodes > plus) to be monitored at a glance with detail readily available as > problems emerged. I just don't have that many nodes that wulfstat > doesn't work locally, and writing the app and debugging it is maybe a > week's work. With such a display and lights that were order of 10-20 > pixels square, one could put a LOT of them onscreen in a neat little > array... There's more to GUI scalability than small icons. The best scalability comes from being able to range from "show me everything all at once" to "roll up the data in various meaningful terms". Possibilities like machine rooms and racks are things I've prototyped and received generally favorable responses. > I don't know if this would be possible in a web browser, at least on a > timescale you could live with. Binary can have portability issues, but > it is as fast as it gets. By timescale, I assume you mean display time? That's where the abstractions come in to play, and, for the all-at-once method, better output preparation, i.e., a single bit image over multiple images. So, I'd suggest that the only thing you lose is live-update, which can be an issue, to be sure. -- David N. Lombard Sr. Staff Software Engineer 1.949.567.4843 Parallel & Distributed Processing Division, Intel Corporation My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From apseyed at bu.edu Fri Jan 9 15:20:39 2004 From: apseyed at bu.edu (apseyed at bu.edu) Date: Fri, 09 Jan 2004 15:20:39 -0500 Subject: [Beowulf] diskless cluster nodes ? Message-ID: <1073679639.3fff0d17d203d@www.bu.edu> What type of jobs would be typical of a scenerio where diskless nodes (nodes with no hard drives/pxe booting the kernel) would be most optimal. I understand that the jobs to be run on our future cluster don't do alot of I/O, i.e. once running it does alot of computations for a long period of time, therefore after loading into memory not much disk access. However, we have a couple users utilizing MPI.. Any thoughts on this appreciated. -P _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Fri Jan 9 15:06:04 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Fri, 09 Jan 2004 15:06:04 -0500 Subject: [Beowulf] profiling memory bandwidth of a job In-Reply-To: <20040109105451.A32360@blnsrv1.science-computing.de> References: <20040109105451.A32360@blnsrv1.science-computing.de> Message-ID: <1073678764.8918.21.camel@squash.scalableinformatics.com> On Fri, 2004-01-09 at 04:54, Michael Arndt wrote: > I.e., if two One-CPU Jobs run on the dual CPU Node, the total run > time is far more than twice the time of one job + e.g 30 % penalty for > ressource competition of both jobs on a SMP Node. So you believe that you are running out of memory bandwidth? > > Question: Is it possible, to show with standard linux profiling commands > that really memory bandwidth is the issue ? You would need to use tools like Oprofile, or Troy Baer's lperfex (works with perfctr, which requires a kernel patch last I looked). > (Memory requirements and disk IO rate of the jobs are moderate, as far > as iostat / top shows) Some notes: 1) 'vmstat 1' is your friend. Use it to see at a gross level, what is going on with the system. Is it possible you are swapping? Is it possible that you are hitting the interrupts hard (don't know why), or doing lots of context switching? 2) The linux tools won't give you a fine grain view, you will need to go to something like Oprofile. 3) Before you do two runs, it might be advisable to create a baseline run, and profile it. Look for the usual suspects (paging, memory, interrupts, ...). Gather profile data. Unless you have source for your commercial code, you probably would not be able to get the code to generate profile data, so you are left with the other tools. > > thx > Micha > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Jan 9 17:24:17 2004 From: becker at scyld.com (Donald Becker) Date: Fri, 9 Jan 2004 17:24:17 -0500 (EST) Subject: [Beowulf] Beowulf List adminstrivia: SPF now published for beowulf.org Message-ID: The beowulf domain now publishes Sender Permitted From records. Detailed information on SPF is at http://spf.pobox.com/ but here is a quick summary: SPF is an advisory field in the domain name record that can help mail receipients reject spam that pretends to come from popular mailers such as beowulf.org. SPF doesn't require you to do anything. It does allow you or your ISP to set up better spam filtering rules, if you choose. To see what SPF "really does", run a DNS name server query on beowulf.org and check out the "additional text" records. There is a record for SPF version 1: beohost$ host -t txt beowulf.org beowulf.org. text "v=spf1 a mx ptr -all" This says "If you get a message from beowulf.org, it must come from A the primary host for beowulf.org MX the mail exchange hosts for beowulf.org PTR any host whose domain name ends in .beowulf.org -all and no other machine I don't expect even the slightest inconvenience from providing SPF records for beowulf.org. Please let me know if it causes any problems. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster systems Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rpickler at dnahelix.com Fri Jan 9 16:59:54 2004 From: rpickler at dnahelix.com (Rich Pickler) Date: Fri, 09 Jan 2004 15:59:54 -0600 Subject: [Beowulf] Recommendations for new cluster Message-ID: <3FFF245A.7070804@dnahelix.com> We're looking a few months into the future and are beginning to design our next renderfarm for production at our studio. Instead of going with a render system tailored specifically for the renderer we're using, like Alfred for Pixar's RenderMan http://www.pixar.com/, we're toying with the idea of a more generalized cluster system which will open up a ton of possibilities for what we can do with the farm. However, having only tinkered with generalized cluster systems during college, I am a bit new to some of the details and what is right for us. After exhausting Google for all I can get out of it, I have some questions over what exactly is right for us. And I would rather have real world experiences with tools instead of someone trying to sell me something. In a nutshell, the questions we have: -What batch system do you recommend for our purposes (Condor, PBSPro, ScalablePBS, SGE)? -What monitoring tools and set up tools have you found and used that reduce down time on faulty nodes (ganglia, management tools, etc.)? -Fail over of the head node is also important to us, but I haven't researched enough on this topic to start asking questions. (So far, I've only found stuff for SGE, and a rumor that PBSPro will add this feature soon) A bit of detail on our requirements: First, we're looking at a cluster of around 400 CPUs, which is getting dangerously close to the limits of OpenPBS. Considering that this number could easily grow down the road, pushing that limit as far back as possible is needed. So far, PBS looks like the most widely used and known system out there, so some variant of that seems ideal for finding prospective employees with cluster experience, although it looks like a lot of you deal with SGE as well. PBSPro and ScalablePBS advertise they get the scalability limit much higher than what we need, but the first is much more expensive. I have yet to find any documentation or rumor of SGE's upper bound. The majority of jobs in our queue will be small and single node (but definitely multithreaded to take advantage of dual cpu nodes). This means intercommunication between nodes is almost nil. However, there will be cases where we need processes to communicate outside the cluster with workstations, but that isn't terribly difficult to design around. One of the problems I have noticed is that some batch managers bog down with a large amount of jobs. Looking at the structure of our jobs, we can easily wind up with 20,000-40,000 jobs in a day with a large amount of them being submitted to the batch around the same time (when everybody leaves work). Any thoughts what might be the best solution for us? Thanks for any recommendations you can give me, and feel free to ask me any more details you want. Rich _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctierney at hpti.com Fri Jan 9 19:07:12 2004 From: ctierney at hpti.com (Craig Tierney) Date: 09 Jan 2004 17:07:12 -0700 Subject: [Beowulf] Latency issues with Itanium2 (Tiger2) and e1000 gigE Message-ID: <1073693232.4266.14.camel@hpti7.fsl.noaa.gov> I am having some problems with some new servers I just got. I have several dual Itanium2 (Intel Tiger2 board) connected via gigE through a Dell Powerconnect 5224 gigE switch. Using mpptest to test latency and bandwidth the results are not so good. Latency varies between 60-200 mu/s for small packets, and the bandwidth tops out at about 45 MB/s. - This happens with both a Debian and White Box (rebuilt from RHAS 3.0 sources). - I have a similar set of Opteron nodes that use the same model switch (different phyiscal switch though) and the latency and bandwidth of the built in tg3 card is 32 mu/s and and 100 MB/s respectively. - I have verified (at least on the hosts) that both are in full-duplex mode. Does anyone have any suggestions on what else to look for? Thanks, Craig _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laurenceliew at yahoo.com.sg Fri Jan 9 18:46:00 2004 From: laurenceliew at yahoo.com.sg (Laurence Liew) Date: Sat, 10 Jan 2004 07:46:00 +0800 Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> References: <3FFF245A.7070804@dnahelix.com> Message-ID: <1073691959.914.11.camel@scalable> We use SGE with much success and little problems. As a builder of clusters, we support many customers.. and installing SGE for customers since late 2001 has meant a lot less hassle for us in terms of support, reliability and robustness.... previously with another scheduler... we had to visit our customers quite a bit.. SGE also supports shadow/secondary master nodes for failover of the master SGE itself.. it may be important to you Condor is also a good scheduler Ganglia for monitoring. We have used Rocks (www.rocksclusters.org) with SGE to quickly build a renderfarm (smallish 64 CPU) in under 1 day incorporating Renderman. If checkpoint and restart is important to you at the head and compute node level, you may wish to take a look at www.meiosys.com. These guys have an interesting virtualisation technology which allows transparent checkpointing and restart mechanism... this means you do not need specialised binaries... most applications should run ontop of their technology. Have fun! Cheers! laurence On Sat, 2004-01-10 at 05:59, Rich Pickler wrote: > We're looking a few months into the future and are beginning to design > our next renderfarm for production at our studio. Instead of going with > a render system tailored specifically for the renderer we're using, like > Alfred for Pixar's RenderMan http://www.pixar.com/, we're toying with > the idea of a more generalized cluster system which will open up a ton > of possibilities for what we can do with the farm. However, having only > tinkered with generalized cluster systems during college, I am a bit new > to some of the details and what is right for us. After exhausting > Google for all I can get out of it, I have some questions over what > exactly is right for us. And I would rather have real world experiences > with tools instead of someone trying to sell me something. > > In a nutshell, the questions we have: > > -What batch system do you recommend for our purposes (Condor, PBSPro, > ScalablePBS, SGE)? > > -What monitoring tools and set up tools have you found and used that > reduce down time on faulty nodes (ganglia, management tools, etc.)? > > -Fail over of the head node is also important to us, but I haven't > researched enough on this topic to start asking questions. (So far, > I've only found stuff for SGE, and a rumor that PBSPro will add this > feature soon) > > A bit of detail on our requirements: > > First, we're looking at a cluster of around 400 CPUs, which is getting > dangerously close to the limits of OpenPBS. Considering that this > number could easily grow down the road, pushing that limit as far back > as possible is needed. So far, PBS looks like the most widely used and > known system out there, so some variant of that seems ideal for finding > prospective employees with cluster experience, although it looks like a > lot of you deal with SGE as well. PBSPro and ScalablePBS advertise they > get the scalability limit much higher than what we need, but the first > is much more expensive. I have yet to find any documentation or rumor > of SGE's upper bound. > > The majority of jobs in our queue will be small and single node (but > definitel multithreaded to take advantage of dual cpu nodes). This > means intercommunication between nodes is almost nil. However, there > will be cases where we need processes to communicate outside the cluster > with workstations, but that isn't terribly difficult to design around. > > One of the problems I have noticed is that some batch managers bog down > with a large amount of jobs. Looking at the structure of our jobs, we > can easily wind up with 20,000-40,000 jobs in a day with a large amount > of them being submitted to the batch around the same time (when > everybody leaves work). > > Any thoughts what might be the best solution for us? > > Thanks for any recommendations you can give me, and feel free to ask me > any more details you want. > > Rich > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Fri Jan 9 21:17:51 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 10 Jan 2004 10:17:51 +0800 (CST) Subject: [Beowulf] Upcoming Beowulf Presentation - Help! In-Reply-To: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Message-ID: <20040110021751.5788.qmail@web16813.mail.tpe.yahoo.com> beowulf is good for HPC clusters running parallel jobs. If you use the cluster like a compute farm, then make sure you also install a DRM (distributed resource management, or a batch system) on top of the beowulf. Common ones like SGE (Sun's open source DRM), SPBS (much much better than PBS), or LSF (commercial). I would suggest new DRM installations to choose SGE instead of LSF, since SGE is free, and SGE 6.0 is coming soon and fixes some limitations that some people encounter. LSF is quite similar to SGE, but it is very expensive (last time I heard it costed $500 US per CPU, and support fee is not cheap), and for the price of 1 CPU license, I can get another Linux box with SGE, and I won't loose much. SPBS is also good, it is free, and used by a lot of HPC sites. They used to use OpenPBS, but OpenPBS's development is basically dead, and a lot of bugs not fixed! Andrew. --- "Baker, Jeffrey" ????> Howdy, all... > > I've just been selected to give a presentation on > 1/27 to the entire IT department about Beowulf's -- > of which I know nothing about! > > I'm interested in finding out what good books or > websites are out there that can get me up to > "presentation speed" in short order ("Beowulfs for > Dummies") ... > > So far, I've found the Beowulf history < > http://beowulf.gsfc.nasa.gov/overview.html > > ... Beowulf FAQ < > http://www.canonical.org/~kragen/beowulf-faq.txt > > ... and Engineering a Beowulf-style Compute Cluster > < http://www.phy.duke.edu/brahma/beowulf_book/ > > ... all which should help tremendously. > > Any other sites I should know about? > > Many thanks! > > Jeff in Dallas > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Fri Jan 9 21:05:36 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 10 Jan 2004 10:05:36 +0800 (CST) Subject: [Beowulf] Apple gets it right this time? Message-ID: <20040110020536.37964.qmail@web16802.mail.tpe.yahoo.com> x-serve with ECC, 42 of those in a rack. http://www.apple.com/xserve/ Also, Apple previews x-grid: http://www.apple.com/acg/xgrid/ Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sat Jan 10 00:20:10 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 10 Jan 2004 13:20:10 +0800 (CST) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> Message-ID: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> --- Rich Pickler ???? > Instead of going with a render system tailored > specifically for the renderer we're using, like > Alfred for Pixar's RenderMan http://www.pixar.com/, You are not alone, Framestore CFC is doing the same thing. They are looking at using SGE 6.0 to replace Alfred. You can follow the mail thread -- "SGE Renderfarm": http://gridengine.sunsource.net/servlets/ReadMsg?msgId=14218&listName=users (Since you are in this graphics industry, can you tell me if Framestore is big? People on the sge list looked so excited.) > And I would rather have real world experiences > with tools instead of someone trying to sell me > something. Use Opensource (r), and no one will ask you for money. > -What batch system do you recommend for our purposes > (Condor, PBSPro, ScalablePBS, SGE)? SGE has array jobs, but all others do not. An array job is submitted as a single job, but each element of the array takes different inputs. It is much faster to submit a single array job than to submit each element as seperately. And you can stop and del the whole array job or just each individual element. AFAIK, this job array feature makes graphics rendering and certain types of bio-computation job management easily -- e.g. different frames are part of the same job array. > -What monitoring tools and set up tools have you > found and used that reduce down time on faulty nodes > (ganglia, management tools, etc.)? ganglia is good, a lot of people use it. I just think setting up a webpage is the overkilled part -- need to run Apache to monitor the cluster -- but just MHO only. If you like command line stuff better, you can use qhost in SGE to see which host is down. > I've only found stuff for SGE, and a rumor that > PBSPro will add this feature soon) Yes, PBSPro 5.4 will support it. > So far, PBS looks like the most widely used and > known system out there, so some variant of that > seems ideal for finding prospective employees with > cluster experience, I find PBS is used more in HPC clusters (MPI, PVM jobs), and SGE is used more in enterprise and biocomputation clusters. The problem with SGE when handling MPI jobs is that it takes time to set up the PE, and PBS has an external interface which is much nicer. People had already written mpiexec to take advantage of the interface. > I have yet to find any documentation or rumor > of SGE's upper bound. SGE scales quite well, I haven't seen people complaining about it, may be that's why you couldn't find it. Just to let you know, Forecast Systems Laboratory has a cluster using SGE, it has 1536-CPUs. They gave a presentation of their cluster last year. SGE 6.0 (coming this year) will scale even further (like multithread SGE, and use a DBMS to handle job spooling), you can take a lot at Sun's presentations at: http://gridengine.sunsource.net/project/gridengine/workshop22-24.09.03/proceedings.html > One of the problems I have noticed is that some > batch managers bog down with a large amount of > jobs. Make sure you have lots of memory in the head node, and no swapping. I used to try how SGE behaves when there are tons of submitted jobs, I went up to 100,000 and it was still running, but of course it was just a toy cluster with several hosts. However, I think 20,000-40,000 can be handled very easily. > Any thoughts what might be the best solution for us? I think the best is to try it yourself, as it all depends on what you want to do. Download the SGE binaries (or source, if you really wanted to), and install it on one of the test machines. It doesn't require root access, and it can be placed any directory. It's free, all you need is time. Also, I highly recommend the mailing lists, a lot of people are on the lists, and can solve problems quickly. http://gridengine.sunsource.net/project/gridengine/maillist.html Like the "SGE Renderfarm" thread, they exchange experience and share knowledge very often. Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 04:52:35 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 10:52:35 +0100 (CET) Subject: [Beowulf] Upcoming Beowulf Presentation - Help! In-Reply-To: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Message-ID: On Fri, 9 Jan 2004, Baker, Jeffrey wrote: > Howdy, all... > > I've just been selected to give a presentation on 1/27 to the entire IT department about Beowulf's -- of which I know nothing about! > > I'm interested in finding out what good books or websites are out there that can get me up to "presentation speed" in short order ("Beowulfs for Dummies") ... > > So far, I've found the Beowulf history < http://beowulf.gsfc.nasa.gov/overview.html > > ... Beowulf FAQ < http://www.canonical.org/~kragen/beowulf-faq.txt > > ... and Engineering a Beowulf-style Compute Cluster < http://www.phy.duke.edu/brahma/beowulf_book/ > > ... all which should help tremendously. > Useful sites indeed. In addition to these, you might look at the books: "Beowulf Cluster Computing with Linux" , editied by Thomas Sterling. published by MIT Press "Linux Clustering" by Charles Bookman, published by New Riders. The second one is an easier introduction. Good Luck with the presentation! _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Sat Jan 10 05:38:32 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: 10 Jan 2004 10:38:32 +0000 Subject: [Beowulf] Latency issues with Itanium2 (Tiger2) and e1000 gigE In-Reply-To: <1073693232.4266.14.camel@hpti7.fsl.noaa.gov> References: <1073693232.4266.14.camel@hpti7.fsl.noaa.gov> Message-ID: <1073731111.2389.22.camel@fermi> On Sat, 2004-01-10 at 00:07, Craig Tierney wrote: > I am having some problems with some new servers I just > got. I have several dual Itanium2 (Intel Tiger2 board) > connected via gigE through a Dell Powerconnect 5224 gigE > switch. > > Using mpptest to test latency and bandwidth the results > are not so good. Latency varies between 60-200 mu/s for > small packets, and the bandwidth tops out at about 45 MB/s. > > - This happens with both a Debian and White Box (rebuilt > from RHAS 3.0 sources). > Does anyone have any suggestions on what else to look for? Yes. I suppose the networking chip is an Intel model - something like a 82546 or 82547 running with the e1000 driver? If this is the case, and the kernel module is inserted w/o any special options, "Interrupt Throttling" kicks in. In the default case Intel has implemented a heuristical approach, which limits the number of interrupts per sec the NIC produces. This is done to lower the CPU overhead in high volume communications. Excellent for large transfers but really bad for low-latency communication :-( What we usually do is setting the InterruptThrottleRate to something other than the default value, 1. Look here for information: http://www.intel.com/support/network/adapter/1000/e1000.htm This does not solve the problem fully though: Setting the ITR too high (50000-75000) makes the system unresponsive under certain network loads. Something like 25000 allows for most communication, like NFS and SSH but still software like LAM-MPI has problems. Setting the value arround 10000 makes most software run smoothly, but latency then is arround 50 us (i think) on our hardware.. But the latency is constant (no 60 to 250us) fluctuations. We have not found an optimal solution (other than using different NIC's) If anyone has any ideas i would be grateful. best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 04:59:43 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 10:59:43 +0100 (CET) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> Message-ID: On Fri, 9 Jan 2004, Rich Pickler wrote: > > In a nutshell, the questions we have: > > -What batch system do you recommend for our purposes (Condor, PBSPro, > ScalablePBS, SGE)? > Over on the SGE list we've had a discussion about this over the last week. Perhaps you could search the archives at gridengine.sunsource.net Notably, there was a contribution from the company I used to work for, Framestore, who are doing a similar exercise - evaluating SGE and PBS etc. for a renderfarm. a few other companies made controbutions. Perhaps it would bee worth joining the SGE list and asking the question there? Also of interest to the rendering community, if you are thinking of Opterons, is that there is a native x86-64 version of SGE now. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 05:20:27 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 11:20:27 +0100 (CET) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> Message-ID: On Sat, 10 Jan 2004, [big5] Andrew Wang wrote: > (Since you are in this graphics industry, can you tell > me if Framestore is big? People on the sge list looked > so excited.) ps. I'm excited myself. I like using SGE, and I think it would be great to see it running a big renderfarm. The Enterprise Edition has I think project-based priority, which might be useful in giving projects coming to a deadline more time. I've no idea how big the Framestore-CFC render farm is at the moment. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 05:13:16 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 11:13:16 +0100 (CET) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> Message-ID: On Sat, 10 Jan 2004, [big5] Andrew Wang wrote: > > You are not alone, Framestore CFC is doing the same > thing. They are looking at using SGE 6.0 to replace > Alfred. > > (Since you are in this graphics industry, can you tell > me if Framestore is big? People on the sge list looked > so excited.) > They are now the biggest post-production house in Europe, I believe. When I worked for them they made many series for Hallmark, the Walking with Dinosaurs series, TV adverts and shots for many feature films. For example the tracking shot through four seasons along the market in Notting Hill, James Bond title sequences. I joined the company just after the merger with Computer Film Company. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Sat Jan 10 08:00:46 2004 From: proksy_me at yahoo.com (prakash borade) Date: Sat, 10 Jan 2004 05:00:46 -0800 (PST) Subject: [Beowulf] help for my beowulf Message-ID: <20040110130047.6599.qmail@web40110.mail.yahoo.com> hello i need your help for my beowulf cluster i am using linux redhat-9.0 mpich-1.2.5.2 implementtaion of mpi i require to enable thw lam environment for my program to run please send me some links that can lead me to the succesfull beowulf or if possible any documenttaion fopr buoilding the cluster thanks in advace proksy_me at yahoo.com __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Sat Jan 10 08:47:11 2004 From: raysonlogin at yahoo.com (Rayson Ho) Date: Sat, 10 Jan 2004 05:47:11 -0800 (PST) Subject: [Beowulf] help for my beowulf In-Reply-To: <20040110130047.6599.qmail@web40110.mail.yahoo.com> Message-ID: <20040110134711.82821.qmail@web11407.mail.yahoo.com> MPICH and LAM are different MPI implementations. Rayson --- prakash borade wrote: > hello > > > i need your help for my beowulf cluster > i am using linux redhat-9.0 > mpich-1.2.5.2 implementtaion of mpi > > > i require to enable thw lam environment for my program > to run ^^^^^^^^^^^^^^^^ > __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Sat Jan 10 08:26:25 2004 From: proksy_me at yahoo.com (prakash borade) Date: Sat, 10 Jan 2004 05:26:25 -0800 (PST) Subject: [Beowulf] (no subject) Message-ID: <20040110132625.9563.qmail@web40110.mail.yahoo.com> mpich implementation can anybody sujects me the best benchmarking suit for the computational cluster __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Sat Jan 10 09:40:14 2004 From: raysonlogin at yahoo.com (Rayson Ho) Date: Sat, 10 Jan 2004 06:40:14 -0800 (PST) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> Message-ID: <20040110144014.74052.qmail@web11410.mail.yahoo.com> > -What batch system do you recommend for our purposes (Condor, PBSPro, > ScalablePBS, SGE)? My take is, if you know PBS, or your users have experience with PBS before, go for SPBS or PBSPro. If you use SPBS, also take a look at Maui, since SPBS scheduling algorithm is limited. If you start clean, use SGE. Condor is also good, but the developers haven't released the source (they said they will), not a big problem. *However*, your cluster information is spied by Condor, sent back to UW, for "research purposes only". I personally like condor a lot (esp the checkpointing feature), but the spying art makes them look like M$. > -Fail over of the head node is also important to us, but I haven't > researched enough on this topic to start asking questions. (So far, > I've only found stuff for SGE, and a rumor that PBSPro will add this > feature soon) I've requested this feature on the SPBS mailing list, I assume the SPBS developers are looking into it. > First, we're looking at a cluster of around 400 CPUs, which is > getting > dangerously close to the limits of OpenPBS. Considering that this > number could easily grow down the road, pushing that limit as far > back > as possible is needed. The problem is not the number of CPUs you have, it is the number of nodes you have. OpenPBS scheduler contacts each node in each scheduling cycle to get the load information. If one of the nodes is dead, then the scheduler hangs. The hack they did was to set up an alarm, wait for the reply from the node, and timeout after 2-3 minutes, and restart. --> like M$... exit all windows and restart :) OpenPBS works OK with less than 300 nodes, but as more and more nodes are added to the cluster, the likelihood of a node failing increases. SPBS fixed this problem 2 months ago, the developers said that they have changed the server <-> mom communication protocol. SGE does not have this problem, AFAIK. As long as the server <-> node communication protocol is designed correctly, you can in theory add as many nodes to the cluster as you want. In the extreme case (several 10,000 nodes?), there are way too many nodes reporting the load information to the master, the master will not do much work but just keeps on accepting load info. And at this point you need to decrease the load report frequency. The bottomline: as long as the communication protocol is OK (SPBS, PBSPro, SGE, LSF), you don't need to worry. But OpenPBS is really something you need to avoid for mid-size clusters. > So far, PBS looks like the most widely used > and > known system out there, so some variant of that seems ideal for > finding > prospective employees with cluster experience, although it looks like > a > lot of you deal with SGE as well. SGE has tons of names, it's a bit confusing. It used to be a research project "DQS", then commercialized, became "gridware", "Codine", "GRD", and then bought by Sun, renamed to "SGE", but some people call it "Sun Grid", while some call it "Gridengine". PBS called itself PBS at the very beginning. > PBSPro and ScalablePBS advertise > they > get the scalability limit much higher than what we need, but the > first > is much more expensive. Yes, PBSPro don't fix the known problems and well-known limitations in OpenPBS, and they ask you to buy PBSPro -- good marketing strategy :) > I have yet to find any documentation or > rumor > of SGE's upper bound. Again, as long as you are not using OpenPBS, don't worry... > One of the problems I have noticed is that some batch managers bog > down > with a large amount of jobs. Looking at the structure of our jobs, > we > can easily wind up with 20,000-40,000 jobs in a day with a large > amount > of them being submitted to the batch around the same time (when > everybody leaves work). Should be fine. The scheduler is CPU-bound (a Linux PC box can handle that amount of scheduling work), while the job holder (qmaster in SGE) is I/O bound. If the spool directory is in an imported NFS partition, then you will find that job submission slow. > > Any thoughts what might be the best solution for us? > > Thanks for any recommendations you can give me, and feel free to ask > me > any more details you want. BTW, subscribe to the SPBS and SGE mailing lists for further info. Rayson > > Rich > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Sun Jan 11 21:45:17 2004 From: lathama at yahoo.com (Andrew Latham) Date: Sun, 11 Jan 2004 18:45:17 -0800 (PST) Subject: [Beowulf] linuxbios Message-ID: <20040112024517.80352.qmail@web60307.mail.yahoo.com> Just got my "Linux Journal" and was reading about the linuxbios. It mentioned tyan (My fav.) I searched tyan.com and found nothing related. (just thought about googling tyan.com) Any Tyan MB Linux BIOS users out there? What are the success stories out there. I am starting to play and am already downloading some stuff I just wondered what the group thinks about this. http://www.clustermatic.org/ http://www.linuxbios.org ===== /---------------------------------------------------------------------------------------------------\ Andrew Latham -LathamA - Penguin Loving, Moralist Agnostic. What Is an agnostic? - An agnostic thinks it impossible to know the truth in matters such as, a god or the future with which religions are concerned with. Or, if not impossible, at least impossible at the present time. LathamA.com - (lay-th-ham-eh) - lathama at lathama.com - lathama at yahoo.com \---------------------------------------------------------------------------------------------------/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Jan 12 03:52:03 2004 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 12 Jan 2004 09:52:03 +0100 (CET) Subject: [Beowulf] linuxbios In-Reply-To: <20040112024517.80352.qmail@web60307.mail.yahoo.com> Message-ID: I've been following it for a couple of years. At the Linux Beer Hike in Ireland in 2002 we did some experiments, and got it running. I've stilll got the PC-chips motherboard which I bought to do the tests. Antony Stone found a UK supplier if the disk-on-chip parts. Antony did a very good article for the UK version Linux Magazine. I haven't had the courage though to try Linuxbios on my precious mini-ITXes. I'm mostly interested in Linuxbios though for the controlability of large clusters. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Mon Jan 12 12:44:00 2004 From: becker at scyld.com (Donald Becker) Date: Mon, 12 Jan 2004 12:44:00 -0500 (EST) Subject: [Beowulf] BWBUG January 2004 -- Sun Microsystems Linux plans Message-ID: Date: January 13, 2003 Time: 2:45 PM Location: Northrop Grumman IT, McLean Virginia Sun Microsystems will provide an in-depth presentation regarding their success with Linux and future strategy to become one of the world leaders in Linux Beowulf systems. That's correct, it is LINUX not Solaris. The presentation will cover current and planned features of Sun Grid Engine, as well as the new Sun Fire V60x Compute Grid Rack System. Sun has installed a number of Linux Beowulf Clusters on the Top 500 List in 2003. This event will be sponsored by the Baltimore-Washington Beowulf Users Group (BWBUG) and will be held at Northrop Grumman Information Technology 7575 Colshire Drive, 2nd floor, McLean Virginia. Please register on line at http://bwbug.org to assure a spot. This event has attracted a great deal of interest. Date: January 13, 2003 Time: 2:45 PM Parking: Free Need to be a member?: No ( guests are welcome ) For more details see listing in Washington Technology: http://www.washingtontechnology.com/events/16835.html T. Michael Fitzmaurice, Jr. Coordinator of the BWBUG 703-205-3132 office 240-475-7877 cell Email michael.fitzmaurice at ngc.com http://www.bwbug.org -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster systems Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Per.Lindstrom at me.chalmers.se Mon Jan 12 14:54:42 2004 From: Per.Lindstrom at me.chalmers.se (=?ISO-8859-1?Q?Per_Lindstr=F6m?=) Date: Mon, 12 Jan 2004 20:54:42 +0100 Subject: [Beowulf] The name Beowulf-cluster Message-ID: <4002FB82.5030101@me.chalmers.se> Hi, Is there some one that know why the clustering technique is named Beowulf? According to a viking saga was Beowulf a king that ruled //the kingdom of Geat, (Gothia, G?taland that is located in west Sweden) around year 500, i.e. about 700 year before the rise of the Swedish nation. Best regards Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gust_77ar at yahoo.com Mon Jan 12 17:36:08 2004 From: gust_77ar at yahoo.com (Gustavo Velardez) Date: Mon, 12 Jan 2004 14:36:08 -0800 (PST) Subject: [Beowulf] G98 on cluster Message-ID: <20040112223608.49919.qmail@web60702.mail.yahoo.com> Hi everybody, I'm trying to run Gaussian98 on one or several nodes in a beowulf cluster. Are there another different option than qsub or similar? Thanks Gustavo __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rpickler at dnahelix.com Mon Jan 12 19:35:35 2004 From: rpickler at dnahelix.com (Rich Pickler) Date: Mon, 12 Jan 2004 18:35:35 -0600 Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> References: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> Message-ID: <40033D57.3050702@dnahelix.com> >(Since you are in this graphics industry, can you tell >me if Framestore is big? People on the sge list looked >so excited.) > > > Yeah, they're big. I talk with a couple people there quite often so I'll drop them a line and pick their brains a little bit. >Use Opensource (r), and no one will ask you for money. > > The OpenSource + pay support hasn't really sold everyone here yet. Seems logical to me, but I guess they need to be won over. Speaking of, one of the guys here was really upset with Sun's support and is leery of Grid Engine because of some issue with a Cobalt Rack system. I was always under the impression Sun's support was quite good, the cobalt stuff was just kind of a red-headed step child they never wanted. What's everybody's perception of Grid Engine's support? I've heard some accounts of great responsiveness from the community and getting bugs fixed very quickly. Unfortunately, it's hard to sell to the bosses based on community support alone. >SGE has array jobs, but all others do not. > > I'll have to check this feature out. Sounds very interesting and well suited for our kind of usage. It hadn't come up in my searches before. >ganglia is good, a lot of people use it. I just think >setting up a webpage is the overkilled part -- need to >run Apache to monitor the cluster -- but just MHO >only. > > People are going to have to go in and see how their jobs are progressing on the farm. We're talking about artists here. Asking them to cut off their arms would be easier than getting them to log into a shell and type a few commands. So far it looks like pretty much every clustering system has hooks for integrating it into a web page. (Luckily this isn't my area to develop). So we ought to have something quite nice for them to use. >The problem with SGE when handling MPI jobs is that it >takes time to set up the PE, and PBS has an external >interface which is much nicer. People had already >written mpiexec to take advantage of the interface. > > MPI is completely a non issue here. Even if I were to write a cluster app in the traditional sense, I wouldn't use it. (Bad memories in college, but I suspect they had it poorly set up) >I think the best is to try it yourself, as it all >depends on what you want to do. > > That's next month when we buy the development cluster. We just need some recommendations before diving in. Everybody, thanks for the help on this last weekend. We start talking to different hardware providers this week, so I needed a quick background in the options before they try and pitch something to us that will only cause headaches down the road. Rich _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Tue Jan 13 00:26:06 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Tue, 13 Jan 2004 13:26:06 +0800 (CST) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <40033D57.3050702@dnahelix.com> Message-ID: <20040113052606.10833.qmail@web16808.mail.tpe.yahoo.com> > Speaking of, one of the guys here was really upset > with Sun's support > and is leery of Grid Engine because of some issue > with a Cobalt Rack system. You can also pick another company to support SGE, there are actually more than 20 for you to choose from: http://wwws.sun.com/software/gridware/partners/index.html > People are going to have to go in and see how their > jobs are progressing > on the farm. We're talking about artists here. > Asking them to cut off > their arms would be easier than getting them to log > into a shell and > type a few commands. A guy switched from another batch system to SGE and wrote a very simple web GUI: http://www.veus.hr/linux/gemonitor.html Also, Sun has "Grid Engine Portal", but it requires the Solaris Portal server: http://gridengine.sunsource.net/project/gridengine/gep/GEP_Intro.html Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Jan 13 03:25:43 2004 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Tue, 13 Jan 2004 09:25:43 +0100 Subject: [Beowulf] The name Beowulf-cluster In-Reply-To: <4002FB82.5030101@me.chalmers.se> ( =?iso-8859-1?q?Per_Lindstr=F6m's_message_of?= "Mon, 12 Jan 2004 20:54:42 +0100") References: <4002FB82.5030101@me.chalmers.se> Message-ID: Per Lindstr?m writes: > Is there some one that know why the clustering technique is named Beowulf? Thomas Sterling claims he sat in his room, trying to think of a name. He looked up at his bookshelf, saw an edition of the Beowulf saga, and thought "'Beowulf', that'll do. Nobody will hear about it, anyways". -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Andrew.Cannon at nnc.co.uk Tue Jan 13 03:40:34 2004 From: Andrew.Cannon at nnc.co.uk (Cannon, Andrew) Date: Tue, 13 Jan 2004 08:40:34 -0000 Subject: [Beowulf] Solaris on x86? Message-ID: Hi there, We are looking to get some x86 based compute power (possibly a number of Athlon based machines) and I was wondering if anyone has any experience of Solaris for x86, specifically, how well does it work in a cluster? Would we be better looking at Red Hat PW or ES or some other linux distro. The company is not too keen on going down the ROCKS etc route and would prefer a "mainstream" solution. TIA. Andy Andrew Cannon, Nuclear Technology (J2), NNC Ltd, Booths Hall, Knutsford, Cheshire, WA16 8QZ. Telephone; +44 (0) 1565 843768 email: mailto:andrew.cannon at nnc.co.uk NNC website: http://www.nnc.co.uk NNC's UK Operating Companies : NNC Holdings Limited (no. 3725076), NNC Limited (no. 1120437), National Nuclear Corporation Limited (no. 2290928), STATS-NNC Limited (no. 4339062) and Technica-NNC Limited (no. 235856). The registered office of each company is at Booths Hall, Chelford Road, Knutsford, Cheshire WA16 8QZ except for Technica-NNC Limited whose registered office is at 6 Union Row, Aberdeen AB10 1DQ. NNC's head office and principal address is Booths Hall and the switchboard number is 01565 633800. The NNC website is www.nnc.co.uk Any information or opinions in this message which do not relate to our business are not authorised by any of the above companies. Where this message does relate to our business the content is sent to you by the relevant company (as above) and is confidential and intended for the use of the individual or entity to whom it is addressed. The contents do not form part of any contract unless so stated. If you have received this e-mail in error please notify the NNC system manager by email at eadm at nnc.co.uk. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmdavis1 at vcu.edu Mon Jan 12 20:58:55 2004 From: jmdavis1 at vcu.edu (Mike Davis) Date: Mon, 12 Jan 2004 20:58:55 -0500 Subject: [Beowulf] G98 on cluster In-Reply-To: <20040112223608.49919.qmail@web60702.mail.yahoo.com> References: <20040112223608.49919.qmail@web60702.mail.yahoo.com> Message-ID: <400350DF.1080508@vcu.edu> You will need a pbs wrapper. In addition, to run in parallel, you need the LINDA libraries. I have a wrapper for GAMESS/G98 and a standalone submission script (interactive, character based). Let me know if you would like them. Mike Gustavo Velardez wrote: >Hi everybody, >I'm trying to run Gaussian98 on one or several nodes >in a beowulf cluster. Are there another different >option than qsub or similar? > >Thanks > > Gustavo > >__________________________________ >Do you Yahoo!? >Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes >http://hotjobs.sweepstakes.yahoo.com/signingbonus >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mas at ucla.edu Tue Jan 13 09:51:18 2004 From: mas at ucla.edu (Michael Stein) Date: Tue, 13 Jan 2004 06:51:18 -0800 Subject: [Beowulf] multiline string literals... In-Reply-To: ; from rgb@phy.duke.edu on Tue, Jan 13, 2004 at 08:29:25AM -0500 References: Message-ID: <20040113065118.A20459@mas1.ats.ucla.edu> > This is a teeny bit off-topic, but I don't know where else to find a > group of knowledgeable systems programmers. > > Does anybody here know WHY multiline literals, e.g. specifically > something like: > > char usage = " > Usage: foobar [-h] [-v] -f barianfile > -h for help > -v for verbose > -f barianfile to specify your foo-bar-barian > "; > > are deprecated in gcc? Probably because it's hard to tell the difference between the above and an unbalanced double quote character? An unbalanced quote could eat a big hunk of your code and possibly without any error messages... Try this instead: char usage = "\n" "Usage: foobar [-h] [-v] -f barianfile\n" " -h for help\n" " -v for verbose\n" " -f barianfile to specify your foo-bar-barian\n" " "; See ANSI C standard section 6.1.4 String literals which says that adjacent strings literals are concatenated into a single string. PS: You need either *usage or usage[]. -Wall is your friend... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 13 08:29:25 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 13 Jan 2004 08:29:25 -0500 (EST) Subject: [Beowulf] multiline string literals... Message-ID: This is a teeny bit off-topic, but I don't know where else to find a group of knowledgeable systems programmers. Does anybody here know WHY multiline literals, e.g. specifically something like: char usage = " Usage: foobar [-h] [-v] -f barianfile -h for help -v for verbose -f barianfile to specify your foo-bar-barian "; are deprecated in gcc? I have a very hard time understanding this. A string is a bunch of bytes in memory. I can put anything I want in it. It is clearly a million times more tedious to code this in one line at a time, and many times more likely to contain a bug as well. Sorry to rant a bit, but I've spent some time trying to understand just what it is that they're saving me from or what low-level aspect of a computer is made ambiguous by multiline literals in C code and come up empty. Seems like pure fascism to me, which is anathema in C. You want rigorous structure, use Pascal... rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Tue Jan 13 10:51:36 2004 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Tue, 13 Jan 2004 16:51:36 +0100 (CET) Subject: [Beowulf] multiline string literals... In-Reply-To: Message-ID: On Tue, 13 Jan 2004, Robert G. Brown wrote: > Does anybody here know WHY multiline literals ... are deprecated in gcc? As written at the top of: http://gcc.gnu.org/gcc-3.3/changes.html they were deprecated for quite some time. So it's a bit late to ask now :-) > A string is a bunch of bytes in memory. I think that this has more to do with (pre)processing the source rather than storing the result in memory. > It is clearly a million times more tedious to code this in one line at a > time, and many times more likely to contain a bug as well. Agreed :-) One other example is inlined ASM... -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Jan 13 09:56:20 2004 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Tue, 13 Jan 2004 15:56:20 +0100 Subject: [Beowulf] multiline string literals... In-Reply-To: (Robert G. Brown's message of "Tue, 13 Jan 2004 08:29:25 -0500 (EST)") References: Message-ID: "Robert G. Brown" writes: > Does anybody here know WHY multiline literals, e.g. specifically > something like: > > char usage = " > Usage: foobar [-h] [-v] -f barianfile > -h for help > -v for verbose > -f barianfile to specify your foo-bar-barian > "; > > are deprecated in gcc? Because they are not C? -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgoornaden at intnet.mu Tue Jan 13 11:43:40 2004 From: rgoornaden at intnet.mu (roudy) Date: Tue, 13 Jan 2004 20:43:40 +0400 Subject: [Beowulf] Linpack Benchmark References: <200401121706.i0CH6Xh18291@NewBlue.scyld.com> Message-ID: <000701c3d9f4$6cdff4d0$080b7bca@roudy> Hello Everybody, I have just build my cluster, and now I want to measure its performance via hpl. I download this software some days back, but I have some difficulty to run it. Below is the several steps I undertake.. tar -xvf hpl.tar -----here it is fine Then I create a file call "Make.cluster" in the directory "/hpl" Afterwards, I use the following command "make arch=cluster" but this does not work...can someone give me a hint of what I should do???? Best Greetings Roud _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nfaerber at penguincomputing.com Tue Jan 13 13:56:35 2004 From: nfaerber at penguincomputing.com (Nate Faerber) Date: 13 Jan 2004 10:56:35 -0800 Subject: [Beowulf] Linpack Benchmark In-Reply-To: <000701c3d9f4$6cdff4d0$080b7bca@roudy> References: <200401121706.i0CH6Xh18291@NewBlue.scyld.com> <000701c3d9f4$6cdff4d0$080b7bca@roudy> Message-ID: <1074020195.29028.10.camel@m10.penguincomputing.com> > Below is the several steps I undertake.. > > tar -xvf hpl.tar -----here it is fine > > Then I create a file call "Make.cluster" in the directory "/hpl" > Afterwards, I use the following command "make arch=cluster" but this does > not work...can someone give me a hint of what I should do???? > First, check that TOPdir is set correctly in your Makefile. Then send the output so we can see where it is breaking. -- Nate Faerber, Engineer Tel: 415-358-2666 Fax: 415-358-2646 Toll Free: 888-PENGUIN PENGUIN COMPUTING www.penguincomputing.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Jan 13 13:46:17 2004 From: becker at scyld.com (Donald Becker) Date: Tue, 13 Jan 2004 13:46:17 -0500 (EST) Subject: [Beowulf] BW-BUG meeting today (Tuesday Jan 13, 2004) in McLean Va Message-ID: Remember that this month's meeting is McLean VA , not Greenbelt MD Baltimore-Washington Beowulf User Group January 2004 Meeting Date: January 13, 2003 Time: 2:45 PM Location: Northrop Grumman IT, McLean Virginia Sun Microsystems will provide an in-depth presentation regarding their success with Linux and future strategy to become one of the world leaders in Linux Beowulf systems. That's correct, it is LINUX not Solaris. The presentation will cover current and planned features of Sun Grid Engine, as well as the new Sun Fire V60x Compute Grid Rack System. Sun has installed a number of Linux Beowulf Clusters on the Top 500 List in 2003. This event will be sponsored by the Baltimore-Washington Beowulf Users Group (BWBUG) and will be held at Northrop Grumman Information Technology 7575 Colshire Drive, 2nd floor, McLean Virginia. Please register on line at http://bwbug.org to assure a spot. This event has attracted a great deal of interest. Date: January 13, 2003 Time: 2:45 PM Parking: Free Need to be a member?: No ( guests are welcome ) For more details see listing in Washington Technology: http://www.washingtontechnology.com/events/16835.html T. Michael Fitzmaurice, Jr. Coordinator of the BWBUG 703-205-3132 office 240-475-7877 cell Email michael.fitzmaurice at ngc.com http://www.bwbug.org -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster systems Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Tue Jan 13 12:56:42 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Tue, 13 Jan 2004 12:56:42 -0500 (EST) Subject: [Beowulf] The name Beowulf-cluster In-Reply-To: <4002FB82.5030101@me.chalmers.se> Message-ID: On Mon, 12 Jan 2004, Per Lindstr?m wrote: > Hi, > > Is there some one that know why the clustering technique is named Beowulf? > > According to a viking saga was Beowulf a king that ruled //the kingdom > of Geat, (Gothia, G?taland that is located in west Sweden) around year > 500, i.e. about 700 year before the rise of the Swedish nation. You may find reading this: http://www.linux-mag.com/2003-06/breakthroughs_01.html useful. It was written by Tom Sterling and provides some history. Doug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Tue Jan 13 16:32:15 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Tue, 13 Jan 2004 13:32:15 -0800 Subject: [Beowulf] Upcoming Beowulf Presentation - Help! In-Reply-To: References: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Message-ID: <20040113213215.GB3343@greglaptop.internal.keyresearch.com> On Tue, Jan 13, 2004 at 03:27:02PM -0500, Douglas Eadline, Cluster World Magazine wrote: > If you can find it Gregory Pfister's "In Search of Clusters" provides a > great technical overview of why clusters are going to be important, BTW > the book is pre-Linux/Beowulf There's a second edition that's post-Beowulf. When reading it, you should keep in mind that he buys the "shared memory programming is easier than message passing" dogma, and he only thinks about fairly small cpu counts. Some of his statements make a lot more sense if you remember where he comes from. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bushnell at chem.ucsb.edu Tue Jan 13 17:59:01 2004 From: bushnell at chem.ucsb.edu (John Bushnell) Date: Tue, 13 Jan 2004 14:59:01 -0800 (PST) Subject: [Beowulf] G98 on cluster In-Reply-To: <400350DF.1080508@vcu.edu> Message-ID: In the Gaussian 03 distribution, there is a script called 'subg03' in the 'bsd' subdirectory. G98 may have something similar. - John On Mon, 12 Jan 2004, Mike Davis wrote: > You will need a pbs wrapper. In addition, to run in parallel, you need > the LINDA libraries. I have a wrapper for GAMESS/G98 and a standalone > submission script (interactive, character based). Let me know if you > would like them. > > Mike > > > Gustavo Velardez wrote: > > >Hi everybody, > >I'm trying to run Gaussian98 on one or several nodes > >in a beowulf cluster. Are there another different > >option than qsub or similar? > > > >Thanks > > > > Gustavo _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 13 20:20:33 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 13 Jan 2004 20:20:33 -0500 (EST) Subject: [Beowulf] multiline string literals... In-Reply-To: <20040113065118.A20459@mas1.ats.ucla.edu> Message-ID: On Tue, 13 Jan 2004, Michael Stein wrote: > Probably because it's hard to tell the difference between the above and > an unbalanced double quote character? An unbalanced quote could > eat a big hunk of your code and possibly without any error messages... Yeah, but the stuff below has 2N-2 more opportunities to fail to close a quote and several other opportunities to screw up, and it isn't any more difficult than balancing {}'s across the miles of intervening code. I was thinking it might have to do with internationalization or something like that and not just somebody's opinion of what is easy or hard to safely code. > Try this instead: > char usage = "\n" > "Usage: foobar [-h] [-v] -f barianfile\n" > " -h for help\n" > " -v for verbose\n" > " -f barianfile to specify your foo-bar-barian\n" > " "; > > See ANSI C standard section 6.1.4 String literals which says that > adjacent strings literals are concatenated into a single string. Sigh. This works and thank you. Of course just a big string WORKS, it just complains every time the file is compiled and this shuts it up. Ah, in the good old days you could often declare a constant string like this (without complaint) and even write back into it without anything complaining. Of course in the good old days, code leaked like a sieve and buffer overwrite attacks existed in nearly every daemon or utility... > PS: You need either *usage or usage[]. -Wall is your friend... :-) I know, pointer to the data block. Just sloppy typing in the example by hand. As you say, that too is what compilers are for -- to help you find little buglets. Thanks again. I'll adopt your suggestion just so the compiler shuts up. I hate programs that don't build perfectly clean. Especially when I'm writing them. So eventually I am nagged out of deprecated habits...:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From root at pixelcatcher.de Wed Jan 14 06:06:03 2004 From: root at pixelcatcher.de (Michael =?ISO-8859-1?Q?Damb=F6ck?=) Date: Wed, 14 Jan 2004 12:06:03 +0100 Subject: [Beowulf] How to set up 2 Node Failover Cluster Message-ID: <1074078362.2246.10.camel@acernote.evonet.de> Hello everybody, I'm told to set up a 2 Node Failover Linux Cluster. Is ist possible to do this using beowulf? I've been google-ing all day but unfortuneately I didn't get what I was looking for. I hope you can help me. If it is possible to do that using beowulf please tell me where I can find Documents describing the way this can be done in general. Alternatively please tell me what other free Software is available for setting up a linux cluster. Thanks a lot! Michael -- http://www.pixelcatcher.de digitalfotografie online _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Wed Jan 14 07:28:19 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Wed, 14 Jan 2004 12:28:19 +0000 (GMT) Subject: [Beowulf] Re: Linpack Benchmark (roudy) In-Reply-To: <200401131703.i0DH3vh22722@NewBlue.scyld.com> References: <200401131703.i0DH3vh22722@NewBlue.scyld.com> Message-ID: >Hello Everybody, I have just build my cluster, and now I want to measure >its performance via hpl. IBM have a redbook on running and tuning HPL for benchmarking clusters. Take a look at: http://www.redbooks.ibm.com/ and look for "Running HPL LINPACK on IBM xSeries Linux Clusters". Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jgreenseid at wesleyan.edu Wed Jan 14 10:34:07 2004 From: jgreenseid at wesleyan.edu (Joe Greenseid) Date: Wed, 14 Jan 2004 10:34:07 -0500 (EST) Subject: [Beowulf] How to set up 2 Node Failover Cluster In-Reply-To: <1074078362.2246.10.camel@acernote.evonet.de> References: <1074078362.2246.10.camel@acernote.evonet.de> Message-ID: This type of clustering isn't really what Beowulf clusters do. You should try checking out the High Availability Linux project at http://linux-ha.org. --Joe On Wed, 14 Jan 2004, Michael [ISO-8859-1] Damb?ck wrote: > Hello everybody, > > I'm told to set up a 2 Node Failover Linux Cluster. > Is ist possible to do this using beowulf? I've been google-ing all day > but unfortuneately I didn't get what I was looking for. > > I hope you can help me. If it is possible to do that using beowulf > please tell me where I can find Documents describing the way this can be > done in general. Alternatively please tell me what other free Software > is available for setting up a linux cluster. > > Thanks a lot! > > Michael > -- > http://www.pixelcatcher.de > digitalfotografie online > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Wed Jan 14 10:32:11 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed, 14 Jan 2004 07:32:11 -0800 Subject: [Beowulf] How to set up 2 Node Failover Cluster Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C041@orsmsx402.jf.intel.com> From: Michael Damb?ck; Wednesday, January 14, 2004 3:06 AM > > Hello everybody, > > I'm told to set up a 2 Node Failover Linux Cluster. > Is ist possible to do this using beowulf? I've been google-ing all day > but unfortuneately I didn't get what I was looking for. No, and yes. No, Beowulf doesn't provide failover; yes, you can use failover within a cluster with good and desirable results. The biggest issue is what the failover will handle, as that determines the software you need. For example, do you want to provide a highly available NFS server? Web server? Cluster headnode? Other "stateless" server? Database? Queuing system? Something else? The NFS and web servers, cluster headnode, and other "stateless" servers can be handled by "heartbeat" at http://www.linux-ha.org/download/ This code allows you to define highly available resources that are managed by heartbeat, the failover manager. You'll typically have heartbeat own a well-known highly available IP address along with any other resources, e.g., shared storage. Heartbeat will only allow one of the two nodes to host the HA resources until that node fails, heartbeat will then start the resources on the other (backup) node and optionally disable the replaced server via a STONITH (shoot the other node in the head) mechanism to prevent a "split-brain" situation where two nodes think they own a resource. The limitation to heartbeat is that it cannot manage any shared state, as you would find in a database server or queuing system. For an HA database or queuing system, you'll need to look at an app that specifically provides that capability, e.g., SGE for queuing. Even in these latter cases, you still may want to use heartbeat to provide a well-known HA IP address to host the HA app. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Wed Jan 14 12:31:07 2004 From: john.hearns at clustervision.com (John Hearns) Date: Wed, 14 Jan 2004 18:31:07 +0100 (CET) Subject: [Beowulf] linuxbios In-Reply-To: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: On Wed, 14 Jan 2004, Erwan Velu wrote: > Can you develop that point ? What is really intresting in such > technologie ? Once your bios are well setup, you don't need to change it > every day..> Of course that's true. It just happens that I was just back from a customer site visit, where Iwas doing some BIOS upgrades. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jlb17 at duke.edu Wed Jan 14 11:01:03 2004 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed, 14 Jan 2004 11:01:03 -0500 (EST) Subject: [Beowulf] Dual Xeon gigabit issues Message-ID: I just moved two dual Xeon servers into a new home. One of them refuses to see a network link, before or after the machine boots. The only difference in them is a slight revision in the motherboard: the one that links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel 82546EB -- dual port gigabit). They're connected to a Cisco Catalyst 4507R. The box *will* link when connected to a 100Mb switch. I also hooked it up to a Dell 5224 (gigabit) switch that I have, and it found a link after about 10 seconds. Any ideas? I've already tried flashing the BIOS and playing with mii-tool a bit. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University _______________________________________________ Dulug mailing list Dulug at lists.dulug.duke.edu https://lists.dulug.duke.edu/mailman/listinfo/dulug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Wed Jan 14 12:36:45 2004 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Wed, 14 Jan 2004 14:36:45 -0300 (ART) Subject: [Beowulf] Nodes with or without HD? Message-ID: <20040114173646.68930.qmail@web12204.mail.yahoo.com> I would like to know how better or worse is a cluster with nodes that have HD. All my nodes in my cluster have HD with the System installed. I cannot see why it could be worse. Well if anyone have the answer, please answer. Thanks Mathias Brito ===== Mathias Brito Universidade Estadual de Santa Cruz - UESC Departamento de Ci?ncias Exatas e Tecnol?gicas Estudante do Curso de Ci?ncia da Computa??o ______________________________________________________________________ Conhe?a a nova central de informa??es anti-spam do Yahoo! Mail: http://www.yahoo.com.br/antispam _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 14 12:58:38 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 14 Jan 2004 12:58:38 -0500 Subject: [Beowulf] Nodes with or without HD? In-Reply-To: <20040114173646.68930.qmail@web12204.mail.yahoo.com> References: <20040114173646.68930.qmail@web12204.mail.yahoo.com> Message-ID: <1074103117.4307.49.camel@pel> On Wed, 2004-01-14 at 12:36, Mathias Brito wrote: > I would like to know how better or worse is a cluster > with nodes that have HD. All my nodes in my cluster > have HD with the System installed. I cannot see why it > could be worse. Well if anyone have the answer, please > answer. There are plus and minuses. If you are not familiar with what it takes to admin a bunch of machines and keep them all in sync together (probablly using something like yum), then you may find it easier to have diskless nodes all sharing a single /usr over NFS so that its easier to update software. Likewise, diskless nodes can be quicker to add to the cluster, however if you're using kickstart and have a good kickstart config setup, nodes with disks aren't that hard to setup either. On the other side.. diskless machines require more network load in general. Unless it happens to be cached, whenever you need to access a file, you have to hit the network. This causes network congestion that can slow down MPI and such. It also means that it takes the slave node wanting the file longer to read it. So it can be quite a performance hit. But that also depends on what your networking needs are based on what kind of networking you have. Note, when you said diskless, I assumed you're talking about using NFS-root. If you're using Scyld, then things are a bit different. Their system of diskless slave nodes is a lot different. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From timm at fnal.gov Wed Jan 14 13:01:43 2004 From: timm at fnal.gov (Steven Timm) Date: Wed, 14 Jan 2004 12:01:43 -0600 (CST) Subject: [Beowulf] Nodes with or without HD? In-Reply-To: <20040114173646.68930.qmail@web12204.mail.yahoo.com> References: <20040114173646.68930.qmail@web12204.mail.yahoo.com> Message-ID: The disadvantage of having disk in the cluster is that you will increase your maintenance cost. we have 600+ nodes at Fermilab, all with system disks and data disks, and disks are almost 70% of all hardware calls that we have. To avoid disk in the cluster there is more technical work that needs to be done, we are investigating that at the moment. Steve ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division/Core Support Services Dept. Assistant Group Leader, Scientific Computing Support Group Lead of Computing Farms Team On Wed, 14 Jan 2004, [iso-8859-1] Mathias Brito wrote: > I would like to know how better or worse is a cluster > with nodes that have HD. All my nodes in my cluster > have HD with the System installed. I cannot see why it > could be worse. Well if anyone have the answer, please > answer. > > Thanks > Mathias Brito > > ===== > Mathias Brito > Universidade Estadual de Santa Cruz - UESC > Departamento de Ci?ncias Exatas e Tecnol?gicas > Estudante do Curso de Ci?ncia da Computa??o > > ______________________________________________________________________ > > Conhe?a a nova central de informa??es anti-spam do Yahoo! Mail: > http://www.yahoo.com.br/antispam > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From moloned at tcd.ie Wed Jan 14 13:46:13 2004 From: moloned at tcd.ie (david moloney) Date: Wed, 14 Jan 2004 18:46:13 +0000 Subject: [Beowulf] Market information on Beowulf clusters? Message-ID: <40058E75.7040409@tcd.ie> Could anybody point me to public and/or commercial market info. on Beowulf clusters especially: Market size (total installations, total nodes) histogram #nodes/cluster performance (GFLOPs) price cluster/node Thanks, - David _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From elfcool at yahoo.com Wed Jan 14 14:03:21 2004 From: elfcool at yahoo.com (Elif Sariarslan) Date: Wed, 14 Jan 2004 11:03:21 -0800 (PST) Subject: [Beowulf] Running MPICH on more than one node Message-ID: <20040114190321.42034.qmail@web40105.mail.yahoo.com> Hi, I have installed mpich on two machines. I am unable to execute my programs on 2 nodes. I have enabled ssh without password requirements and I have added both machines to machines.LINUX file but I still get the following error message when I execute the following command: [root at node02 basic]# ../../bin/mpirun -np 2 systest node00: Connection refused p0_26120: p4_error: Child process exited while making connection to remote process on node00.dl805.eng.ohio-state.edu: 0 ../../bin/mpirun: line 1: 26120 Broken pipe /root/mpich-1.2.5.2/examples/basic/systest -p4pg /root/mpich-1.2.5.2/examples/basic/PI26032 -p4wd /root/mpich-1.2.5.2/examples/basic Thanks, Elif __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bari at onelabs.com Wed Jan 14 14:07:47 2004 From: bari at onelabs.com (Bari Ari) Date: Wed, 14 Jan 2004 13:07:47 -0600 Subject: [Beowulf] linuxbios In-Reply-To: <20040112024517.80352.qmail@web60307.mail.yahoo.com> References: <20040112024517.80352.qmail@web60307.mail.yahoo.com> Message-ID: <40059383.4070004@onelabs.com> Andrew Latham wrote: > Just got my "Linux Journal" and was reading about the linuxbios. > It mentioned tyan (My fav.) > I searched tyan.com and found nothing related. > (just thought about googling tyan.com) > Any Tyan MB Linux BIOS users out there? > > What are the success stories out there. I am starting to play and am already > downloading some stuff I just wondered what the group thinks about this. Fwd from the LinuxBIOS list: "If you want to order a Tyan board with LinuxBIOS pre-loaded, you might need to go through some special ordering process... try asking Charlie Ding (cding at tyan.com). Currently we're working on the S2885. VGA works to an extent now, we can use the testbios utility to init the VGA BIOS on a GeForce FX and X will run with the "nv" driver. The binary driver will load and X will start, but the screen goes corrupt. So long as it's not a driver issue, this might be fixed in the near future. Getting the CLI should be first, and that will be nice since Ollie just got keyboard input working." "well there is the largest K8 cluster in the world, here at LANL, 1408 nodes. Built by linuxbios, arima mobo, linuxbios support from linux networx. Tyan will sell you K8 mobos with linuxbios on them. Hopefully the tyan people reading this will get in touch with you. There are a few other companies that will do K8 mobos with linuxbios but nothing public yet." _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tod at gust.sr.unh.edu Wed Jan 14 14:41:30 2004 From: tod at gust.sr.unh.edu (Tod Hagan) Date: 14 Jan 2004 14:41:30 -0500 Subject: [Beowulf] Running MPICH on more than one node In-Reply-To: <20040114190321.42034.qmail@web40105.mail.yahoo.com> References: <20040114190321.42034.qmail@web40105.mail.yahoo.com> Message-ID: <1074109290.16051.5.camel@haze.sr.unh.edu> Mpich uses rsh by default. Did you set 'export RSHCOMMAND=ssh' when compiling mpich? Can you run the mpich tests ('make testing') successfully? Before running mpich tests programs, you must be able to ssh in to the remote node yourself WITHOUT typing a password. See http://www.puddingonline.com/~dave/publications/SSH-with-Keys-HOWTO/document/html/SSH-with-Keys-HOWTO.html On Wed, 2004-01-14 at 14:03, Elif Sariarslan wrote: > Hi, > > I have installed mpich on two machines. I am > unable to execute my programs on 2 nodes. I have > enabled ssh without password requirements and I have > added both machines to machines.LINUX file but I still > get the following error message when I execute the > following command: > > [root at node02 basic]# ../../bin/mpirun -np 2 systest > > > node00: Connection refused > p0_26120: p4_error: Child process exited while making > connection to remote process on > node00.dl805.eng.ohio-state.edu: 0 > ../../bin/mpirun: line 1: 26120 Broken pipe > /root/mpich-1.2.5.2/examples/basic/systest -p4pg > /root/mpich-1.2.5.2/examples/basic/PI26032 -p4wd > /root/mpich-1.2.5.2/examples/basic > > Thanks, > > Elif > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 14 15:52:39 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 14 Jan 2004 15:52:39 -0500 (EST) Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: Message-ID: On Wed, 14 Jan 2004, Joshua Baker-LePain wrote: > I just moved two dual Xeon servers into a new home. One of them refuses > to see a network link, before or after the machine boots. The only > difference in them is a slight revision in the motherboard: the one that > links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't > has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel > 82546EB -- dual port gigabit). They're connected to a Cisco Catalyst > 4507R. You might first try looking for any messages in /var/log/messages when you load the driver module. I assume you are using e1000.o. Are the versions identical on both systems ? The output from things like uname -a lsmod may be a good place to start. BTW, with Intel chips-sets showing link to a switch does not necessarily imply that the kernel module is working. Doug > > The box *will* link when connected to a 100Mb switch. I also hooked it > up to a Dell 5224 (gigabit) switch that I have, and it found a link after > about 10 seconds. > > Any ideas? I've already tried flashing the BIOS and playing with mii-tool > a bit. > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From avkon at imm.uran.ru Wed Jan 14 15:03:14 2004 From: avkon at imm.uran.ru (Alexandr Konovalov) Date: Thu, 15 Jan 2004 01:03:14 +0500 Subject: [Beowulf] Bonding in 2.4.22-1.2149 Linux kernel Message-ID: <4005A082.1020701@imm.uran.ru> Hi, we have somewhat non-standart configuration of our computing cluster. Inernal computing nodes are connected by 2 (bonded) ethernet networks, but host connected only to one ethernet (i.e., connection to host not bonded for the all computing nodes). For the Linux kernel 2.4.9 - 2.4.20 it's possible to run such configuration if we use static arp tables and static routes (connection to host should be routed directly to the slave device, say eth1, not to bond0). But when we install 2.4.22-1.2149 (from Fedora 1), things are changed. Now routing to slave device became impossibe with diagnostics like ioctl(3, 0x890b, 0xbfe90b10) = -1 ENODEV (No such device) And there is no more slave devices in route output. Googling give us some reasons for this graceful changing of (yes, undocumented and unspecified) behaviour. It's compitely unacceptable for us to abandon bonding, so after drivers/net/bonding/bond_main.c reading we (probably) found a solution. Out idea is simply create new bonding policy. If a packet destination is in the bonded network it will call bond_xmit_roundrobin, if not then packed will be xmited directly to an appropriate slave device. MAC adresses for the direct xmiting will received by modiles as the load parameters. Probably we miss some opportunities and make some desing mistakes, because our hardware configuration is not very rare, so someone perhaps solve all our problems. We are interested in any feedback from the Beowulf community. -- Thanks for your attention, Alexandr Konovalov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Wed Jan 14 16:22:54 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed, 14 Jan 2004 13:22:54 -0800 Subject: [Beowulf] Market information on Beowulf clusters? In-Reply-To: <40058E75.7040409@tcd.ie> References: <40058E75.7040409@tcd.ie> Message-ID: <20040114212254.GA2127@greglaptop.internal.keyresearch.com> On Wed, Jan 14, 2004 at 06:46:13PM +0000, david moloney wrote: > Could anybody point me to public and/or commercial market info. on > Beowulf clusters especially: Two sources come to mind: 1) The Top500 list, which only has big clusters 2) Information from analysts like IDC For (2) you'll have to poke around; they usually charge $$, but they give out summaries for free sometimes. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jlb17 at duke.edu Wed Jan 14 16:29:28 2004 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed, 14 Jan 2004 16:29:28 -0500 (EST) Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: References: Message-ID: On Wed, 14 Jan 2004 at 11:01am, Joshua Baker-LePain wrote > I just moved two dual Xeon servers into a new home. One of them refuses > to see a network link, before or after the machine boots. The only > difference in them is a slight revision in the motherboard: the one that > links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't > has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel > 82546EB -- dual port gigabit). They're connected to a Cisco Catalyst > 4507R. > > The box *will* link when connected to a 100Mb switch. I also hooked it > up to a Dell 5224 (gigabit) switch that I have, and it found a link after > about 10 seconds. > > Any ideas? I've already tried flashing the BIOS and playing with mii-tool > a bit. Thanks to all who responded (on and off list). It turns out that the most likely culprit is a design flaw on certain SuperMicro dual Xeon boards that renders them unable to talk to some switches. Including Ciscos. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Wed Jan 14 16:51:08 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed, 14 Jan 2004 13:51:08 -0800 Subject: [Beowulf] Solaris on x86? In-Reply-To: References: Message-ID: <20040114215108.GA2292@greglaptop.internal.keyresearch.com> On Tue, Jan 13, 2004 at 08:40:34AM -0000, Cannon, Andrew wrote: > We are looking to get some x86 based compute power (possibly a number of > Athlon based machines) and I was wondering if anyone has any experience of > Solaris for x86, specifically, how well does it work in a cluster? Would we > be better looking at Red Hat PW or ES or some other linux distro. The > company is not too keen on going down the ROCKS etc route and would prefer a > "mainstream" solution. I see no one commented on this one. Solaris x86 is much less used for Linux clusters, with the main weaknesses being device driver support and compilers. It's certainly much less mainstream than RedHat Linux in this space, but Unix is Unix, and MPICH and most of our other favorite software runs on Solaris. As for ROCKS, it's using a pure-GPL version of RedHat Enterprise something or other. So it's less odd than you might think. You might want to explore what your company thinks "mainstream" means... I don't think I get the sense in which they use the word. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bioinformaticist at mn.rr.com Wed Jan 14 10:42:50 2004 From: bioinformaticist at mn.rr.com (Eric R Johnson) Date: Wed, 14 Jan 2004 09:42:50 -0600 Subject: [Beowulf] Schedulers for use with Scyld Version 28 Message-ID: <4005637A.2050908@mn.rr.com> I was wondering what the scheduler options are for me if I am running a Scyld version 28 machine. From everything I am finding, it appears that I only have a few choices. 1) at/bbq that ships with Scyld 2) PBS Pro 3) Clubmask/Maui I am running a very small cluster right now (16 processors) out of a University lab and PBS Pro is really out of our price range right now. I would like to add more nodes to the machine, but scheduling jobs is already a problem, so I don't think that bbq is the right choice. I have tried installing clubmask/Maui, but I ran into problems while compiling it. I was advised that it was due to the age of BProc that ships with Scyld. I was told that 3.2.5 is required. Can anybody give me some suggestions? Thanks, Eric -- ******************************************************************** Eric R A Johnson University Of Minnesota tel: (612) 626 5115 Dept. of Laboratory Medicine & Pathology fax: (612) 625 1121 7-230 BSBE e-mail: john4482 at umn.edu 312 Church Street web: www.eric-r-johnson.com Minneapolis, MN 55455 USA _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Wed Jan 14 09:42:56 2004 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Wed, 14 Jan 2004 09:42:56 -0500 Subject: [Beowulf] How to set up 2 Node Failover Cluster In-Reply-To: <1074078362.2246.10.camel@acernote.evonet.de> References: <1074078362.2246.10.camel@acernote.evonet.de> Message-ID: <20040114144255.GA15091@central.cis.upenn.edu> Just 2 nodes/servers, both are exactly the same but one is the "master" and the other is the "backup"? You're looking for a High Availability setup. Stop googling for Beowulf. Start searching for High Availability, and Failover. HTH, Dan W. > I'm told to set up a 2 Node Failover Linux Cluster. > Is ist possible to do this using beowulf? -- -- Daniel Widyono http://www.cis.upenn.edu/~widyono -- Liniac Project, CIS Dept., SEAS, University of Pennsylvania -- Mail: CIS Dept, 302 Levine 3330 Walnut St Philadelphia, PA 19104 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From avkon at imm.uran.ru Wed Jan 14 14:40:58 2004 From: avkon at imm.uran.ru (Alexandr Konovalov) Date: Thu, 15 Jan 2004 00:40:58 +0500 Subject: [Beowulf] Bonding in 2.4.22-1.2149 Linux kernel Message-ID: <40059B4A.6060701@imm.uran.ru> Hi, we have somewhat non-standart configuration of our computing cluster. Inernal computing nodes are connected by 2 (bonded) ethernet networks, but host connected only to one ethernet (i.e., connection to host not bonded for the all computing nodes). For the Linux kernel 2.4.9 - 2.4.20 it's possible to run such configuration if we use static arp tables and static routes (connection to host should be routed directly to the slave device, say eth1, not to bond0). But when we install 2.4.22-1.2149 (from Fedora 1), things are changed. Now routing to slave device became impossibe with diagnostics like ioctl(3, 0x890b, 0xbfe90b10) = -1 ENODEV (No such device) And there is no more slave devices in route output. Googling give us some reasons for this graceful changing of (yes, undocumented and unspecified) behaviour. It's compitely unacceptable for us to abandon bonding, so after drivers/net/bonding/bond_main.c reading we (probably) found a solution. Out idea is simply create new bonding policy. If a packet destination is in the bonded network it will call bond_xmit_roundrobin, if not then packed will be xmited directly to an appropriate slave device. MAC adresses for the direct xmiting will received by modiles as the load parameters. Probably we miss some opportunities and make some desing mistakes, because our hardware configuration is not very rare, so someone perhaps solve all our problems. We are interested in any feedback from the Beowulf community. -- Thanks for your attention, Alexandr Konovalov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Wed Jan 14 11:04:52 2004 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 14 Jan 2004 17:04:52 +0100 Subject: [Beowulf] linuxbios In-Reply-To: References: Message-ID: <1074096292.310.43.camel@revolution.mandrakesoft.com> > I'm mostly interested in Linuxbios though for the controlability of > large clusters. Can you develop that point ? What is really intresting in such technologie ? Once your bios are well setup, you don't need to change it every day.. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 14 20:36:16 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 14 Jan 2004 17:36:16 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) Message-ID: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> I'm building a cluster using wireless interconnects, and it is getting to be a royal pain to figure out what sort of adapters/interfaces to use. I'm casting some bread upon the waters here to see if anyone has fooled with this sort of thing. I want to go 802.11a because of spectrum usage, not because of data rate. If I go to 5GHz, I don't have to worry about accidentally connecting to JPL's 802.11b (2.4GHz) network infrstructure, for instance, which will keep netops off my back. The processors in the cluster are Mini-ITX widgets with Compact Flash (CF) drives, and, while booting off the net might be nice, I'm going to boot off CF for now. Here are some issues that have come up: 1) there's two ways to get the node to talk to the net: via the ethernet connector and an external bridge via a PCI card with a 802.11a adapter (most likely a 802.11a/b/g, since that's what's available) (D=Link, Netgear, and Linksys all have them) In all cases, I'd have an "access point" of some sort to talk to my head node/NFS, etc. Ideally, I'd like to set up the network in "ad-hoc" mode, where any node can talk to any other directly, without having to be routed through an access point. In "infrastructure" mode, many clients can talk to the access point, but clients cannot talk to clients, except by going through the access point, creating a single failure point (probably not important for my initial work, but philosophically "bad"). 2) It's unclear whether there are Linux drivers for any of the PCI based 802.11a cards. The mfrs don't seem to want to fool with that market, and, chipset mfrs are quite reticent about releasing the intellectual property needed to do a good job writing the drivers. 3) I could go with external bridging adapters (perhaps with integrated routers, in case I add another ethernet device to the node, or, to hook up a sniffer). Here the problem is that not all mfrs appear to support bridging, at least with more than 2 (i.e. they can set up a point to point bridge, but not a many to many bridge) From some reading of the various manuals, it appears that some "access points" can be set up to appear to be a "client" in infrastructure mode, however that's a problem philosophically (and in terms of latency). So, does anyone know which "access points" (i.e. a 802.11x to ethernet box) can look like a client in an ad-hoc network. (possible candidates: Netgear FWAG114, D-link DWL-774, DWL-7000AP, Linksys WAP54A*, WAP51AB, WRT51AB. *Linksys says that the WAP54A doesn't do bridging) Part 2 of the quest................. I'm also looking for suggestions on performance and timing tests to run on this cluster once it's assembled. Aside from the usual network throughput (benchmark program recommendations requested), I'm interested in techniques to look at latency, latency distribution, and dropped packets/retries, since I suspect that wireless networks will have very "unusual" statistics compared to the usual cluster interconnects. And, bearing in mind our recent lengthy thread on timing and clocks, you can be sure that I will do those sorts of tests too. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Wed Jan 14 22:08:23 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 15 Jan 2004 11:08:23 +0800 (CST) Subject: [Beowulf] Scalable PBS running at teragrid.org (900+ nodes)!! In-Reply-To: <020b01c3dae0$ea21a9b0$28b358c7@titan> Message-ID: <20040115030823.92280.qmail@web16806.mail.tpe.yahoo.com> >From the maintainer of xCAT: --- Egan Ford ???? > xCAT 1.2.0-pre4 has support for SPBS. There is an > SPBS maker and a newer Maui > Maker that works with 3.2.5 and 3.2.6. Other > scripts have been adjusted to > support SPBS and OpenPBS and the newer Maui's. > > We tested SPBS at scale at teragrid.org (900+ > nodes), but used early release Maui. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at iplink.net Wed Jan 14 23:07:37 2004 From: alvin at iplink.net (Alvin Starr) Date: Wed, 14 Jan 2004 23:07:37 -0500 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> Message-ID: <40061209.7060205@iplink.net> Jim Lux wrote: > I'm building a cluster using wireless interconnects, and it is getting > to be a royal pain to figure out what sort of adapters/interfaces to > use. I'm casting some bread upon the waters here to see if anyone has > fooled with this sort of thing. > > I want to go 802.11a because of spectrum usage, not because of data > rate. If I go to 5GHz, I don't have to worry about accidentally > connecting to JPL's 802.11b (2.4GHz) network infrstructure, for > instance, which will keep netops off my back. > > The processors in the cluster are Mini-ITX widgets with Compact Flash > (CF) drives, and, while booting off the net might be nice, I'm going to > boot off CF for now. > > Here are some issues that have come up: > 1) there's two ways to get the node to talk to the net: > via the ethernet connector and an external bridge > via a PCI card with a 802.11a adapter (most likely a 802.11a/b/g, since > that's what's available) (D=Link, Netgear, and Linksys all have them) take a look at the Actiontec cards. They have a line of prism based cards that are supported by linux and when I was talking with them they made sounds like they were going to stay with the prism chipset for a while. > > In all cases, I'd have an "access point" of some sort to talk to my head > node/NFS, etc. > > Ideally, I'd like to set up the network in "ad-hoc" mode, where any node > can talk to any other directly, without having to be routed through an > access point. In "infrastructure" mode, many clients can talk to the > access point, but clients cannot talk to clients, except by going > through the access point, creating a single failure point (probably not > important for my initial work, but philosophically "bad"). > > 2) It's unclear whether there are Linux drivers for any of the PCI > based 802.11a cards. The mfrs don't seem to want to fool with that > market, and, chipset mfrs are quite reticent about releasing the > intellectual property needed to do a good job writing the drivers. > > 3) I could go with external bridging adapters (perhaps with integrated > routers, in case I add another ethernet device to the node, or, to hook > up a sniffer). Here the problem is that not all mfrs appear to support > bridging, at least with more than 2 (i.e. they can set up a point to > point bridge, but not a many to many bridge) > > From some reading of the various manuals, it appears that some "access > points" can be set up to appear to be a "client" in infrastructure mode, > however that's a problem philosophically (and in terms of latency). > > So, does anyone know which "access points" (i.e. a 802.11x to ethernet > box) can look like a client in an ad-hoc network. > > (possible candidates: Netgear FWAG114, D-link DWL-774, DWL-7000AP, > Linksys WAP54A*, WAP51AB, WRT51AB. *Linksys says that the WAP54A > doesn't do bridging) > > > > > Part 2 of the quest................. > > I'm also looking for suggestions on performance and timing tests to run > on this cluster once it's assembled. Aside from the usual network > throughput (benchmark program recommendations requested), I'm interested > in techniques to look at latency, latency distribution, and dropped > packets/retries, since I suspect that wireless networks will have very > "unusual" statistics compared to the usual cluster interconnects. I think you will want to play with the payload size. It seems to make a big difference for what we are doing(net boot over wireless). > > And, bearing in mind our recent lengthy thread on timing and clocks, you > can be sure that I will do those sorts of tests too. > > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin at iplink.net || _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 00:05:24 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 14 Jan 2004 21:05:24 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <400606F4.3090802@iwantka.com> Message-ID: <001a01c3db26$bd9ed890$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Nathan Littlepage" To: "Jim Lux" Sent: Wednesday, January 14, 2004 7:20 PM Subject: Re: [Beowulf] Wireless clusters (Bridging 802.11a) > Jim, > > Just to add my two cents. Why not create your own AP with a linux > distribution thats available. There are many out there that can work on > hardware you already have. One that I have used, not in your situation, > but as an AP is from NYWireless called Pebble. > http://www.nywireless.net/pebble/. An added cent, I use ebtables to > screen MAC address to better secure access. > > Hope that helps in some manner. But, then, I'd still need some form of 802.11a wireless interface that's supported by Linux. I'd also have to spend some (scarce) time fooling with building APs. I'd rather have a basically off-the-shelf solution. I'm not real, real interested in understanding all the subtleties of 802.11 drivers under Linux. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 00:10:46 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 14 Jan 2004 21:10:46 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <40061209.7060205@iplink.net> Message-ID: <001b01c3db26$be0e6340$36a8a8c0@LAPTOP152422> > > take a look at the Actiontec cards. They have a line of prism based cards that are supported by linux and when I was > talking with them they made sounds like they were going to stay with the prism chipset for a while. While the prism chipset is interesting (for other reasons, particularly the Prism2, not so much 2.5 and 3 or even newer ones), they do not support the 802.11a at 5GHz form of wireless LAN. (for what it's worth, the prism set is interesting because it was originally a Harris product, then Intersil, and now GlobespanVirata. However, Harris has a long history of making military/space qualified components, and it turns out that the Prism 2 chip set is made on a fab line that is compatible with radiation tolerant processes) > > > > > In all cases, I'd have an "access point" of some sort to talk to my head > > node/NFS, etc. > > > > > Part 2 of the quest................. > > > > I'm also looking for suggestions on performance and timing tests to run > > on this cluster once it's assembled. Aside from the usual network > > throughput (benchmark program recommendations requested), I'm interested > > in techniques to look at latency, latency distribution, and dropped > > packets/retries, since I suspect that wireless networks will have very > > "unusual" statistics compared to the usual cluster interconnects. > > I think you will want to play with the payload size. It seems to make a big difference for what we are doing(net boot > over wireless). > > > Thanks.. I suspect that measurements vs packet size are of some interest. (something like HINT, except for size instead of computation). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xavier at zeth.ciencias.uchile.cl Thu Jan 15 00:00:49 2004 From: xavier at zeth.ciencias.uchile.cl (Xavier Andrade) Date: Thu, 15 Jan 2004 02:00:49 -0300 (CLST) Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <4005637A.2050908@mn.rr.com> References: <4005637A.2050908@mn.rr.com> Message-ID: On Wed, 14 Jan 2004, Eric R Johnson wrote: > I was wondering what the scheduler options are for me if I am running a > Scyld version 28 machine. > From everything I am finding, it appears that I only have a few choices. > 1) at/bbq that ships with Scyld > 2) PBS Pro > 3) Clubmask/Maui > > I am running a very small cluster right now (16 processors) out of a > University lab and PBS Pro is really out of our price range right now. > I would like to add more nodes to the machine, but scheduling jobs is > already a problem, so I don't think that bbq is the right choice. I > have tried installing clubmask/Maui, but I ran into problems while > compiling it. I was advised that it was due to the age of BProc that > ships with Scyld. I was told that 3.2.5 is required. > > Can anybody give me some suggestions? > Try sun gridengine ( http://gridengine.sunsource.net ). Xavier _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Jan 15 00:38:38 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 15 Jan 2004 13:38:38 +0800 (CST) Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <4005637A.2050908@mn.rr.com> Message-ID: <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> Depending on your requirements, you can choose either gridengine (SGE) or spbs. SGE is opensource, and is the most popular batch system. Just ask any questions on its mailing lists and you will get answers very quickly, the people on the lists are helpful and friendly. http://gridengine.sunsource.net/ SPBS is a version of OpenPBS developed by another group of developers. It may be easier for you if you know PBS already. ** And now SPBS is called StORM (scalable open resource manager) http://supercluster.org/projects/storm/ Andrew. --- Eric R Johnson ???? > I was wondering what the scheduler options are for > me if I am running a > Scyld version 28 machine. > From everything I am finding, it appears that I > only have a few choices. > 1) at/bbq that ships with Scyld > 2) PBS Pro > 3) Clubmask/Maui > > I am running a very small cluster right now (16 > processors) out of a > University lab and PBS Pro is really out of our > price range right now. > I would like to add more nodes to the machine, but > scheduling jobs is > already a problem, so I don't think that bbq is the > right choice. I > have tried installing clubmask/Maui, but I ran into > problems while > compiling it. I was advised that it was due to the > age of BProc that > ships with Scyld. I was told that 3.2.5 is > required. > > Can anybody give me some suggestions? > > Thanks, > Eric > > -- > ******************************************************************** > Eric R A Johnson > University Of Minnesota tel: > (612) 626 5115 > Dept. of Laboratory Medicine & Pathology fax: > (612) 625 1121 > 7-230 BSBE e-mail: > john4482 at umn.edu > 312 Church Street web: > www.eric-r-johnson.com > Minneapolis, MN 55455 > USA > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shaeffer at neuralscape.com Thu Jan 15 02:20:13 2004 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed, 14 Jan 2004 23:20:13 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) In-Reply-To: <001a01c3db26$bd9ed890$36a8a8c0@LAPTOP152422> References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <400606F4.3090802@iwantka.com> <001a01c3db26$bd9ed890$36a8a8c0@LAPTOP152422> Message-ID: <20040115072013.GA26730@synapse.neuralscape.com> On Wed, Jan 14, 2004 at 09:05:24PM -0800, Jim Lux wrote: > > ----- Original Message ----- > From: "Nathan Littlepage" > To: "Jim Lux" > Sent: Wednesday, January 14, 2004 7:20 PM > Subject: Re: [Beowulf] Wireless clusters (Bridging 802.11a) > > > > Jim, > > > > Just to add my two cents. Why not create your own AP with a linux > > distribution thats available. There are many out there that can work on > > hardware you already have. One that I have used, not in your situation, > > but as an AP is from NYWireless called Pebble. > > http://www.nywireless.net/pebble/. An added cent, I use ebtables to > > screen MAC address to better secure access. > > > > Hope that helps in some manner. > > But, then, I'd still need some form of 802.11a wireless interface that's > supported by Linux. I'd also have to spend some (scarce) time fooling with > building APs. I'd rather have a basically off-the-shelf solution. I'm not > real, real interested in understanding all the subtleties of 802.11 drivers > under Linux. Hello, I did fool around with Linux hosted 802.11b APs. It is straightforward. I use the hostap code from: http://hostap.epitest.fi/ works like a charm. It's actually fun stuff. You've pointed out that these prism based systems do not support 802.11a. Obviously a big problem for you. But, you may not be aware that the prism chips do support the full range of operating modes, including some obscure test modes that are not published. That hostap software supports many of them. Good luck, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer at neuralscape.com http://www.neuralscape.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From pesch at attglobal.net Thu Jan 15 13:01:13 2004 From: pesch at attglobal.net (pesch at attglobal.net) Date: Thu, 15 Jan 2004 10:01:13 -0800 Subject: [Beowulf] linuxbios References: Message-ID: <4006D569.5AC38A66@attglobal.net> The time is ripe for a "cluster bios" - it could make life a lot easier if it had all the right features. What would these features be in your opinion? Paul Schenker John Hearns wrote: > I've been following it for a couple of years. > > At the Linux Beer Hike in Ireland in 2002 we did some experiments, > and got it running. I've stilll got the PC-chips motherboard which > I bought to do the tests. Antony Stone found a UK supplier if the > disk-on-chip parts. > Antony did a very good article for the UK version Linux Magazine. > > I haven't had the courage though to try Linuxbios on my precious > mini-ITXes. > > I'm mostly interested in Linuxbios though for the controlability of > large clusters. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Thu Jan 15 07:27:21 2004 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Thu, 15 Jan 2004 13:27:21 +0100 Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> References: <4005637A.2050908@mn.rr.com> <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> Message-ID: <20040115132721U.hanzl@unknown-domain> > SGE is opensource, and is the most popular batch > system. Just ask any questions on its mailing lists > and you will get answers very quickly, the people on > the lists are helpful and friendly. > > http://gridengine.sunsource.net/ SGE is superb but did anybody actually make it to work with bproc-based cluster? Is there already something better than my hack: http://noel/magi/sge+bproc.html ? If yes, please let me know, I would be happy to document any success on this page. Regards Vaclav _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 15 07:58:24 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 15 Jan 2004 07:58:24 -0500 (EST) Subject: [Beowulf] Solaris on x86? In-Reply-To: <20040114215108.GA2292@greglaptop.internal.keyresearch.com> Message-ID: On Wed, 14 Jan 2004, Greg Lindahl wrote: > As for ROCKS, it's using a pure-GPL version of RedHat Enterprise > something or other. So it's less odd than you might think. You might > want to explore what your company thinks "mainstream" means... I don't > think I get the sense in which they use the word. I think it must be like the IBM ad with the spooky kid, the orphan that nobody owned, that was adopted by the world and put to all sorts of heavy lifting work (oh child labor laws, where are you now). Maybe they take the kid out in some kind of stream to drown him? Very confusing. [And what's wrong with penguins, I want to know... what's with this "kid" metaphor that looks like it belongs on the SciFi channel?] Seriously, linux is both mainstream and pretty all-around excellent in this (cluster/server) space. You could argue that it isn't mainstream on the office desktop or as a home computer (although the kid is inexorably moving out into that stream as well in many places) but in the server room linux has been a solid player for a long time and in clusters it is by far the OS of choice. I mean, it IS only the most stable, scalable and maintainable operating system the world has ever seen... rgb, trying his coffee withOUT sugar this morning Time to go take my, um, "vitamins". -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 15 08:48:49 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 15 Jan 2004 08:48:49 -0500 (EST) Subject: [Beowulf] benchmarking network and performance In-Reply-To: <001b01c3db26$be0e6340$36a8a8c0@LAPTOP152422> Message-ID: On Wed, 14 Jan 2004, Jim Lux wrote: > Thanks.. I suspect that measurements vs packet size are of some interest. > (something like HINT, except for size instead of computation). netpipe. netperf, too, but nobody loves it any more. netpipe will let you transparently test raw TCP, PVM and MPI connections side by side, for varying packet sizes, with streams of packets long enough to get decent statistics on them, and autogenerate a nice little report on same. Most of what you learn from such a test on ethernet is that small packets are latency bound and the interface sends as many of them per second as it can (given the latency) largely independent of packet size. Consequently bandwidth grows linearly with packet size. At some point the volume of data being transmitted STARTS to actually run into the wire bandwidth (if you like, the linear growth curve gets close to the wirespeed saturation/constant line) and the linear growth curve gently bends over and saturates sort-of-exponentially, typically at some 90% of wirespeed (for a decent interface), typically when packet sizes reach roughly 1K in size (maybe 2/3 of the standard MTU). Sometimes there is interesting "structure" in the curve -- "often" and "egregious" structure for poor interfaces, less often for good ones but sometimes you'll see something like dips or peaks around certain packet size boundaries. Oh, and lmbench has a variety of network latency and bw tests (tcp and udp and localhost) mixed in with the rest of its microbenchmarks. Maybe a bit clunky to use compared to netpipe but you wanted other benchmark tools as well and this is a suite of them. Finally, for other benchmark tools there is cpu_rate, which includes stream (with malloc and a sweep of vector sizes) as well as the savage benchmark and some other stream-like arithmetic tests. If your organization participates in SPEC, there is SPEC, and of course the list archives contain pointers to various parallel performance benchmarks. rgb > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Thu Jan 15 10:46:22 2004 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Thu, 15 Jan 2004 16:46:22 +0100 Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <20040115132721U.hanzl@unknown-domain> References: <4005637A.2050908@mn.rr.com> <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> <20040115132721U.hanzl@unknown-domain> Message-ID: <20040115164622K.hanzl@unknown-domain> > SGE is superb but did anybody actually make it to work with > bproc-based cluster? > > Is there already something better than my hack: > > http://noel/magi/sge+bproc.html > > ? The URL should have been: http://noel.feld.cvut.cz/magi/sge+bproc.html Sorry for my stupid mistake. Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 10:05:19 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 15 Jan 2004 07:05:19 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <40061209.7060205@iplink.net> <001b01c3db26$be0e6340$36a8a8c0@LAPTOP152422> <400698AF.4050409@iplink.net> Message-ID: <002901c3db79$431fd030$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "alvin" To: "Jim Lux" Cc: Sent: Thursday, January 15, 2004 5:42 AM Subject: Re: [Beowulf] Wireless clusters (Bridging 802.11a) > J>> > >> > >prism chipset for a while. > > > >While the prism chipset is interesting (for other reasons, particularly the > >Prism2, not so much 2.5 and 3 or even newer ones), they do not support the > >802.11a at 5GHz form of wireless LAN. (for what it's worth, the prism set is > >interesting because it was originally a Harris product, then Intersil, and > >now GlobespanVirata. However, Harris has a long history of making > >military/space qualified components, and it turns out that the Prism 2 chip > >set is made on a fab line that is compatible with radiation tolerant > >processes) > > > > > Just took a cruse through the GlobespanVirata site and it looks like > they have a new chipset that will do most of the various 802.11 protocols. > I wonder if there are linux drivers or programming information available > yet. I hope they have not gone the same way of the other chipset > manufacturers bykeeping the programming inerface propritary. You'll note that there are no datasheets on the Globespan Virata web site. Just block diagrams, and a page to contact marketing. Their latest offerings are highly integrated, and the details of the onchip processor doing the backend stuff are proprietary. > > > will you require a radation hardened product if you get to production? Probably not radiation hard, just radiation tolerant. TID of, say, 20-100 kRad, no latchup (or, some decent form of latchup mitigation). For now, though, this is just a proof of concept prototype to show that a wireless interconnect can work in this particular application. Down the road a bit, moderate integration RFICs (such as in the Prism II chipset) that are radiation tolerant would be of great interest (of course, by then, the masks will be 10 years old and you'll have to buy the fab line on E-bay.. such is life when designing flight hardware) > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 15 08:31:42 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 15 Jan 2004 08:31:42 -0500 (EST) Subject: [Beowulf] Wireless clusters (Bridging 802.11a) In-Reply-To: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 14 Jan 2004, Jim Lux wrote: > I'm building a cluster using wireless interconnects, and it is getting to > be a royal pain to figure out what sort of adapters/interfaces to use. I'm > casting some bread upon the waters here to see if anyone has fooled with > this sort of thing. > > I want to go 802.11a because of spectrum usage, not because of data > rate. If I go to 5GHz, I don't have to worry about accidentally connecting > to JPL's 802.11b (2.4GHz) network infrstructure, for instance, which will > keep netops off my back. I don't think I can help you with anything below, but I can point out an interesting and so far unexplained anomaly I've encountered with 802.11b. While working on a PVM article for CWM I often naturally end up sitting in my den downstairs working on my laptop where I can keep an eye on our neurotic and fey border collie puppy so he doesn't eat a table or rip a hole in a wall or something. That's fine, I've got switched 100BT systems a.k.a. "nodes" all over the house and when they aren't running Diablo II on top of winex they even have respectable performance. I can run PVM on the laptop just fine. I can login with ssh transparently (no password) to all the systems in the household just fine. I can login to one of those systems, crank PVM on it, and add the laptop (over the wireless connection) just fine. BUT, when I run PVM on the laptop and try to add the SAME host that worked fine the other way, I get: rgb at lilith|T:125>pvm pvm> add lucifer add lucifer libpvm [t40001]: pvm_addhosts(): Pvmd system error This is both not terribly informative and frustrating. PVM has an internal autodiagnostic mode these days that warns you of e.g. leftover lockfiles, inability to ssh and so forth -- this doesn't even make it that far. Running pvm -d0xffff (to get a debug trace) isn't much more helpful although it might be to somebody who really knows the PVM internals -- all the daemon's progress in setting up the connection proceeds just as it ought up to the final libpvm [t40001] mxinput() pkt src t80000000 len 36 ff 3 libpvm [t40001] mxinput() src t80000000 route t80000000 ctx 524286 tag TM_ADDHOST len 4 libpvm [t40001] mesg_input() src t80000000 ctx 524286 tag TM_ADDHOST len 4 libpvm [t40001] mxfer() txfp 0 gotem 1 tt_rxf 0 libpvm [t40001] msendrecv() from t80000000 tag TM_ADDHOST libpvm [t40001]: pvm_addhosts(): Pvmd system error and then it fails where in an identical trace going the other way it succeeds, no indication why in either case. At a guess it is either: a) I'm doing something stupid and silly somewhere. Always a possibility, although I >>have<< been using PVM since 1993 and generally can make it behave, especially since they added all the nifty diagnostics to it. b) Something "odd" about the wireless networking stack or the PCMCIA bus where (perhaps) PVM does something at a low level for speed/efficiency that just won't work with wireless. The latter is a bit disconcerting to me -- one would think that all transactions occur on top of virtual devices that keep wireless from being anything but "a network interface", although there COULD be either timeouts or packet reliability problems, of course. And the wireless network on the laptop works flawlessly otherwise as long as signal strength is adequate, and of course PVM works TO the wireless system but not FROM the wireless system. Anyway, I offer this up not because I want or need help with it (it would be silly to use my wireless laptop as a pvm master node in anything BUT writing an article or an EP task of some sort and there are other ways to accomplish both of these that are a lot less work than hacking the PVM sources to debug the failure) but because YOU might encounter similar problems and have to invest the extra time to debug the failure. Moral of story, don't assume that just because a wireless interface has device support under linux and works perfectly with simple network tests it will necessarily work with PVM, MPI, tools that access a network stack at a low level for "efficiency" reasons and presume e.g. ethernet. rgb > > The processors in the cluster are Mini-ITX widgets with Compact Flash (CF) > drives, and, while booting off the net might be nice, I'm going to boot off > CF for now. > > Here are some issues that have come up: > 1) there's two ways to get the node to talk to the net: > via the ethernet connector and an external bridge > via a PCI card with a 802.11a adapter (most likely a 802.11a/b/g, since > that's what's available) (D=Link, Netgear, and Linksys all have them) > > In all cases, I'd have an "access point" of some sort to talk to my head > node/NFS, etc. > > Ideally, I'd like to set up the network in "ad-hoc" mode, where any node > can talk to any other directly, without having to be routed through an > access point. In "infrastructure" mode, many clients can talk to the access > point, but clients cannot talk to clients, except by going through the > access point, creating a single failure point (probably not important for > my initial work, but philosophically "bad"). > > 2) It's unclear whether there are Linux drivers for any of the PCI based > 802.11a cards. The mfrs don't seem to want to fool with that market, and, > chipset mfrs are quite reticent about releasing the intellectual property > needed to do a good job writing the drivers. > > 3) I could go with external bridging adapters (perhaps with integrated > routers, in case I add another ethernet device to the node, or, to hook up > a sniffer). Here the problem is that not all mfrs appear to support > bridging, at least with more than 2 (i.e. they can set up a point to point > bridge, but not a many to many bridge) > > From some reading of the various manuals, it appears that some "access > points" can be set up to appear to be a "client" in infrastructure mode, > however that's a problem philosophically (and in terms of latency). > > So, does anyone know which "access points" (i.e. a 802.11x to ethernet box) > can look like a client in an ad-hoc network. > > (possible candidates: Netgear FWAG114, D-link DWL-774, DWL-7000AP, Linksys > WAP54A*, WAP51AB, WRT51AB. *Linksys says that the WAP54A doesn't do bridging) > > > > > Part 2 of the quest................. > > I'm also looking for suggestions on performance and timing tests to run on > this cluster once it's assembled. Aside from the usual network throughput > (benchmark program recommendations requested), I'm interested in techniques > to look at latency, latency distribution, and dropped packets/retries, > since I suspect that wireless networks will have very "unusual" statistics > compared to the usual cluster interconnects. > > And, bearing in mind our recent lengthy thread on timing and clocks, you > can be sure that I will do those sorts of tests too. > > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 10:13:13 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 15 Jan 2004 07:13:13 -0800 Subject: [Beowulf] Re: benchmarking network and performance References: Message-ID: <003101c3db7a$18cac1e0$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Robert G. Brown" To: "Jim Lux" Cc: "Alvin Starr" ; ; Sent: Thursday, January 15, 2004 5:48 AM Subject: benchmarking network and performance > On Wed, 14 Jan 2004, Jim Lux wrote: > > > Thanks.. I suspect that measurements vs packet size are of some interest. > > (something like HINT, except for size instead of computation). > > netpipe. netperf, too, but nobody loves it any more. netpipe will let > you transparently test raw TCP, PVM and MPI connections side by side, > for varying packet sizes, with streams of packets long enough to get > decent statistics on them, and autogenerate a nice little report on > same. > > Most of what you learn from such a test on ethernet is that small > packets are latency bound and the interface sends as many of them per > second as it can (given the latency) largely independent of packet size. > Consequently bandwidth grows linearly with packet size. At some point > the volume of data being transmitted STARTS to actually run into the > wire bandwidth (if you like, the linear growth curve gets close to the > wirespeed saturation/constant line) and the linear growth curve gently > bends over and saturates sort-of-exponentially, typically at some 90% of > wirespeed (for a decent interface), typically when packet sizes reach > roughly 1K in size (maybe 2/3 of the standard MTU). Of course, since it's a wireless channel, the "wire bandwidth" is somewhat ill defined, at least in terms of a classical shared media ethernet type situation. 802.11 is also half duplex, so even though the wirespeed (the over the air bit rate) is 11 Mbps, the actual through put could be quite a bit less, especially if it's an "ack/nak" kind of protocol being used in the transfer. UDP could be a LOT faster than TCP, for instance, because the UDP packets could just stream on out. > > Sometimes there is interesting "structure" in the curve -- "often" and > "egregious" structure for poor interfaces, less often for good ones but > sometimes you'll see something like dips or peaks around certain packet > size boundaries. > > Oh, and lmbench has a variety of network latency and bw tests (tcp and > udp and localhost) mixed in with the rest of its microbenchmarks. Maybe > Thanks for the suggestions Jim _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 15 16:01:22 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 15 Jan 2004 14:01:22 -0700 Subject: [Beowulf] linuxbios In-Reply-To: <1074096292.310.43.camel@revolution.mandrakesoft.com> References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: <4006FFA2.3080907@lanl.gov> Erwan Velu wrote: >>I'm mostly interested in Linuxbios though for the controlability of >>large clusters. > > Can you develop that point ? What is really intresting in such > technologie ? Once your bios are well setup, you don't need to change it > every day.. Setting up BIOS is a pain. Managing 100+ machines and making sure that they all run the same version, with the same options, and upgrading BIOS if it isn't quite right is *tons* of fun, particularly if you need to plug in a monitor and keyboard to do it. By contrast, LinuxBIOS gives you much better control (see http://www.linuxbios.org/). Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Thu Jan 15 15:52:57 2004 From: nixon at nsc.liu.se (Leif Nixon) Date: Thu, 15 Jan 2004 21:52:57 +0100 Subject: [Beowulf] linuxbios In-Reply-To: <1074096292.310.43.camel@revolution.mandrakesoft.com> (Erwan Velu's message of "Wed, 14 Jan 2004 17:04:52 +0100") References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: Erwan Velu writes: >> I'm mostly interested in Linuxbios though for the controlability of >> large clusters. > Can you develop that point ? What is really intresting in such > technologie ? Once your bios are well setup, you don't need to change it > every day.. Right now I'd like to evaluate the performance of a 100 node cluster with and without hyperthreading activated. I don't look forward to fiddle those BIOS settings, I can tell you. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From domingo at netcomp.com.br Thu Jan 15 08:45:50 2004 From: domingo at netcomp.com.br (Domingo Antonio) Date: Thu, 15 Jan 2004 11:45:50 -0200 Subject: [Beowulf] Re: MySQL clustering Message-ID: <001501c3db6d$e37db350$2101a8c0@blue> Hi all!! Is mysql capable to work with beowulf ? I?m intend to make a beowulf cluster to make my website more fast! Actualy a have an apache + mysql server, but they not suport requests... I tried, but no suces, to make mysql replication.. the performance was good but not good enoght... Is possible to make mysql work over beowulf? Thanks a lot!!! Domingo Antonio _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgoornaden at intnet.mu Thu Jan 15 23:12:04 2004 From: rgoornaden at intnet.mu (roudy) Date: Fri, 16 Jan 2004 08:12:04 +0400 Subject: [Beowulf] HPL References: <200401151701.i0FH1ph31020@NewBlue.scyld.com> Message-ID: <000a01c3dbe7$1358f070$b8097bca@roudy> Hi Can someone tell me what cause the installation process of HPL to stop, below is the copy of what has been done so far. Thanks Roudy [root at localhost hpl]# make arch=cluster make -f Make.top startup_dir arch=cluster make[1]: Entering directory `/home/hpl' mkdir include/cluster mkdir lib mkdir: cannot create directory `lib': File exists make[1]: [startup_dir] Error 1 (ignored) mkdir lib/cluster mkdir bin mkdir: cannot create directory `bin': File exists make[1]: [startup_dir] Error 1 (ignored) mkdir bin/cluster make[1]: Leaving directory `/home/hpl' make -f Make.top startup_src arch=cluster make[1]: Entering directory `/home/hpl' make -f Make.top leaf le=src/auxil arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/auxil ; mkdir cluster ) ( cd src/auxil/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/blas arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/blas ; mkdir cluster ) ( cd src/blas/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/comm arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/comm ; mkdir cluster ) ( cd src/comm/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/grid arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/grid ; mkdir cluster ) ( cd src/grid/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/panel arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/panel ; mkdir cluster ) ( cd src/panel/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/pauxil arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/pauxil ; mkdir cluster ) ( cd src/pauxil/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/pfact arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/pfact ; mkdir cluster ) ( cd src/pfact/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/pgesv arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/pgesv ; mkdir cluster ) ( cd src/pgesv/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make[1]: Leaving directory `/home/hpl' make -f Make.top startup_tst arch=cluster make[1]: Entering directory `/home/hpl' make -f Make.top leaf le=testing/matgen arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/matgen ; mkdir cluster ) ( cd testing/matgen/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/timer arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/timer ; mkdir cluster ) ( cd testing/timer/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/pmatgen arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/pmatgen ; mkdir cluster ) ( cd testing/pmatgen/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/ptimer arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/ptimer ; mkdir cluster ) ( cd testing/ptimer/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/ptest arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/ptest ; mkdir cluster ) ( cd testing/ptest/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_src arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.auxil src/auxil/cluster/Makefile cp makes/Make.blas src/blas/cluster/Makefile cp makes/Make.comm src/comm/cluster/Makefile cp makes/Make.grid src/grid/cluster/Makefile cp makes/Make.panel src/panel/cluster/Makefile cp makes/Make.pauxil src/pauxil/cluster/Makefile cp makes/Make.pfact src/pfact/cluster/Makefile cp makes/Make.pgesv src/pgesv/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_tst arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.matgen testing/matgen/cluster/Makefile cp makes/Make.timer testing/timer/cluster/Makefile cp makes/Make.pmatgen testing/pmatgen/cluster/Makefile cp makes/Make.ptimer testing/ptimer/cluster/Makefile cp makes/Make.ptest testing/ptest/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_src arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.auxil src/auxil/cluster/Makefile cp makes/Make.blas src/blas/cluster/Makefile cp makes/Make.comm src/comm/cluster/Makefile cp makes/Make.grid src/grid/cluster/Makefile cp makes/Make.panel src/panel/cluster/Makefile cp makes/Make.pauxil src/pauxil/cluster/Makefile cp makes/Make.pfact src/pfact/cluster/Makefile cp makes/Make.pgesv src/pgesv/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_tst arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.matgen testing/matgen/cluster/Makefile cp makes/Make.timer testing/timer/cluster/Makefile cp makes/Make.pmatgen testing/pmatgen/cluster/Makefile cp makes/Make.ptimer testing/ptimer/cluster/Makefile cp makes/Make.ptest testing/ptest/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top build_src arch=cluster make[1]: Entering directory `/home/hpl' ( cd src/auxil/cluster; make ) make[2]: Entering directory `/home/hpl/src/auxil/cluster' Makefile:47: Make.inc: No such file or directory make[2]: *** No rule to make target `Make.inc'. Stop. make[2]: Leaving directory `/home/hpl/src/auxil/cluster' make[1]: *** [build_src] Error 2 make[1]: Leaving directory `/home/hpl' make: *** [build] Error 2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From domingo2 at netcomp.com.br Wed Jan 14 19:07:14 2004 From: domingo2 at netcomp.com.br (Domingo Antonio) Date: Wed, 14 Jan 2004 22:07:14 -0200 Subject: [Beowulf] MySQL clustering Message-ID: <00df01c3dafb$88094060$2101a8c0@blue> Hi all!! Is mysql capable to work with beowulf ? I?m intend to make a beowulf cluster to make my website more fast! Actualy a have an apache + mysql server, but they not suport requests... I tried, but no suces, to make mysql replication.. the performance was good but not good enoght... Is possible to make mysql work over beowulf? Thanks a lot!!! Domingo Antonio -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars at meshtechnologies.com Fri Jan 16 01:48:20 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: 16 Jan 2004 06:48:20 +0000 Subject: [Beowulf] linuxbios In-Reply-To: References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: <1074235700.2279.2.camel@fermi> On Thu, 2004-01-15 at 20:52, Leif Nixon wrote: > Right now I'd like to evaluate the performance of a 100 node cluster > with and without hyperthreading activated. I don't look forward to > fiddle those BIOS settings, I can tell you. Well, the easy solution is to disable SMP support in the kernel. That can be done remote, unlike tedious BIOS config :-) best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Fri Jan 16 04:34:50 2004 From: nixon at nsc.liu.se (Leif Nixon) Date: Fri, 16 Jan 2004 10:34:50 +0100 Subject: [Beowulf] linuxbios In-Reply-To: <1074235700.2279.2.camel@fermi> (Lars Henriksen's message of "16 Jan 2004 06:48:20 +0000") References: <1074096292.310.43.camel@revolution.mandrakesoft.com> <1074235700.2279.2.camel@fermi> Message-ID: Lars Henriksen writes: > On Thu, 2004-01-15 at 20:52, Leif Nixon wrote: > >> Right now I'd like to evaluate the performance of a 100 node cluster >> with and without hyperthreading activated. I don't look forward to >> fiddle those BIOS settings, I can tell you. > > Well, the easy solution is to disable SMP support in the kernel. > That can be done remote, unlike tedious BIOS config :-) Not if the initial setting is hyperthreading deactivated. Which it is in my case. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Fri Jan 16 07:58:23 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: Fri, 16 Jan 2004 12:58:23 +0000 Subject: [Beowulf] linuxbios In-Reply-To: References: <1074096292.310.43.camel@revolution.mandrakesoft.com> <1074235700.2279.2.camel@fermi> Message-ID: <1074257903.4988.4.camel@tp1.mesh-hq> On Fri, 2004-01-16 at 09:34, Leif Nixon wrote: > >> Right now I'd like to evaluate the performance of a 100 node cluster > >> with and without hyperthreading activated. I don't look forward to > >> fiddle those BIOS settings, I can tell you. > > > > Well, the easy solution is to disable SMP support in the kernel. > > That can be done remote, unlike tedious BIOS config :-) > > Not if the initial setting is hyperthreading deactivated. Which it is > in my case. No, of course not, my bad. I assumed that it was enabled :-) In that case I can understand your point. best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5230 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Fri Jan 16 08:11:08 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Fri, 16 Jan 2004 05:11:08 -0800 (PST) Subject: [Beowulf] my beowulf problem Message-ID: <20040116131108.94673.qmail@web21506.mail.yahoo.com> hello friends i am very new at beowulf cluster i need your help i have tried following the initial advises but still unsuccesfull i get a following error i dont understand why is the lam environment req for mpirun it would be requ for lam/mpim implementataion [mpich at tung /]$ mpirun -np 4 cpilog ----------------------------------------------------------------------------- It seems that there is no lamd running on this host, which indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "mpirun" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. ----------------------------------------------------------------------------- __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 16 07:21:13 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 16 Jan 2004 07:21:13 -0500 (EST) Subject: [Beowulf] MySQL clustering In-Reply-To: <00df01c3dafb$88094060$2101a8c0@blue> Message-ID: On Wed, 14 Jan 2004, Domingo Antonio wrote: > Hi all!! > > Is mysql capable to work with beowulf ? > > I?m intend to make a beowulf cluster to make my website more fast! > > Actualy a have an apache + mysql server, but they not suport > requests... I tried, but no suces, to make mysql replication.. the > performance was good but not good enoght... > > Is possible to make mysql work over beowulf? To some extent yes, but this is the wrong list venue for the question. What you want is "high availability computing", not "high performance computing". There is a linux site devoted to it, and of course it is of great interest to and in significant use in corporate america. There is even a bit of crossover with this list -- distributing some sort of serious computation (as opposed to database transactions) front ended by a web interface, for example, but most of what HA linux focusses on is failover/reliability, delivery speed, handling lots of "transactions". BTW, two more comments. One is that in many cases web performance is at least partly bottlenecked by your net interface or I/O subsystem (disk), not the CPU. These are often difficult to share, and without sharing and parallelization there is little point in multiplexing your server arrangement and you'll do a lot of work for nothing. Multiplexing DB transactions IS one of the places where it can matter, if you can distribute multiple copies of the DB and keep them in sync as transactions proceed. A second comment is that on one or two of the lists that I'm on (not this one) I've heard significant rants against mysql as a professional quality DB. Note that this is NOT such a rant -- I use mysql myself for a small DB-driven php website, but note the key word: "small". Being a packrat, I squirrelled away the links. One is: http://openacs.org/philosophy/why-not-mysql.html and another is: http://sql-info.de/mysql/gotchas.html#1_1 A rant against mysql in general, and a fairly specific list of gotchas that have bitten other developers who looked for some specific SQL feature and found it lacking or broken. They tend to recommend using either postgres or oracle for anything more serious than alphabetizing your record collection (I use it to track campus clusters, which is just about that complex, and use SQL with a book in one hand anyway, so to me it matters little). Just FYI. rgb > > > Thanks a lot!!! > > Domingo Antonio -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wade.hampton at nsc1.net Fri Jan 16 08:17:21 2004 From: wade.hampton at nsc1.net (Wade Hampton) Date: Fri, 16 Jan 2004 08:17:21 -0500 Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: References: Message-ID: <4007E461.6050705@nsc1.net> Douglas Eadline, Cluster World Magazine wrote: >On Wed, 14 Jan 2004, Joshua Baker-LePain wrote: > > > >>I just moved two dual Xeon servers into a new home. One of them refuses >>to see a network link, before or after the machine boots. The only >>difference in them is a slight revision in the motherboard: the one that >>links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't >>has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel >>82546EB -- dual port gigabit). They're connected to a Cisco Catalyst >>4507R. >> >> > >You might first try looking for any messages in /var/log/messages >when you load the driver module. I assume you are using e1000.o. >Are the versions identical on both systems ? > >The output from things like > >uname -a >lsmod > >may be a good place to start. BTW, with Intel chips-sets showing link to a >switch does not necessarily imply that the kernel module is working. > >Doug > > > >>The box *will* link when connected to a 100Mb switch. I also hooked it >>up to a Dell 5224 (gigabit) switch that I have, and it found a link after >>about 10 seconds. >> >>Any ideas? I've already tried flashing the BIOS and playing with mii-tool >>a bit. >> >> I have several SuperMicro dual 2.2 xeon boxes and am having NIC problems with one as well. It is also an E7500 chipset with 82546EB NIC. The eth0 would not connect to a GB switch (don't have the make right now). Eth1worked fine so I had to use eth1 instead of eth0. When I get the box back next week, I'll find out if this was a hardware problem or similar to the one reported. Note: This box was borrowed from my cluster and is running Fedora Core 1.0 with the 2115 kernel and is somewhat unstable (several crashes) - yet another issue. Sorry I don't have more information at this time as the box is at another location. -- Wade Hampton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Fri Jan 16 08:10:13 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Fri, 16 Jan 2004 08:10:13 -0500 (EST) Subject: [Beowulf] linuxbios In-Reply-To: Message-ID: On Thu, 15 Jan 2004, Leif Nixon wrote: > Erwan Velu writes: > > >> I'm mostly interested in Linuxbios though for the controlability of > >> large clusters. > > Can you develop that point ? What is really intresting in such > > technologie ? Once your bios are well setup, you don't need to change it > > every day.. > > Right now I'd like to evaluate the performance of a 100 node cluster > with and without hyperthreading activated. I don't look forward to > fiddle those BIOS settings, I can tell you. Depending on what you want to do, you can probably get a "feel" for the HT vs no HT with a handful of machines or even singel machine. Basically, HT is not a win for cluster/parallel applications. Things can actually run slower. Doug > > -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine Desk: 610.865.6061 Cell: 610.390.7765 Redefining High Performance Computing Fax: 610.865.6618 www.clusterworld.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Fri Jan 16 08:16:48 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Fri, 16 Jan 2004 05:16:48 -0800 (PST) Subject: [Beowulf] do lam and mpich go together Message-ID: <20040116131648.80093.qmail@web21502.mail.yahoo.com> hello everybody can anybody help me to solve mt problem i have tried mpich-1.2.5.2 implimentation of mpi but i get the foll error for mpirun do we req lam environment for mpirun comand acc to my knowld it is req for lam/mpi implementation of mpi can any body send me the link or doc for perfect instalation of mpich-1.2.5.2 on cluster of 3 machine with red hat 9.0 [mpich at tung /]$ mpirun -np 4 cpilog ----------------------------------------------------------------------------- It seems that there is no lamd running on this host, which indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "mpirun" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. ----------------------------------------------------------------------------- __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmoyer at redhat.com Fri Jan 16 08:26:09 2004 From: jmoyer at redhat.com (Jeff Moyer) Date: Fri, 16 Jan 2004 08:26:09 -0500 Subject: [Beowulf] linuxbios In-Reply-To: References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: <16391.58993.771197.287433@segfault.boston.redhat.com> ==> Regarding Re: [Beowulf] linuxbios; Leif Nixon adds: nixon> Erwan Velu writes: >>> I'm mostly interested in Linuxbios though for the controlability of >>> large clusters. >> Can you develop that point ? What is really intresting in such >> technologie ? Once your bios are well setup, you don't need to change it >> every day.. nixon> Right now I'd like to evaluate the performance of a 100 node cluster nixon> with and without hyperthreading activated. I don't look forward to nixon> fiddle those BIOS settings, I can tell you. Well, once you've enabled HT in the BIOS, there's at leaset a kernel command line switch to disable it in the kernel (noht). -Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bkmacy at dali.ppco.com Fri Jan 16 09:58:59 2004 From: bkmacy at dali.ppco.com (Brian Macy) Date: Fri, 16 Jan 2004 08:58:59 -0600 Subject: [Beowulf] do lam and mpich go together References: <200401161329.i0GDTrh09638@NewBlue.scyld.com> Message-ID: <4007FC33.5050905@dali.ppco.com> beowulf-request at scyld.com wrote: >Message: 12 >Date: Fri, 16 Jan 2004 05:16:48 -0800 (PST) >From: prakash borade >To: beowulf at beowulf.org >Subject: [Beowulf] do lam and mpich go together > > >hello everybody > >can anybody help me to solve mt problem >i have tried mpich-1.2.5.2 implimentation of mpi > >but i get the foll error for mpirun >do we req lam environment for mpirun comand >acc to my knowld it is req for lam/mpi implementation >of mpi >can any body send me the link or doc for perfect >instalation of mpich-1.2.5.2 on cluster of 3 machine >with red hat 9.0 > > > >[mpich at tung /]$ mpirun -np 4 cpilog >----------------------------------------------------------------------------- >It seems that there is no lamd running on this host, >which indicates >that the LAM/MPI runtime environment is not operating. > The LAM/MPI >runtime environment is necessary for the "mpirun" >command. > >Please run the "lamboot" command the start the LAM/MPI >runtime >environment. See the LAM/MPI documentation for how to >invoke >"lamboot" across multiple machines. >----------------------------------------------------------------------------- > > > Prakash, It looks like you have LAM installed as well, and that you're picking up the mpirun from LAM instead of MPICH. Do a "which mpirun" to see where mpirun lives. You could provide the full path to mpirun for MPICH to ensure you're getting the right one; e.g., /usr/local/mpich-1.2.5.2/bin/mpirun -np 4 cpilog HTH, Brian _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From luis.licon at cimav.edu.mx Fri Jan 16 11:42:48 2004 From: luis.licon at cimav.edu.mx (luis.licon at cimav.edu.mx) Date: Fri, 16 Jan 2004 09:42:48 -0700 (MST) Subject: [Beowulf] Re: MySQL clustering In-Reply-To: <001501c3db6d$e37db350$2101a8c0@blue> References: <001501c3db6d$e37db350$2101a8c0@blue> Message-ID: <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> You have two options, the first one is DNS Round Robin, and the second one is Load Balance, the second one has a better performance indeed, you could try this option with PEN, and you can get it on: http://siag.nu/pen Cheerz, Luis > Hi all!! > > Is mysql capable to work with beowulf ? > I?m intend to make a beowulf cluster to make my website more fast! > Actualy a have an apache + mysql server, but they not suport > requests... I tried, but no suces, to make mysql replication.. the > performance was good but not good enoght... > > Is possible to make mysql work over beowulf? > > > Thanks a lot!!! > > Domingo Antonio > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Fri Jan 16 11:43:33 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Fri, 16 Jan 2004 16:43:33 +0000 (GMT) Subject: [Beowulf] MySQL clustering In-Reply-To: <200401161332.i0GDWfh10058@NewBlue.scyld.com> References: <200401161332.i0GDWfh10058@NewBlue.scyld.com> Message-ID: >quality DB. Note that this is NOT such a rant -- I use mysql myself for >a small DB-driven php website, but note the key word: "small". It depends on your value of small: df -k /mysql* Filesystem Size Used Available Capacity Mounted on mysqla#mysqla 209715200 136332008 73363504 66% /mysqla mysqlb#mysqlb 209715200 192189552 17504864 92% /mysqlb mysqlc#mysqlc 209715200 127955469 81740544 62% /mysqlc mysqld#mysqld 209715200 171582111 38133089 82% /mysqld mysqle#mysqle 209715200 100388182 109311880 48% /mysqle mysqlf#mysqlf 209715200 182784033 26910192 88% /mysqlf Mysql4 may not be fully Ansi SQL compliant, but it is fast, which if you have 1000 nodes of cluster blatting results back into it, may be a more critical requirement. Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Fri Jan 16 13:51:40 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Fri, 16 Jan 2004 10:51:40 -0800 (PST) Subject: [Beowulf] need MPI Programs Message-ID: <20040116185140.61764.qmail@web12708.mail.yahoo.com> hello all, trying to learn LAM/MPI for my final year project. I've gone through the tutorials found on the net and the MPI:Complete Reference book(which i obtained online through the beowulf mailing list.thanks) Anyway, I would be grateful if anyone would share their MPI programs with me, so that I can run/learn/test them on my cluster. Any help would be appreciated. Thanks, Toufeeq ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From atp at piskorski.com Fri Jan 16 14:38:20 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Fri, 16 Jan 2004 14:38:20 -0500 Subject: [Beowulf] MySQL clustering In-Reply-To: <200401161701.i0GH1Sh06443@NewBlue.scyld.com> References: <200401161701.i0GH1Sh06443@NewBlue.scyld.com> Message-ID: <20040116193820.GA63814@piskorski.com> > Date: Fri, 16 Jan 2004 16:43:33 +0000 (GMT) > From: Guy Coates > >quality DB. Note that this is NOT such a rant -- I use mysql myself for > >a small DB-driven php website, but note the key word: "small". > Mysql4 may not be fully Ansi SQL compliant, but it is fast, which if you > have 1000 nodes of cluster blatting results back into it, may be a more > critical requirement. >From what I've read, MySQL is only fast under conditions of no or low concurrency. Get up above 5 or so concurrent users, and performance starts going into the toilet very fast due to table lock contention. Note that 5 concurrent users is a trivial number for any large public website. SourceForge switched from MySQL to PostgreSQL a while back (two years ago?) for exactly that reason, and had some nice little articles about it at the time, showing the scaling graphs of both databases running the actual SourceForge software, etc. Better databases (Oracle, PostgreSQL, Firebird, DB2, Sybase, MS SQL Server, etc. etc.), particularly those using MVCC, will scale well to much higher numbers of concurrent users. If you don't need ACIDity, much concurrency, or powerful "real RDBMS" style features, then MySQL may be a good choice. And there are plenty of applications that either fit that, or are maybe close enough. Of course I might argue that even for most of those applications, there are likely still better choices than MySQL, but as I've never actually built such a maybe-MySQL-would-be-ok app, I've never had any reason to make a real investigation into the different options there. [shrug] YMMV. -- Andrew Piskorski http://www.piskorski.com/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From vallard at CSUA.Berkeley.EDU Fri Jan 16 15:16:45 2004 From: vallard at CSUA.Berkeley.EDU (Vallard Benincosa) Date: Fri, 16 Jan 2004 12:16:45 -0800 (PST) Subject: [Beowulf] HPL Message-ID: <20040116121329.I88241-100000@soda.csua.berkeley.edu> Just had this same problem. HPL uses a symbolic link for your $HOME directory while it's building. So since you are root, your $HOME is /root/. If you extract the tarball in /root/ instead of /home/ then it will work. Alternatively, you can take out references of $HOME in the Make.arch file. Vallard _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brbarret at lam-mpi.org Fri Jan 16 18:49:57 2004 From: brbarret at lam-mpi.org (Brian Barrett) Date: Fri, 16 Jan 2004 15:49:57 -0800 Subject: [Beowulf] my beowulf problem In-Reply-To: <20040116131108.94673.qmail@web21506.mail.yahoo.com> References: <20040116131108.94673.qmail@web21506.mail.yahoo.com> Message-ID: On Jan 16, 2004, at 5:11 AM, prakash borade wrote: > hello friends > i am very new at beowulf cluster > > i get a following error > > i dont understand why is the lam environment req for > mpirun > > it would be requ for lam/mpim implementataion It looks like the mpirun that appears in your path first is the mpirun from LAM/MPI. Both MPICH and LAM/MPI provide an "mpirun" (as well as mpicc, mpiCC, and mpif77). For LAM/MPI to operate, you must "lamboot" first, starting a small daemon on each node. You might want to take a look at our user manual: http://www.lam-mpi.org/using/docs/ As for why you are using LAM/MPI as opposed to MPICH (which is what I'm assuming you meant to ask), it's probably because LAM/MPI is installed by default on most Red Hat and Mandrake Linux installations. Hope this helps, Brian -- Brian Barrett LAM/MPI developer and all around nice guy Have a LAM/MPI day: http://www.lam-mpi.org/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Fri Jan 16 01:12:02 2004 From: gerry.creager at tamu.edu (Gerry Creager (N5JXS)) Date: Fri, 16 Jan 2004 00:12:02 -0600 Subject: [Beowulf] Looking for WRF/MM5 modellers Message-ID: <400780B2.5080908@tamu.edu> I've some questions for folks who've recently built clusters for WRF and MM5 model work. Off-list responses would probably be better. Thanks! gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Office: 979.458.4020 FAX: 979.847.8578 Cell: 979.229.5301 Pager: 979.228.0173 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mperez at delta.ft.uam.es Fri Jan 16 07:11:21 2004 From: mperez at delta.ft.uam.es (Manuel J) Date: Fri, 16 Jan 2004 13:11:21 +0100 Subject: [Beowulf] Mac address collector v 1.1 Message-ID: <200401161311.21575.mperez@delta.ft.uam.es> Dear all, I am happy to announce the first version of the new mac-1.1. Mac stands for "MAC address collector", and it is a tool to automathically collect Mac addresses of a set of computers. For more information about mac and its usage, please check its man page. Mac is part of the Quattor project ( www.quattor.org ). Quattor is a system administration toolkit providing a powerful, portable and modular toolsuite for the automated installation, configuration and management of clusters and farms running UNIX derivates like Linux and Solaris. You can download mac-1.1 from http://datagrid.in2p3.fr/cvsweb/edg-quattor/util/mac/mac.cin and rename it as mac. Any comments, suggestions and critics are wellcome!! Best Regards. Manuel J. Perez Rubio Universidad Autonoma Madrid. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ken_i_m at fatair.net Fri Jan 16 23:07:59 2004 From: ken_i_m at fatair.net (ken_i_m at fatair.net) Date: Fri, 16 Jan 2004 21:07:59 -0700 Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: <4007E461.6050705@nsc1.net> References: <4007E461.6050705@nsc1.net> Message-ID: <20040117040759.GH3362@hyper> On Fri, Jan 16, 2004 at 08:17:21AM -0500, Wade Hampton (wade.hampton at nsc1.net) wrote: > >>(Intel 82546EB -- dual port gigabit). They're connected to a Cisco > >>Catalyst 4507R. Spent an hour with a brand new HP DL380 last week that would not talk to the network dispite dmesg show both ports were detected. Once I plugged working cables into both they started to work (instead of just the one I usually use on the workbench). Remove one and networking dies... Not nice. -- Ken Dyke "Bits at the speed of light" _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From whately at lcp.coppe.ufrj.br Fri Jan 16 14:30:30 2004 From: whately at lcp.coppe.ufrj.br (Lauro Whately) Date: Fri, 16 Jan 2004 17:30:30 -0200 Subject: [Beowulf] Re: MySQL clustering In-Reply-To: <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> References: <001501c3db6d$e37db350$2101a8c0@blue> <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> Message-ID: <40083BD6.5070904@lcp.coppe.ufrj.br> Domingo, take a look in the Linux Virtual Server project (www.linuxvirtualserver.org) or try ultramonkey (www.ultramonkey.org) Lauro. luis.licon at cimav.edu.mx wrote: >You have two options, the first one is DNS Round Robin, and the second one >is Load Balance, the second one has a better performance indeed, you could >try this option with PEN, and you can get it on: http://siag.nu/pen > >Cheerz, > >Luis > > >>Hi all!! >> >> Is mysql capable to work with beowulf ? >> I?m intend to make a beowulf cluster to make my website more fast! >> Actualy a have an apache + mysql server, but they not suport >>requests... I tried, but no suces, to make mysql replication.. the >>performance was good but not good enoght... >> >> Is possible to make mysql work over beowulf? >> >> >>Thanks a lot!!! >> >>Domingo Antonio >> >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >>http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >> > > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jaimeco at dyer.rc.arizona.edu Fri Jan 16 18:02:39 2004 From: jaimeco at dyer.rc.arizona.edu (jaimeco) Date: Fri, 16 Jan 2004 16:02:39 -0700 (MST) Subject: [Beowulf] enforce resources_max.nodes Message-ID: <200401162302.i0GN2dA9002931@dyer.rc.arizona.edu> We have a cluster of alpha machines and we are running openpbs. We have created a queue with a max number of 32 cpus. We want these jobs to run on aura2 and aura4 as can be seen below. openpbs is not enforcing this unless the name of the node is used (-l nodes=aura2:ppn=32). We would like users to be more generic and use -l nodes=1:ppn=32 so that openpbs knows that this queue should be assigned to nodes 2 and 4 and will enforce it. It is not happening and jobs will usually go to aura0 --if there are enough cpus there. Does anyone know of a way to enforce this or have any experience with this? I appreciate your responses. Thanks create queue aura_32p set queue aura_32p queue_type = Execution set queue aura_32p Priority = 100 set queue aura_32p max_running = 4 set queue aura_32p acl_user_enable = True set queue aura_32p resources_max.ncpus = 32 set queue aura_32p resources_max.nodes = aura2+aura4:ppn=32 set queue aura_32p resources_default.ncpus = 16 set queue aura_32p resources_available.ncpus = 128 set queue aura_32p max_user_run = 1 set queue aura_32p enabled = True set queue aura_32p started = True --- Jaime E. Combariza Manager, Research Computing CCIT, University of Arizona (520) 626-4050 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tmerritt at email.arizona.edu Fri Jan 16 14:55:16 2004 From: tmerritt at email.arizona.edu (Todd Merritt) Date: Fri, 16 Jan 2004 12:55:16 -0700 Subject: [Beowulf] MySQL clustering In-Reply-To: <20040116193820.GA63814@piskorski.com> References: <200401161701.i0GH1Sh06443@NewBlue.scyld.com> <20040116193820.GA63814@piskorski.com> Message-ID: <1074282915.23849.50.camel@hive.CCIT.Arizona.EDU> On Fri, 2004-01-16 at 12:38, Andrew Piskorski wrote: > From what I've read, MySQL is only fast under conditions of no or low > concurrency. Get up above 5 or so concurrent users, and performance > starts going into the toilet very fast due to table lock contention. > Note that 5 concurrent users is a trivial number for any large public > website. > That may be true of multiple concurrent writes, but for mostly read access, mysql performance is stellar. We use it as the preference backend for our webmail system, as well as maintaining user mappings for postfix for ~50,000 email accounts. At peak times, there are ~1200 simultaneous webmail users and mysql has never been a bottleneck here. Todd _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Sat Jan 17 01:28:27 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Fri, 16 Jan 2004 22:28:27 -0800 (PST) Subject: [Beowulf] NEDD OPENPBS HELP In-Reply-To: <200401162302.i0GN2dA9002931@dyer.rc.arizona.edu> Message-ID: <20040117062827.10830.qmail@web21502.mail.yahoo.com> hello i am trying openpbs for the first time . iam not able to get the pbs admin_guid pdf file ( it cannot be downloaded from their page) i don't know how to make my openpbs know all my nodes it does'nt lists all my nodes in the cluster __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Sat Jan 17 07:12:53 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Sat, 17 Jan 2004 04:12:53 -0800 (PST) Subject: [Beowulf] need MPI Programs In-Reply-To: <20040117083752.83681.qmail@web12706.mail.yahoo.com> Message-ID: <20040117121253.92292.qmail@web21509.mail.yahoo.com> hello thanks for your help i have found out the bug and unistalled lam an now trying the mpirun from mpich but still un able to include my all machines though i have made entries in machine.LINUX file i get something like this about the programs yes we have lots of codes to be written usin mpi but before that i must have my clkuster working we'll work together to help each other thanks in advance [mpich]#mpirun -np 2 cpi p0_21223: (62.933408) Procgroup: p0_21223: (62.933510) entry 0: tung.unipune.ernet.in 0 0 /home/mpich/cpi mpich p0_21223: (62.933539) entry 1: dyp.unipune.ernet.in 1 1 /home/mpich/cpi mpich p0_21223: p4_error: Could not gethostbyname for host dyp.unipune.ernet.in; may be invalid name __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sat Jan 17 08:33:07 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 17 Jan 2004 21:33:07 +0800 (CST) Subject: [Beowulf] NEDD OPENPBS HELP In-Reply-To: <20040117062827.10830.qmail@web21502.mail.yahoo.com> Message-ID: <20040117133307.2013.qmail@web16807.mail.tpe.yahoo.com> OpenPBS is dead, a lot of the bugs are not fixed. If you go to the "OpenPBS vs. PBS Pro Comparison" page on www.openpbs.org, it says: "OpenPBS works up to 32 nodes with 100s of jobs, but doesn't scale well above that due to fault tolerance problems. PBS Pro is in production use at sites with thousands of CPUS, 30,000+ jobs." You should use Scalable PBS (aka storm), it is actually the opensource version OpenPBS but with lots of fixes and new features. You can download it here: http://supercluster.org/downloads/spbs/ In storm-1.0.1p3.tar.gz, you will get admin_guide.ps in the doc/ directory. Use "ps2pdf" if you don't have the ps reader. Also, Gridengine is another good batch system and with even more features. They have the pre-compiled binaries available so you don't need to go through the source compile process, which sometimes is a pain to get all the dependencies. You can also give it a try: http://gridengine.sunsource.net/ Andrew. --- prakash borade ????> > hello > i am trying openpbs for the first > time . iam not able to get the pbs admin_guid pdf > file ( it cannot be downloaded from their page) > > i don't know how to make my openpbs know all my > nodes > it does'nt lists all my nodes in the cluster > > > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" > Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nathan at iwantka.com Sat Jan 17 14:04:32 2004 From: nathan at iwantka.com (Nathan Littlepage) Date: Sat, 17 Jan 2004 13:04:32 -0600 Subject: [Beowulf] Re: MySQL clustering In-Reply-To: <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> Message-ID: <00f201c3dd2c$cd2bdfd0$6c45a8c0@ntbrt.bigrivertelephone.com> To add an additional option that would work in conjunction with the other two. Utilize the replication ability of mySQL. http://www.mysql.com/doc/en/Replication.html > -----Original Message----- > From: beowulf-admin at scyld.com > [mailto:beowulf-admin at scyld.com] On Behalf Of luis.licon at cimav.edu.mx > Sent: Friday, January 16, 2004 10:43 AM > To: Domingo Antonio > Cc: beowulf at beowulf.org > Subject: Re: [Beowulf] Re: MySQL clustering > > > You have two options, the first one is DNS Round Robin, and > the second one > is Load Balance, the second one has a better performance > indeed, you could > try this option with PEN, and you can get it on: http://siag.nu/pen > > Cheerz, > > Luis > > Hi all!! > > > > Is mysql capable to work with beowulf ? > > I?m intend to make a beowulf cluster to make my > website more fast! > > Actualy a have an apache + mysql server, but they not suport > > requests... I tried, but no suces, to make mysql replication.. the > > performance was good but not good enoght... > > > > Is possible to make mysql work over beowulf? > > > > > > Thanks a lot!!! > > > > Domingo Antonio > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From meetsunil80x86 at yahoo.co.in Sat Jan 17 22:36:48 2004 From: meetsunil80x86 at yahoo.co.in (=?iso-8859-1?q?sunil=20kumar?=) Date: Sun, 18 Jan 2004 03:36:48 +0000 (GMT) Subject: [Beowulf] MPI Message-ID: <20040118033648.79360.qmail@web8307.mail.in.yahoo.com> Hello everybody, I am a newbie in MPI programming. Please help me. truely, sunil ________________________________________________________________________ Yahoo! India Mobile: Download the latest polyphonic ringtones. Go to http://in.mobile.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 00:59:22 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 06:59:22 +0100 Subject: [Beowulf] Selection of Beowulf cluster type Message-ID: <400B723A.4070701@telia.com> Hi, I am going to set up my first beowulf cluster and I have no experience of this. The cluster will be built of two dual CPU workstations. Box 1 MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe 4 x 512 MB DDR SDRAM PC2100 Box 2. Intel model SE7505VB2 ATX PCI, FSB 533MHz 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB 2 x 1 GB DDR SDRAM PC2100 Reg ECC The cluster is intended for numerical simulations (FEA) with ABAQUS Please advise the type of cluster you think I shall build and what package to use. For the time being I am considering following solutions: - Warewulf 1.16 (http://www.warewulf-cluster.org) - OSCAR 3.0 ( http://oscar.openclustergroup.org/tiki-index.php) Yours faithfully Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Mon Jan 19 01:42:27 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Sun, 18 Jan 2004 22:42:27 -0800 (PST) Subject: [Beowulf] can't include all machines in my beowulf Message-ID: <20040119064227.51163.qmail@web21503.mail.yahoo.com> i was able to run cpi program on the single machine with only master included in machines.LINUX but after i have made entries of my all nodes in machines.LINUX file i get the following error i have properly configure rsh also i have mounted my mpich installed directory on all machines using NFS and all uses the same path [mpich]#mpirun -np 2 cpi > p0_21223: (62.933408) Procgroup: > p0_21223: (62.933510) entry 0: > tung.unipune.ernet.in 0 0 /home/mpich/cpi mpich > p0_21223: (62.933539) entry 1: > dyp.unipune.ernet.in 1 1 /home/mpich/cpi mpich > p0_21223: p4_error: Could not gethostbyname for > host > dyp.unipune.ernet.in; may be invalid name > __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Jan 19 08:54:55 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 19 Jan 2004 08:54:55 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400B723A.4070701@telia.com> Message-ID: On Mon, 19 Jan 2004, Per Lindstrom wrote: > Hi, > > I am going to set up my first beowulf cluster and I have no experience > of this. > > The cluster will be built of two dual CPU workstations. > > Box 1 > MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) > 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe > 4 x 512 MB DDR SDRAM PC2100 > > Box 2. > Intel model SE7505VB2 ATX PCI, FSB 533MHz > 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB > 2 x 1 GB DDR SDRAM PC2100 Reg ECC > > The cluster is intended for numerical simulations (FEA) with ABAQUS > > Please advise the type of cluster you think I shall build and what > package to use. > > For the time being I am considering following solutions: > > - Warewulf 1.16 (http://www.warewulf-cluster.org) > > - OSCAR 3.0 ( http://oscar.openclustergroup.org/tiki-index.php) You sound like you are on a good track. I've never tried warewulf but expect that it is fine. OSCAR is very popular and seems to be very easy for a novice to get going with. rgb > > Yours faithfully > Per Lindstr?m > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Mon Jan 19 12:28:55 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: 19 Jan 2004 17:28:55 +0000 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400B723A.4070701@telia.com> References: <400B723A.4070701@telia.com> Message-ID: <1074533335.2278.10.camel@fermi> On Mon, 2004-01-19 at 05:59, Per Lindstrom wrote: > Box 1 > MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) > 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe > 4 x 512 MB DDR SDRAM PC2100 > > Box 2. > Intel model SE7505VB2 ATX PCI, FSB 533MHz > 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB > 2 x 1 GB DDR SDRAM PC2100 Reg ECC > > The cluster is intended for numerical simulations (FEA) with ABAQUS First off: I do not really know ABAQUS, but i assume, that like most FE solvers, it is based on a MPI parallelization. By nature it has a strictly parallel execution flow. This would probably mean that the speed of which the execution takes place (if the problem is indeed CPU bound), is dictated by the slowest CPU in the system (beeing the 1,13 GHz P3) - You might get better performance with running on just the dual Xeon. It can also be stated as follows: Does the program execute faster with 4 x P3 1,13 GHz than on 2 x 2,8GHz Xeons communicating through memory? Though the 4 CPU setup might give better cache utilization... I'm not saying that you shouldn't build a cluster, but i i'm very interested in your findings in this case. Please correct me if i'm rambling here :-) best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 12:07:07 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 18:07:07 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <1074533335.2278.10.camel@fermi> References: <400B723A.4070701@telia.com> <1074533335.2278.10.camel@fermi> Message-ID: <400C0EBB.8030905@telia.com> Lars Henriksen wrote: >On Mon, 2004-01-19 at 05:59, Per Lindstrom wrote: > > >>Box 1 >>MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) >>2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe >>4 x 512 MB DDR SDRAM PC2100 >> >>Box 2. >>Intel model SE7505VB2 ATX PCI, FSB 533MHz >>2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB >>2 x 1 GB DDR SDRAM PC2100 Reg ECC >> >>The cluster is intended for numerical simulations (FEA) with ABAQUS >> >> > >First off: I do not really know ABAQUS, but i assume, that like most FE >solvers, it is based on a MPI parallelization. By nature it has a >strictly parallel execution flow. This would probably mean that the >speed of which the execution takes place (if the problem is indeed CPU >bound), is dictated by the slowest CPU in the system (beeing the 1,13 >GHz P3) - You might get better performance with running on just the dual >Xeon. It can also be stated as follows: Does the program execute faster >with 4 x P3 1,13 GHz than on 2 x 2,8GHz Xeons communicating through >memory? > >Though the 4 CPU setup might give better cache utilization... > >I'm not saying that you shouldn't build a cluster, but i i'm very >interested in your findings in this case. > >Please correct me if i'm rambling here :-) > >best regards >Lars > > Tanks for your thoughts Lars, :-) You are most probably right about that the Xeon machine alone will be the fastest solution, I will do some test and let you know the result and the problem used. The main plan with the cluster exercise is, in the first order, to gain know how and experience of building, managing and solving FEA (ABAQUS) problems on a cluster and/or MPI network. If everything vent well and it feels comfortable, the next step will be to replace the Tualatin machine with an identical Xeon machine i.e. giving a 4 x 2,8 GHz setup. (i.e. applied cash flow management ;-) ) Best regards Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 12:35:41 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 18:35:41 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <187D3A7CAB42A54DB61F1D05F012572201D4C05A@orsmsx402.jf.intel.com> References: <187D3A7CAB42A54DB61F1D05F012572201D4C05A@orsmsx402.jf.intel.com> Message-ID: <400C156D.90303@telia.com> Lombard, David N wrote: >From: Per Lindstrom; Sunday, January 18, 2004 9:59 PM > > >>Hi, >> >>I am going to set up my first beowulf cluster and I have no experience >>of this. >> >>The cluster will be built of two dual CPU workstations. >> >>Box 1 >>MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) >>2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe >>4 x 512 MB DDR SDRAM PC2100 >> >>Box 2. >>Intel model SE7505VB2 ATX PCI, FSB 533MHz >>2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB >>2 x 1 GB DDR SDRAM PC2100 Reg ECC >> >>The cluster is intended for numerical simulations (FEA) with ABAQUS >> >> > >The most important factor is how ABAQUS runs your jobs on the two >systems individually and on the cluster as a whole. > >If ABAQUS doesn't do any sort of load balancing, e.g., it simply cuts >the problem in half (or quarters), you will likely see the effect that >the problem solution time is controlled (limited) by the speed of the >slower system. > >You also haven't discussed the available disk (storage). I do not know >what the disk requirements of ABAQUS are. If it has significant disk >requirements, e.g., disk bandwidth, the disks will also affect the >solution time. I don't expect disk latency to be an issue with any FEA >app, and presume you have sufficiently large disks. > >Finally, the network may have some impact when running a job across the >nodes. > >Bottom line. Once you get ABAQUS running, you will likely benefit from >analyzing how well it runs in serial, SMP, and DMP modes on your >cluster. For example, you may find the "best" use is to run the cluster >in throughput (running one job per node, possibly under a queuing >system), rather than parallel performance mode (running one job across >both nodes). I put best in quotes because that quality depends on your >needs. > > > Thanks for your advices David, All the best Per Lindstro"m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 12:43:26 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 18:43:26 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400C173E.1040803@telia.com> Robert G. Brown wrote: >On Mon, 19 Jan 2004, Per Lindstrom wrote: > > > >>Hi, >> >>I am going to set up my first beowulf cluster and I have no experience >>of this. >> >>The cluster will be built of two dual CPU workstations. >> >>Box 1 >>MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) >>2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe >>4 x 512 MB DDR SDRAM PC2100 >> >>Box 2. >>Intel model SE7505VB2 ATX PCI, FSB 533MHz >>2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB >>2 x 1 GB DDR SDRAM PC2100 Reg ECC >> >>The cluster is intended for numerical simulations (FEA) with ABAQUS >> >>Please advise the type of cluster you think I shall build and what >>package to use. >> >>For the time being I am considering following solutions: >> >>- Warewulf 1.16 (http://www.warewulf-cluster.org) >> >>- OSCAR 3.0 ( http://oscar.openclustergroup.org/tiki-index.php) >> >> > >You sound like you are on a good track. I've never tried warewulf but >expect that it is fine. OSCAR is very popular and seems to be very easy >for a novice to get going with. > > rgb > > > >>Yours faithfully >>Per Lindstr?m >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> > > > I will most probably go for the Warewulf as it should be possible to use the Linux distro I'm used to (Slackware) and I think that is quite important as it allways cost time to change "environment". Or is that a blind trace? Best regards Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Mon Jan 19 12:09:18 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Mon, 19 Jan 2004 09:09:18 -0800 Subject: [Beowulf] Selection of Beowulf cluster type Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C05A@orsmsx402.jf.intel.com> From: Per Lindstrom; Sunday, January 18, 2004 9:59 PM > > Hi, > > I am going to set up my first beowulf cluster and I have no experience > of this. > > The cluster will be built of two dual CPU workstations. > > Box 1 > MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) > 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe > 4 x 512 MB DDR SDRAM PC2100 > > Box 2. > Intel model SE7505VB2 ATX PCI, FSB 533MHz > 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB > 2 x 1 GB DDR SDRAM PC2100 Reg ECC > > The cluster is intended for numerical simulations (FEA) with ABAQUS The most important factor is how ABAQUS runs your jobs on the two systems individually and on the cluster as a whole. If ABAQUS doesn't do any sort of load balancing, e.g., it simply cuts the problem in half (or quarters), you will likely see the effect that the problem solution time is controlled (limited) by the speed of the slower system. You also haven't discussed the available disk (storage). I do not know what the disk requirements of ABAQUS are. If it has significant disk requirements, e.g., disk bandwidth, the disks will also affect the solution time. I don't expect disk latency to be an issue with any FEA app, and presume you have sufficiently large disks. Finally, the network may have some impact when running a job across the nodes. Bottom line. Once you get ABAQUS running, you will likely benefit from analyzing how well it runs in serial, SMP, and DMP modes on your cluster. For example, you may find the "best" use is to run the cluster in throughput (running one job per node, possibly under a queuing system), rather than parallel performance mode (running one job across both nodes). I put best in quotes because that quality depends on your needs. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 15:46:27 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 21:46:27 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400C4223.7050103@telia.com> Robert G. Brown wrote: >On Mon, 19 Jan 2004, Per Lindstrom wrote: > > > >>I will most probably go for the Warewulf as it should be possible to use >>the Linux distro I'm used to (Slackware) and I think that is quite >>important as it allways cost time to change "environment". >> >>Or is that a blind trace? >> >> > >Speaking personally as a person who started with slackware (a fairly >long time ago) and moved on, I think that you should STRONGLY consider >moving to an RPM-based distribution. Red Hat, or in the future Fedora >(for example) installed with kickstart via PXE and maintained with yum >approach the theoretical limit in operating system scalability. As in, >to install a system one turns it on. To update a system one does >nothing (a nightly cron script does it for you). To install or browse >new software from an RPM repository one uses a single command with a >very efficient interface. > >Debian also scales much better than slackware. Slackware was OK for >single systems or small networks, but it won't do for hundreds of LAN >boxes and beowulf nodes per administrator. RPM-based distros will. > > rgb > > > >>Best regards >>Per Lindstr?m >> >> >> > > > It gives that I should jump from Slack to Fedora or Debian, which distro is scaling best? Best regards Per Lindstrom _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Jan 19 15:27:26 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 19 Jan 2004 15:27:26 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400C173E.1040803@telia.com> Message-ID: On Mon, 19 Jan 2004, Per Lindstrom wrote: > I will most probably go for the Warewulf as it should be possible to use > the Linux distro I'm used to (Slackware) and I think that is quite > important as it allways cost time to change "environment". > > Or is that a blind trace? Speaking personally as a person who started with slackware (a fairly long time ago) and moved on, I think that you should STRONGLY consider moving to an RPM-based distribution. Red Hat, or in the future Fedora (for example) installed with kickstart via PXE and maintained with yum approach the theoretical limit in operating system scalability. As in, to install a system one turns it on. To update a system one does nothing (a nightly cron script does it for you). To install or browse new software from an RPM repository one uses a single command with a very efficient interface. Debian also scales much better than slackware. Slackware was OK for single systems or small networks, but it won't do for hundreds of LAN boxes and beowulf nodes per administrator. RPM-based distros will. rgb > > Best regards > Per Lindstr?m > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Jan 19 17:44:57 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 19 Jan 2004 17:44:57 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400C4223.7050103@telia.com> Message-ID: On Mon, 19 Jan 2004, Per Lindstrom wrote: > It gives that I should jump from Slack to Fedora or Debian, which distro > is scaling best? Ah, the kind of question that can start a religious war. I will say that I personally prefer RH (and soon expect to prefer Fedora) because kickstart + PXE + yum is pretty incredible. However, Debianites feel that apt and friends are even better (or at least just as good). On this list there is at least one Mandrake-person who will point out that Mandrake is a good, RPM-based distribution (so yum will work) with its own RPM manipulation tools and its own cluster distro. Finally -- well, probably not finally;-) -- there is SuSE, used by a lot of Europe and reasonably popular over here where yum also should work. There. That's very ecumenical. My advise is to look over the distributions with regard to cost, how current they stay, and what other tools you want to use expect. OSCAR, for example, installs on top of one of the RH revisions, so if you go that way you'll probably want to go RH/Fedora. rgb > > Best regards > Per Lindstrom > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at comcast.net Mon Jan 19 19:00:00 2004 From: laytonjb at comcast.net (Jeffrey B. Layton) Date: Mon, 19 Jan 2004 19:00:00 -0500 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400C6F80.6060600@comcast.net> Robert G. Brown wrote: >On Mon, 19 Jan 2004, Per Lindstrom wrote: > > > >>It gives that I should jump from Slack to Fedora or Debian, which distro >>is scaling best? >> >> > >Ah, the kind of question that can start a religious war. > That's true! Although sometimes I get a little devious and like to start them (fun to watch the opinions come out). If you're goint to use Warewulf, you'll need a rpm-based distrbution. Something like RH, Mandrake, Suse, cAos, Whitebox, Fedora, etc. Warewulf gives you all of the details about PXE (don't need kickstart since there's nothing on the disk). I think yum has been added to cAos, but in general yum is pretty cool. As for performance, I didn't know one was better than the other (never heard of the problems with slackware before). Oh, I think you can add rpm capability to slack anyway (anyone comment on this?). Good Luck! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 00:50:22 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Mon, 19 Jan 2004 21:50:22 -0800 (PST) Subject: [Beowulf] can't include all machines in my beowulf In-Reply-To: Message-ID: <20040120055022.18675.qmail@web21506.mail.yahoo.com> hey tahnks for your advice i was able to find the bug thank you very much do you know what is MPE and what role does it play in mpich where cpi don't require it but cpilog requires it __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 01:54:49 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Mon, 19 Jan 2004 22:54:49 -0800 (PST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) Message-ID: <20040120065449.33171.qmail@web21507.mail.yahoo.com> hey can anybody tell me what do i choose for programing either GNU c or JAVA for implementing some networkprograms using some linux systemcalls __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Tue Jan 20 04:56:47 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Tue, 20 Jan 2004 10:56:47 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400C6F80.6060600@comcast.net> References: <400C6F80.6060600@comcast.net> Message-ID: <400CFB5F.4010104@telia.com> Jeffrey B. Layton wrote: > Robert G. Brown wrote: > >> On Mon, 19 Jan 2004, Per Lindstrom wrote: >> >> >> >>> It gives that I should jump from Slack to Fedora or Debian, which >>> distro is scaling best? >>> >> >> Ah, the kind of question that can start a religious war. >> > > That's true! Although sometimes I get a little devious and like to > start them (fun to watch the opinions come out). > > If you're goint to use Warewulf, you'll need a rpm-based distrbution. > Something like RH, Mandrake, Suse, cAos, Whitebox, Fedora, etc. > Warewulf gives you all of the details about PXE (don't need > kickstart since there's nothing on the disk). I think yum has been > added to cAos, but in general yum is pretty cool. > > As for performance, I didn't know one was better than the other > (never heard of the problems with slackware before). Oh, I think > you can add rpm capability to slack anyway (anyone comment on > this?). > ---------------------------- Throwing in the statement that Slackware has a bad scalability to a Slack fundamentalist forum, should indeed start a religious war. ; ) Here I am not running for Slack for its own sake so I will use Slacware up to two (2) nodes and when shifting to Debian when I'm connecting the remaining nodes. I think it will stop at about 10 CPUs. Until I have learned to manage Debian I will use the "slack-rpm-transfer-tool" and it will be very interesting to benchmark the two distros with each other. Thanks a lot for all help and support in his case. Best regards Per Lindstrom _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 06:59:19 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Tue, 20 Jan 2004 03:59:19 -0800 (PST) Subject: [Beowulf] can i increase priority of process on beowulf with mpich Message-ID: <20040120115919.1940.qmail@web21502.mail.yahoo.com> hello proksy here can we change the priority of the process on the cluser using mpich because i have seen that when i run the process as a root with priority -10 it is maintained only on master how can i maintain same priority on all machines __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 06:58:34 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Tue, 20 Jan 2004 03:58:34 -0800 (PST) Subject: [Beowulf] can i increase priority of process on beowulf with mpich Message-ID: <20040120115834.71935.qmail@web21505.mail.yahoo.com> hello proksy here can we change the priority of the process on the cluser using mpich because i have seen that when i run the process as a root with priority -10 it is maintained only on master how can i maintain same priority on all machines __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Tue Jan 20 08:05:30 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: Tue, 20 Jan 2004 13:05:30 +0000 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400CFB5F.4010104@telia.com> References: <400C6F80.6060600@comcast.net> <400CFB5F.4010104@telia.com> Message-ID: <1074603930.2417.8.camel@tp1.mesh-hq> On Tue, 2004-01-20 at 09:56, Per Lindstrom wrote: > Here I am not running for Slack for its own sake so I will use Slacware > up to two (2) nodes and when shifting to Debian when I'm connecting the > remaining nodes. I think it will stop at about 10 CPUs. > > Until I have learned to manage Debian I will use the > "slack-rpm-transfer-tool" and it will be very interesting to benchmark > the two distros with each other. If you want to use Debian as the base distribution, I can recommend SytemImager as an installation tool. http://www.systemimager.org/ Sytemimager allows you to replicate an installed client via PXE/rsync to the remaining nodes. It also allows for updates to be distributed via rsync. A feature that i really like, is that it allows for the use of several different images. Also it scales pretty well (I had the pleasure of using it in a couple of 16-32 node configurations, as well as on our own cluster) IMO it is the easiest way to deal with smaller clusters (except maybe CLIC) best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5230 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 20 08:05:45 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 20 Jan 2004 08:05:45 -0500 (EST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120065449.33171.qmail@web21507.mail.yahoo.com> Message-ID: On Mon, 19 Jan 2004, prakash borade wrote: > hey > > can anybody tell me > what do i choose for programing > either GNU c or JAVA for implementing some > networkprograms using some linux systemcalls Gads, that's two questions in two days with the distinct whiff of crowds of screaming peasants waving swords and torches... Let it be Jihad, then. The answer technically is "it depends". This is to make the vast horde of java programmers feel loved, and to acknowledge that for writing web applications their primary scripting language (like perl, python and many another before it) has a purpose in God's world. There. Let us bask for a moment in the serenity of our knowledge that we have the complete freedom to choose, and that there are no wrong answers. Now we can give the correct answer, which is "C". In order to give it, I have to infer an implicit "better than" and a certain degree of generality in your question is in "which language is BETTER suited to write an efficient networking program using linux systemscalls, in GENERAL". With this qualifier, C wins hands down. A variety of reasons: a) The operating system is written in C (plus a bit of assembler) b) Nearly all network daemons and utilities and program clients are written in C or perhaps an extension of C such as C++ c) Nearly all decent books on network programming (e.g. Stevens) provide excellent C templates for doing lots of network-based things d) You can do "anything" with C plus (in a few, very rare cases, a bit of inlined assembler) e) C is a compiler and produces (in the hands of a good programmer) code that runs very efficiently. Java is an interpreter and does not. f) C is an open ANSI standard implemented in linux with fully open source tools with no restrictions or covenants on their usage and with no "vendors" in any sort of position of control over the language itself or its core libraries. Java was invented by Sun, and Sun (AFAIK) still controls the distribution of its core libraries, and as an interpreted language, those libraries are sine qua non for portability. Sun is relatively linux friendly, but I personally tend not to trust giant multibillion dollar companies that own core components of something I need to rely on (or incorporate those components into my code), as they have an annoying tendency to turn around and want money for it. g) Or find themselves imbroiled in a corporate war with someboy ELSE with deep pockets advancing a different "standard"; for example I'd guess that Microsoft would like nothing better than to co-opt Java for themselves and turn around and want money for it. C is co-opt proof, and GCC is a highly reliable platform upon which to base code in the certain knowledge that the core compiler and libraries will remain royalty free. h) ...and I could go on. Note that I do not address java's OO nature, as OO C variants exist AND as OO programming is largely a matter of programming style, not language per se anyway. To conclude with a more ecumenical note because it DOES depend, I should point out that my acquaintances who code in java seem to be able to write fairly complex structured programs with the language, and I'd guess that most posix systems calls are wrapped for use within a java program as they are in perl and python and etc (I could be wrong, as java has "sandbox" issues that might well block certain systems calls from being run at all in which case this is another argument IN GENERAL for C, although there may well be SPECIFIC cases where it is an argument in favor of Java). Note also that my diatribe above is clearly biased as I don't code in Java, largely become of all of the reasons above (and then some). Speed, power, portability, open standard and source, and available code base and learning support all argue in favor of C about 10 to 1. UNLESS you are trying to write a powerful web-distributed application, which is a PITA in C and what Java was designed for. Books with good to excellent support for network programming in C: W. Richard Stevens, Networking God (deceased): http://www.kohala.com/start/ (pick almost any title, but I like "Unix Network Programming"). Even if you code in Java, you'll likely want this as I'd bet that Java just wraps up the C systems calls in interpreted tokens. Unix for Programmers and Users, Graham Glass (the current edition is with King Ables as well, but the older edition is just fine). Various websites dedicated to network programming, with examples. That's what Google is for -- I have the contents of several squirrelled away already for myself, but the web is amorphous and changes often, so you'll need to do your own search. Any of a LOT of GPL programs that can serve as code templates to get you started. xmlsysd/wulfstat, for example, contain templated code for a network daemon that can run either in forking mode or as an xinetd-managed daemon plus a client that connects to the daemon. Glass and Stevens both have related code snippets (in fact, guess where my code is derived from). But there are also a zillion other daemons in any linux distro, most of them GPL and readily available in source. I think you'd have a hard time finding 1/10 as many ready-made examples (especially ones with highly complex functionality) in Java. LOL, rgb P.S. -- I >>tried<< to be mild, but I suppose I'll need to spend the rest of the day in a bunker wearing my lead underwear and a tinfoil skullcap. Really, I love YAPL's as they pop up from year to year. I do. Marshmellows...ready! Flame on... -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Jan 20 07:45:20 2004 From: nixon at nsc.liu.se (Leif Nixon) Date: Tue, 20 Jan 2004 13:45:20 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <1074603930.2417.8.camel@tp1.mesh-hq> (Lars Henriksen's message of "Tue, 20 Jan 2004 13:05:30 +0000") References: <400C6F80.6060600@comcast.net> <400CFB5F.4010104@telia.com> <1074603930.2417.8.camel@tp1.mesh-hq> Message-ID: Lars Henriksen writes: > On Tue, 2004-01-20 at 09:56, Per Lindstrom wrote: >> Here I am not running for Slack for its own sake so I will use Slacware >> up to two (2) nodes and when shifting to Debian when I'm connecting the >> remaining nodes. I think it will stop at about 10 CPUs. >> >> Until I have learned to manage Debian I will use the >> "slack-rpm-transfer-tool" and it will be very interesting to benchmark >> the two distros with each other. > > If you want to use Debian as the base distribution, I can recommend > SytemImager as an installation tool. > > http://www.systemimager.org/ Systemimager is not Debian specific. We use it together with Red Hat. Very nice indeed. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 20 09:06:55 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 20 Jan 2004 09:06:55 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400CFB5F.4010104@telia.com> Message-ID: On Tue, 20 Jan 2004, Per Lindstrom wrote: > Until I have learned to manage Debian I will use the > "slack-rpm-transfer-tool" and it will be very interesting to benchmark > the two distros with each other. > > Thanks a lot for all help and support in his case. Just to be clear, the issue is not "benchmarking two distros" for performance. It is strictly manageability and scalability. Suppose I want to (re)install a system upstairs in my house. I walk up to the system and reboot it (power cycling is OK). When it boots, a PXE prompt pops up. I tell it to boot "linux-install". The system issues a query to determine who it is (gets an IP number and various other data from a DHCP server), retrieves a kernel, and boots into a network-based install. The install is fully scripted -- it will even build/rebuild an MD RAID for me. Every package I have listed for the host is installed, its keyboard, mouse, monitor, time base, and more is set up, and in the post install routine all required account support (/etc/passwd, NFS mounts, printers, whatever) are set up. At the end the system reboots itself into production. On a fast network connection and with a fast repository server, the time required for me to fully (re)install a system can be as little as two or three minutes, at most perhaps thirty minutes (where I need be in attendance only for the first thirty seconds). In fact, I don't NEED to be in attendance. By running a script, I can tell grub on an installed system to boot linux-install by default and reboot; linux-install will end by putting a new grub in place to go back to the normal default kernel. Once installed, provided that I installed from a yummified server and installed the yum package (appropriately configured for my network and repository) the system will wake up sometime in the night, query the server over anonymous http or ftp to see if there are any packages in the repository that are more recent (have a more advanced revision number) than what is installed. If there are, it grabs the RPMs and retrieves them (plus any needed dependencies) and updates the system's installation. If I'm managing five systems or five thousand systems, to keep ALL OF THEM CURRENT within 24 hours to the latest patches and package updates requires only that I drop an RPM onto the repository and do nothing. I don't even need root privileges on all those derived clients -- yum is a client-pull tool, not a root-push tool. So if RH announces a security bug in apache, what do I care? I take the patched RPM (generally released at the same time as the bug announcement -- security bugs are not tolerated for long in linux) and drop it into my repository and by the next morning every webserver in an INSTITUTION scale linux network is once again secure, even if some of the individuals running linux-based webservers are complete idiots who don't have the foggiest clue about the need to keep the institution's network secure by means of regular system maintenance. The institution still needs a genuine clueful expert, but it only needs ONE. yum has lots of other lovely features. yum will list all packages in all repositories a system is configured to use, broken down into "installed" and "available" lists. yum will install anything that is installed, and uninstall anything that you want to remove, AND will augment or prune the dependency tree just the right amount in either case. yum always asks before doing things (unless you tell it not to in e.g. a script). yum was designed by systems adminstrators for systems administrators, so it is feature rich precisely where sysadmins need it to be and conservative in its design throughout. Some sites that maintain repositories for particular GPL tools have started to "yummify" those repositories so that you can literally get updates hot off the primary site from any network-connected system. Note that if you replace the "kickstart" step above with whatever primary installation mode an RPM-based distro favors, the other RPM-based distros are equally yummifiable. I happen to like kickstart because I've used it enough to learn it and because it is sufficiently feature rich that 95% or better of all systems can be templated to be installed without human hands (99% or better if you sensibly choose the hardware and put sane restrictions on the software) but yum actually CREATES space for new installation methodology. For example, one hassle with kickstart is that if you run it over a slow connection, e.g. DSL, it takes forever to install several GB worth of software and if you interrupt it at any point it has to start over. With yum, you can install an ABSOLUTELY MINIMAL system that takes only a few minutes to install even over DSL, and then "finish" the system with yum. Unlike kickstart, yum is perfectly happy to be interrupted and will just start where it left off until it finishes installing a given list of packages. THIS is where slackware failed -- it is package based, but its packages tended to lag, in some cases significantly, the more aggressively maintained commercial distros (as did/do Debian's packages, actually). The thing that originally recommended it was that it could be installed from floppy disks in "sets", which is precisely why I started with it back in the mid-90's. It is still distributed that way, for reasons that are now somewhat difficult to fathom. Its tgz packages don't manage dependencies and dependency issues as well as .deb or .rpm packages, and it is very, very easy for systems on a slackware-based LAN to "drift" from their installed configuration and become heterogeneous and difficult to update and maintain. Package rebuilding is also much more of an issue. It just takes more human TIME to install and maintain EACH SYSTEM on a LAN using slackware than it does using an RPM-based distro or Debian. As far as performance goes, well, performance is generally all but identical across linuces because they all tend to use the same code base EXCEPT that the most aggressively maintained distros will typically have a small edge because their underlying packages are more up to date and better maintained. A distro with an openly out of date gcc/glibc might underperform one with gcc/glibc up to date (with e.g. the newer SSE instuctions supported), for example. So for security AND performance, one typically wants an aggressively maintained distribution, yet one that doesn't advance so rapidly that it thereby becomes unstable because it lives at the bleeding edge. The major commercial linuces try to tread precisely this line, and that is in fact the primary added value in what they sell. The free linuces have to face the same issue. Debian recognizes this internally with stable and unstable versions, where unstable likely has better performance but requires a certain degree of expertise to be comfortable with. I suspect that the fedora project is now RH's "RH unstable" version, as they will no longer be able to tolerate any significant degree of instability in their moderately expensive commercial release. Caosity and a few others will likely closely track Fedora in level of relative (in)stability. I don't know what slackware does currently -- because of its relatively high degree of aggregation and relatively weak dependency resolution, I'm guessing that it tends to be very slowly maintained and advanced -- each release is stable and is advanced only very incrementally with bugfixes only until the next release, because of the difficulty in rebuilding just a single node and its dependencies without unexpected side effects. In the old days this was a plague -- every time I'd add something new to a slackware box and build it, if it required newer libraries than slackware had installed it was a nightmare to rebuild everything required to make the new package happy. So the REAL issue is not performance. It is human time, specifically your time. If you do your job well, you'll end up with a network that damn near takes care of itself, where you do work once and it is transparently replicated many times throughout your LAN. Therefore pick a distro that you are reasonably certain: a) Has an aggressive package update schedule, especially wrt security updates but also performance. b) Has intelligent package dependency resolution (in my opinion, this is rpm-based or debian). c) Similarly, makes it easy to rebuild packages from source. d) Has intelligent package management and maintenance tools, e.g. apt, yum, urpmi. e) Supports the notion of a "package repository", ideally local repositories that you can maintain yourself and that can provide high bandwidth controlled load services to your clients WITHIN your LAN. f) Supports PXE-driven fully automated installation of one sort or another. g) Has some sort of bug-feedback-resolution mechanism, e.g. bugzilla. h) Is aggressively open source and charges FAIR PRICES per client, if it charges at all. Alas, IMHO a number of the commercial distros are smelling large profits and are starting to charge like Microsoft for a product they don't even own. You get what you pay for, sure, but be sure you pay for the VALUE of what you get relative to the absolutely free linux distros if you go for any of the distros that seek to charge you per seat. i) Is used by enough people that there exists a good online support forum. Good "enough" for your level of expertise, really -- I don't mean to discourage people, especially competent system programmer people, from choosing caosity or fedora just because they are so new they don't have as many adopters yet as commercial distros or debian. Linux is the quintessential "community" product; we all own it, and we all are responsible for making it work. Adopt one tiny bit of it and help out, if you can. There are probably more features I should add, but I've got to move on. Busy list today, and also a busy teaching day for rgb (starting in one hour). rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From award at correu.andorra.ad Tue Jan 20 09:16:19 2004 From: award at correu.andorra.ad (Alan Ward i Koeck) Date: Tue, 20 Jan 2004 15:16:19 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) References: Message-ID: <400D3833.F1D97ABB@correu.andorra.ad> Well, my own preference goes to Python, which - has a syntax I prefer a lot over C - while being an interpreted language, manages to be reasonably speedy, unlike Java - has a fair amount of predefined library objects you can use to access C POSIX system routines but I do recognize that low-level stuff definately should be coded in C. Now, as to just what "low-level" means ... Best regards, Alan Ward "Robert G. Brown" wrote: (snip) > > Let it be Jihad, then. > > The answer technically is "it depends". This is to make the vast horde > of java programmers feel loved, and to acknowledge that for writing web > applications their primary scripting language (like perl, python and > many another before it) has a purpose in God's world. > > There. Let us bask for a moment in the serenity of our knowledge that > we have the complete freedom to choose, and that there are no wrong > answers. > > Now we can give the correct answer, which is "C". (snip) > -- > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gropp at mcs.anl.gov Tue Jan 20 08:47:02 2004 From: gropp at mcs.anl.gov (William Gropp) Date: Tue, 20 Jan 2004 07:47:02 -0600 Subject: [Beowulf] can't include all machines in my beowulf In-Reply-To: <20040120055022.18675.qmail@web21506.mail.yahoo.com> References: <20040120055022.18675.qmail@web21506.mail.yahoo.com> Message-ID: <6.0.0.22.2.20040120073814.02186048@localhost> At 11:50 PM 1/19/2004, prakash borade wrote: >hey tahnks for your advice >i was able to find the bug > >thank you very much > >do you know what is MPE and what role does it play in >mpich > where cpi don't require it >but cpilog requires it cpilog uses the logging features provided by the MPE (MultiProcessing Extensions) library, which is included with MPICH and works with all MPI implementations. cpi is a basic MPI program and can be built with any MPI implementation; cpilog makes calls to the routines in the MPE library to log MPI and other operations. Bill _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Tue Jan 20 12:16:13 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Tue, 20 Jan 2004 18:16:13 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400D625D.9000902@telia.com> Robert G. Brown wrote: >On Tue, 20 Jan 2004, Per Lindstrom wrote: > > > >>Until I have learned to manage Debian I will use the >>"slack-rpm-transfer-tool" and it will be very interesting to benchmark >>the two distros with each other. >> >>Thanks a lot for all help and support in his case. >> >> > >Just to be clear, the issue is not "benchmarking two distros" for >performance. It is strictly manageability and scalability. > > > I got your point "total operation cost in relation to total performance". Thanks again for the knowledge, experience and information you are sharing with us. Regards Per _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Tue Jan 20 13:15:59 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Tue, 20 Jan 2004 10:15:59 -0800 (PST) Subject: [Beowulf] LAM/MPI Setup Message-ID: <20040120181559.73844.qmail@web12703.mail.yahoo.com> hi all, Have been trying (unsuccessfully) to run LAM/MPI on my 2 node network. But when I run lamboot, it says Permission Denied. the and the network is configured properly. Also the problem is with ssh. Earlier I had the same problem with rsh. I sorted it out by adding: in.rshd 192.168.0. and in.rlogind 192.168.0. although I have also added sshd 192.168.0. to the /etc/hosts.allow file I get the ssh error. I got the above from Google, if anyone has a better solution please help. Thank You Toufeeq ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From irene at syndeocorp.com Tue Jan 20 15:53:27 2004 From: irene at syndeocorp.com (Irene Koo) Date: Tue, 20 Jan 2004 12:53:27 -0800 Subject: [Beowulf] Bonding driver in Active-standby mode Message-ID: <012401c3df97$74b79570$39231504@irenexppc> Hi, I am new to using the channel bonding driver. So, any help would be appreciated. Thx. Basically, I have one 3Com 3c905c (using 3c59x driver) and an integrated Intel NIC using e100 driver on one system. Both of them are ifenslaved to bond0 with parameters mode=1 arp_interval=20 arp_target_ip=10.1.5.105 bonding.c:v2.2.5 (May 22, 2003) bond0 registered without MII link monitoring, in fault-tolerance (active-backup) mode. bond0 registered with ARP monitoring set to 20 ms with 1 target(s): 10.1.5.105 On another system, I have one 3Com 3c905b and an integrated Intel NIC using e100 driver. Both of these NICs are ifenslaved to bond0 with parameters mode=1 arp_interval=20 arp_target_ip=10.1.5.100 Two cross-over cables connect these 2 systems. The question I have is when the backup interface gets activated or when a recovered interface (the link failed and the up) get activated, the kernel recorded over 300 times of the following messages: bond0: backup interface eth0 is now up bond0: backup interface eth0 is now down bond0: backup interface eth0 is now up bond0: backup interface eth0 is now down bond0: backup interface eth0 is now up bond0: backup interface eth0 is now down .... or something like bond0: backup interface eth2 is now up last message repeated 384 times During the time these messages were displayed, traffic from ping does not go through. After about 30 sec, messages such as bond0: eth0 is up and now the active interface or bond0: eth2 is up and now the active interface was displayed, then traffic would go through again. What is causing all those repeated messages? It looks like the bonding driver was in a loop. From the experiment, I can see that there is a 30 sec down time. Is there something I can do to get ride of the down time? I also used mii-diag --monitor to monitor both interfaces. All "down/negotiating/up" messages are shown almost instantuously after the link went down/up. I also tested with arp_interval=200. Results from this set is pretty simular to the ones above To make things more interesting, I also tried miimon (without arp monitoring). Bond driver setting was miimon=20 and miimon=200 For 3Com 3c905b/c, it took close to 50 sec for the bond driver to switch over; For Intel integrated NIC, it took a bit over 1 sec. Any ideas what is causing the delay? Thanks, Irene K. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Tue Jan 20 16:01:55 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Tue, 20 Jan 2004 16:01:55 -0500 (EST) Subject: [Beowulf] can i increase priority of process on beowulf with mpich In-Reply-To: <20040120115919.1940.qmail@web21502.mail.yahoo.com> Message-ID: On Tue, 20 Jan 2004, prakash borade wrote: > hello > proksy here > > can we change the priority of the process on the > cluser using mpich > > because i have seen that when i run the process as a > root with priority -10 it is maintained only on master > how can i maintain same priority on all machines > Running MPI programs as root is not considered a good idea. Are you using a batch scheduler? Doug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From atp at piskorski.com Tue Jan 20 17:47:46 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Tue, 20 Jan 2004 17:47:46 -0500 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <200401201701.i0KH1Oh22025@NewBlue.scyld.com> References: <200401201701.i0KH1Oh22025@NewBlue.scyld.com> Message-ID: <20040120224741.GA81067@piskorski.com> On Tue, Jan 20, 2004 at 08:05:45AM -0500, Robert G. Brown wrote: > On Mon, 19 Jan 2004, prakash borade wrote: > > can anybody tell me > > what do i choose for programing > > either GNU c or JAVA for implementing some > > networkprograms using some linux systemcalls Hehe. Um, is any of this supposed to be Beowulf related? > Gads, that's two questions in two days with the distinct whiff of crowds > of screaming peasants waving swords and torches... > Now we can give the correct answer, which is "C". > W. Richard Stevens, Networking God (deceased): > http://www.kohala.com/start/ Excellent advice. The three books of his that I have are very good. Now, my own contribution to the jihad. :) None of this is particularly specific to "network programs" by the way: The conventional wisdom is that C is a "portable assembly language" for low-level bit twiddling, performance criticial library functions, and writing byte code compiles for scripting languages, and my own experience seems to bear that out. So, IMNSHO, you don't want to do all your programaming in C, you don't even want to do most of your programming in C. In fact, C should viewed like habaneros, or perhaps caviar - use where needed and called for, no more. So what other language to use? Well: - For any project of reasonable size, you HAVE to know C, because it is ubiquitous, and you will have to use it somewhere. (Your OS calls are in C, 95% of the other APIs you'll want to call are in C, etc. etc.) - Therefore pick a very high level language suitable for the most rapid development possible, and use that for as much of your code as possible. But the ability to easily interface with C is also very important. - Use C only where you really need it, which means: 1) Small amounts of performance critical code. 2) Where you have to in order to talk to other C code - vast numbers of 3rd paty libraries, etc. - For the "very high level language", so far I've found that a scripting language like Tcl works very well. (For more math and especially statistics intensive use, S/S-Plus/R can be good.) - Proper use of an RDBMS (Oracle, PostgreSQL, etc.) can make a huge difference, as it provides a high level interface to many powerful features that are hard to find elsewhere - ACIDity, concurrency, constraints to enforce data correctness, etc. A corrolary of the above is that Java isn't interesting. Java is higher level than C but not very much. I'm already using C for any really low level stuff, I want something a LOT higher level for anything else. Java would probably look pretty attractive if doing ALL my coding in C was my only other option, but it isn't. Therefore, my very limited contact with Java says, Java isn't it. There you go, the answer is Tcl. :) With it's Threads Extension (or in AOLserver), it's particularly fabulous for multi-threaded programming. (Ironically, Tcl guru's have traditonally emphasized concurrent programming using the event loop rather than threads.) And I can always drop down and implement a new Tcl command in C if I need to. I've heard other potential answers ("Standard ML for everything!", "we just need a new Lisp dialect..."), but about those, well, I can't really say. -- Andrew Piskorski http://www.piskorski.com/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Tue Jan 20 18:50:10 2004 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 21 Jan 2004 00:50:10 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: References: <20040120065449.33171.qmail@web21507.mail.yahoo.com> Message-ID: <20040120235009.GD30005@unthought.net> On Tue, Jan 20, 2004 at 08:05:45AM -0500, Robert G. Brown wrote: > On Mon, 19 Jan 2004, prakash borade wrote: > > > hey > > > > can anybody tell me > > what do i choose for programing > > either GNU c or JAVA for implementing some > > networkprograms using some linux systemcalls > > Gads, that's two questions in two days with the distinct whiff of crowds > of screaming peasants waving swords and torches... > > Let it be Jihad, then. It is better to light a flame thrower than curse the darkness. - Terry Prachett, "Men at Arms" Just keep in mind that *my* religion is *the* superior one, no matter what the infidels want you to believe! ;) > The answer technically is "it depends". This is to make the vast horde > of java programmers feel loved, and to acknowledge that for writing web > applications their primary scripting language (like perl, python and > many another before it) has a purpose in God's world. I beg to differ. It does not depend. Not when Java is in the discussion. Once Java is out, we can do the huggy-feely 'it depends' and "just because it sucks it doesn't mean it doesn't have a purpose" type discussions. For any systems related programming task, C or C++ is to prefer over Java. For the following reasons: *) C and C++ have consistent run-time performance, which Java *cannot* have due to its automatic garbage collecting (yes, programmers can apply various tricks to lessen the performance penalty and unpredictability - but that's more work for the programmer and for the administrator, and it simply cannot be as good as a well written C or C++ program) *) Portability - well-written C and C++ programs can fairly easily be ported between various UN*X type systems - and new systems can fairly easily be supported by the existing codebase. Larger Java programs are often not portable among various JVMs, and adding portability is not convenient in Java (since it lacks the "dirty" macro-processor of C and C++). In other words; wrt. portability Java is a very "pure" language, since it lacks the constructs (preprocessor) that would be unnecessary in a "perfect world" - unfortunately for Java, the world is not perfect, and therefore the "dirtier" languages have a major advantage here. *) Readability - for larger systems I recommend C++ over C - it allows you to write very efficient type-safe code, unlike C (which would either not be type-safe, or require more code) and Java (which is inefficient by design, and will do much of its type checking at run-time, decreasing programmer productivity, program performance and ease of testing) *) Reliability - well written C++ tends to be very stable - most problems will display themselves at compile time, and you have excellent tools to verify that your code is running as it should at run-time (valgrind, ElectricFence, LeakChecker), and to inspect what went wrong when your program crashed (core dumps and GDB). Most of this holds true for C as well, except that fewer errors show themselves at compile time, because the C type system is less strict than the C++ type system. > > There. Let us bask for a moment in the serenity of our knowledge that > we have the complete freedom to choose, and that there are no wrong > answers. I find your lack of faith disturbing... ;) > > Now we can give the correct answer, which is "C". Typing a little fast there, I think... The correct answer for anything larger than 1KLOC is "C++" - of course, you knew that, you were just a little fast on the keyboard ;) (KLOC = Kilo-Lines-Of-Code) > > In order to give it, I have to infer an implicit "better than" and a > certain degree of generality in your question is in "which language is > BETTER suited to write an efficient networking program using linux > systemscalls, in GENERAL". > > With this qualifier, C wins hands down. A variety of reasons: > > a) The operating system is written in C (plus a bit of assembler) This ought not to be a good argument - I guess what makes this a good argument is, that the operating system provides very good C APIs. So, any language that provides easy direct access to those C APIs have a chance of being "the one true language". > b) Nearly all network daemons and utilities and program clients are > written in C or perhaps an extension of C such as C++ I prefer to think of "C++" as "A better C", rather than a "C extension", as not all C is valid C++, and therefore C++ is not really an extension. > c) Nearly all decent books on network programming (e.g. Stevens) > provide excellent C templates for doing lots of network-based things > d) You can do "anything" with C plus (in a few, very rare cases, a bit > of inlined assembler) Amen! This goes for C++ as well though. > e) C is a compiler and produces (in the hands of a good programmer) > code that runs very efficiently. Java is an interpreter and does not. Java is not necessarily interpreted - but that doesn't help (it is still garbage collected, and it's general library routines such as lists etc. still work on "Object" types, not the real types of the elements, which makes it run-time inefficient by design). So I absolutely agree that Java sucks, just for slightly different reasons :) > f) C is an open ANSI standard implemented in linux with fully open > source tools with no restrictions or covenants on their usage and with > no "vendors" in any sort of position of control over the language itself > or its core libraries. Java was invented by Sun, and Sun (AFAIK) still > controls the distribution of its core libraries, and as an interpreted > language, those libraries are sine qua non for portability. Sun is > relatively linux friendly, but I personally tend not to trust giant > multibillion dollar companies that own core components of something I > need to rely on (or incorporate those components into my code), as they > have an annoying tendency to turn around and want money for it. > g) Or find themselves imbroiled in a corporate war with someboy ELSE > with deep pockets advancing a different "standard"; for example I'd > guess that Microsoft would like nothing better than to co-opt Java for > themselves and turn around and want money for it. C is co-opt > proof, and GCC is a highly reliable platform upon which to base code in > the certain knowledge that the core compiler and libraries will remain > royalty free. > h) ...and I could go on. Me too, oh, me too ;) > > Note that I do not address java's OO nature, as OO C variants exist AND > as OO programming is largely a matter of programming style, not language > per se anyway. Let me then. Java *forces* OO. C makes OO possible, but cumbersome and error prone. C++ makes OO both possible and safe. The fact that Java forces OO down your throat, whether you want it or not, makes it an inferior language alone. See, most real world problems are not stricly OO, or strictly procedural. They tend to be a good mix. This is why C becomes cumbersome in many larger projects, since it does not address the OO parts well. And this is why Java is hopeless for larger projects, since it simply forces a re-write of the would-be simple procedural code into something sufficiently OO-like for the Java language to be able to comprehend it. C++ on the other hand allows (and even encourages) a mix-and-match approach - using the proper "tools" for the job at hand. Oh, and not only does Java force OO, it forces inferior OO. It cannot do multiple inheritance, which makes a lot of the good arguments for OO vanish (single-inheritance encourages fat bases, which is all that OO is *not* about). > ... > > Note also that my diatribe above is clearly biased as I don't code in > Java, largely become of all of the reasons above (and then some). > Speed, power, portability, open standard and source, and available code > base and learning support all argue in favor of C about 10 to 1. UNLESS > you are trying to write a powerful web-distributed application, which is > a PITA in C and what Java was designed for. Yes - I've been told that that is what Java does well. In generall, I am pretty much of the oppinion that C++ does 99% of all problems "well" or "very well", and the remaining 1% somewhere between "poor" and "well". Java on the other hand will do 99% of all problems somewhere between "crap" and "not at all", where it will do the remaining 1% "extremely well". Unless you have a problem in that last 1%, there is probably better languages than Java. C, in my oppinion, would be somewhat like C++, except for larger problems it doesn't fare qute as well (not poorly by any means, just not as well). The only very very large problem with C++ is, that almost no people know the language. There is a mountain of absolutely crap learning material out there. This is why you see examples of "C versus C++" where the C++ code is several times larger or even less efficient than the C example, because the author felt that shovelling bad OO and braindead design after the problem seemed like a good idea. I believe that with the state of compilers today, nobody should have to start a large project in C - except if it absolutely needs to run on a platform for which no decent C++ compiler is available (maybe Novell NetWare - but that's the only such platform that comes to mind...) And I firmly believe, that in order to justify starting a project in Java, you must have *thorough* understanding of not just the problem and Java, but also about possible alternative languages - there really should be very few excuses for using the language for anything at all. ... > P.S. -- I >>tried<< to be mild, but I suppose I'll need to spend the > rest of the day in a bunker wearing my lead underwear and a tinfoil > skullcap. Really, I love YAPL's as they pop up from year to year. I > do. Marshmellows...ready! Flame on... My office is cold, and I like the flames :) Seriously though, I think that the language-flamewars are fun and informative, since so much happens in the space of compilers and real-world projects out there. So, I think it's useful to get an update every now and then, from people who have strong feelings about their langauges - oh, and a discussion in the form of a friendly flame-fest is always good fun too ;) Well, the bait is out, let's see if someone bites ;) / jakob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Wed Jan 21 01:39:57 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed, 21 Jan 2004 01:39:57 -0500 Subject: [Beowulf] fully rpm-ified NCBI toolkit available Message-ID: <1074667197.4344.28.camel@protein.scalableinformatics.com> Hi folks: I wanted to fix up my original packaging of the ncbi toolkit to make full use of the RPM mechanism for easier compilation and installation on clusters and other systems. I built a spec file for it and created a source RPM which you can pull down from http://downloads.scalableinformatics.com/ specifically http://downloads.scalableinformatics.com/downloads/NCBI-2.2.7-1.src.rpm There is nothing OS or target specific in this RPM. I will add in the Opteron patches and some other tweaks in short order, and get an updated source RPM. To build this RPM into a set of binary RPMs, do the following as root user: rpmbuild --rebuild NCBI-2.2.7-1.src.rpm If you want to build for a specific target processor, use the --target option, as in rpmbuild --rebuild --target i686 NCBI-2.2.7-1.src.rpm Please let me know if you find any problems. It has rebuilt fine on a number of machines thus far. I would like to see what happens on non-native RPM architectures (OS-X, and other Unices). -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From camm at enhanced.com Wed Jan 21 09:22:54 2004 From: camm at enhanced.com (Camm Maguire) Date: 21 Jan 2004 09:22:54 -0500 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <1074603930.2417.8.camel@tp1.mesh-hq> References: <400C6F80.6060600@comcast.net> <400CFB5F.4010104@telia.com> <1074603930.2417.8.camel@tp1.mesh-hq> Message-ID: <541xptmldt.fsf@intech19.enhanced.com> Greetings! Just a brief plug for Debian here -- we've been using it on our commercial cluster for many years and across several hardware upgrades, and, after the initial installation, never futzing with the software by hand other than 'apt-get install' and/or 'apt-get dist-upgrade'! We're literally running the 'same installation' automatically upgraded via apt through at least 3 or 4 Debian release generations. And the quantity of parallel software available 'out of the box' is enormous! I'll never forget the day I could just 'apt-get install scalapack-lam-dev' and diagonalize huge matrices across the cluster with hardly a hitch. As for the initial installation, some people like FAI, though I have never tried it myself, as I've only had to install once, and that was somewhere around 1995 .... Take care, Lars Henriksen writes: > On Tue, 2004-01-20 at 09:56, Per Lindstrom wrote: > > Here I am not running for Slack for its own sake so I will use Slacware > > up to two (2) nodes and when shifting to Debian when I'm connecting the > > remaining nodes. I think it will stop at about 10 CPUs. > > > > Until I have learned to manage Debian I will use the > > "slack-rpm-transfer-tool" and it will be very interesting to benchmark > > the two distros with each other. > > If you want to use Debian as the base distribution, I can recommend > SytemImager as an installation tool. > > http://www.systemimager.org/ > > Sytemimager allows you to replicate an installed client via PXE/rsync to > the remaining nodes. It also allows for updates to be distributed via > rsync. A feature that i really like, is that it allows for the use of > several different images. > > Also it scales pretty well (I had the pleasure of using it in a couple > of 16-32 node configurations, as well as on our own cluster) > > IMO it is the easiest way to deal with smaller clusters (except maybe > CLIC) > > best regards > Lars > -- > Lars Henriksen | MESH-Technologies A/S > Systems Manager & Consultant | Forskerparken 10 > www.meshtechnologies.com | DK-5230 Odense M, Denmark > lars at meshtechnologies.com | mobile: +45 2291 2904 > direct: +45 6315 7310 | fax: +45 6315 7314 > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > -- Camm Maguire camm at enhanced.com ========================================================================== "The earth is but one country, and mankind its citizens." -- Baha'u'llah _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Wed Jan 21 10:44:18 2004 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Wed, 21 Jan 2004 16:44:18 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120224741.GA81067@piskorski.com> References: <200401201701.i0KH1Oh22025@NewBlue.scyld.com> <20040120224741.GA81067@piskorski.com> Message-ID: <20040121164418Z.hanzl@unknown-domain> > scripting language like Tcl works very well I think tcl(/tk) fulfills most of the promisses heard about Java :-) Those tempted by Java hype might therefore well try tcl (and C/C++ of course). I use tcl for various purposes throughout the cluster and I am very happy with it. Superb integration ability. There is no better glue than tcl. But it is a pleasing language on its own. Excellent ability to go to meta-levels like adding new control structures (comparable to FORTH if anybody remembers it). Making GUI for some cluster management tool is a fun with tcl/tk. And few bonuses: - it even works on M$ Windows and Mac (shock your friends ;-) - if you ever dreamt about goodies from LISP without having to deal with Lots of Silly Idiotic Parenthesis, these dreams come true in tcl One strong detergent: Syntax which does not follow usual conventions. But you will appreciate it later, for sure. There is a very strong logic behind it. (Those who ever tried deep TeX programming: tcl syntax shock is say 10% of the TeX syntax shock ;-) Just my religious 2 cents... Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 21 11:46:15 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 21 Jan 2004 11:46:15 -0500 (EST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120235009.GD30005@unthought.net> Message-ID: On Wed, 21 Jan 2004, Jakob Oestergaard wrote: > > Well, the bait is out, let's see if someone bites ;) > Having been accused of having early alzheimers and forgetting some silly little symbol, what was it, oh yeah, a "++" in my even handed and totally objective diatribe, I'll have to at least nibble:-) > It is better to light a flame thrower than curse the darkness. > - Terry Prachett, "Men at Arms" Alas, my flame thrower is in the shop. The best I can do is make a nifty lamp with a handy wine bottle, some gasoline, and some detergent flakes. There, let me stick this handy rag in the neck like this, now where were we again? My memory is failing me. Oh yes. The pluses. > > There. Let us bask for a moment in the serenity of our knowledge that > > we have the complete freedom to choose, and that there are no wrong > > answers. > > I find your lack of faith disturbing... ;) Trying an old jedi mind trick on me, are you? At least your metaphor is correct. You are indeed attracted to the Power of the Dark Side...:-) > > Now we can give the correct answer, which is "C". > > Typing a little fast there, I think... The correct answer for anything > larger than 1KLOC is "C++" - of course, you knew that, you were just a > little fast on the keyboard ;) > > (KLOC = Kilo-Lines-Of-Code) Well, we COULD make a list of all the major programs involving > 1KLOC (exclusive of comments, even) that have been written entirely in C (plus a small admixture, in some cases, of assembler). Unfortunately, the list would be too long (I'd be here literally all day typing), and would include things like the kernels of pretty much all the major operating systems (certainly all the Unix derivatives and variants) and gcc itself, including its g++ extension. To be fair, that doesn't prove that your observation is wrong -- there is a rather huge list of fortran sources available in my primary field of endeavor, for example, which doesn't stop fortran from being a cruel joke. However, to be fair the OTHER way most of that fortran was written by scientists (a.k.a. "complete idiots where advanced computer programming is concerned") to do what amounts to simple arithmetic problems, or maybe even complex arithmetic problems, fine, with a trivial data interface (read the data in, write the data out). Quite a bit of the aforementioned C sources and operating systems and advanced toos were written not only by computer professionals, but by "brilliant" computing professionals. Class O stars in a sea of A, B, F and G (where fortran programmers are at best class M, or maybe white dwarfs). What you probably mean is that IF everybody knew how to program in C++, they would have written all of this in C++, right? Let's see, is there a major operating system out there that favors C++? Hmmmm, I believe there is. Are it and its applications pretty much perpetually broken, to the extent that a lot of its programmers have bolted from C++ and use things like Visual Basic instead? Could be. This isn't intended to be a purely a joke observation. I would rather advance it as evidence that contrary to expectations it is MORE difficult to write and maintain a complex application in C++. The very features of C++ that are intended to make an application "portable" and "extensible" are a deadly trap, because portability and extensibility are largely an illusion. The more you use them, the more difficult it is to go in and work under the hood, and if you DON'T go in and work under the hood, things you've ported or extended often break. To go all gooey and metaphorical, programming seems to be a holistic enterprise with lots of layering and feathering of the brush strokes to achieve a particular effect with C providing access to the entire palette (it was somebody on this list, I believe, that referred to C as a "thin veneer of upper-level language syntax on top of naked assembler"). C++, in the same metaphor, is paint by numbers. It ENCOURAGES you to create blocks to be filled in with particular colors, and adds a penalty to those that want to go in and feather. In some cases, those paint-by-numbers blocks can without doubt be very useful, I'm not arguing that. It is a question of balance. A very good C++ programmer (and I'm certain that you are one:-) very likely has developed a very good sense of this balance, as suggested by your observation that a good C++ programmer writes what amounts to procedural C where it is appropriate (which IMHO is a LARGE block of most code) and reserves C++ extensions for where its structural blocking makes sense. Who could argue with that? Of course, a very good programmer in ANY language is going to use procedural methodology where appropriate, and create "objects" where they make sense. We must not make the mistake of comparing good programming practice with the language features. Here are the real questions. Presuming "good programmers, best practice" in all cases: a) Do C's basic object features (typedefs, structs and unions, for example) suffice to meet the needs for object creation and manipulation in those code segments where they are needed? I would say of course they do. IMO, protection and inheritance are a nuisance more often than an advantage because, as I observed, real "objects" are almost never portable between programs (exceptions exist for graphical objects -- graphics is one place where OO methodology is very appropriate -- and MAYBE for DB applications where data objects are being manipulated by multiple binaries in a set). "protection" in particular I think of as being a nuisance and more likely to lead to problems than to solutions. In a large project it often translates into "live with the bugs behind this layer, you fool, mwaahhahahaha". Result: stagnation, ugly hacks, bitterness and pain. In single-developer projects, just who and what are you protecting your structs FROM? Yourself? b) Do C++'s basic object features increase or decrease the efficiencies of the eventual linked binaries one produces? As you say, C++ is more than an "extension" of C, it has some real differences. In particular, it forces (or at least, "encourages") a certain programming discipline on the programmer, one that pushes them away from the "thin veneer" aspect of pure C. I think it is clear that each additional layer of contraint INCREASES the intrinsic complexity of the compiler itself, thickens the veneer, and narrows the range of choices into particular channels. Those channels in C++ were deliberately engineered to favor somebody's idea of "good programming practice", which amounts to a particular tradeoff between ultimate speed and flexibility and code that can be extended, scaled, maintained. So I would expect C++ to be as efficient as C only in the limit that one programs it like C and to deviate away as one uses the more complex and narrow features. Efficient "enough", sure, why not? CPU clocks and Moore's Law give us cycles to burn in most cases. And when one uses objects in many cases, they ARE cases where bleeding edge efficiency IS less important than ability to create an API and "reuse" data objects via library calls (for example). Still I think C would have to win here. c) Do C's type-checking, etc. features suffice to make programming particularly easy or safe? Here I will give C++ a win, as the answer for C at least is no. C is not particularly easy and it most definitely is not safe. It is a THIN veneer on top of assembler, and assembler is as unsafe as it gets (short of writing in naked machine code), so thin that one can easily inline assembler to access and manipulate CPU registers etc and have a pretty good idea of how to smoothly move data around and switch from one programming mode to the other. Again the art metaphor is appropriate -- with color-by-numbers you are "safer" from creating really, really ugly pictures (although it is always possible, of course:-). With nothing but the canvas, the paints, and a set of brushes and palette knives you can create anything from Da Vinci or Rembrandt to a three-year-old's picture of a dog (big blob of muddy brown in the middle of the canvas that may or may not even have "eyes"). This is why I think that the C++ vs C issue is almost entirely determined by an individual's personal taste and preferences, along with (maybe) the amount and kind of code that they write. I personally prefer to eat my vegetables (objects) raw -- if I need a struct, I make a struct. If I want to allocate a struct, I use malloc or write a constructor, depending on the complexity of the struct. If I want to de-allocate a struct, I either use free or I write a destructor, again depending on the complexity (whether or not I have to recursively free the contents of the struct, and at how many levels, how many times in the code). If I want to change the struct (either structurally or by accessing or altering its contents), I change the struct, and am of course responsible for changing all the points in my program that are affected by the change. If I want to "protect" the struct, well, I don't change it, or write to it, or read it, or whatever. My choice, unchanneled by the compiler. Do I have to deal with sometimes screwing up data typing? Absolutely. Do I have to occasionally deal with working my way backwards through my programs to redo all my structs? Of course (I'm doing it right now with a program I'm writing). This process is INEVITABLE in real-world program design because we DON'T know a priori what the best shape is for the data objects is until we've programmed them at least once the wrong way, in most cases. The really interesting question is whether or not this PROCESS is any easier in C++ than in C. I can't see why it would be, but perhaps it is. I suspect it is still mostly a matter of skill and style (and how one apportions time between active code development and "planning" a program in the first place) and, of course, personal taste. > > In order to give it, I have to infer an implicit "better than" and a > > certain degree of generality in your question is in "which language is > > BETTER suited to write an efficient networking program using linux > > systemscalls, in GENERAL". > > > > With this qualifier, C wins hands down. A variety of reasons: > > > > a) The operating system is written in C (plus a bit of assembler) > > This ought not to be a good argument - I guess what makes this a good > argument is, that the operating system provides very good C APIs. > > So, any language that provides easy direct access to those C APIs have a > chance of being "the one true language". You miss my point entirely. The argument is that C is a thin veneer on top of assembler -- so thin that one CAN write an operating system in it. Imagine writing an operating system in LISP. Wait, don't do that. The results are too horrible to imagine. Imagine doing it on top of fortran, instead. That's bad, but you won't have nightmares for more than a week or two afterwards. (IIRC, somebody actually did this once.) The API issue is moot. Hell, perl and python have direct access to most of the C APIs. IN THE CONTEXT of the reply, of course, there was also the suggestion that the C APIs are a good way of writing network code since the network drivers and kernel structs those APIs provide access to were all written in C, so your access is pretty much "naked". You can often read or write directly any register or value or memory location that isn't in the protected part of the kernel, if you dare (or need to to achieve enough efficiency in your particular application). But C++ I'm sure provides the same degree of naked access and C and C++ share a common underlying data organization, really, and even Fortran (with somewhat different data organization) probably does pretty well. In other languages (especially scripting languages e.g. java, perl, python), the access is typically "wrapped" in a translation layer that is required because one has NO CONTROL over the way the interpreter actually creates data objects. They are the "ultimate" in OO programming -- instantly created by the interpreter in real time, manipulable according to a wide set of rules, they go away when they aren't being used without leaking memory (usually:-) but heaven help you if you look beneath the hood and try to manipulate the raw memory addresses. This is a lovely tradeoff in a lot of cases, which is why I hesitate to actually stick a rod up java's ass and barbeque it on a spit. I really do believe the BASIC argument associated with OO programming, that the data and what you want to do with it should dictate program design features, including choice of language and programming technique. For many problems perl or python provide a very good data interface where one DOESN'T have to mess with lots of data issues that are a major chore in C. Wanna make a hash? Use it. Wanna make your hash into an array? Change the little % TO A @ and hack your dereferencing a bit. Is your hash actually stored as a linked list, as an array of structs? Don't ask, don't tell, don't try to look. All this and direct syntactical manipulation of regular expressions, what's not to love? This is what I was trying to keep an open mind to WRT java. Perhaps there are aspects of its data manipulation methodologies and programming features that are excellent fits to particular problems. I just don't know, and would rather have my teeth drilled with a half-charged portable black and decker screwdriver (allen bit) than learn YAPL just to find out. Hell, I'm only starting to learn python under extreme duress as I swore perl was going to be the last language I ever learned and that was before PHP and now python. Somebody would have to pay me a LOT OF MONEY to get me to learn java. Yessir, a whole lot. [Anybody reading this who happens to have a lot of money is welcome to contact me to arrange for a transfer...;-)] > I prefer to think of "C++" as "A better C", rather than a "C extension", > as not all C is valid C++, and therefore C++ is not really an extension. Funny that. I tend to think of C++ as "A broken C extension" for exactly the same reason;-) If they hadn't broken it, then there really would be no reason not to use it, as if C were a strict subset of C++ all the way down to the compiler level, so that pure C code was compiled as efficiently as pure C is anyway then sure, I'd replace all the gcc's in my makefiles with g++'s. That would certainly lower the barrier to using C++; C programmers could program in C to their heart's content and IF AND WHEN their program needs a "C++ object" they could just add it without breaking or even tweaking the rest of their code. Breaking C was (IMO) a silly design decision. So was adding crap like a completely different stdin/stdout interface, which makes most C++ code, even that which doesn't use objects in any way, most non-back-portable-or-compatible to C. It is very clear that these were all deliberate design decisions INTENDED to break C and FORCE programmers to make what amounts to a religious choice instead of smoothly extend the palette of their programming possibilities. I'd honestly have to say that it is this particular aspect of C++ more than any other that irritates me the most. There is a HUGE CODE BASE in C. For a reason. > > c) Nearly all decent books on network programming (e.g. Stevens) > > provide excellent C templates for doing lots of network-based things > > d) You can do "anything" with C plus (in a few, very rare cases, a bit > > of inlined assembler) > > Amen! This goes for C++ as well though. Y'know, you won't believe this, but I actually added the (and C++) references in my original reply just thinking of you...;-) Last time we had this discussion you were profound and passionate and highly articulate in advancing C++ -- so much so that you almost convinced me. Alas, that silly barrier (which I recall your saying took you YEARS to get over yourself)... I just bounce right off of it in a state of total irritation every time I try. Another problem is that every time I work with a student C++ programmer coming out of Duke's CPS department (which now teaches C++ as standard fare) I observe that while they are just great at creating objects and so forth they are clueless about things like pointers and malloc and the actual structure of a multidimensional array of structs and how to make one and manipulate its contents. As a consequence I have to spend weeks teaching them about all of the virtues of "C as a thin veneer" in order to get them to where they can cut advanced code at all, in either one. You may not have encountered this as you did C first. Or maybe you never have cause to allocate triangular arrays of structs or to build your own linked list of structs, or maybe C++ does this and eats your meatloaf for you and I'm just to ignorant to know how. Its funny. Back in the 80's all the CPS departments taught pascal because it provided strong type checking and absolutely forced one to use a rigorous bottom-up program design methodology. Precisely the same features that caused all real programmers to run from it screaming, as somehow all the programmers I've ever met work top down initially, then top, bottom, middle, whereever until the project is finished, often with a few epiphany-driven total project reorganizations in the middle as experience teaches one what the REAL data objects are that they are manipulating and whether they need easily maintained code that can run relatively slowly or whether they need the core loop to run faster than humanly possible if they have to hand-code it in assembler to get it there. Now they teach C++, and as you observed, teach it badly. I humbly submit that the REASON they teach it badly is that they aren't teaching "C++" at all, they are teaching structured programming and programming discipline with a language that simply doesn't permit (or if you prefer encourage) the student to use methodologies with far more power but far less externally enforced structure. Pure pedagogy, in other words. I personally think the world would be a better place if they FIRST taught the students to code in naked C with no nets, and taught them that the reason for learning and using good programming discipline is BECAUSE the bare machine that they are working with comes with no nets. Then by all means, teach them C++ and object oriented design principles. I suspect that students who learn C++ in this order are, as you appear to be, really good programmers who can get the most out of C++ and its object oriented features without "forgetting" how to manipulate raw blocks of memory without any sort of OO interface per se when the code structure and extensibility requirements don't warrant all the extra overhead associated with setting one up. > C, in my oppinion, would be somewhat like C++, except for larger > problems it doesn't fare qute as well (not poorly by any means, just not > as well). For CERTAIN larger problems, you could be right. I do consider things like writing operating systems and compilers to be "larger problems" though, and C seems to do very well here;-) Your arguments from last time were very compelling, as I said. > The only very very large problem with C++ is, that almost no people know > the language. There is a mountain of absolutely crap learning material > out there. This is why you see examples of "C versus C++" where the C++ > code is several times larger or even less efficient than the C example, > because the author felt that shovelling bad OO and braindead design > after the problem seemed like a good idea. Again, I think that is because most C++ programmers learn C++ >>first<< as part of a course on structured programming, which teaches them to use C++ the way it was "intended" to be used. If one learns C first, on the other hand, you already KNOW how to write tight code and can freely switch over to using C++ constructs where they make sense. In fact, since you actually understand what a struct IS and how data allocation WORKS (a thing that, believe it or not, is largely hidden from the student in most C++ classrooms as this is precisely the under-the-hood stuff considered anathema in a world where programmers are expected to be commodity items and write commodity --interchangeable -- code) you can probably even do very clever end runs around most of the silliness that they teach as good practice whenever and whereever it suits you. > I believe that with the state of compilers today, nobody should have to > start a large project in C - except if it absolutely needs to run on a > platform for which no decent C++ compiler is available (maybe Novell > NetWare - but that's the only such platform that comes to mind...) Give me a few days paring C++ down to the bare minimum extension of C so that it is a pure superset (so ALL C CODE just plain compiles perfectly and even produces the same runfile, so that the I/O language "extensions" are moved off into a library that you can link or not, mostly not, and so that using things like classes and protection and inheritance are suddenly pure and transparent extensions of the basic ideas of structs and unions and typedefs, and I wouldn't even argue. In fact, I'd say that the argument itself becomes silly and moot. One wouldn't even have to change Makefiles, as C++ would just be a feature of gcc that is automatically processed when and where it is encountered. So the kernel would suddenly be written in "C++", sort of, and there would be a very low barrier to converting C into C++ because basically, there would be no conversion at all, only extension. This of course will never happen. Until it does, then an excellent reason to choose C instead of C++ is if you happen to know C very well but not know (or care for) C++ as well. Or if your project is derived from a code base already written in C, and you don't feel like going back and fixing everything to make a C++ compiler happy with it (especially when doing so might well "break" it so that a C compiler is no longer happy with it). Note that this isn't a "theoretical" argument about should -- it is a real-world argument on why the choice you describe is NOT made, over and over again, by most people writing code (judging strictly on the basis of the number of GPL projects written in C vs C++). Even in a world where most schools have been TEACHING C++ for close to a decade, the majority of people who become computer scientists and systems programmers and to work on the guts of computer systems underneath the hood seem to at some point crossover to C and stay there, while the majority of people who stick with C++ end up programming for Windows for some big corporation (which might well be Microsoft) -- and produce shitty, broken, huge, inefficient code as often as not. My next door neighbor is an interesting example that comes to mind. He is a professional programmer. A decade ago he taught CPS at Duke, but got irritated because he had to teach the students to program in C++, and in such a way that they never learned how data structures really work. Believe it or not, I've had to actually teach SEVERAL students who have FINISHED the intro computer courses here just how memory on a computer works -- it is deliberately taught in such a way that one DOESN'T learn that, one learns instead to "think" only about the compiler-provided memory schema. He once spent a whole afternoon in my yard ranting at me about how a struct or union was all the object support any programmer could ever need. So he quit and took over Scholastic Books software division, made a modest fortune, and STILL lives next door writing software and clipping coupons. Other faculty I know seem to view C++ the same way -- a good thing to use to teach students because it forces them to code a certain way, but THEY don't use it for THEIR research projects. Professionals seem more often than not to prefer having the entire palette and all the brushes, and don't like even MAKING color by numbers codelets. This is very definitely not exhaustive or definitive. I also know students where they exact opposite is true -- they learned C++, learned about data structs and malloc and C, and continue to code in C++ (as you do) with strictly enriched abilities. Of course they are (like you;-) Very Bright Guys...and could probably even write good Fortran code if they ever learned it. No, wait, that is an oxymoron -- couldn't happen. > Seriously though, I think that the language-flamewars are fun and > informative, since so much happens in the space of compilers and > real-world projects out there. So, I think it's useful to get an update > every now and then, from people who have strong feelings about their > langauges - oh, and a discussion in the form of a friendly flame-fest is > always good fun too ;) > > / jakob I agree, and hope you realize that I'm deliberately overarguing the case for C above for precisely that reason. I really do believe that it is as much a matter of taste and your particular experience and application space as anything else and wouldn't be surprised if some java coder DOES speak up for java as that's what THEY are really good at and they've discovered some feature that makes it all worthwhile. We've already heard from the python crowd, after all, and THAT'S a language where they even eschew such obviously useful constructs as {}'s and line terminators such as ; in favor of indentation as a matter of deliberate design/dogma. It really does come down in the end to religious belief/personal taste. In the best spirit of tongue-in-cheek, I cannot do better than end with: http://www.phy.duke.edu/~rgb/Beowulf/c++_interview/c++_interview.html Read it and weep...:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 21 12:24:50 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 21 Jan 2004 09:24:50 -0800 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040121164418Z.hanzl@unknown-domain> References: <20040120224741.GA81067@piskorski.com> <200401201701.i0KH1Oh22025@NewBlue.scyld.com> <20040120224741.GA81067@piskorski.com> Message-ID: <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> At 04:44 PM 1/21/2004 +0100, you wrote: >- if you ever dreamt about goodies from LISP without having to deal > with Lots of Silly Idiotic Parenthesis, these dreams come true in > tcl Surely you ment Lots of Idiotic Silly Parenthesis (after all, isn't LISP an acronym) And, pulling some old Intel i/PSC documentation off the shelf (what is an Intel i/PSC but the ancestor of a modern Beowulf... 8 MHz 80286 CPU (with an optional 80287), Ethernet for interprocessor comms (the venerable 82586), Xenix as the base OS).. I find a whole raft of stuff advocating CCLISP for all sorts of things: battle management, dining philosophers problems, optimizing loads of livestock and produce across a river in a small boat, all using stuff like "message passing", node streams, etc. Even such useful stuff as "Sorting a List of N elements" using things like the Sample sort of Felton, Karlin, and Otto. Lest we get too enamored with CCLISP, there's also: (roll of drums) "Concurrent Prolog" (Crash of cymbals, choir of angels) Having not actually looked at anything in Prolog for the last 15+ years, I thought it would be interesting to repeat a short program here. (I have no idea if it would work, since I don't recall CP Syntax) quicksort([X|Xs],Ys):- partition (Xs?,X,Smaller,Larger), quicksort (Smaller?,Ss), quicksort (Larger?,Ls), append(Ss>,[X|Ls?],Ys). quicksort([],[]). partition([Y|ln],X,[Y|Smaller],Larger):- X>=Y|partition(ln?,X,Smaller,Larger). partition([Y|ln],X,Smaller,[Y|Larger]):- X indicates communications name? => readonly variable, process waits until there is a value in the name before proceeding (presumably this is some sort of barrier sync?) ---------- In all of this, they make the very appropriate comments along the lines of: sure it's horribly inefficient, but, you've got: a) tons of computer power potentially available (I mean, you could have 128 of those 8 MHz 286's spinning away) b) you've got a moderately transparent way to distribute the load across those processors without worrying too much about subtlety c) human hours expensive|machine hours cheap d) The computers WILL get faster, so don't agonize about CPU resources too much.. time and the market will improve performance per dollar or wall clock hour much faster than toiling over a hot keyboard ekeing out algorithm tuning improvements. Spend your time on better high level algorithms to solve "your problem". Hey, this was in 1986,1987 or so.. In the last 15 years, I don't think much has really changed, philosophically. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From saravalli.simone at student.unife.it Wed Jan 21 14:06:20 2004 From: saravalli.simone at student.unife.it (Simone Saravalli) Date: Wed, 21 Jan 2004 20:06:20 +0100 Subject: [Beowulf] my first time Message-ID: <001601c3e051$a87182d0$68221997@simc942uomw5a0> Hi to everybody, I'm an italian student in computer science and this is the first time that I write on this mailing list. I hope that you will help this poor newbie with his first experiences with clusters. Thank you and excuse me if my english isn't so good. Simone Saravalli _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 21 17:31:13 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 21 Jan 2004 14:31:13 -0800 Subject: apropos of language wars wsa Re: [Beowulf] Which is better In-Reply-To: References: <20040120235009.GD30005@unthought.net> Message-ID: <5.2.0.9.2.20040121141457.0305ad88@mailhost4.jpl.nasa.gov> At 11:46 AM 1/21/2004 -0500, Robert G. Brown wrote: < a whole pile of stuff in response to Jakob's whole pile of stuff, all quite interesting, but irrelevant to my comments inserted below> > So, any language that provides easy direct access to those C APIs have a > > chance of being "the one true language". > >You miss my point entirely. The argument is that C is a thin veneer on >top of assembler -- so thin that one CAN write an operating system in >it. Imagine writing an operating system in LISP. Wait, don't do that. >The results are too horrible to imagine. Imagine doing it on top of >fortran, instead. That's bad, but you won't have nightmares for more >than a week or two afterwards. (IIRC, somebody actually did this once.) I should imagine that more than one person has written what amounts to an operating system (or, at least the kernel, meaning file system and scheduler/task management) in FORTRAN or other less than ideal languages. Certainly I have done both (although, thankfully, not in the same system!..).. (Gosh, I had to write a good part of an OS in MIXAL as an assignment in college, something I wouldn't wish on an enemy) Many, many laboratory/instrument control kinds of applications have been written in Basic or FORTRAN, and they tend to have all the sorts of things one expects in a scheduler (variable rate events, threads, timed execution, priorities, etc.), although, granted, with a somewhat adhoc arrangement of interfaces and so forth (and with varying degrees of independent contexts (ah, the memories of using Blank COMMON, and implementing linked lists and heaps in FORTRAN arrays). You use what's available. Some of the low level BASIC's out there for microcontrollers like the BASICStamp are quite close to a "formatted assembler". I'd also point you to ESPOL, which was the native language of Burroughs mainframes in the 70's. The OS (Master Control Program, MCP, in my recollection) was written in (only) ESPOL. It's a variant of Algol, uses polish notation for arithmetic, etc. There was no "assembler" per se for these machines (although, there must have been some binary equivalent for the ESPOL statements. In any case, the machine was designed to be programmed in high level languages, and didn't leave such essential things like array bounds checking, linked list manipuation, or memory allocation to some hacker flinging out assembler in a library. Nope, it's in the CPU as a native opcode. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From karthik at siptech.co.in Wed Jan 21 00:26:42 2004 From: karthik at siptech.co.in (Karthik) Date: Wed, 21 Jan 2004 10:56:42 +0530 Subject: [Beowulf] Re: [Ilugc] LAM/MPI Setup In-Reply-To: <20040120181559.73844.qmail@web12703.mail.yahoo.com> References: <20040120181559.73844.qmail@web12703.mail.yahoo.com> Message-ID: <200401211056.43276.karthik@siptech.co.in> I have had a successful LAM/MPI cluster some time back. Here are the steps that I have figured out after many unsuccessful attempts. These steps are my quickest way to have a LAM/MPI cluster. 1.Install Linux in the Master node 2.Set Hostname and domainname 3.Setup the Master node as NIS server 4.Export /usr, /home directories. 5.Add a user for cluster, say kluster. 6.Update the NIS maps. 7.Install lam-mpi in Master under /usr/local 8.Install NPB in /usr/npb 9.Check the path for mpicc and lam* executables. 10.In Master: login as kluster. 11.Create a text file lamhosts in ~/ with: LMIT3 #Master Node Node1 #Slave Node 12.Install Linux in Slave node node1. 13.Make slave to mount /home, /usr with NFS. 14.Setup NIS ypbind in node1. 15.Check whether NIS is OK by using ypcat. 16.In node1: Login as kluster. 17.Type recon ?v lamhosts in master to check whether the cluster is bootable. 18.Type lamboot ?v lamhosts in Master to start lam on the cluster. 19.From Master, type tping ?c5 N to ping the entire cluster (use this to check the cluster is OK). 20.In Master: mpirun N progname or mpirun ?lamd N progname to run the progname on all nodes. The second command used lamd to run the program. On Tuesday 20 January 2004 23:45, 70uf33q Hu5541n wrote: > hi all, > > Have been trying (unsuccessfully) to run LAM/MPI on my > 2 node network. But when I run lamboot, it says > Permission Denied. > > the and the network is configured properly. > > Also the problem is with ssh. Earlier I had the same > problem with rsh. I sorted it out by adding: > > in.rshd 192.168.0. and in.rlogind 192.168.0. > > although I have also added sshd 192.168.0. to the > /etc/hosts.allow file I get the ssh error. > > I got the above from Google, if anyone has a > better solution please help. > > Thank You > > Toufeeq > > > ===== > "Love is control,I'll die if I let go > I will only let you breathe > My air that you receive > Then we'll see if I let you love me." > -James Hetfield > All Within My Hands,St.Anger > Metallica > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > To unsubscribe, email ilugc-request at aero.iitm.ernet.in with > "unsubscribe address" > in the subject or body of the message. > http://www.aero.iitm.ernet.in/mailman/listinfo/ilugc > > . -- Karthik M System Administrator - Infrastructure Team SIP Technologies & Exports Limited, G4, Elnet Software City, CPT Road, Taramani, Chennai 600 113, INDIA. Phone: +91-44-22541401 +91-44-22541031 Fax: +91-44-22541475 Website: http://www.siptech.com . _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Wed Jan 21 02:23:02 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Tue, 20 Jan 2004 23:23:02 -0800 (PST) Subject: [Beowulf] Re: [Ilugc] LAM/MPI Setup In-Reply-To: <200401211056.43276.karthik@siptech.co.in> Message-ID: <20040121072302.97063.qmail@web12702.mail.yahoo.com> hi karthick, will try out your solution. anyway the error I got was ssh_exchange_identification : Coneection refused by host. I edited my /etc/sshd/sshd_config entries to "no" where all Authentication was needed.Still same problem need help badly. Toufeeq --- Karthik wrote: > I have had a successful LAM/MPI cluster some time > back. Here are the steps > that I have figured out after many unsuccessful > attempts. These steps are my > quickest way to have a LAM/MPI cluster. > > 1.Install Linux in the Master node > 2.Set Hostname and domainname > 3.Setup the Master node as NIS server > 4.Export /usr, /home directories. > 5.Add a user for cluster, say kluster. > 6.Update the NIS maps. > 7.Install lam-mpi in Master under /usr/local > 8.Install NPB in /usr/npb > 9.Check the path for mpicc and lam* executables. > 10.In Master: login as kluster. > 11.Create a text file lamhosts in ~/ with: > LMIT3 #Master Node > Node1 #Slave Node > 12.Install Linux in Slave node node1. > 13.Make slave to mount /home, /usr with NFS. > 14.Setup NIS ypbind in node1. > 15.Check whether NIS is OK by using ypcat. > 16.In node1: Login as kluster. > 17.Type recon ???v lamhosts in master to check > whether the cluster is bootable. > 18.Type lamboot ???v lamhosts in Master to start lam > on the cluster. > 19.From Master, type tping ???c5 N to ping the > entire cluster (use this to check > the cluster is OK). > 20.In Master: mpirun N progname or mpirun ???lamd N > progname to run the progname > on all nodes. The second command used lamd to run > the program. > > > > On Tuesday 20 January 2004 23:45, 70uf33q Hu5541n > wrote: > > hi all, > > > > Have been trying (unsuccessfully) to run LAM/MPI > on my > > 2 node network. But when I run lamboot, it says > > Permission Denied. > > > > the and the network is configured > properly. > > > > Also the problem is with ssh. Earlier I had the > same > > problem with rsh. I sorted it out by adding: > > > > in.rshd 192.168.0. and in.rlogind 192.168.0. > > > > although I have also added sshd 192.168.0. to the > > /etc/hosts.allow file I get the ssh error. > > > > I got the above from Google, if anyone has a > > better solution please help. > > > > Thank You > > > > Toufeeq > > > > > > ===== > > "Love is control,I'll die if I let go > > I will only let you breathe > > My air that you receive > > Then we'll see if I let you love me." > > -James Hetfield > > All Within My Hands,St.Anger > > Metallica > > > > __________________________________ > > Do you Yahoo!? > > Yahoo! Hotjobs: Enter the "Signing Bonus" > Sweepstakes > > http://hotjobs.sweepstakes.yahoo.com/signingbonus > > _______________________________________________ > > To unsubscribe, email > ilugc-request at aero.iitm.ernet.in with > > "unsubscribe address" > > in the subject or body of the message. > > > http://www.aero.iitm.ernet.in/mailman/listinfo/ilugc > > > > . > > -- > Karthik M > System Administrator - Infrastructure Team > > SIP Technologies & Exports Limited, > G4, Elnet Software City, CPT Road, > Taramani, Chennai 600 113, > INDIA. > > Phone: +91-44-22541401 > +91-44-22541031 > Fax: +91-44-22541475 > > Website: http://www.siptech.com > > > . > > _______________________________________________ > To unsubscribe, email > ilugc-request at aero.iitm.ernet.in with > "unsubscribe address" > in the subject or body of the message. > http://www.aero.iitm.ernet.in/mailman/listinfo/ilugc ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raiders at phreaker.net Tue Jan 20 10:06:07 2004 From: raiders at phreaker.net (Shawn Raider) Date: Tue, 20 Jan 2004 23:06:07 +0800 Subject: [Beowulf] SGE vs LSF vs PBS... Message-ID: <200401202306.07794.raiders@phreaker.net> I am expanding an .edu cluster of Xeons, adding more racks. The older racks run LSF alongside all the unix boxes in the campus and the admin staff really love LSF, esp interactive logins. But we can't afford that kind of license pricing. I would be thankful if you could share your experiences in replacing LSF with SGE. I did study features etc from the respective docs/sites etc. before asking here. I am looking specifically for user experiences. Thanks, Shawn _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From afant at geekmail.cc Tue Jan 20 11:57:29 2004 From: afant at geekmail.cc (Andrew Fant) Date: Tue, 20 Jan 2004 11:57:29 -0500 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <400D3833.F1D97ABB@correu.andorra.ad> References: <400D3833.F1D97ABB@correu.andorra.ad> Message-ID: <82620000.1074617849@flux.usg.tufts.edu> Right on, Alan. Lately, I have come to think that python is a "good thing (tm)". I attended the two python tutorials at SC2003 last November, and I was very impressed with the extensions that have been made. SciPy now allows direct calls to the Atlas blas implementation from python, and offers a slew of optimized c routines as well for numerical work in python. f2py provides a mostly automatic way to wrap old fortran code so it can be loaded as a python module. Perhaps most exciting to me is the development work going on with MPI bindings in python. While I do not think python will overtake optimized C and Fortran for the bleeding-edge of scientific computing, I think that it has the potential to capture a lot of the middle ground. For teaching numerical simulations and parallel computing, it reduces the detail of the C API to a level that a part-time developer can keep straight while they are worrying about the science behind the code or that a new user can start using rapidly. Andy --On Tuesday, January 20, 2004 15:16:19 +0100 Alan Ward i Koeck wrote: > Well, my own preference goes to Python, which > > - has a syntax I prefer a lot over C > - while being an interpreted language, manages to be reasonably speedy, > unlike Java > - has a fair amount of predefined library objects you can use to access > C > POSIX system routines > > but I do recognize that low-level stuff definately should be > coded in C. Now, as to just what "low-level" means ... > > Best regards, > Alan Ward > > > "Robert G. Brown" wrote: > (snip) >> >> Let it be Jihad, then. >> >> The answer technically is "it depends". This is to make the vast horde >> of java programmers feel loved, and to acknowledge that for writing web >> applications their primary scripting language (like perl, python and >> many another before it) has a purpose in God's world. >> >> There. Let us bask for a moment in the serenity of our knowledge that >> we have the complete freedom to choose, and that there are no wrong >> answers. >> >> Now we can give the correct answer, which is "C". > (snip) >> -- >> Robert G. Brown http://www.phy.duke.edu/~rgb/ >> Duke University Dept. of Physics, Box 90305 >> Durham, N.C. 27708-0305 >> Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu >> >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From victor_ms at brturbo.com Wed Jan 21 14:38:34 2004 From: victor_ms at brturbo.com (victor_ms at brturbo.com) Date: Wed, 21 Jan 2004 17:38:34 -0200 (BRST) Subject: [Beowulf] About Linpack cluster Message-ID: <25760380.1074713914199.JavaMail.nobody@webmail1.brturbo.com> Hello everbody. I need some help. I have 17 Linux cluster Nodes with Mosix and MPI and I need some help on Linpack software. Anyone where knows where can I download some documents about Linpack? I have the Linpack Software install on my machine, but when I start it, some errors appear so, I need some documents about Linpack. Best Resguards. From camm at enhanced.com Wed Jan 21 19:14:34 2004 From: camm at enhanced.com (Camm Maguire) Date: 21 Jan 2004 19:14:34 -0500 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> References: <20040120224741.GA81067@piskorski.com> <200401201701.i0KH1Oh22025@NewBlue.scyld.com> <20040120224741.GA81067@piskorski.com> <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> Message-ID: <54oeswalg5.fsf@intech19.enhanced.com> Greetings! Jim Lux writes: > At 04:44 PM 1/21/2004 +0100, you wrote: > > > >- if you ever dreamt about goodies from LISP without having to deal > > with Lots of Silly Idiotic Parenthesis, these dreams come true in > > tcl > > Surely you ment Lots of Idiotic Silly Parenthesis (after all, isn't > LISP an acronym) > While I certainly agree that nothing comes close at present to C/Fortran for parallel numerical work, I do wonder what it would take to get LISP to be the best 'higher order' language alternative for this task. After all, 1) it has an *open* standard, 2) it compiles to native machine code, 3) speeds often within 20% of C, and 4) it can easily call libraries written in other languages. In fact, GCL already has a binding to MPI, and there has been discussion about using LISP's powerful macro system to automatically insert blas calls in place of written out floating point matrix code, and then relying on an optimized blas setup as in Debian, where several atlas packages transparently provide accelerated blas without recompilation, to get decent performance. The downside of course is that LISP has lost a lot of mindshare. Take care, -- Camm Maguire camm at enhanced.com ========================================================================== "The earth is but one country, and mankind its citizens." -- Baha'u'llah _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From webhat at xs4all.nl Wed Jan 21 06:29:28 2004 From: webhat at xs4all.nl (=?unknown-8bit?B?RGFuaetsIFcu?= Crompton) Date: Wed, 21 Jan 2004 12:29:28 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120235009.GD30005@unthought.net> References: <20040120065449.33171.qmail@web21507.mail.yahoo.com> <20040120235009.GD30005@unthought.net> Message-ID: <20040121112928.GA8296@troy.rphh.org> On Wed, Jan 21, 2004 at 12:50:10AM +0100, Jakob Oestergaard wrote: > On Tue, Jan 20, 2004 at 08:05:45AM -0500, Robert G. Brown wrote: > > On Mon, 19 Jan 2004, prakash borade wrote: < ... snipped for brevity ... > > > Now we can give the correct answer, which is "C". > Typing a little fast there, I think... The correct answer for anything > larger than 1KLOC is "C++" - of course, you knew that, you were just a > little fast on the keyboard ;) While I can find litte fault with your posts I should note that there are a number of points which I would like to mention. Firstly, obviously YHBT YHL HAND. Now I've got that out of the way, there was a light weight comparative benchmark[1] on OSNews. Obviously, as you will see, they only tested some simple things, but the tests showed Sun's Java performed quite well when compaired to GCC. Compairing it to GCC is a little cheap though, as we know from the benchmarks in October edition of Dr Dobbs Journal[2]: GCC, it may have the best ANSI C/C++ support produces the bulkiest code and, although not always, slower code that a processor specific design such as Intel's C++ compiler. The have been a lot of myths[3] floating around about Java performance, and although IMO floating point precision sucks in Java, it is not as bad as people think. The main reason that I vote for Java and not my favorite C or my second favorite C++ is that the person who asks this question will just be adding more bad/exploitable C/C++ network code to the world. And I don't really think we should be encouraging that. [1] Nine Language Performance Round-up: Benchmark Math & File I/O - C.W. Cowell-Shah http://osnews.com/story.php?news_id=5602 [2] Compairing C/C++ Compilers - M. Wilson http://www.ddj.com/articles/2003/0310/ [3] Urban Performance Myths - B. Goetz http://www-106.ibm.com/developerworks/java/library/j-jtp04223.html?ca=dgr-lnxw01JavaUrbanLegends yours, Daniel (ever the debunker) blaze your trail -- redhat 'I am become Shiva, destroyer of worlds' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 21 22:28:36 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 21 Jan 2004 22:28:36 -0500 (EST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> Message-ID: > > Lest we get too enamored with CCLISP, there's also: > (roll of drums) > "Concurrent Prolog" > (Crash of cymbals, choir of angels) > > Having not actually looked at anything in Prolog for the last 15+ years, I > thought it would be interesting to repeat a short program here. (I have no > idea if it would work, since I don't recall CP Syntax) > > quicksort([X|Xs],Ys):- > partition (Xs?,X,Smaller,Larger), > quicksort (Smaller?,Ss), > quicksort (Larger?,Ls), > append(Ss>,[X|Ls?],Ys). > quicksort([],[]). > > partition([Y|ln],X,[Y|Smaller],Larger):- > X>=Y|partition(ln?,X,Smaller,Larger). > partition([Y|ln],X,Smaller,[Y|Larger]):- > X partition([],X,[],[]). > > append([X|Xs],Ys,[X|Zs]):- > append(Xs?,Ys,Zs). > append([],Xs,Xs). > > There is a scribbled note (not in my handwriting) that mentions the following: > - => indicates communications > name? => readonly variable, process waits until there is a value in the > name before proceeding (presumably this is some sort of barrier sync?) I had actually helped write a parallel version of Prolog that would automatically execute the standard Prolog quicksort in parallel. It had a very interesting run-time system that automatically load balanced the parallelization - at least I thought it was cool in a "self organizing Sante Fe Institute kind of way". There was (and still is to some degree) a good argument for such languages. For those who do not understand (or hate) Prolog, it operates by interpreting the program using a "resolution mechanism". The resolution mechanism is total transparent to the user. So if you parallelize the resolution mechanism, you have parallelized all Prolog programs (which does not necessarily mean they will run faster by the way). Indeed, it is even possible (and quite simple) to write meta-interpreters in Prolog to interpret Prolog in parallel - and thus play with the parallel resolution mechanism. The runtime performance issue has also been solved so that Prolog programs can run quite fast. The trouble is that it is very difficult for most programmers to get their mind around Prolog, but once you do, it can provide a very expressive and concise language in which to program (particularly database stuff). There is a gnu Prolog compiler that works rather well (plus a few other free versions as well). Alas, most discussions of "high level" approaches are not well received these days. Seems there is certain fashion to computer languages. Finally, if you are interested, some old papers on this subject can found at http://www.clusterworld.com/deadline.html. Doug > > ---------- > > In all of this, they make the very appropriate comments along the lines of: > sure it's horribly inefficient, but, you've got: > a) tons of computer power potentially available (I mean, you could have 128 > of those 8 MHz 286's spinning away) > b) you've got a moderately transparent way to distribute the load across > those processors without worrying too much about subtlety > c) human hours expensive|machine hours cheap > d) The computers WILL get faster, so don't agonize about CPU resources too > much.. time and the market will improve performance per dollar or wall > clock hour much faster than toiling over a hot keyboard ekeing out > algorithm tuning improvements. Spend your time on better high level > algorithms to solve "your problem". > > Hey, this was in 1986,1987 or so.. In the last 15 years, I don't think much > has really changed, philosophically. > > > > > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine Desk: 610.865.6061 Cell: 610.390.7765 Redefining High Performance Computing Fax: 610.865.6618 www.clusterworld.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Jan 22 00:42:07 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 22 Jan 2004 13:42:07 +0800 (CST) Subject: [Beowulf] SGE vs LSF vs PBS... In-Reply-To: <200401202306.07794.raiders@phreaker.net> Message-ID: <20040122054207.141.qmail@web16807.mail.tpe.yahoo.com> I did not buy LSF, but I did read the manuals of it. I compared the features of LSF and SGE, found that most of them are the same. One thing you need to understand is that if you want to have advanced scheduling algorithms, you need to use SGEEE. SGEEE is also free, but I remember there is a switch to turn on _during installation_ to get SGEEE, you can send a mail to the SGE mailing list. Another big difference is that LSF has global queues, but SGE's queue concept is different -- a queue is just a "container" of jobs. So the configuration may be different than your LSF cluster in some cases. HOWEVER, SGE 6.0 will provide LSF like queues to help people to migrate more easily, and it is coming soon too :) Since how easy or difficult it is to migrate to SGE depends on what you are looking for, the best is still to provide more info, but I think the beowulf list is not the best place to discuss about your batch system requirements. The SGE mailing list is a better place, as there are a lot of people there who used LSF but switched to SGE, like the author of GE monitor: http://www.veus.hr/linux/gemonitor.html SGE mailing list: http://gridengine.sunsource.net/project/gridengine/maillist.html Also, don't forget to find out more about SPBS. I don't have much experiece with SPBS, may be others who used LSF and SPBS can provide more info. Andrew. --- Shawn Raider ???? > I would be thankful if you could share your > experiences in replacing LSF with > SGE. I did study features etc from the respective > docs/sites etc. before > asking here. I am looking specifically for user > experiences. > > Thanks, > Shawn ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Jan 22 00:49:12 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 22 Jan 2004 13:49:12 +0800 (CST) Subject: [Beowulf] Fwd: Bio-cluster Linux distribution (BioBrew announcement) Message-ID: <20040122054912.82737.qmail@web16813.mail.tpe.yahoo.com> based on Rocks cluster. Andrew. --- Glen Otero wrote: > Hi folks- > > I just wanted to announce the release of BioBrew v. 1.0 for the x86 > architecture. The three CD iso images are freely available for > download > at > http://bioinformatics.org/ftp/biobrew, a Bioinformatics.org sponsored > > and > hosted website. > > BioBrew is an open source Linux cluster distribution based on the > NPACI > Rocks (www.rocksclusters.org) cluster distribution and enhanced for > bioinformaticists and life scientists. While BioBrew looks, feels, > and > operates like ordinary Red Hat Linux, it includes popular open source > clustering and bioinformatics software e.g. the NCBI toolkit, BLAST, > mpiBLAST, HMMER, ClustalW, GROMACS, PHYLIP, WISE, FASTA, and EMBOSS. > Installing BioBrew is a simple, straightforward procedure that allows > Linux beginners and gurus alike to go from bare metal to cluster in > the > shortest time possible. > > I used this version of BioBrew to build this Infiniband cluster for > Intel > and Mellanox for Supercomputing 2003: > http://www.hpcwire.com/hpcwire/hpcwireWWW/03/1114/106388.html. > > BioBrew v. 1.0 has the following features: > > - GM 1.6 and 2.0 support for Myrinet > - Infiniband support (binary drivers only) > - Automated compute node installation > - RPM software packaging and installation methods > - Cluster management tools > - MySQL database for storing cluster node information > - Cluster status monitoring with Ganglia > - SGE batch processing and job scheduler software > - Message passing interface software (MPICH & LAM) > - Parallel virtual machine (PVM) > - Parallel virtual filesystem support (PVFS) > - Virtual Machine Interface (VMI) 2.0. > - MPICH built for VMI > > The big improvement for 1.0 is the inclusion of VMI. I can't say > enough > about VMI. I encourage everybody that uses MPICH over multiple > interconnects to read the VMI docs I've posted. Basically, by > virtualizing > all interconnects, VMI allows you to compile your application once > against > a standard version of MPICH (mpich-vmi2), and run it over any > interconnnect. VMI supports ethernet, myrinet, and Infiniband. > > The README and installation pictorial provide sufficient installation > instructions. I'll be posting the complete User's Guide in a week or > so. > > BioBrew 3.0 for x86, ia64, and x86_64 is in early beta, but will be > posted > in mid to late February. > > Biobrew-Discuss at bioinformatics.org is an email discussion list for > all > things BioBrew. You can subscribe to the list at > http://bioinformatics.org/lists/biobrew-discuss. > > Glen > > (Sorry about any double posting) > > > Glen Otero, Ph.D. > Linux Prophet > 619.917.1772 > > _______________________________________________ > Bioclusters maillist - Bioclusters at bioinformatics.org > https://bioinformatics.org/mailman/listinfo/bioclusters ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian.dobbins at yale.edu Thu Jan 22 11:57:02 2004 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Thu, 22 Jan 2004 11:57:02 -0500 (EST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: Hi Brent, A while back, I had done a portion of what you were looking for, just to make my job easier while testing new kernels on our nodes. We had a small cluster connected to an APC masterswitch which could reboot the nodes (by powering them off, then on again). The nodes themselves were diskless, and the process was: a) If a newer image was located on the file server, reboot with the new image b) If we don't get a status OK from the node after 5 minutes (done via a simple file write in rc.local in the ramdisk), revert back to the old image. (This was all done on a fairly small scale, mind you.) For us, with diskless nodes, this was very easy - we didn't have to modify GRUB's boot parameters at all. I'm not even remotely knowledgeable about GRUB's options, but perhaps there's an easy way to do configure it to boot a specified kernel just once, then revert back to a 'safe' one. (You can do this in startup, if you're guaranteed startup will succeed, but that's not likely in your situation!) If there's a chance you can do this disklessly in terms of booting, let me know, I can see if I have any of the old configuration files still around. This doesn't mean you can't use disks, but with RAM so cheap these days, booting from a ramdisk doesn't cost you much, and makes some things a lot easier. You can always mount the disk during init. (Ie, you'd have the script called from the job manager execute a 'mkelf-linux' or 'mknbi' binary with a standard ramdisk, and the only thing that changes is the bzImage file supplied to the command. From memory, something like:) mkelf-linux --append="root=/dev/ram" ${bzImage_custom} RamDisk.gz mem=xG > /tftpboot/${user}_${date}_${bzImage_custom}.elf Unfortunately, I can't help you on the job manager stuff, but I'm sure there are plenty of people here who know heaps more and will certainly chime in. Good luck! - Brian Brian Dobbins Yale University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Thu Jan 22 10:29:09 2004 From: bclem at rice.edu (Brent M. Clements) Date: Thu, 22 Jan 2004 09:29:09 -0600 (CST) Subject: [Beowulf] Hypothetical Situation Message-ID: We have a request by some of our research scientists to form a beowulf cluster that provides the following abilities: 1. It must be part of a shared computing facility controlled by a batch queuing system. 2. A normal user must be able to compile a customized kernel, then submit a job with a variable pointing to that kernel. The batch queuing system must then load that kernel onto the allocated nodes and reboot the nodes allocated. 2a. If the node doesn't come back up after rebooting, the job must be canceled and the node rebuilt automatically with a stable kernel/image. 3. When the job is finished, the node must be rebuilt automatically using a stable kernel/image. The admin's here have come up with a name for this: "user provisioned os imaging". Our gut feeling is that this can be done but it will be a highly customized system. What is everyone's thoughts, experiences, etc etc concerning doing something like this? Thanks, Brent Clements Linux Technology Specialist Information Technology Rice University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 22 12:25:31 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 22 Jan 2004 09:25:31 -0800 Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> At 09:29 AM 1/22/2004 -0600, Brent M. Clements wrote: >We have a request by some of our research scientists to form a beowulf >cluster that provides the following abilities: > >1. It must be part of a shared computing facility controlled by a batch >queuing system. >2. A normal user must be able to compile a customized kernel, then submit >a job with a variable pointing to that kernel. The batch queuing system >must then load that kernel onto the allocated nodes and reboot the nodes >allocated. >2a. If the node doesn't come back up after rebooting, the job must be >canceled and the node rebuilt automatically with a stable kernel/image. >3. When the job is finished, the node must be rebuilt automatically using >a stable kernel/image. > >The admin's here have come up with a name for this: "user >provisioned os imaging". Our gut feeling is that this can be done but it >will be a highly customized system. What is everyone's thoughts, >experiences, etc etc concerning doing something like this? I think this cries out for diskless nodes!! The only moderately tricky part will be the mangement of the cluster so that you can feed off the right images when the node reboots and asks for its image. Some fairly simple database table of MAC, Node ID, image, etc. would probably take care of it. Obviously, you'll need some sort of remote controlled booting (power cycle?). Possibly, you might want to have centrally managed network switches so you can isolate the subclusters, in case someone builds a kernel that runs amok. You'll need to spend a few work weeks/months building some nice quasi automated scripts and documentation to help the folks build their kernels and put them into an appropriate form. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Florent.Calvayrac at univ-lemans.fr Thu Jan 22 12:11:15 2004 From: Florent.Calvayrac at univ-lemans.fr (Florent Calvayrac) Date: Thu, 22 Jan 2004 18:11:15 +0100 Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: <40100433.4000903@univ-lemans.fr> Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? > > with PXE and/or bproc, (two kernel monte) I see it easily. If the node doesn"t come up, just send a magic packet to reboot it and download another kernel. -- Florent Calvayrac | Tel : 02 43 83 26 26 Laboratoire de Physique de l'Etat Condense | Fax : 02 43 83 35 18 UMR-CNRS 6087 | http://www.univ-lemans.fr/~fcalvay Universite du Maine-Faculte des Sciences | 72085 Le Mans Cedex 9 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 22 13:00:11 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 22 Jan 2004 13:00:11 -0500 (EST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: On Thu, 22 Jan 2004, Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? See the "COD" (Cluster on Demand) project in the Duke CPS department (Jeff Chase, primarily, along with his student Justin Moore). It is PRECISELY what you are looking for. I mean precisely. I don't know that current status of the project, but that's just what COD is designed for. Justin Moore, BTW, is the ex-protege of this list's own Greg Lindahl, and a very savvy computer guy. http://issg.cs.duke.edu/cod/ >From its summary: Users of a shared cluster should be free to select the software environments that best support their needs. Cluster-on-Demand (COD) is a system to enable rapid, automated, on-the-fly partitioning of a physical cluster into multiple independent virtual clusters. A virtual cluster (vcluster) is a group of machines (physical or virtual) configured for a common purpose, with associated user accounts and storage resources, a user-specified software environment, and a private IP address block and DNS naming domain. COD vclusters are dynamic; their node allotments may change according to demand or resource availability (This includes shipping each node its own custom kernel and filesystem image -- full setup and boot into cluster, followed by teardown and restoration of the resource to the pool when done, "arbitrary" kernels, operating systems, and runtime node images.) I'm very peripherally connected to this project, as in we all talk about it, but I haven't contributed any code, only (maybe) a very few ideas. Justin is probably who you'd want to talk to: http://www.cs.duke.edu/~justin/ He's on this list; maybe he'll speak up if he's around. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From epaulson at cs.wisc.edu Thu Jan 22 11:21:06 2004 From: epaulson at cs.wisc.edu (Erik Paulson) Date: Thu, 22 Jan 2004 10:21:06 -0600 Subject: [Beowulf] Hypothetical Situation In-Reply-To: ; from bclem@rice.edu on Thu, Jan 22, 2004 at 09:29:09AM -0600 References: Message-ID: <20040122102105.A5743@chopin.cs.wisc.edu> On Thu, Jan 22, 2004 at 09:29:09AM -0600, Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? > Auto rebuilding with custom images isn't difficult. The Utah folks probably have the best complete setup for this: http://www.emulab.net/index.php3 Read "An Integrated Experimental Environment for Distributed Systems and Networks" from OSDI 2002, it's a good overview. Their setup is aimed more at building a network-emulation testbed, but it will provision up machines, install the right OS, reboot nodes, muck with switches, etc etc. Tossing a batch system into that image wouldn't be hard (it'd probably be a bit tricky with PBS, since it doesn't like nodes coming and going, Condor would work fine, and I know nothing about LSF or SGE.) -Erik _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmoyer at redhat.com Thu Jan 22 12:59:14 2004 From: jmoyer at redhat.com (Jeff Moyer) Date: Thu, 22 Jan 2004 12:59:14 -0500 Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: <16400.3954.325319.622862@segfault.boston.redhat.com> ==> Regarding [Beowulf] Hypothetical Situation; "Brent M. Clements" adds: bclem> We have a request by some of our research scientists to form a bclem> beowulf cluster that provides the following abilities: bclem> 1. It must be part of a shared computing facility controlled by a bclem> batch queuing system. 2. A normal user must be able to compile a bclem> customized kernel, then submit a job with a variable pointing to bclem> that kernel. The batch queuing system must then load that kernel bclem> onto the allocated nodes and reboot the nodes allocated. 2a. If the bclem> node doesn't come back up after rebooting, the job must be canceled bclem> and the node rebuilt automatically with a stable kernel/image. bclem> 3. When the job is finished, the node must be rebuilt automatically bclem> using a stable kernel/image. Check out STP. Besides being a pretty good band, it stands for Scalable Test Platform. http://sourceforge.net/projects/stp/ -Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Thu Jan 22 13:58:26 2004 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Thu, 22 Jan 2004 19:58:26 +0100 (CET) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: On Thu, 22 Jan 2004, Brent M. Clements wrote: > 1. It must be part of a shared computing facility controlled by a batch > queuing system. Use the "epilogue" facility to run a script that installs a sane image and then reboots or first reboots and PXE+something (recent thread mentioned what this something can be :-)) takes care of the rest. The trickier part is starting the user compiled kernel, because the "epilogue" of a job would need to know about the requirements of the next job. Making the change in the "prologue" of the user's job might not work properly because the node is rebooted in the process of changing the kernel and this breaks the link to the batch system daemon... > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. I did something similar for a test and after I realized what are the implications I never put it into production, even though I would have allowed only kernels compiled by myself :-) Basically, user compiled kernel means that the kernel can do anything; that means reading the content of /etc/shadow and dumping it into an unprotected file, disturbing the network activity on the switch(es) where it's connected, overloading file-servers, making connectiong from priviledged ports (which can upset the batch system :-)), destroying local file-systems. It can even go to destroying hardware by driving it with out-of-spec parameters. Completely hijacking the node is another thing that can be done; there's no guarantee with a user compiled kernel that the batch system daemons are started, for example, so the master node might loose any control over the node, requiring a human to press a button. Now if these kernels come only from trusted persons, there are still many things that can go wrong. For example, running with a RHEL 3.0 user space and a non-NTPL kernel will break threaded applications. Compiling most (all) drivers in-kernel while user-space expects modules might break the whole system. Not including IP autoconfig when root FS is over NFS will not allow booting. And so on... > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. "doesn't come back up after rebooting" is a pretty vague description. Probably a watchdog (even the software one) would help, but might still not catch all cases. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. If the user compiled kernel plays nice and gives back the node to the batch system... just reboot and instruct PXE to give a sane kernel/image. You might have better chances of keeping things under control using some kind of virtualization technique, UML or VMWare/Bochs. Limits can be imposed on the host system, but performance might go down a lot. One solution that I used at some point and will probably use again in the near future is to have a (small) set of sane kernels that are allowed for such purposes; that would allow for example conflicting drivers (like GM and PM for Myrinet cards) to be used - for each job the kernel that corresponds to the right driver is chosen. This greatly reduces the risks as the kernels are compiled only by the admin who is supposed to know what is doing :-) -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Jan 22 12:30:21 2004 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 22 Jan 2004 18:30:21 +0100 (CET) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: On Thu, 22 Jan 2004, Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? I've always wondered if there is a place for UML (User Mode Linux) in cluster or grid computing. This might be an interesting area to apply it - though I admit it would take effort. If you are not familiar with the concept, UML allows Linux to be run in a virtual machine. SO the real machine can be running the standard distro. The users get their own machine, and run it under their control. So what they would have to do is cmpile up their own kernel, and UML instance. On a more conventional Beowulf setup, what you would have to do is make a software image with the custom kernel. Reboot the machine, install that image. Run the job, change the image which should be installed on that node then reboot. We support per-node chhsing of the software image to be installed, and I'm sure lots of others do also. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Thu Jan 22 14:02:48 2004 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Thu, 22 Jan 2004 20:02:48 +0100 (CET) Subject: [Beowulf] Hypothetical Situation In-Reply-To: <40100433.4000903@univ-lemans.fr> Message-ID: On Thu, 22 Jan 2004, Florent Calvayrac wrote: > If the node doesn"t come up, just send a magic packet to reboot it Magic packet only wakes up a computer. There is no magic packet to reboot or turn off (or at least put back to sleep) a computer. Imagine the security consequences... -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at comcast.net Thu Jan 22 16:22:02 2004 From: laytonjb at comcast.net (Jeffrey B. Layton) Date: Thu, 22 Jan 2004 16:22:02 -0500 Subject: [Beowulf] Hypothetical Situation In-Reply-To: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> Message-ID: <40103EFA.6060900@comcast.net> Jim Lux wrote: > At 09:29 AM 1/22/2004 -0600, Brent M. Clements wrote: > >> We have a request by some of our research scientists to form a beowulf >> cluster that provides the following abilities: >> >> 1. It must be part of a shared computing facility controlled by a batch >> queuing system. >> 2. A normal user must be able to compile a customized kernel, then >> submit >> a job with a variable pointing to that kernel. The batch queuing system >> must then load that kernel onto the allocated nodes and reboot the nodes >> allocated. >> 2a. If the node doesn't come back up after rebooting, the job must be >> canceled and the node rebuilt automatically with a stable kernel/image. >> 3. When the job is finished, the node must be rebuilt automatically >> using >> a stable kernel/image. >> >> The admin's here have come up with a name for this: "user >> provisioned os imaging". Our gut feeling is that this can be done but it >> will be a highly customized system. What is everyone's thoughts, >> experiences, etc etc concerning doing something like this? > > > I think this cries out for diskless nodes!! The only moderately tricky > part will be the mangement of the cluster so that you can feed off the > right images when the node reboots and asks for its image. Some fairly > simple database table of MAC, Node ID, image, etc. would probably take > care of it. Obviously, you'll need some sort of remote controlled > booting (power cycle?). Possibly, you might want to have centrally > managed network switches so you can isolate the subclusters, in case > someone builds a kernel that runs amok. I've got just the thing for this - Warewulf. WW can already handle hetero diskless clusters. I'm working on a write-up on how to do. I've tested it on my home cluster and it works fine. Enjoy! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Thu Jan 22 18:33:12 2004 From: bclem at rice.edu (Brent M. Clements) Date: Thu, 22 Jan 2004 17:33:12 -0600 (CST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: We have actually come up with an idea called a "kernel preload" system based on ideas from alot of places. Basically what we have is as follows: The system admins provide the source code to a bunch of "stock" kernels known to work with the hardware we are using in the cluster. The user takes a copy of this source code and makes his/her changes. We have a script that's called skj(submit kernel job). This is a perl script that does the following 1. Makes a reservation within job scheduling system for 1 hour, reserving what ever number of nodes the user requests. 2. Creates a patch from their modified source code and then builds the modified kernel. Please note that the script will do tests to make sure that the patch file does not remove any needed hardware drivers to ensure that the machine will actually boot. 4. The script makes it a tftp'd kernel and modifies the dhcp file to reflect the kernel changes for the nodes reserved. 5. The machines are power cycled/rebooted 6. The script waits up to 15 minutes for the machines that have the new kernel installed to come back up. If after 15 minutes there are no responses from any of the machines, the script reblasts the system with the stock kernel and outputs an error message. We will actually speed up the boot process by installing linuxbios, so our waittime will be like 5 minutes. 7. But If everything goes well, the script modifies the reservation and creates a pbs job script, or gives them an interactive session to the allocated nodes. Voila, we have solved our issue. This could also work if we wanted to actually reblast an entirely new image onto the allocated nodes rather than just the kernel. Remember our requirements/issue was that we needed this to work easily with our job schedulars(pbs/maui). I am going to be posting a document outlining the requirements,and the solution next week if anyone is interested. Thanks to everyone who helped out! Sincerely, Brent Clements Linux Technology Specialist Information Technology Rice University On Thu, 22 Jan 2004, Mark Hahn wrote: > > You'll need to spend a few work weeks/months building some nice quasi > > automated scripts and documentation to help the folks build their kernels > > and put them into an appropriate form. > > I can't imagine why this would take even a day of work, > assuming you use plain old PXE to boot the nodes, already > have a table of MAC addrs, and are comfortable with simple > suid script programming: > > pushkernel(machinename, kernelfile): > > $mac = grep $machinename mac-table-file > ln -s $kernelfile /var/tftp/pxelinux.cfg/$mac > powercontrol -r $machinename > sleep 180 > if (cant ping $machinename) > ln -s fallbackimage /var/tftp/pxelinux.cfg/$mac > > (powercontrol is a simple perl script I use to talk to my APC power thingies.) > hooking this into a batch system shouldn't be hard. > > physically installing the kernel on a machine and booting it locally > seems pointless. "lilo -R" is great for trying out new kernels, though. > I understand someone has added this feature to grub, but haven't seen it myself. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Thu Jan 22 18:18:05 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Thu, 22 Jan 2004 18:18:05 -0500 (EST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> Message-ID: > You'll need to spend a few work weeks/months building some nice quasi > automated scripts and documentation to help the folks build their kernels > and put them into an appropriate form. I can't imagine why this would take even a day of work, assuming you use plain old PXE to boot the nodes, already have a table of MAC addrs, and are comfortable with simple suid script programming: pushkernel(machinename, kernelfile): $mac = grep $machinename mac-table-file ln -s $kernelfile /var/tftp/pxelinux.cfg/$mac powercontrol -r $machinename sleep 180 if (cant ping $machinename) ln -s fallbackimage /var/tftp/pxelinux.cfg/$mac (powercontrol is a simple perl script I use to talk to my APC power thingies.) hooking this into a batch system shouldn't be hard. physically installing the kernel on a machine and booting it locally seems pointless. "lilo -R" is great for trying out new kernels, though. I understand someone has added this feature to grub, but haven't seen it myself. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From anand at mecheng.iisc.ernet.in Fri Jan 23 00:10:43 2004 From: anand at mecheng.iisc.ernet.in (Anand TNC) Date: Fri, 23 Jan 2004 10:40:43 +0530 (IST) Subject: [Beowulf] cluster on suse In-Reply-To: <40103EFA.6060900@comcast.net> Message-ID: Hi, I'm new to clustering...does anyone know of some clustering software which works on Suse 8.2 or Suse 9.0? Thanks regards, Anand -- Anand TNC PhD Student, Engine Research Laboratory U-55 IISc Hostels, Dept. of Mechanical Engg., Indian Institute of Science, Indian Institute of Science, Bangalore 560 012. Bangalore 560 012. Ph: 080 293 2591 Lab Ph: 293 2352 080 293 2624 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From siegert at sfu.ca Fri Jan 23 12:37:34 2004 From: siegert at sfu.ca (Martin Siegert) Date: Fri, 23 Jan 2004 09:37:34 -0800 Subject: [Beowulf] cluster on suse In-Reply-To: References: <40103EFA.6060900@comcast.net> Message-ID: <20040123173734.GA17568@stikine.ucs.sfu.ca> Hi Anand, On Fri, Jan 23, 2004 at 10:40:43AM +0530, Anand TNC wrote: > I'm new to clustering...does anyone know of some clustering software which > works on Suse 8.2 or Suse 9.0? You actually do not have to look very far: SuSE comes with a lot of software for clusters. Here is a list of SuSE-9.0 rpms you may want to look at: mpich-1.2.5-241.i586.rpm mpich-devel-1.2.5-241.i586.rpm lam-7.0-77.i586.rpm lam-devel-7.0-77.i586.rpm fftw-mpi-2.1.3-960.i586.rpm fftw-mpi-devel-2.1.3-960.i586.rpm OpenPBS-2.3.16-466.i586.rpm OpenPBS-clients-2.3.16-466.i586.rpm OpenPBS-devel-2.3.16-466.i586.rpm OpenPBS-mom-2.3.16-466.i586.rpm OpenPBS-scheduler-2.3.16-466.i586.rpm OpenPBS-server-2.3.16-466.i586.rpm OpenPBS-utils-2.3.16-466.i586.rpm and probably more. As an aside: that version of OpenPBS is actually by far superior to the version that you can download from the PBS web site. The SuSE rpm has patches for many bugs in the (unsupported) openpbs source code. Thus, even if you are not running SuSE you may want to download their OpenPBS src.rpm and recompile it for your distribution (alternatively you could use SPBS). Cheers, Martin -- Martin Siegert Manager, Research Services WestGrid Site Manager Academic Computing Services phone: (604) 291-4691 Simon Fraser University fax: (604) 291-4242 Burnaby, British Columbia email: siegert at sfu.ca Canada V5A 1S6 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From afant at geekmail.cc Thu Jan 22 11:29:30 2004 From: afant at geekmail.cc (Andrew Fant) Date: Thu, 22 Jan 2004 11:29:30 -0500 Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: <34120000.1074788970@flux.usg.tufts.edu> Brent, Good to see someone from Rice on here. I spent a couple years at Rice in the mid-ninetys, myself in Biochemistry and Cell Biology. To be clear, are you wanting to use a "real" beowulf with single system image and unified process space, or are you more interested in a compute farm, where each system runs it's own copy of the OS? If it's the former, I can't be much help. If it is the latter, is there a reason why user-mode linux is not an option for the researchers? Are they part of the CITI crew and doing hardcore OS work, or are they computational science types who have software that is just very anal about kernel, etc. versioning? It would probably be far easier to keep a stable minimal OS on each node and mount loopback filesystems that the users keep their custom environments in. Alternately, if you have to go the bare-metal route, systemimager might be an option, if the delay in time to reimage the system after job submission is not deemed excessive; in any case you may need to examine a fast dedicated network for reimaging to keep the time cost low. Normally, I am a big fan of LSF, but from the sounds of the demands, you would probably do better to work with SGE (Grid Engine), since the situation you describe would require a couple of reboots if real installs are needed. I don't think any available batch system can handle reboots as part of a prelude/prologue script cleanly, but with SGE, you at least have a chance to hack the code to try and fix it. Have you given any thought to which distro you intend to use? I will admit to being a Gentoo fanatic, but it sounds as if you might want to look into it. Gentoo is a source-based Linux distribution that allows the installation to be fairly tightly tailored to the application at hand. If the users are demanding their preferred kernels, I wouldn't be surprised if they started wanting their own tool-chains and application versions in the near future. Gentoo would allow them to build exactly what they want in a chrooted directory that could then either be used as the loopback filesystem for uml, or as the golden client to upload to systemimager. You can find out more from www.gentoo.org or check on irc.freenode.net in #gentoo-server. This sounds like an interesting problem. I'd be glad to answer more questions or serve as a sounding board if you want. Is Cathy F. still working on networking stuff in Mudd? If she is, and you see her, please give her my regards. Hope this Helps, Andrew Fant Tufts University --On Thursday, January 22, 2004 09:29:09 -0600 "Brent M. Clements" wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? > > Thanks, > > Brent Clements > Linux Technology Specialist > Information Technology > Rice University > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Fri Jan 23 23:09:59 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 24 Jan 2004 12:09:59 +0800 (CST) Subject: [Beowulf] SGE vs LSF vs PBS... Message-ID: <20040124040959.85793.qmail@web16806.mail.tpe.yahoo.com> I almost forgot, Platform Computing is willing to cut the price of LSF, what used to be $1000 is now $50 per CPU. I think you might want to stay with LSF if your admin likes it, since switching means new trainings, service intereptions, and user complains. You can get that discount provided that you are from an edu site and buy enough. http://www.bohnsack.com/lists/archives/xcat-user/2104.html I believe they will cut the price even more when SGE 6.0 comes out later this year, or SPBS gets more popular. This is good news since we will benefit. Andrew. > But we can't afford that kind of license > pricing. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Fri Jan 23 09:41:28 2004 From: jakob at unthought.net (Jakob Oestergaard) Date: Fri, 23 Jan 2004 15:41:28 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: References: <20040120235009.GD30005@unthought.net> Message-ID: <20040123144127.GA814@unthought.net> On Wed, Jan 21, 2004 at 11:46:15AM -0500, Robert G. Brown wrote: ... > > I find your lack of faith disturbing... ;) > > Trying an old jedi mind trick on me, are you? > > At least your metaphor is correct. You are indeed attracted to the > Power of the Dark Side...:-) Hey! Making jokes when we're having an *argument* - now that's just plain unfair! ;) ... > Well, we COULD make a list of all the major programs involving > 1KLOC > (exclusive of comments, even) that have been written entirely in C (plus > a small admixture, in some cases, of assembler). Unfortunately, the > list would be too long (I'd be here literally all day typing), and would > include things like the kernels of pretty much all the major operating > systems (certainly all the Unix derivatives and variants) and gcc > itself, including its g++ extension. For good reason too See - C++ was standardized in 1998 - but until around 2001 or so, there were no decent compilers generally available. Even today, "certain" platforms still have problems on the C++ compiler front. How many compilers and operating system kernels have been written from scratch since, say, 2001? Yeah, probably some, but none that I know of either :) As I see it, C++ is a very new language when you consider languages that are "mature" and "practically usable". That's why I understand (and appreciate) that big things like the Linux kernel or GCC are written in C - anything else at that time would have been plain silly. > > To be fair, that doesn't prove that your observation is wrong -- there > is a rather huge list of fortran sources available in my primary field > of endeavor, for example, which doesn't stop fortran from being a cruel > joke. Did he just say the F-word? ;) (oh, the memories of parallelizing serial chemistry code in Fortran pre-77 from Cray to SP/2 with MPI.... The guy who wrote it had kept the Amdahl-induced line serial numbers after column 73 in his code as a sort of revision control - so he could see which lines he wrote before and after 1970... I have a big scar on my chest from back then... The scars in my mind will probably never heal though ;) ... > What you probably mean is that IF everybody knew how to program in C++, > they would have written all of this in C++, right? *if* they knew the language and *if* they started over from scratch today, then 100% affirmative *yes*. (Starting a new project was the spawn of this thread) > Let's see, is there > a major operating system out there that favors C++? Hmmmm, I believe > there is. Are it and its applications pretty much perpetually broken, > to the extent that a lot of its programmers have bolted from C++ and use > things like Visual Basic instead? Could be. What operating system would that be? :) I make a living from writing a piece of commercial software that runs on a bunch of different operating systems, one of them being the one you mention. We use the Intel C++ compiler on that platform. We have to. Because the opearting system vendor do not have a C++ compiler themselves - they do have a compiler called "C++", but it does not understand the language. We only have two major problems left with C++ support on that platform; that Intel rely on the OS vendor's linker and STL - those are the two weak links that are left. So, honestly, I cannot say that they favour C++. If they do, they have one sick way of showing it ;) They do favour "bad C with extensions" though - I've seen a lot of such code (on many platforms) being labelled as C++, so that might be where the confusion comes from ;) Now, you can look at my statements as I'm arguing for a solution which would fit in an ideal world (which we do not live in) - this is not my intention. I am just trying to point out that things have changed the past 10 years, and what was true about a language under development 10 years ago just may not be the entire truth about what that language has evolved into today. > This isn't intended to be a purely a joke observation. I would rather > advance it as evidence that contrary to expectations it is MORE > difficult to write and maintain a complex application in C++. I think this boils down to how well written the application is - and that will hold true for any language out there. I have seen unreadable ML - something I was told at university was impossible (yeah, I didn't believe them either). > The very > features of C++ that are intended to make an application "portable" and > "extensible" are a deadly trap, because portability and extensibility > are largely an illusion. Maybe I misunderstand, or maybe you know a dark corner of C++ that I don't (that's not impossible at all :) But for platform portability I use makefile rules to compile different implementations of code prototyped in a common header file. Same as I do in C. ... > To go all gooey and metaphorical, programming seems to be a holistic > enterprise with lots of layering and feathering of the brush strokes to > achieve a particular effect with C providing access to the entire > palette (it was somebody on this list, I believe, that referred to C as > a "thin veneer of upper-level language syntax on top of naked > assembler"). C++, in the same metaphor, is paint by numbers. It > ENCOURAGES you to create blocks to be filled in with particular colors, > and adds a penalty to those that want to go in and feather. Hmm.... I would say, C++ allows you to write "quite" efficient code really darn easily. If "quite" is not good enough, it allows you to do what C would have forced you to; to write the stuff yourself. With no run-time performance difference between C and C++. You can say C++ might encourage people to accept the "quite" efficient solution, whereas a "real" C programmer would have spent that extra week writing yet another AVL tree implementation instead of using the existing RB tree provided by the STL. But there is nothing that stops you from spending that week and re-inventing the AVL tree in C++ either. Now some C programmers would say "but there are standard implementations for all those data structures for C as well!" - the really funny part being, that all generic data structure implementations in C that I have seen, use void pointers to point to key/data members, which makes it impossible for the compiler to do clever inlining, and therefore a type-agnostic C implementation of an RB tree will actually be run-time *slower* than the corresponding STL C++ RB tree ;) This is the sort of thing that made me move from C to C++ (where feasible - yes, part of my job is also writing software that will actually function on a number of platforms including NetWare, so I do write and maintain a good piece of C code as well ;) ... > a) Do C's basic object features (typedefs, structs and unions, for > example) suffice to meet the needs for object creation and manipulation > in those code segments where they are needed? I would say of course > they do. I agree. But I must add, that for example having "destructors" can take a lot of pain out of memory management, when used cleverly and appropriately. I have not found any usable substitute for that in C. (Yes, you can write "destructor functions" and call them instead of free(), it will work but it is less convenient). Once you start using exceptions in error handling, destructors can become quite essential in assuring that memory leaks cannot happen. > IMO, protection and inheritance are a nuisance more often than > an advantage because, as I observed, real "objects" are almost never > portable between programs (exceptions exist for graphical objects -- > graphics is one place where OO methodology is very appropriate -- and > MAYBE for DB applications where data objects are being manipulated by > multiple binaries in a set). "protection" in particular I think of as > being a nuisance and more likely to lead to problems than to solutions. > In a large project it often translates into "live with the bugs behind > this layer, you fool, mwaahhahahaha". Result: stagnation, ugly hacks, > bitterness and pain. In single-developer projects, just who and what > are you protecting your structs FROM? Yourself? Never heard that argument before - but it's a good one. I do not see protection as being a powerful feature in C++ (but I have not thought about it before now - scary, you make me think! ;) I see protection more as a way of commenting your code, with a compiler assisted check that someone read your 'comment'. For example, "private" means, "when you use this class/struct, you're not supposed to be referring directly to this member". Someone new to the project can look at the struct/class in question, and will know (assuming it is well written code), that probably the original author had a good reason for not wanting others to address those members directly. So, I guess one could think of protection as "enforced comments". In a sense one could argue that this is an unnecessary language feature, because it make up for three reserved words (private, protected, public), and only allows us to "enforce comments" in one particular (but rather common) case. Anyway, it's there. It does not give run-time overhead. Use it if you want to. Or just use 'struct' if you feel better with that. These two type declarations are *identical* (will yield exactly the same code, memory layout, run-time performance, everything...): struct A { B b; }; class A { public: B b; }; If what you need is the first, it'd be silly to write the longer one ;) I have quite a lot of C++ code which obviously uses the first and shorter example. Anything else would just be plain stupid. > > b) Do C++'s basic object features increase or decrease the efficiencies > of the eventual linked binaries one produces? As you say, C++ is more > than an "extension" of C, it has some real differences. In particular, > it forces (or at least, "encourages") a certain programming discipline on > the programmer, one that pushes them away from the "thin veneer" aspect > of pure C. I disagree that it "forces" anything. This is a delusion, but a common one at that. I do agree though, that it can be difficult to keep people "sharp", when you use a language that allows you to write really bloated code easily. In C it is more difficult to write code that magically results in about a billion nested function calls, even more dynamic memory allocations, etc. etc. In C++ this can be a trap. In order to write lean code in C++, you need to have some sort of dicipline (or a screaming violent psychopath in the corner of the office - that's me ;) So, in a way I agree that C++ can be more dangerous wrt. writing poorly performing code. But it is not a language limitation, it is a disciplinary problem :) The same features that lead to this trap, are the same features that allow you to write code that is as fast as "C the way you would have written it if you had had the time", correct and maintainable. If the programmer does not understand the language (ie. when to use what), then I guess he has a good chance of writing something that's really crap. But to a large extent that would go for any other language as well. The real incompatibilities from C to C++ (except for a few reserved words which do cause real (but easy to remedy) problems), are mostly C braindamages that noone really depends on anyway - at least this is my experience... If anyone has counter examples, I'm always happy to be enlightened. For example: struct A { struct B { int a; }; }; struct B b; The above is valid C and invalid C++, because C++ does not forget scope checking in this special case. C++ would require you to write: struct A::B b; Now, I'd be rather surprised if something like this is going to make all your C code not compile with a C++ compiler ;) > I think it is clear that each additional layer of contraint > INCREASES the intrinsic complexity of the compiler itself, thickens the > veneer, and narrows the range of choices into particular channels. It absolutely increases the complexity of the compiler front-end. Compiling C++ just takes longer. That's a real and practical problem (but a good excuse for a Beowulf ;) However, the language was carefully designed so that "you pay for what you use". If you take a piece of C code that happens to also be valid C++, and compile it with g++ instead of gcc, there is no valid reason for the resulting code to be slower than the original gcc-compiled code. > Those channels in C++ were deliberately engineered to favor somebody's > idea of "good programming practice", which amounts to a particular > tradeoff between ultimate speed and flexibility and code that can be > extended, scaled, maintained. So I would expect C++ to be as efficient > as C only in the limit that one programs it like C and to deviate away > as one uses the more complex and narrow features. One very powerful feature of C++ is templates. It has no run-time overhead, compared to how you would solve the problem in C. They do allow more complex problems to be solved, where in C you would fall back to using void pointers and casts (instead of the proper types), which would give you a *slower* C implementation than what C++ makes feasible. Let me throw out a challenge: "Write a program in C that reads from standard input until EOF, and prints the number of distinct words found in that input" I give you my solution here, in C++: ---------------------------------------- #include #include #include int main(int argc, char **argv) { // Declare and Initialize some variables std::string word; std::set wordcount; // Read words and insert in rb-tree while (std::cin >> word) wordcount.insert(word); // Print the result std::cout << "Words: " << wordcount.size() << std::endl; return 0; } ---------------------------------------- I use an RB-tree to store the words I find, and print the number of elements in the final tree. A comparable C solution shouldn't be more than a few thousand lines ;) Oh, right, I cheated and used STL. Well, STL is a big part of why C++ is really handy, but go right ahead and use "glib" or whatever. I still guarantee you two things: 1) Your code will be longer 2) Your program will be slower As always, I love to be proven wrong ;) > Efficient "enough", > sure, why not? CPU clocks and Moore's Law give us cycles to burn in > most cases. And when one uses objects in many cases, they ARE cases > where bleeding edge efficiency IS less important than ability to create > an API and "reuse" data objects via library calls (for example). Still > I think C would have to win here. I'm just dying to see that piece of C which cannot possibly be made to parse as C++, without fundamentally changing the structure of the code. ...[lots of good arguments nipped for brevity]... > unchanneled by the compiler. Yup, I understand what you mean. And I'm not arguing that C++ is some magic silver bullet either. All I'm saying is, that I think for 99% of the larger projects you start today, C++ can provide you with some really convenient benefits over C, with no practical downsides. That is, if you know the language, if the people who will maintain your code knows the language, and if you can rely on having a reasonably good compiler. These are definitely practical obstacles out there in the scary real world of coding, and I am aware of that. Just trying to point out that *if* those practical obstacles can be ovecome, then there *is* practical benefits. And today, just *maybe* it is not universally a bad idea to at least consider whether the practical obstacles can be overcome in order to reap the benefits that can be provided. Uh, I'm starting to sound like some middle-ground-searching tree-hugging weed-smoking hippie. That was not my intention! ;) > Do I have to deal with sometimes screwing up data typing? Absolutely. > Do I have to occasionally deal with working my way backwards through my > programs to redo all my structs? Of course (I'm doing it right now with > a program I'm writing). This process is INEVITABLE in real-world > program design because we DON'T know a priori what the best shape is for > the data objects is until we've programmed them at least once the wrong > way, in most cases. The really interesting question is whether or not > this PROCESS is any easier in C++ than in C. I can't see why it would > be, but perhaps it is. In my experience, not much, if at all. You screw up, decide to rewrite, and rewrite. One thing which could give C++ an edge (depending on your program of course), is, if your code is 1/10th the size in C++ - you would have less code to rewrite :) ... > This is a lovely tradeoff in a lot of cases, which is why I hesitate to > actually stick a rod up java's ass and barbeque it on a spit. I really > do believe the BASIC argument associated with OO programming, that the > data and what you want to do with it should dictate program design > features, including choice of language and programming technique. For > many problems perl or python provide a very good data interface where > one DOESN'T have to mess with lots of data issues that are a major chore > in C. Wanna make a hash? Use it. Wanna make your hash into an array? > Change the little % TO A @ and hack your dereferencing a bit. Is your > hash actually stored as a linked list, as an array of structs? Don't > ask, don't tell, don't try to look. All this and direct syntactical > manipulation of regular expressions, what's not to love? A hash of hashes, is not to love in perl ;) At least, that's one thing they got right with STL and templates - templates may not have the prettiest syntax imaginable, but at least it's consistent. A map from int to int: std::map A; a[42] = 5; A map from int to a map from int to int: std::map > A; A[42][5] = 8; I can't remember how many explicit '{' and '%' you'd need in perl to do that last one, but I do remember it wasn't pretty ;) To be fair, I have never seen anything remotely as powerful as Perl's regular expression handling. That's just sweet. Unreadable, but sweet. ... > > > I prefer to think of "C++" as "A better C", rather than a "C extension", > > as not all C is valid C++, and therefore C++ is not really an extension. > > Funny that. I tend to think of C++ as "A broken C extension" for > exactly the same reason;-) Good one! > > If they hadn't broken it, then there really would be no reason not to > use it, as if C were a strict subset of C++ all the way down to the > compiler level, so that pure C code was compiled as efficiently as pure > C is anyway then sure, I'd replace all the gcc's in my makefiles with > g++'s. That would certainly lower the barrier to using C++; C > programmers could program in C to their heart's content and IF AND WHEN > their program needs a "C++ object" they could just add it without > breaking or even tweaking the rest of their code. Have you actually tried this and found that your code would not compile with g++ ? Anyway, I'm not arguing that all that is C should become C++. C has its uses (sort of like automatic weapons ;) I think for new projects C++ is worth considering. If I had a 10KLOC C program that worked really well, I probably wouldn't bother to convert it (even if no conversion was necessary). Why risk to break the application on some obscure platform out there, if you are not going to take advantage of the new language features anyway? I've never tried such a conversion on a large program, and for the small programs where I did it I ended up pretty much rewriting them anyway (having 60% void* "generic" code with casts, and 40% template-based code is just *messy* to look at). > Breaking C was (IMO) a silly design decision. Well, yes, but I believe the largest brekage is due to the new keywords, which really coulnd't have been avoided (unless "static" should have gotten yet another five-ten different meanings depending on which context you use it in ;) The other breakage (my scope example earlier) is more like a braindamage in C that virtually noone depends on which was fixed. I like that. > So was adding crap like a > completely different stdin/stdout interface, which makes most C++ code, > even that which doesn't use objects in any way, most > non-back-portable-or-compatible to C. The C library is readily available, and writing fprintf() instead of std::cout in your C++ code does not make you a bad person :) > It is very clear that these were > all deliberate design decisions INTENDED to break C Ok, Bjarne explains quite well in his books why each breakage was introduced - they did deliberately break compatibility, but they paid a lot of attention to not breaking "more than absolutely necessary". You may disagree with the definition of "absolutely necessary" of course - I do too at times. > and FORCE > programmers to make what amounts to a religious choice instead of > smoothly extend the palette of their programming possibilities. I'd > honestly have to say that it is this particular aspect of C++ more than > any other that irritates me the most. Ok. I just do not see the breakage as such a deliberate attempt to piss people off ;) ... > > > > Amen! This goes for C++ as well though. > > Y'know, you won't believe this, but I actually added the (and C++) > references in my original reply just thinking of you...;-) Hehe, thanks ;) > Last time we had this discussion you were profound and passionate and > highly articulate in advancing C++ -- so much so that you almost > convinced me. :) > Alas, that silly barrier (which I recall your saying took > you YEARS to get over yourself)... I just bounce right off of it in a > state of total irritation every time I try. I understand. I don't know of any easy way to do that. Having too much time on ones hands is a good start I think. Yeah, I had that once, I can almost remember it, if I think about it hard ;) It was worse when I moved from Pascal to C though (quite a few years earlier, I guess I was 16 back then - time does fly ;) > Another problem is that > every time I work with a student C++ programmer coming out of Duke's CPS > department (which now teaches C++ as standard fare) I observe that while > they are just great at creating objects and so forth they are clueless > about things like pointers and malloc and the actual structure of a > multidimensional array of structs and how to make one and manipulate its > contents. As a consequence I have to spend weeks teaching them about > all of the virtues of "C as a thin veneer" in order to get them to where > they can cut advanced code at all, in either one. Some people argue that it is better to start with C++, then "move on" to C if necessary (or at least the subset of C++ that is also C). I think they have many valid points. And so do you. I don't know. For sure, you cannot be an excellent "computer programmer" if you do not understand in detail how memory works, and all those other "details" that *can* be abstracted away in C++. Those details will still bite you, no matter how far they are abstracted away, and if you do not understand them, you cannot hope to solve the problem. I guess we agree on that :) > You may not have encountered this as you did C first. Or maybe you > never have cause to allocate triangular arrays of structs or to build > your own linked list of structs, or maybe C++ does this and eats your > meatloaf for you and I'm just to ignorant to know how. C++ does not make the need for understanding go away. Other "more pure" languages make understanding irrelevant though - no matter how clever you are, you cannot solve the problem because the language won't let you ;) > Its funny. Back in the 80's all the CPS departments taught pascal > because it provided strong type checking and absolutely forced one to > use a rigorous bottom-up program design methodology. Precisely the same > features that caused all real programmers to run from it screaming, as > somehow all the programmers I've ever met work top down initially, then > top, bottom, middle, whereever until the project is finished, often with > a few epiphany-driven total project reorganizations in the middle as > experience teaches one what the REAL data objects are that they are > manipulating and whether they need easily maintained code that can run > relatively slowly or whether they need the core loop to run faster than > humanly possible if they have to hand-code it in assembler to get it > there. The difference, as I see it, is that Pascal *forces* you to use a very rigorous (and limiting) type system, or, work around it in extremely unsafe ways. C++ *allows* you to use a fairly rigorous type system, and "fall back" to casting where required. But even the casts have been made "safer", for example, the static_cast will not cast away const, you must explicitly add a const_cast if you really want to do that. Just an example. Typing the longer casts can be a little tedous - but hey, \textbf{} is that long because if you used it frequently your text would look ugly (don't remember if it was Lamport or Knuth who gave that explanation). Same goes with cast-ridden code. The longer and more difficult a cast is to type, the better ;) But if you disagree with me, C++ allows you to use C-style casts as well. Just don't come back complaining to me when they bite you ;) > Now they teach C++, and as you observed, teach it badly. I humbly > submit that the REASON they teach it badly is that they aren't teaching > "C++" at all, they are teaching structured programming and programming > discipline with a language that simply doesn't permit (or if you prefer > encourage) the student to use methodologies with far more power but far > less externally enforced structure. Pure pedagogy, in other words. Thank you, I do prefer "encourage", and I would argue that "permit" would be false. ;) Handing a person a copy of Webster's will not make him a poet. I'm sure we can agree on that last one (we can, right? ;) It's a pity, really, if teaching of "real programming languages" is really in such a sad state. I got some learning material from the univeristy here (have a cousin who started there recently), and that was *bad*. It was C though, at least I think it was meant to be (yes, it was hard to tell). Giving 3rd semester students example code where you do inter-process communication by writing to a file from one process, and polling it from another (hoping that other write finished completely), now that's just sick. Best part; the delay function (from keeping the polling from taking too much CPU time or whatever) was a busy wait - yes, you know, a for() loop which counts to a gazillion and a half, with a neat little arithmetic function in it to keep the compiler from optimizing the loop away. I almost cried. Then came the rage. Oh, and the voices in my head. [...good stuff snipped...] > I suspect that students who learn C++ in this order are, as you appear > to be, really good programmers who can get the most out of C++ and its > object oriented features without "forgetting" how to manipulate raw > blocks of memory without any sort of OO interface per se when the code > structure and extensibility requirements don't warrant all the extra > overhead associated with setting one up. I really don't know how one ought to teach people "properly". But I absolutely agree with you that one must understand the machine that one intends to program. At least to a resonable extend (and that, in my book as well as in yours it seems, includes knowing about layout of datastructures in memory, knowing that caches exist, etc.) ...[much more good stuff snipped]... > Give me a few days paring C++ down to the bare minimum extension of C so > that it is a pure superset (so ALL C CODE just plain compiles perfectly > and even produces the same runfile, so that the I/O language > "extensions" are moved off into a library that you can link or not, You would need to take out the 'template' reserved word (among others) to gain that full compatiblity. That alone would make C++ woth only half as much. Templates are *extremely* powerful (when in combination with either existing C and new C++ constructs). Again, I believe that there were good reasons for the breakage introduced. Sure, you can probably point out things that were stupid, and I may even agree with you, but a major part of the breakage was definitely a "lesser evil", at least the way that I see it :) > mostly not, and so that using things like classes and protection and > inheritance are suddenly pure and transparent extensions of the basic > ideas of structs and unions and typedefs, and I wouldn't even argue. In > fact, I'd say that the argument itself becomes silly and moot. Exactly! >From where I stand, in many situations (in particular, the situation that spawned this very thread - the start of a new project), we may already be at the point where there is no practical downside to using C++ as compared to C. That leaves you with some possible benefits you can use where you see fit. Assuming, as before, that the author knows the language etc. etc. And the whole argument becomes silly and moot :) > One > wouldn't even have to change Makefiles, as C++ would just be a feature > of gcc that is automatically processed when and where it is encountered. [albatross:joe] $ g++ -o wc wc.cc [albatross:joe] $ gcc -o wc wc.cc -lstdc++ We're there already - sort of ;) Just rename your files and you're done (the wc.cc file is the "challenge" from above) ;) > So the kernel would suddenly be written in "C++", sort of, and there > would be a very low barrier to converting C into C++ because basically, > there would be no conversion at all, only extension. Well, yes and no. Thinking about all this, I am actually not so much of a proponent of converting existing C projects into C++. I believe it is better to make a "clean break" - when starting a new project, go C++ from the beginning. Do templates. Use STL. Manage your memory safely. Compared to a hybrid project, where some percentage of the code is "very" safe, some percentage is "rather unsafe", and some of the safe code may need to call unsafe code and end up being unsafe itself. I don't know actually - I have a gut feeling that such a conversion would be a pain. The problem is not the conversion itself, the problem is that you end up with a so-called C++ program, which has never taken advantage of the advantages that C++ could have provided. So, in effect, your old "Good C" program becomes a "Bad C++" program. I am not so sure that this is a change to the better. In fact, you may even make me say that it is probably not... And this may be a good reason to keep the good ol' "gcc" for a few more decades. For all the code that is "Good C". > This of course will never happen. Until it does, then an excellent > reason to choose C instead of C++ is if you happen to know C very well > but not know (or care for) C++ as well. That's a good reason for sure! :) It's even language neutral ;) [...good stuff snipped yet again...] > majority of people who stick with C++ end up programming for Windows for > some big corporation (which might well be Microsoft) -- and produce > shitty, broken, huge, inefficient code as often as not. Hey, I will claim that: *) "Bad C++" has a strange attraction to crappy programmers *) Windows has a strange attraction to crappy programmers Thus, I end up observing the very same thing as you; that a lot of people write crappy C++ on windows. I'd just rather attack people than languages ;) (uh, *some* languages that is... Don't get me started on the J-word ;) [...more good stuff snipped yet again...] > lives next door writing software and clipping coupons. Other faculty I > know seem to view C++ the same way -- a good thing to use to teach > students because it forces them to code a certain way, but THEY don't > use it for THEIR research projects. I think it's a matter of time until more people get to learn the language and get used to it. Sure, a struct may be all you need. But then one day you have a struct which just needs a dynamically allocated member. And then, maybe, you find that it is convenient that you can put the deallocation of that member in the struct destructor. It's the little things, at first. Small conveniences that just come in handy. As far as my arguments have gone, C++ is more like a three year old language, in the "mature languages" world. No wonder it's not ubiquitous. Especially when looking at production systems or even research projects that may have been 10 or 20 years in development. > > Professionals seem more often than not to prefer having the entire > palette and all the brushes, and don't like even MAKING color by numbers > codelets. I continue to claim, that you do have the whole palette. It's just, that for the little trivial things where colour by number is more convenient and fully accomplishes the task at hand, you can actually do it, instead of starting to cut out the support frame for your future canvas on which you will eventually paint your masterpiece. You should still know how to produce a canvas, and how to paint properly. But you should (in my oppinion) not be absolutely required to start from scratch for every little trivial problem that needs solving. Any large project will solve countless small trivial problems, and a number of absolutely nontrivial problems too probably. It's darn convenient to be able to solve the trivial ones trivially ;) It will take a long time before C++ is generally accepted as "A Better C", I'm sure, but the time will come eventually (bets, anyone?) ;) > > This is very definitely not exhaustive or definitive. I also know > students where they exact opposite is true -- they learned C++, learned > about data structs and malloc and C, and continue to code in C++ (as you > do) with strictly enriched abilities. Of course they are (like you;-) > Very Bright Guys... *blush* ;) > and could probably even write good Fortran code if > they ever learned it. No, wait, that is an oxymoron -- couldn't happen. Point! ... > I agree, and hope you realize that I'm deliberately overarguing the case > for C above for precisely that reason. I really do believe that it is > as much a matter of taste and your particular experience and application > space as anything else and wouldn't be surprised if some java coder DOES > speak up for java as that's what THEY are really good at and they've > discovered some feature that makes it all worthwhile. I see it as serious fun :) Besides, it's no fun to hear about a langauge from someone who doesn't care about it. You need to poke people, call them sissies, tell them their language sucks and things about their mother, and *then* you can get a good language discussion going ;) A good language discussion needs *passion*, *anger*, *love*, true feelings. Or maybe that's just me... ;) > We've already heard from the python crowd, after all, and THAT'S a > language where they even eschew such obviously useful constructs as {}'s > and line terminators such as ; in favor of indentation as a matter of > deliberate design/dogma. It really does come down in the end to > religious belief/personal taste. They do that to some extent in Haskell as well - I code a little in Haskell from time to time, but haven't gotten around to Python yet. The indentation idea is, well, "interesting", I'd say :) > > In the best spirit of tongue-in-cheek, I cannot do better than end with: > > http://www.phy.duke.edu/~rgb/Beowulf/c++_interview/c++_interview.html > > Read it and weep...:-) Will do. Both, I fear ;) Cheers, / jakob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amb_lew at yahoo.com Sat Jan 24 14:14:20 2004 From: amb_lew at yahoo.com (ambrose lewis) Date: Sat, 24 Jan 2004 11:14:20 -0800 (PST) Subject: [Beowulf] MPI_File experience??? Message-ID: <20040124191420.82560.qmail@web14204.mail.yahoo.com> Hi All: I have a small cluster running a remote sensing code. I'm using MPICH on top of Mac OS X. Right now, my parallel performance isn't so great, because my communication time is far outweighing my computations. Does anybody have experience/suggestions/lessons learned about using the various MPI-IO routines??? I'm particularly interesting in learning more about successful use of the non-blocking read and write calls. Any pointers to sample codes would be greatly appreciated! THANXS! amb __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Sat Jan 24 16:36:06 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Sat, 24 Jan 2004 22:36:06 +0100 Subject: [Beowulf] MPI_File experience??? In-Reply-To: <20040124191420.82560.qmail@web14204.mail.yahoo.com> References: <20040124191420.82560.qmail@web14204.mail.yahoo.com> Message-ID: <4012E546.8010309@telia.com> Ambrose, This sounds interesting, how is the price/prestanda compared to PCs? /pelle ambrose lewis wrote: >Hi All: >I have a small cluster running a remote sensing code. >I'm using MPICH on top of Mac OS X. >Right now, my parallel performance isn't so great, >because my communication time is far outweighing my >computations. >Does anybody have experience/suggestions/lessons >learned about using the various MPI-IO routines??? >I'm particularly interesting in learning more about >successful use of the non-blocking read and write >calls. >Any pointers to sample codes would be greatly >appreciated! >THANXS! >amb > >__________________________________ >Do you Yahoo!? >Yahoo! SiteBuilder - Free web site building tool. Try it! >http://webhosting.yahoo.com/ps/sb/ >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kums at mpi-softtech.com Sun Jan 25 13:23:00 2004 From: kums at mpi-softtech.com (Kumaran Rajaram) Date: Sun, 25 Jan 2004 12:23:00 -0600 (CST) Subject: [Beowulf] MPI_File experience??? In-Reply-To: <20040124191420.82560.qmail@web14204.mail.yahoo.com> Message-ID: Ambrose, If your application is such that only root process performs file I/O and distributes raw data/ aggregates processed data to/from other processes, its recommended to shift to the MPI-2 model and use MPI-IO interface to perform file I/O. In order to gain performance from the MPI-IO non-blocking I/O interface, the middleware/implementation needs to as well have a good asynchronous model. Please read the following for more info: http://www.mpi-softtech.com/company/publications/files/Mercutio.pdf You should be able to find sample MPI-IO codes and other information in the following website: http://www.cs.dartmouth.edu/pario/ Hope this helps. Cheers, -Kums On Sat, 24 Jan 2004, ambrose lewis wrote: > Hi All: > I have a small cluster running a remote sensing code. > I'm using MPICH on top of Mac OS X. > Right now, my parallel performance isn't so great, > because my communication time is far outweighing my > computations. > Does anybody have experience/suggestions/lessons > learned about using the various MPI-IO routines??? > I'm particularly interesting in learning more about > successful use of the non-blocking read and write > calls. > Any pointers to sample codes would be greatly > appreciated! > THANXS! > amb > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shriram1976 at yahoo.com Sun Jan 25 16:40:51 2004 From: shriram1976 at yahoo.com (Shriram R) Date: Sun, 25 Jan 2004 13:40:51 -0800 (PST) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node Message-ID: <20040125214051.84151.qmail@web11408.mail.yahoo.com> Hi, We have a 24 node/48 procs linux cluster running Redhat. The queueing system that we use is PBS. One of the nodes, node15, conked out completely and is not restarting. "pbsnodes -a" shows the state of the node15 as "down". However, jobs which had been running on node15 still show up when I do a "qstat". I tried to use "qdel" and "qsig" to delete these jobs, but the server complains that it is unable to contact pbs_mom, which is obvious since node15 is down. Can someone tell me how do I delete these jobs from the output of "qstat" ? TIA. -shriram __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Sun Jan 25 17:56:26 2004 From: bclem at rice.edu (Brent M. Clements) Date: Sun, 25 Jan 2004 16:56:26 -0600 (CST) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node In-Reply-To: <20040125214051.84151.qmail@web11408.mail.yahoo.com> References: <20040125214051.84151.qmail@web11408.mail.yahoo.com> Message-ID: YOu have to remove the job files from both the mom_priv/jobs directory on the moms that the job was running on as well as the server_priv/jobs directory on the pbs server. -B Brent Clements Linux Technology Specialist Information Technology Rice University On Sun, 25 Jan 2004, Shriram R wrote: > Hi, > > We have a 24 node/48 procs linux cluster running > Redhat. The queueing system that we use is PBS. One > of the nodes, node15, conked out completely and is not > restarting. > > "pbsnodes -a" shows the state of the node15 as "down". > > However, jobs which had been running on node15 still > show up when I do a "qstat". > > I tried to use "qdel" and "qsig" to delete these jobs, > but the server complains that it is unable to contact > pbs_mom, which is obvious since node15 is down. > > Can someone tell me how do I delete these jobs from > the output of "qstat" ? > > TIA. > -shriram > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Mon Jan 26 00:43:53 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Mon, 26 Jan 2004 00:43:53 -0500 Subject: [Beowulf] mpiBLAST 1.2.0 RPM's available Message-ID: <1075095833.4481.542.camel@protein.scalableinformatics.com> Hi Folks: We built some source and binary RPM's for mpiBLAST 1.2.0. You can find them at http://downloads.scalableinformatics.com/downloads/mpiblast/ or linked off of our main page http://scalableinformatics.com . These RPM's assume you have an operational and functional mpich installation. They should work with most architectures, though I am working on fixing the Opteron build. We created these as people seemed to have some trouble building the package, and installing it on their systems. If you grab the binary version, please edit the /usr/local/etc/mpiblast.conf file as per the instructions on the mpiBLAST page at http://mpiblast.lanl.gov . Please send us a note on any problems you encounter with this packaging. To install the binary version, download the appropriate binary and as root, run rpm -Uvh mpiBLAST-1.2.0-3.i686.rpm This assumes you have grabbed the i686 architecture binary. If you would prefer to build from source, make sure you have a functional mpich install with mpe, have it in your path and be able to compile with mpiCC, and then run: rpmbuild --rebuild mpiBLAST-1.2.0-3.src.rpm to build a binary. Then use the resulting binary RPM to install. Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shriram1976 at yahoo.com Mon Jan 26 01:03:30 2004 From: shriram1976 at yahoo.com (Shriram R) Date: Sun, 25 Jan 2004 22:03:30 -0800 (PST) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node In-Reply-To: Message-ID: <20040126060330.30554.qmail@web11411.mail.yahoo.com> Hi, This is with ref the reply below. I deleted the files from server_priv/jobs. But the node on which the jobs were running has completely crashed - doesnt even reboot. So, cant delete the files from mom_priv/jobs on the corresponding node. Hence the jobs are still showing up in "qstat". Any suggestions ? TIA -shriram --- "Brent M. Clements" wrote: > YOu have to remove the job files from both the > mom_priv/jobs directory on > the moms that the job was running on as well as the > server_priv/jobs > directory on the pbs server. > -B > > Brent Clements > Linux Technology Specialist > Information Technology > Rice University > > > On Sun, 25 Jan 2004, Shriram R wrote: > > > Hi, > > > > We have a 24 node/48 procs linux cluster running > > Redhat. The queueing system that we use is PBS. > One > > of the nodes, node15, conked out completely and is > not > > restarting. > > > > "pbsnodes -a" shows the state of the node15 as > "down". > > > > However, jobs which had been running on node15 > still > > show up when I do a "qstat". > > > > I tried to use "qdel" and "qsig" to delete these > jobs, > > but the server complains that it is unable to > contact > > pbs_mom, which is obvious since node15 is down. > > > > Can someone tell me how do I delete these jobs > from > > the output of "qstat" ? > > > > TIA. > > -shriram > > > > __________________________________ > > Do you Yahoo!? > > Yahoo! SiteBuilder - Free web site building tool. > Try it! > > http://webhosting.yahoo.com/ps/sb/ > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or > unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From franz.marini at mi.infn.it Mon Jan 26 04:41:02 2004 From: franz.marini at mi.infn.it (Franz Marini) Date: Mon, 26 Jan 2004 10:41:02 +0100 (CET) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node In-Reply-To: <20040126060330.30554.qmail@web11411.mail.yahoo.com> References: <20040126060330.30554.qmail@web11411.mail.yahoo.com> Message-ID: Hi, On Sun, 25 Jan 2004, Shriram R wrote: > Hi, > > > This is with ref the reply below. I deleted the files > from server_priv/jobs. But the node on which the jobs > were running has completely crashed - doesnt even > reboot. So, cant delete the files from mom_priv/jobs > on the corresponding node. > > Hence the jobs are still showing up in "qstat". Any > suggestions ? Have you restarted pbs_server *after* deleting the jobs under server_priv/jobs ? (and maybe even maui, if you use it) Franz --------------------------------------------------------- Franz Marini Sys Admin and Software Analyst, Dept. of Physics, University of Milan, Italy. email : franz.marini at mi.infn.it --------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Mon Jan 26 20:17:35 2004 From: shewa at inel.gov (Andrew Shewmaker) Date: Mon, 26 Jan 2004 18:17:35 -0700 Subject: [Beowulf] C vs C++ challenge In-Reply-To: <20040123144127.GA814@unthought.net> References: <20040120235009.GD30005@unthought.net> <20040123144127.GA814@unthought.net> Message-ID: <4015BC2F.8030405@inel.gov> Jakob Oestergaard wrote: > Let me throw out a challenge: > > "Write a program in C that reads from standard input until EOF, and > prints the number of distinct words found in that input" > > > I use an RB-tree to store the words I find, and print the number of > elements in the final tree. > > A comparable C solution shouldn't be more than a few thousand lines ;) > > Oh, right, I cheated and used STL. Well, STL is a big part of why C++ > is really handy, but go right ahead and use "glib" or whatever. > > I still guarantee you two things: > 1) Your code will be longer > 2) Your program will be slower > > As always, I love to be proven wrong ;) Here's a similar program...word frequency...instead of word count. http://www.cs.bell-labs.com/cm/cs/pearls/sec151.html His C++ code is shorter, but his C code is faster. With a trivial modification, his C code can meet your challenge too. http://www.cs.bell-labs.com/cm/cs/pearls/wordfreq.c I downloaded the KJV of the Bible, Shakespeare's Complete Works, War and Peace, part of the human genome, and a few other files off of Project Gutenberg. The above C code was much faster on small files and twice as fast when I lumped everything together into a 35 MB file with 460000+ distinct words. The only test I ran where C++ beat C (by 20+%) was a 187 MB file with over 2.5 million distinct words, where I used PGI's hash_set instead of its regular set. Have a great day, Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 27 01:04:37 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Mon, 26 Jan 2004 22:04:37 -0800 (PST) Subject: [Beowulf] How to enable MPE environment in MPICH-1.2.5.2 In-Reply-To: <6.0.0.22.2.20040120073814.02186048@localhost> Message-ID: <20040127060437.45877.qmail@web21501.mail.yahoo.com> how to i enable MPE environment in my cluster do i have do build it explicitly or just some changes while installing mpich __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From swap_project at yahoo.com Tue Jan 27 01:40:27 2004 From: swap_project at yahoo.com (swapna ghosh) Date: Mon, 26 Jan 2004 22:40:27 -0800 (PST) Subject: [Beowulf] Clustering on Redhat9.0 In-Reply-To: Message-ID: <20040127064027.41294.qmail@web40808.mail.yahoo.com> Hi I am a newbee to the clustering and my next project is clustering with max 2 nodes. The servers are Redhat 9.0. I was going through few clustering procedure.. - beowolf with PVM - openMosix - SSI Cluster for linux There are lots many but i need the *free* software... Though the last one(SSI Clustering..) the document i read , it is on Redhat 9.0 openMosix is kernel patch oriented.. So my little knowledge says that it will be difficult everytime to upgrade if not rpm...[ i may be wrong ] beowolf with PVM, i did not get any such document(installation steps) till yet on Redhat 9.0. It will be really appreciated if anybody points me the document/url of the procedure *installation of clustering with beowolf PVM on RH-9.0* Lastly can anybody suggest me which one will be suitable for redhat 9.0? Thanks in advance -Swapna __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 27 04:50:19 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Tue, 27 Jan 2004 01:50:19 -0800 (PST) Subject: [Beowulf] Remote procedure call (error) Message-ID: <20040127095019.30138.qmail@web21506.mail.yahoo.com> have any body tried RPC programing on Linux i get rpc unknown protocol error i am trying to diaply the message on the remote host __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Tue Jan 27 09:56:04 2004 From: bclem at rice.edu (Brent M. Clements) Date: Tue, 27 Jan 2004 08:56:04 -0600 (CST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: <20040127095019.30138.qmail@web21506.mail.yahoo.com> References: <20040127095019.30138.qmail@web21506.mail.yahoo.com> Message-ID: Are the rpc daemons started up or enabled in xinetd/inetd? -Brent Brent Clements Linux Technology Specialist Information Technology Rice University On Tue, 27 Jan 2004, prakash borade wrote: > have any body tried RPC programing on Linux > i get > rpc unknown protocol error > i am trying to diaply the message on the remote host > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Tue Jan 27 10:37:19 2004 From: shewa at inel.gov (Andrew Shewmaker) Date: Tue, 27 Jan 2004 08:37:19 -0700 Subject: [Beowulf] Clustering on Redhat9.0 In-Reply-To: <20040127064027.41294.qmail@web40808.mail.yahoo.com> References: <20040127064027.41294.qmail@web40808.mail.yahoo.com> Message-ID: <401685AF.2000901@inel.gov> swapna ghosh wrote: > Hi > > I am a newbee to the clustering and my next project is clustering > with max 2 nodes. > > The servers are Redhat 9.0. I was going through few clustering > procedure.. > > - beowolf with PVM > - openMosix > - SSI Cluster for linux > > There are lots many but i need the *free* software... > > Though the last one(SSI Clustering..) the document i read , it is on > Redhat 9.0 > > openMosix is kernel patch oriented.. So my little knowledge says > that it will be difficult everytime to upgrade if not rpm...[ i may be > wrong ] OpenSSI also involves a patched kernel, so it sounds like you'll probably want to avoid it too, if you are avoiding openMosix. > beowolf with PVM, i did not get any such document(installation > steps) till yet on Redhat 9.0. It will be really appreciated if anybody > points me the document/url of the procedure *installation of clustering > with beowolf PVM on RH-9.0* RH-9 includes pvm-3.4.4-12.i386.rpm on the second CD and pvm-gui-3.4.4-12.i386.rpm on the third. You still have to set it up to use ssh or rsh, though. Search through the archives at http://marc.theaimsgroup.com/?l=beowulf&r=1&w=2v and you will find links to howtos or descriptions on how to set up rsh. I recommend learning how to set up ssh for host based authentication. http://www.omega.telia.net/vici/openssh/ has good instructions, but note that the latest OpenSSH requires "EnableSSHKeysign yes" to be set in /ssh_config. Also, you can simplify the instructions if you treat the cluster as a single system and copy the same host keys to all of the machines. Rocks does it that way. If you don't want to learn about that sort of thing, then install a cluster environment/distribution. > Lastly can anybody suggest me which one will be suitable for redhat > 9.0? You should look at Rocks or OSCAR. Rocks is a distribution based on RH and OSCAR can be installed on top of RH-9. They will set up PVM and MPI for you. http://rocksclusters.org http://oscar.openclustergroup.org Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Tue Jan 27 09:55:34 2004 From: bclem at rice.edu (Brent M. Clements) Date: Tue, 27 Jan 2004 08:55:34 -0600 (CST) Subject: [Beowulf] How to enable MPE environment in MPICH-1.2.5.2 In-Reply-To: <20040127060437.45877.qmail@web21501.mail.yahoo.com> References: <20040127060437.45877.qmail@web21501.mail.yahoo.com> Message-ID: By default, I believe mpe is built. But you may want to check the output from ./configure --help -B Brent Clements Linux Technology Specialist Information Technology Rice University On Mon, 26 Jan 2004, prakash borade wrote: > > how to i enable MPE environment in my cluster > do i have do build it explicitly or just some changes > while installing mpich > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tkrewson at korrnet.org Tue Jan 27 14:15:55 2004 From: tkrewson at korrnet.org (Tom A Krewson) Date: Tue, 27 Jan 2004 14:15:55 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade Message-ID: Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Tue Jan 27 16:35:40 2004 From: dtj at uberh4x0r.org (Dean Johnson) Date: Tue, 27 Jan 2004 15:35:40 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <1075239339.8252.2.camel@terra> On Tue, 2004-01-27 at 13:15, Tom A Krewson wrote: > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > They say that the upgraded machine will give off less heat. If I am not mistaken, its the same damn cpus in a different box and the power supply will be roughly the same ratings. Am missing something? -- -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at fnal.gov Tue Jan 27 17:33:53 2004 From: djholm at fnal.gov (Don Holmgren) Date: Tue, 27 Jan 2004 16:33:53 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> References: <1075239660.18659.14.camel@localhost.localdomain> Message-ID: Xserve boxes use new G5's based on a 90nm process. See http://www.theregister.co.uk/content/39/35057.html The claim is that power consumption per processor drops to ~ 25 Watt at 2 GHz from ~ 50 Watt at 1.8 GHz. Don Holmgren On Tue, 27 Jan 2004, Vann H. Walke wrote: > "The new system will take much less power and generate less heat and > free up space." > > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Removing video cards / hard drives? Do the video cards > really use a significant amount of power if they aren't being used? Or, > is there something fundamentally different about the Xserve? > > Vann > > On Tue, 2004-01-27 at 14:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jamesr at best.com Tue Jan 27 17:25:45 2004 From: jamesr at best.com (James Rogers) Date: 27 Jan 2004 14:25:45 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> References: <1075239660.18659.14.camel@localhost.localdomain> Message-ID: <1075242345.14220.7.camel@localhost.localdomain> On Tue, 2004-01-27 at 13:41, Vann H. Walke wrote: > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Removing video cards / hard drives? Do the video cards > really use a significant amount of power if they aren't being used? Or, > is there something fundamentally different about the Xserve? Different fab processes for the CPUs. The PPC970 XServes use a 90nm PPC970 versus the 130nm PPC970 used in the G5 towers. These are the first chips off the new process line, and use about half the power of the 130nm ones IIRC. -James Rogers jamesr at best.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jim at ks.uiuc.edu Tue Jan 27 17:33:38 2004 From: jim at ks.uiuc.edu (Jim Phillips) Date: Tue, 27 Jan 2004 16:33:38 -0600 (CST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239339.8252.2.camel@terra> Message-ID: As I understand, the Xserve G5 is using a newer process G5 that gives off less heat (and can therefore fit in a 1U chassis). -Jim On Tue, 27 Jan 2004, Dean Johnson wrote: > On Tue, 2004-01-27 at 13:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > > > They say that the upgraded machine will give off less heat. If I am not > mistaken, its the same damn cpus in a different box and the power supply > will be roughly the same ratings. Am missing something? > > -- > > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From robl at mcs.anl.gov Tue Jan 27 18:02:17 2004 From: robl at mcs.anl.gov (Robert Latham) Date: Tue, 27 Jan 2004 17:02:17 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> <1075239339.8252.2.camel@terra> References: <1075239660.18659.14.camel@localhost.localdomain> <1075239339.8252.2.camel@terra> Message-ID: <20040127230217.GK25328@mcs.anl.gov> On Tue, Jan 27, 2004 at 03:35:40PM -0600, Dean Johnson wrote: > They say that the upgraded machine will give off less heat. If I am not > mistaken, its the same damn cpus in a different box and the power supply > will be roughly the same ratings. Am missing something? On Tue, Jan 27, 2004 at 04:41:01PM -0500, Vann H. Walke wrote: > "The new system will take much less power and generate less heat and > free up space." > > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Different cpus. http://www.theregister.co.uk/content/39/35057.html new 90nm chips throw off 24W at 1.4 GHz, down from 51 W at 1.8 GHz in the older 130nm models ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Labs, IL USA B29D F333 664A 4280 315B _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mbanck at gmx.net Tue Jan 27 17:54:49 2004 From: mbanck at gmx.net (Michael Banck) Date: Tue, 27 Jan 2004 23:54:49 +0100 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239339.8252.2.camel@terra> References: <1075239339.8252.2.camel@terra> Message-ID: <20040127225449.GM638@blackbird.oase.mhn.de> On Tue, Jan 27, 2004 at 03:35:40PM -0600, Dean Johnson wrote: > On Tue, 2004-01-27 at 13:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > They say that the upgraded machine will give off less heat. If I am not > mistaken, its the same damn cpus in a different box and the power supply > will be roughly the same ratings. Am missing something? The latest XServe are reportedly equipped with 90nm CPUs, not 120nm CPUs like the G5, thus apparently spreading far less heat. Michael _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xyzzy at speakeasy.org Tue Jan 27 18:04:11 2004 From: xyzzy at speakeasy.org (Trent Piepho) Date: Tue, 27 Jan 2004 15:04:11 -0800 (PST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message-ID: On Tue, 27 Jan 2004, Joel Jaeggli wrote: > one wonders why they didn't just wait g5 the close relationship to apple > and the fact that xserves were basically justaround the corner. unless of > course it had to be in this years top-500 list and the motivation is as > much political/econmic as practical... Exactly. They said before that one of the reasons they went with apple was because they could get them the system in the timeframe (top-500 list it would seem) that they needed. The xserve has ECC memory, which will bring the performance of their cluster up to what they say it is. Their performance now is half the published value, since they need to run everything at least twice to have a reasonable assurance that their results are correct. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From walkev at presearch.com Tue Jan 27 16:41:01 2004 From: walkev at presearch.com (Vann H. Walke) Date: Tue, 27 Jan 2004 16:41:01 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <1075239660.18659.14.camel@localhost.localdomain> "The new system will take much less power and generate less heat and free up space." Ok, the space reduction is obvious. Anyone know why the power/heat is going down? Removing video cards / hard drives? Do the video cards really use a significant amount of power if they aren't being used? Or, is there something fundamentally different about the Xserve? Vann On Tue, 2004-01-27 at 14:15, Tom A Krewson wrote: > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Tue Jan 27 17:22:05 2004 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Tue, 27 Jan 2004 14:22:05 -0800 (PST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> Message-ID: radeon 9600's are probably good for 6-10w idle at the default clock speed... not all that much... just from the specs the dual g5 cluster node specifices 5.5amp peak vs 6.5amp for the dual g5 dekstop. I can't imagine that the difference would be too massive however between a dekstop and a node each with one drive. the space savings have to be enormous thought g5 towers are basically 5u even in the best circumstances.... one wonders why they didn't just wait g5 the close relationship to apple and the fact that xserves were basically justaround the corner. unless of course it had to be in this years top-500 list and the motivation is as much political/econmic as practical... joelja On Tue, 27 Jan 2004, Vann H. Walke wrote: > "The new system will take much less power and generate less heat and > free up space." > > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Removing video cards / hard drives? Do the video cards > really use a significant amount of power if they aren't being used? Or, > is there something fundamentally different about the Xserve? > > Vann > > On Tue, 2004-01-27 at 14:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From canon at nersc.gov Tue Jan 27 19:00:36 2004 From: canon at nersc.gov (canon at nersc.gov) Date: Tue, 27 Jan 2004 16:00:36 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message from Dean Johnson of "Tue, 27 Jan 2004 15:35:40 CST." <1075239339.8252.2.camel@terra> Message-ID: <200401280000.i0S00ax0015946@pookie.nersc.gov> I think the Xserve boxes use a version of the PPC with smaller feature size (90nm vs 130 or something like that). So its a different version of the same damn chip. :-) --Shane ------------------------------------------------------------------------ Shane Canon voice: 510-486-6981 PSDF Project Lead fax: 510-486-7520 National Energy Research Scientific Computing Center 1 Cyclotron Road Mailstop 943-256 Berkeley, CA 94720 canon at nersc.gov ------------------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Tue Jan 27 21:09:33 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Tue, 27 Jan 2004 21:09:33 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message-ID: I draw several conclusions from the VT cluster: - HPL is very cache-friendly. the dual-G5 has an impressive peak flops rating, but pretty mediocre memory bandwidth. anyone have a Stream result for a G5? I'd guess around 2.5 GB/s. - HPL also doesn't penalize fractional bisection bandwidth clusters, which is pretty interesting, since it *does* seem to reward post-gigabit interconnect. latency, perhaps? - the press isn't smart enough to notice that the VT cluster was far too unreliable (no ECC) and used an aweful lot of floorspace. it *is* neat that clusters are commodities nowadays, but is that news? somehow I don't think very many future clusters will be built by pizza-motivated volunteers... - a good PR stunt buys a lot of vendor discount. remember: the G5s list at close to $5M total, and the IB kit is pushing $2M. I'm guessing that VT can sell the G5's at a 30% discount. and Xserves cost about 50% more than desktops, so VT winds up with around $10M from Apple, for around $3M investment, nice! > The xserve has ECC memory, which will bring the performance of their cluster > up to what they say it is. oh, come on. MTBF is ~10 hours, so short jobs are perfectly fine ;) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 00:14:32 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 00:14:32 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <4015BC2F.8030405@inel.gov> Message-ID: On Mon, 26 Jan 2004, Andrew Shewmaker wrote: > Jakob Oestergaard wrote: > > I still guarantee you two things: > > 1) Your code will be longer > > 2) Your program will be slower > > > > As always, I love to be proven wrong ;) > > Here's a similar program...word frequency...instead of word count. > > http://www.cs.bell-labs.com/cm/cs/pearls/sec151.html > > His C++ code is shorter, but his C code is faster. > > With a trivial modification, his C code can meet your challenge too. > > http://www.cs.bell-labs.com/cm/cs/pearls/wordfreq.c There are two or three very interesting observations to make about both Jakob's challenge and this response. One is that the code is indeed longer. Or is it? One obvious feature of the C++ code is that it relies heavily on the STL. Is this cheating? Perhaps not exactly, or not quite, but neither is it totally fair WHEN COMPARING CODE LENGTH. One could encapsulate all the code in ANY such program into a single subroutine call and then say "look how short my program is", and although perhaps the library IS a standard feature of C++, it is also very much a library whose functionality could, in all probability, be pretty much duplicated in C. And of course as expected C++ appears to lose the speed challenge (which should be no great surprise, as the C code that DOESN'T encapsulate a solution is pretty close to as efficient as assembler and is in any event dominated by I/O. So let's give C++ points for having a powerful library. Let's take points away because it runs slower and because HAVING the library, especially a POWERFUL library, tends to channel your program design in ways that can take advantage of the library but that might not be horribly efficient. Let's also not give it too MANY points for shorter program length, both because the library hides a LOT of code, you betcha, and because using the library renders the program totally opaque to somebody that doesn't know C++ and the library. As in maybe it's just me, but when I look at Jakob's code I have no idea how it works. What are these routines? What do they do? Where's the logic? Oh, I can read the comments and see the keywords and sort of guess that all sorts of dark magic is being done; I just have no idea how it does it, any more than I would if one took Jakob's code or my code or anybody's code for the same task and wrapped it up in a single routine: int count_words_from_stdin(); main() { int count; count = count_words_from_stdin(); printf("I counted %d words\n",count); } Simple code -- pretty readable without any comments at all. However, not terribly understandable or useful, without being able to see what the routine count_words_from_stdin() in fact does in however many lines of code it hides. It could even be written IN assembler, and take a LOT of lines and be VERY efficient. It would be my hope and expectation that C++ is more than just a big library with some nifty stuff added on top of the already fairly hefty (but incrementally includeable, you only link to what you need) and relatively low level library set one has to work with in C. In fact, to convince me to try C++ you'd have to show me code that is NOT this compact -- code that in fact defines all the various class/objects used the hard way, uses (and I mean really USES) inheritance and protection and so forth, in ways that cannot be trivially duplicated in C code (and that, in fact, doesn't look very much LIKE the associated C code). The fundamental issue is whether or not "objects" are somehow managed better in C++. Beyond that, even its relatively strong typing and powerful library are not language issues, they are features added to C++ as a more recent revision of C and could probably be added to C as well, if anybody really cared. As the KR to ANSI transition shows, perhaps somebody one day will care, so this isn't an entirely moot point. You might argue, perhaps with complete accuracy, that WRITING such a high level library in C would be difficult because it exploits many of the language features of C++. I'd argue back that projects such as the GSL suggest otherwise -- the GSL code is certainly written in a very "object-oriented" way, and there are constructors and destructors and so forth where they make sense, but underlying those entities are ordinary C structs as the essential objects, and you can read the GSL headers and work directly with the structs if you prefer to work with the naked data for reasons of speed even in cases where they do provide a get/set method for accessing components of an object. In my own code I typically use lots of objects, and when it is reasonable to do so (when they are very complex or have multiple layers of pointers that have to be freed in a particular order) I use constructors and destructors for those objects. Lots of times I don't. I NEVER use "methods" for objects -- if I want to operate on the contents of a struct I operate on the contents of a struct, with a subroutine or without it as the needs of the code dictate. I think such a argument would eventually end up back where I left us. C++ a) shapes code in certain ways that go beyond the mere use of "objects" or "object oriented methods" in programming which is ALWAYS an option in any language that supports some sort of implementation of a struct or union or (in the worst possible case) even a raw data block with suitable offsets (fortran, anyone:-); b) provides strong typing, which I'm willing to acknowledge could be a real advantage (although one CAN cast void types in C and accomplish very subtle things one does have to be a programming god to do it and not occasionally get bitten by it); c) provides library extensions (quite possibly powerful extensions) that encapsulate a variety of useful tasks so that they can be done with a relatively few lines of the concept of YOUR code (plus all the hidden lines of library code, of course); d) at the cost of transparency and user-controllable logic and user access to some components of the code. This is just restating a) again, and around we'd go. If I were to write code to read lines from stdin, parse them into words on the basis of a set of separators e.g. space, tab, comma, period, put them into a vector of strings (uniqued or not) and count and print the final result on EOF it would certainly be longer than the C++, no doubt, but it would also be literally as fast as it could be at execution (probably comparing decently to assembler) and any programmer, even someone that didn't program in C but minimally understood the concept of the malloc family for memory management, could likely make out the logic. Preferring this to writing a program that just calls STL functions or somebody else's count_words_from_stdin() is doubtless a matter of taste, and there are doubtless programmers that span the range (taste-wise) from "real programmers only program in assembler" to perl or python (where the code would also be very short -- a single transparent loop and an output statement in perl) to "I like LISP". So tastes differ, experiences differ, and needs differ as well. Is the C++/C debate about taste, experiences, needs? When I talk to CPS faculty, they make no bones about it -- they teach C++ so that their students can learn structured programming in an environment with strong typing and lots of structure, and they feel that in today's programming world performance is less important than ease of programming and structure and having a powerful library. They compromise helping students develop skill in the "details" of procedural programming for the ability to quickly put together projects using big building blocks in a collegial/corporate programming environment. They KNOW (in many cases) that they are doing this; they just consider ability to be a good, diligent programmer that follows the rules and colors within the lines to be more important than knowing how pointers and malloc works. Is this a good tradeoff? I dunno. This is a bit different from when I learned to code -- there WERE no powerful libraries, and a whole lot of our programming exercises were what amounted to writing code to process lists of words or numbers in various ways, to sort, to build little code machines on a microscopic level within the confines of the language. In one course we even wrote an assembler simulator/interpreter and a compiler interpreter WRITTEN in our assembler interpreter (all punched out on cards for our IBM 370 in the upper level language du jour, which I think was PL/1 at the time). Useless for developing programming skills, unless you think that good programming skills are developed writing a really complex project that forces you to duplicate the way a computer really really works, and then write a complex program that works on your simulated computer... A lot of the kids learning programming now don't learn "how can one sort a list"; they learn how to call a routine named "sort", and extensions thereof (at least until they are a relatively upper level class). This works just marvey, of course, as long as the entity they need to sort and the sort criterion can be fed to the subroutine successfully in a form that it can work with... but one cannot help but feel that something is lost, just as something was lost even with C when people who needed random numbers just naively called "rand()" and were a bit surprised when, um, the results weren't horribly random... So perhaps my problem is that I'm just an Old Guy(tm) and punched too many cards in too many mediocre languages in my day. For me learning C was liberating, exhilerating, empowering, and I continue to discover clever little things it can do to this very day. Portable assembler, that's what C is, plus a bit of upper-level dressing (some of which COULD be a lot nicer, no doubt). But I will, just maybe, give C++ another try some day in the future when I have time and have found a really good book on it. Every time in the past I was working without a decent text and with a questionable (Microsoft or Borland) compiler. If g++ is, as you suggest, a pure superset of gcc (a few extra keywords don't matter) I can always give using a class a try sometime when I want to make a new struct, and see just how many hoops I have to jump through to do what I want to do. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 00:47:47 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 00:47:47 -0500 (EST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: <20040127095019.30138.qmail@web21506.mail.yahoo.com> Message-ID: On Tue, 27 Jan 2004, prakash borade wrote: > have any body tried RPC programing on Linux > i get > rpc unknown protocol error > i am trying to diaply the message on the remote host To do rpm programming you probably need a bunch of stuff configured and running that might or might not be turned off for security reasons. Check, for example, to see if the portmapper is running. Also check iptables and/or ipchains to see if they are running and if so if any rules block access to the portmapper. rpcinfo is also your friend: rgb at lilith|T:3#rpcinfo -p lilith program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 32768 status 100024 1 tcp 32768 status 391002 2 tcp 32769 sgi_fam As a general rule, I'd consider the portmapper and rpc stuff to be a moderate security risk; lilith lives inside firewalls only. I have had direct experience of systems cracked in the past through portmapper bugs, which perhaps makes me a bit paranoid. What exactly are you trying to do? rgb > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Wed Jan 28 01:26:17 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Tue, 27 Jan 2004 22:26:17 -0800 (PST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: Message-ID: hi ya robert On Wed, 28 Jan 2004, Robert G. Brown wrote: > rpcinfo is also your friend: > > rgb at lilith|T:3#rpcinfo -p lilith > program vers proto port > 100000 2 tcp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 32768 status > 100024 1 tcp 32768 status > 391002 2 tcp 32769 sgi_fam > > As a general rule, I'd consider the portmapper and rpc stuff to be a > moderate security risk; lilith lives inside firewalls only. I have had > direct experience of systems cracked in the past through portmapper > bugs, which perhaps makes me a bit paranoid. did you run the secure versions of each ?? secure rpc secure portmap secure nfs http://www.Linux-Sec.net/FileSystem/#NFS and yes, guess if security is an issue, dont use "nfs" but than again, what's the options .. at least encrypt as much as one can (afford) .. including the filesystem at "*.duke.edu" guess you folks would be a prime target... c ya alvin _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 00:42:54 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 00:42:54 -0500 (EST) Subject: [Beowulf] Clustering on Redhat9.0 In-Reply-To: <20040127064027.41294.qmail@web40808.mail.yahoo.com> Message-ID: On Mon, 26 Jan 2004, swapna ghosh wrote: > beowolf with PVM, i did not get any such document(installation > steps) till yet on Redhat 9.0. It will be really appreciated if anybody > points me the document/url of the procedure *installation of clustering > with beowolf PVM on RH-9.0* > > Lastly can anybody suggest me which one will be suitable for redhat > 9.0? Find the pvm and pvm-gui binary rpm's on whatever RH 9 medium you used for the install or from their website or from a mirror (such as the one at mirror.dulug.duke.edu). Download them if necessary. CD into their directory. As root, run: rpm -Uvh pvm*.rpm Configure ssh to permit login with no password. Use google to find a micro-HOWTO on how to do this, if you can't follow the man pages -- it is straightforward. Configure your environment to support PVM. It will work best/easiest if your two nodes share a home directory filesystem (NFS exported from one to the other) and have identical accounts. That's all there is to it, really. Login to one system, start pvm or xpvm (the latter is more fun) and add the other as a node. Write code, have fun. There is a functional pvm program template on my personal website under general. I think it is even semi-documented. Give it a try. This is also going to be the focus of my column in Cluster World in March and April. Consider subscribing. It has also been covered in past issues of Forrest's column in Linux Magazine (IIRC). Search LM's archive. Finally, there is a good book on introductory PVM from MIT press. Consider buying it via e.g. Amazon. rgb > > Thanks in advance > -Swapna > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Wed Jan 28 04:09:12 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Wed, 28 Jan 2004 01:09:12 -0800 (PST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: Message-ID: <20040128090912.51497.qmail@web21506.mail.yahoo.com> rgb thanks for your support well i am trying to wite an client server application ( monitoring sofware ) for my beowulf cluster tell me what would be efficient either sockets or RPC would open socket on each machine put an extra load on the cluster or is it managable --- "Robert G. Brown" wrote: > On Tue, 27 Jan 2004, prakash borade wrote: > > > have any body tried RPC programing on Linux > > i get > > rpc unknown protocol error > > i am trying to diaply the message on the remote > host > > To do rpm programming you probably need a bunch of > stuff configured and > running that might or might not be turned off for > security reasons. > Check, for example, to see if the portmapper is > running. Also check > iptables and/or ipchains to see if they are running > and if so if any > rules block access to the portmapper. > > rpcinfo is also your friend: > > rgb at lilith|T:3#rpcinfo -p lilith > program vers proto port > 100000 2 tcp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 32768 status > 100024 1 tcp 32768 status > 391002 2 tcp 32769 sgi_fam > > As a general rule, I'd consider the portmapper and > rpc stuff to be a > moderate security risk; lilith lives inside > firewalls only. I have had > direct experience of systems cracked in the past > through portmapper > bugs, which perhaps makes me a bit paranoid. > > What exactly are you trying to do? > > rgb > > > > > __________________________________ > > Do you Yahoo!? > > Yahoo! SiteBuilder - Free web site building tool. > Try it! > > http://webhosting.yahoo.com/ps/sb/ > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or > unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- > Robert G. Brown > http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 > email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctibirna at giref.ulaval.ca Wed Jan 28 09:49:10 2004 From: ctibirna at giref.ulaval.ca (Cristian Tibirna) Date: Wed, 28 Jan 2004 09:49:10 -0500 Subject: [Beowulf] C vs C++ challenge In-Reply-To: References: Message-ID: <200401280949.10516.ctibirna@giref.ulaval.ca> Le Mercredi 28 Janvier 2004 00:14, Robert G. Brown a ?crit: > As in maybe it's just me, but when I look at Jakob's code I have no idea > how it works. What are these routines? What do they do? Where's the > logic? Oh, I can read the comments and see the keywords and sort of > guess that all sorts of dark magic is being done; I just have no idea > how it does it, any more than I would if one took Jakob's code or my > code or anybody's code for the same task and wrapped it up in a single > routine: And here I argue that this is where C++'s strength becomes evident. STL is _standard_ (hence its name). Thus, for a C++ programmer, Jakob's code is comprehensible in one 15second long glimpse and this programmer can immediately use (trust?) the code. With the equivalent C code from the posted link, I personally have the problem that I need at least 15 minutes to read and understand it, and even then, I can't be sure I identified all gotchas and thus I can use it without major crashes and problems. I argue that the biggest problem with C up until glib and gsl (thus very recently) was exactly the lack of high level routines collected in a ubiquous library. > A lot of the kids learning programming now don't learn "how can one sort > a list"; they learn how to call a routine named "sort", and extensions > thereof (at least until they are a relatively upper level class). This > works just marvey, of course, as long as the entity they need to sort > and the sort criterion can be fed to the subroutine successfully in a > form that it can work with... And once again C++ fills the expectation here, IMO. Paraphrasing you: "This works just marvey... with C++ exactly because any... entity they need to sort and the sort criterion can be fed to the subroutine successfully and the subroutine will happily work with". This, of course, thanks to C++'s templates. And the result is even highly readable and quite compact usually (and this is the reason I am ready to cope with performance penalties, because I can put out code 10-20 times faster than in C -- OK, this is a subjective measure -- and that code will only run 2-3 times slower -- citing another poster "that's what clusters are for" ;-). > But I will, just maybe, give C++ > another try some day in the future when I have time and have found a > really good book on it. Lippman's "C++ Primer". Thanks for listening. -- Cristian Tibirna (1-418-) 656-2131 / 4340 Universit? Laval - Qu?bec, CAN ... http://www.giref.ulaval.ca/~ctibirna Professionnel de recherche GIREF ... ctibirna at giref.ulaval.ca ?tudiant PhD G?nie Chimique ... tibirna at gch.ulaval.ca _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Wed Jan 28 08:54:02 2004 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 28 Jan 2004 14:54:02 +0100 Subject: [Beowulf] C vs C++ challenge In-Reply-To: References: <4015BC2F.8030405@inel.gov> Message-ID: <20040128135402.GM814@unthought.net> On Wed, Jan 28, 2004 at 12:14:32AM -0500, Robert G. Brown wrote: ... Guys, thanks a lot for your replies! Interesting, and a challenge. One I intend to follow up on. The C example ran 10 times faster than my C++ example here. So that's a 1-0 for "the world" vs. "me". However, I had to extend the stack-allocated variable that holds a temporary string, in order to stop the program from segfaulting. The only way to fix the problem was manual code inspection, as neither ElectricFence, Valgrind or plain GDB could display the problem because of the significant stack scribbling. So, I'll take the liberty of calling it 1-1 so far - I do intend to follow up on this to make it a win though ;) I have *tons* of work to do before saturday, where I board a plane for france, to spend next eight days skiing in the alps. I don't think I will be able to give you a proper followup until after the vacation. Say, in about one and a half week from now. However, once back, I'll be fully energized and ready to take on the world - my asbestos underwear will be ready, my tinfoil hat polished up, and by then I think my lithium prescription should have run out again. All in all, I'll be back! :) Cheers, / jakob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Wed Jan 28 11:19:09 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed, 28 Jan 2004 11:19:09 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <200401280949.10516.ctibirna@giref.ulaval.ca> Message-ID: > And here I argue that this is where C++'s strength becomes evident. STL is > _standard_ (hence its name). Thus, for a C++ programmer, Jakob's code is > comprehensible in one 15second long glimpse and this programmer can I'm less enthusiastic about STL for a couple reasons. first, it's not always obvious whether it's being used correctly. STL is pretty big - more in the way that Java is basically a huge library, incidentally glued together with a mediocre scripting language and what amounts to emacs bytecodes. STL is probably a valuable toolkit for people who've invested in learning it. frankly, I haven't, though I'm a C++ programmer. I find that most of my experimental coding happens in Perl (which is quite a lot handier than STL, I think). and for non-experimental code, I'm usually too concerned with space or time efficiency to want to trust a big library like STL. > > another try some day in the future when I have time and have found a > > really good book on it. > > Lippman's "C++ Primer". I loved this in its first edition (more than a few years ago). in fact, I must have raved about it a lot, since someone made off with my copy. I looked at a later edition in a bookstore and was un-enthralled, though. perhaps I'm just surlier than I used to be, or perhaps the book changed. I remember being very impressed with Coplien's "Advanced C++". Stroustrup's books are also worth reading, at least parts of. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Tue Jan 27 23:43:54 2004 From: bill at math.ucdavis.edu (Bill Broadley) Date: Tue, 27 Jan 2004 20:43:54 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <20040128044354.GA10117@cse.ucdavis.edu> On Tue, Jan 27, 2004 at 09:09:33PM -0500, Mark Hahn wrote: > I draw several conclusions from the VT cluster: > > - HPL is very cache-friendly. the dual-G5 has an impressive peak > flops rating, but pretty mediocre memory bandwidth. anyone have a > Stream result for a G5? I'd guess around 2.5 GB/s. gcc -v # default for osx-10.3 Reading specs from /usr/libexec/gcc/darwin/ppc/3.3/specs Thread model: posix gcc version 3.3 20030304 (Apple Computer, Inc. build 1495) gcc -O1 stream_d.c second_wall.c Total memory required = 183.1 MB. Function Rate (MB/s) RMS time Min time Max time Copy: 2040.0621 0.0629 0.0627 0.0632 Scale: 2055.3702 0.0624 0.0623 0.0631 Add: 2141.2825 0.0901 0.0897 0.0908 Triad: 2100.9326 0.0917 0.0914 0.0921 -O2 Function Rate (MB/s) RMS time Min time Max time Copy: 2058.1437 0.0626 0.0622 0.0630 Scale: 1987.8218 0.0646 0.0644 0.0647 Add: 2150.3450 0.0897 0.0893 0.0902 Triad: 2155.9465 0.0893 0.0891 0.0895 -O3 Function Rate (MB/s) RMS time Min time Max time Copy: 2053.5855 0.0625 0.0623 0.0629 Scale: 1985.6382 0.0646 0.0645 0.0651 Add: 2148.7557 0.0897 0.0894 0.0901 Triad: 2155.1214 0.0894 0.0891 0.0896 -- Bill Broadley Computational Science and Engineering UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 13:01:19 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 13:01:19 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <20040128135402.GM814@unthought.net> Message-ID: On Wed, 28 Jan 2004, Jakob Oestergaard wrote: > I have *tons* of work to do before saturday, where I board a plane for > france, to spend next eight days skiing in the alps. I don't think I > will be able to give you a proper followup until after the vacation. > Say, in about one and a half week from now. > > However, once back, I'll be fully energized and ready to take on the > world - my asbestos underwear will be ready, my tinfoil hat polished up, > and by then I think my lithium prescription should have run out again. > > All in all, I'll be back! :) Awwwww, that hash-table example has tons of unnecessary overhead. Are you gonna make me actually WRITE a C variant to compete? Maybe I should start by posting a related program I wrote just for fun. I like to play hangman, and strategically a knowledge of letter frequency in the English language seemed like a useful thing to have. After all, everybody knows that 'e' is the most common letter, but what about the second, third, twenty-third most common letter? So one day, bored, I wrote letcount, which takes a filename as an argument, opens the file, reads it, and counts the letters. It was the first program I ever wrote that compiled and worked perfectly the first try! Not a lot of lines, but still...:-) Note that if all one wishes is to count the words, this program could be trivially modified with a snippet to determine word boundaries and increment a counter. If one wishes to count the words and save them (non-uniqued) in order, an additional snippet that moves each non-word-break character into a wor buffer, together with a realloc and a malloc (realloc the **char word array, malloc the spac for the word) would yield a scalable and storage-minimal solution. If one REALLY wants performance (and don't care about memory usage, sensible on a large memory modern system) one could realloc in largish blocks or simply malloc "more than enough" space (storing a million words less than or equal to rgb at lilith|T:144>echo supercalifragilisticexpialadotious | wc 1 1 35 in length requires a whopping 36 MB of memory) hard allocation of char words[1000000][36]; would yield the fastest possible access with no indirected pointer address resolution at all at the expense of being horribly wasteful in space that is probably idle anyway. Note that this wouldn't be efficient for smaller files as it probably takes the kernel some time to free up a block of memory that size; page size requests would presumably be much faster up to a point. Again, in addition to being perhaps the fastest possible code (note that scanning the entire stream of characters inline in this way probably would beat using strtok(), although not by much since this is likely pretty much what strtok does:-) the code (included below) is READABLE! At least I think so. I didn't even comment it the way I usually do -- open the file, stream it in, count the letters, trivially and inefficiently sort the count vector, print the sorted result in place. As the result of running the code on the linux words dictionary shows, it is pretty fast -- it literally returns a count as my finger is lifting from depressing the enter key, and the bulk of the DELAY is writing to stdout at 9600 baud or whatever an xterm supports, not the 0.03 seconds it actually takes to run. So there are LOTS of ways to solve your challenge problem in C; static memory allocation (wasteful but fast, especially for big files/streams), dynamic memory allocation (probably optimal, especially if one allocates roughly a page at a time on demand as the previous page is filled and maintain your own table of word boundaries as pointer offsets in an otherwise linear buffer), dynamic allocation the simpler and probably more readable/portable way (a word at a time in a char **words list), dynamic allocation of the words as "simple objects" (build a linked list struct of some sort to hold words and perhaps indices for use in a later sort or uniqueing step), and of course the hash. In each case the performance tradeoffs for using one memory management approach or the other are fairly transparent, and in each case one can choose between using a tool like e.g. strtok or doing it yourself. Of course, some of these have significant risks associated with them in the hands of a novice programmer or result in code that isn't horribly portable -- writing the incoming stream as rapidly as possible into a resize-on-demand buffer and recording word offsets in a linear integer (pointer) vector requires visualizing the way the memory space is laid out and is very close to the way one would likely do this in assembler, but accessing a word as (char *) buffer[offset[13]] isn't everybody's cup of tea (including mine, unless the code HAD to be really fast). So sure, give it your best shot in C++ using C++ constructs, but I guarantee you won't beat the best one can do in C and probably won't come terribly close unless you write the C++ code, after all, in C. I'll see if I can spare an hour or so this afternoon to hack letcount into wordcount using one or another of the buffer storage mechanisms I mentioned above -- the buffer[offset[]] actually seems kinda cute, come to think of it, but I may use strtok() instead of my code as a base to make the code appear shorter. Especially since it RETURNS the offset in buffer -- I think my entire program would be reading the file into a big buffer and a single while loop where strtok extracts the offsets, puts them into a vector, and counts. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu rgb at lilith|T:141>/usr/bin/time ./letcount /usr/share/dict/linux.words Opening /usr/share/dict/linux.words for counting e: 42715 i: 31459 s: 29652 a: 28984 r: 27064 n: 26981 t: 24614 o: 21613 l: 19484 c: 15018 d: 13857 u: 11723 g: 10342 p: 10073 m: 9807 h: 7812 b: 7371 y: 6006 f: 4932 v: 3973 k: 3209 w: 3073 z: 1631 x: 1054 j: 727 q: 682 Command exited with non-zero status 7 0.03user 0.00system 0:00.40elapsed 7%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (93major+14minor)pagefaults 0swaps %< snip snip ======================================================= #include main(int argc, char **argv) { char filename[128],buf[256]; int i,j,biggest,bigind,count[26]; FILE *infile; if(argc != 2){ fprintf(stderr,"Usage: letcount filename\n"); exit(1); } argv++; sprintf(filename,"%s",*argv); printf("Opening %s for counting\n",filename); infile = fopen(filename,"r"); for(i=0;i<26;i++) count[i] = 0; while(fgets(buf,256,infile)!= NULL){ /* printf("%s",buf); */ i = 0; while(buf[i] != 0){ if(buf[i] > 64 && buf[i] < 92) count[buf[i]-65]++; if(buf[i] > 96 && buf[i] < 124) count[buf[i]-97]++; i++; } } for(i=0;i<26;i++){ biggest = 0; for(j=0;j<26;j++){ if(count[j] > biggest) { biggest = count[j]; bigind = j; } } printf("%c: %d\n",bigind+97,biggest); count[bigind] = 0; } } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at fnal.gov Wed Jan 28 13:16:26 2004 From: djholm at fnal.gov (Don Holmgren) Date: Wed, 28 Jan 2004 12:16:26 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: With gcc 2.95.3, -O3, I get on a dual 2.0 GHz G5: Total memory required = 183.1 MB, walltime clock ------------------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Copy: 2848.7459 0.0450 0.0449 0.0457 Scale: 2851.4134 0.0453 0.0449 0.0457 Add: 3470.4630 0.0556 0.0553 0.0557 Triad: 3456.0981 0.0557 0.0556 0.0561 and with the beta IBM compiler (VAC 6.0, -O5): ------------------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Copy: 2923.5762 0.0439 0.0438 0.0441 Scale: 2566.9303 0.0500 0.0499 0.0502 Add: 2304.2857 0.0834 0.0833 0.0836 Triad: 2339.1539 0.0822 0.0821 0.0823 The data bus is a bit unconventional, in that it's split - 32 bits read-only, 32 bits write-only. The two halves can run at the same time. For moving data, that configuration is nicely balanced. For computations, it may not be so well balanced. For example, most of the flops in one of our codes occur doing small complex matrix-vector multiplies. One typical kernel reads 96 single precision operands, and writes only 6 single precision results. The effective memory bandwidth for this is much reduced - instead of a 1 GHz FSB, it's more like an effective 500 MHz FSB. Don Holmgren Fermilab On Tue, 27 Jan 2004, Mark Hahn wrote: > I draw several conclusions from the VT cluster: > > - HPL is very cache-friendly. the dual-G5 has an impressive peak > flops rating, but pretty mediocre memory bandwidth. anyone have a > Stream result for a G5? I'd guess around 2.5 GB/s. > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 13:55:42 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 13:55:42 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <200401280949.10516.ctibirna@giref.ulaval.ca> Message-ID: On Wed, 28 Jan 2004, Cristian Tibirna wrote: > And once again C++ fills the expectation here, IMO. Paraphrasing you: "This > works just marvey... with C++ exactly because any... entity they need to sort > and the sort criterion can be fed to the subroutine successfully and the > subroutine will happily work with". This, of course, thanks to C++'s > templates. And the result is even highly readable and quite compact usually > (and this is the reason I am ready to cope with performance penalties, > because I can put out code 10-20 times faster than in C -- OK, this is a > subjective measure -- and that code will only run 2-3 times slower -- citing > another poster "that's what clusters are for" ;-). Well, let's get serious here. A cluster is a large and costly instrument. You don't really mean that you are willing to waste 1/2-2/3 of the practical capacity of up to a million dollars worth of hardware by writing in C++'s generic objects in order to save a few weeks of writing truly efficient code in C. That makes no sense from a CBA point of view -- especially when it requires one to purchase 2-3x as many nodes to get the same time of completion of the job OR requires the project to run 2-3x as long (3 year instead of one, for example). You can write and debug a hell of a lot of C code in a year. You can also avoid being burned in effigy by competing groups wanting access to the resource. Everything else you say, well, that's just precisely the attitude of many C++ programmers and teachers of same and is perfectly OK with me. In fact, it is the attitude of most perl and python coders as well -- why bother with all the overhead of a compiler when you can have run-time checking, dynamically self-rewriting code (there's a clever perl trick -- code that writes itself!) threads and more, especially string parsing and regular expressions in conditionals), and never have to actually manage memory at all? For many tasks they are only a few times slower than compilers (they effectively tokenize/"compile" and then run library calls just like a compiled program) but they can make developing various classes of applications must faster. In fact, if I were to actually want to write a word counter to parse stdin (where I/O is already a rate limiter and speed is irrelevant) I'd almost certainly use perl myself, just as others (who like a more OO interface) would use python. Neither perl nor python is suitable for use on a beowulf to do real work; C++ doubtless is, but only if it is used "correctly" so that parts and data structures that need to be in plain old C for reasons of efficiency are. This is why I'm ecumenical, however strong my own prejud-- I mean "calmly reasoned and wise opinions":-). There is a spectrum of languages, and there are tradeoffs on ease of use and encapsulation across the spectrum. As computers get ever faster and larger, the overhead cost associated with using interpreters and higher-level languages and libraries goes down not in relative terms but in absolute ones, often to the point of irrelevancy. Notable exceptions include HIGH PERFORMANCE COMPUTING (the principle topic of this list:-) where performance is a PRIMARY focus, and a whole variety of tasks that tend to be studied by computer scientists because of their unfavorable scaling properties and consequent large demands on whatever system is running them. These categories are far from static as Moore's Law does its thing -- the reason PL/1 is a vanished language is in part because it WAS a high-overhead language for its era (IBM mainframe computing) and the resource it wasted (CPU time at truly exorbitant rates) was EXPENSIVE. The fact that the compiler would take monkeytype and turn it into SOME sort of program with its syntax error correction probably didn't help...given that one paid for the bad runs along with the good.;-) Also, Real Computer Scientists work out algorithms and port code and turn it into a library routine and suddenly a difficult problem DOES have a pretty good and even efficient solution in a given language/library. And then there is ATLAS, where the library isn't just good, it is dynamically self-optimizing. So by all means, seek your own comfort level, and optimize tradeoffs in accord with your own abilities and needs... > Lippman's "C++ Primer". > > Thanks for listening. Thanks for the pointer to the book. I'll give it a look next time I'm in B&N. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 11:29:18 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 11:29:18 -0500 (EST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: Message-ID: On Tue, 27 Jan 2004, Alvin Oga wrote: > > hi ya robert > > On Wed, 28 Jan 2004, Robert G. Brown wrote: > > > rpcinfo is also your friend: > > > > rgb at lilith|T:3#rpcinfo -p lilith > > program vers proto port > > 100000 2 tcp 111 portmapper > > 100000 2 udp 111 portmapper > > 100024 1 udp 32768 status > > 100024 1 tcp 32768 status > > 391002 2 tcp 32769 sgi_fam > > > > As a general rule, I'd consider the portmapper and rpc stuff to be a > > moderate security risk; lilith lives inside firewalls only. I have had > > direct experience of systems cracked in the past through portmapper > > bugs, which perhaps makes me a bit paranoid. > > did you run the secure versions of each ?? > secure rpc > secure portmap > secure nfs This was a few years ago. Remember, I've been a Unix sysadmin since, oh, 1986 or 1987 or thereabouts. As in I remember the Morris worm, a slew of sendmail and lpd attacks, an emacs bug, syslogd attacks, the good old days where any user armed with a userspace nfs kit could access any user's NFS space, and much, much, more. The portmapper exploit was particularly annoying and was one of the things (to remain vaguely on topic:-) that propelled us from Slackware to Red Hat as a departmental base -- our previous head sysadmin (by this time I was a "part time" sysadmin of our department network and our primary sysadmin kept pretty tight control of the LAN so I basically did ver little actual admin work) had left our department, and his first replacement turned out to be a midlevel disaster and permitted sigificant spread in the actual set of installed packages on all the various systems in the department. As in there were probably three or four different revisions and patchlevels of portmap running on different systems. Well, THAT sysadmin went away, I found myself once again in charge of the network literally overnight until we hired a replacement, and almost the first thing that happened was that about three or four departmental systems turned out to be vulnerable to a portmap buffer overwrite attack, were exploited, and I spent week of my life cleaning up. This is a salutary lesson to the novices out there; to quote from A Thousand Nights and a Night (various places) "it should be graven on the eyelid corners as a warner to whomso would be warned". It is pretty much essential to: a) keep systems patched and up to date, especially wrt security fixes but of course performance and bugfixes are also important. b) keep all systems in a LAN patched at the SAME level -- all it takes is one bad system and your whole network is compromised, as many attackers who get a user account can find a way to promote to root, and a user who gets root on one system in a LAN has many routes to eventually promote to root on the department servers and LAN. Heterogeneity is an enemy of good systems administration, as it dilutes effort and limits scalability. c) In case it isn't obvious, this suggests that sound management practice includes something like: pick a packaged distribution that is aggressively maintained, especially wrt security; use kickstart or other scripting/imaging tools to construct system templates so that all your systems will be as similar as possible (same packages, same general configuration); use yum or related tools in other distributions to fully automate system updates from a common package repository, so that keeping the package repository up to date is precisely equivalent to keeping all your systems up to date. d) Minimize exposure. This means that unless you need a daemon or service, don't run it. Buffer overwrite attacks via "essential" systems daemons have probably been the single most common avenue of successful attack in the history of the non-Windows internet, and they have potentially disasterous scaling properties as the Morris worm (and more contemporaneously, the Slammer) clearly demonstrated. Worms like this might be classified as dangerous terrorist munitions, as they can cause more economic damage in a day than a hundred car bombs, especially with the Internet at this point being the base of a huge amount of economic activity. e) Since there are always daemons and services that are more or less REQUIRED inside a LAN in order not to lose productivity THAT way run some sort of firewall to isolate access to the exposed ports to only the hosts that need the access. It is relatively easy to police these hosts and their users with tool-enhanced vigilance (syslog-ng etc) and a sucker rod (read man syslogd if the reference escapes you:-) to school any local users who want to try any 1337 crap or mad skills). iptables or ipchains permit one remarkably fine-grained control of port access rights even within a LAN -- every host runs its own mini-firewall. NFS in particular and most RPC services are very definition, BTW, of services that should have carefully regulated access. NFS should pretty much NEVER be used over a WAN (this may have changed, of course, but Old Guys will always be paranoid and not really believe that it has:-). f) A lot of folks organize their internal LAN in security layers, with highly restricted access to servers, very limited privileges to clients (to make it more difficult for a successful exploit of a client to be promotable to the entire LAN and especially to the servers, ESPECIALLY to install repositories). There is no substitute for vigilance, though -- most exploits are discovered by admins or users noting and reporting something "odd" or something "broken" and following up aggressively. Again kickstart/yum/PXE are your friend, as cleanup nowadays is just Ctrl-Alt-Del into a full reinstall of a compromised client, coupled with a scrubbing of any affected user accounts. g) Keep Good Backups. With highly automated install/maintenance tools, cleanup is relatively cheap, but you may well have to restore corrupted files or home directories or (in the worst of cases) rebuild servers or repositories. Security is always a trade-off; you give up the ability to do certain kinds of work easily and quickly in order to achieve a "hardened" status, but the lost work and productivity is also a "cost", one that accrues linearly in time, and can easily exceed the "cost" of a successful exploit. Many linuces now install by default into what amounts to a tuckee-tail hard firewalled mode with access to virtually all ports blocked via iptables and turned off in chkconfig. This is a good thing, but it can mean that lots of things "mysteriously" don't work unless you know what your are doing and know how to enable them. Which is ALSO a good thing; if you don't know how to turn them on, could you possibly know enough to be vigilant and keep a system with them enabled secure? Learning the one should always be accompanied by learning the other... > at "*.duke.edu" guess you folks would be a prime target... I think that everybody is a prime target. Many, even most, exploits these days are driven by fully automated engines; there are relatively few incidents of pimply-faced cracker script-kiddies targeting systems one at a time (working through the usual string of intermediary successes). Even the script-kids use tools that simply sweep IP spaces probing for open/vulnerable ports, and the worms combine a successful sweep with exploitation of the port, implanting of a clone of the worm, and a continuation of propagation process. At Duke, at least, we have a VERY competent security officer (Chris Cramer) who does a damn fine job coping with and minimizing our exposure on an institutional basis. He's a beowulfer, BTW, and might even read (eventually) these words of glowing praise and wonder what the hell I want out of him now:-). (Lunch, Chris, Lunch. Love those free lunches:-) He is backed up by an equally excellent collective sysadmin group; they keep mot of the organized LAN spaces professionally secured without his constant browbeating so he can focus on the bleeding, supperating wound that is Windows. Today is of course not a good day for him on that front as YAWE (yet another windows exploit) got enough traction yesterday that answering my email was distinctly tedious (note that I do NOT use the word Hi in my subject line above, not wishing to be procmailed out by others in the same pickle). I'm certain that it is popping up in the dorms if nowhere else, as we obviously have a hard time controlling "private" student systems in the dorms, who are both "inside" and "outside" Duke's trusted space (as outside as we can make them, but nevertheless possessing trusted access to certain resources). So sure, Duke is a major target by virtue of its large IP-space volume if nothing else, but it is also fairly aggressively and competently policed. rgb > > c ya > alvin > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rmiguel at usmp.edu.pe Wed Jan 28 14:46:31 2004 From: rmiguel at usmp.edu.pe (Richard Miguel) Date: Wed, 28 Jan 2004 14:46:31 -0500 Subject: [Beowulf] about compilers for cluster beowulf References: Message-ID: <000001c3e5db$8d6d1280$1101000a@cpn.senamhi.gob.pe> Hi ... this is my first post . So, i have a beowulf cluster with 27 nodes PIV 2.4 Ghz Intel, my question is if is possible that Fortran Intel compiler and PGI compiler are installed in a same machine without conflicts (server node). thanks for yor answers.. R. Miguel _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nordquist at geosci.uchicago.edu Wed Jan 28 15:13:46 2004 From: nordquist at geosci.uchicago.edu (Russell Nordquist) Date: Wed, 28 Jan 2004 14:13:46 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <1075320826.5224.7.camel@vitalstatistix> in other news here is an article written by the dean of the college of engineering on the bigMac http://news.com.com/2010-7337-5149087.html i finally know what is was built to do....solve "grand challenge" problems. BTW, does anyone out there own a smallMac cluster that would be willing to field some questions offline. were looking at smallish (~30 cpus) g5 cluster. we have several linux clusters but this is my would be my first stab at Xserves. russell _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 28 17:47:37 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 28 Jan 2004 17:47:37 -0500 (EST) Subject: [Beowulf] About a survey request In-Reply-To: Message-ID: Has anyone else received an HPC survey request from: microsoftresearch at surveysite.com I received one today. I declined the survey as they will not make the results available to the participants/community. Just curious. Doug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 11:42:05 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 11:42:05 -0500 (EST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: <20040128090912.51497.qmail@web21506.mail.yahoo.com> Message-ID: On Wed, 28 Jan 2004, prakash borade wrote: > rgb thanks for your support > > well i am trying to wite an client server application > ( monitoring sofware ) for my beowulf cluster > tell me what would be efficient either sockets or RPC Raw sockets. Eschew RPC. You should also just visit http://www.phy.duke.edu/~rgb/Beowulf/beowulf.php or brahma and pull xmlsysd and wulfstat. It IS client/server monitoring software, fully GPL, and will build and install and (hopefully) just plain work for you. If you are interested in this as a "learning project" and not because you just need a monitoring package, consider using xmlsysd as a base and contributing to the project rather than reinventing all the wheels that it contains. For example, it already contains forking daemon and xinetd code (so the daemon part is "done". It already knows how to open most of the "interesting" files in /proc, read them and encapsulate their contents in XML and send them to clients on demand, and even uses tricks such as rewinding the files instead of closing and reopening them to minimize the fairly significant overhead associated with opening a file in the first place. I think you'll find that xmlsysd is pretty efficiently written and lightweight and stable and functional and non-leaky and (due to its use of xml to encapsulate results) scalable and accessible to a variety of clients. It is even moderately extensible -- it is fairly easy to add a new thing to be monitored, once you know the code (not quite "follow the instructions" easy, though -- at this point I'm probably the only person who could do it without extensively studying the code). The thing that really needs to be done is develop more monitoring (client) tools. wulfstat isn't bad, really, especially for smallish clusters, but it would be lovely to have "gwulfstat" (gtk) and "wwulfstat" (web) monitoring tools. It would also be lovely to have some command line tools that could be backgrounded and poll results in a timed loop and write them out to a named logfile (both as a historical log trace of node load and to facilitate the creation of tools that could graph the traces and generate reports). SO if you are interested at all in helping out and joining the project, I could certainly use the help. I have too many projects running all at the same time to give this one the full attention it deserves. rgb > > would open socket on each machine put an extra load > on the cluster or is it managable > > --- "Robert G. Brown" wrote: > > On Tue, 27 Jan 2004, prakash borade wrote: > > > > > have any body tried RPC programing on Linux > > > i get > > > rpc unknown protocol error > > > i am trying to diaply the message on the remote > > host > > > > To do rpm programming you probably need a bunch of > > stuff configured and > > running that might or might not be turned off for > > security reasons. > > Check, for example, to see if the portmapper is > > running. Also check > > iptables and/or ipchains to see if they are running > > and if so if any > > rules block access to the portmapper. > > > > rpcinfo is also your friend: > > > > rgb at lilith|T:3#rpcinfo -p lilith > > program vers proto port > > 100000 2 tcp 111 portmapper > > 100000 2 udp 111 portmapper > > 100024 1 udp 32768 status > > 100024 1 tcp 32768 status > > 391002 2 tcp 32769 sgi_fam > > > > As a general rule, I'd consider the portmapper and > > rpc stuff to be a > > moderate security risk; lilith lives inside > > firewalls only. I have had > > direct experience of systems cracked in the past > > through portmapper > > bugs, which perhaps makes me a bit paranoid. > > > > What exactly are you trying to do? > > > > rgb > > > > > > > > __________________________________ > > > Do you Yahoo!? > > > Yahoo! SiteBuilder - Free web site building tool. > > Try it! > > > http://webhosting.yahoo.com/ps/sb/ > > > _______________________________________________ > > > Beowulf mailing list, Beowulf at beowulf.org > > > To change your subscription (digest mode or > > unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > > -- > > Robert G. Brown > > http://www.phy.duke.edu/~rgb/ > > Duke University Dept. of Physics, Box 90305 > > Durham, N.C. 27708-0305 > > Phone: 1-919-660-2567 Fax: 919-660-2525 > > email:rgb at phy.duke.edu > > > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or > > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctibirna at giref.ulaval.ca Wed Jan 28 15:17:54 2004 From: ctibirna at giref.ulaval.ca (Cristian Tibirna) Date: Wed, 28 Jan 2004 15:17:54 -0500 Subject: [Beowulf] C vs C++ challenge In-Reply-To: References: Message-ID: <200401281517.54712.ctibirna@giref.ulaval.ca> Le Mercredi 28 Janvier 2004 13:55, Robert G. Brown a ?crit: > > 2-3 times slower -- citing another poster "that's what clusters are for" > > ;-). > > Well, let's get serious here. A cluster is a large and costly > instrument. You don't really mean that you are willing to waste 1/2-2/3 > of the practical capacity of up to a million dollars worth of hardware > by writing in C++'s generic objects in order to save a few weeks of > writing truly efficient code in C. Of course not. Thus the smiley. And of course I (very often) get frustrated at the "laziness" of the popular C++ compilers as well at the performance drop (that sometimes becomes unacceptable by any standards). But getting frustrated at compilers is a bearable psychosociological inconvenience compared with the rage (accompanied sometimes with physical violence ;-) that badly written code in a too flexible (or too poor) language can inflict on me. There's the aspect of software engineering in temporally, spacially and knoweldgeably sparse teams. In projects with many millions LOCs, developed over many (>5) years, employing 30 to 40 persons (many of them passage students with brief interventions), especially projects with a pedagogic penchant, anything less strict and with a lesser standardized high level library than C++ wouldn't cut it. But you of course already touched the topic of pedagogic intents before in this thread. I dare extract a conclusion (as the thread drifts fastly;-): "the right tool for the job at hand" remains the essential wisdom to keep in mind when comes to choosing languages. -- Cristian Tibirna (1-418-) 656-2131 / 4340 Laval University - Quebec, CAN ... http://www.giref.ulaval.ca/~ctibirna Research professional at GIREF ... ctibirna at giref.ulaval.ca PhD Student - Chemical Engng ... tibirna at gch.ulaval.ca _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Wed Jan 28 14:58:16 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Wed, 28 Jan 2004 11:58:16 -0800 (PST) Subject: [Beowulf] C vs C++ challenge - hw too In-Reply-To: Message-ID: hi ya robert On Wed, 28 Jan 2004, Robert G. Brown wrote: > On Wed, 28 Jan 2004, Cristian Tibirna wrote: > > > (and this is the reason I am ready to cope with performance penalties, > > because I can put out code 10-20 times faster than in C -- OK, this is a > > subjective measure -- and that code will only run 2-3 times slower -- citing > > another poster "that's what clusters are for" ;-). yupperz... people will do things the fastest/easiest way .. :-) > Well, let's get serious here. A cluster is a large and costly > instrument. You don't really mean that you are willing to waste 1/2-2/3 > of the practical capacity of up to a million dollars worth of hardware > by writing in C++'s generic objects in order to save a few weeks of > writing truly efficient code in C. That makes no sense from a CBA point > of view -- especially when it requires one to purchase 2-3x as many > nodes to get the same time of completion of the job OR requires the > project to run 2-3x as long (3 year instead of one, for example). You > can write and debug a hell of a lot of C code in a year. You can also > avoid being burned in effigy by competing groups wanting access to the > resource. sometimes ... people dont do the "full total analysis" and/or it can be swayed one way vs another sometimes cheaper to hire another programmer to do the work .. while one carries on with one's own area of expertise -- and dont forget to add the issues of hw costs to the list of too - soemtimes i wonder why people want dual-cpu or quad-cpu - found 1 guy ( company ) that says that their custom code runs linearly with increasing number of cpu's ... ( they write custom finite element stuff ) - otherwise it might be cheapr to run 2 jobs on 2 machines than 1.5 jobs on one dual-cpu machines ( aka 2 machines ) have fun alvin -- latest fun puzzle ... one can fit 4 1Us in one 4U shelf space one can also fit 10 1Us in one 4U shelf space ( magic of atx blades .. 2.5x the capacity in the same volume ) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kdunder at sandia.gov Wed Jan 28 15:35:49 2004 From: kdunder at sandia.gov (Keith D. Underwood) Date: Wed, 28 Jan 2004 13:35:49 -0700 Subject: [Beowulf] C vs C++ challenge In-Reply-To: <200401280949.10516.ctibirna@giref.ulaval.ca> References: <200401280949.10516.ctibirna@giref.ulaval.ca> Message-ID: <1075305637.4039.31.camel@thinkpad> > And once again C++ fills the expectation here, IMO. Paraphrasing you: "This > works just marvey... with C++ exactly because any... entity they need to sort > and the sort criterion can be fed to the subroutine successfully and the > subroutine will happily work with". This, of course, thanks to C++'s > templates. And the result is even highly readable and quite compact usually > (and this is the reason I am ready to cope with performance penalties, > because I can put out code 10-20 times faster than in C -- OK, this is a > subjective measure -- and that code will only run 2-3 times slower -- citing > another poster "that's what clusters are for" ;-). So, in some very real examples, the performance difference between an existing Fortran code and the first version of a C++ code was more like a factor of 10. When you look at the impact of that in real dollars on, say, an ASCI scale computing platform that costs $90+ million, it is huge. If that code is used for very long, you are wasting 90% of the machine or $80+ million (and we won't count the very significant cost of electricity or cooling for a machine of that scale). So, then the programmers get to spend their days trying to get some of that performance back. In one example, a Fortran code was being replaced by a C++ code. The C++ code was way too slow and the core computational routines were replaced with their Fortran equivalent. The code was still a factor of 2 to 3 behind. By the end of it, a lot more effort had been spent on trying to make the C++ code run fast than was spent when the C++ code was scrapped and the Fortran code was updated instead. Ease of programming is good, but the performance implications cannot be ignored. Even a factor of 2 is a lot of money when you talk about a large computing resource. Even a large computing resources as inexpensive as the Virginia Tech Mac cluster. Keith _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 28 17:59:37 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 28 Jan 2004 17:59:37 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message-ID: The first thought I had was "what will they do with all the old systems?" Then it hit me. They put a fancy sticker on each box that says "This machine was part of the third fastest supercomputer on the planet Nov. 2003" or something similar. Also put a serial number on the tag and provide a "certificate of authenticity" from VT. My guess is they can make a little bit on the whole deal. I wager they would sell rather quickly. Alumni eat this kind of thing up. For those interested, my old www.cluster-rant.com site has morphed into the new www.clusterworld.com site. You can check out issue contents, submit stories, check out the polls, and rant about clusters. Doug -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From patrick at myri.com Wed Jan 28 19:03:27 2004 From: patrick at myri.com (Patrick Geoffray) Date: Wed, 28 Jan 2004 19:03:27 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075320826.5224.7.camel@vitalstatistix> References: <1075320826.5224.7.camel@vitalstatistix> Message-ID: <40184DCF.8090800@myri.com> Hi Russel, Russell Nordquist wrote: > to field some questions offline. were looking at smallish (~30 cpus) g5 > cluster. we have several linux clusters but this is my would be my first > stab at Xserves. You can also look at the JS-20 blades from IBM, 28 cpus in 7U. Maybe a little pricey though. Patrick -- Patrick Geoffray Myricom, Inc. http://www.myri.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nordquist at geosci.uchicago.edu Wed Jan 28 20:58:42 2004 From: nordquist at geosci.uchicago.edu (Russell Nordquist) Date: Wed, 28 Jan 2004 19:58:42 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <40184DCF.8090800@myri.com> References: <1075320826.5224.7.camel@vitalstatistix> <40184DCF.8090800@myri.com> Message-ID: <1075341521.5224.34.camel@vitalstatistix> On Wed, 2004-01-28 at 18:03, Patrick Geoffray wrote: > Hi Russel, > > Russell Nordquist wrote: > > to field some questions offline. were looking at smallish (~30 cpus) g5 > > cluster. we have several linux clusters but this is my would be my first > > stab at Xserves. > > You can also look at the JS-20 blades from IBM, 28 cpus in 7U. Maybe a > little pricey though. > I am working with an IBM rep to get a login to a JS-20 blade. From the ibm website a JS20 blade w/512M ram is $2,699.00. The chassis is 2,789.00 so.... around $40.5K. i think i'm missing a GiGe switch module though. in comparison, for g5 Xserves are $3k x 14 = $42. so it's similar. however the g5's are 2Ghz the JS-20's are 1.6Ghz..... on a related note....in the next day I am going to post (or prompt someone to post) about an interesting issue we are having with some code. basically it will run on g4's and g5's, but not Xeons/Athlons due to what appears to be underflow issues. the odd thing is that it doesn't work on an Opteron (in 64 bit mode) either. call this a teaser since i am going to get the author of the code to explain a bit more about it. this particular problem has me a bit vexed....i can't figure if it's a architecture or compiler issue. more on this tomorrow. russell > Patrick _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Wed Jan 28 21:20:22 2004 From: bill at math.ucdavis.edu (Bill Broadley) Date: Wed, 28 Jan 2004 18:20:22 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <40184DCF.8090800@myri.com> References: <1075320826.5224.7.camel@vitalstatistix> <40184DCF.8090800@myri.com> Message-ID: <20040129022022.GA19317@cse.ucdavis.edu> On Wed, Jan 28, 2004 at 07:03:27PM -0500, Patrick Geoffray wrote: > Hi Russel, > > Russell Nordquist wrote: > >to field some questions offline. were looking at smallish (~30 cpus) g5 > >cluster. we have several linux clusters but this is my would be my first > >stab at Xserves. > > You can also look at the JS-20 blades from IBM, 28 cpus in 7U. Maybe a > little pricey though. An IBM guy said (informally) approximately $2500 list per blade. Same ball park as the .edu dual g5 1u compute node. -- Bill Broadley Information Architect Computational Science and Engineering UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dc at hemeris.com Thu Jan 29 09:55:55 2004 From: dc at hemeris.com (dc) Date: Thu, 29 Jan 2004 15:55:55 +0100 Subject: [Beowulf] C vs C++ challenge (java version) Message-ID: Jakob Oestergaard wrote: > Let me throw out a challenge: > > "Write a program in C that reads from standard input until EOF, and > prints the number of distinct words found in that input" > > > I use an RB-tree to store the words I find, and print the number of > elements in the final tree. > > A comparable C solution shouldn't be more than a few thousand lines ;) > > Oh, right, I cheated and used STL. Well, STL is a big part of why C++ > is really handy, but go right ahead and use "glib" or whatever. > > I still guarantee you two things: > 1) Your code will be longer > 2) Your program will be slower > > As always, I love to be proven wrong ;) Here is another try at that, this time in Java. C++ code compiled with g++ 3.3 (-O3), times are 'real time' reported by the bash time function, the client and server columns correspond to java runs using the -client and -server options respectively. Java compiler and VM are the hotspot 1.4. file size C++ j client j server wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s big.txt 39389424 0m29.120s 0m13.972s 0m12.776s vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s Oh right, I cheated and used Java standard collections ;-) Seriously tough, Java is not as slow as some seem to believe. And optimizing the code at run-time (hotspot, code morphing,...) has the advantage that the optimizer has access to potentially useful information, both about the code itself and about the executing environment. -------------------------------------------------------------- import java.util.*; import java.util.regex.Pattern; import java.io.*; public class wordcount { public static void main (String args[]) { HashSet hs = new HashSet(); try { BufferedReader input = new BufferedReader(new InputStreamReader(System.in)); String s, words[]; Pattern p = Pattern.compile("\\s"); while ((s = input.readLine()) != null) { words = p.split(s); for (int i = 0; i < words.length; i++) hs.add(words[i]); } hs.remove(""); } catch (Throwable e) { e.printStackTrace(); } System.out.println("Words: " + hs.size()); } } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at fnal.gov Thu Jan 29 14:45:05 2004 From: djholm at fnal.gov (Don Holmgren) Date: Thu, 29 Jan 2004 13:45:05 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <40192F0C.5010809@lmco.com> References: <40192F0C.5010809@lmco.com> Message-ID: Here you go: 1 Copy of the IBM VAC version: Function Rate (MB/s) RMS time Min time Max time Copy: 2956.1260 0.0434 0.0433 0.0435 Scale: 2620.4804 0.0491 0.0488 0.0495 Add: 2308.5529 0.0833 0.0832 0.0837 Triad: 2371.3380 0.0810 0.0810 0.0812 2 simultaneous copies: Function Rate (MB/s) RMS time Min time Max time Copy: 1671.2143 0.0820 0.0766 0.0886 Scale: 1462.9393 0.0915 0.0875 0.0941 Add: 1296.6226 0.1519 0.1481 0.1575 Triad: 1378.7155 0.1425 0.1393 0.1507 Function Rate (MB/s) RMS time Min time Max time Copy: 1569.2244 0.0845 0.0816 0.0893 Scale: 1470.4361 0.0891 0.0870 0.0929 Add: 1324.9507 0.1505 0.1449 0.1545 Triad: 1481.1049 0.1448 0.1296 0.1509 These results are consistent with the Fortran MPI version, run with np=1 and np=2: bash-2.05a$ mpirun -np 1 stream_mpi ... ---------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 2910.9423 .0220 .0220 .0220 Scale: 2577.4299 .0249 .0248 .0249 Add: 2385.3296 .0403 .0402 .0404 Triad: 2456.4980 .0391 .0391 .0392 bash-2.05a$ mpirun -np 2 stream_mpi ... ---------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 2780.1300 .0465 .0460 .0469 Scale: 2779.6478 .0464 .0460 .0469 Add: 2649.7750 .0728 .0725 .0736 Triad: 2777.8189 .0698 .0691 .0709 Don On Thu, 29 Jan 2004, Jeff Layton wrote: > Don, > > Is there any way you could run two copies of Stream > at the same time, one on each CPU? I'd like to see the > effect of the memory subsystem. > > Thanks! > > Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeffrey.b.layton at lmco.com Thu Jan 29 11:04:28 2004 From: jeffrey.b.layton at lmco.com (Jeff Layton) Date: Thu, 29 Jan 2004 11:04:28 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <40192F0C.5010809@lmco.com> Don, Is there any way you could run two copies of Stream at the same time, one on each CPU? I'd like to see the effect of the memory subsystem. Thanks! Jeff > > With gcc 2.95.3, -O3, I get on a dual 2.0 GHz G5: > > > Total memory required = 183.1 MB, walltime clock > ------------------------------------------------------------- > Function Rate (MB/s) RMS time Min time Max time > Copy: 2848.7459 0.0450 0.0449 0.0457 > Scale: 2851.4134 0.0453 0.0449 0.0457 > Add: 3470.4630 0.0556 0.0553 0.0557 > Triad: 3456.0981 0.0557 0.0556 0.0561 > > and with the beta IBM compiler (VAC 6.0, -O5): > > ------------------------------------------------------------- > Function Rate (MB/s) RMS time Min time Max time > Copy: 2923.5762 0.0439 0.0438 0.0441 > Scale: 2566.9303 0.0500 0.0499 0.0502 > Add: 2304.2857 0.0834 0.0833 0.0836 > Triad: 2339.1539 0.0822 0.0821 0.0823 > > > The data bus is a bit unconventional, in that it's split - 32 bits > read-only, 32 bits write-only. The two halves can run at the same time. > > For moving data, that configuration is nicely balanced. For > computations, it may not be so well balanced. For example, most of the > flops in one of our codes occur doing small complex matrix-vector > multiplies. One typical kernel reads 96 single precision operands, and > writes only 6 single precision results. The effective memory bandwidth > for this is much reduced - instead of a 1 GHz FSB, it's more like an > effective 500 MHz FSB. > > Don Holmgren > Fermilab > > > On Tue, 27 Jan 2004, Mark Hahn wrote: > > > I draw several conclusions from the VT cluster: > > > > - HPL is very cache-friendly. the dual-G5 has an impressive peak > > flops rating, but pretty mediocre memory bandwidth. anyone > have a > > Stream result for a G5? I'd guess around 2.5 GB/s. > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Dr. Jeff Layton Aerodynamics and CFD Lockheed-Martin Aeronautical Company - Marietta _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 29 15:25:09 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 29 Jan 2004 12:25:09 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075341521.5224.34.camel@vitalstatistix> References: <1075320826.5224.7.camel@vitalstatistix> <40184DCF.8090800@myri.com> <1075341521.5224.34.camel@vitalstatistix> Message-ID: <20040129202509.GB1630@greglaptop.internal.keyresearch.com> On Wed, Jan 28, 2004 at 07:58:42PM -0600, Russell Nordquist wrote: > on a related note....in the next day I am going to post (or prompt > someone to post) about an interesting issue we are having with some > code. basically it will run on g4's and g5's, but not Xeons/Athlons due > to what appears to be underflow issues. The G5 in particular has the usual IBM feature that some float computations are done as double, with the answer only being truncated when variables are stored to memory. The IBM compiler has a switch that rounds things more normally. If this feature allows your code to run, then your code is probably the most guilty party. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 29 15:07:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 29 Jan 2004 15:07:08 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (C version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, dc wrote: > > I still guarantee you two things: > > 1) Your code will be longer > > 2) Your program will be slower > > > > As always, I love to be proven wrong ;) > > > Here is another try at that, this time in Java. > > file size C++ j client j server ... > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s And here is a version in C. It is longer, no question. It does its own memory management, in detail, in pages (which should be nearly optimally efficient). It is moderately robust, and smart enough to recognize all sorts of separators (when counting words, separators matter -- hence this program will find more words than e.g. wc because it splits things differently). It's timing (compared to a REALLY efficient wc): rgb at ganesh|T:406>/usr/bin/time wordcount shaks12.txt Opening shaks12.txt for counting Total Number of Words = 902303 Command exited with non-zero status 31 0.46user 0.02system 0:00.47elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (93major+2334minor)pagefaults 0swaps rgb at ganesh|T:407>/usr/bin/time wc shaks12.txt 124456 901325 5582655 shaks12.txt 0.17user 0.02system 0:00.19elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (130major+21minor)pagefaults 0swaps 0.47 wall clock time is far, far faster than C++ or java. It is still 2x and change wc, but it is far from optimally efficient, for many reasons: a) It uses fgets() instead of raw read(). It would be easy to change this but I'm lazy and haven't bothered as even the version I've come up with blows away the competition. At a guess, given an I/O dominated process it would cut processing time roughly in half if I did. b) It uses strsep() to tokenize lines of input read directly into a buffer (but a line at a time). wc works with the character stream directly (and splits words on fewer characters) and is much faster. c) It saves all the words as separate strings in the buffer and creates a list of offsets so each word can be addressed by its own index. This meets the requirement that not only do I count the words, I read them into a list where they can be accessed individually and (presumably) processed further. d) As noted, I'm lazy. If anyone complains about accessing the words via words+word_offset[i], I should point out that it is absolutely trivial to repack the offsets (now that we have them) in a char **wordlist so that the words can be addressed as wordlist[i]. If anyone complains that we "just got a char ** list of words", I should ALSO point out that with trivial modifications the code could allocate a struct { char *word; int index; int length; } *wordlist; and fill IT in so that each word is in a word "object". Neither of these steps would significantly increase the time required (more than a few percent). As for the code, well, 144 lines including all comments isn't horrible, and if I cut the comments and empty lines it's probably about 80. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu #include #include #include #include #define PAGE 4096 #define K 1024 static char delim[] = { ' ', /* blank */ (char) 9, /* tab */ ',', /* comma */ (char) 10, /* LF */ (char) 13, /* CR */ '.', /* . */ '!', /* ! */ '?', /* ? */ '"', /* " */ '*', /* * */ '<', /* < */ '>', /* > */ '@', /* @ */ '[', ']', '(', ')', '{', '}', '+', '=', '/', '\\', '#', ':', (char) 0 /* terminator */ }; main(int argc, char **argv) { char filename[128],*word,*words,*words_head,*words_tail,*words_last; int *word_offset; size_t offset,word_length,words_length,wordlist_length,word_count,i; FILE *infile; if(argc != 2){ fprintf(stderr,"Usage: wordcount filename\n"); exit(1); } argv++; sprintf(filename,"%s",*argv); printf("Opening %s for counting\n",filename); infile = fopen(filename,"r"); /* * The word list as a single buffer. We allocate two pages to start with, and * reallocate whenever the free space in the buffer drops below one page. We * also maintain a pointer to the beginning of the unused part of this buffer. */ words_length = 2*PAGE; words = (char *) malloc(words_length); words_head = words; words_tail = words + words_length; /* * We do the exact same thing here, sort of. We allocate a PAGE of offsets * (relative to the words head!) and reallocate whenever the number remaining * drops below K. i will both index the wordlist and count the words as we go. */ wordlist_length = PAGE; word_offset = (int *) malloc(wordlist_length*sizeof(int)); word_count = 0; /* * Each pass through the while loop reads a single line from * infile into words at offset wptr (current tail). We will * write the code to manage lines up to 1K in length. */ word_offset[0] = 0; while(fgets(words_head,(words_tail - words_head),infile)!= NULL){ /* * Check to add a PAGE to the words vector/buffer. Since realloc * may move pointer, we have to recompute offsets. */ if((words_tail - words_head) < PAGE) { words_length += PAGE; /* save offset */ offset = words_head - words; words = (char *) realloc(words,words_length); /* displace by saved offset */ words_head = words + offset; words_tail = words + words_length; } /* * Check to add a PAGE to the wordlist vector/buffer. Here we * don't have to worry about offsets. */ if((wordlist_length - word_count) < K) { wordlist_length += PAGE; word_offset = (int *) realloc(word_offset,wordlist_length*sizeof(int)); } /* * Now we rip through the line. */ while(1){ /* Get a word */ word = strsep(&words_head,delim); if(words_head == NULL) break; /* Save its address for after we are done with this line */ words_last = words_head; /* Save its offset in the offset vector */ if(word[0] != (char)0){ word_offset[word_count++] = word - words; /* for debugging offsets etc */ /* printf("%d: word at %0x = %s, offset is %d\n",word_count,word,word,word_offset[word_count - 1]); */ } } /* * strsep finishes parsing a line with a NULL. Restore the right pointer. */ words_head = words_last; } /* * That's about it. We print the results. * (or not, in a speed trial). for(i=0;i (Alvin Oga's message of "Tue, 27 Jan 2004 22:26:17 -0800 (PST)") References: Message-ID: Alvin Oga writes: > On Wed, 28 Jan 2004, Robert G. Brown wrote: > >> As a general rule, I'd consider the portmapper and rpc stuff to be a >> moderate security risk; lilith lives inside firewalls only. I have had >> direct experience of systems cracked in the past through portmapper >> bugs, which perhaps makes me a bit paranoid. > > did you run the secure versions of each ?? > secure rpc > secure portmap > secure nfs Well, "secure" in that context means "encrypted". That might protect your traffic against snoopers, but you're still as exposed to security holes due to portmapper bugs. I've seen too many RPC related buffer overrun holes in the past to expose any RPC services to the Internet at large, whether "secure" or not. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Thu Jan 29 15:47:38 2004 From: agrajag at dragaera.net (Jag) Date: Thu, 29 Jan 2004 15:47:38 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075320826.5224.7.camel@vitalstatistix> References: <1075320826.5224.7.camel@vitalstatistix> Message-ID: <1075409258.4438.19.camel@pel> On Wed, 2004-01-28 at 15:13, Russell Nordquist wrote: > in other news here is an article written by the dean of the college of > engineering on the bigMac > > http://news.com.com/2010-7337-5149087.html > Interesting article. It seems they don't know what their cluster is called. The article wants to call it 'Virginia Tech X', whereas the website for the cluster wants to call it Terascale: http://computing.vt.edu/research_computing/terascale/ Another interesting thing is the article mentions Deja Vu, a proprietary piece of software. If you check this site (http://mac.excaliburworld.com/teracluster.html), it seems that Deja Vu is written in house. I actually went up and saw the cluster last month. When I was there, they still weren't production ready, and didn't have any real users. Now they're suddenly upgrading everything.. sounds very curious to me. Btw, when I was there, it was indicated that the Deja Vu software was written by a faculty member in their computer science department. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 29 18:22:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 29 Jan 2004 18:22:08 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, Selva Nair wrote: > But this one does not count unique words, does it? Hmmm. I read "distinct words" as being words distinguished from one another by separators, not as unique words. As in "two and two" are one line, three distinct words, but only two unique words. I'll have to hack a bit tomorrow to make the C do only distinct words. For that the hash table solution is probably optimal, but I might as well see how much extra time the loops take. > Here is the script: > > #!/bin/awk -f > { > for(i = 1; i <= NF; i++) { > if (words[$i]) continue; > words[$i] = 1 ; > ++nwords; > } > } > END { > printf "Number of distinct words = %i\n", nwords; > } I always have liked awk;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 29 17:56:41 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 29 Jan 2004 17:56:41 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: <1075408742.7848.73.camel@squash.scalableinformatics.com> Message-ID: On Thu, 29 Jan 2004, Joe Landman wrote: > On Thu, 2004-01-29 at 09:55, dc wrote: > > file size C++ j client j server > > wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s > > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > > big.txt 39389424 0m29.120s 0m13.972s 0m12.776s > > vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s > > Where did these files come from? Would be nice to try out > non-C++/non-Java solutions with. At least shaks12.txt is from project gutenberg and shows up as a direct link to there, first thing, from Google on a "shaks12 txt" search. I haven't tried the rest of them -- if a tool works for Shakespeare it SHOULD work for any of the rest. The only dicey point will arise when the filesize starts to compete with available free memory, as one of the rules of the "competition" is to save the list of words in addressable form. At least my algorithm should scale boringly linearly up to where the system is forced to really clear buffers and cache and then swap. Theirs appear to as well. So I'd suggest using Shakespeare from Gutenberg and maybe /usr/share/dict/linux_words as a baseline, although the latter might vary from version to version of linux. I'd also suggest that each contestant run their code at least once or twice with commented out lines that demonstrate that their code actually is parsing the text and saving the words (and that what it saves ARE words, or at least word-like tokens) in an addressable retrievable form. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Thu Jan 29 18:59:37 2004 From: raysonlogin at yahoo.com (Rayson Ho) Date: Thu, 29 Jan 2004 15:59:37 -0800 (PST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <20040129202509.GB1630@greglaptop.internal.keyresearch.com> Message-ID: <20040129235937.80631.qmail@web11404.mail.yahoo.com> IBM did not release its compilers for the G4, so I believe he used gcc to compiler his code. May be it's due to the little vs big-endian issue?? Rayson --- Greg Lindahl wrote: > The G5 in particular has the usual IBM feature that some float > computations are done as double, with the answer only being truncated > when variables are stored to memory. The IBM compiler has a switch > that rounds things more normally. If this feature allows your code to > run, then your code is probably the most guilty party. > > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From selva.nair at utoronto.ca Thu Jan 29 17:59:13 2004 From: selva.nair at utoronto.ca (Selva Nair) Date: Thu, 29 Jan 2004 17:59:13 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, rgb wrote: > On Thu, 29 Jan 2004, dc wrote: > > > > I still guarantee you two things: > > > 1) Your code will be longer > > > 2) Your program will be slower > > > > > > As always, I love to be proven wrong ;) > > > > > > Here is another try at that, this time in Java. > > > > file size C++ j client j server > ... > > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > > And here is a version in C. It is longer, no question. It does its own > memory management, in detail, in pages (which should be nearly optimally > efficient). It is moderately robust, and smart enough to recognize all > sorts of separators (when counting words, separators matter -- hence > this program will find more words than e.g. wc because it splits things > differently). But this one does not count unique words, does it? Here is my version in awk. It beats C++ by 1 line in length and 1.5 times in speed (1.86s versus 2.83s elapsed time) with shaks12.txt as input. [selva at scala distinct_words]$ wc shaks12.txt 124456 901325 5458199 shaks12.txt This copy of shaks12.txt has been filtered by dos2unix. Timings: First my awk script (with GNU awk 3.1.0) [selva at scala distinct_words]$ /usr/bin/time ./dwc.awk shaks12.txt Number of distinct words = 67505 1.82user 0.04system 0:01.86elapsed 99%CPU Now the original C++ code (compiled by g++ 2.96). [selva at scala distinct_words]$ /usr/bin/time ./dwc < shaks12.txt Words: 67505 2.79user 0.04system 0:02.83elapsed 100%CPU Here is the script: #!/bin/awk -f { for(i = 1; i <= NF; i++) { if (words[$i]) continue; words[$i] = 1 ; ++nwords; } } END { printf "Number of distinct words = %i\n", nwords; } Selva _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eccf at super.unam.mx Thu Jan 29 20:16:13 2004 From: eccf at super.unam.mx (Eduardo Cesar Cabrera Flores) Date: Thu, 29 Jan 2004 19:16:13 -0600 (CST) Subject: [Beowulf] PVM BEOLIN & SMP Message-ID: Are there anybody that could run sucessfully PVM BEOLIN conf into a cluster using both of the processors of the dual machines? Cause i got the next, output (i'm using OpenPBS to submit my job): [eccf at mixbaal BEOLIN]$ more ~/pvml.10006 [pvmd pid1213] 01/29 19:43:50 PROC_LIST=nodo13:nodo13:nodo12:nodo12:nodo11:nodo1 1:nodo10:nodo10: [pvmd pid1213] 01/29 19:43:50 8 nodes in list. [t80040000] 01/29 19:43:50 nodo13 (192.168.1.13:32770) BEOLIN 3.4.4 [t80040000] 01/29 19:43:50 ready Thu Jan 29 19:43:50 2004 [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60001 [t80040000] 01/29 19:43:50 mpp_find: Task not found [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60001 [t80040000] 01/29 19:43:50 mpp_find: Task not found [t80040000] 01/29 19:43:50 mpp_new() sp=806f510 sp->n_link=806f510 [t80040000] 01/29 19:43:50 mpp_new() sp=806f510 sp->n_link=80828e8 [t80040000] 01/29 19:43:50 mppload(): Forking to nodo13 [t80040000] 01/29 19:43:50 mppload(): Forking to nodo13 [t80040000] 01/29 19:43:50 mppload(): Forking to nodo12 [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60002 [t80040000] 01/29 19:43:50 [t60000] [t80040000] 01/29 19:43:50 [t60000] Estoy en el nodo nodo13 [t80040000] 01/29 19:43:50 mpp_free() sp=80828e8 sp->n_link=806f510 [t80040000] 01/29 19:43:50 mpp_free() n_ptype = 0, ptype = 0 [eccf at mixbaal BEOLIN]$ And that's it. The process aparently is running but nothing else happens!!!! Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 9947.mixbaal.su jlgr q_250m_2 chel 1247 2 -- 200mb 08:00 R 01:42 nodo04/0+nodo03/0 9952.mixbaal.su eccf q_500m_4 myjob.tcsh 1246 4 -- 500mb 16:00 R 00:00 nodo09/1+nodo09/0+nodo08/1+nodo08/0+nodo07/1+nodo07/0+nodo05/1+nodo05/0 9953.mixbaal.su eccf q_500m_4 shel 1146 4 -- 500mb 16:00 R 00:00 nodo13/1+nodo13/0+nodo12/1+nodo12/0+nodo11/1+nodo11/0+nodo10/1+nodo10/0 This is my shell scrit: #!/bin/bash #PBS -q q_500m_4n_2h #PBS -l mem=500Mb #PBS -l nodes=4:ppn=2 #PBS -o temporal1 #PBS -e tiempo_temporal cd /home/staff/eccf/pvm3/bin/BEOLIN cat $PBS_NODEFILE > salida export PROC_LIST=`gawk 'BEGIN{ORS=":"} {getline a < FILENAME; print a}' salida` echo "El valor de PROC_LIST = `echo $PROC_LIST`" /local/pvm3/lib/BEOLIN/pvm /home/staff/eccf/pvm3/bin/BEOLIN/master1 echo "halt" | pvm Any idea? If i run 8 processes in 8 different nodes it perfectly runs!!!! Thanks a lot cafe _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Jan 29 15:39:02 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Thu, 29 Jan 2004 15:39:02 -0500 Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: References: Message-ID: <1075408742.7848.73.camel@squash.scalableinformatics.com> On Thu, 2004-01-29 at 09:55, dc wrote: > file size C++ j client j server > wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > big.txt 39389424 0m29.120s 0m13.972s 0m12.776s > vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s Where did these files come from? Would be nice to try out non-C++/non-Java solutions with. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Thu Jan 29 14:56:34 2004 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Thu, 29 Jan 2004 13:56:34 -0600 Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: References: Message-ID: <20040129195634.GG23892@mikee.ath.cx> On Thu, 29 Jan 2004, dc wrote: > Jakob Oestergaard wrote: > > > Let me throw out a challenge: > > > > "Write a program in C that reads from standard input until EOF, and > > prints the number of distinct words found in that input" > > > > > > > I use an RB-tree to store the words I find, and print the number of > > elements in the final tree. > > > > A comparable C solution shouldn't be more than a few thousand lines ;) > > > > Oh, right, I cheated and used STL. Well, STL is a big part of why C++ > > is really handy, but go right ahead and use "glib" or whatever. > > > > I still guarantee you two things: > > 1) Your code will be longer > > 2) Your program will be slower > > > > As always, I love to be proven wrong ;) > > > Here is another try at that, this time in Java. > > C++ code compiled with g++ 3.3 (-O3), times are 'real time' reported by the > bash time function, the client and server columns correspond to java runs using > the -client and -server options respectively. > Java compiler and VM are the hotspot 1.4. > > > file size C++ j client j server > wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > big.txt 39389424 0m29.120s 0m13.972s 0m12.776s > vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s > > Oh right, I cheated and used Java standard collections ;-) > > Seriously tough, Java is not as slow as some seem to believe. > > And optimizing the code at run-time (hotspot, code morphing,...) has the > advantage that the optimizer has access to potentially useful information, both > about the code itself and about the executing environment. > > -------------------------------------------------------------- > import java.util.*; > import java.util.regex.Pattern; > import java.io.*; > > > public class wordcount { > > public static void main (String args[]) { > HashSet hs = new HashSet(); > try { > BufferedReader input = new BufferedReader(new InputStreamReader(System.in)); > > String s, words[]; > Pattern p = Pattern.compile("\\s"); > > while ((s = input.readLine()) != null) { > words = p.split(s); > for (int i = 0; i < words.length; i++) > hs.add(words[i]); > } > hs.remove(""); > } > catch (Throwable e) { > e.printStackTrace(); > } > System.out.println("Words: " + hs.size()); > } > } Do you have some time to run the same test using the gcc front-end to java and posting those results? Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From scyld at jasons.us Fri Jan 30 00:44:26 2004 From: scyld at jasons.us (scyld at jasons.us) Date: Fri, 30 Jan 2004 00:44:26 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075409258.4438.19.camel@pel> References: <1075320826.5224.7.camel@vitalstatistix> <1075409258.4438.19.camel@pel> Message-ID: <20040130004018.J28799@torgo.bigbroncos.org> On Thu, 29 Jan 2004, Jag wrote: > Interesting article. It seems they don't know what their cluster is > called. The article wants to call it 'Virginia Tech X', whereas the > website for the cluster wants to call it Terascale: > http://computing.vt.edu/research_computing/terascale/ When I was there last month they referred to it as "System X" and as part of the Terascale project. YMMV. -Jason ----- Jason K. Schechner - check out www.cauce.org and help ban spam-mail. "All HELL would break loose if time got hacked." - Bill Kearney 02-04-03 ---There is no TRUTH. There is no REALITY. There is no CONSISTENCY.--- ---There are no ABSOLUTE STATEMENTS I'm very probably wrong.--- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Fri Jan 30 08:16:19 2004 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Fri, 30 Jan 2004 07:16:19 -0600 Subject: [Beowulf] [OT] HA clusters vs. grids? Message-ID: <20040130131619.GA32707@mikee.ath.cx> Ok, so this is on the edge of the mailing list.... I have been told I must suggest a plan for the purchase of a new server, that I can't low-ball the situation by only adding core to a few servers. One pair of servers is in a AIX HA cluster, one node is the primary and the other is simply a hot-spare (always in standby). I understand what the HA is doing with its heartbeats, fail over, and such. If I have to buy a new server, is it possible I could improve uptime by having a grid? This is a homegrown application, the primary application in the company, and it uses SNA, ISAM files, and lots of serial connections. Can this type of application (that needs these external devices) work in a grid environment? Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dc at hemeris.com Fri Jan 30 04:31:45 2004 From: dc at hemeris.com (Didier Carlier) Date: Fri, 30 Jan 2004 10:31:45 +0100 Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: References: Message-ID: <1F08D319-5307-11D8-84E2-003065E5B6BA@hemeris.com> On Jan 29, 2004, at 23:56, Robert G. Brown wrote: > On Thu, 29 Jan 2004, Joe Landman wrote: > >> On Thu, 2004-01-29 at 09:55, dc wrote: >>> file size C++ j client j server >>> wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s >>> shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s >>> big.txt 39389424 0m29.120s 0m13.972s 0m12.776s >>> vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s >> >> Where did these files come from? Would be nice to try out >> non-C++/non-Java solutions with. > > At least shaks12.txt is from project gutenberg and shows up as a direct > link to there, first thing, from Google on a "shaks12 txt" search. I > haven't tried the rest of them -- if a tool works for Shakespeare it > SHOULD work for any of the rest. The only dicey point will arise when > the filesize starts to compete with available free memory [...] Exactly. The rnpc10.txt is "War and Peace" (also from Gutenberg), big.txt is simply 12 copies of wrnp10.txt concatenated, and vbig,txt is 60 copies of wrnp10.txt concatenated. I included the 2 large files because Java makes little sense for run-times of a couple of seconds where more time is spent compiling the code than executing it. Also I didn't think that run-times of 1 or 2 seconds were of particular interest on this list. Looking at the (elapsed time)/(file size) ratio for the 2 big files, both the C++ and the Java elapsed times seem to scale linearly with the file size, but (* in this particular test *) java beats c++ by a factor of 2.3 for large files and only 1.6 for the smaller one. Didier _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Fri Jan 30 13:37:53 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Fri, 30 Jan 2004 13:37:53 -0500 Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: References: Message-ID: <1075487873.8580.36.camel@squash.scalableinformatics.com> On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > Note that dwc finds 67505 "distinct words". This is very interesting, This is what I find in the Perl version as well. I wish I could get that Java code compiling on this platform so I could compare the two. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From selva.nair at utoronto.ca Fri Jan 30 16:39:19 2004 From: selva.nair at utoronto.ca (Selva Nair) Date: Fri, 30 Jan 2004 16:39:19 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: <1075487873.8580.36.camel@squash.scalableinformatics.com> Message-ID: On Fri, 30 Jan 2004, Joe Landman wrote: > On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > > > Note that dwc finds 67505 "distinct words". This is very interesting, > > This is what I find in the Perl version as well. I wish I could get > that Java code compiling on this platform so I could compare the two. The awk script I posted earlier counts tokens separated by [:blank:] and I guess that is what the C++ version also does -- tokens not words as rgb carfully pointed out. I was trying to mimic the C++ code, so case and punctuations were not ignored. Anyway, there is no easy answer to what a word is -- e.g, simplistic removal of punctuations makes 3 "words" out of 100,000,000. But all that is beside the point and the point itself has nothing to do with beowulf. It is still tempting to to change the awk script to match RGB's C version ;-) My first attempt with 3 extra lines of awk gives: [selva at scala distinct_words]$ /usr/bin/time ./dwc.awk < shaks12.txt Total number of words = 902299 Number of distinct words = 31384 2.04user 0.03system 0:02.06elapsed 100%CPU Only 10% slower than the earlier version that was not punctuation/capitalisation-agnostic. The total number of words is still off by 4 and unique words are about 6000 smaller than the C version (see footnote). It may not be hard to hunt down the difference, but that does not appear worthwhile. I think the point has been made -- use your favourite scripting language for quick answers or even for testing algorithms. "For everything else there is C" -- be it for extensibility or performance or all the other reasons eloguantly expressed by rgb. I can not see any reason to learn or use C++ simply for sets or hashes or trees in STL. By the way, the performance of C++ STL appears to vary by up to a factor of 10 between different versions of g++. The number I quoted (2.8 s) was the fastest I could get and that was with g++ 2.96. For some reason version 3.0.x runs much slower here.. Selva Footnotes: 1. Googling shows that Shakespeare used 31,534 different words and a total of 884,647 words in his complete works (http://www-math.cudenver.edu/~wbriggs/qr/shakespeare.html) And some statistical projections indicate he probably knew that many more words but did not use. In any case, the awk result is surprisingly close to 31,534 inspite of the much smaller sample we used -- there is some over counting due to numeric tokens and other "junk" in the Gutenberg text, but I am still surprised by the close agreement. Also I wonder why the C version comes up with as high as 37,000+ unique words. 2. FWIW, here is the new script: #!/bin/awk -f # Count number of distinct or unique words in a file # Usage: dwc.awk inputfile { $0 = tolower($0); # be case insensitive gsub(/[\r\n\t,.!?;"*<>@\[\](){}+=/\\\#:]/, " "); # remove punctuations etc.. # the above list is chosen to match the C version -- ', ` etc missing $0 = $0; # hack to re-split $0 for(i = 1; i <= NF; ++i) { # printf "line:%i word:%i %s\n", NR, i, $i; # DEBUG ++nwords; # increment total number of words if (words[$i]) continue; # word is not new - skip words[$i] = 1; ++uwords; # increment number of unique words } } END { printf("Total number of words = %i\n", nwords); printf("Number of distinct or unique words = %i\n", uwords); } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From selva.nair at utoronto.ca Fri Jan 30 16:55:11 2004 From: selva.nair at utoronto.ca (Selva Nair) Date: Fri, 30 Jan 2004 16:55:11 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) -- correction In-Reply-To: Message-ID: Replying to my own post.. On Fri, 30 Jan 2004, Selva Nair wrote: > knew that many more words but did not use. In any case, the awk > result is surprisingly close to 31,534 inspite of the > much smaller sample we used -- there is some over counting Ignore that comment about "smaller smaple". I did not realize that shaks12.txt indeed contains the complete works including even the sonnets. Sorry for the confusion. Selva _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 30 16:31:18 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 30 Jan 2004 16:31:18 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: <1075487873.8580.36.camel@squash.scalableinformatics.com> Message-ID: On Fri, 30 Jan 2004, Joe Landman wrote: > On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > > > Note that dwc finds 67505 "distinct words". This is very interesting, > > This is what I find in the Perl version as well. I wish I could get > that Java code compiling on this platform so I could compare the two. It might also be worthwhile to look at the list of words it produces to see what it is counting as a word. As I said, even my version where I'm controlling the split into words contains lots of non-words. Of course, I could have a bug in my code, as well:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Fri Jan 30 19:27:59 2004 From: rossini at blindglobe.net (A.J. Rossini) Date: Fri, 30 Jan 2004 16:27:59 -0800 Subject: [Beowulf] HVAC and room cooling... Message-ID: <85k739x8q8.fsf@servant.blindglobe.net> I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual athlon or opteron machines). Can anyone suggest where I might look to find information for this? Also, does anyone have any brilliant thoughts for cooling an internal room that can't affordably get chilled water? (I've been suggesting to people that it isn't possible, but someone brought up "portable liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying to get stable systems, not instability :-). best, -tony -- rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 30 18:46:44 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 30 Jan 2004 18:46:44 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Fri, 30 Jan 2004, Selva Nair wrote: > 1. Googling shows that Shakespeare used 31,534 different > words and a total of 884,647 words in his complete works > (http://www-math.cudenver.edu/~wbriggs/qr/shakespeare.html) > And some statistical projections indicate he probably > knew that many more words but did not use. In any case, the awk > result is surprisingly close to 31,534 inspite of the > much smaller sample we used -- there is some over counting > due to numeric tokens and other "junk" in the Gutenberg text, > but I am still surprised by the close agreement. Also I wonder > why the C version comes up with as high as 37,000+ unique words. I didn't use toupper or tolower, simply said that to get closer to a correct count of unique "words" (as opposed to tokens) one should. If I do (trivial addition, completed) then I get 31384 words, which is again curious because a) I don't strip off the Gutenberg headers -- this is on raw shaks12.txt, so it contains words like cdroms and etexts and of course this is one reason the raw word count is high as well; and b) yeah, it still miscounts hyphenations and apostrophes and various other constructions I see scanning the actual list of unique words. It appears that the net effect is negative. OR, as always, I could have a bug. Or the google produced count could be wrong -- depends on how it was made and the sources, after all. If you (or anyone) care I can put the wordlist it generates on my website or something so you can grab it. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Fri Jan 30 20:00:52 2004 From: bclem at rice.edu (Brent M. Clements) Date: Fri, 30 Jan 2004 19:00:52 -0600 (CST) Subject: [Beowulf] Authentication within beowulf clusters. Message-ID: So, we were just going through out standards and procedures on how we do various things on our beowulf clusters. One of the procedures that we are looking at is the way to do authentication. My question is this, how does everyone handle authentication? Do you run standalone ldap or nis services on the master/management node of their beowulf clusters? We'd like to get an idea of how everyone else handles authentication on their clusters. If your company/university uses a centralized authentication service...please explain how you get your private network nodes to work with that configuration. THanks, Brent Clements Linux Technology Specialist Information Technology Rice University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bushnell at chem.ucsb.edu Fri Jan 30 20:16:59 2004 From: bushnell at chem.ucsb.edu (John Bushnell) Date: Fri, 30 Jan 2004 17:16:59 -0800 (PST) Subject: [Beowulf] HVAC and room cooling... Message-ID: >I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual >athlon or opteron machines). > >Can anyone suggest where I might look to find information for this? You can probably find some approx. formula on a Beowulf site. I know I've seen one on this list, which I found useful. (list archive) (So many watts) times 'x' equals how many "tons" of AC. Multiply by at least two of course ;-) >Also, does anyone have any brilliant thoughts for cooling an internal >room that can't affordably get chilled water? (I've been suggesting >to people that it isn't possible, but someone brought up "portable >liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >to get stable systems, not instability :-). You can have an external heat exchanger. If you are lucky and are, say, on the first floor somewhere close to an external wall, it is pretty simple to run a small pipe between the internal AC and the heat exchanger outside. Don't know how far it is practical to run one though. We have one in our computer room, but it is only six feet or so from the exchanger outside. Our newer AC runs on chilled water which was quoted for a lot less than another inside/outside combo, but we already had a leftover chilled water supply in the computer room. Hope that helps - John > >best, >-tony > >-- >rossini at u.washington.edu http://www.analytics.washington.edu/ >Biomedical and Health Informatics University of Washington >Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research >Center >UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable >FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Fri Jan 30 20:31:16 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Fri, 30 Jan 2004 20:31:16 -0500 Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: References: Message-ID: <1075512676.4915.207.camel@protein.scalableinformatics.com> On Fri, 2004-01-30 at 16:39, Selva Nair wrote: > On Fri, 30 Jan 2004, Joe Landman wrote: > > > On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > > > > > Note that dwc finds 67505 "distinct words". This is very interesting, > > > > This is what I find in the Perl version as well. I wish I could get > > that Java code compiling on this platform so I could compare the two. > > The awk script I posted earlier counts tokens separated by [:blank:] and I > guess that is what the C++ version also does -- tokens not words as rgb > carfully pointed out. I was trying to mimic the C++ code, so case and Ok. I changed mine slightly. Very naive case here Perl 5.8.3 wrnpc11.txt 1.21 +/- 0.01 s shaks12.txt 2.09 +/- 0.04 s big.txt 14.31 +/- 0.09 s vbig.txt 1m11.49 +/- 0.52 s I could easily optimize it more (do the work on a larger buffer at a once), but I think enough waste heat has been created here. This is a simple 2500+ Athlon XP box (nothing fancy) running 2.4.24-pre3. ---- #!/opt/perl583/bin/perl my $infile; my $filename = shift; my (%word,$buffer); die "Usage: wcount2.pl filename\n" if (!defined($filename)); open($infile,"<".$filename); printf "opened file ...\n"; while ( $buffer = lc(<$infile>) ) { @word{($buffer=~ /(\w+)/g)}=1; } printf "Words: %s\n",1 + keys %word; ---- [...] > eloguantly expressed by rgb. I can not see any reason to learn or use C++ > simply for sets or hashes or trees in STL. The view I have is to use the most appropriate tool for the task. Some tools are in search of tasks for which they are appropriate. Marketing has in some cases supplanted common sense. > By the way, the performance of C++ STL appears to vary by up to a factor > of 10 between different versions of g++. The number I quoted (2.8 s) was > the fastest I could get and that was with g++ 2.96. For some reason > version 3.0.x runs much slower here.. Alex Stepanov had a benchmark on the cost of the STL and the advanced C++ features. Some compilers did rather poorly as I remember. You might get better results from Intel's compiler or the PathScale compilers (for the Opteron). > > Selva > > Footnotes: > > 1. Googling shows that Shakespeare used 31,534 different > words and a total of 884,647 words in his complete works Hmmm... time ./wcount2.pl shaks12.txt opened file ... Words: 23903 2.000u 0.020s 0:02.04 99.0% 0+0k 0+0io 360pf+0w I am using /(\w+)/g which will grab words into capture buffers. A word is a collection of alphanumeric plus others. I could build other regexes. Using /(\S+)/g gives 59509. Using /\b(.*)\b/g I get 110557. > (http://www-math.cudenver.edu/~wbriggs/qr/shakespeare.html) > And some statistical projections indicate he probably > knew that many more words but did not use. In any case, the awk > result is surprisingly close to 31,534 inspite of the > much smaller sample we used -- there is some over counting > due to numeric tokens and other "junk" in the Gutenberg text, > but I am still surprised by the close agreement. Also I wonder > why the C version comes up with as high as 37,000+ unique words. It goes to what the definition of "word" is. I am lower-casing everything. Without lower-casing I get 29673. Not sure what the issue is, but not willing to spend more time on it. Of course this has little to do with Beowulf-ery. Language choice is largely dictated by the problem, the developers etc. Force fitting solutions is often a recipe for disaster (systems designed to fail often do). Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Fri Jan 30 22:32:20 2004 From: rossini at blindglobe.net (A.J. Rossini) Date: Fri, 30 Jan 2004 19:32:20 -0800 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: (John Bushnell's message of "Fri, 30 Jan 2004 17:16:59 -0800 (PST)") References: Message-ID: <85y8rosshn.fsf@servant.blindglobe.net> John Bushnell writes: > (So many watts) times 'x' equals how many "tons" of AC. Multiply > by at least two of course ;-) Or 3, sigh... >>Also, does anyone have any brilliant thoughts for cooling an internal >>room that can't affordably get chilled water? (I've been suggesting >>to people that it isn't possible, but someone brought up "portable >>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>to get stable systems, not instability :-). > > You can have an external heat exchanger. If you are lucky and are, > say, on the first floor somewhere close to an external wall, it is > pretty simple to run a small pipe between the internal AC and the > heat exchanger outside. Don't know how far it is practical to run > one though. We have one in our computer room, but it is only six > feet or so from the exchanger outside. Our newer AC runs on chilled > water which was quoted for a lot less than another inside/outside > combo, but we already had a leftover chilled water supply in the > computer room. I've looked at the chilled-water approach. They estimated between $40k-$80k. oops (this room is REALLY in the middle of the building. Great for other computing purposes, but not for cooling). I'm looking for the proverbial vent-free A/C. Sort of like frictionless tables and similar devices I recall from undergraduate physics... Thanks for the comments! best, -tony -- rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Sat Jan 31 00:18:15 2004 From: bclem at rice.edu (Brent M. Clements) Date: Fri, 30 Jan 2004 23:18:15 -0600 (CST) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <85y8rosshn.fsf@servant.blindglobe.net> References: <85y8rosshn.fsf@servant.blindglobe.net> Message-ID: I have found that the best thing to do is outsource the colocation of your equipment. The cost of installing and maintaining the proper type of cooling and ventilation for mid-large size clusters costs more than to colocate. We are currently exploring placing our larger clusters in colocation facilities right now. The only downside that we have is that we can't find colocation facilities that will give us 24/7 physical access to our equipment. As you all know...researchers push beowulf hardware to the limits and the meantime to failure is higher. -B Brent Clements Linux Technology Specialist Information Technology Rice University On Fri, 30 Jan 2004, A.J. Rossini wrote: > John Bushnell writes: > > > (So many watts) times 'x' equals how many "tons" of AC. Multiply > > by at least two of course ;-) > > Or 3, sigh... > > >>Also, does anyone have any brilliant thoughts for cooling an internal > >>room that can't affordably get chilled water? (I've been suggesting > >>to people that it isn't possible, but someone brought up "portable > >>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > >>to get stable systems, not instability :-). > > > > You can have an external heat exchanger. If you are lucky and are, > > say, on the first floor somewhere close to an external wall, it is > > pretty simple to run a small pipe between the internal AC and the > > heat exchanger outside. Don't know how far it is practical to run > > one though. We have one in our computer room, but it is only six > > feet or so from the exchanger outside. Our newer AC runs on chilled > > water which was quoted for a lot less than another inside/outside > > combo, but we already had a leftover chilled water supply in the > > computer room. > > I've looked at the chilled-water approach. They estimated between > $40k-$80k. oops (this room is REALLY in the middle of the building. > Great for other computing purposes, but not for cooling). > > I'm looking for the proverbial vent-free A/C. Sort of like > frictionless tables and similar devices I recall from undergraduate > physics... > > Thanks for the comments! > > best, > -tony > > -- > rossini at u.washington.edu http://www.analytics.washington.edu/ > Biomedical and Health Informatics University of Washington > Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center > UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable > FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email > > CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be > confidential and privileged. If you received this message in error, > please destroy it and notify the sender. Thank you. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 31 06:22:44 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 31 Jan 2004 12:22:44 +0100 (CET) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <85k739x8q8.fsf@servant.blindglobe.net> Message-ID: On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > to people that it isn't possible, but someone brought up "portable > liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > to get stable systems, not instability :-). > Daft idea. What happens when the liquid nitrogen vapourises? It turns to nitrogen gas. And if you are unlucky anyone in the room chokes... I was reminded of this in the lift of a chemistry department where we have an opteron cluster - a large sign reminding that LNO2 and dangerous chemicals are not allowed in passenger lifts. BTW, when arranging air flows try to get cold air ducted into the front of the rack - node tend to draw air in the front and exhaust to the rear. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Sat Jan 31 06:57:49 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Sat, 31 Jan 2004 12:57:49 +0100 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: References: <85y8rosshn.fsf@servant.blindglobe.net> Message-ID: <401B983D.8070801@telia.com> Hi, As a Marine Engineer is the cooling aspect the absolute smallest problem with a Linux cluster. The cooling capacity required is calculated with by means of the "Funnel principle" i.e. every thing that you are putting in will come out. This give that you have to calculate the total heat in-put in your cluster-room as that is what you want transport out of the room by means of an air-refrigerant equipment. The simplest way to found out the cooling effect required will be to: A) Ask a professional air refrigerant company to recommend the capacity for the actual1 location when it is used as an office. (They should have experience and know-how to calculate the heat in-put from the sun-shine and climate in your area) B) Calculate your cluster's maximum power consumption 3 Calculate the total cooling capacity required by the formula: ? > (A+B) x 1.1 Best regards Per Lindstro"m Brent M. Clements wrote: >I have found that the best thing to do is outsource the colocation of your >equipment. The cost of installing and maintaining the proper type of >cooling and ventilation for mid-large size clusters costs more than to >colocate. > >We are currently exploring placing our larger clusters in colocation >facilities right now. > >The only downside that we have is that we can't find colocation facilities >that will give us 24/7 physical access to our equipment. As you all >know...researchers push beowulf hardware to the limits and the meantime to >failure is higher. > >-B > >Brent Clements >Linux Technology Specialist >Information Technology >Rice University > > >On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > >>John Bushnell writes: >> >> >> >>>(So many watts) times 'x' equals how many "tons" of AC. Multiply >>>by at least two of course ;-) >>> >>> >>Or 3, sigh... >> >> >> >>>>Also, does anyone have any brilliant thoughts for cooling an internal >>>>room that can't affordably get chilled water? (I've been suggesting >>>>to people that it isn't possible, but someone brought up "portable >>>>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>>>to get stable systems, not instability :-). >>>> >>>> >>> You can have an external heat exchanger. If you are lucky and are, >>>say, on the first floor somewhere close to an external wall, it is >>>pretty simple to run a small pipe between the internal AC and the >>>heat exchanger outside. Don't know how far it is practical to run >>>one though. We have one in our computer room, but it is only six >>>feet or so from the exchanger outside. Our newer AC runs on chilled >>>water which was quoted for a lot less than another inside/outside >>>combo, but we already had a leftover chilled water supply in the >>>computer room. >>> >>> >>I've looked at the chilled-water approach. They estimated between >>$40k-$80k. oops (this room is REALLY in the middle of the building. >>Great for other computing purposes, but not for cooling). >> >>I'm looking for the proverbial vent-free A/C. Sort of like >>frictionless tables and similar devices I recall from undergraduate >>physics... >> >>Thanks for the comments! >> >>best, >>-tony >> >>-- >>rossini at u.washington.edu http://www.analytics.washington.edu/ >>Biomedical and Health Informatics University of Washington >>Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center >>UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable >>FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email >> >>CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be >>confidential and privileged. If you received this message in error, >>please destroy it and notify the sender. Thank you. >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 30 11:37:52 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 30 Jan 2004 11:37:52 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, Selva Nair wrote: > But this one does not count unique words, does it? > > Here is my version in awk. It beats C++ by 1 line in length and > 1.5 times in speed (1.86s versus 2.83s elapsed time) with shaks12.txt as > input. > A modified (hashing/unique word) version of the C yields: rgb at lilith|T:245>/usr/bin/time wordcount shaks12.txt Opening shaks12.txt for counting Total Number of Words = 902303; total number of UNIQUE words = 37499 Command exited with non-zero status 69 1.15user 0.12system 0:01.29elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (100major+4093minor)pagefaults 0swaps This is using gnu's hcreate, hsearch hash functions (ordinary POSIX library functions, as it were) and simply adds a secondary loop through all the words to do the count. This is still not terribly close to efficient -- the reason it has to be done in a secondary loop is because realloc moves the base pointer to the primary data buffer where all the words are stored, which makes hcreate/hsearch unhappy in midstream. The best way around this are to split the code into a segment to read the whole file into memory efficiently (probably using read()/realloc in 32K chunks) and a single loop to strsep() parse into words and to hsearch() them into the hash. I don't have a good feel for how much time this would save, but at a guess one could get the overall time down to maybe a second from 1.29 seconds wallclock, mostly on the I/O side. Oh, and one wouldn't need to maintain a secondary indexing vector at all since the hash would do it all, although I'm sure it isn't horribly expensive. Of course this STILL doesn't REALLY count unique "words", only unique "tokens". To do words we'd have to toupper() all the words before saving/hashing them, as The and the aren't really unique. In Shakespeare, things are even worse as he uses his own characteristic blend of "interesting" contractions, what say'st though? Idiomatic, as it were. So lots of tokens included a ' and are contractions, others are caps, still others are real mistakes -- hyphenated words split across a line but recorded as "two unique words". So the real count is more likely a few thousand smaller. Still, Shakespeare seems pretty literate -- ~38K unique tokens when linux_words only has order of 45K total. I don't know that I'll bother to redo this one more time to speed it any further as described above -- it IS a toy project. The main point is that adding hashing required about 13 lines of code, and in a rewrite that eliminated the indexing vector I'd get those back and change. So using POSIX/Gnu functions (sloppily) it still isn't a horribly long program, and is still fairly efficient (running in less than half the time of C++). One final point: > [selva at scala distinct_words]$ /usr/bin/time ./dwc < shaks12.txt > Words: 67505 > 2.79user 0.04system 0:02.83elapsed 100%CPU > > Here is the script: > > #!/bin/awk -f > { > for(i = 1; i <= NF; i++) { > if (words[$i]) continue; > words[$i] = 1 ; > ++nwords; > } > } > END { > printf "Number of distinct words = %i\n", nwords; > } Note that dwc finds 67505 "distinct words". This is very interesting, as wordcount only finds 38000-odd distinct words, where "distinct" has to do with just how words are parsed in the first place and where I can watch, via inserted printf commands, to be dead certain that the process of word selection and uniquifying is proceeding correctly. In fact in C this is ALL strictly under my control -- I can alter the list of word delimiters as I wish, and I can even (with a bit of effort but not a huge penalty in time or lines of code) "fix" the hyphenation problem: a) stop parsing on LF/CR (currently they are delimiters b) IF a word ends on LF and/or CR AND the preceeding character is a hyphen c) THEN shift the following word left N spaces (to be computed) in the buffer to reconnect the word halves. d) AND save the word as the reconnected halves (via offsets or hash or both. Fix >>this<< in C++ using the code already presented -- hmmm, not so easy. Fix it in awk (or even figure out what is wrong with the script above that it is returning almost twice as many words as I count, where my script and e.g. wc return precisely the same count on /usr/share/dict/linux_words and where I can as noted watch it do what it does and hence have a fair bit of confidence in it). Fix it so that (in both cases) one still has the entire text in memory, indexed by and retrievable by words one at a time AND has access to the hash of unique words only without significantly increasing the time or storage requirement. I bring these up simply to reiterate that real world code is rarely simple. This is hardly a "complex project" at first glance, but if one examines what one actually GETS as unique words when one just reads in a block of arbitrary (but real) text, parses it badly (or at least in a way that is out of your control) one finds that it is pretty unlikely to return what you really want if you use a one-size-fits-all library function to do all the work. To reassert control and fix special cases without microscopic access to the raw data stream requires more and more contortion, and can easily introduce multiple passes across the data to do first one thing, then another, then another, when they could all be done in a single loop if one did it "right". It is precisely this that I was referring to when I said that using a library, ESPECIALLY a powerful high level library, shapes your code, sometimes in highly inefficient ways. This isn't surprising -- a variety of philosophers e.g. Dorothy Lee and Bronislaw Malinoski have long argued that the very way we think is governed by the language we think in, and this is obviously even more true when it comes to computer "languages", where commands encapsulate entire algorithms with very specific and complex results. Similar tradeoffs are visible in pictographic languages (where every "thought" has its own unique token that must be learned in order to read it or write it) and phonetic languages (where every thought has a nearly unique VERBAL token, but where a relatively small set of building blocks can represent all the phonemic encodings so that one can learn to read and write years earlier, and where the language is far more flexible and capable of growth and evolution at a far greater rate). So my final conclusion (this thread has probably gone on long enough and at least SOME people are likely bored with it by now:-) is my original one -- raw C (or C++ written as procedural C with minimal and sensible use of the extensions, a la Mark Hahn:-) will nearly always result in the fastest execution times, and gives one COMPLETE control of your data and execution loops. Using objects and object-oriented methodology is still entirely possible and I'd even say best practice in BOTH language variants. C++ (including the STL library) has a wider set of "standard" tools, although those tools come with a very distinct ease of programming vs execution time tradeoff, and with another more subtle tradeoff in the way that they shape your code (which may be in ways that are just fine, or MAY be in ways that are moderately detrimental when the library calls aren't terribly good fits to what you want to do but your reliance on them has slowed the development of skills to do what you want without them). Again, this isn't an intrinsic criticism of C++ -- it is just an empirical observation and I'd make an entirely similar observation about many other high level tools including markup languages vs word processors and more. If you use a nail gun all the time doing rough carpentry, don't be surprised if you hit your thumb and leave divots in the work when it comes time to hammer finish nails into the trim or cut nails into a cinder block wall with an actual hammer. (Don't you just love metaphors?:-) Other languages have similar tradeoffs -- few will beat C in executable speed, although it is possible (even humble fortran might beat C in certain tasks, partly because the fixed data language structure permits some things to be done by the compiler very efficiently where C uses library calls on more arbitrary pointer based data structures less efficiently). Many have things C does not that make PROGRAMMING a lot easier and may even enable certain things (such as writing a self-modifying program) POSSIBLE that are horribly difficult in raw C. SO I like C, but I also like PHP, Perl, might one day like C++ and python. I even like awk, although aside from very short awk programs to do very specialized things perl is infinitely easier. I therefore remain highly ecumenical -- I might tease people for using other languages and do think that one should match language to task with some care, but I can hardly justify being overly zealous in advancing any single language as being THE best as it is always a matter of which tradeoffs one prefers or are led to by one's problem. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu #include #include #include #include #define _GNU_SOURCE #include #define PAGE 4096 #define K 1024 static char delim[] = { ' ', /* blank */ (char) 9, /* tab */ ',', /* comma */ (char) 10, /* LF */ (char) 13, /* CR */ '.', /* . */ '!', /* ! */ '?', /* ? */ ';', /* ; */ '"', /* " */ '*', /* * */ '<', /* < */ '>', /* > */ '@', /* @ */ '[', ']', '(', ')', '{', '}', '+', '=', '/', '\\', '#', ':', (char) 0 /* terminator */ }; static char dummy[] = "aapvoinwe"; main(int argc, char **argv) { char filename[128],*word,*words,*words_head,*words_tail,*words_last; int *word_offset; size_t offset,word_length,words_length,wordlist_length, word_count,unique_count,i,j,nel; FILE *infile; ENTRY item,*search_result; if(argc != 2){ fprintf(stderr,"Usage: wordcount filename\n"); exit(1); } argv++; sprintf(filename,"%s",*argv); printf("Opening %s for counting\n",filename); infile = fopen(filename,"r"); /* * The word list as a single buffer. We allocate two pages to start with, and * reallocate whenever the free space in the buffer drops below one page. We * also maintain a pointer to the beginning of the unused part of this buffer. */ words_length = 2*PAGE; words = (char *) malloc(words_length); words_head = words; words_tail = words + words_length; /* * We do the exact same thing here, sort of. We allocate a PAGE of offsets * (relative to the words head!) and reallocate whenever the number remaining * drops below K. i will both index the wordlist and count the words as we go. */ wordlist_length = PAGE; word_offset = (int *) malloc(wordlist_length*sizeof(int)); word_count = 0; /* * Each pass through the while loop reads a single line from * infile into words at offset wptr (current tail). We will * write the code to manage lines up to 1K in length. */ word_offset[0] = 0; /* * Set up a hash table to extract/count only UNIQUE words. 300000 * words could come close to doing the OED, at least OTC versions. * Not much speed penalty for making nel bigger, BIG penalty for * making it too small... */ nel = 300000; hcreate(nel); unique_count = 0; while(fgets(words_head,(words_tail - words_head),infile)!= NULL){ /* * Check to add a PAGE to the words vector/buffer. Since realloc * may move pointer, we have to recompute offsets. */ if((words_tail - words_head) < PAGE) { words_length += PAGE; /* save offset */ offset = words_head - words; words = (char *) realloc(words,words_length); /* displace by saved offset */ words_head = words + offset; words_tail = words + words_length; /* printf("reallocating words at %0x, new length = %d\n",words,words_length); */ } /* * Check to add a PAGE to the wordlist vector/buffer. Here we * don't have to worry about offsets. */ if((wordlist_length - word_count) < K) { wordlist_length += PAGE; word_offset = (int *) realloc(word_offset,wordlist_length*sizeof(int)); /* printf("reallocating words_offset at %0x, new length = %d\n",word_offset,wordlist_length); */ } /* * Now we rip through the line. */ while(1){ /* Get a word */ word = strsep(&words_head,delim); if(words_head == NULL) break; /* Save its address for after we are done with this line */ words_last = words_head; /* Save its offset in the offset vector */ if(word[0] != (char)0){ word_offset[word_count] = word - words; /* for debugging offsets etc */ /* printf("%d: word at %0x = %s, offset is %d\n",word_count,word,word,word_offset[word_count]); */ word_count++; } } /* * strsep finishes parsing a line with a NULL. Restore the right pointer. */ words_head = words_last; } /* * Let's see, we can try sorting and uniq'ing here. As expected, it is * gawdawful slow and scales terribly with size and even though we COULD * move the first loop up into the code above easily enough, this is a * poor way to do it. We'll leave the fragment here, disabled. for(i=0;i Message-ID: On Fri, 30 Jan 2004, A.J. Rossini wrote: > John Bushnell writes: > > > (So many watts) times 'x' equals how many "tons" of AC. Multiply > > by at least two of course ;-) > > Or 3, sigh... I should have posted this in the earlier reply, sorry -- 3500 watts/ton, within a watt or four. From the sound of it, you need at least one ton, but probably not over two. BTW, I should have said, but consider talking to an architect or HVAC engineer. They are professionals, after all...:-) > I'm looking for the proverbial vent-free A/C. Sort of like > frictionless tables and similar devices I recall from undergraduate > physics... Hahahahahaha... (ROTFL). Send me one if you find it. Then I can build my perpetual motion heat engine. rgb > > Thanks for the comments! > > best, > -tony > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 10:03:56 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 07:03:56 -0800 Subject: [Beowulf] HVAC and room cooling... References: <85k739x8q8.fsf@servant.blindglobe.net> Message-ID: <002d01c3e80b$809b94f0$36a8a8c0@LAPTOP152422> There were some posts on this last summer, and rgb's book on the Brahma site at Duke has a chapter that runs through the calculations. You just ballpark the power consumption (figure a couple hundred watts/processor to be conservative)... 24*200 = 4800W... convert that to BTU/hr and you know how big an a/c you'll need. All airconditioning just moves heat from one place to another, whether by moving water or by pumping Freon or ammonia around. Where are you going to reject that 5kW of heat? The hallway outside? (it's been done..) There's a variety of a/c equipment that only requires about a 2" diameter hole/passage between cold end and hot end. ----- Original Message ----- From: "A.J. Rossini" To: Sent: Friday, January 30, 2004 4:27 PM Subject: [Beowulf] HVAC and room cooling... > > I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual > athlon or opteron machines). > > Can anyone suggest where I might look to find information for this? > > Also, does anyone have any brilliant thoughts for cooling an internal > room that can't affordably get chilled water? (I've been suggesting > to people that it isn't possible, but someone brought up "portable > liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > to get stable systems, not instability :-). > > best, > -tony > > -- > rossini at u.washington.edu http://www.analytics.washington.edu/ > Biomedical and Health Informatics University of Washington > Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center > UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable > FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email > > CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be > confidential and privileged. If you received this message in error, > please destroy it and notify the sender. Thank you. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Sat Jan 31 10:32:58 2004 From: agrajag at dragaera.net (Jag) Date: 31 Jan 2004 10:32:58 -0500 Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: References: Message-ID: <1075563177.2560.4.camel@loiosh> On Fri, 2004-01-30 at 20:00, Brent M. Clements wrote: > So, we were just going through out standards and procedures on how we do > various things on our beowulf clusters. One of the procedures that we are > looking at is the way to do authentication. > > My question is this, how does everyone handle authentication? > > Do you run standalone ldap or nis services on the master/management node > of their beowulf clusters? > > We'd like to get an idea of how everyone else handles authentication on > their clusters. If your company/university uses a centralized > authentication service...please explain how you get your private network > nodes to work with that configuration. I work at a large university that has a central kerberos server. How we do it is we use NIS locally in the cluster, but *only* for name service information. We then configure PAM to use kerberos off the main campus servers to do the actual authentication. We do this so we can use the central campus resources, but still remap home directories and such. The main key to that is making sure that the usernames on the cluster match the campus username. In order to use this on the compute nodes, we use hostbased authentication with ssh. This is along the same lines of doing hosts.equiv with rsh. We have 4 head nodes that are on the public network and the internal cluster network. A user simply logs into there, and when their mpich jobs call ssh, ssh lets them log into any of the cluster nodes without asking for a password. We can get away with this as we have a private LAN for all the nodes, and physical access to the cluster is restricted. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Fri Jan 30 20:11:22 2004 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Fri, 30 Jan 2004 20:11:22 -0500 Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: References: Message-ID: <20040131011122.GA19068@central.cis.upenn.edu> We have one form of authentication (based on University standards) to login to the head node. We use our own system for authenticating within the internal network (which is only used by the cluster). We distribute /etc/passwd for account information via cron (and via useradd and userdel wrappers). We update an authentication token on each node as it becomes assigned to a user by the scheduling system. Ssh checks this authentication token (via PAM module) before executing on a node, and Bproc uses an ownership database on the server side. Regards, Dan W. On Fri, Jan 30, 2004 at 07:00:52PM -0600, Brent M. Clements wrote: > > So, we were just going through out standards and procedures on how we do > various things on our beowulf clusters. One of the procedures that we are > looking at is the way to do authentication. > > My question is this, how does everyone handle authentication? > > Do you run standalone ldap or nis services on the master/management node > of their beowulf clusters? > > We'd like to get an idea of how everyone else handles authentication on > their clusters. If your company/university uses a centralized > authentication service...please explain how you get your private network > nodes to work with that configuration. > > THanks, > > > Brent Clements > Linux Technology Specialist > Information Technology > Rice University > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- -- Daniel Widyono http://www.cis.upenn.edu/~widyono -- Liniac Project, CIS Dept., SEAS, University of Pennsylvania -- Mail: CIS Dept, 302 Levine 3330 Walnut St Philadelphia, PA 19104 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Jan 31 10:25:58 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 31 Jan 2004 10:25:58 -0500 (EST) Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: <20040131011122.GA19068@central.cis.upenn.edu> Message-ID: On Fri, 30 Jan 2004, Daniel Widyono wrote: > We have one form of authentication (based on University standards) to login > to the head node. > > We use our own system for authenticating within the internal network (which > is only used by the cluster). We distribute /etc/passwd for account > information via cron (and via useradd and userdel wrappers). We update an > authentication token on each node as it becomes assigned to a user by the > scheduling system. Ssh checks this authentication token (via PAM module) > before executing on a node, and Bproc uses an ownership database on the > server side. ... > > My question is this, how does everyone handle authentication? > > > > Do you run standalone ldap or nis services on the master/management node > > of their beowulf clusters? NIS works fine for many purposes as well, but be warned -- in certain configurations and for certain tasks it becomes a very high overhead protocol. In particular, it adds an NIS hit to every file stat, for example, so that it can check groups and permissions. I think it was Greg who pointed out many moons ago that there are was around this (IIRC, making every node an NIS slave so that the hit is local?). NIS also at one point in time was a bleeding wound as far as security is concerned. How fondly I remember reaching out across campus and grabbing the encrypted passwd's of whole departments with a simple userspace tool to feed into crack and grab the passwords of the five silly beanies who used something like "password" for their password. Or using tftp to accomplish the same thing on systems configured out of the box to provide it. (Shame on you! I wasn't really cracking, I was a white hat checking the nets in question with permission...:-) Even though they've largely fixed it, I believe NIS remains susceptible to a vast range of attacks -- snooping (sending stuff in cleartext where it can be read by anyone on a shared line), spoofing (a host is known by IP number only) and more. So I'd strongly urge using it only inside a firewall on a trusted LAN space. I suspect LDAP has the same issues, but we've never used it so I'm not certain. Kerberos is a lot better for this sort of thing, especially if the cluster you're talking about is a "public cluster" to be accessed from a WAN or a large LAN with multiple internal LAN boundaries (e.g. department LANs) of varying degrees of trust and quality of administration. It's also fairly straightforward to use e.g. rsync to keep passwd, groups, shadow in sync across a cluster, but one does have the problem of propagating a passwd change, which NIS manages for you transparently. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Sat Jan 31 11:30:56 2004 From: agrajag at dragaera.net (Jag) Date: 31 Jan 2004 11:30:56 -0500 Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: References: Message-ID: <1075566655.2560.8.camel@loiosh> On Sat, 2004-01-31 at 10:25, Robert G. Brown wrote: > NIS works fine for many purposes as well, but be warned -- in certain > configurations and for certain tasks it becomes a very high overhead > protocol. In particular, it adds an NIS hit to every file stat, for > example, so that it can check groups and permissions. A good way around this is to run nscd (Name Services Caching Daemon). Its part of glibc and will cache a lot of that info on the local nodes. That way if you're stating a bunch of files all owned by the same user, it'll go over the network once to get the info, but all future requests will be handled by nscd, and you won't see any network traffic for it. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jj.renes at hccnet.nl Sat Jan 31 19:28:19 2004 From: jj.renes at hccnet.nl (joop renes) Date: Sat, 31 Jan 2004 16:28:19 -0800 Subject: [Beowulf] C/C++ discussion Message-ID: <200401311628.19673.jj.renes@hccnet.nl> Hi, I would like to offer the attached remarks to that discussion. best regards. ++jopa -------------- next part -------------- A non-text attachment was scrubbed... Name: discussion1.lyx Type: text/x-lyx Size: 2522 bytes Desc: not available URL: From daniel.pfenniger at obs.unige.ch Sat Jan 31 12:40:33 2004 From: daniel.pfenniger at obs.unige.ch (Daniel Pfenniger) Date: Sat, 31 Jan 2004 18:40:33 +0100 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: References: Message-ID: <401BE891.708@obs.unige.ch> John Hearns wrote: > On Fri, 30 Jan 2004, A.J. Rossini wrote: >>to people that it isn't possible, but someone brought up "portable >>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>to get stable systems, not instability :-). >> > > Daft idea. > What happens when the liquid nitrogen vapourises? It turns to nitrogen > gas. And if you are unlucky anyone in the room chokes... > I was reminded of this in the lift of a chemistry department where > we have an opteron cluster - a large sign reminding that LNO2 and > dangerous chemicals are not allowed in passenger lifts. When liquid N2 vaporizes in a closed room the natural fraction of N2 in air, already 78% by volume, just increases. N2 is thus not toxic, the only risk by increasing N2 volume is to reach a too low O2 concentration for mammals beings breathing in the room. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Sat Jan 31 12:54:25 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat, 31 Jan 2004 12:54:25 -0500 (EST) Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: Message-ID: > My question is this, how does everyone handle authentication? our clusters are accessible via ssh, with no auth within. > Do you run standalone ldap or nis services on the master/management node > of their beowulf clusters? no, rsync'ed passwd/shadow files; we'll eventually consider switching to ldap or a simple sql-backed passwd/shadow generator. NIS is out of the question. > their clusters. If your company/university uses a centralized > authentication service...please explain how you get your private network > nodes to work with that configuration. I don't see the advantage of "centralized" auth, except has a form of centralized control (which is inherently bad). consider that a random user *never* has access to every resource on campus, so whoever admins the central auth has to be constantly managing permissions for random resources that pop up. not to mention that it creates a single point of failure and a network hotspot (yes, yes, those can be engineered around.) single point of compromise, too. what's the benefit? ssh already does password-less, secure logins, so there's no advantage there. if everything were not ssh-based, I'm sure the analysis would be different. I'd sooner consider going to pure ssh-key logins, rather than involving some external authentication oracle. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Sat Jan 31 13:38:26 2004 From: bclem at rice.edu (Brent M. Clements) Date: Sat, 31 Jan 2004 12:38:26 -0600 (CST) Subject: [Beowulf] How do vendors win you over? Message-ID: Hi Everyone, In addition to our beowulf standards that we are working on, I'm streamlining our research and purchasing process of vendors for beowulf clusters within IT. 1. What are the qualities in vendors that you look for. 2. What requirements of a vendor do you require before you make a purchase? 3. Would you go with a relatively new company? If so, what would make you choose them over someone who is more knowledgeble about clusters or who has been in the computer business for a good number of years. 4. What type of support options do you all require. Do you just require warranty support or do you require full support(software and hardware, patches, etc etc) 5. Finally, what is the number one thing that always sways your vote towards a particular vendor? Thanks Alot! Brent Clements Brent Clements Information Technology Rice University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Sat Jan 31 14:54:47 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Sat, 31 Jan 2004 20:54:47 +0100 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <401BE891.708@obs.unige.ch> References: <401BE891.708@obs.unige.ch> Message-ID: <401C0807.4000209@telia.com> ATTENTION! Liquid Nitrogen is deadly dangerous as it will shut off the breathing function of the human body when the Oxygen content falls below 19.5 % and thereby causing suffocation. This is not an unusual accident type in the Oil- & Gas- as well as in the Shipping industry. For the sake of your safety I'm recommending the use of an Industrial Safe Working Practice as a guideline before any work commence. A reference: **http://www.medicine.uiowa.edu/biochemstores/Pages/ln2msds.htm Best regards Per Lindstrom Daniel Pfenniger wrote: > John Hearns wrote: > >> On Fri, 30 Jan 2004, A.J. Rossini wrote: > > >>> to people that it isn't possible, but someone brought up "portable >>> liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>> to get stable systems, not instability :-). >>> >> >> Daft idea. >> What happens when the liquid nitrogen vapourises? It turns to >> nitrogen gas. And if you are unlucky anyone in the room chokes... >> I was reminded of this in the lift of a chemistry department where >> we have an opteron cluster - a large sign reminding that LNO2 and >> dangerous chemicals are not allowed in passenger lifts. > > > When liquid N2 vaporizes in a closed room the natural fraction of N2 > in air, already 78% by volume, just increases. N2 is thus not toxic, > the only risk by increasing N2 volume is to reach a too low O2 > concentration > for mammals beings breathing in the room. > > Dan > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel.pfenniger at obs.unige.ch Sat Jan 31 16:59:26 2004 From: daniel.pfenniger at obs.unige.ch (Pfenniger Daniel) Date: Sat, 31 Jan 2004 21:59:26 +0000 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <401C0807.4000209@telia.com> References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> Message-ID: <401C253E.9040206@obs.unige.ch> Per Lindstrom wrote: > ATTENTION! > > Liquid Nitrogen is deadly dangerous as it will shut off the breathing > function of the human body when the Oxygen content falls below 19.5 % > and thereby causing suffocation. > Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly dangerous when inhaled in too large quantities. The quoted 19.5% are probably referring to the human body (blood?) concentration, not the inhaled air. Otherwise most of the people in pubs or small rooms would rapidly die as the normal O2 concentration in air is 20%. Note that in the responded message John was confusing N2 and NO2. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Sat Jan 31 17:30:47 2004 From: gerry.creager at tamu.edu (Gerry Creager (N5JXS)) Date: Sat, 31 Jan 2004 16:30:47 -0600 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <401C253E.9040206@obs.unige.ch> References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> <401C253E.9040206@obs.unige.ch> Message-ID: <401C2C97.8020903@tamu.edu> At the risk of turning this technical, the real problem is usually the partial pressure of available oxygen. Generally, if the PaO2 (arterial blood partial pressure) drops below about 80 mmHg, one begins to suffer the effects of hypoxia. Most folks (University administrators seem to be immune to this) start dying at or below a PaO2 of 40 mmHg. In general, if one considers a standard pressure to be 760 mmHg at sea level, and noting that, in fact the normal concentration of oxygen available ambiently is roughly 20.95%, one sees the ambient PO2 at sea level is approximately 159 mmHg. Upon inspiration, one adds significant water vapor to the inhaled air (for reference, the medical and aviation world tend to assume the partial pressure of water in ambient conditions is about 4 mmHg), usually, about 40 mmHg at saturation... which is how the inspired air is supposed to be when it hits the lungs. So, taking that off-the-top from the partial pressure of oxygen in inspired air we have 159 - 40= 119 mmHg. Now, there's a gradient that helps control the transfer of oxygen across the alveolar-capillary barrier, where gas exchange occurs. Further, there's some transfer at the same interface of carbon dioxide leaving the blood cells and plasma. This combines to a net pressure gradient (I'm working from memory here, and I've not had to give this lecture for several years, so the exact numbers could be off) of 20-25 mmHg. The net result is simple: The PaO2 (arterial O2 concentration) is about 95 mmHg for normal individuals. Nitrogen in an enclosed space tends to act simply as an oxygen displacer. Thus, you go from roughly 78% N2 in ambient air to however much you ramp it up, and the O2 concentration decreases from a nominal 21% (rounding error, you understand) to whatever's left. If you note, while there's trace gases in ambient air, we can generally ignore them and worry about N2 and O2. YMMV for increased concentrations of CO2 and CO. That's the end of gas exchange physiology I. There will be a short quiz Monday. We'll continue with the next module. I encourage everyone to have read the Pulmonary Medicine chapters in Harrison's for the next lecture. Gerry Pfenniger Daniel wrote: > Per Lindstrom wrote: > >> ATTENTION! >> >> Liquid Nitrogen is deadly dangerous as it will shut off the breathing >> function of the human body when the Oxygen content falls below 19.5 % >> and thereby causing suffocation. >> > Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly > dangerous when inhaled in too large quantities. > The quoted 19.5% are probably referring to the human body (blood?) > concentration, not the inhaled air. Otherwise most of the people in > pubs or small rooms would rapidly die as the normal O2 concentration > in air is 20%. > > Note that in the responded message John was confusing N2 and NO2. > > > Dan > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Office: 979.458.4020 FAX: 979.847.8578 Cell: 979.229.5301 Pager: 979.228.0173 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 18:26:55 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 15:26:55 -0800 Subject: [Beowulf] HVAC and room cooling... References: <401BE891.708@obs.unige.ch> Message-ID: <005d01c3e851$b78dbba0$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Daniel Pfenniger" To: "John Hearns" Cc: ; Sent: Saturday, January 31, 2004 9:40 AM Subject: Re: [Beowulf] HVAC and room cooling... > John Hearns wrote: > > On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > > Daft idea. > > What happens when the liquid nitrogen vapourises? It turns to nitrogen > > gas. And if you are unlucky anyone in the room chokes... > > I was reminded of this in the lift of a chemistry department where > > we have an opteron cluster - a large sign reminding that LNO2 and > > dangerous chemicals are not allowed in passenger lifts. > > When liquid N2 vaporizes in a closed room the natural fraction of N2 in > air, already 78% by volume, just increases. N2 is thus not toxic, the > only risk by increasing N2 volume is to reach a too low O2 concentration > for mammals beings breathing in the room. Which is precisely the problem. You can fill a room with nitrogen and there's no obvious sign of it. Someone opens the door, walks in, and is unconscious in a remarkably short time. The problem is that you've also displaced the CO2, so there's no "breathe reflex", and you just breath faster trying to get the PO2 in your blood up, which aggravates the problem, then you pass out. All that oxygen poor air is now flowing out through the door so eventually you'll get oxygen into the room, as well, but you're really relying on diffusion (there's not a huge density difference). This is one of the classic "silent but deadly" hazards. This is what "confined space" training is all about. People have climbed down into a tank or crawlspace and not climbed back out, for instance. Here at JPL, even in ordinary labs, if there's nitrogen plumbed in (we use it to purge "dry boxes" or temperature test chambers, e.g. to prevent frost from forming when it's cooled), we have oxygen monitors and big placards on the door. The risk is that the hose comes off, and that even a fairly low flow (say, 1-2 CFM, typically) can flush all the oxygen out overnight, for instance. Say the lab is 20x20x8 feet.. that's 3200 cubic feet, less a few hundred for the stuff that's in the lab (shelves, file cabinets, equipment racks, benches, etc). The hose pops off unnoticed and 12 hours later at a couple CGM, and the lab's full of nitrogen when you arrive the next morning. I can't recall the exact number for where you go unconscious, but certainly, somewhere around 10% O2 would probably be sufficient. You'd only have to displace half the air to get there. With LN2 there's also the problem that the gas is cold, so it pools more readily. LN2 and elevators is a bad combination if you get a catastrophic leak, because the elevator shaft fills. People do, in fact, die from this, every year. (though, thankfully, not at JPL).. There's cases of stagehands quietly suffocating below stage when they're using dryice foggers (CO2 is very much more dense and pools quite nicely). In stage shows where they use large quantities of LN2 for making fog (we're talking hundreds of gallons here!), they generally use a mix of LN2 and LO2 for just this reason. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Sat Jan 31 19:52:27 2004 From: rossini at blindglobe.net (A.J. Rossini) Date: Sat, 31 Jan 2004 16:52:27 -0800 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <007701c3e855$6c6c58d0$36a8a8c0@LAPTOP152422> (Jim Lux's message of "Sat, 31 Jan 2004 15:53:27 -0800") References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> <401C253E.9040206@obs.unige.ch> <007701c3e855$6c6c58d0$36a8a8c0@LAPTOP152422> Message-ID: <85llnnr584.fsf@servant.blindglobe.net> Thanks, all! Looks like LN2 isn't as bad as I thought. I still don't think we'll be using it as a temporary cooling measure, though :-). Looks like I'll be doing a local co-lo, and getting my collaborators to pay for the computing lab infrastructure... best, -tony "Jim Lux" writes: > Found some definitions: > An oxygen-deficient atmosphere means that there is not enough oxygen in the > space. > Normal air is made up of 20.8 percent oxygen. > > An oxygen-deficient atmosphere has less than 19.5 percent of oxygen. > > Levels below 10 percent can cause unconsciousness and levels below 8 > percent can quickly cause death. > > That's probably where Per's 19.5% came from > > And, a delightful paragraph from OSHA: > Oxygen deprivation is one form of asphyxiation. While it is desirable to > maintain the atmospheric oxygen level at 21% by volume, the body can > tolerate deviation from this ideal. When the oxygen level falls to 17%, the > first sign of hypoxia is a deterioration to night vision which is not > noticeable until a normal oxygen concentration is restored. Physiologic > effects are increased breathing volume and accelerated heartbeat. Between > 14-16% physiologic effects are increased breathing volume, accelerated > heartbeat, very poor muscular coordination, rapid fatigue, and intermittent > respiration. Between 6-10% the effects are nausea, vomiting, inability to > perform, and unconsciousness. Less than 6%, spasmatic breathing, convulsive > movements, and death in minutes. > > > ----- Original Message ----- > From: "Pfenniger Daniel" > To: "Per Lindstrom" > Cc: "John Hearns" ; > ; > Sent: Saturday, January 31, 2004 1:59 PM > Subject: Re: [Beowulf] HVAC and room cooling... > > >> Per Lindstrom wrote: >> > ATTENTION! >> > >> > Liquid Nitrogen is deadly dangerous as it will shut off the breathing >> > function of the human body when the Oxygen content falls below 19.5 % >> > and thereby causing suffocation. >> > >> Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly >> dangerous when inhaled in too large quantities. >> The quoted 19.5% are probably referring to the human body (blood?) >> concentration, not the inhaled air. Otherwise most of the people in >> pubs or small rooms would rapidly die as the normal O2 concentration >> in air is 20% > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 18:53:27 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 15:53:27 -0800 Subject: [Beowulf] HVAC and room cooling... References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> <401C253E.9040206@obs.unige.ch> Message-ID: <007701c3e855$6c6c58d0$36a8a8c0@LAPTOP152422> Found some definitions: An oxygen-deficient atmosphere means that there is not enough oxygen in the space. Normal air is made up of 20.8 percent oxygen. An oxygen-deficient atmosphere has less than 19.5 percent of oxygen. Levels below 10 percent can cause unconsciousness and levels below 8 percent can quickly cause death. That's probably where Per's 19.5% came from And, a delightful paragraph from OSHA: Oxygen deprivation is one form of asphyxiation. While it is desirable to maintain the atmospheric oxygen level at 21% by volume, the body can tolerate deviation from this ideal. When the oxygen level falls to 17%, the first sign of hypoxia is a deterioration to night vision which is not noticeable until a normal oxygen concentration is restored. Physiologic effects are increased breathing volume and accelerated heartbeat. Between 14-16% physiologic effects are increased breathing volume, accelerated heartbeat, very poor muscular coordination, rapid fatigue, and intermittent respiration. Between 6-10% the effects are nausea, vomiting, inability to perform, and unconsciousness. Less than 6%, spasmatic breathing, convulsive movements, and death in minutes. ----- Original Message ----- From: "Pfenniger Daniel" To: "Per Lindstrom" Cc: "John Hearns" ; ; Sent: Saturday, January 31, 2004 1:59 PM Subject: Re: [Beowulf] HVAC and room cooling... > Per Lindstrom wrote: > > ATTENTION! > > > > Liquid Nitrogen is deadly dangerous as it will shut off the breathing > > function of the human body when the Oxygen content falls below 19.5 % > > and thereby causing suffocation. > > > Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly > dangerous when inhaled in too large quantities. > The quoted 19.5% are probably referring to the human body (blood?) > concentration, not the inhaled air. Otherwise most of the people in > pubs or small rooms would rapidly die as the normal O2 concentration > in air is 20% _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 18:38:36 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 15:38:36 -0800 Subject: [Beowulf] HVAC and room cooling... References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> Message-ID: <006e01c3e853$592f0850$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Per Lindstrom" To: "Daniel Pfenniger" Cc: "John Hearns" ; ; Sent: Saturday, January 31, 2004 11:54 AM Subject: Re: [Beowulf] HVAC and room cooling... > ATTENTION! > > Liquid Nitrogen is deadly dangerous as it will shut off the breathing > function of the human body when the Oxygen content falls below 19.5 % > and thereby causing suffocation. > That high? I'm not so sure.. Consider for instance that conscious ness has mostly to do with partial pressure of oxygen in the gas being breathed, which then transfers to the PO2 in your blood. As a rule of thumb, the pressure drops by half with every 18,000 feet of elevation change (5500 m). Most people can tolerate a sudden pressure drop to an equivalent altitude of 5000 ft without losing consciousness, and, for instance, the Federal Aviation Administration, uses 10,000 ft as a cutoff. 19.5% oxygen, compared to the usual 21%, is a partial O2 pressure change of about 2000 ft or so (600m). I would go for 9.5% (it's well established that suddenly taking someone to 5000 meters will cause unconsciousness, although not that quickly... you have to go to something like 15,000 m/40,000 ft to get rapid unconsciousness). They do this sort of training in altitude chambers for pilots to impress on them the need to "keep that mask fastened tightly, even if you think you're getting sick" We're sort of off topic here, but think in terms of those little oxygen masks that drop "in case of a sudden cabin depressurization"... Not only will there be nosebleeds and broken eardrums a plenty if this happens, but I think you'd better move pretty fast to get that mask on. By the way, if you know the loss of oxygen is coming, you can hold your breath (or just not breathe) and be conscious for quite a while (30 seconds to minute would be fairly easy..). The problem with the nitrogen purge is that you don't realize it, and you keep on breathing, flushing the oxygen from your lungs, until it's too late. My math may be all wrong here, but that's the general idea. > This is not an unusual accident type in the Oil- & Gas- as well as in > the Shipping industry. For the sake of your safety I'm recommending the > use of an Industrial Safe Working Practice as a guideline before any > work commence. > > A reference: **http://www.medicine.uiowa.edu/biochemstores/Pages/ln2msds.htm > > Best regards > Per Lindstrom > > > Daniel Pfenniger wrote: > > > John Hearns wrote: > > > >> On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > > > >>> to people that it isn't possible, but someone brought up "portable > >>> liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > >>> to get stable systems, not instability :-). > >>> > >> > >> Daft idea. > >> What happens when the liquid nitrogen vapourises? It turns to > >> nitrogen gas. And if you are unlucky anyone in the room chokes... > >> I was reminded of this in the lift of a chemistry department where > >> we have an opteron cluster - a large sign reminding that LNO2 and > >> dangerous chemicals are not allowed in passenger lifts. > > > > > > When liquid N2 vaporizes in a closed room the natural fraction of N2 > > in air, already 78% by volume, just increases. N2 is thus not toxic, > > the only risk by increasing N2 volume is to reach a too low O2 > > concentration > > for mammals beings breathing in the room. > > > > Dan > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Jan 31 08:34:55 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 31 Jan 2004 08:34:55 -0500 (EST) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <85k739x8q8.fsf@servant.blindglobe.net> Message-ID: On Fri, 30 Jan 2004, A.J. Rossini wrote: > > I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual > athlon or opteron machines). > > Can anyone suggest where I might look to find information for this? There are some documents on the brahma site (and my personal website) that might help. Quite a bit of their content has been drawn from extended threads on this list, so a look through the list archives is also in order. My book on the brahma site and an article in Linux Magazine and I think one other document should tell you how to work out required capacity and so forth. > Also, does anyone have any brilliant thoughts for cooling an internal > room that can't affordably get chilled water? (I've been suggesting > to people that it isn't possible, but someone brought up "portable > liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > to get stable systems, not instability :-). How big a room? Is it ducted for air conditioning of any sort? Does it have any windows or outside walls (probably not, given that "internal room", eh?). It isn't a matter of brilliance. The electrical cables running into the room can be thought of as pipes bringing in pure heat and pouring it out onto the floor. Some of that heat will leak out, driven by temperature differences, through the walls and ceiling, and the temperature of the room will rise until equilibrium of a sort is established (probably stabilizing too hot for most hardware). To run systems in there, you have to provide a means for removing that heat. Chilled water can be heated and removed/recycled. Chilled air can be heated and removed/recycled. Cooler air from surrounding air conditioned rooms can be forcably driven through, heated, and indirectly removed/recycled. You will have to provide some sort of mechanism for removing the heat, and they'll all involve moving cooled mass in and warmed mass out. Sorry. rgb > > best, > -tony > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Sat Jan 31 21:51:01 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Sat, 31 Jan 2004 18:51:01 -0800 (PST) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: Message-ID: hi ya On Sat, 31 Jan 2004, John Hearns wrote: > What happens when the liquid nitrogen vapourises? It turns to nitrogen > gas. And if you are unlucky anyone in the room chokes... > I was reminded of this in the lift of a chemistry department where > we have an opteron cluster - a large sign reminding that LNO2 and > dangerous chemicals are not allowed in passenger lifts. > > > BTW, when arranging air flows try to get cold air ducted into the front of > the rack - node tend to draw air in the front and exhaust to the rear. it'd be better, if one could, to make it bottom to top air flow and all inside a sealed ( reasonably closed ) rack if co-location is an option... the best choice would be alsaka or canada and leave the 3rd/4th story window open to the outside ?? ( wonder how it'd run during the short summers ) - and yes.. lots of people forget about regulatory issues when building clusters or too many pc's in the office - not enough building power being the first major "oops" - fire hazard being the 2nd violations ( they(fire dept) don't allow daisy chained power strips ) - not enough hvac or costs of cooling .. ( sometimes big fans, in the right direction, is enough to keep the pc cooler ) - too much regulations also kills the project :-0 c ya alvin _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Fri Jan 2 03:27:20 2004 From: proksy_me at yahoo.com (prakash borade) Date: Fri, 2 Jan 2004 00:27:20 -0800 (PST) Subject: [Beowulf] i have problem in my beowulf Message-ID: <20040102082720.81089.qmail@web40104.mail.yahoo.com> hello everybody i am very new to clustering environ ment and also to beowulf i need some guid in how to build a beowul cluster i m trying it with mpich 1.2.5.2 is it neccesary to initialize the lam environmnet what i know is that lam/mpi and mpich are to diffrent implementtations of mpi i get following error for the standard cpilog program after i have initialized lam environment using lamboot [mpich at tung mpich]$ mpirun -np 2 cpilog Process 0 running on tung Process 0 running on dyp pi is approximately 3.1415926535897634, Error is 0.0000000000000298 wall clock time = 0.768179 ----------------------------------------------------------------------------- It seems that [at least] one of processes that was started with mpirun did not invoke MPI_INIT before quitting (it is possible that more than one process did not invoke MPI_INIT -- mpirun was only notified of the first one, which was on node n0). mpirun can *only* be used with MPI programs (i.e., programs that invoke MPI_INIT and MPI_FINALIZE). You can use the "lamexec" program to run non-MPI programs over the lambooted nodes. ----------------------------------------------------------------------------- __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Sat Jan 3 01:42:21 2004 From: proksy_me at yahoo.com (prakash borade) Date: Fri, 2 Jan 2004 22:42:21 -0800 (PST) Subject: [Beowulf] i have problem in my beowulf Message-ID: <20040103064221.4142.qmail@web40112.mail.yahoo.com> > hi glen > first of all thanks a lot for replying > me. > as mentioned that mpirun command would not require > lam > > > but at my machines seems that unless and until i > enable lam environment i cannot run mpirun > > also > > the output i.e > process 0 of 0 at node00 > process 0 of 0 at node01 > > is incorrect > and it should be as > > process 0 of 0 at node00 > process 1 of 0 at node01 > > also all my paths are well set > > i have also added all my hosts to the file > /etc/lam/lam-bhosts.def > as > > node00 > node01 > > is thist thinf necessary > > because it has not been mentioned > in any of the beowulf building manual > > > > > > __________________________________ > Do you Yahoo!? > Find out what made the Top Yahoo! Searches of 2003 > http://search.yahoo.com/top2003 __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Kris.Boutilier at scrd.bc.ca Sun Jan 4 23:50:44 2004 From: Kris.Boutilier at scrd.bc.ca (Kris Boutilier) Date: Sun, 4 Jan 2004 20:50:44 -0800 Subject: [Beowulf] Offtopic: Emulex cLan 1000 support under kernels 2.4.x or 2.6.x? Message-ID: <1174450A1968D111AAAF00805FC162AEBE1344@deep_thought.secure.scrd.bc.ca> This seems like the most probable place to find someone who will know first hand - Is anyone aware of step-by-step instructions, updated code or any other means of getting the cLan 1000 VIA hardware running under anything higher than a 2.2 kernel? Emulex still offer binaries for Redhat 6.x on their site, and they offer source code for v1.3.1 of the driver, but I understand it to now be considered very obsolete... certainly it doesn't compile out of the box against 2.4.x Suggestions anyone? _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tony at mpi-softtech.com Mon Jan 5 10:32:28 2004 From: tony at mpi-softtech.com (Anthony Skjellum) Date: Mon, 5 Jan 2004 09:32:28 -0600 Subject: [Beowulf] Offtopic: Emulex cLan 1000 support under kernels 2.4.x or 2.6.x? References: <1174450A1968D111AAAF00805FC162AEBE1344@deep_thought.secure.scrd.bc.ca> Message-ID: <00c901c3d3a1$211a8f50$b600a8c0@ciswinnet.cis.uab.edu> Hi, we have a Yahoo! support group for cLan/Emulex VIA hardware... please see http://groups.yahoo.com/group/giganet-network-users/ There are 57 members, and people have successfully run on recent RH kernels, and you are likely to get feedback from those folks. Tony ----- Original Message ----- From: "Kris Boutilier" To: Sent: Sunday, January 04, 2004 10:50 PM Subject: [Beowulf] Offtopic: Emulex cLan 1000 support under kernels 2.4.x or 2.6.x? > This seems like the most probable place to find someone who will know first > hand - Is anyone aware of step-by-step instructions, updated code or any > other means of getting the cLan 1000 VIA hardware running under anything > higher than a 2.2 kernel? Emulex still offer binaries for Redhat 6.x on > their site, and they offer source code for v1.3.1 of the driver, but I > understand it to now be considered very obsolete... certainly it doesn't > compile out of the box against 2.4.x > > Suggestions anyone? > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at comcast.net Mon Jan 5 18:30:07 2004 From: laytonjb at comcast.net (Jeffrey B. Layton) Date: Mon, 05 Jan 2004 18:30:07 -0500 Subject: [Beowulf] Article on First Clusters from Around the World Message-ID: <3FF9F37F.2050207@comcast.net> Hello, I'm working on an article for ClusterWorld magazine about people's first clusters from around the World. The intent of the article is to find out why people started using clusters, what problems they are trying to solve, how they learned about clusters, and what their experiences have been. If you have built a small cluster recently, I'd like to interview you for the article. Just email me at the return address on this posting. Thanks! Jeff Layton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rene.storm at emplics.com Tue Jan 6 08:20:19 2004 From: rene.storm at emplics.com (Rene Storm) Date: Tue, 6 Jan 2004 14:20:19 +0100 Subject: [Beowulf] Benchmark results In-Reply-To: <200312301523.SAA06085@nocserv.free.net> References: <200312301523.SAA06085@nocserv.free.net> Message-ID: <20040106142019.48d39cee.rene.storm@emplics.com> Dear cluster folks, I saw some weird results of different benchmarks on my systems. My problem is to verify these results. I've played with hpl benchmark on our clusters (8 CPU Intel, 8 CPU Opteron) and wasn't able to get two times the same result. Of course no configuration changes. Difference round 5-10%. So I went down to one machine, but the same behavior. Standart mpich, per-complied lapack, two processor on an smp machine. 10 times the same benchmark gives back 10 different results. Same trouble with stream memory benchmark, dgemv matrix calculation and others. There was no network, but it was a big installation, so it could be, that there are some running jobs (eg cron-jobs) which disturb my benchmarks results. Next Step: I created a self-booting cdrom from the scratch - added litte bit of X and a gui my benchmarks. 1) cpi - calculating pi with mpich on smp (ch_p4 on loopback) 2+3) dgemv - 5kx5k matrix calculation , single and smp 4) crafty chess benchmark - 2 threads 5) /bin/true - calling true via fork(), taking the time 6) PMB-MPI1 - Pallas benchmark There is only sshd for mpich running, everything is loaded from the cdrom into a ramdisk. BUT same result for the results. 5-10% unsteady. Please let me know, if you could see same behavior on your machines. Does someone know a reason for that? If you would like to check this cd out -> http://212.12.58.172/bench/bench_on_cd.iso.gz 40 MB download. You will need at least 512MB and no usb mice, keyboard. Thanks in advance Rene Storm _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Tue Jan 6 09:10:04 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Tue, 06 Jan 2004 09:10:04 -0500 Subject: [Beowulf] Benchmark results In-Reply-To: <20040106142019.48d39cee.rene.storm@emplics.com> References: <200312301523.SAA06085@nocserv.free.net> <20040106142019.48d39cee.rene.storm@emplics.com> Message-ID: <3FFAC1BC.6020201@scalableinformatics.com> Hi Rene: How long were the runs in comparison to the timer resolution? What other processes (if any) were running? Did various cron-jobs light off during any portion of this? For opterons, there is an a processor affinity issue whereby you can get different performance if the memory allocated and used is not associated with that processor. I have heard that there are patches which deal with this, though I haven't had time to try them. I do seem to remember an alignment issue for Xeon as well. I would not expect absolutely identical timing data each time the benchmark were run, but I would expect a tight grouping around a mean value. The relative size of the error would be related to a number of factors, including timer resolution, machine state, etc. For runs of reasonable length (longer than 30 minutes), the timer resolution effects should be minimal. If you have other processes lighting off, consuming memory, processor cycles, cache, bandwidth, interrupts, it is likely that your distribution will reflect this. Joe Rene Storm wrote: >Dear cluster folks, > >I saw some weird results of different benchmarks on my systems. >My problem is to verify these results. > >I've played with hpl benchmark on our clusters (8 CPU Intel, 8 CPU Opteron) and wasn't able to get two times the same result. Of course no configuration changes. Difference round 5-10%. > >So I went down to one machine, but the same behavior. >Standart mpich, per-complied lapack, two processor on an smp machine. >10 times the same benchmark gives back 10 different results. >Same trouble with stream memory benchmark, dgemv matrix calculation and others. > >There was no network, but it was a big installation, so it could be, that there are some running jobs (eg cron-jobs) which disturb my benchmarks results. > >Next Step: I created a self-booting cdrom from the scratch - added litte bit of X and a gui my benchmarks. > >1) cpi - calculating pi with mpich on smp (ch_p4 on loopback) >2+3) dgemv - 5kx5k matrix calculation , single and smp >4) crafty chess benchmark - 2 threads >5) /bin/true - calling true via fork(), taking the time >6) PMB-MPI1 - Pallas benchmark > >There is only sshd for mpich running, everything is loaded from the cdrom into a ramdisk. >BUT same result for the results. 5-10% unsteady. > >Please let me know, if you could see same behavior on your machines. >Does someone know a reason for that? > > >If you would like to check this cd out -> >http://212.12.58.172/bench/bench_on_cd.iso.gz >40 MB download. You will need at least 512MB and no usb mice, keyboard. > >Thanks in advance >Rene Storm > > > > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 6 09:51:25 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 6 Jan 2004 09:51:25 -0500 (EST) Subject: [Beowulf] Benchmark results In-Reply-To: <3FFAC1BC.6020201@scalableinformatics.com> Message-ID: On Tue, 6 Jan 2004, Joe Landman wrote: > Hi Rene: > > How long were the runs in comparison to the timer resolution? What > other processes (if any) were running? Did various cron-jobs light off > during any portion of this? > > For opterons, there is an a processor affinity issue whereby you can > get different performance if the memory allocated and used is not > associated with that processor. I have heard that there are patches > which deal with this, though I haven't had time to try them. I do seem > to remember an alignment issue for Xeon as well. > > I would not expect absolutely identical timing data each time the > benchmark were run, but I would expect a tight grouping around a mean > value. The relative size of the error would be related to a number of > factors, including timer resolution, machine state, etc. For runs of > reasonable length (longer than 30 minutes), the timer resolution effects > should be minimal. If you have other processes lighting off, consuming > memory, processor cycles, cache, bandwidth, interrupts, it is likely > that your distribution will reflect this. > > Joe Ya. This is one of the things motivating the bproc beowulf design -- the ability to build a "stripped" machine with much more controlled and predictable state and with an absolutely minimal set of competing "system" tasks running on an otherwise quiescent system. You can check out Scyld (open source commercial) or Clustermatic (non-commercial/GPL) if you want to pursue this route. The run length issue is pure central limit theorem. If you do short runs (or internally time short loops), expect large fluctuations and a relatively broad distribution of times as e.g. a task swap or context switch in the middle of a timing loop will lead to wildly different times on a per-case basis. If you run "long" times you can average more effectively over the fluctuating/uncontrolled part of the system state and your mean times SHOULD home in their true mean value, with a distribution of means that narrows roughly with the inverse square root of the run time. Exceptions to this are, as Joe noted, persistent state differences between runs. These should be investigated and understood, as 5% runtime difference is like getting a free node in twenty and worth pursuing. ps and top are your friend. They'll eat some cycles themselves (good old Heisenberg, sort of:-), but they'll also give you a realtime picture of the tasks that are competing. xmlsysd/wulfstat might help as well (sort of like vmstat and/or netstat run across the whole cluster at once). The other thing I can think of that might result in a large persistent variance (besides memory/motherboard issues) is substantial difference in network communication pattern. If your communications aren't carefully organized, they may self-organize depending on a nucleation condition (who talks to who and and gets through first). I can easily imagine distinct communication patterns emerging with a large variance. Actually, if I were a REAL computer scientist, this would be a fairly interesting topic for a research project -- spontaneous emergence of persistent patterns with very different efficiencies in complex networks. If you look (and upon examination it looks like this might be the problem) there are probably research papers out there on it already. Can't imagine that this hasn't been studied. rgb > > Rene Storm wrote: > > >Dear cluster folks, > > > >I saw some weird results of different benchmarks on my systems. > >My problem is to verify these results. > > > >I've played with hpl benchmark on our clusters (8 CPU Intel, 8 CPU Opteron) and wasn't able to get two times the same result. Of course no configuration changes. Difference round 5-10%. > > > >So I went down to one machine, but the same behavior. > >Standart mpich, per-complied lapack, two processor on an smp machine. > >10 times the same benchmark gives back 10 different results. > >Same trouble with stream memory benchmark, dgemv matrix calculation and others. > > > >There was no network, but it was a big installation, so it could be, that there are some running jobs (eg cron-jobs) which disturb my benchmarks results. > > > >Next Step: I created a self-booting cdrom from the scratch - added litte bit of X and a gui my benchmarks. > > > >1) cpi - calculating pi with mpich on smp (ch_p4 on loopback) > >2+3) dgemv - 5kx5k matrix calculation , single and smp > >4) crafty chess benchmark - 2 threads > >5) /bin/true - calling true via fork(), taking the time > >6) PMB-MPI1 - Pallas benchmark > > > >There is only sshd for mpich running, everything is loaded from the cdrom into a ramdisk. > >BUT same result for the results. 5-10% unsteady. > > > >Please let me know, if you could see same behavior on your machines. > >Does someone know a reason for that? > > > > > >If you would like to check this cd out -> > >http://212.12.58.172/bench/bench_on_cd.iso.gz > >40 MB download. You will need at least 512MB and no usb mice, keyboard. > > > >Thanks in advance > >Rene Storm > > > > > > > > > >_______________________________________________ > >Beowulf mailing list, Beowulf at beowulf.org > >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > -- > Joseph Landman, Ph.D > Scalable Informatics LLC, > email: landman at scalableinformatics.com > web : http://scalableinformatics.com > phone: +1 734 612 4615 > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Per.Lindstrom at me.chalmers.se Tue Jan 6 14:54:49 2004 From: Per.Lindstrom at me.chalmers.se (=?ISO-8859-1?Q?Per_Lindstr=F6m?=) Date: Tue, 06 Jan 2004 20:54:49 +0100 Subject: [Beowulf] Website for Linux workstations and small clusters Message-ID: <3FFB1289.90800@me.chalmers.se> Hi I wish you and your near and dear a Happy New Year. We, Department of Naval Architecture and Ocean Engineering, Chalmers University of Technology will set up a Web site with instructions how to build, install, configure and operate Linux workstations and small Linux clusters as well as the installation and the basic use of calculation and simulation softwares. The site's structure will be very much influenced by the Materials Algorithms Project, http://www.msm.cam.ac.uk/map/mapmain.html It will take some weeks until we are up and running. Please contact me if you want to contribute with your experience and/or have some ideas how this should be done Best regards Per Lindstr?m, Industrial Doctoral Student Per.Lindstrom at me.chalmers.se www.na.chalmers.se Per Lindstr?m M.Sc, CEng, SenMWeldI, IWE, Welding Engineer to Kvaerner Power AB +46 (0)31 50 10 14 Per.Lindstrom at akerkvaerner.com http://akerkvaerner.com/Internet/IndustriesAndServices/Power/default.htm Jim Lux wrote: >I fully agree... I suspect that most readers of this list start with a small >cluster, and a historical record of what it took to get it up and running is >quite useful, especially the hiccups and problems that you inevitably >encounter. (e.g. what do you mean the circuit breaker just tripped on the >plug strip when we plugged all those things into it?) > >----- Original Message ----- >From: "Per Lindstr?m" >To: "D. Scott" ; "Anand TNC" > >Cc: "Beowulf" ; "Josh Moore" ; "Per" > >Sent: Monday, December 29, 2003 11:59 AM >Subject: [Beowulf] Websites for small clusters > > > > >>Hi Dan, >> >>It should be great if you publish your cluster work instructions on a >>website. I have found that there is need for a such place. >> >>The site http://www.msm.cam.ac.uk/map/mapmain.html is a good example on >>how a website sharing scientific and/or professional experience can be >>aranged. >> >>If it not allready exist, shall we arrange something similar for few >>node clusters? (Few node clusters 2 - 30 nodes?) >> >>Best regards >>Per Lindstr?m >> >> >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >> >> >http://www.beowulf.org/mailman/listinfo/beowulf > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mwill at penguincomputing.com Tue Jan 6 12:53:13 2004 From: mwill at penguincomputing.com (Michael Will) Date: Tue, 6 Jan 2004 09:53:13 -0800 (PST) Subject: [Beowulf] Re: Benchmark results Message-ID: <40318.64.243.132.186.1073411593.squirrel@imap.penguincomputing.com> Rene, Do you see that fluctuation only on the opterons or also on the intel cpus (I assume Xeon?) If it was only on the opteron, then you will have node memory interleave turned off in your bios, and sometimes the data you work with is close to your CPU and sometimes far. (each CPU has a bank of RAM and accesses the other CPUs bank through hypertransport, which is slower) When running a memory and floatpoint intensive benchmark (neuronal network training program like SNNS), we got two consistent results: They were alternating, depending on which CPU the code ran, and they were 20% off of each other for that setup. Once we put in the same amount of memory in both banks, and turned on 'node memory interleave' in the bios, they were consistently between the two, a bit closer to the faster value. I will take a closer look at your CD later, too. Michael Will PS: Did I mention the opterons blew away the xeons by far? -- Michael Will, Linux Sales Engineer Tel: 415.358.2673 Toll Free: 888-PENGUIN PENGUIN COMPUTING - The World's Most Reliable Linux System www.penguincomputing.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rene.storm at emplics.com Tue Jan 6 18:00:18 2004 From: rene.storm at emplics.com (Rene Storm) Date: Wed, 7 Jan 2004 00:00:18 +0100 Subject: [Beowulf] ...Re: Benchmark results Message-ID: <29B376A04977B944A3D87D22C495FB230127BB@vertrieb.emplics.com> HI, first of all, thanks for the responses.... It seems that there are some more points I have to observe. For example the hpl benchmark: The job was running on two processors (smp) on loopback. Used time round about 20min. There was no cronjob, no X, only a niced top. The machines environments are almost constant (temperature->lm_sensors). A reboot after every single run. And yes, the differences on the opterons are greater as on i386. I will take a look at interleaving. But why should the memory allocation change between two runs? So it seems, I will to go a little bit deeper into that and try to get some more results. What I do expect is some sort of Gaussian distribution on the results. I will take the hpl and throw the results into gnuplot. One question about the walltime. Could this be a problem? Does someone know how this really works? I read something about HZ and jiffies and so, but it was a bit confusing. Maybe this could be the point were old, unfair heisenberg could have had a chance. Rene Storm -----Urspr?ngliche Nachricht----- Von: Michael Will [mailto:mwill at penguincomputing.com] Gesendet: Dienstag, 6. Januar 2004 18:53 An: beowulf at beowulf.org Betreff: [Beowulf] Re: Benchmark results Rene, Do you see that fluctuation only on the opterons or also on the intel cpus (I assume Xeon?) If it was only on the opteron, then you will have node memory interleave turned off in your bios, and sometimes the data you work with is close to your CPU and sometimes far. (each CPU has a bank of RAM and accesses the other CPUs bank through hypertransport, which is slower) When running a memory and floatpoint intensive benchmark (neuronal network training program like SNNS), we got two consistent results: They were alternating, depending on which CPU the code ran, and they were 20% off of each other for that setup. Once we put in the same amount of memory in both banks, and turned on 'node memory interleave' in the bios, they were consistently between the two, a bit closer to the faster value. I will take a closer look at your CD later, too. Michael Will PS: Did I mention the opterons blew away the xeons by far? -- Michael Will, Linux Sales Engineer Tel: 415.358.2673 Toll Free: 888-PENGUIN PENGUIN COMPUTING - The World's Most Reliable Linux System www.penguincomputing.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Tue Jan 6 19:44:31 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Tue, 06 Jan 2004 19:44:31 -0500 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <29B376A04977B944A3D87D22C495FB230127BB@vertrieb.emplics.com> References: <29B376A04977B944A3D87D22C495FB230127BB@vertrieb.emplics.com> Message-ID: <1073436271.15823.30.camel@protein.scalableinformatics.com> On Tue, 2004-01-06 at 18:00, Rene Storm wrote: > HI, > first of all, thanks for the responses.... > > It seems that there are some more points I have to observe. > For example the hpl benchmark: > The job was running on two processors (smp) on loopback. Used time round about 20min. > There was no cronjob, no X, only a niced top. > The machines environments are almost constant (temperature->lm_sensors). > A reboot after every single run. One of the odder bits about benchmarking... Most folks I know don't reboot machines between real end user runs. Or umount/mount after a heavy IO run. If you are really aiming for the maximal in limited interference, you have probably turned everything off anyway, and are running in single user mode (or just above it with networking enabled). Doesn't guarantee much more than some default services are not run. You might want to turn of atd and disable stuff in /etc/cron.d . Not that this really makes much sense from a "realism" point of view, as end users are not (normally) allowed to do that. > And yes, the differences on the opterons are greater as on i386. > I will take a look at interleaving. But why should the memory allocation change between two runs? In short, when a process launches the NUMA nature of its memory could mean that the memory system where the thread gets its physical allocations differs from the memory system which is tied to the CPU running the code. For a NUMA system, this gives you an extra "hop" for memory accesses. There is the concept of process affinity whereby the thread is tied to a particular CPU it has been running on, which tends to prevent the cache from being rendered less effective by having the process hop from CPU to CPU. Tying an allocation to a CPU exists in various Unixen (IRIX in particular with the old dplace and similar directives), and I have heard rumors of a "runon" command (applying both a process/thread affinity, and pegging the memory allocated by the thread to the CPU running the thread). > So it seems, I will to go a little bit deeper into that and try to get some more results. > What I do expect is some sort of Gaussian distribution on the results. You should get some distribution. I would be reluctant to discuss the shape without at least a reasonable empirical theory. It does seem that the allocation locality would cause at least a bimodal shape (rough guess), something like 2 overlapping distributions. Other effects are going to spread this a bit. > I will take the hpl and throw the results into gnuplot. > > One question about the walltime. Could this be a problem? > Does someone know how this really works? I read something about HZ and jiffies and so, but it was a bit confusing. > Maybe this could be the point were old, unfair heisenberg could have had a chance. Physics 101: Take your precision of your clock. Your uncertainty in using the clock as a measuring device is +/- 1/2 of the clocks precision (call this delta_t), or 1/2 of the smallest interval that this clock can measure. As you are really interested in time intervals, you may need to appeal to basic error analysis. That said, if you look at a really rough estimate of the contribution of the random error of the clock for your runs, have a look at the ratio of delta_t to the total run time. For a run of 30 minutes, t ~ 1800 seconds, while using 0.01 seconds as the clocks precision gives you about 3 x 10**(-6) (or 1 part in 360000). That is, the contribution to the error due to the clock is small here. Even for a run of 60 seconds it is still in the 0.01% regime. To get a 5% error out of that, you would need delta_t/t ~ 0.05, or t about 0.1 second for your total benchmark run time. You wouldn't even need another process taking up time, the kernel will happily do this for you with some of its housekeeping chores (bdflush, kswapd, ...). Servicing lots of interrupts is a good way to slow things down, as are doing many small page allocations where one large one would suffice, etc. > > Rene Storm > > > -----Urspr?ngliche Nachricht----- > Von: Michael Will [mailto:mwill at penguincomputing.com] > Gesendet: Dienstag, 6. Januar 2004 18:53 > An: beowulf at beowulf.org > Betreff: [Beowulf] Re: Benchmark results > > Rene, > > Do you see that fluctuation only on the opterons or also on the intel cpus > (I assume Xeon?) > > If it was only on the opteron, then you will have node memory interleave > turned off in your bios, and sometimes the data you work with is close to > your CPU and sometimes far. (each CPU has a bank of RAM and accesses the > other CPUs bank through hypertransport, which is slower) > > When running a memory and floatpoint intensive benchmark (neuronal network > training program like SNNS), we got two consistent results: They were > alternating, depending on which CPU the code ran, and they were 20% off of > each other for that setup. > > Once we put in the same amount of memory in both banks, and turned on > 'node memory interleave' in the bios, they were consistently between the > two, a bit closer to the faster value. > > I will take a closer look at your CD later, too. > > Michael Will > PS: Did I mention the opterons blew away the xeons by far? -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From maikel_punie at hotmail.com Wed Jan 7 10:16:07 2004 From: maikel_punie at hotmail.com (Maikel Punie) Date: Wed, 07 Jan 2004 16:16:07 +0100 Subject: [Beowulf] Need help Message-ID: hallo everyone, well for my final project at school i need to build a beowulf, well i'm this far that all the node (diskless system) boot from the main node (ntfs) and there config and stuff from there, the main node can see all off his nodes. Now my question is whats the next step? How do i make the nodes do the calculations and stuff for the main node, how do i realy integrate the parrallel processing? Thanks a lot Maikel _________________________________________________________________ Al aan je vakantie gedacht? http://www.msn.be/reizen _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 7 11:18:33 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 7 Jan 2004 11:18:33 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <1073436271.15823.30.camel@protein.scalableinformatics.com> Message-ID: On Tue, 6 Jan 2004, Joe Landman wrote: > Physics 101: > > Take your precision of your clock. Your uncertainty in using the clock > as a measuring device is +/- 1/2 of the clocks precision (call this > delta_t), or 1/2 of the smallest interval that this clock can measure. > As you are really interested in time intervals, you may need to appeal > to basic error analysis. That said, if you look at a really rough > estimate of the contribution of the random error of the clock for your > runs, have a look at the ratio of delta_t to the total run time. For a > run of 30 minutes, t ~ 1800 seconds, while using 0.01 seconds as the > clocks precision gives you about 3 x 10**(-6) (or 1 part in 360000). > That is, the contribution to the error due to the clock is small here. > > Even for a run of 60 seconds it is still in the 0.01% regime. To get a > 5% error out of that, you would need delta_t/t ~ 0.05, or t about 0.1 > second for your total benchmark run time. A Brief Digression on Benchmark Timing (Yes, I lie like a rug. It is not brief. If you already know about timing, or don't care, hit 'd' now...:-) To add a bit of pure CPS 1 empiricism to Joe's remarks, I've played quite a bit with timers while writing cpu_rate and using lmbench and so forth. There are two reasonably accurate "simple" timing mechanisms available in linux, one simpler (but less accurate) than the other. gettimeofday returns a timeval struct with seconds and microseconds since the Epoch. Typical use in a timing loop saves the timeval at an interval start, runs the loop, saves the timeval at the stop, and computes the delta. This generates wall clock delta and EMPIRICALLY is accurate to about two microseconds, no more. This makes it nearly useless for timing short loops -- say a few hundred iterations of a multiply -- because multiplies take order of a nanosecond (depending on locality and clock) and one has to execute thousands and thousands of them to use up more time than the resolution of your clock. A timing of 2 usec +/- 2 usec isn't very useful. If you execute a few billion of them (so the internal loop takes order of seconds to complete) things are better -- 2 sec +/- 2 usec isn't too bad, precision wise. However now there is another tradeoff. A timed interval that occupies only a hundred microseconds or so has an excellent chance of not being interrupted (depending, of course, on system environment -- I'm assuming a basically quiescent network and no other toplevel userspace tasks running). One that occupies two seconds will be interrupted hundreds of times, because if nothing else the system timer will issue interrupts that the kernel has to handle. If you run "vmstat 1" on your box, you'll note that there are roughly 100 interrupts per second steady state on a quiet system. Associated with these are 100+ context switches. These interrupts and context switches are quite "expensive", and one is tempted to get off into a discussion of the kernel design and upper/lower half handlers and ways the system tries to bundle up its work to minimize impact on running processes, but it would be a moderate digression. Suffice it to say that if you time LOTS of 100 microsecond loops separately in a continuous process and look at the results, you're likely to see the bimodal distribution Joe refers to (or even a fully multimodal distribution) because SOME of those intervals will be interrupted and the interrupt/context switch will be "timed" along with the swapped out loop by the wall-clock timer. This can be somewhat ameliorated by adding a usleep right before each timing loop iteration, as the kernel will often take advantage of a task pause to complete pending interrupts. However, it alas isn't a magic bullet because not all interrupts are "pendable". gettimeofday is actually a mediocre timer for nearly all microbenchmarking purposes as it is so much slower than the processes that it times that getting good precision between these competing limits is not at all easy. It is just fine for macroscopic, large scale completion times that average over all sorts of system state changes (e.g. all those int/css). A much better timer is available on nearly all intel/amd cpus. You can actually read the CPU's onboard cycle counter with a suitable bit of assembler magic. By also reading the CPU's clock (from /proc/cpuinfo) and doing a bit of division, you can obtain a "time" accurate (empirically, again) to roughly 10-60 nanoseconds. Very roughly, as this depends strongly on the clock and other systems architecture differences that can vary a bit from system to system and whether or not the clock is wrapped up quasi-portably in a subroutine call (adding a bit of overhead) or inlined as raw assembler on both sides of the timing loop -- this is just what I measure. This nanotimer is FAR more suitable for use in microbenchmarking -- relatively small loops where one devoutly hopes to avoid getting nailed by a context switch as one is interested in "how many floating point multiplies can the system perform on numbers in its registers/L1 cache/L2 cache/memory per second". A loop with a few microseconds worth of operations, fairly likely not to be interrupted if you start immediately after a timer click, can yield < 1% accuracy. Multimodality in a distribution of such short loop timings yields "interesting" information about the system's CPU, cache, memory architecture, and their interaction with the kernel. I've wrapped both of the timers up in a single file in a somewhat portable form. Because there can be a LOT of clocks in a second (several billion, obviously) the nanotimer is a bit complex -- on the first call it saves the base clock registers as a baseline that it subtracts from subsequent calls BEFORE converting the results to a double precision nanosecond count. Done in the wrong order, you can end up with a significant precision loss subtracting two large numbers to get a very small number, hence the runaround. I'm pretty sure it is done right at this point. I attach the timer source below for anyone who cares. rgb > You wouldn't even need another process taking up time, the kernel will > happily do this for you with some of its housekeeping chores (bdflush, > kswapd, ...). Servicing lots of interrupts is a good way to slow things > down, as are doing many small page allocations where one large one would > suffice, etc. Absolutely. In fact, one of the "fun" things about microbenchmarking is that a good microbenchmarking suite and/or a set of good vmstat-like tools can really help you understand and "see" the actual costs of stuff like this. Josip Loncaric (on this list) worked very hard some years ago to run down a moderately hienous intermittant latency hit in the (IIRC) TCP networking stack, for example. Every now and then instead of a predictable relatively short packet latency a really long (again IIRC, 2 msec or the like compared to timings on the order of tens of usec) hit would appear. Sending lots of small packets is "bad". Doing lots of malloc is "bad". Doing any sort of interrupt is bad, in part because the system has a tendency to swap in waiting/competing tasks (if any) when the interrupt returns and of course because of the relatively high cost of the interrupt itself. Lots of system calls can have the same effect -- select, for example, has about the same cost as gettimeofday, and AFAICT nanosleep is really usleep -- the ONLY way I've found to "pause" for a nanoscale-specified time is to use a nanotimer-based polling loop. FORTUNATELY, most of this isn't horribly difficult to understand or to avoid in actual code, and FORTUNATELY most people really are interested in macroscopic wall-clock completion times of large scale code, where "date" at the shell is accurate enough to get decent timings. Where it makes a difference is in optimizing core loops or understanding how you spent all this money on a 3 GHz CPU system that IN PRINCIPLE has a 6 GFLOPS peak but in practice you are lucky to get 1 GFLOPS on a good day with a tail wind. [Parenthetical Aside: Youngsters don't know HOW lucky you are to get that 1 GFLOPS in a $1000 system, of course. Suffice it to say that as of a few years ago, your desktop would have been classified as a weapon for purposes of export, and sending one outside of the US would have taken a special license, and a decade ago a GFLOPS system (even a cluster) would have cost order of $1 million. Gotta love Moore's Law (and the fact that Our Government, helped enormously by the beowulf movement, finally wised up to Moore's Law and moved any remaining weapons boundaries out so far that they are invisible).] rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu nanotimer: COPYING is GPL v2b, so play on through. You can get the whole package in context in cpu_rate from my personal website and I think on brahma. %< snip snip snip ======================================================= /* *======================================================================== * $Id: nanotimer.h,v 1.1 2003/06/10 15:41:27 rgb Exp $ * * See copyright in copyright.h and the accompanying file COPYING *======================================================================== */ #include "copyright.h" #define K 1024 /* * The following are the struct components of the timer(s). They * should let us write one bit of code and use it with either * timer transparently. */ typedef struct { double (*timer)(); double start; double stop; double delta; double nsec_granularity; char name[K]; } Nanotimer; /* * Built in cycle counter timer (Intel or AMD) resolution * roughly 1 CPU clock (sub-nanosecond on modern CPUs). */ static double nsec_per_cycle; static unsigned long ax_first,dx_first; static unsigned long long count_first; void nanotimer_rdtsc_init(Nanotimer *new_nanotimer); double nanotimer_rdtsc(); /* * Fallback timer using gettimeofday. Resolution of roughly * 2 usec on a good day, but the way we scale test intervals * it doesn't usually matter. */ static struct timeval tv_first; void nanotimer_gttod_init(Nanotimer *nanotimer); double nanotimer_gttod(); %< snip snip snip ======================================================= /* * $Id: nanotimer.c,v 1.9 2003/02/28 21:18:04 rgb Exp $ * * See copyright in copyright.h and the accompanying file COPYING * */ /* *======================================================================== * timing and utility sources. tv_start and tv_stop are globals. *======================================================================== */ #include "cpu_rate.h" /* * The following to programs constitute a NONportable, but very accurate, * intel/amd timer for pentia and athlons (tsc flag in /proc/cpuinfo). * The init call basically extracts the CPU clock from /proc/cpuinfo * and converts it into nanoseconds/cpu clock cycle. The second reads * the cpu cycle counter directly and converts it into nanoseconds. * * To avoid potential problems with loss of precision on the interval * subtraction, this program stores the register contents on the * first call and only counts the nanoseconds from the first call, not * since the last boot. * * It is quite fast (order of 40-50 nsec). */ void nanotimer_rdtsc_init(Nanotimer *new_nanotimer) { int i,imax,l,numfields; char statbuf[K]; char delim[2],*nextval; FILE *cpuinfo_fd; double loop_delta_1000,loop_delta_1000000,loop_ratio; snprintf(new_nanotimer->name,K,"cpu cycle counter nanotimer"); if(verbose == V_TIMER){ printf("#========================================================================\n"); printf("# Initializing timer %s\n",new_nanotimer->name); } new_nanotimer->timer = nanotimer_rdtsc; /* * First we extract the number of nanoseconds per cycle (required!) * from /proc/cpuinfo. AFAICT, this is the easiest/only way to get * it, although I don't know about its precision. This is fairly * standard parsing code (ripped from xmlsysd). */ delim[0] = ':'; /* separator */ delim[1] = (char) NULL; /* string terminator */ cpuinfo_fd = fopen("/proc/cpuinfo","r"); while(-1){ /* Normal EOF causes break from while loop */ if((fgets(statbuf,K,cpuinfo_fd) == NULL)) break; if(strncmp(statbuf,"cpu MHz",7) == 0) { nextval = strtok(statbuf,delim); /* first field skip */ nextval = strtok((char *)NULL,delim); /* second field is it */ nsec_per_cycle = 1000.0/atof(nextval); break; } } fclose(cpuinfo_fd); if(nsec_per_cycle == 0.0){ fprintf(stderr,"Error: Cannot parse out the cpu MHz from /proc/cpuinfo.\n"); fprintf(stderr,"Cannot use built-in CPU cycle counter as timer.\n"); fprintf(stderr,"Try the -g flag to use gettimeofday instead.\n"); exit(0); } /* * Now we set the key elements of the nanotimer struct and TEST its * call granularity. * * We subtract off the time base to ensure that times of * order seconds (and then some) have resolvable differences * in double precision. This is therefore the initial call to * the cpu cycle timer to extract the initial time base and * MUST precede the use of the timer. We do NOT CHECK to keep * the timer fast! */ asm volatile("rdtsc" : "=a" (ax_first), "=d" (dx_first)); count_first = dx_first; count_first = count_first<<32; count_first += ax_first; /* * To determine the granularity, we just start calling it in a loop until * we've accumlated a few zillion calls, and see how long they took on * average. We also crudely time the loop itself, and make sure that * its timing SCALES WITH SIZE. */ new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Empty loop. The printf(""); seems more reliable than anything * including a nanosleep etc. at ensuring that the test start on a * clock cycle break, yielding the most consistent results. */ printf(""); new_nanotimer->start = nanotimer_rdtsc(); for(i=0;i<1000;i++){ } new_nanotimer->stop = nanotimer_rdtsc(); loop_delta_1000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000 = %f\n",loop_delta_1000); printf(""); new_nanotimer->start = nanotimer_rdtsc(); for(i=0;i<1000000;i++){ } new_nanotimer->stop = nanotimer_rdtsc(); loop_delta_1000000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000000 = %f\n",loop_delta_1000000); loop_ratio = loop_delta_1000000/loop_delta_1000; if(verbose == V_TIMER) printf("# loop_ratio = %f (should be near 1000)\n",loop_ratio); /* * It should get bigger. I don't much care how MUCH bigger */ if(loop_ratio < 10){ fprintf(stderr,"Warning: Empty loop measurement not scaling in nanotimer_rdtsc_init()\n"); fprintf(stderr," The compiler may optimize away empty loops!\n"); } new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Full loop. We start right after a clock tick, we hope, and * we subtract out the loop_delta from above. */ printf(""); new_nanotimer->start = nanotimer_rdtsc(); for(i=0;i<1000000;i++){ new_nanotimer->stop = nanotimer_rdtsc(); } new_nanotimer->delta = new_nanotimer->stop - new_nanotimer->start - loop_delta_1000000; if(verbose == V_TIMER) printf("# nanotimer delta = %f for 1000000 iterations\n",new_nanotimer->delta); new_nanotimer->nsec_granularity = new_nanotimer->delta/1000000.0; if(verbose == V_TIMER){ printf("# %s: granularity = %f nanoseconds\n", new_nanotimer->name,new_nanotimer->nsec_granularity); } } double nanotimer_rdtsc() { unsigned long ax, dx; unsigned long long count; double nanotime; asm volatile("rdtsc" : "=a" (ax), "=d" (dx)); count = dx; count = count<<32; count += ax; count -= count_first; nanotime = (double) count; nanotime *= nsec_per_cycle; /* printf("nanotime = %f\n",nanotime); */ return(nanotime); } /* * This is a portable nanosecond timer. It uses gettimeofday (wall clock * time) with the time of the first call subtracted off to keep intervals * from horribly overflowing the double with irrelevant numbers (causing * a loss of precision). Note that my direct measurements show that * gettimeofday() itself takes about 2 usec to complete. */ void nanotimer_gttod_init(Nanotimer *new_nanotimer) { int i; double loop_delta_1000,loop_delta_1000000,loop_ratio; snprintf(new_nanotimer->name,K,"gettimeofday nanotimer"); if(verbose == V_TIMER){ printf("#========================================================================\n"); printf("# Initializing timer %s\n",new_nanotimer->name); } new_nanotimer->timer = nanotimer_gttod; /* * Now we set the key elements of the nanotimer struct and TEST its * call granularity. * * To determine the granularity, we just start calling it in a loop until * we've accumlated a few zillion calls, and see how long they took on * average. We also crudely time the loop itself, and make sure that * its timing SCALES WITH SIZE. */ new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Empty loop. The printf(""); seems more reliable than anything * including a nanosleep etc. at ensuring that the test start on a * clock cycle break, yielding the most consistent results. */ printf(""); new_nanotimer->start = nanotimer_gttod(); for(i=0;i<1000;i++){ } new_nanotimer->stop = nanotimer_gttod(); loop_delta_1000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000 = %f\n",loop_delta_1000); printf(""); new_nanotimer->start = nanotimer_gttod(); for(i=0;i<1000000;i++){ } new_nanotimer->stop = nanotimer_gttod(); loop_delta_1000000 = new_nanotimer->stop - new_nanotimer->start; if(verbose == V_TIMER) printf("# loop_delta_1000000 = %f\n",loop_delta_1000000); loop_ratio = loop_delta_1000000/loop_delta_1000; if(verbose == V_TIMER) printf("# loop_ratio = %f (should be near 1000)\n",loop_ratio); /* * It should get bigger. I don't much care how MUCH bigger */ if(loop_ratio < 10){ fprintf(stderr,"Warning: Empty loop measurement not scaling in nanotimer_gttod_init()\n"); fprintf(stderr," The compiler may optimize away empty loops!\n"); } new_nanotimer->stop = 0.0; new_nanotimer->delta = 0.0; /* * Full loop. We start right after a clock tick, we hope, and * we subtract out the loop_delta from above. */ printf(""); new_nanotimer->start = nanotimer_gttod(); for(i=0;i<1000000;i++){ new_nanotimer->stop = nanotimer_gttod(); } new_nanotimer->delta = new_nanotimer->stop - new_nanotimer->start - loop_delta_1000000; if(verbose == V_TIMER) printf("# nanotimer delta = %f for 1000000 iterations\n",new_nanotimer->delta); new_nanotimer->nsec_granularity = new_nanotimer->delta/1000000.0; if(verbose == V_TIMER){ printf("# %s: granularity = %f nanoseconds\n", new_nanotimer->name,new_nanotimer->nsec_granularity); } } double nanotimer_gttod() { struct timeval tv_now; double nanotime; /* * This avoids potential precision problems by computing the starting * time as of the first call, and measuring all subsequent times * relative to this. Gets rid of a LOT of seconds. */ if((tv_first.tv_sec == 0) && (tv_first.tv_usec == 0)){ gettimeofday(&tv_first, (struct timezone *) NULL); } gettimeofday(&tv_now, (struct timezone *) NULL); nanotime = (double)(tv_now.tv_sec - tv_first.tv_sec); nanotime += 1.0e-6*(double)(tv_now.tv_usec - tv_first.tv_usec); /* return nanoseconds */ nanotime *= 1.e+9; return(nanotime); } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 7 11:41:12 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 7 Jan 2004 11:41:12 -0500 (EST) Subject: [Beowulf] Need help In-Reply-To: Message-ID: On Wed, 7 Jan 2004, Maikel Punie wrote: > > hallo everyone, > > well for my final project at school i need to build a beowulf, well i'm this > far that all the node (diskless system) boot from the main node (ntfs) and > there config and stuff from there, the main node can see all off his nodes. > > Now my question is whats the next step? How do i make the nodes do the > calculations and stuff for the main node, how do i realy integrate the > parrallel processing? > > Thanks a lot Visit http://www.phy.duke.edu/~rgb and grab the PVM project template under General. Install PVM on all the nodes and configure it to run. Get a PVM book (there is a good one from MIT press). Run the PVM example programs. OR do the same thing for MPI. OR learn to write raw socket code for doing IPC's. AND consider buying e.g. Ian Foster's book on parallel programming (available online for free in the short run), linked to http://www.phy.duke.edu/brahma under one of the resources links. Two final resources: Subscribe to Cluster World Magazine for more getting started code and tips and so forth. You can get three free trial issues if you live in the US by visiting its website. Subscribe to Linux Magazine ditto -- Forrest Hoffman has been writing an excellent column on cluster computing there for years now, and recent issues (within the last year) have had complete getting started in MPI instructions, for example, with sample code. HTH, rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ken_i_m at fatair.net Wed Jan 7 12:12:54 2004 From: ken_i_m at fatair.net (ken_i_m at fatair.net) Date: Wed, 7 Jan 2004 10:12:54 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <1073436271.15823.30.camel@protein.scalableinformatics.com> Message-ID: <20040107171254.GB4848@hyper> On Wed, Jan 07, 2004 at 11:18:33AM -0500, Robert G. Brown (rgb at phy.duke.edu) wrote:These interrupts and context switches are quite "expensive", > and one is tempted to get off into a discussion of the kernel design and > upper/lower half handlers and ways the system tries to bundle up its > work to minimize impact on running processes, but it would be a moderate > digression. Whereas this digression is merely brief. :-) -- Ken Dyke "Bits at the speed of light" _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Wed Jan 7 11:57:30 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed, 7 Jan 2004 08:57:30 -0800 Subject: [Beowulf] Need help Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C012@orsmsx402.jf.intel.com> From: Maikel Punie; Wednesday, January 07, 2004 7:16 AM > > well for my final project at school i need to build a beowulf, well i'm > this > far that all the node (diskless system) boot from the main node (ntfs) and > there config and stuff from there, the main node can see all off his > nodes. Congratulations! > Now my question is whats the next step? How do i make the nodes do the > calculations and stuff for the main node, how do i realy integrate the > parrallel processing? Try running an MPI or PVM-based program. There are many such codes out there, the biggest problem, though is "how accessible (understandable) are the concepts to those you will present these results too?" While somewhat mundane in terms of parallel processing, try running POV-Ray in serial and parallel modes. For standard POV-Ray, try http://www.povray.org/ and for a parallel version, try http://pvmpov.sourceforge.net/ Alternatively, look at http://chromium.sourceforge.net/ -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From maikel_punie at hotmail.com Wed Jan 7 12:02:59 2004 From: maikel_punie at hotmail.com (Maikel Punie) Date: Wed, 07 Jan 2004 18:02:59 +0100 Subject: [Beowulf] Need help Message-ID: >Try running an MPI or PVM-based program. There are many such codes out >there, the biggest problem, though is "how accessible (understandable) >are the concepts to those you will present these results too?" well thats not my problem :) nah i know it will be hard to explain it, but its a fun project. Maikel _________________________________________________________________ MSN Messenger je favoriete software van 2003? Stem op http://pcmagazine.zdnet.be/pvhj.cfm _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Jan 7 13:41:58 2004 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 07 Jan 2004 11:41:58 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: References: Message-ID: <3FFC52F6.6080503@lanl.gov> Robert G. Brown wrote: > A Brief Digression on Benchmark Timing > > (Yes, I lie like a rug. It is not brief. If you already know about > timing, or don't care, hit 'd' now...:-) Thank you for providing this excellent article. I'd just like to add a few details on fine grained timings, which are indeed useful in tracking down some performance quirks... > gettimeofday is actually a mediocre timer for nearly all > microbenchmarking purposes While gettimeofday() is quite portable as a function call, its timing accuracy is not. On some machines (e.g. Linux PCs) the times are reliable to within a few microseconds. On other systems, true granularity can be 1000 microseconds, apparently related to the scheduler frequency. This seriously undermines the portability of fine grained timings done with gettimeofday(). > A much better timer is available on nearly > all intel/amd cpus. Indeed -- and thank you for your code examples. One minor simplifying comment (I learned this recently) is that GNU C has the ability to read the full 64-bit time stamp counter as follows: unsigned long long tsc; asm volatile ("rdtsc": "=A" (tsc)); where the capitalized "=A" tells gcc that the operand constraints of RDTSC involve both a and d registers, as in: "A -- Specifies the a or d registers. This is primarily useful for 64-bit integer values (when in 32-bit mode) intended to be returned with the d register holding the most significant bits and the a register holding the least significant bits." --http://gcc.gnu.org/onlinedocs/gcc-3.3.2/gcc/Machine-Constraints.html Regarding the RDTSC instruction, those interested in more detail can consult http://cedar.intel.com/software/idap/media/pdf/rdtscpm1.pdf but I'd like to add a few observations: (1) Pentiums execute instructions out-of-order. Although RDTSC instruction will read the 64-bit Time Stamp Counter which advances at the CPU's clock frequency (~3GHz these days), the meaning of the count might not be exactly what you expected. For maximum reliability, one can issue a serializing instruction such as CPUID before RDTSC, at the cost of about 300 clock cycles (CPUID can take variable amount of time to execute). BTW, just doing the RDTSC and moving its result to other registers takes about 86 cycles on a Pentium III. Intel's document suggests some compensation techniques, but in many cases 300 cycle resolution (about 100 nanoseconds) is already OK, so CPUID is optional. Also, older Pentiums (plain or MMX) do not need CPUID. (2) CPU time stamp counters exist on other processors, e.g. PowerPC and Alpha. While 64-bit counters won't overflow for centuries, machines with 32-bit counters (Alpha) experience counter rollovers every few seconds. This must be accounted for in interpreting the results. (3) Thermal management on recent Pentiums duty cycles the clock whenever the processor is in danger of overheating. The clock frequency does not slow down, but the CPU core sees normal/stopped clock with a thermally controlled duty cycle. This begs the question: Is the time stamp counter affected? I hope not, but I do not know for sure. There are reasons for both approaches: Counting either only the cycles seen by the core, or counting all clock cycles; I do not know which way Intel's engineers went. (4) SMP processors present a special challenge in using CPU time stamp counters for precision timings, because the timing process can migrate from one CPU to another, where the counter might have a different offset. Fortunately, Linux checks this on startup, and synchronizes CPU time stamp counters when booted. Provided that all CPUs count the same clock cycles, they should remain synchronized, so process migration need not affect timings. However, this can be a serious problem if the operating system does not ensure that CPU counters are closely synchronized, or if the individual CPU counters drift apart due to some reason (e.g. counting thermally duty-cycled clock, as in (3) above). (5) Precision timings are sometimes available from other sources, e.g. some high end network cards (Quadrics elan_clock() returns nanoseconds). Also, MPI specifies portable MPI_Wtime() and MPI_Wtick() calls, but again, their actual performance varies by MPI implementation and platform (e.g. some MPI implementations use gettimeofday() to return MPI_Wtime() in seconds). (6) All of the above applies to measurements of *local* time only. The notion of *global* time across a cluster is much fuzzier than it ought to be, even though it should be technically possible to keep node clocks synchronized to microseconds across a cluster. NTP helps, but even better synchronization is needed. One approach to collecting fine grained global timings is to work with local timers, then correct their offsets in a post-processing step (this can be tricky, since estimating individual offsets assumes that there were enough common timed events during the run). Here is my challenge to the Beowulf and MPI communities: Achieve microsecond-level *global* synchronization and tracking of all system clocks within a cluster, for application use via portable calls gettimeofday() and/or MPI_Wtime(). > Absolutely. In fact, one of the "fun" things about microbenchmarking is > that a good microbenchmarking suite and/or a set of good vmstat-like > tools can really help you understand and "see" the actual costs of stuff > like this. Josip Loncaric (on this list) worked very hard some years > ago to run down a moderately hienous intermittant latency hit in the > (IIRC) TCP networking stack, for example. Every now and then instead of > a predictable relatively short packet latency a really long (again IIRC, > 2 msec or the like compared to timings on the order of tens of usec) hit > would appear. In Linux kernels 2.0.x, this periodic TCP performance hit was 20 ms (2000:1), in 2.2.x it was 10 ms (1000:1), but Linux 2.4.x finally seems to have fixed this problem without my patches. While my old web page at ICASE went away, history of this Linux TCP quirk can be found at: http://jl-icase.home.comcast.net/LinuxTCP.html http://jl-icase.home.comcast.net/LinuxTCP2.html http://jl-icase.home.comcast.net/LinuxTCP-patches.html Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gropp at mcs.anl.gov Wed Jan 7 13:00:04 2004 From: gropp at mcs.anl.gov (William Gropp) Date: Wed, 07 Jan 2004 12:00:04 -0600 Subject: [Beowulf] Need help In-Reply-To: References: Message-ID: <6.0.0.22.2.20040107115455.0256dd30@localhost> At 09:16 AM 1/7/2004, Maikel Punie wrote: >hallo everyone, > >well for my final project at school i need to build a beowulf, well i'm >this far that all the node (diskless system) boot from the main node >(ntfs) and there config and stuff from there, the main node can see all >off his nodes. > >Now my question is whats the next step? How do i make the nodes do the >calculations and stuff for the main node, how do i realy integrate the >parrallel processing? At the risk of sounding a tiny bit commercial, you should have a look at the second edition of the Beowulf Cluster Computing with Linux (http://mitpress.mit.edu/catalog/item/default.asp?sid=DD198EEC-B4B8-48BD-B728-C28DF093FBD4&ttype=2&tid=9947). The second edition has a great deal of new material on managing and using clusters, including using other approaches to parallelism besides MPI and PVM and using existing software libraries instead of writing your own. Royalties go to a fund to support student travel to IEEE Cluster. Bill _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 7 13:10:45 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 07 Jan 2004 10:10:45 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <1073436271.15823.30.camel@protein.scalableinformatics.com> Message-ID: <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> At 11:18 AM 1/7/2004 -0500, Robert G. Brown wrote: >A Brief Digression on Benchmark Timing >Suffice it to say that if you time LOTS of 100 microsecond loops >separately in a continuous process and look at the results, you're >likely to see the bimodal distribution Joe refers to (or even a fully >multimodal distribution) because SOME of those intervals will be >interrupted and the interrupt/context switch will be "timed" along with >the swapped out loop by the wall-clock timer. This can be somewhat >ameliorated by adding a usleep right before each timing loop iteration, >as the kernel will often take advantage of a task pause to complete >pending interrupts. However, it alas isn't a magic bullet because not >all interrupts are "pendable". That very distribution might be useful as a diagnostic tool, by the way... >This nanotimer is FAR more suitable for use in microbenchmarking -- >relatively small loops where one devoutly hopes to avoid getting nailed >by a context switch as one is interested in "how many floating point >multiplies can the system perform on numbers in its registers/L1 >cache/L2 cache/memory per second". A loop with a few microseconds worth >of operations, fairly likely not to be interrupted if you start >immediately after a timer click, can yield < 1% accuracy. Multimodality >in a distribution of such short loop timings yields "interesting" >information about the system's CPU, cache, memory architecture, and >their interaction with the kernel. This is along the lines of what that nifty benchmark (the name of which eludes me right now) that produces a graph of speed vs computation size does.. You can "see" the effects of cache etc as the problems space gets bigger. (Edward Tufte would be pleased with the results) >I've wrapped both of the timers up in a single file in a somewhat >portable form. Because there can be a LOT of clocks in a second >(several billion, obviously) the nanotimer is a bit complex -- >on the first call it saves the base clock registers as a baseline that >it subtracts from subsequent calls BEFORE converting the results to a >double precision nanosecond count. Done in the wrong order, you can end >up with a significant precision loss subtracting two large numbers to >get a very small number, hence the runaround. I'm pretty sure it is >done right at this point. I attach the timer source below for anyone >who cares. But, except for diagnostic purposes, code tuning, research into the performance of CPU architectures, and the like, are such nanoscale measurements really useful. The real figure of merit is "how long does it take to run my problem" and that's a wall clock time issue. Given a suitably scalable problem, is it worth spending much time making the code 10% faster, or are you better off just adding 10% more CPU resources? > rgb > > > You wouldn't even need another process taking up time, the kernel will > > happily do this for you with some of its housekeeping chores (bdflush, > > kswapd, ...). Servicing lots of interrupts is a good way to slow things > > down, as are doing many small page allocations where one large one would > > suffice, etc. > >Absolutely. In fact, one of the "fun" things about microbenchmarking is >that a good microbenchmarking suite and/or a set of good vmstat-like >tools can really help you understand and "see" the actual costs of stuff >like this. Josip Loncaric (on this list) worked very hard some years >ago to run down a moderately hienous intermittant latency hit in the >(IIRC) TCP networking stack, for example. Every now and then instead of >a predictable relatively short packet latency a really long (again IIRC, >2 msec or the like compared to timings on the order of tens of usec) hit >would appear. This sort of nondeterministic latency is the bane of high performance computing, particularly for those of us with real-time or embedded applications. In these, you typically need to have an absolute bound on maximum time. >Sending lots of small packets is "bad". Doing lots of malloc is "bad". >Doing any sort of interrupt is bad, in part because the system has a >tendency to swap in waiting/competing tasks (if any) when the interrupt >returns and of course because of the relatively high cost of the >interrupt itself. Lots of system calls can have the same effect -- >select, for example, has about the same cost as gettimeofday, and AFAICT >nanosleep is really usleep -- the ONLY way I've found to "pause" for a >nanoscale-specified time is to use a nanotimer-based polling loop. It might be "bad" in the overall performance sense, but it might be "good" in the reduction of latency variability standpoint. If you had a very tightly coupled parallel stream with lots of barrier synchronization, the overall problem might actually run faster with a longer, but more consistent, or more fine grained, latencies, rather than something that usually runs real fast, but occasionally takes a lot longer. Things like heaps and garbage collection are notorious for this kind of thing. >FORTUNATELY, most of this isn't horribly difficult to understand or to >avoid in actual code, and FORTUNATELY most people really are interested >in macroscopic wall-clock completion times of large scale code, where >"date" at the shell is accurate enough to get decent timings. Where it >makes a difference is in optimizing core loops or understanding how you >spent all this money on a 3 GHz CPU system that IN PRINCIPLE has a 6 >GFLOPS peak but in practice you are lucky to get 1 GFLOPS on a good day >with a tail wind. Folks who are squeezing out the last operation per clock cycle typically use external measurement equipment (i.e. a logic analyzer or oscilloscope), because you wouldn't want to waste any precious cycles on internally instrumenting the code. We did a substantial amount of debugging on a loosely coupled cluster of DSPs just looking at the supply current to the processors on a scope, since instantaneous power consumption is very tightly tied to processor utilization. You can see the end of each pass of butterflies in the FFT (very high current) as you set up the next pass(low current), and we could see interrupt handling (low current) at inopportune times causing context switching in the middle of a dense pipelined code section (because we hadn't masked the interrupt). >[Parenthetical Aside: Youngsters don't know HOW lucky you are to get >that 1 GFLOPS in a $1000 system, of course. Suffice it to say that as >of a few years ago, your desktop would have been classified as a weapon >for purposes of export, and sending one outside of the US would have >taken a special license, and a decade ago a GFLOPS system (even a >cluster) would have cost order of $1 million. Gotta love Moore's Law >(and the fact that Our Government, helped enormously by the beowulf >movement, finally wised up to Moore's Law and moved any remaining >weapons boundaries out so far that they are invisible).] Those non-exportable video games with Motorola 68000's were "arms" or "munitions", or even "dual use technologies" , not "weapons"... Such distinctions are important to the folks at Commerce and State. Besides, being an "international arms dealer" has a much better sound to it than "gun runner". Then too, such formerly silly rules helped create news stories about "bad guys" building immense clusters from networked Playstation 2s. I suspect nobody's ever successfully done this; at least to a practical state, running any sort of useful parallel processing. There was a lot of interest in it for a while, and if you go back through the archives, you can probably find it. >James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ken_i_m at fatair.net Wed Jan 7 14:02:07 2004 From: ken_i_m at fatair.net (ken_i_m at fatair.net) Date: Wed, 7 Jan 2004 12:02:07 -0700 Subject: [Beowulf] Need help In-Reply-To: <6.0.0.22.2.20040107115455.0256dd30@localhost> References: <6.0.0.22.2.20040107115455.0256dd30@localhost> Message-ID: <20040107190207.GC5273@hyper> On Wed, Jan 07, 2004 at 12:00:04PM -0600, William Gropp (gropp at mcs.anl.gov) wrote: > the second edition of the Beowulf Cluster Computing with Linux Dang, didn't know the 2nd ed was coming out. Bought the 1st not 2 months ago... -- Ken Dyke "Bits at the speed of light" _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From twhitcomb at apl.washington.edu Wed Jan 7 15:34:36 2004 From: twhitcomb at apl.washington.edu (Timothy R. Whitcomb) Date: Wed, 7 Jan 2004 12:34:36 -0800 (PST) Subject: [Beowulf] beostatus not showing master node Message-ID: Any ideas why the "beostatus" program would show only slave nodes for a while (several months) then display the master node (node -1) for a while (another several months) and then go back to displaying only the slave nodes again with no obvious changes in the software? TRW _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From twhitcomb at apl.washington.edu Wed Jan 7 15:32:12 2004 From: twhitcomb at apl.washington.edu (Timothy R. Whitcomb) Date: Wed, 7 Jan 2004 12:32:12 -0800 (PST) Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) Message-ID: First of all, thanks to everyone who responded to my first message. I received many helpful suggestions to test, which I have documented here. As a quick review of my (still unsolved) situation, we are running a weather model on a 5-node, 10-processor Scyld Beowulf cluster. As long as we use a single processor per machine, performance is fine - but when we use both processors on a single node then the node freezes after a variable length of time (sometimes minutes, sometimes hours). The suggestions I received are summarized as I have tested them: -Make sure kernel is compiled for athlon architecture > Kernel is indeed compiled for Athlon and SMP -Try 1 IDE disk/cable > There is only 1 disk/cable in the slave nodes -Manually ventilate cases > no change in results when cases were opened up and fanned -Replace NIC cables > replaced all NIC cables with no change in results -Tyan 2466 motherboards have stability issues > This is not a 2466 motherboard -Check with the Navy and see if this is run with SMP > The Naval Research Lab runs the model using SMP machines and a Linux cluster (though not necessarily Scyld) -Check thermal problems > no change -Flash update the BIOS > no change, BIOS updated to most recent version -Run CPUBURN for several days > ran over the weekend on all nodes, system seemed stable -Try new network drivers > Updated network drivers to Becker's most recent 3c59x for the dual 3Com 3c982 Server Tornado built-in Ethernet, but with no apparent change The suggestions that I have not been able to try as yet: -Try upgrading the kernel and bproc package Since this is a Scyld system purchased from a retailer, the kernel has all sorts of patches in it. Can anyone point me to anywhere detailing how to do a kernel upgrade on one of these systems? I've built kernels before but usually only using the kernel.org sources. I'd also like to know if it is possible for the cluster interconnects to be "too slow" for the software - there is a 100BaseT ethernet between the nodes but there is a _lot_ of data being passed around. Something else I have noticed is that if I run a 2-processor run on 2 nodes, it goes fairly quickly. However, if I switch it to 2 processors on 1 machine, it stalls for very long periods of time. >From the memory usage, it looks like there's quite a bit in the swap memory (there's 512MB/node of RAM and 1GB of swap) but there is little to no CPU access on either processor for very long periods of time (on the order of several minutes and up) whereas in the 2-node case there are brief pauses, but CPU usage jumps up to 100% fairly quickly. On the 1-node case, one processor will go up a little but then fall back down quickly. Are there any other ideas as to what would be causing these nodes to freeze up? Thanks again for all the help I've received. Tim Whitcomb _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 7 16:04:43 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 07 Jan 2004 16:04:43 -0500 Subject: [Beowulf] beostatus not showing master node In-Reply-To: References: Message-ID: <1073509483.4697.8.camel@pel> On Wed, 2004-01-07 at 15:34, Timothy R. Whitcomb wrote: > Any ideas why the "beostatus" program would show only slave nodes for a > while (several months) then display the master node (node -1) for a while > (another several months) and then go back to displaying only the slave > nodes again with no obvious changes in the software? What version of Scyld are you running? You may want to contact Scyld and see if they have a newer version. I recall this problem and also recall fixing it when I was still at Scyld. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From twhitcomb at apl.washington.edu Wed Jan 7 16:11:23 2004 From: twhitcomb at apl.washington.edu (Timothy R. Whitcomb) Date: Wed, 7 Jan 2004 13:11:23 -0800 (PST) Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: <1073509387.4697.5.camel@pel> References: <1073509387.4697.5.camel@pel> Message-ID: No, I'm not using myrinet. It's a Scyld 28cz4 with 100BaseT interconnects, dual-processor Tyan AMD boards. TRW On Wed, 7 Jan 2004, Jag wrote: > On Wed, 2004-01-07 at 15:32, Timothy R. Whitcomb wrote: > > First of all, thanks to everyone who responded to my first message. I > > received many helpful suggestions to test, which I have documented > > here. > > > > As a quick review of my (still unsolved) situation, we are running a > > weather model on a 5-node, 10-processor Scyld Beowulf cluster. As > > long as we use a single processor per machine, performance is fine - > > but when we use both processors on a single node then the node freezes > > after a variable length of time (sometimes minutes, sometimes hours). > > Unfortunately I missed your first post. Are you using myrinet? I seem > to recall some past problems with two MPI jobs on one host running myri > on Scyld systems. > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 7 16:03:07 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 07 Jan 2004 16:03:07 -0500 Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: References: Message-ID: <1073509387.4697.5.camel@pel> On Wed, 2004-01-07 at 15:32, Timothy R. Whitcomb wrote: > First of all, thanks to everyone who responded to my first message. I > received many helpful suggestions to test, which I have documented > here. > > As a quick review of my (still unsolved) situation, we are running a > weather model on a 5-node, 10-processor Scyld Beowulf cluster. As > long as we use a single processor per machine, performance is fine - > but when we use both processors on a single node then the node freezes > after a variable length of time (sometimes minutes, sometimes hours). Unfortunately I missed your first post. Are you using myrinet? I seem to recall some past problems with two MPI jobs on one host running myri on Scyld systems. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wade.hampton at nsc1.net Wed Jan 7 16:36:23 2004 From: wade.hampton at nsc1.net (Wade Hampton) Date: Wed, 07 Jan 2004 16:36:23 -0500 Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: References: <1073509387.4697.5.camel@pel> Message-ID: <3FFC7BD7.3020402@nsc1.net> Timothy R. Whitcomb wrote: >No, I'm not using myrinet. It's a Scyld 28cz4 with 100BaseT >interconnects, dual-processor Tyan AMD boards. > I too missed your first post, but this thread reminded me of problems I had 6 or so months ago. We had some node hangs with Scyld our cluster, however we were using Tyan dual XEON boards, not AMD. We also had 28cz4. We were using the 100T interface (not the GB). A reboot of the node fixed the problem. We've had problems with this cluster recently, but it is now using the GB interfaces and I have not seen this problem. (Note: getting GB working in this config was a PITA.) Is the node locking up or is it just losing networking? Can you attach monitors to the nodes and see if it is still running when it "locks up"? My head node (same Tyan dual MB as my nodes, hyperthread enabled), would sometimes lose the eepro100 100T connection (outside world), but the e1000 (cluster) interface kept working. However, I would still have to reboot the head node. This would happen every few months initially, but then happened almost every day. I had to install a PCI NIC and disable the onboard 100T interface. It worked fine for months after that (until I lost my RAID -- another story). -- Wade Hampton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 7 16:53:21 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 07 Jan 2004 13:53:21 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFC52F6.6080503@lanl.gov> References: Message-ID: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> A >Here is my challenge to the Beowulf and MPI communities: Achieve >microsecond-level *global* synchronization and tracking of all system >clocks within a cluster, for application use via portable calls >gettimeofday() and/or MPI_Wtime(). Is added hardware on each node legal along with distributing some periodic time mark (like a 1 pulse per second from a GPS receiver)? All you need is a free running counter which can be latched in an atomic operation, driven from a clock of suitable speed (for submicrosecond timing, the clock must be >1MHz), derived from the clock you wish to track. There must be something suitable on the run of the mill PC these days. I haven't looked in detail at this recently, but, on an original PC which used an 8253 timer chip for a number of purposes, one could do this fairly easily. Perhaps the generally unused "game port" on the mobo might work for this. One could certainly build a very inexpensive unit with a USB interface to do what's needed. One of the inexpensive USB data acquisition pods could probably do it, although Linux support seems to be a bit spotty on these, and you'd need to cobble up an interface from the processor board clock to the pod. I'm looking into seeing if the IrDA interface available on many mobos these days might provide a convenient way to distribute the reference time marks (optical, diffuse coupling) With hardware that happened to support this kind of thing (i.e. a DSP with a specialized peripheral interface chip that had UARTs, timers, etc. on it), we were able to synchronize multiple processors and their associated analog to digital converters by this technique to substantially better than 100 nanoseconds absolute. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 7 16:29:59 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 07 Jan 2004 16:29:59 -0500 Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: References: <1073509387.4697.5.camel@pel> Message-ID: <1073510998.4697.19.camel@pel> On Wed, 2004-01-07 at 16:11, Timothy R. Whitcomb wrote: > No, I'm not using myrinet. It's a Scyld 28cz4 with 100BaseT > interconnects, dual-processor Tyan AMD boards. Hm.. I believe they have 29cz out now. If nothing else, that should solve the beostatus problem. (The beostatus problem is actually a bug that's potentially triggered based on the date/time) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rodmur at maybe.org Wed Jan 7 18:25:56 2004 From: rodmur at maybe.org (Dale Harris) Date: Wed, 7 Jan 2004 15:25:56 -0800 Subject: [Beowulf] pvm and bproc 4.0 Message-ID: <20040107232556.GR11808@maybe.org> Thought I would try this list to see if anyone has some possible insights. Basically I'm trying to get PVM to work with bproc 4.0, but having some problems. Initially I was a having some little problems with some node naming conventions changing with bproc 4.0, but I've worked around those problems (instead of identifying nodes like .1 for gethostbyname() they use n1). However, the problem I've ran into now is that I can get PVM to work with bproc 4.0 if I run PVM as root, but it doesn't work for normal users. Which is kind of a problem. It appears that pvm isn't able to access a socket, that it should be able to do. So I see errors like: [t80040000] 01/07 18:30:28 n-1 (192.168.2.1:33095) BEOSCYLD 3.4.4 [t80040000] 01/07 18:30:28 ready Wed Jan 7 18:30:28 2004 [t80000000] 01/07 18:30:32 stderr at n40: /proc/self/fd/3: /proc/self/fd/3: Permission denied [t80000000] 01/07 18:30:32 stdout at n40: EOF fd3 points to a Unix socket. Anyone have any insights? PVM isn't supported like it used to be, unfortunately. -- Dale Harris rodmur at maybe.org /.-) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Wed Jan 7 19:18:46 2004 From: josip at lanl.gov (Josip Loncaric) Date: Wed, 07 Jan 2004 17:18:46 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> Message-ID: <3FFCA1E6.9070905@lanl.gov> Jim Lux wrote: > Josip Loncaric wrote: > >> Here is my challenge to the Beowulf and MPI communities: Achieve >> microsecond-level *global* synchronization and tracking of all system >> clocks within a cluster, for application use via portable calls >> gettimeofday() and/or MPI_Wtime(). > > > Is added hardware on each node legal along with distributing some > periodic time mark (like a 1 pulse per second from a GPS receiver)? I do not think this would be required, especially on clusters which already have a high performance network. These days, 1/2 round trip time can be as low as 2.4 microseconds (Quadrics). Even Fast Ethernet may prove sufficient (over time) if the delays are not too noisy. BTW, the objective here is to support parallel timings within a cluster, so "global time" is still "local to this cluster only", although one may want to run NTP on the head node to stay close to the official time. Back when VXIbus was introduced, I really liked its extra trigger lines which could synchronize multiple instruments within a few nanoseconds, but it is unlikely that people would distribute clock that way in a cluster. Despite various software/hardware network layers and longer distances involved, it should be possible to get microsecond synchronization through the network by gradually tweaking local clock offset and drift... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 7 19:35:36 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 07 Jan 2004 16:35:36 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFCA1E6.9070905@lanl.gov> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: >>Josip Loncaric wrote: >> >>>Here is my challenge to the Beowulf and MPI communities: Achieve >>>microsecond-level *global* synchronization and tracking of all system >>>clocks within a cluster, for application use via portable calls >>>gettimeofday() and/or MPI_Wtime(). >> >>Is added hardware on each node legal along with distributing some >>periodic time mark (like a 1 pulse per second from a GPS receiver)? > >I do not think this would be required, especially on clusters which >already have a high performance network. These days, 1/2 round trip time >can be as low as 2.4 microseconds (Quadrics). Even Fast Ethernet may >prove sufficient (over time) if the delays are not too noisy. BTW, the >objective here is to support parallel timings within a cluster, so "global >time" is still "local to this cluster only", although one may want to run >NTP on the head node to stay close to the official time. > >Back when VXIbus was introduced, I really liked its extra trigger lines >which could synchronize multiple instruments within a few nanoseconds, but >it is unlikely that people would distribute clock that way in a >cluster. Despite various software/hardware network layers and longer >distances involved, it should be possible to get microsecond >synchronization through the network by gradually tweaking local clock >offset and drift... Most of the more recent implementations of NTP clients can do this sort of thing, sending messages both ways across the link to measure the latency (and latency distribution) There's a fairly big literature on synchronizing clocks over noisy, unreliable, non-deterministic communication links. (Some of the basic theory is tied into what's called the "Byzantine Generals" problem.) I wasn't so much advocating synchronization in an absolute sense, but that one can accurately synchronize a cluster to much better than a microsecond, if one can distribute a common time mark to all processors in some way. A really crummy communications network, plus a single time hack, can do it. In some ways, I guess this could be considered a classic barrier synchronization scheme. Conceptually, the time synchronization process in all threads gets to the barrier and waits for the pulse to release them. In my own implementation, part of the work was done by moderately specialized hardware. While reasonably deterministic in their execution time, even DSPs weren't good enough. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 7 19:46:46 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 7 Jan 2004 19:46:46 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 7 Jan 2004, Jim Lux wrote: > This is along the lines of what that nifty benchmark (the name of which > eludes me right now) that produces a graph of speed vs computation size > does.. You can "see" the effects of cache etc as the problems space gets > bigger. (Edward Tufte would be pleased with the results) Well, that's one of the things cpu_rate does (the package from which these timers were lifted. It run one of a variety of stream-like tests but uses malloc instead of a hard data allocation so that one can vary the vector size for vector tests. In fact, all the stream tests are embedded in cpu_rate, even though they are more correctly viewable as memory tests, or maybe integrated architecture (CPU and memory together) tests. cpu_rate also contains the old savage benchmark and a stream extension that includes division (division is typically MUCH SLOWER than multiplication and addition, making it "unsexy" to include in a benchmark designed to show how fast your system is, although I find that my own code has plenty of divisions in it:-). It doesn't provide a graph per se -- it provides a bogomegarate. However, there is a script included that will usually generate a file of rates at a range of vector sizes that can then easily be graphed. All available on brahma or my personal website. lmbench is also a very good microbenchmark, by the way -- arguably better than cpu_rate. Larry has been in the benchmark business longer and Carl Staelin also is very smart. For a while, though, Larry had some "odd" rules on the benchmark (designed not to close the source but rather to keep vendors from corrupting its application to their systems). I just prefer a straight GPL, so I developed a VERY old "FLOPS" benchmark suite of mine (dating back to the late 80's or early 90's -- I think I originally wrote it using DeSmet C if any old PC users recognize it:-) into a microbenchmark I could use and publish without care. > But, except for diagnostic purposes, code tuning, research into the > performance of CPU architectures, and the like, are such nanoscale > measurements really useful. The real figure of merit is "how long does it > take to run my problem" and that's a wall clock time issue. Given a > suitably scalable problem, is it worth spending much time making the code > 10% faster, or are you better off just adding 10% more CPU resources? Agreed. Although ATLAS is a pretty clear demonstration that sometimes it's a matter of 200-300% faster. Algorithm and tuning CAN matter a lot, and it is worth learning "enough" to know whether that is likely to be the case for your code. Just something like inverting a value so you can multiply it many times inside a core loop instead of divide it many times inside a core loop can make a BIG difference in core loop timing... > Folks who are squeezing out the last operation per clock cycle typically > use external measurement equipment (i.e. a logic analyzer or oscilloscope), > because you wouldn't want to waste any precious cycles on internally > instrumenting the code. We did a substantial amount of debugging on a > loosely coupled cluster of DSPs just looking at the supply current to the > processors on a scope, since instantaneous power consumption is very > tightly tied to processor utilization. You can see the end of each pass of > butterflies in the FFT (very high current) as you set up the next pass(low > current), and we could see interrupt handling (low current) at inopportune > times causing context switching in the middle of a dense pipelined code > section (because we hadn't masked the interrupt). Now THAT'S serious performance optimization. > Those non-exportable video games with Motorola 68000's were "arms" or > "munitions", or even "dual use technologies" , not "weapons"... Such > distinctions are important to the folks at Commerce and State. Besides, > being an "international arms dealer" has a much better sound to it than > "gun runner". Then too, such formerly silly rules helped create news > stories about "bad guys" building immense clusters from networked > Playstation 2s. I suspect nobody's ever successfully done this; at least to > a practical state, running any sort of useful parallel processing. There > was a lot of interest in it for a while, and if you go back through the > archives, you can probably find it. They were just worried that bad persons/countries would use the systems to simulate critical nuclear cores so that they could build small/efficient devices without having to actually test them. Correctly worried -- I imagine that my laptop would work just fine to simulate at least some aspects of a nuclear chain reaction, given the right code. I personally never worried much about this. The nuclear bomb is 1940's technology. Give me access to U235 or plutonium and I'll build one. Moderately efficient non-implosion uranium bombs are trivial to build -- literally garage stuff. Implosion/plutonium bombs are more difficult -- explosive lenses and so forth -- but c'mon, this is 2004. Saddam had functional test pits at the end of the FIRST gulf war -- he just didn't have the nuclear grade material (from what was reported at the time). Conventional explosives are pretty well known by now. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 08:01:16 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 08:01:16 -0500 (EST) Subject: [Beowulf] SMP Nodes [still] Freezing w/ Scyld (follow-up, long) In-Reply-To: Message-ID: On Wed, 7 Jan 2004, Timothy R. Whitcomb wrote: > Something else I have noticed is that if I run a 2-processor run on 2 > nodes, it goes fairly quickly. However, if I switch it to 2 > processors on 1 machine, it stalls for very long periods of time. > >From the memory usage, it looks like there's quite a bit in the swap > memory (there's 512MB/node of RAM and 1GB of swap) but there is little > to no CPU access on either processor for very long periods of time (on > the order of several minutes and up) whereas in the 2-node case there > are brief pauses, but CPU usage jumps up to 100% fairly quickly. On > the 1-node case, one processor will go up a little but then fall back > down quickly. > > Are there any other ideas as to what would be causing these nodes to > freeze up? Thanks again for all the help I've received. I'm assuming that you've looked at and for memory leaks or just plain running out of memory. It is difficult for me to think offhand why a dual CPU job would leak and a single CPU job wouldn't (except leak faster and maybe crash sooner. It is pretty easy to believe that one job uses (at some point) 60% of available memory and the second job (at the same point) uses 60% of available memory -- or whatever -- enough to push it over a critical threshold. IF your system ever really runs out of memory it will definitely die shortly after the kernel requests a page and can't get it. Often die horribly. If this were 1997 or 1998, I'd also suspect an SMP kernel deadlock on some interrupt controlled resource. In 2003 that's a bit tougher to imagine, but it still isn't impossible to imagine either a memory management issue with your particular kernel snapshot (the one in the Scyld version you bought) or something LIKE a deadlock associated with your particular hardware combination. Over the years, I have very definitely seen certain motherboard/memory combinations that were just poison to linux kernels -- sometimes in common enough usage that they'd get fixed; in a few cases they took their instability with them to their early graves. SMP kernels are generally marginally less stable than UP kernels, although honestly it has been maybe 2-3 years since I last had an issue with any linux kernel at all (and I run lots of SMP systems). Still, two things to try are: a) Try a different version of Scyld, either earlier or later (one with a different kernel). See if problems magically vanish. Yes, even earlier kernels can be more stable -- some problems don't exist, then appear in a particular snapshot but make it through the "stable" selection process to make it out into the world as a plague, then vanish with a later kernel as somebody puts the offending variable or struct or algorithm back in order. Oh, and the issue isn't just kernel -- there have been significant variations in e.g. glibc over the last year plus -- any linked library could be the offender, although some bugs would cause more mundane/recoverable crashes in userspace. b) Try a non-Scyld linux on one or two boxes, e.g. a vanilla RH 9. Run your job to see if it crashes. This gives you the ability to play a LOT more with kernels, including building new stable or unstable kernels, and gives you a base that is in active use with patch feedback on a LOT more platforms and hence a lot less likely to contain unpatched critical bugs. Editorial comment: The ONE THING that I think really hurts packages like Scyld and Mosix is the fact that they tend to significantly lag commercial linux kernel update streams, and the kernel update streams are one of the best of the various reasons to consider commercial linuces in the first place. Kernel updates in the commercial streams tend to fairly rapidly address critical instabilities that sneak into a released kernel snapshot, and they also address the rare but real kernel-level security hole that sneak through ditto. Over the years I've played the roll-your-own kernel game (quite extensively) and although it is fairly straightforward on a per-system basis, it isn't terribly easy to come up with a configuration schema for a good general purpose/portable/installable kernel that can be built "identically" for smp, amd, i386, i686, etc and installed across a LAN AND to keep it regularly rebuilt and updated. In fact it is a lot of work -- work it is worth it to pay somebody else to do for you. It is perhaps a silly vision, but I still dream of the day when linux comes prewrapped to be automagically built, top to bottom, by fully automated tools so that updating a particular package is a matter of dropping in a new e.g. src rpm and doing nothing. I'm hoping that future revisions of yum and/or caosity eventually move it in this direction. Of course, this still begs the question of testing and validation for commercial application and won't remove the raison d'etre of the big commercial linuces, but it will mean that perhaps the smaller commercial linuces like Scyld can lock their offerings more tightly to the current patch levels of the larger linuces. I mean, these are COMPUTERS we're talking about, for God's sake. They run PROGRAMS. PROGRAMS are these nifty things that do work for you in an automated and moderately reproducible way. So why does building stuff still require so much handiwork? rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 09:10:24 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 09:10:24 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 7 Jan 2004, Jim Lux wrote: > >Here is my challenge to the Beowulf and MPI communities: Achieve > >microsecond-level *global* synchronization and tracking of all system > >clocks within a cluster, for application use via portable calls > >gettimeofday() and/or MPI_Wtime(). > > Is added hardware on each node legal along with distributing some periodic > time mark (like a 1 pulse per second from a GPS receiver)? IIRC, we discussed this on list within the last couple of years. There were only two approaches that seemed worth considering -- a network based schema that used ntp or an ntp-like daemon to gradually refine the system local clock to be accurate within a microsecond across a network. This isn't horribly easy on a network with a latency on the order of hundreds of microseconds, but it is doable -- it just takes a lot of packets and complete control of the network. This synchronization won't be accurate to absolute time unless the master clock is, but it would be synchronization. It seems reasonable to do this as a grant funded project. I've got a couple or three student hackers handy that might tackle it if I could get it funded (with a very small grant). The nice thing about this approach is that it could be a fully GPL, fully portable solution that could be built right into e.g. a bproc cluster. Note also that all that is really required in terms of counters is to be able to match the distributed/refined time base to a particular value of the CPU cycle counter. Then a trivial revision of the non-portable CPU-based nanoscale gettimeofday I posted (or an architecture-aware replacement of the archaic gettimeofday library functions) would make this effectively transparent. The second approach is hardware, which seems perfectly "legal" but which requires an additional investment and which pushes one a bit away from the "commodity" ideal and into the "hardware hacker" ideal which is not everybody's cup of tea. GPS is (as was discussed) a good way to get a very accurate time base, with the caveat that GPS signals are very weak (my handheld GPS won't work indoors, or even inside a car, let alone in a basement server room underneat a few hundred tons of steel and concrete). GPS is getting to be VERY cheap -- almost all cell phones now apparently have built in GPS, and for $45 or so I bought a GPS-sync'd clock from Wal-Mart (which won't work inside the house because GPS signals are so weak, sigh). GPS thus seems like something that would actually be fairly tricky to get working inside a lot of server room environments unless an external antenna were used and piped down to the device, which would add to the expense and hassle and be moderately inefficient on a per-system basis. I have my own reasons for liking the design you indicate below, and would like it very much if two PCs separated by tens of kilometers and not even ON a common network could synchronize an internal clock to less than a microsecond, better yet to less than 100 nanoseconds. For this to happen, the whole thing REALLY needs to move onto the motherboard and be tightly integrated with the system design, although a PCI card or USB device both are alternatives that would likely "work", especially if one wrote kernel drivers for them that could do the entire job of reading and setting the internal cycle-counter-to-absolute-time variable as an atomic operation AND actually correct the result for the time it took to do the set. If one could build a hardware device that kept nanosecond-accurate time, it should be possible to write a gettimeofday or inline assembler fragment that is accurate to within a few nanoseconds (the minimal overhead associated with reading the counter and saving the result as a timestamp for later conversion). Perhaps Intel and/or AMD would consider (or are considering) adding a GPS receiver to their future mobo designs. Add a single port on the back for an external antenna, or figure out how to use some of the considerable quantity of metal attached to the system as a default antenna. Sell amplified GPS antennae as an inexpensive add-on. Or ditto on a PCI card, but again one has to find a manufacturer who will make and sell it for $25, or maybe $35, or it won't be worth it to most people and the price then has to rise. Personally I can think of all sorts of benefits to having each and every PC clock in the universe set and locked to real, absolute time within a few nanoseconds, and similarly located in space to within a few meters (those same nanoseconds times c). The little radiotelescope project I've got a student (still, sigh) working on being one of them, but far from the only one. I'd really like to be able to sit at my PC and have it go "happy new year" at PRECISELY the new year, for example. While we are at it sailing around in the blue sky, can we also put an entropy generator on the add-on card? Something that can generate random bits (in parallel) at oh, 32 GHz, buffer them internally, and deliver the entire buffer to memory via DMA on a single interrupt? My systems get SO tired of generating pseudorands...:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 10:44:42 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 08:44:42 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> Message-ID: <3FFD7AEA.4060709@lanl.gov> Jim Lux wrote: > > At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: > >> Despite various software/hardware network >> layers and longer distances involved, it should be possible to get >> microsecond synchronization through the network by gradually tweaking >> local clock offset and drift... > > > Most of the more recent implementations of NTP clients can do this sort > of thing, sending messages both ways across the link to measure the > latency (and latency distribution) I love NTP, but in its current state it does not deliver microsecond level synchronization. I just checked my old cluster (which runs NTP) and the compute nodes are off by 500-25000 microseconds. Actually, most are within 1000 microseconds, but about 20% are off by up to 25000 microseconds. This happens over Fast Ethernet where ntpdate usually reports timing dispersion of 0-20 microseconds (based on 7 ping-pong measurements). In other words, network timing is quite precise, but we'd need about 1000 times better clock synchronization. NTP may need to be streamlined and tuned for clusters... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kalyanigoli at yahoo.com Thu Jan 8 03:35:32 2004 From: kalyanigoli at yahoo.com (kalyani goli) Date: Thu, 8 Jan 2004 00:35:32 -0800 (PST) Subject: [Beowulf] have u encountered this problem Message-ID: <20040108083532.89213.qmail@web20203.mail.yahoo.com> Hi all, Iam new to this mailing list. Iam doing a project related to MPI. when running the testcase programs in mpich, ch_p4 device ,example cpi program iam getting the following error. could u plz tell me how to solve this problem. [kalyani at mpi1 basic]$ mpirun -np 5 systest Process 3 is alive on mpi1 Process 1 is alive on mpi2 Process 0 is alive on mpi1 Process 2 is alive on mpi3 Process 4 is alive on mpi2 p3_2940: p4_error: Timeout in establishing connection to remote process: 0 bm_list_2926: (303.941183) wakeup_slave: unable to interrupt slave 0 pid 2925 bm_list_2926: (303.941283) wakeup_slave: unable to interrupt slave 0 pid 2925 p1_1579: p4_error: net_recv read: probable EOF on socket: 1 p4_1593: p4_error: net_recv read: probable EOF on socket: 1 Iam running this program on three machines. mpi,mpi2,mpi3.Iam able to do ssh on all the machines. Thank you all in advance __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 8 11:29:46 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 8 Jan 2004 08:29:46 -0800 Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results References: Message-ID: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> ----- Original Message ----- Subject: Re: [Beowulf] ...Re: Benchmark results > On Wed, 7 Jan 2004, Jim Lux wrote: > > > >Here is my challenge to the Beowulf and MPI communities: Achieve > > >microsecond-level *global* synchronization and tracking of all system > > >clocks within a cluster, for application use via portable calls > > >gettimeofday() and/or MPI_Wtime(). > > > > Is added hardware on each node legal along with distributing some periodic > > time mark (like a 1 pulse per second from a GPS receiver)? > > IIRC, we discussed this on list within the last couple of years. There > were only two approaches that seemed worth considering -- a network > based schema that used ntp or an ntp-like daemon to gradually refine the > system local clock to be accurate within a microsecond across a network. > This isn't horribly easy on a network with a latency on the order of > hundreds of microseconds, but it is doable -- it just takes a lot of > packets and complete control of the network. This synchronization won't > be accurate to absolute time unless the master clock is, but it would be > synchronization. > > It seems reasonable to do this as a grant funded project. I've got a > couple or three student hackers handy that might tackle it if I could > get it funded (with a very small grant). The nice thing about this > approach is that it could be a fully GPL, fully portable solution that > could be built right into e.g. a bproc cluster. Note also that all that > is really required in terms of counters is to be able to match the > distributed/refined time base to a particular value of the CPU cycle > counter. Then a trivial revision of the non-portable CPU-based > nanoscale gettimeofday I posted (or an architecture-aware replacement of > the archaic gettimeofday library functions) would make this effectively > transparent. But, does the CPU cycle counter run at constant speed? Or does it ratchet up and down with power management, for instance. This was a curse for us in the DSP world, where we did alter CPU clock rate(as in stopping the processor) to reduce power during "idle time", which is why we went to using a counter that happened to be in a peripheral chip. > > The second approach is hardware, which seems perfectly "legal" but which > requires an additional investment and which pushes one a bit away from > the "commodity" ideal and into the "hardware hacker" ideal which is not > everybody's cup of tea. GPS is (as was discussed) a good way to get a > very accurate time base, with the caveat that GPS signals are very weak > (my handheld GPS won't work indoors, or even inside a car, let alone in > a basement server room underneat a few hundred tons of steel and > concrete). I think that one can probably do it with no mods to the motherboard or outside hardware. For instance, the NTP servers can bring in the 1pps on one of the serial port lines. And, "legacy free" notwithstanding, most motherboards still have a parallel printer port. Perhaps something along the lines of PAPERS? > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > have built in GPS, In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest widget, a GPS receiver that's about 10x15 mm, and can do what's called "assisted" fixes, where the cell site does a lot of the work in terms of keeping track of satellite ephemerides, and so forth. (Greatly reducing the processing, and power, in the phone). It's all part of the E911 mandate which requires cellphone providers to give locations within a few meters when someone makes a 911 call on their cellphone. (Such position information is also quite commercially valuable.. they could send you customized SMS text messages: "Looking for Starbucks? 2 blocks ahead on the left?") and for $45 or so I bought a GPS-sync'd clock from > Wal-Mart (which won't work inside the house because GPS signals are so > weak, sigh). GPS thus seems like something that would actually be > fairly tricky to get working inside a lot of server room environments > unless an external antenna were used and piped down to the device, which > would add to the expense and hassle and be moderately inefficient on a > per-system basis. That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least most of the widely advertised "atomic clocks" do that...) You don't need to have GPS on each computer (in fact, that would be an egregious waste of money), just the sync signal piped to them all. A single wire, pulsing at 1pps, is sufficient. Put the GPS somewhere convenient. You can send the 1pps down cat5 wire, or on a fiber optic link, or almost anything. Even better (if you're into this kind of thing), is a GPS timing receiver like the surplus Z3801, which use GPS to "discipline" a very high quality quartz oscillator, which then generates a very clean and precise 10MHz, as well as the 1 pps. You can also generate precision IRIG (or AES/EBU or SMPTE) time code, and distribute that. > > I have my own reasons for liking the design you indicate below, and > would like it very much if two PCs separated by tens of kilometers and > not even ON a common network could synchronize an internal clock to less > than a microsecond, better yet to less than 100 nanoseconds. Such things are available off the shelf now, just not as a commodity. I would think, though, that a Z3801 type receiver ($250 surplus) hooked to each PC can do it to microsecond precision. Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all these sorts of things, with external trigger inputs and so forth. For this > to happen, the whole thing REALLY needs to move onto the motherboard and > be tightly integrated with the system design, although a PCI card or USB > device both are alternatives that would likely "work", especially if one > wrote kernel drivers for them that could do the entire job of reading > and setting the internal cycle-counter-to-absolute-time variable as an > atomic operation AND actually correct the result for the time it took to > do the set. If one could build a hardware device that kept > nanosecond-accurate time, it should be possible to write a gettimeofday > or inline assembler fragment that is accurate to within a few > nanoseconds (the minimal overhead associated with reading the counter > and saving the result as a timestamp for later conversion). This might already exist on the motherboard.. one just needs to find something that latches the counter deterministically. I'd look at the game port interface. > > Perhaps Intel and/or AMD would consider (or are considering) adding a > GPS receiver to their future mobo designs. Add a single port on the > back for an external antenna, or figure out how to use some of the > considerable quantity of metal attached to the system as a default > antenna. Sell amplified GPS antennae as an inexpensive add-on. Nahh.. have the user buy a GPS receiver pod that is connected via a power/data cable and keep all the RF stuff at the antenna. Like the DeLorme EarthMate or TripMate widgets. The key is getting the 1pps tick into the box in a "nice" fashion. > > Personally I can think of all sorts of benefits to having each and every > PC clock in the universe set and locked to real, absolute time within a > few nanoseconds, and similarly located in space to within a few meters > (those same nanoseconds times c). The little radiotelescope project > I've got a student (still, sigh) working on being one of them, but far > from the only one. I'd really like to be able to sit at my PC and have > it go "happy new year" at PRECISELY the new year, for example. This is what LOFAR and the SKA are all about... I think the Allen Telescope Array up in Hat Creek is also using GPS disciplined clocks to sync everything. > > While we are at it sailing around in the blue sky, can we also put an > entropy generator on the add-on card? Something that can generate > random bits (in parallel) at oh, 32 GHz, buffer them internally, and > deliver the entire buffer to memory via DMA on a single interrupt? My > systems get SO tired of generating pseudorands...:-) How about a free running counter, latched by the output from a geiger counter instead of the 1pps.. (Tough to get numbers at 32 Gnumbers/second though.. but rates in the 10s of kHz would be easy to get with a few microcuries of Am241 and a suitable detector tube/amplifier. ) Or, wide band noise fed to a video A/D and digitized. (you'd have some spectral whitening issues and A/D nonlinearities to deal with) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 8 11:46:21 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 8 Jan 2004 08:46:21 -0800 Subject: [Beowulf] ...Re: Benchmark results References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> Message-ID: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Josip Loncaric" To: "Jim Lux" Cc: Sent: Thursday, January 08, 2004 7:44 AM Subject: Re: [Beowulf] ...Re: Benchmark results > Jim Lux wrote: > > > > At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: > > > >> Despite various software/hardware network > >> layers and longer distances involved, it should be possible to get > >> microsecond synchronization through the network by gradually tweaking > >> local clock offset and drift... > > > > > > Most of the more recent implementations of NTP clients can do this sort > > of thing, sending messages both ways across the link to measure the > > latency (and latency distribution) > > I love NTP, but in its current state it does not deliver microsecond > level synchronization. > > I just checked my old cluster (which runs NTP) and the compute nodes are > off by 500-25000 microseconds. Actually, most are within 1000 > microseconds, but about 20% are off by up to 25000 microseconds. This > happens over Fast Ethernet where ntpdate usually reports timing > dispersion of 0-20 microseconds (based on 7 ping-pong measurements). In > other words, network timing is quite precise, but we'd need about 1000 > times better clock synchronization. > > NTP may need to be streamlined and tuned for clusters... Longer integration times in the smoothing algorithm? The limiting thing might be the instability of the clock oscillator on the motherboard. They're not necessarily designed for low phase noise, and may have terrible short term statistics. The other thing that will bite you with the NTP approach is the asynchronous nature of the Ethernet interface against the CPU. The Ethernet card has it's own clock, it's own logic, etc., so the time from "receving a message" over the wire to the CPU knowing about it could vary quite widely, and the statistics might be non-stationary, which would kill any sort of statistical averaging scheme. You're probably right.. NTP, using just ethernet, will probably never get to sub microsecond without external help, or cleverly designed specialized ethernet hardware (use the bit clock in the PHY layer as the time transfer medium) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 11:34:26 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 11:34:26 -0500 (EST) Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFD7AEA.4060709@lanl.gov> Message-ID: On Thu, 8 Jan 2004, Josip Loncaric wrote: > Jim Lux wrote: > > > > At 05:18 PM 1/7/2004 -0700, Josip Loncaric wrote: > > > >> Despite various software/hardware network > >> layers and longer distances involved, it should be possible to get > >> microsecond synchronization through the network by gradually tweaking > >> local clock offset and drift... > > > > > > Most of the more recent implementations of NTP clients can do this sort > > of thing, sending messages both ways across the link to measure the > > latency (and latency distribution) > > I love NTP, but in its current state it does not deliver microsecond > level synchronization. > > I just checked my old cluster (which runs NTP) and the compute nodes are > off by 500-25000 microseconds. Actually, most are within 1000 > microseconds, but about 20% are off by up to 25000 microseconds. This > happens over Fast Ethernet where ntpdate usually reports timing > dispersion of 0-20 microseconds (based on 7 ping-pong measurements). In > other words, network timing is quite precise, but we'd need about 1000 > times better clock synchronization. > > NTP may need to be streamlined and tuned for clusters... I'm not sure that the problem is NTP itself -- from what I recall reading in its documentation it is CAPABLE of achieving 1 usec synchronization, over time. However, in order to do that it needs better tools than the current gettimeofday (and other time calls) which seem to deliver no better than perhaps millisecond resolution. It also very likely needs to be done not by a userspace daemon but by the kernel, as only the kernel can decide to do something in a true "single user mode" and for specified brief periods pretty much ignore interrupts and pending tasks and internal sources of delay and noise. If one used the internal hardware cycle counter as the timer (with sub-nanosecond hardware resolution) and worked solely on iteratively refining a single clock set/sync variable (the offset of the CPU cycle counter relative to the epoch, basically) then one could IN PRINCIPLE get below a microsecond even with fast ethernet with enough handshaking. It might take a few tens of thousands of packets per node on an otherwise quiet network, but eventually both ends would agree on packet timings, reject exchanges that fell outside the empirically determined minimum/optimum packet timing for the particular hardware, and settle down to let the central limit theorem and enough exchanges reduce the error bars to the desired level of precision. That's why I think it is a reasonable project for a bright undergraduate. It doesn't seem to me that it would be that difficult, at least not at the cluster/LAN/one-hop level where the networking hardware can be at least moderately deterministic. ntp is really a WAN tool and is robust and portable at the expense of potentially achievable precision. rgb > > Sincerely, > Josip > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Jan 8 12:08:36 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Thu, 08 Jan 2004 12:08:36 -0500 Subject: [Beowulf] tracking down NaN's in an mpi job Message-ID: <1073581716.5027.83.camel@protein.scalableinformatics.com> Hi folks: I am trying to help a customer track down the spontaneous appearance of NaN's in a StarCD job. When submitted in parallel (using mpich 1.2.4 as supplied by Adapco) from their head node on their cluster, they get instability in later iterations, and eventually NaN's start cropping up. When run on one CPU, it appears to be stable (on the head node and compute nodes). They are using the p4 mechanism with rsh. I don't know if others have seen anything like this. Is it possible that messages are being missed somehow in the parallel run? The head node and compute nodes are running two different versions of the Linux kernel (same distribution). Suggestions/hints welcome. Joe -- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 13:13:54 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 11:13:54 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> References: <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> Message-ID: <3FFD9DE2.10303@lanl.gov> Jim Lux wrote: > From: "Josip Loncaric" >> >>NTP may need to be streamlined and tuned for clusters... > > Longer integration times in the smoothing algorithm? Better yet, extended Kalman filtering (given my control theory background). BTW, there was a highly relevant paper 9 years ago: "On Efficiently Implementing Global Time for Performance Evaluation on Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of Parallel and Distributed Computing, No. 28, pp. 84-93 (1995). Their experimental analysis of the relationship between the system clock and the reference clock over a 50-minute period showed a good fit to a linear model, within about 50-100 microseconds, with slow 16-minute periodicity apparently related to thermal swings in the computer room (A/C cycling). These measurements were done using their T-Node "Meganode" system (128 T800 transputers) on the Transputer network, older than 1995 vintage, probably with clock frequency of about 3.8 MHz. Current networks are much faster, although clock drift behavior may still be similar. Presumably, clock drift coefficients could be correlated to motherboard temperature measurements. Over time, one should be able to construct a linearized model of clock dynamics, linking the local clock drift, motherboard temperature, and possibly even power supply voltages. This should overcome the main sources of timing uncertainty. Sincerely, Josip P.S. Internally, clusters often have a single broadcast domain. Even if the head node broadcast an augmented NTP packet every second, this would still not impact the network much -- but may generate enough timing data so that each node's filter would have at least some good measurements to work with... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 13:25:53 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 10:25:53 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFD9DE2.10303@lanl.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> Message-ID: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: > >>From: "Josip Loncaric" >>> >>>NTP may need to be streamlined and tuned for clusters... >>Longer integration times in the smoothing algorithm? > >Better yet, extended Kalman filtering (given my control theory background). > >BTW, there was a highly relevant paper 9 years ago: > >"On Efficiently Implementing Global Time for Performance Evaluation on >Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of Parallel >and Distributed Computing, No. 28, pp. 84-93 (1995). I'll get a copy and take a look. >Their experimental analysis of the relationship between the system clock >and the reference clock over a 50-minute period showed a good fit to a >linear model, within about 50-100 microseconds, with slow 16-minute >periodicity apparently related to thermal swings in the computer room (A/C >cycling). > >These measurements were done using their T-Node "Meganode" system (128 >T800 transputers) on the Transputer network, older than 1995 vintage, >probably with clock frequency of about 3.8 MHz. Current networks are much >faster, although clock drift behavior may still be similar. What was the measurement/integration interval for their frequency measurements? Robert (and I, and others) are interested in stability over fairly short time scales (milliseconds?). >Presumably, clock drift coefficients could be correlated to motherboard >temperature measurements. Over time, one should be able to construct a >linearized model of clock dynamics, linking the local clock drift, >motherboard temperature, and possibly even power supply voltages. This >should overcome the main sources of timing uncertainty. I wouldn't be too sure that you could build a suitable model. One could certainly make a very accurate predictor in the long term sense: it would very accurately model things like how many cycles of the processor clock there are in every 10 or 60 second interval. But at shorter time scales, the clocks MIGHT be pretty bad, with random jumps and stuff. I don't know, but since it's relevant to my current research, I'm going to find out, starting today.. The run of the mill clock oscillators you see on a motherboard do not have very good short term characteristics. For instance, you can't use them for timing digital communications interfaces like SONET or OC-3 or ATM, where they use much higher quality oscillators, with better jitter specs. A typical processor isn't going to much care if there's, say, 10% jitter (which is really hideously bad) in the clock supplied to it , providing the clock multiplier loops on the chip can stay locked. Likewise, since the processor is, generally, synchronously clocked logic (except for those of you running PDP-8 based clusters!), instantaneous variation in clock rates don't do much, other than set a bound on timing margins (more jitter means you need more timing margin in the propagation paths on chip). If you plot the frequency versus time for an oscillator, a "nice" oscillator will show a fairly smooth monotonic drift with aging (usually they drift up with time) and smooth variations with temperature and things like gravitational force or acceleration(if you change the orientation of the oscillator). A "bad" oscillator will have spontaneous jumps up and down. We're talking parts per million here, though, just to keep things in perspective. >Sincerely, >Josip > > >P.S. Internally, clusters often have a single broadcast domain. Even if >the head node broadcast an augmented NTP packet every second, this would >still not impact the network much -- but may generate enough timing data >so that each node's filter would have at least some good measurements to >work with... James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 8 14:05:57 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 8 Jan 2004 14:05:57 -0500 (EST) Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> Message-ID: On Thu, 8 Jan 2004, Jim Lux wrote: > But, does the CPU cycle counter run at constant speed? Or does it ratchet > up and down with power management, for instance. This was a curse for us in > the DSP world, where we did alter CPU clock rate(as in stopping the > processor) to reduce power during "idle time", which is why we went to > using a counter that happened to be in a peripheral chip. One would have to check the Intel or AMD specs to be sure, but I recall that it is a hardware counter of the stabilized baseline clock crystal for the system and is hence very, very stable and quite accurate. However, I admit that I'm not certain -- clock drift in gettimeofday on some systems USED to be a really annoying problem and a load dependent one at that. Also, I have no idea what happens to it on a CPU that does cycle throttling when it gets hot (e.g. P4). The problem with putting it in a peripheral chip is that it inevitably increases the overhead associated with reading it, probably to hundreds of nanoseconds or worse. Probably adequate to achieve usec resolution, but not adequate to get nsec resolution. > > The second approach is hardware, which seems perfectly "legal" but which > > requires an additional investment and which pushes one a bit away from > > the "commodity" ideal and into the "hardware hacker" ideal which is not > > everybody's cup of tea. GPS is (as was discussed) a good way to get a > > very accurate time base, with the caveat that GPS signals are very weak > > (my handheld GPS won't work indoors, or even inside a car, let alone in > > a basement server room underneat a few hundred tons of steel and > > concrete). > > I think that one can probably do it with no mods to the motherboard or > outside hardware. For instance, the NTP servers can bring in the 1pps on one > of the serial port lines. And, "legacy free" notwithstanding, most > motherboards still have a parallel printer port. Perhaps something along > the lines of PAPERS? Again, one has to navigate the device driver for the peripheral and likely hack the kernel or suffer random delays. If you're going this route one might as well just use the network and work harder, as either way you'll need the same techniques to overcome the various sources of random latency and refine an average to the desired resolution. > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > > have built in GPS, > > In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest > widget, a GPS receiver that's about 10x15 mm, and can do what's called > "assisted" fixes, where the cell site does a lot of the work in terms of > keeping track of satellite ephemerides, and so forth. (Greatly reducing the > processing, and power, in the phone). > It's all part of the E911 mandate which requires cellphone providers to give > locations within a few meters when someone makes a 911 call on their > cellphone. (Such position information is also quite commercially valuable.. > they could send you customized SMS text messages: "Looking for Starbucks? 2 > blocks ahead on the left?") I was pretty surprised, but then my Christmas present in 2002 was a cell-phone sized GPS that cost about $100, complete with nifty screen, built in maps, leedle controls -- and it is still mostly battery, as I take it the GPS signals from the various satellites are quite weak and require a lot of amplification. > and for $45 or so I bought a GPS-sync'd clock from > > Wal-Mart (which won't work inside the house because GPS signals are so > > weak, sigh). GPS thus seems like something that would actually be > > fairly tricky to get working inside a lot of server room environments > > unless an external antenna were used and piped down to the device, which > > would add to the expense and hassle and be moderately inefficient on a > > per-system basis. > > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least > most of the widely advertised "atomic clocks" do that...) Hmmm, checking it out on the web it does indeed. I'm probably getting my wife's new cell phone and the various new PDA's scrambled with the thermometer/clock (Acurite Wireless Weather and Atomic Clock Station, $22.44 at Wal-Mart:-). Silly me -- I assume(d) that it just meant that somebody in China had cloned a single-chip GPS receiver and was selling it for $0.50 to the clock's maker. I didn't realize that the cell phone devices were also indirectly being run through the cell towers instead of directly from satellites, although it makes sense since cellular signals work indoors but GPS's don't. At $22.44 I'm tempted to buy one and hack it up and see if one can leech the clock register contents out in a machine readable form. Might not get microsecond resolution that way, but it would be good enough for an NTP server for my house and beat the hell out of the $400 cards sold for that purpose over the counter...;-). I've been TEMPTED to see if my $100 GPS will do the same thing -- somewhere in there it must be reading the time base(s) to do the location computation. > You don't need to have GPS on each computer (in fact, that would be an > egregious waste of money), just the sync signal piped to them all. A single Egregious at current prices, yes. > wire, pulsing at 1pps, is sufficient. Put the GPS somewhere convenient. > You can send the 1pps down cat5 wire, or on a fiber optic link, or almost > anything. > > Even better (if you're into this kind of thing), is a GPS timing receiver > like the surplus Z3801, which use GPS to "discipline" a very high quality > quartz oscillator, which then generates a very clean and precise 10MHz, as > well as the 1 pps. You can also generate precision IRIG (or AES/EBU or > SMPTE) time code, and distribute that. And read it with a $400 card? Alas, I'm not into this kind of thing at that level. What I REALLY want is an over the counter device that sells for $22.95, attaches to a USB port or a PCI slot, works in a server room deep in the bowels of the world, and magically keeps the best clock available to linux on a given architecture accurate to a few nsec. Not too much, really. Couple of years, mass market GPS, prices drop, or somebody will figure out a way to use cell phone towers to achieve the same thing. > > I have my own reasons for liking the design you indicate below, and > > would like it very much if two PCs separated by tens of kilometers and > > not even ON a common network could synchronize an internal clock to less > > than a microsecond, better yet to less than 100 nanoseconds. > > Such things are available off the shelf now, just not as a commodity. I > would think, though, that a Z3801 type receiver ($250 surplus) hooked to > each PC can do it to microsecond precision. > > Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and > reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all > these sorts of things, with external trigger inputs and so forth. Ya, for prices that seem to exceed $500 a pop by a goodly amount. Even a card to read a time reference signal from a provided standard interface is $400 from e.g. masterclock, even a WWVB card is $400 from e.g. beagle. This is nuts (especially for the WWVB card). If Wal-Mart can sell a WWVB clock thermometer with remote wireless thermometer included for less than $25 full retail, and a PCI card base costs a few dollars, I can't believe that one couldn't build and sell a WWVB PCI time base card (and hell, throw temperature in too with a remote sensor probe that plugs into the back or onto the card internally) for considerably less than $100. I suppose there is a limited market but that in part is CREATED by the high prices. I might buy a card for my home network for $100. I might buy a card for every system in my home network for $25. Prices of $1000 (ballpark of a lot of the GPS-based offerings) might be ok for a departmental LAN or an institution, but they are not by any means "consumer" level offerings. Odd at a time that radio clocks are over the counter available and even becoming "common" at less than $50 full retail. > > to happen, the whole thing REALLY needs to move onto the motherboard and > > be tightly integrated with the system design, although a PCI card or USB > > device both are alternatives that would likely "work", especially if one > > wrote kernel drivers for them that could do the entire job of reading > > and setting the internal cycle-counter-to-absolute-time variable as an > > atomic operation AND actually correct the result for the time it took to > > do the set. If one could build a hardware device that kept > > nanosecond-accurate time, it should be possible to write a gettimeofday > > or inline assembler fragment that is accurate to within a few > > nanoseconds (the minimal overhead associated with reading the counter > > and saving the result as a timestamp for later conversion). > > This might already exist on the motherboard.. one just needs to find > something that latches the counter deterministically. I'd look at the game > port interface. Wouldn't one need something that could manage pulses with sub-microsecond rise times and trigger reliably? I'd expect a game interface to be way too slow. I'd even worry about RS232. As I said, a lot of the $300-500 cards sold appear to be nothing but an interface that can reliably manage a time pulse accurate to within a usec or so. > > Personally I can think of all sorts of benefits to having each and every > > PC clock in the universe set and locked to real, absolute time within a > > few nanoseconds, and similarly located in space to within a few meters > > (those same nanoseconds times c). The little radiotelescope project > > I've got a student (still, sigh) working on being one of them, but far > > from the only one. I'd really like to be able to sit at my PC and have > > it go "happy new year" at PRECISELY the new year, for example. > > This is what LOFAR and the SKA are all about... I think the Allen Telescope > Array up in Hat Creek is also using GPS disciplined clocks to sync > everything. nsec syncronization on a global scale has lots of benefits, some of which could enable projects that would dwarf SETI at home. The heck with turning every cellular tower in the country into a giant radiotelescope array -- turn every suitably equipped PC in the country into a receiver in a really enormous radiotelescope array. Now there is a project for a GRID! Frankly, it would probably work better for SETI as well. Maybe even tabletop relativity experiments as well. > > While we are at it sailing around in the blue sky, can we also put an > > entropy generator on the add-on card? Something that can generate > > random bits (in parallel) at oh, 32 GHz, buffer them internally, and > > deliver the entire buffer to memory via DMA on a single interrupt? My > > systems get SO tired of generating pseudorands...:-) > > How about a free running counter, latched by the output from a geiger > counter instead of the 1pps.. > (Tough to get numbers at 32 Gnumbers/second though.. but rates in the 10s of > kHz would be easy to get with a few microcuries of Am241 and a suitable > detector tube/amplifier. ) > Or, wide band noise fed to a video A/D and digitized. (you'd have some > spectral whitening issues and A/D nonlinearities to deal with) Oh, there are people looking and and even implementing a lot of these sorts of things. Unfortunately, a lot of "random" physical sources turn out not be random at all, just to have a relatively short autocorrelation function and to have distributions that are not uniform as well. A lot of the noise-based generators fall into this category, but quantum generators can as well (one has to look at second and higher order correlation functions, for example -- even resonance fluorescence, a textbook example of a "random quantum process" turns out to be photon antibunched, and there is also the venerable work of Hanbury Brown and Twiss. It turns out not to be horribly easy to build a hardware generator that passes diehard or the NIST suite, let alone one that works fast. Lots of effort is actually expended here in corporate america as random numbers are a key element of cryptography, and some devices are on the market. They're just expensive and slow and in some cases not terribly high quality -- "unpredictable" but not necessarily "random". Unpredictable can be made random, sort of, sometimes -- but only via transformations that slow the generator down. Right now I don't know of any process or device that can deliver uniform deviates as fast or faster (or of any better quality) than the faster/better pseudorandom generators, although there may be some under development or out there as the world keeps changing. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 8 15:08:07 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 8 Jan 2004 12:08:07 -0800 Subject: [Beowulf] tracking down NaN's in an mpi job In-Reply-To: <1073581716.5027.83.camel@protein.scalableinformatics.com> References: <1073581716.5027.83.camel@protein.scalableinformatics.com> Message-ID: <20040108200807.GB3000@greglaptop.internal.keyresearch.com> On Thu, Jan 08, 2004 at 12:08:36PM -0500, Joe Landman wrote: > I am trying to help a customer track down the spontaneous appearance > of NaN's in a StarCD job. The usual fix is to compile with floating point options set such that the application faults instead of generating NaNs. This sounds like a turnkey commercial app, though, so why not just file a bug report? -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 14:38:03 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 11:38:03 -0800 Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> Message-ID: <5.2.0.9.2.20040108111528.031bde38@mailhost4.jpl.nasa.gov> At 02:05 PM 1/8/2004 -0500, Robert G. Brown wrote: >On Thu, 8 Jan 2004, Jim Lux wrote: > > > > > I think that one can probably do it with no mods to the motherboard or > > outside hardware. For instance, the NTP servers can bring in the 1pps > on one > > of the serial port lines. And, "legacy free" notwithstanding, most > > motherboards still have a parallel printer port. Perhaps something along > > the lines of PAPERS? > >Again, one has to navigate the device driver for the peripheral and >likely hack the kernel or suffer random delays. If you're going this >route one might as well just use the network and work harder, as either >way you'll need the same techniques to overcome the various sources of >random latency and refine an average to the desired resolution. hacking the kernel is not necessarily evil. I suppose it's a tradeoff: hack the kernel and use commodity hardware; use stock kernel and use hacked hardware. > > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > > > have built in GPS, > > > > In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest > > widget, a GPS receiver that's about 10x15 mm, and can do what's called >I was pretty surprised, but then my Christmas present in 2002 was a >cell-phone sized GPS that cost about $100, complete with nifty screen, >built in maps, leedle controls -- and it is still mostly battery, as I >take it the GPS signals from the various satellites are quite weak and >require a lot of amplification. yep.. as a standalone device, there are lots of $50 (retail) GPS units available in one form or another. The basic module, with antenna, runs about $30-$50 in single quantities. But, there aren't a whole lot of phones with GPS in them yet. No real customer demand yet (although, I can certainly see a "On-Star" type service being attractive to people.. I'm lost, how do I get to the freeway) > > > > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at > least > > most of the widely advertised "atomic clocks" do that...) > >Hmmm, checking it out on the web it does indeed. I'm probably getting >my wife's new cell phone and the various new PDA's scrambled with the >thermometer/clock (Acurite Wireless Weather and Atomic Clock Station, >$22.44 at Wal-Mart:-). Silly me -- I assume(d) that it just meant that >somebody in China had cloned a single-chip GPS receiver and was selling >it for $0.50 to the clock's maker. I didn't realize that the cell phone >devices were also indirectly being run through the cell towers instead >of directly from satellites, although it makes sense since cellular >signals work indoors but GPS's don't. Sort of a combination of the two.. Rather than have to do the entire fix in the phone, they can just receive the GPS signals and make some raw measurements of code and carrier phase, then send it to the cell site, where it can get turned into position. >At $22.44 I'm tempted to buy one and hack it up and see if one can leech >the clock register contents out in a machine readable form. Might not >get microsecond resolution that way, but it would be good enough for an >NTP server for my house and beat the hell out of the $400 cards sold >for that purpose over the counter...;-). I've been TEMPTED to see if my >$100 GPS will do the same thing -- somewhere in there it must be reading >the time base(s) to do the location computation. Check out the documentation for the various Time Servers software.. Almost all GPS receivers have a 1pps output, and many synchronize the NMEA serial output string to the 1pps (that is, you can use the start bit of the first character in the sentence for sync). The highly integrated GPS chips do NOT make it easy to read the internal registers, for a variety of reasons. http://www.synergy-gps.com/ is a big dealer of Motorola Oncore modules and has a lot of documentation. Googling for the ST Microdevices ST20GP6 will find a data sheet for a typical GPS receiver/processor chip. (several years old design, though.. ) Search for "Totally Accurate Clock" and you'll find stuff about using WWVB and/or GPS for this sort of thing. > > You don't need to have GPS on each computer (in fact, that would be an > > egregious waste of money), just the sync signal piped to them all. A > single > >Egregious at current prices, yes. > > > wire, pulsing at 1pps, is sufficient. Put the GPS somewhere convenient. > > You can send the 1pps down cat5 wire, or on a fiber optic link, or almost > > anything. > > > > Even better (if you're into this kind of thing), is a GPS timing receiver > > like the surplus Z3801, which use GPS to "discipline" a very high quality > > quartz oscillator, which then generates a very clean and precise 10MHz, as > > well as the 1 pps. You can also generate precision IRIG (or AES/EBU or > > SMPTE) time code, and distribute that. > >And read it with a $400 card? Read it with software and a single bit input. If you can latch that register on the edges, you're all set, because the time code itself is quite slow, and because time marches forward in a fairly regular fashion, you can take a lot of time decoding the time code sequence (you could do 1 bit per second, and after a few minutes, have the complete sequence locked up). You can certainly decode most audio timecode signals (i.e. IRIG) (but not synchronize to microseconds) with a sound card. >Alas, I'm not into this kind of thing at that level. What I REALLY want >is an over the counter device that sells for $22.95, attaches to a USB >port or a PCI slot, works in a server room deep in the bowels of the >world, and magically keeps the best clock available to linux on a given >architecture accurate to a few nsec. Not too much, really. > >Couple of years, mass market GPS, prices drop, or somebody will figure >out a way to use cell phone towers to achieve the same thing. You could do it today for <$100, and a free serial port, and enough wire to get the GPS receiver to a view of the sky. > > Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and > > reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all > > these sorts of things, with external trigger inputs and so forth. > >Ya, for prices that seem to exceed $500 a pop by a goodly amount. Even >a card to read a time reference signal from a provided standard >interface is $400 from e.g. masterclock, even a WWVB card is $400 from >e.g. beagle. This is nuts (especially for the WWVB card). If Wal-Mart >can sell a WWVB clock thermometer with remote wireless thermometer >included for less than $25 full retail, and a PCI card base costs a few >dollars, I can't believe that one couldn't build and sell a WWVB PCI >time base card (and hell, throw temperature in too with a remote sensor >probe that plugs into the back or onto the card internally) for >considerably less than $100. You could, if you could convince Wal-Mart that they could sell millions of them. >Odd at a time that radio clocks are over the counter available and even >becoming "common" at less than $50 full retail. Wireless LAN interfaces used to be $200, and are now <$30... it might happen. > > > > This might already exist on the motherboard.. one just needs to find > > something that latches the counter deterministically. I'd look at the game > > port interface. > >Wouldn't one need something that could manage pulses with >sub-microsecond rise times and trigger reliably? I'd expect a game >interface to be way too slow. The original game interface latched a counter driven off the processor clock. Yes, overkill to measure a potentiometer position to 1 part in 100, but that's the hardware available. The question would be how slavishly have the chipset designers held to the original design concept. > I'd even worry about RS232. As I said, a >lot of the $300-500 cards sold appear to be nothing but an interface >that can reliably manage a time pulse accurate to within a usec or so. For RS232, it's the interrupt latency on the RS232 status line that determines the performance, which can be quite impressive. Essentially a kernel hack, but someone's already done it for the NTB clock drivers. Sub microsecond is fairly easy. All the ISR has to do is latch the processor time counter and set a flag, so it can run unmasked, etc. Likewise the IRQ7 line coming in from the printer port can be used. Although it's low priority, you don't need perfect accuracy.. you just throw out the interrupts that are obviously late, having been overridden by a higher priority interrupt. For those of you working with Win NT, take a look at AboutTime, which has a kernel driver doing just this sort of thing. I don't know how deterministic the interrupt latency is on the modern x-ium and x-lon processors though. It kind of depends on if the designer wanted to flush the pipeline before servicing the interrupt, or if they stall it. It might be a bit tricky to even find out from the x-ium documentation exactly what it does, since for most users, the few nanoseconds difference doesn't make any difference. There's probably also some latching pipeline logic running at fairly low clock rates between the IRQ line and the actually processor. Theinterrupt controller could introduce some random latency depending on the relative phases of the clock latching the interrupt request and the processor seeing the IRQ asserted on the chip. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dag at sonsorol.org Thu Jan 8 14:12:20 2004 From: dag at sonsorol.org (Chris Dagdigian) Date: Thu, 08 Jan 2004 14:12:20 -0500 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <3FFDAB94.2070805@sonsorol.org> {Forwarded to this list on behalf of a friend with some email troubles...} > I am in the process of trying to get a stopgap perfomance monitoring > system going on a 64 CPU Linux cluster with LSF. Ultimately, I hope to > use PCP for data collection, but since nobody seems to be doing this yet, > we are going to be rolling our own solution. To meet some of their needs, > management has asked for an interim solution that gives them a web page > with aggregate usage statistics and such. > > Unfortunately, ganglia is a non-starter because our networking group can't > enable multicast for the private network the cluster lives on (it would > break almost everything else). > > Does anyone have any suggestions for an alternative that can be quickly > implemented, doesn't rely on multicast, and that can generate graphs of aggregate > statistics on the fly? Regards, Chris BioTeam.net -- Chris Dagdigian, Independent life science IT & research computing consulting Office: 617-666-6454, Mobile: 617-877-5498, Fax: 425-699-0193 Work: http://BioTeam.net PGP KeyID: 83D4310E Yahoo IM: craffi _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Thu Jan 8 15:03:00 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Thu, 8 Jan 2004 12:03:00 -0800 (PST) Subject: [Beowulf] MPI noob prog, needs help Message-ID: <20040108200300.60882.qmail@web12705.mail.yahoo.com> hello all, Can I implement MPI usage in common C programs directly by MPI_Init and MPI_Finalize and run it on slave nodes. Also is MPI:Complete Reference available in pdf format. I stay in India and its not printed locally here. Online prices are too high and the shipping is too much , total cost ends up at around 60 $. will be glad if someone can optimize the following C program to run on multiple nodes in LAM/MPI. Its for a project. thanks in advance, toufeeq ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus -------------- next part -------------- A non-text attachment was scrubbed... Name: prime_sieve.c Type: application/octet-stream Size: 1344 bytes Desc: prime_sieve.c URL: From josip at lanl.gov Thu Jan 8 16:43:30 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 14:43:30 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <3FFDCF02.3000501@lanl.gov> Jim Lux wrote: > At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote: > >> "On Efficiently Implementing Global Time for Performance Evaluation on >> Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of >> Parallel and Distributed Computing, No. 28, pp. 84-93 (1995). > > What was the measurement/integration interval for their frequency > measurements? Robert (and I, and others) are interested in stability > over fairly short time scales (milliseconds?). Alas, their sampling interval was about 6 seconds. They had 500 message exchanges in 50 minutes. However, drift due to independent random events should grow like sqrt(time), so in absolute terms it should be lower at smaller time scales (even though it may be larger in relative terms). In other words, if you had 6 microseconds of uncompensated drift in 6 seconds (1ppm), and you assume that the uncompensated part was due to a random walk process, you'd see about 0.06 microseconds of drift in 0.6 milliseconds. This is not bad in absolute terms, even though in relative terms it looks bad (100ppm). >> Presumably, clock drift coefficients could be correlated to >> motherboard temperature measurements. Over time, one should be able >> to construct a linearized model of clock dynamics, linking the local >> clock drift, motherboard temperature, and possibly even power supply >> voltages. This should overcome the main sources of timing uncertainty. > > > I wouldn't be too sure that you could build a suitable model. One could > certainly make a very accurate predictor in the long term sense: it > would very accurately model things like how many cycles of the processor > clock there are in every 10 or 60 second interval. But at shorter time > scales, the clocks MIGHT be pretty bad, with random jumps and stuff. I > don't know, but since it's relevant to my current research, I'm going to > find out, starting today.. Hmmm... I'm no expert, but I believe that typical clock oscillators become quite good when temperature compensation is used. I am not sure if there is any significant supply voltage dependence, particularly since there may be some voltage stabilization in the clock circuit. I assume we're talking about a crystal oscillator plus clock multipliers; there may be significant phase jitter but ultimately one cares only about cycle counting. These days, there can be over 3000 cycles in a microsecond. If good quality synchronization pulses can be provided every second, then microsecond requirement means that one can afford 1 ppm of uncompensated clock drift. Over a year, the uncompensated drift could be 31.5 seconds. Clock in my car does almost as well, despite being subjected to 60 deg. C temperature swings and variable accelerations. Perhaps I'm overly optimistic, but writing a kernel module to achieve cluster-wide microsecond-level clock synchronization just might be feasible... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amacater at galactic.demon.co.uk Thu Jan 8 17:53:52 2004 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Thu, 8 Jan 2004 22:53:52 +0000 Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: References: <001501c3d604$a350f170$36a8a8c0@LAPTOP152422> Message-ID: <20040108225352.GA29682@galactic.demon.co.uk> On Thu, Jan 08, 2004 at 02:05:57PM -0500, Robert G. Brown wrote: > On Thu, 8 Jan 2004, Jim Lux wrote: > > > > The second approach is hardware, which seems perfectly "legal" but which > > > requires an additional investment and which pushes one a bit away from > > > the "commodity" ideal and into the "hardware hacker" ideal which is not > > > everybody's cup of tea. GPS is (as was discussed) a good way to get a > > > very accurate time base, with the caveat that GPS signals are very weak > > > (my handheld GPS won't work indoors, or even inside a car, let alone in > > > a basement server room underneat a few hundred tons of steel and > > > concrete). > > > > I think that one can probably do it with no mods to the motherboard or > > outside hardware. For instance, the NTP servers can bring in the 1pps on one > > of the serial port lines. And, "legacy free" notwithstanding, most > > motherboards still have a parallel printer port. Perhaps something along > > the lines of PAPERS? > > Again, one has to navigate the device driver for the peripheral and > likely hack the kernel or suffer random delays. If you're going this > route one might as well just use the network and work harder, as either > way you'll need the same techniques to overcome the various sources of > random latency and refine an average to the desired resolution. Maplin electronics did a kit - cheaply - to run off the time code from 60kHz clocks and put it into the serial/parallel port. Know any radio hams - a lot are taking a temperature controlled crystal oscillator and phase locking to WWWV or equivalent. > > > > GPS is getting to be VERY cheap -- almost all cell phones now apparently > > > have built in GPS, > > > > In the GPS mfrs dreams maybe!... Soon. Motorola just released their latest > > widget, a GPS receiver that's about 10x15 mm, and can do what's called > > "assisted" fixes, where the cell site does a lot of the work in terms of > > keeping track of satellite ephemerides, and so forth. (Greatly reducing the > > processing, and power, in the phone). > > It's all part of the E911 mandate which requires cellphone providers to give > > locations within a few meters when someone makes a 911 call on their > > cellphone. (Such position information is also quite commercially valuable.. > > they could send you customized SMS text messages: "Looking for Starbucks? 2 > > blocks ahead on the left?") > > I was pretty surprised, but then my Christmas present in 2002 was a > cell-phone sized GPS that cost about $100, complete with nifty screen, > built in maps, leedle controls -- and it is still mostly battery, as I > take it the GPS signals from the various satellites are quite weak and > require a lot of amplification. Correlation of the satellite constellation is what takes the processing power. > > > and for $45 or so I bought a GPS-sync'd clock from > > > Wal-Mart (which won't work inside the house because GPS signals are so > > > weak, sigh). GPS thus seems like something that would actually be > > > fairly tricky to get working inside a lot of server room environments > > > unless an external antenna were used and piped down to the device, which > > > would add to the expense and hassle and be moderately inefficient on a > > > per-system basis. > > > > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least > > most of the widely advertised "atomic clocks" do that...) At 60kHz, the clocks you get in the States have warnings on - not necessarily reliable on the Eastern seaboard because of co-channel interference from Rugby, UK. Here in Europe, we also have/had DCF77 on approx 77kHz from Germany. > > nsec syncronization on a global scale has lots of benefits, some of > which could enable projects that would dwarf SETI at home. The heck with > turning every cellular tower in the country into a giant radiotelescope > array -- turn every suitably equipped PC in the country into a receiver > in a really enormous radiotelescope array. Now there is a project for a > GRID! Haven't you heard - the Govt. is using them to track individuals by their individual radar fingerprint. They can watch me wherever I walk and they're using the system to mess with my mind ... :) > > Frankly, it would probably work better for SETI as well. > > Maybe even tabletop relativity experiments as well. > > > > > While we are at it sailing around in the blue sky, can we also put an > > > entropy generator on the add-on card? Something that can generate > > > random bits (in parallel) at oh, 32 GHz, buffer them internally, and > > > deliver the entire buffer to memory via DMA on a single interrupt? My > > > systems get SO tired of generating pseudorands...:-) > > > Unfortunately, a lot of "random" physical sources turn > out not be random at all, just to have a relatively short > autocorrelation function and to have distributions that are not uniform > as well. A lot of the noise-based generators fall into this category, > but quantum generators can as well (one has to look at second and higher > order correlation functions, for example -- even resonance fluorescence, > a textbook example of a "random quantum process" turns out to be photon > antibunched, and there is also the venerable work of Hanbury Brown and > Twiss. It turns out not to be horribly easy to build a hardware > generator that passes diehard or the NIST suite, let alone one that > works fast. > The Knuth random number generator that converged to a value - the first example in Seminumerical Algorithms springs to mind :) > Lots of effort is actually expended here in corporate america as random > numbers are a key element of cryptography, and some devices are on the > market. They're just expensive and slow and in some cases not terribly > high quality -- "unpredictable" but not necessarily "random". > Unpredictable can be made random, sort of, sometimes -- but only via > transformations that slow the generator down. Right now I don't know of > any process or device that can deliver uniform deviates as fast or > faster (or of any better quality) than the faster/better pseudorandom > generators, although there may be some under development or out there as > the world keeps changing. > > rgb > > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From atp at piskorski.com Thu Jan 8 16:48:35 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Thu, 8 Jan 2004 16:48:35 -0500 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <200401081951.i08JpDh12979@NewBlue.scyld.com> References: <200401081951.i08JpDh12979@NewBlue.scyld.com> Message-ID: <20040108214835.GA34465@piskorski.com> On Thu, Jan 08, 2004 at 11:34:26AM -0500, Robert G. Brown wrote: > On Thu, 8 Jan 2004, Josip Loncaric wrote: > > I love NTP, but in its current state it does not deliver microsecond > > level synchronization. > > NTP may need to be streamlined and tuned for clusters... > > I'm not sure that the problem is NTP itself -- from what I recall > reading in its documentation it is CAPABLE of achieving 1 usec > synchronization, over time. However, in order to do that it needs > better tools than the current gettimeofday (and other time calls) which > seem to deliver no better than perhaps millisecond resolution. It also > very likely needs to be done not by a userspace daemon but by the > kernel, as only the kernel can decide to do something in a true "single Would DJB's clockspeed, taiclock, etc. stuff be any better than ntp for that? His website mentions using CPU hardware tick or nanosecond counters, but doesn't define "precise" when it says that taiclock is "... suitable for precise time synchronization over a local area network ...": http://cr.yp.to/clockspeed.html http://cr.yp.to/libtai.html -- Andrew Piskorski http://www.piskorski.com/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 8 18:20:00 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 8 Jan 2004 15:20:00 -0800 Subject: [Beowulf] MPI noob prog, needs help In-Reply-To: <20040108200300.60882.qmail@web12705.mail.yahoo.com> References: <20040108200300.60882.qmail@web12705.mail.yahoo.com> Message-ID: <20040108232000.GA5350@greglaptop.internal.keyresearch.com> On Thu, Jan 08, 2004 at 12:03:00PM -0800, 70uf33q Hu5541n wrote: > Also is MPI:Complete Reference available in pdf > format. It looks like the complete contents are: http://www.netlib.org/utk/papers/mpi-book/mpi-book.html Is this just the Standard? -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 17:37:19 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 15:37:19 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <3FFDDB9F.3050201@lanl.gov> Jim Lux wrote: > > We're talking parts per million here, though, just to keep things in > perspective. I just checked some quartz crystal oscillator specs, and they range all over the place. The cheap ones recommended for Gigabit Ethernet devices claim 100 ppm stability, but good quality ones can be ordered with 0.1 ppm specs. HP application node 200-2 "Fundamentals of Quartz Oscillators" (see http://www.measurement.tm.agilent.com/main.cgi?PSP_NEXT=Redirect&Document:CONTENT_ID=6980) says that temperature effects are the dominant influence (2.5 ppm within 0-50 deg. C). This is followed by aging (<0.3ppm/month), line voltage change (<0.1ppm/10% change), and various other smaller effects. A good crystal may deliver <2.5 ppm frequency variation over a 0 to 50 deg. C range, but the temperature dependence may be quite nonlinear. Temperature compensated crystals can deliver 0.5 ppm stability over the same range. Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 8 18:33:28 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 08 Jan 2004 16:33:28 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <20040108214835.GA34465@piskorski.com> References: <200401081951.i08JpDh12979@NewBlue.scyld.com> <20040108214835.GA34465@piskorski.com> Message-ID: <3FFDE8C8.40700@lanl.gov> Andrew Piskorski wrote: > Would DJB's clockspeed, taiclock, etc. stuff be any better than ntp > for that? Interestingly enough, DJB says that after he applied clockspeed compensation, his machine drifted only 0.21 seconds in three months. This would imply long term clock stability of about 0.027 ppm, but as Jim already pointed out, short term stability usually isn't as good. Using the random walk assumption, DJB's clock could drift 75 microseconds in either direction each second (at random), and still meet these long term stability results... While full NTP may be too much, some NTP-like approach is probably needed... Sincerely, Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 19:14:38 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 16:14:38 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <3FFDCF02.3000501@lanl.gov> References: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20040108160818.031eb830@mailhost4.jpl.nasa.gov> At 02:43 PM 1/8/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: > >>At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote: >> >>>"On Efficiently Implementing Global Time for Performance Evaluation on >>>Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of >>>Parallel and Distributed Computing, No. 28, pp. 84-93 (1995). >>What was the measurement/integration interval for their frequency >>measurements? Robert (and I, and others) are interested in stability >>over fairly short time scales (milliseconds?). > >Alas, their sampling interval was about 6 seconds. They had 500 message >exchanges in 50 minutes. > >However, drift due to independent random events should grow like >sqrt(time), so in absolute terms it should be lower at smaller time scales >(even though it may be larger in relative terms). In other words, if you >had 6 microseconds of uncompensated drift in 6 seconds (1ppm), and you >assume that the uncompensated part was due to a random walk process, you'd >see about 0.06 microseconds of drift in 0.6 milliseconds. This is not bad >in absolute terms, even though in relative terms it looks bad (100ppm). Unfortunately, the errors aren't nice additive white gaussian processes... You could have substantial short term variations that average out in the long run. >>>Presumably, clock drift coefficients could be correlated to motherboard >>>temperature measurements. Over time, one should be able to construct a >>>linearized model of clock dynamics, linking the local clock drift, >>>motherboard temperature, and possibly even power supply voltages. This >>>should overcome the main sources of timing uncertainty. >> >>I wouldn't be too sure that you could build a suitable model. One could >>certainly make a very accurate predictor in the long term sense: it would >>very accurately model things like how many cycles of the processor clock >>there are in every 10 or 60 second interval. But at shorter time scales, >>the clocks MIGHT be pretty bad, with random jumps and stuff. I don't >>know, but since it's relevant to my current research, I'm going to find >>out, starting today.. > >Hmmm... I'm no expert, but I believe that typical clock oscillators >become quite good when temperature compensation is used. I am not sure if >there is any significant supply voltage dependence, particularly since >there may be some voltage stabilization in the clock circuit. I assume >we're talking about a crystal oscillator plus clock multipliers; there may >be significant phase jitter but ultimately one cares only about cycle >counting. These days, there can be over 3000 cycles in a microsecond. Yes, they're reasonably stable in terms of frequency, measured over a long time, but over short time spans (tau in the timing literature) they can be much worse. This kind of thing is what separates run of the mill GPS receivers (which have 1pps jitters of around 100ns) from so-called timing grade receivers (which have 1pps jitter of <10-20 ns). >If good quality synchronization pulses can be provided every second, then >microsecond requirement means that one can afford 1 ppm of uncompensated >clock drift. Over a year, the uncompensated drift could be 31.5 >seconds. Clock in my car does almost as well, despite being subjected to >60 deg. C temperature swings and variable accelerations. The 32.768 kHz crystal in your car clock is a good example of an oscillator that's not too hot in the short run (seconds) but is very good in the long run. Of course, they run at really, really low powers, which doesn't help. >Perhaps I'm overly optimistic, but writing a kernel module to achieve >cluster-wide microsecond-level clock synchronization just might be feasible... I am entirely certain that this can be done, and may have already been done. It's such a useful thing to "know, exactly, what time it is".... James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Thu Jan 8 19:12:06 2004 From: shewa at inel.gov (Andrew Shewmaker) Date: Thu, 08 Jan 2004 17:12:06 -0700 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> References: <1073436271.15823.30.camel@protein.scalableinformatics.com> <5.2.0.9.2.20040107094930.030fd040@mailhost4.jpl.nasa.gov> Message-ID: <3FFDF1D6.4040309@inel.gov> Jim Lux wrote: > This is along the lines of what that nifty benchmark (the name of which > eludes me right now) that produces a graph of speed vs computation size > does.. You can "see" the effects of cache etc as the problems space gets > bigger. (Edward Tufte would be pleased with the results) Are you referring to HINT? The original Ameslab home page is gone, but it looks like there is a mirror at http://hint.byu.edu An example of the memory regime graph is at http://hint.byu.edu/tutorials/graphs/regime/regime.html I'm sure Mr. Tufte would have some very good suggestions to improve the graph on the above page, though. Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 19:07:34 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 16:07:34 -0800 Subject: [Beowulf] more about crystal oscillators than you ever wanted to know, was Re: Benchmark results In-Reply-To: <3FFDDB9F.3050201@lanl.gov> References: <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> Message-ID: <5.2.0.9.2.20040108153952.031c8ba0@mailhost4.jpl.nasa.gov> Fascinating as this is, it is straying pretty far afield from Beowulfery though. I've put some comments on crystal oscillators below Josip's comments.. To keep it somewhat list related, I've looked over some documentation on early PCs, and it appears that the 8253(PC)/8254 (AT and later) didn't have any of the gate inputs brought out conveniently, so you can't use it for timing. And, the game port is a latched input port, with some 555 type timers, triggered by another port output, so you can't jam anything in there either. Looks like IRQs off the serial or parallel port is your best bet. I haven't been able to fully determine how the usual chipsets these days implement the 8259 interrupt controller logic, and how the average P4 handles the interrupt in the first place. So, what one really needs is something that will field the IRQ, grab the CPU counter, and stuff it in a known location somewhere. I should think that this would have very much sub microsecond scatter, even if the entire operation takes several microseconds to do. (and, on modern processors, that's a bunch of ops you're giving up to do the timing...) At 03:37 PM 1/8/2004 -0700, Josip Loncaric wrote: >Jim Lux wrote: >>We're talking parts per million here, though, just to keep things in >>perspective. > >I just checked some quartz crystal oscillator specs, and they range all >over the place. The cheap ones recommended for Gigabit Ethernet devices >claim 100 ppm stability, but good quality ones can be ordered with 0.1 ppm >specs. > >HP application node 200-2 "Fundamentals of Quartz Oscillators" (see >http://www.measurement.tm.agilent.com/main.cgi?PSP_NEXT=Redirect&Document:CONTENT_ID=6980) >says that temperature effects are the dominant influence (2.5 ppm within >0-50 deg. C). This is followed by aging (<0.3ppm/month), line voltage >change (<0.1ppm/10% change), and various other smaller effects. > >A good crystal may deliver <2.5 ppm frequency variation over a 0 to 50 >deg. C range, but the temperature dependence may be quite nonlinear. >Temperature compensated crystals can deliver 0.5 ppm stability over the >same range. There's also the issue of phase noise.. you could have a fairly unstable (in the long term sense), but very clean, oscillator. A cesium beam standard is exactly the opposite.. very, very stable in the long term, really noisy in the short term, so the usual standard source uses a very clean quartz oscillator that is locked to the cesium reference (or to GPS, nowadays.. GPS uses Cesium and Rubidium sources on board). It kind of depends on what time scale you want to measure things over. If it's down in the microseconds for the sort of nanobenchmarking that Robert was talking about, then you need a good low jitter source. In a typical Ethernet receiver, there's a tracking loop that acquires the frequency (bit timing) of the incoming signal during the preamble, and it can tolerate a fairly wide offset. However, it really depends on the bit to bit timing being very stable, so it can make the decisions. The internal oscillator in a lab time interval counter might be pretty poor, noise wise,but very stable, since its function is to measure frequency. An oscillator in a receiver, transmitter, or spectrum analyzer may have poorer frequency stability, but be "cleaner" because noise in the oscillator output shows up in the signal being transmitted or received because of reciprocal mixing. There's a tradeoff between aging and temperature dependence as well. A temperature compensated oscillator uses a temperature dependent capacitor in combination with the crystal to pull the crystal slightly to compensate the variations in its resonant frequency. However, the fact that the crystal can be pulled means that it has to be lower Q, and hence, poorer close in noise. VCXOs (Voltage controlled oscillators) have the same general problem.. if you can adjust it, it implies that "other things" can also adjust it, causing random variations. Aging is more a matter of drive power and mechanical stress. Driving the crystal harder gives you a cleaner output (because your farther above the thermal noise floor), but also heats the crystal, causing it to age faster. The ideal scheme is to have a system that does not depend on the crystal frequency being at a particular value, but, rather, can adapt to changing frequencies. Then, you can have a very high quality (stiff) crystal (so the environmental and loading effects don't affect it, and so it has low phase noise), drive the heck out of it (causing it to drift and age rapidly), and measure it periodically against an accurate clock (i.e. Cs, Rb, or GPS). This is basically the NTP approach... measure your computer's clock against a higher stratum reference, and change the "ticks per second" number to adjust so that the clock runs true. You then apply a calibration factor, or retune the radio, or whatever, to account for the known frequency of the oscillator. By the way, this is what they do in GPS receivers.. one of the variables the receiver solves for (in addition to pseudo range and pseudo range rate) is local clock offset and offset rate. It also means you need more satellites to get a fix, though. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 8 19:48:16 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 08 Jan 2004 16:48:16 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <20040108214835.GA34465@piskorski.com> References: <200401081951.i08JpDh12979@NewBlue.scyld.com> <200401081951.i08JpDh12979@NewBlue.scyld.com> Message-ID: <5.2.0.9.2.20040108164704.031c4aa8@mailhost4.jpl.nasa.gov> At 04:48 PM 1/8/2004 -0500, Andrew Piskorski wrote: >On Thu, Jan 08, 2004 at 11:34:26AM -0500, Robert G. Brown wrote: > > On Thu, 8 Jan 2004, Josip Loncaric wrote: > > > > I love NTP, but in its current state it does not deliver microsecond > > > level synchronization. > > > > NTP may need to be streamlined and tuned for clusters... > > > > I'm not sure that the problem is NTP itself -- from what I recall > > reading in its documentation it is CAPABLE of achieving 1 usec > > synchronization, over time. However, in order to do that it needs > > better tools than the current gettimeofday (and other time calls) which > > seem to deliver no better than perhaps millisecond resolution. It also > > very likely needs to be done not by a userspace daemon but by the > > kernel, as only the kernel can decide to do something in a true "single > >Would DJB's clockspeed, taiclock, etc. stuff be any better than ntp >for that? His website mentions using CPU hardware tick or nanosecond >counters, but doesn't define "precise" when it says that taiclock is >"... suitable for precise time synchronization over a local area >network ...": > > http://cr.yp.to/clockspeed.html > http://cr.yp.to/libtai.html Nope.. from what I can see in the source code, DJB doesn't have a hardware "tick" interface. I suppose, over a long enough time, the uncertainty introduced by grabbing the time from NTP, etc., would average out, but there's still a sync problem. >-- >Andrew Piskorski >http://www.piskorski.com/ >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 8 20:52:51 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 8 Jan 2004 17:52:51 -0800 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: <5.2.0.9.2.20040108160818.031eb830@mailhost4.jpl.nasa.gov> References: <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107134111.031ba668@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040107162049.03100008@mailhost4.jpl.nasa.gov> <3FFD7AEA.4060709@lanl.gov> <004e01c3d606$f29932e0$36a8a8c0@LAPTOP152422> <5.2.0.9.2.20040108101519.0310a300@mailhost4.jpl.nasa.gov> <5.2.0.9.2.20040108160818.031eb830@mailhost4.jpl.nasa.gov> Message-ID: <20040109015251.GA6567@greglaptop.internal.keyresearch.com> On Thu, Jan 08, 2004 at 04:14:38PM -0800, Jim Lux wrote: > >Perhaps I'm overly optimistic, but writing a kernel module to achieve > >cluster-wide microsecond-level clock synchronization just might be > >feasible... > > I am entirely certain that this can be done, and may have already been > done. It's such a useful thing to "know, exactly, what time it is".... Octiga Bay claimed to have done it at SC. However, they made it sound like they did a lot of work to get it going, as what they were doing was enabling MPI programs to run in lock-step on microsecond timescales. That's much harder than just getting global times. No doubt SGI has done something clever on their big Altix boxes. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jcownie at etnus.com Fri Jan 9 04:44:48 2004 From: jcownie at etnus.com (James Cownie) Date: Fri, 09 Jan 2004 09:44:48 +0000 Subject: [Beowulf] ...Re: Benchmark results In-Reply-To: Message from Greg Lindahl of "Thu, 08 Jan 2004 17:52:51 PST." <20040109015251.GA6567@greglaptop.internal.keyresearch.com> Message-ID: <1AetCC-1YI-00@etnus.com> Greg Lindahl wrote :- > On Thu, Jan 08, 2004 at 04:14:38PM -0800, Jim Lux wrote: > > > >Perhaps I'm overly optimistic, but writing a kernel module to achieve > > >cluster-wide microsecond-level clock synchronization just might be > > >feasible... > > > > I am entirely certain that this can be done, and may have already been > > done. It's such a useful thing to "know, exactly, what time it is".... > > Octiga Bay claimed to have done it at SC. However, they made it sound > like they did a lot of work to get it going, as what they were doing > was enabling MPI programs to run in lock-step on microsecond > timescales. That's much harder than just getting global times. But the OctigaBay system has a single network clock from which all the machine clocks are derived, so all of the processors are already synchronised. Given than architecture they can then easily ensure that timer interrupts are also synchronised on all nodes, and it's no big leap from there to co-scheduling the MPI processes. That's rather different from the problem being discussed here, which is how to get clock synchonisation in a Beowulf where you do _not_ have any common clock. (Indeed many of the solutions to that problem which are being suggested are basically to provide a common timing signal (clock input) to all the nodes...) -- Jim James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Fri Jan 9 04:54:43 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Fri, 9 Jan 2004 09:54:43 +0000 (GMT) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <200401090009.i0909Vh10376@NewBlue.scyld.com> References: <200401090009.i0909Vh10376@NewBlue.scyld.com> Message-ID: > --__--__-- > > > I am in the process of trying to get a stopgap perfomance monitoring > > system going on a 64 CPU Linux cluster with LSF. Just use LSF's own built in data monitoring. "lsload -l" will give you the load, memory, swap usage etc of every node in the cluster. You can extend the data collection with your own scripts (elim) to measure whatever else you want. For example, we monitor temperature and some database load information as well as the usual machine statistics. Data collection is done via a very simple perl script which collects lsload info at regular intervals, and then splats the data into rrdtool (www.rrdtool.com) for storing the historical data and producing graphs. Cheers, Guy Coates. -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 08:29:23 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 08:29:23 -0500 (EST) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <3FFDAB94.2070805@sonsorol.org> Message-ID: On Thu, 8 Jan 2004, Chris Dagdigian wrote: > > {Forwarded to this list on behalf of a friend with some email troubles...} > > > I am in the process of trying to get a stopgap perfomance monitoring > > system going on a 64 CPU Linux cluster with LSF. Ultimately, I hope to > > use PCP for data collection, but since nobody seems to be doing this yet, > > we are going to be rolling our own solution. To meet some of their needs, > > management has asked for an interim solution that gives them a web page > > with aggregate usage statistics and such. > > > > Unfortunately, ganglia is a non-starter because our networking group can't > > enable multicast for the private network the cluster lives on (it would > > break almost everything else). > > > > Does anyone have any suggestions for an alternative that can be quickly > > implemented, doesn't rely on multicast, and that can generate graphs of aggregate > > statistics on the fly? > > Regards, > Chris > BioTeam.net > > > http://www.phy.duke.edu/brahma/Resources/xmlsysd.php http://www.phy.duke.edu/brahma/Resources/wulfstat.php xmlsysd is a daemon that runs on each client. It parses /proc and runs certain systems commands on demand, extracts the requested data, wraps it up in an XML tagging that is fairly self-explanatory and easy/trivial to parse with any of a bunch of XML toolsets and libraries in a variety of scripting and compiler languages, and sends it back via a standard point-to-point TCP connection (no multicast) to the connecting host. It is throttleable -- it only returns the information you want to monitor, in fairly coarse-grained blocks (e.g. it tends to deliver all the information from a given /proc file all at once as the primary overhead is rewinding and reading the file at all, not parsing any given line once it is read, and the burden of sending a packet of 300 bytes and 750 bytes is similarly nearly identical). wulfstat is a provided tty/ncurses monitoring program that reads in an XML-based cluster descriptor/config file (it is very easy to define a cluster using what amount to scanf/printf wildcards and numerical ranges), connects to xmlsysd on all hosts specified in the config file, and builds a display that updates every five seconds or so of the entire cluster in a tty window that can be scrolled or paged. The default display currently presents a vmstat-like set of data, but one can also monitor only 5/10/15 load average, network stats, memory stats, cpu and time info (including duty cycle) and userspace running jobs (which can additionally be masked in the config file with -- you guessed it -- an XML-based set of tags). Wulfstat runs doesn't generate graphs of statistics on the fly, but because xmlsysd is trivial to connect to and control (you can telnet to the port and type in its control commands and see exactly what it returns and how to control/throttle configure it online) and because it returns xml-wrapped data, it would be would be easy, I think, to make it generate graphs and it will definitely scale to 64 nodes. In fact I think one could probably write a simple perl or python script that connects to the cluster xmlsysds, polls the cluster every (say) minute, extracts the quantities you wish to monitor, and write them out on a per-node basis to a file where any graphing utility you like could plot them. There are also almost certainly web/php widgets for plotting -- I just haven't had the need for them and so haven't written this myself (my nodes tend to go to load 1 per CPU and stay there for a month or so, which is pretty boring -- the "interesting" exceptions are easy to spot in real time in wulfstat). If you go this route and implement the fully GPL (v2b, where the b means you owe me a beer if and when we ever meet if you use it:-) toolset, please consider sending me any also GPL cool tools you develop that extend the usefulness of the basic xmlsysd and I'll wrap them up in the package for others. In fact, GPL v2b them and I'll have to buy YOU a beer back if/when we ever meet in human person with a bar handy...;-) rgb P.S. - I don't know exactly how many sites are using xmlsysd/wulfstat, but there are a few -- the tools have been around for a while and I do actively maintain them and have been known to add displays to wulfstat on request just to make a given user happy. One day I'll have LOTS of time on my hands I'm sure and will even port wulfstat to a gtk "real" GUI form, but the nice thing about a tty based display is that it works on a basic tty console if no X is running at all (which may well happen) and everybody has a vast range of xterm choices under X, so it is definitely the common denominator of monitor interfaces. -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 09:06:52 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 09:06:52 -0500 (EST) Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results In-Reply-To: <20040108225352.GA29682@galactic.demon.co.uk> Message-ID: On Thu, 8 Jan 2004, Andrew M.A. Cater wrote: > Maplin electronics did a kit - cheaply - to run off the time code from > 60kHz clocks and put it into the serial/parallel port. Know any > radio hams - a lot are taking a temperature controlled crystal > oscillator and phase locking to WWWV or equivalent. This is useful to know. I'd really like to set up a local ntp server in my house and it wouldn't hurt to have one in the department at Duke, but not for hundreds of dollars in either case. A 60 KHz signal really sounds like one should be able to tune it and amplify it and deliver it as a pulse accurate within an easy millisecond with a five dollars worth of electronics (coil, cap, rectifier, amplifier and maybe a serial interface chip). I just don't have the time to do a lot of hobby electronics -- I have a really nifty thermal sensor kit upstairs that I've had for six months now without having had time to build it, for example. > > nsec syncronization on a global scale has lots of benefits, some of > > which could enable projects that would dwarf SETI at home. The heck with > > turning every cellular tower in the country into a giant radiotelescope > > array -- turn every suitably equipped PC in the country into a receiver > > in a really enormous radiotelescope array. Now there is a project for a > > GRID! > > Haven't you heard - the Govt. is using them to track individuals by > their individual radar fingerprint. They can watch me wherever I walk > and they're using the system to mess with my mind ... :) Time to take your medication, there, there, it will be all right...;-) > > Unfortunately, a lot of "random" physical sources turn > > out not be random at all, just to have a relatively short > > autocorrelation function and to have distributions that are not uniform > > as well. A lot of the noise-based generators fall into this category, > > but quantum generators can as well (one has to look at second and higher > > order correlation functions, for example -- even resonance fluorescence, > > a textbook example of a "random quantum process" turns out to be photon > > antibunched, and there is also the venerable work of Hanbury Brown and > > Twiss. It turns out not to be horribly easy to build a hardware > > generator that passes diehard or the NIST suite, let alone one that > > works fast. > > > > The Knuth random number generator that converged to a value - the first > example in Seminumerical Algorithms springs to mind :) It is certainly worth remembering that in a deterministic universe (and we DO live in a deterministic universe according to physics, regardless of what one is tempted to conclude by looking at a quantum system that is "inside" a generalized master equation and is analyzed with non-relativistic quantum as well) there are no random number generators or random events, period. There is only entropy, which the way I learned it is the log of the missing information. Missing information is an expression of our degree of ignorance of true state given a limited/projective view of a large, multivariate space, not a statement that the information is in any sense not there. I do wish that more quantum classes taught the GME and quantum statistics properly, I do I do... Marsaglia apparently shares this view. IIRC it is him that I'm semiquoting when I say that hardware RNG's often aren't, at least according to diehard. Even with entropy generators and thermal noise, the RNG has to be BOTH unpredictable AND uniform at the bit level to all orders, and the latter is actually not terribly easy to accomplish without a pure random binary process at hand. I've written this nifty random number generator rating program (as always, alas, functional but unfinished in terms of original design goals) that when finished will encapsulate all the diehard tests and all the NIST STS/FIPS tests in an shell that permits them to be automagically applied to any GSL-wrapped RNG. It also permits the RNGs to be timed for benchmarking purposes (using the cpu_rate timer I published on list a few days ago). Leaving ppm effects aside :-) one discovers that the linux /dev/random entropy generator is pretty good but slow as molasses (not surprising). The Mersenne twister rocks. GSL rocks -- there are over 50 RNG's pre-wrapped and ready to go, including lots of the old/famous failures. It's kind of fun to test some of the original rand()'s and watch them fail all over the place... rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 10:52:26 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 07:52:26 -0800 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C01E@orsmsx402.jf.intel.com> From: Robert G. Brown; Friday, January 09, 2004 5:29 AM > > On Thu, 8 Jan 2004, Chris Dagdigian wrote: > > > > > {Forwarded to this list on behalf of a friend with some email > troubles...} > > > > > I am in the process of trying to get a stopgap perfomance monitoring > > > system going on a 64 CPU Linux cluster with LSF. Ultimately, I hope > to > > > use PCP for data collection, but since nobody seems to be doing this > yet, > > > we are going to be rolling our own solution. To meet some of their > needs, > > > management has asked for an interim solution that gives them a web > page > > > with aggregate usage statistics and such. I'm not sure what is meant by "nobody seems to be doing this yet". Is "this" read as "using pcp" or "using pcp with LSF"? At any rate, CluMon http://clumon.ncsa.uiuc.edu/ uses PCP to collect data, along with Apache, PHP, and MySQL to manage the data. The only issue with the specific request is that it's tied to PBS as the queuing system. That shouldn't be too hard to eliminate or port to LSF. Alternatively, just use the data collection facilities of LSF directly, and while you're at it, use the GUI facilities of LSF to display the data. Alternatively, if the LSF GUI doesn't work for you, just build the GUI you want using the LSF data. Is there a compelling reason to collect the data twice? > > > Unfortunately, ganglia is a non-starter because our networking group > can't > > > enable multicast for the private network the cluster lives on (it > would > > > break almost everything else). I guess the network isn't that private to the cluster then. Otherwise, how would activities on a private network, at best accessible only via the routing on the headnode, impact the rest of the network? But, I digress, challenging IT policy ;^) > One day I'll have LOTS of > time on my hands I'm sure and will even port wulfstat to a gtk "real" > GUI form, but the nice thing about a tty based display is that it works > on a basic tty console if no X is running at all (which may well happen) > and everybody has a vast range of xterm choices under X, so it is > definitely the common denominator of monitor interfaces. I'd argue the web interface is the common denominator. Viewable via text browser, plus you get out of the OS wars... Yes, I know that real sys admins *never* use a web interface... -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From M.Arndt at science-computing.de Fri Jan 9 04:54:51 2004 From: M.Arndt at science-computing.de (Michael Arndt) Date: Fri, 9 Jan 2004 10:54:51 +0100 Subject: [Beowulf] profiling memory bandwidth of a job Message-ID: <20040109105451.A32360@blnsrv1.science-computing.de> Hello, Problem: to get the characteristics of a commercial numerical CFD code on a Cluster with respect to memory bandwidth. The Application is running single cpu jobs on SMP Nodes, no parallel application. The runtimes show symptomes of concurring about memory bandwith. I.e., if two One-CPU Jobs run on the dual CPU Node, the total run time is far more than twice the time of one job + e.g 30 % penalty for ressource competition of both jobs on a SMP Node. Question: Is it possible, to show with standard linux profiling commands that really memory bandwidth is the issue ? (Memory requirements and disk IO rate of the jobs are moderate, as far as iostat / top shows) thx Micha _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 11:57:41 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 11:57:41 -0500 (EST) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <187D3A7CAB42A54DB61F1D05F012572201D4C01E@orsmsx402.jf.intel.com> Message-ID: On Fri, 9 Jan 2004, Lombard, David N wrote: > I'd argue the web interface is the common denominator. Viewable via > text browser, plus you get out of the OS wars... For users, David, I agree, although we tend to be linux-centric and wulfstat is popular enough. procstatd had a perl-cgi web interface written by a friend here at Duke and I'm sure it would be easy enough to port. > Yes, I know that real sys admins *never* use a web interface... It isn't never, but I did write this as a pretty high end administrative/monitoring interface that would be useful to sysadmins running lans and not just clustering people running clusters. I also wrote/designed it so that it would be easy to write things like web interfaces for it. I just haven't got the time (or TOO much motivation) to do so myself, as I don't have users clamoring for it. Somebody (I can't offhand recall who) was looking at it to possibly wrap it up to work within ganglia -- one way to get a web interface. From what I've seen of the ganglia interface, though, wulfstat is MUCH more practically useful for most cluster users or administrators as it stands, xterm or not. Icons and nifty bar charts are pretty, but take up a lot of room. A grid of regularly updated number cells is ugly, but oh, so informative and compact. The current default display is something like (sorry for the ncurses cut/paste ugliness, but hey, it does cut and paste and the column/boxes do actually align:-): name |st|Load 1,5,15|eth0 rx|eth0 tx| si/so | pi/po |ctxt|intr|users ======================================================================== ganesh |up|0.0,0.0,0.0| 8415 | 882 | 0/0 | 0/2 | 156| 160| x c00 |up|2.0,2.0,2.0| 330 | 676 | 0/0 | 0/2 | 9| 105| x c01 |up|2.0,2.0,2.0| 330 | 675 | 0/0 | 0/2 | 9| 105| x ... status, load, network traffic (ganesh's is relatively large because it is receiving all of the xmlsysd data from the many monitored nodes), swapin/swapout, pagein/pageout, context switches, interrupts (users doesn't do anything yet but is intended to somehow indicate what and whose jobs are running...already available in detail on another display). At a glance and without a mouse or even a keyboard used for paging, one has a very good chance of identifying cluster problems on as many hosts as will fit in a tty window on as many tty windows as one can fit on a screen -- hundreds, at least. Usually problems show up as downed nodes, anomalous load averages, unusual network traffic, lots of swapping or paging (indicating memory problems) or lots of ctxt/intr activity (indicating a possible thrashing kernel task, I/O where there shouldn't be, etc.). I do have a mental vision of an even more efficient "panel of lights" display that would run under X as a GTK process that used "temperature" metaphor colors to encode the number values, green to red and still presented the number values if the mouse pointer hovered over a light -- such a display could probably allow a really big cluster (1024 nodes plus) to be monitored at a glance with detail readily available as problems emerged. I just don't have that many nodes that wulfstat doesn't work locally, and writing the app and debugging it is maybe a week's work. With such a display and lights that were order of 10-20 pixels square, one could put a LOT of them onscreen in a neat little array... I don't know if this would be possible in a web browser, at least on a timescale you could live with. Binary can have portability issues, but it is as fast as it gets. rgb > > -- > David N. Lombard > > My comments represent my opinions, not those of Intel Corporation. > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 11:16:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 11:16:08 -0500 (EST) Subject: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results (fwd) Message-ID: ---------- Forwarded message ---------- Date: Fri, 09 Jan 2004 09:22:44 -0600 From: Gerry Creager N5JXS To: Robert G. Brown Subject: Re: Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results Check http://www.synergy-gps.com/ and see what they have in their used and refurbed sales pile for an OnCore. There's plenty of info on NTP using an OcCore receiver with 1pps output. Alternately, look for Tom Clark's TAC-2 (Totally Accurate Clock) and an OnCore board or Garmin (but the Garmin's more expensive). Tom's reported some stellar examples of short- and long-term accuracy. If this is interesting enough, please feel free to repost it for me. If not, flush it... I'm off to Seattle to act like a meteorologist for the next week... TTFN, gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 Seems worth it to me, rgb _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 14:15:48 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 11:15:48 -0800 Subject: [Beowulf] profiling memory bandwidth of a job Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C024@orsmsx402.jf.intel.com> From: Michael Arndt; Friday, January 09, 2004 1:55 AM > > Hello, > > Problem: to get the characteristics of a commercial numerical > CFD code on a Cluster with respect to memory bandwidth. > > The Application is running single cpu jobs on SMP Nodes, no > parallel application. > > The runtimes show symptomes of concurring about memory bandwith. > > I.e., if two One-CPU Jobs run on the dual CPU Node, the total run > time is far more than twice the time of one job + e.g 30 % penalty for > ressource competition of both jobs on a SMP Node. > > Question: Is it possible, to show with standard linux profiling commands > that really memory bandwidth is the issue ? > > (Memory requirements and disk IO rate of the jobs are moderate, as far > as iostat / top shows) I'm not sure about profiling tools, but running the job on systems with differing memory configurations (and nothing else) can certainly quantify the impact of memory bandwidth. You also state that disk I/O is a potentially significant factor. If it is, which is a little odd for a CFD app, you also need to consider that. Also look at memory size as an issue potentially affecting multiple jobs. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeffrey.baker at lmco.com Fri Jan 9 14:47:02 2004 From: jeffrey.baker at lmco.com (Baker, Jeffrey) Date: Fri, 09 Jan 2004 13:47:02 -0600 Subject: [Beowulf] Upcoming Beowulf Presentation - Help! Message-ID: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Howdy, all... I've just been selected to give a presentation on 1/27 to the entire IT department about Beowulf's -- of which I know nothing about! I'm interested in finding out what good books or websites are out there that can get me up to "presentation speed" in short order ("Beowulfs for Dummies") ... So far, I've found the Beowulf history < http://beowulf.gsfc.nasa.gov/overview.html > ... Beowulf FAQ < http://www.canonical.org/~kragen/beowulf-faq.txt > ... and Engineering a Beowulf-style Compute Cluster < http://www.phy.duke.edu/brahma/beowulf_book/ > ... all which should help tremendously. Any other sites I should know about? Many thanks! Jeff in Dallas _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 9 14:58:01 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 9 Jan 2004 14:58:01 -0500 (EST) Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast In-Reply-To: <187D3A7CAB42A54DB61F1D05F012572201D4C020@orsmsx402.jf.intel.com> Message-ID: On Fri, 9 Jan 2004, Lombard, David N wrote: > By timescale, I assume you mean display time? That's where the > abstractions come in to play, and, for the all-at-once method, better > output preparation, i.e., a single bit image over multiple images. So, > I'd suggest that the only thing you lose is live-update, which can be an > issue, to be sure. I don't think that we disagree, really. I freely admit there are times where a web view of a cluster is adequate and they certainly tend to be very easy to understand. It is really the quintessential Macintoad tradeoff, epitomized in the epigrammatic "You can learn to use a Mac in a day, then pay for that knowledge for the rest of your life" (anon). Really, this applies to GUIs in general, not just Macs -- there are certain things for which a GUI abstraction are really good for, but often at the expense of speed, or power, or complexity. A command line interface and Unix itself are described as being "expert friendly" -- difficult to learn, complex, but SO powerful once you do learn them. A GUI browser is very lovely for certain kids of browsing, but if you want to move all the files in a directory with the glob foo*.bar.?arian-[0-9]* into the trash, gimme a command line every time, especially if there are a few thousand of them. Still, try EXPLAINING how to do that to a thirteen year old compared to saying "pick up the file with the mouse and drag it into the trash, silly" and the virtue of GUIs as well becomes clear. Each has its place. xmlsysd is designed to support both -- it is divorced from the display mechanism entirely. Connect to it, talk to it, ask it nicely, it spits out xml-tagged anonymous data you might be interested in monitoring. Do whatever you like with this data -- present it via TTY, pipe it to a file, graph it, whatever. I invested my time in the tty interface THIS time because I made the mistake of building a GUI interface first LAST time with procstatd (xmlsysd's predecessor) and spent a year or two later kicking myself for a variety of reasons. The tty interface is fast and information-dense, but has flaws as well (it is a pain to rearrange and customize, for example). If I ever have time, as I said, I'll write a better interface (and maybe even two or three). In the meantime, anyone who wants them sooner can play through -- it is really pretty simple;-) rgb > > -- > David N. Lombard Sr. Staff Software Engineer 1.949.567.4843 > Parallel & Distributed Processing Division, Intel Corporation > > My comments represent my opinions, not those of Intel Corporation. > > Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 15:57:12 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 12:57:12 -0800 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C027@orsmsx402.jf.intel.com> From: Robert G. ; Friday, January 09, 2004 11:58 AM > > On Fri, 9 Jan 2004, Lombard, David N wrote: > > > By timescale, I assume you mean display time? That's where the > > abstractions come in to play, and, for the all-at-once method, better > > output preparation, i.e., a single bit image over multiple images. So, > > I'd suggest that the only thing you lose is live-update, which can be an > > issue, to be sure. > > I don't think that we disagree, really. Actually, you're preaching to the choir. > Really, this applies to GUIs in general, not just Macs -- there are > certain things for which a GUI abstraction are really good for, but > often at the expense of speed, or power, or complexity. I agree fully. A GUI is the best way to display massive amounts of info, especially where you want to be able to quickly vary the level of detail, like in a comprehensive view of a cluster with the ability to drill down at many points. Nothing new -- use text when you want relatively small amounts of detail, and graphics when you want trends and large amounts of information. > A command line interface and Unix itself are described as being "expert > friendly" -- difficult to learn, complex, but SO powerful once you do > learn them. Amen brother. I very much prefer Linux, and have a long and pleasent history in UNIX. > A GUI browser is very lovely for certain kids of browsing, > but if you want to move all the files in a directory with the glob > foo*.bar.?arian-[0-9]* into the trash, gimme a command line every time, > especially if there are a few thousand of them. Still, try EXPLAINING > how to do that to a thirteen year old compared to saying "pick up the > file with the mouse and drag it into the trash, silly" and the virtue of > GUIs as well becomes clear. Each has its place. Violently agreed, both points. > xmlsysd is designed to support both -- it is divorced from the display > mechanism entirely. Connect to it, talk to it, ask it nicely, it spits > out xml-tagged anonymous data you might be interested in monitoring. Do > whatever you like with this data -- present it via TTY, pipe it to a > file, graph it, whatever. I invested my time in the tty interface THIS > time because I made the mistake of building a GUI interface first LAST > time with procstatd (xmlsysd's predecessor) and spent a year or two > later kicking myself for a variety of reasons. The tty interface is > fast and information-dense, but has flaws as well (it is a pain to > rearrange and customize, for example). That was exactly my motivation for the web comment. I spent a fair amount of time putting together web-based interfaces for command-line driven tools, and I've found that's the ideal combination for both system administration and users. Develop the tool in an environment that favors utility and (most importantly) composability, and then _optionally_ interact with the tool in an environment that favors usability and accessibility. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Fri Jan 9 13:47:51 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Fri, 9 Jan 2004 10:47:51 -0800 Subject: [Beowulf] query: aggregate cluster performance monitoring without multicast Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C020@orsmsx402.jf.intel.com> From: Robert G. Brown [mailto:rgb at phy.duke.edu]; Friday, January 09, 2004 > > I do have a mental vision of an even more efficient "panel of lights" > display that would run under X as a GTK process that used "temperature" > metaphor colors to encode the number values, green to red and still > presented the number values if the mouse pointer hovered over a light -- > such a display could probably allow a really big cluster (1024 nodes > plus) to be monitored at a glance with detail readily available as > problems emerged. I just don't have that many nodes that wulfstat > doesn't work locally, and writing the app and debugging it is maybe a > week's work. With such a display and lights that were order of 10-20 > pixels square, one could put a LOT of them onscreen in a neat little > array... There's more to GUI scalability than small icons. The best scalability comes from being able to range from "show me everything all at once" to "roll up the data in various meaningful terms". Possibilities like machine rooms and racks are things I've prototyped and received generally favorable responses. > I don't know if this would be possible in a web browser, at least on a > timescale you could live with. Binary can have portability issues, but > it is as fast as it gets. By timescale, I assume you mean display time? That's where the abstractions come in to play, and, for the all-at-once method, better output preparation, i.e., a single bit image over multiple images. So, I'd suggest that the only thing you lose is live-update, which can be an issue, to be sure. -- David N. Lombard Sr. Staff Software Engineer 1.949.567.4843 Parallel & Distributed Processing Division, Intel Corporation My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From apseyed at bu.edu Fri Jan 9 15:20:39 2004 From: apseyed at bu.edu (apseyed at bu.edu) Date: Fri, 09 Jan 2004 15:20:39 -0500 Subject: [Beowulf] diskless cluster nodes ? Message-ID: <1073679639.3fff0d17d203d@www.bu.edu> What type of jobs would be typical of a scenerio where diskless nodes (nodes with no hard drives/pxe booting the kernel) would be most optimal. I understand that the jobs to be run on our future cluster don't do alot of I/O, i.e. once running it does alot of computations for a long period of time, therefore after loading into memory not much disk access. However, we have a couple users utilizing MPI.. Any thoughts on this appreciated. -P _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Fri Jan 9 15:06:04 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Fri, 09 Jan 2004 15:06:04 -0500 Subject: [Beowulf] profiling memory bandwidth of a job In-Reply-To: <20040109105451.A32360@blnsrv1.science-computing.de> References: <20040109105451.A32360@blnsrv1.science-computing.de> Message-ID: <1073678764.8918.21.camel@squash.scalableinformatics.com> On Fri, 2004-01-09 at 04:54, Michael Arndt wrote: > I.e., if two One-CPU Jobs run on the dual CPU Node, the total run > time is far more than twice the time of one job + e.g 30 % penalty for > ressource competition of both jobs on a SMP Node. So you believe that you are running out of memory bandwidth? > > Question: Is it possible, to show with standard linux profiling commands > that really memory bandwidth is the issue ? You would need to use tools like Oprofile, or Troy Baer's lperfex (works with perfctr, which requires a kernel patch last I looked). > (Memory requirements and disk IO rate of the jobs are moderate, as far > as iostat / top shows) Some notes: 1) 'vmstat 1' is your friend. Use it to see at a gross level, what is going on with the system. Is it possible you are swapping? Is it possible that you are hitting the interrupts hard (don't know why), or doing lots of context switching? 2) The linux tools won't give you a fine grain view, you will need to go to something like Oprofile. 3) Before you do two runs, it might be advisable to create a baseline run, and profile it. Look for the usual suspects (paging, memory, interrupts, ...). Gather profile data. Unless you have source for your commercial code, you probably would not be able to get the code to generate profile data, so you are left with the other tools. > > thx > Micha > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Fri Jan 9 17:24:17 2004 From: becker at scyld.com (Donald Becker) Date: Fri, 9 Jan 2004 17:24:17 -0500 (EST) Subject: [Beowulf] Beowulf List adminstrivia: SPF now published for beowulf.org Message-ID: The beowulf domain now publishes Sender Permitted From records. Detailed information on SPF is at http://spf.pobox.com/ but here is a quick summary: SPF is an advisory field in the domain name record that can help mail receipients reject spam that pretends to come from popular mailers such as beowulf.org. SPF doesn't require you to do anything. It does allow you or your ISP to set up better spam filtering rules, if you choose. To see what SPF "really does", run a DNS name server query on beowulf.org and check out the "additional text" records. There is a record for SPF version 1: beohost$ host -t txt beowulf.org beowulf.org. text "v=spf1 a mx ptr -all" This says "If you get a message from beowulf.org, it must come from A the primary host for beowulf.org MX the mail exchange hosts for beowulf.org PTR any host whose domain name ends in .beowulf.org -all and no other machine I don't expect even the slightest inconvenience from providing SPF records for beowulf.org. Please let me know if it causes any problems. -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster systems Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rpickler at dnahelix.com Fri Jan 9 16:59:54 2004 From: rpickler at dnahelix.com (Rich Pickler) Date: Fri, 09 Jan 2004 15:59:54 -0600 Subject: [Beowulf] Recommendations for new cluster Message-ID: <3FFF245A.7070804@dnahelix.com> We're looking a few months into the future and are beginning to design our next renderfarm for production at our studio. Instead of going with a render system tailored specifically for the renderer we're using, like Alfred for Pixar's RenderMan http://www.pixar.com/, we're toying with the idea of a more generalized cluster system which will open up a ton of possibilities for what we can do with the farm. However, having only tinkered with generalized cluster systems during college, I am a bit new to some of the details and what is right for us. After exhausting Google for all I can get out of it, I have some questions over what exactly is right for us. And I would rather have real world experiences with tools instead of someone trying to sell me something. In a nutshell, the questions we have: -What batch system do you recommend for our purposes (Condor, PBSPro, ScalablePBS, SGE)? -What monitoring tools and set up tools have you found and used that reduce down time on faulty nodes (ganglia, management tools, etc.)? -Fail over of the head node is also important to us, but I haven't researched enough on this topic to start asking questions. (So far, I've only found stuff for SGE, and a rumor that PBSPro will add this feature soon) A bit of detail on our requirements: First, we're looking at a cluster of around 400 CPUs, which is getting dangerously close to the limits of OpenPBS. Considering that this number could easily grow down the road, pushing that limit as far back as possible is needed. So far, PBS looks like the most widely used and known system out there, so some variant of that seems ideal for finding prospective employees with cluster experience, although it looks like a lot of you deal with SGE as well. PBSPro and ScalablePBS advertise they get the scalability limit much higher than what we need, but the first is much more expensive. I have yet to find any documentation or rumor of SGE's upper bound. The majority of jobs in our queue will be small and single node (but definitely multithreaded to take advantage of dual cpu nodes). This means intercommunication between nodes is almost nil. However, there will be cases where we need processes to communicate outside the cluster with workstations, but that isn't terribly difficult to design around. One of the problems I have noticed is that some batch managers bog down with a large amount of jobs. Looking at the structure of our jobs, we can easily wind up with 20,000-40,000 jobs in a day with a large amount of them being submitted to the batch around the same time (when everybody leaves work). Any thoughts what might be the best solution for us? Thanks for any recommendations you can give me, and feel free to ask me any more details you want. Rich _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctierney at hpti.com Fri Jan 9 19:07:12 2004 From: ctierney at hpti.com (Craig Tierney) Date: 09 Jan 2004 17:07:12 -0700 Subject: [Beowulf] Latency issues with Itanium2 (Tiger2) and e1000 gigE Message-ID: <1073693232.4266.14.camel@hpti7.fsl.noaa.gov> I am having some problems with some new servers I just got. I have several dual Itanium2 (Intel Tiger2 board) connected via gigE through a Dell Powerconnect 5224 gigE switch. Using mpptest to test latency and bandwidth the results are not so good. Latency varies between 60-200 mu/s for small packets, and the bandwidth tops out at about 45 MB/s. - This happens with both a Debian and White Box (rebuilt from RHAS 3.0 sources). - I have a similar set of Opteron nodes that use the same model switch (different phyiscal switch though) and the latency and bandwidth of the built in tg3 card is 32 mu/s and and 100 MB/s respectively. - I have verified (at least on the hosts) that both are in full-duplex mode. Does anyone have any suggestions on what else to look for? Thanks, Craig _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laurenceliew at yahoo.com.sg Fri Jan 9 18:46:00 2004 From: laurenceliew at yahoo.com.sg (Laurence Liew) Date: Sat, 10 Jan 2004 07:46:00 +0800 Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> References: <3FFF245A.7070804@dnahelix.com> Message-ID: <1073691959.914.11.camel@scalable> We use SGE with much success and little problems. As a builder of clusters, we support many customers.. and installing SGE for customers since late 2001 has meant a lot less hassle for us in terms of support, reliability and robustness.... previously with another scheduler... we had to visit our customers quite a bit.. SGE also supports shadow/secondary master nodes for failover of the master SGE itself.. it may be important to you Condor is also a good scheduler Ganglia for monitoring. We have used Rocks (www.rocksclusters.org) with SGE to quickly build a renderfarm (smallish 64 CPU) in under 1 day incorporating Renderman. If checkpoint and restart is important to you at the head and compute node level, you may wish to take a look at www.meiosys.com. These guys have an interesting virtualisation technology which allows transparent checkpointing and restart mechanism... this means you do not need specialised binaries... most applications should run ontop of their technology. Have fun! Cheers! laurence On Sat, 2004-01-10 at 05:59, Rich Pickler wrote: > We're looking a few months into the future and are beginning to design > our next renderfarm for production at our studio. Instead of going with > a render system tailored specifically for the renderer we're using, like > Alfred for Pixar's RenderMan http://www.pixar.com/, we're toying with > the idea of a more generalized cluster system which will open up a ton > of possibilities for what we can do with the farm. However, having only > tinkered with generalized cluster systems during college, I am a bit new > to some of the details and what is right for us. After exhausting > Google for all I can get out of it, I have some questions over what > exactly is right for us. And I would rather have real world experiences > with tools instead of someone trying to sell me something. > > In a nutshell, the questions we have: > > -What batch system do you recommend for our purposes (Condor, PBSPro, > ScalablePBS, SGE)? > > -What monitoring tools and set up tools have you found and used that > reduce down time on faulty nodes (ganglia, management tools, etc.)? > > -Fail over of the head node is also important to us, but I haven't > researched enough on this topic to start asking questions. (So far, > I've only found stuff for SGE, and a rumor that PBSPro will add this > feature soon) > > A bit of detail on our requirements: > > First, we're looking at a cluster of around 400 CPUs, which is getting > dangerously close to the limits of OpenPBS. Considering that this > number could easily grow down the road, pushing that limit as far back > as possible is needed. So far, PBS looks like the most widely used and > known system out there, so some variant of that seems ideal for finding > prospective employees with cluster experience, although it looks like a > lot of you deal with SGE as well. PBSPro and ScalablePBS advertise they > get the scalability limit much higher than what we need, but the first > is much more expensive. I have yet to find any documentation or rumor > of SGE's upper bound. > > The majority of jobs in our queue will be small and single node (but > definitel multithreaded to take advantage of dual cpu nodes). This > means intercommunication between nodes is almost nil. However, there > will be cases where we need processes to communicate outside the cluster > with workstations, but that isn't terribly difficult to design around. > > One of the problems I have noticed is that some batch managers bog down > with a large amount of jobs. Looking at the structure of our jobs, we > can easily wind up with 20,000-40,000 jobs in a day with a large amount > of them being submitted to the batch around the same time (when > everybody leaves work). > > Any thoughts what might be the best solution for us? > > Thanks for any recommendations you can give me, and feel free to ask me > any more details you want. > > Rich > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Fri Jan 9 21:17:51 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 10 Jan 2004 10:17:51 +0800 (CST) Subject: [Beowulf] Upcoming Beowulf Presentation - Help! In-Reply-To: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Message-ID: <20040110021751.5788.qmail@web16813.mail.tpe.yahoo.com> beowulf is good for HPC clusters running parallel jobs. If you use the cluster like a compute farm, then make sure you also install a DRM (distributed resource management, or a batch system) on top of the beowulf. Common ones like SGE (Sun's open source DRM), SPBS (much much better than PBS), or LSF (commercial). I would suggest new DRM installations to choose SGE instead of LSF, since SGE is free, and SGE 6.0 is coming soon and fixes some limitations that some people encounter. LSF is quite similar to SGE, but it is very expensive (last time I heard it costed $500 US per CPU, and support fee is not cheap), and for the price of 1 CPU license, I can get another Linux box with SGE, and I won't loose much. SPBS is also good, it is free, and used by a lot of HPC sites. They used to use OpenPBS, but OpenPBS's development is basically dead, and a lot of bugs not fixed! Andrew. --- "Baker, Jeffrey" ????> Howdy, all... > > I've just been selected to give a presentation on > 1/27 to the entire IT department about Beowulf's -- > of which I know nothing about! > > I'm interested in finding out what good books or > websites are out there that can get me up to > "presentation speed" in short order ("Beowulfs for > Dummies") ... > > So far, I've found the Beowulf history < > http://beowulf.gsfc.nasa.gov/overview.html > > ... Beowulf FAQ < > http://www.canonical.org/~kragen/beowulf-faq.txt > > ... and Engineering a Beowulf-style Compute Cluster > < http://www.phy.duke.edu/brahma/beowulf_book/ > > ... all which should help tremendously. > > Any other sites I should know about? > > Many thanks! > > Jeff in Dallas > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Fri Jan 9 21:05:36 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 10 Jan 2004 10:05:36 +0800 (CST) Subject: [Beowulf] Apple gets it right this time? Message-ID: <20040110020536.37964.qmail@web16802.mail.tpe.yahoo.com> x-serve with ECC, 42 of those in a rack. http://www.apple.com/xserve/ Also, Apple previews x-grid: http://www.apple.com/acg/xgrid/ Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sat Jan 10 00:20:10 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 10 Jan 2004 13:20:10 +0800 (CST) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> Message-ID: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> --- Rich Pickler ???? > Instead of going with a render system tailored > specifically for the renderer we're using, like > Alfred for Pixar's RenderMan http://www.pixar.com/, You are not alone, Framestore CFC is doing the same thing. They are looking at using SGE 6.0 to replace Alfred. You can follow the mail thread -- "SGE Renderfarm": http://gridengine.sunsource.net/servlets/ReadMsg?msgId=14218&listName=users (Since you are in this graphics industry, can you tell me if Framestore is big? People on the sge list looked so excited.) > And I would rather have real world experiences > with tools instead of someone trying to sell me > something. Use Opensource (r), and no one will ask you for money. > -What batch system do you recommend for our purposes > (Condor, PBSPro, ScalablePBS, SGE)? SGE has array jobs, but all others do not. An array job is submitted as a single job, but each element of the array takes different inputs. It is much faster to submit a single array job than to submit each element as seperately. And you can stop and del the whole array job or just each individual element. AFAIK, this job array feature makes graphics rendering and certain types of bio-computation job management easily -- e.g. different frames are part of the same job array. > -What monitoring tools and set up tools have you > found and used that reduce down time on faulty nodes > (ganglia, management tools, etc.)? ganglia is good, a lot of people use it. I just think setting up a webpage is the overkilled part -- need to run Apache to monitor the cluster -- but just MHO only. If you like command line stuff better, you can use qhost in SGE to see which host is down. > I've only found stuff for SGE, and a rumor that > PBSPro will add this feature soon) Yes, PBSPro 5.4 will support it. > So far, PBS looks like the most widely used and > known system out there, so some variant of that > seems ideal for finding prospective employees with > cluster experience, I find PBS is used more in HPC clusters (MPI, PVM jobs), and SGE is used more in enterprise and biocomputation clusters. The problem with SGE when handling MPI jobs is that it takes time to set up the PE, and PBS has an external interface which is much nicer. People had already written mpiexec to take advantage of the interface. > I have yet to find any documentation or rumor > of SGE's upper bound. SGE scales quite well, I haven't seen people complaining about it, may be that's why you couldn't find it. Just to let you know, Forecast Systems Laboratory has a cluster using SGE, it has 1536-CPUs. They gave a presentation of their cluster last year. SGE 6.0 (coming this year) will scale even further (like multithread SGE, and use a DBMS to handle job spooling), you can take a lot at Sun's presentations at: http://gridengine.sunsource.net/project/gridengine/workshop22-24.09.03/proceedings.html > One of the problems I have noticed is that some > batch managers bog down with a large amount of > jobs. Make sure you have lots of memory in the head node, and no swapping. I used to try how SGE behaves when there are tons of submitted jobs, I went up to 100,000 and it was still running, but of course it was just a toy cluster with several hosts. However, I think 20,000-40,000 can be handled very easily. > Any thoughts what might be the best solution for us? I think the best is to try it yourself, as it all depends on what you want to do. Download the SGE binaries (or source, if you really wanted to), and install it on one of the test machines. It doesn't require root access, and it can be placed any directory. It's free, all you need is time. Also, I highly recommend the mailing lists, a lot of people are on the lists, and can solve problems quickly. http://gridengine.sunsource.net/project/gridengine/maillist.html Like the "SGE Renderfarm" thread, they exchange experience and share knowledge very often. Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 04:52:35 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 10:52:35 +0100 (CET) Subject: [Beowulf] Upcoming Beowulf Presentation - Help! In-Reply-To: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Message-ID: On Fri, 9 Jan 2004, Baker, Jeffrey wrote: > Howdy, all... > > I've just been selected to give a presentation on 1/27 to the entire IT department about Beowulf's -- of which I know nothing about! > > I'm interested in finding out what good books or websites are out there that can get me up to "presentation speed" in short order ("Beowulfs for Dummies") ... > > So far, I've found the Beowulf history < http://beowulf.gsfc.nasa.gov/overview.html > > ... Beowulf FAQ < http://www.canonical.org/~kragen/beowulf-faq.txt > > ... and Engineering a Beowulf-style Compute Cluster < http://www.phy.duke.edu/brahma/beowulf_book/ > > ... all which should help tremendously. > Useful sites indeed. In addition to these, you might look at the books: "Beowulf Cluster Computing with Linux" , editied by Thomas Sterling. published by MIT Press "Linux Clustering" by Charles Bookman, published by New Riders. The second one is an easier introduction. Good Luck with the presentation! _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Sat Jan 10 05:38:32 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: 10 Jan 2004 10:38:32 +0000 Subject: [Beowulf] Latency issues with Itanium2 (Tiger2) and e1000 gigE In-Reply-To: <1073693232.4266.14.camel@hpti7.fsl.noaa.gov> References: <1073693232.4266.14.camel@hpti7.fsl.noaa.gov> Message-ID: <1073731111.2389.22.camel@fermi> On Sat, 2004-01-10 at 00:07, Craig Tierney wrote: > I am having some problems with some new servers I just > got. I have several dual Itanium2 (Intel Tiger2 board) > connected via gigE through a Dell Powerconnect 5224 gigE > switch. > > Using mpptest to test latency and bandwidth the results > are not so good. Latency varies between 60-200 mu/s for > small packets, and the bandwidth tops out at about 45 MB/s. > > - This happens with both a Debian and White Box (rebuilt > from RHAS 3.0 sources). > Does anyone have any suggestions on what else to look for? Yes. I suppose the networking chip is an Intel model - something like a 82546 or 82547 running with the e1000 driver? If this is the case, and the kernel module is inserted w/o any special options, "Interrupt Throttling" kicks in. In the default case Intel has implemented a heuristical approach, which limits the number of interrupts per sec the NIC produces. This is done to lower the CPU overhead in high volume communications. Excellent for large transfers but really bad for low-latency communication :-( What we usually do is setting the InterruptThrottleRate to something other than the default value, 1. Look here for information: http://www.intel.com/support/network/adapter/1000/e1000.htm This does not solve the problem fully though: Setting the ITR too high (50000-75000) makes the system unresponsive under certain network loads. Something like 25000 allows for most communication, like NFS and SSH but still software like LAM-MPI has problems. Setting the value arround 10000 makes most software run smoothly, but latency then is arround 50 us (i think) on our hardware.. But the latency is constant (no 60 to 250us) fluctuations. We have not found an optimal solution (other than using different NIC's) If anyone has any ideas i would be grateful. best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 04:59:43 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 10:59:43 +0100 (CET) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> Message-ID: On Fri, 9 Jan 2004, Rich Pickler wrote: > > In a nutshell, the questions we have: > > -What batch system do you recommend for our purposes (Condor, PBSPro, > ScalablePBS, SGE)? > Over on the SGE list we've had a discussion about this over the last week. Perhaps you could search the archives at gridengine.sunsource.net Notably, there was a contribution from the company I used to work for, Framestore, who are doing a similar exercise - evaluating SGE and PBS etc. for a renderfarm. a few other companies made controbutions. Perhaps it would bee worth joining the SGE list and asking the question there? Also of interest to the rendering community, if you are thinking of Opterons, is that there is a native x86-64 version of SGE now. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 05:20:27 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 11:20:27 +0100 (CET) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> Message-ID: On Sat, 10 Jan 2004, [big5] Andrew Wang wrote: > (Since you are in this graphics industry, can you tell > me if Framestore is big? People on the sge list looked > so excited.) ps. I'm excited myself. I like using SGE, and I think it would be great to see it running a big renderfarm. The Enterprise Edition has I think project-based priority, which might be useful in giving projects coming to a deadline more time. I've no idea how big the Framestore-CFC render farm is at the moment. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 10 05:13:16 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 10 Jan 2004 11:13:16 +0100 (CET) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> Message-ID: On Sat, 10 Jan 2004, [big5] Andrew Wang wrote: > > You are not alone, Framestore CFC is doing the same > thing. They are looking at using SGE 6.0 to replace > Alfred. > > (Since you are in this graphics industry, can you tell > me if Framestore is big? People on the sge list looked > so excited.) > They are now the biggest post-production house in Europe, I believe. When I worked for them they made many series for Hallmark, the Walking with Dinosaurs series, TV adverts and shots for many feature films. For example the tracking shot through four seasons along the market in Notting Hill, James Bond title sequences. I joined the company just after the merger with Computer Film Company. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Sat Jan 10 08:00:46 2004 From: proksy_me at yahoo.com (prakash borade) Date: Sat, 10 Jan 2004 05:00:46 -0800 (PST) Subject: [Beowulf] help for my beowulf Message-ID: <20040110130047.6599.qmail@web40110.mail.yahoo.com> hello i need your help for my beowulf cluster i am using linux redhat-9.0 mpich-1.2.5.2 implementtaion of mpi i require to enable thw lam environment for my program to run please send me some links that can lead me to the succesfull beowulf or if possible any documenttaion fopr buoilding the cluster thanks in advace proksy_me at yahoo.com __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Sat Jan 10 08:47:11 2004 From: raysonlogin at yahoo.com (Rayson Ho) Date: Sat, 10 Jan 2004 05:47:11 -0800 (PST) Subject: [Beowulf] help for my beowulf In-Reply-To: <20040110130047.6599.qmail@web40110.mail.yahoo.com> Message-ID: <20040110134711.82821.qmail@web11407.mail.yahoo.com> MPICH and LAM are different MPI implementations. Rayson --- prakash borade wrote: > hello > > > i need your help for my beowulf cluster > i am using linux redhat-9.0 > mpich-1.2.5.2 implementtaion of mpi > > > i require to enable thw lam environment for my program > to run ^^^^^^^^^^^^^^^^ > __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From proksy_me at yahoo.com Sat Jan 10 08:26:25 2004 From: proksy_me at yahoo.com (prakash borade) Date: Sat, 10 Jan 2004 05:26:25 -0800 (PST) Subject: [Beowulf] (no subject) Message-ID: <20040110132625.9563.qmail@web40110.mail.yahoo.com> mpich implementation can anybody sujects me the best benchmarking suit for the computational cluster __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Sat Jan 10 09:40:14 2004 From: raysonlogin at yahoo.com (Rayson Ho) Date: Sat, 10 Jan 2004 06:40:14 -0800 (PST) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <3FFF245A.7070804@dnahelix.com> Message-ID: <20040110144014.74052.qmail@web11410.mail.yahoo.com> > -What batch system do you recommend for our purposes (Condor, PBSPro, > ScalablePBS, SGE)? My take is, if you know PBS, or your users have experience with PBS before, go for SPBS or PBSPro. If you use SPBS, also take a look at Maui, since SPBS scheduling algorithm is limited. If you start clean, use SGE. Condor is also good, but the developers haven't released the source (they said they will), not a big problem. *However*, your cluster information is spied by Condor, sent back to UW, for "research purposes only". I personally like condor a lot (esp the checkpointing feature), but the spying art makes them look like M$. > -Fail over of the head node is also important to us, but I haven't > researched enough on this topic to start asking questions. (So far, > I've only found stuff for SGE, and a rumor that PBSPro will add this > feature soon) I've requested this feature on the SPBS mailing list, I assume the SPBS developers are looking into it. > First, we're looking at a cluster of around 400 CPUs, which is > getting > dangerously close to the limits of OpenPBS. Considering that this > number could easily grow down the road, pushing that limit as far > back > as possible is needed. The problem is not the number of CPUs you have, it is the number of nodes you have. OpenPBS scheduler contacts each node in each scheduling cycle to get the load information. If one of the nodes is dead, then the scheduler hangs. The hack they did was to set up an alarm, wait for the reply from the node, and timeout after 2-3 minutes, and restart. --> like M$... exit all windows and restart :) OpenPBS works OK with less than 300 nodes, but as more and more nodes are added to the cluster, the likelihood of a node failing increases. SPBS fixed this problem 2 months ago, the developers said that they have changed the server <-> mom communication protocol. SGE does not have this problem, AFAIK. As long as the server <-> node communication protocol is designed correctly, you can in theory add as many nodes to the cluster as you want. In the extreme case (several 10,000 nodes?), there are way too many nodes reporting the load information to the master, the master will not do much work but just keeps on accepting load info. And at this point you need to decrease the load report frequency. The bottomline: as long as the communication protocol is OK (SPBS, PBSPro, SGE, LSF), you don't need to worry. But OpenPBS is really something you need to avoid for mid-size clusters. > So far, PBS looks like the most widely used > and > known system out there, so some variant of that seems ideal for > finding > prospective employees with cluster experience, although it looks like > a > lot of you deal with SGE as well. SGE has tons of names, it's a bit confusing. It used to be a research project "DQS", then commercialized, became "gridware", "Codine", "GRD", and then bought by Sun, renamed to "SGE", but some people call it "Sun Grid", while some call it "Gridengine". PBS called itself PBS at the very beginning. > PBSPro and ScalablePBS advertise > they > get the scalability limit much higher than what we need, but the > first > is much more expensive. Yes, PBSPro don't fix the known problems and well-known limitations in OpenPBS, and they ask you to buy PBSPro -- good marketing strategy :) > I have yet to find any documentation or > rumor > of SGE's upper bound. Again, as long as you are not using OpenPBS, don't worry... > One of the problems I have noticed is that some batch managers bog > down > with a large amount of jobs. Looking at the structure of our jobs, > we > can easily wind up with 20,000-40,000 jobs in a day with a large > amount > of them being submitted to the batch around the same time (when > everybody leaves work). Should be fine. The scheduler is CPU-bound (a Linux PC box can handle that amount of scheduling work), while the job holder (qmaster in SGE) is I/O bound. If the spool directory is in an imported NFS partition, then you will find that job submission slow. > > Any thoughts what might be the best solution for us? > > Thanks for any recommendations you can give me, and feel free to ask > me > any more details you want. BTW, subscribe to the SPBS and SGE mailing lists for further info. Rayson > > Rich > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lathama at yahoo.com Sun Jan 11 21:45:17 2004 From: lathama at yahoo.com (Andrew Latham) Date: Sun, 11 Jan 2004 18:45:17 -0800 (PST) Subject: [Beowulf] linuxbios Message-ID: <20040112024517.80352.qmail@web60307.mail.yahoo.com> Just got my "Linux Journal" and was reading about the linuxbios. It mentioned tyan (My fav.) I searched tyan.com and found nothing related. (just thought about googling tyan.com) Any Tyan MB Linux BIOS users out there? What are the success stories out there. I am starting to play and am already downloading some stuff I just wondered what the group thinks about this. http://www.clustermatic.org/ http://www.linuxbios.org ===== /---------------------------------------------------------------------------------------------------\ Andrew Latham -LathamA - Penguin Loving, Moralist Agnostic. What Is an agnostic? - An agnostic thinks it impossible to know the truth in matters such as, a god or the future with which religions are concerned with. Or, if not impossible, at least impossible at the present time. LathamA.com - (lay-th-ham-eh) - lathama at lathama.com - lathama at yahoo.com \---------------------------------------------------------------------------------------------------/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Mon Jan 12 03:52:03 2004 From: john.hearns at clustervision.com (John Hearns) Date: Mon, 12 Jan 2004 09:52:03 +0100 (CET) Subject: [Beowulf] linuxbios In-Reply-To: <20040112024517.80352.qmail@web60307.mail.yahoo.com> Message-ID: I've been following it for a couple of years. At the Linux Beer Hike in Ireland in 2002 we did some experiments, and got it running. I've stilll got the PC-chips motherboard which I bought to do the tests. Antony Stone found a UK supplier if the disk-on-chip parts. Antony did a very good article for the UK version Linux Magazine. I haven't had the courage though to try Linuxbios on my precious mini-ITXes. I'm mostly interested in Linuxbios though for the controlability of large clusters. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Mon Jan 12 12:44:00 2004 From: becker at scyld.com (Donald Becker) Date: Mon, 12 Jan 2004 12:44:00 -0500 (EST) Subject: [Beowulf] BWBUG January 2004 -- Sun Microsystems Linux plans Message-ID: Date: January 13, 2003 Time: 2:45 PM Location: Northrop Grumman IT, McLean Virginia Sun Microsystems will provide an in-depth presentation regarding their success with Linux and future strategy to become one of the world leaders in Linux Beowulf systems. That's correct, it is LINUX not Solaris. The presentation will cover current and planned features of Sun Grid Engine, as well as the new Sun Fire V60x Compute Grid Rack System. Sun has installed a number of Linux Beowulf Clusters on the Top 500 List in 2003. This event will be sponsored by the Baltimore-Washington Beowulf Users Group (BWBUG) and will be held at Northrop Grumman Information Technology 7575 Colshire Drive, 2nd floor, McLean Virginia. Please register on line at http://bwbug.org to assure a spot. This event has attracted a great deal of interest. Date: January 13, 2003 Time: 2:45 PM Parking: Free Need to be a member?: No ( guests are welcome ) For more details see listing in Washington Technology: http://www.washingtontechnology.com/events/16835.html T. Michael Fitzmaurice, Jr. Coordinator of the BWBUG 703-205-3132 office 240-475-7877 cell Email michael.fitzmaurice at ngc.com http://www.bwbug.org -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster systems Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Per.Lindstrom at me.chalmers.se Mon Jan 12 14:54:42 2004 From: Per.Lindstrom at me.chalmers.se (=?ISO-8859-1?Q?Per_Lindstr=F6m?=) Date: Mon, 12 Jan 2004 20:54:42 +0100 Subject: [Beowulf] The name Beowulf-cluster Message-ID: <4002FB82.5030101@me.chalmers.se> Hi, Is there some one that know why the clustering technique is named Beowulf? According to a viking saga was Beowulf a king that ruled //the kingdom of Geat, (Gothia, G?taland that is located in west Sweden) around year 500, i.e. about 700 year before the rise of the Swedish nation. Best regards Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gust_77ar at yahoo.com Mon Jan 12 17:36:08 2004 From: gust_77ar at yahoo.com (Gustavo Velardez) Date: Mon, 12 Jan 2004 14:36:08 -0800 (PST) Subject: [Beowulf] G98 on cluster Message-ID: <20040112223608.49919.qmail@web60702.mail.yahoo.com> Hi everybody, I'm trying to run Gaussian98 on one or several nodes in a beowulf cluster. Are there another different option than qsub or similar? Thanks Gustavo __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rpickler at dnahelix.com Mon Jan 12 19:35:35 2004 From: rpickler at dnahelix.com (Rich Pickler) Date: Mon, 12 Jan 2004 18:35:35 -0600 Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> References: <20040110052010.53307.qmail@web16806.mail.tpe.yahoo.com> Message-ID: <40033D57.3050702@dnahelix.com> >(Since you are in this graphics industry, can you tell >me if Framestore is big? People on the sge list looked >so excited.) > > > Yeah, they're big. I talk with a couple people there quite often so I'll drop them a line and pick their brains a little bit. >Use Opensource (r), and no one will ask you for money. > > The OpenSource + pay support hasn't really sold everyone here yet. Seems logical to me, but I guess they need to be won over. Speaking of, one of the guys here was really upset with Sun's support and is leery of Grid Engine because of some issue with a Cobalt Rack system. I was always under the impression Sun's support was quite good, the cobalt stuff was just kind of a red-headed step child they never wanted. What's everybody's perception of Grid Engine's support? I've heard some accounts of great responsiveness from the community and getting bugs fixed very quickly. Unfortunately, it's hard to sell to the bosses based on community support alone. >SGE has array jobs, but all others do not. > > I'll have to check this feature out. Sounds very interesting and well suited for our kind of usage. It hadn't come up in my searches before. >ganglia is good, a lot of people use it. I just think >setting up a webpage is the overkilled part -- need to >run Apache to monitor the cluster -- but just MHO >only. > > People are going to have to go in and see how their jobs are progressing on the farm. We're talking about artists here. Asking them to cut off their arms would be easier than getting them to log into a shell and type a few commands. So far it looks like pretty much every clustering system has hooks for integrating it into a web page. (Luckily this isn't my area to develop). So we ought to have something quite nice for them to use. >The problem with SGE when handling MPI jobs is that it >takes time to set up the PE, and PBS has an external >interface which is much nicer. People had already >written mpiexec to take advantage of the interface. > > MPI is completely a non issue here. Even if I were to write a cluster app in the traditional sense, I wouldn't use it. (Bad memories in college, but I suspect they had it poorly set up) >I think the best is to try it yourself, as it all >depends on what you want to do. > > That's next month when we buy the development cluster. We just need some recommendations before diving in. Everybody, thanks for the help on this last weekend. We start talking to different hardware providers this week, so I needed a quick background in the options before they try and pitch something to us that will only cause headaches down the road. Rich _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Tue Jan 13 00:26:06 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Tue, 13 Jan 2004 13:26:06 +0800 (CST) Subject: [Beowulf] Recommendations for new cluster In-Reply-To: <40033D57.3050702@dnahelix.com> Message-ID: <20040113052606.10833.qmail@web16808.mail.tpe.yahoo.com> > Speaking of, one of the guys here was really upset > with Sun's support > and is leery of Grid Engine because of some issue > with a Cobalt Rack system. You can also pick another company to support SGE, there are actually more than 20 for you to choose from: http://wwws.sun.com/software/gridware/partners/index.html > People are going to have to go in and see how their > jobs are progressing > on the farm. We're talking about artists here. > Asking them to cut off > their arms would be easier than getting them to log > into a shell and > type a few commands. A guy switched from another batch system to SGE and wrote a very simple web GUI: http://www.veus.hr/linux/gemonitor.html Also, Sun has "Grid Engine Portal", but it requires the Solaris Portal server: http://gridengine.sunsource.net/project/gridengine/gep/GEP_Intro.html Andrew. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Jan 13 03:25:43 2004 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Tue, 13 Jan 2004 09:25:43 +0100 Subject: [Beowulf] The name Beowulf-cluster In-Reply-To: <4002FB82.5030101@me.chalmers.se> ( =?iso-8859-1?q?Per_Lindstr=F6m's_message_of?= "Mon, 12 Jan 2004 20:54:42 +0100") References: <4002FB82.5030101@me.chalmers.se> Message-ID: Per Lindstr?m writes: > Is there some one that know why the clustering technique is named Beowulf? Thomas Sterling claims he sat in his room, trying to think of a name. He looked up at his bookshelf, saw an edition of the Beowulf saga, and thought "'Beowulf', that'll do. Nobody will hear about it, anyways". -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Andrew.Cannon at nnc.co.uk Tue Jan 13 03:40:34 2004 From: Andrew.Cannon at nnc.co.uk (Cannon, Andrew) Date: Tue, 13 Jan 2004 08:40:34 -0000 Subject: [Beowulf] Solaris on x86? Message-ID: Hi there, We are looking to get some x86 based compute power (possibly a number of Athlon based machines) and I was wondering if anyone has any experience of Solaris for x86, specifically, how well does it work in a cluster? Would we be better looking at Red Hat PW or ES or some other linux distro. The company is not too keen on going down the ROCKS etc route and would prefer a "mainstream" solution. TIA. Andy Andrew Cannon, Nuclear Technology (J2), NNC Ltd, Booths Hall, Knutsford, Cheshire, WA16 8QZ. Telephone; +44 (0) 1565 843768 email: mailto:andrew.cannon at nnc.co.uk NNC website: http://www.nnc.co.uk NNC's UK Operating Companies : NNC Holdings Limited (no. 3725076), NNC Limited (no. 1120437), National Nuclear Corporation Limited (no. 2290928), STATS-NNC Limited (no. 4339062) and Technica-NNC Limited (no. 235856). The registered office of each company is at Booths Hall, Chelford Road, Knutsford, Cheshire WA16 8QZ except for Technica-NNC Limited whose registered office is at 6 Union Row, Aberdeen AB10 1DQ. NNC's head office and principal address is Booths Hall and the switchboard number is 01565 633800. The NNC website is www.nnc.co.uk Any information or opinions in this message which do not relate to our business are not authorised by any of the above companies. Where this message does relate to our business the content is sent to you by the relevant company (as above) and is confidential and intended for the use of the individual or entity to whom it is addressed. The contents do not form part of any contract unless so stated. If you have received this e-mail in error please notify the NNC system manager by email at eadm at nnc.co.uk. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmdavis1 at vcu.edu Mon Jan 12 20:58:55 2004 From: jmdavis1 at vcu.edu (Mike Davis) Date: Mon, 12 Jan 2004 20:58:55 -0500 Subject: [Beowulf] G98 on cluster In-Reply-To: <20040112223608.49919.qmail@web60702.mail.yahoo.com> References: <20040112223608.49919.qmail@web60702.mail.yahoo.com> Message-ID: <400350DF.1080508@vcu.edu> You will need a pbs wrapper. In addition, to run in parallel, you need the LINDA libraries. I have a wrapper for GAMESS/G98 and a standalone submission script (interactive, character based). Let me know if you would like them. Mike Gustavo Velardez wrote: >Hi everybody, >I'm trying to run Gaussian98 on one or several nodes >in a beowulf cluster. Are there another different >option than qsub or similar? > >Thanks > > Gustavo > >__________________________________ >Do you Yahoo!? >Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes >http://hotjobs.sweepstakes.yahoo.com/signingbonus >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mas at ucla.edu Tue Jan 13 09:51:18 2004 From: mas at ucla.edu (Michael Stein) Date: Tue, 13 Jan 2004 06:51:18 -0800 Subject: [Beowulf] multiline string literals... In-Reply-To: ; from rgb@phy.duke.edu on Tue, Jan 13, 2004 at 08:29:25AM -0500 References: Message-ID: <20040113065118.A20459@mas1.ats.ucla.edu> > This is a teeny bit off-topic, but I don't know where else to find a > group of knowledgeable systems programmers. > > Does anybody here know WHY multiline literals, e.g. specifically > something like: > > char usage = " > Usage: foobar [-h] [-v] -f barianfile > -h for help > -v for verbose > -f barianfile to specify your foo-bar-barian > "; > > are deprecated in gcc? Probably because it's hard to tell the difference between the above and an unbalanced double quote character? An unbalanced quote could eat a big hunk of your code and possibly without any error messages... Try this instead: char usage = "\n" "Usage: foobar [-h] [-v] -f barianfile\n" " -h for help\n" " -v for verbose\n" " -f barianfile to specify your foo-bar-barian\n" " "; See ANSI C standard section 6.1.4 String literals which says that adjacent strings literals are concatenated into a single string. PS: You need either *usage or usage[]. -Wall is your friend... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 13 08:29:25 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 13 Jan 2004 08:29:25 -0500 (EST) Subject: [Beowulf] multiline string literals... Message-ID: This is a teeny bit off-topic, but I don't know where else to find a group of knowledgeable systems programmers. Does anybody here know WHY multiline literals, e.g. specifically something like: char usage = " Usage: foobar [-h] [-v] -f barianfile -h for help -v for verbose -f barianfile to specify your foo-bar-barian "; are deprecated in gcc? I have a very hard time understanding this. A string is a bunch of bytes in memory. I can put anything I want in it. It is clearly a million times more tedious to code this in one line at a time, and many times more likely to contain a bug as well. Sorry to rant a bit, but I've spent some time trying to understand just what it is that they're saving me from or what low-level aspect of a computer is made ambiguous by multiline literals in C code and come up empty. Seems like pure fascism to me, which is anathema in C. You want rigorous structure, use Pascal... rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Tue Jan 13 10:51:36 2004 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Tue, 13 Jan 2004 16:51:36 +0100 (CET) Subject: [Beowulf] multiline string literals... In-Reply-To: Message-ID: On Tue, 13 Jan 2004, Robert G. Brown wrote: > Does anybody here know WHY multiline literals ... are deprecated in gcc? As written at the top of: http://gcc.gnu.org/gcc-3.3/changes.html they were deprecated for quite some time. So it's a bit late to ask now :-) > A string is a bunch of bytes in memory. I think that this has more to do with (pre)processing the source rather than storing the result in memory. > It is clearly a million times more tedious to code this in one line at a > time, and many times more likely to contain a bug as well. Agreed :-) One other example is inlined ASM... -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Jan 13 09:56:20 2004 From: nixon at nsc.liu.se (nixon at nsc.liu.se) Date: Tue, 13 Jan 2004 15:56:20 +0100 Subject: [Beowulf] multiline string literals... In-Reply-To: (Robert G. Brown's message of "Tue, 13 Jan 2004 08:29:25 -0500 (EST)") References: Message-ID: "Robert G. Brown" writes: > Does anybody here know WHY multiline literals, e.g. specifically > something like: > > char usage = " > Usage: foobar [-h] [-v] -f barianfile > -h for help > -v for verbose > -f barianfile to specify your foo-bar-barian > "; > > are deprecated in gcc? Because they are not C? -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgoornaden at intnet.mu Tue Jan 13 11:43:40 2004 From: rgoornaden at intnet.mu (roudy) Date: Tue, 13 Jan 2004 20:43:40 +0400 Subject: [Beowulf] Linpack Benchmark References: <200401121706.i0CH6Xh18291@NewBlue.scyld.com> Message-ID: <000701c3d9f4$6cdff4d0$080b7bca@roudy> Hello Everybody, I have just build my cluster, and now I want to measure its performance via hpl. I download this software some days back, but I have some difficulty to run it. Below is the several steps I undertake.. tar -xvf hpl.tar -----here it is fine Then I create a file call "Make.cluster" in the directory "/hpl" Afterwards, I use the following command "make arch=cluster" but this does not work...can someone give me a hint of what I should do???? Best Greetings Roud _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nfaerber at penguincomputing.com Tue Jan 13 13:56:35 2004 From: nfaerber at penguincomputing.com (Nate Faerber) Date: 13 Jan 2004 10:56:35 -0800 Subject: [Beowulf] Linpack Benchmark In-Reply-To: <000701c3d9f4$6cdff4d0$080b7bca@roudy> References: <200401121706.i0CH6Xh18291@NewBlue.scyld.com> <000701c3d9f4$6cdff4d0$080b7bca@roudy> Message-ID: <1074020195.29028.10.camel@m10.penguincomputing.com> > Below is the several steps I undertake.. > > tar -xvf hpl.tar -----here it is fine > > Then I create a file call "Make.cluster" in the directory "/hpl" > Afterwards, I use the following command "make arch=cluster" but this does > not work...can someone give me a hint of what I should do???? > First, check that TOPdir is set correctly in your Makefile. Then send the output so we can see where it is breaking. -- Nate Faerber, Engineer Tel: 415-358-2666 Fax: 415-358-2646 Toll Free: 888-PENGUIN PENGUIN COMPUTING www.penguincomputing.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From becker at scyld.com Tue Jan 13 13:46:17 2004 From: becker at scyld.com (Donald Becker) Date: Tue, 13 Jan 2004 13:46:17 -0500 (EST) Subject: [Beowulf] BW-BUG meeting today (Tuesday Jan 13, 2004) in McLean Va Message-ID: Remember that this month's meeting is McLean VA , not Greenbelt MD Baltimore-Washington Beowulf User Group January 2004 Meeting Date: January 13, 2003 Time: 2:45 PM Location: Northrop Grumman IT, McLean Virginia Sun Microsystems will provide an in-depth presentation regarding their success with Linux and future strategy to become one of the world leaders in Linux Beowulf systems. That's correct, it is LINUX not Solaris. The presentation will cover current and planned features of Sun Grid Engine, as well as the new Sun Fire V60x Compute Grid Rack System. Sun has installed a number of Linux Beowulf Clusters on the Top 500 List in 2003. This event will be sponsored by the Baltimore-Washington Beowulf Users Group (BWBUG) and will be held at Northrop Grumman Information Technology 7575 Colshire Drive, 2nd floor, McLean Virginia. Please register on line at http://bwbug.org to assure a spot. This event has attracted a great deal of interest. Date: January 13, 2003 Time: 2:45 PM Parking: Free Need to be a member?: No ( guests are welcome ) For more details see listing in Washington Technology: http://www.washingtontechnology.com/events/16835.html T. Michael Fitzmaurice, Jr. Coordinator of the BWBUG 703-205-3132 office 240-475-7877 cell Email michael.fitzmaurice at ngc.com http://www.bwbug.org -- Donald Becker becker at scyld.com Scyld Computing Corporation http://www.scyld.com 914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster systems Annapolis MD 21403 410-990-9993 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Tue Jan 13 12:56:42 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Tue, 13 Jan 2004 12:56:42 -0500 (EST) Subject: [Beowulf] The name Beowulf-cluster In-Reply-To: <4002FB82.5030101@me.chalmers.se> Message-ID: On Mon, 12 Jan 2004, Per Lindstr?m wrote: > Hi, > > Is there some one that know why the clustering technique is named Beowulf? > > According to a viking saga was Beowulf a king that ruled //the kingdom > of Geat, (Gothia, G?taland that is located in west Sweden) around year > 500, i.e. about 700 year before the rise of the Swedish nation. You may find reading this: http://www.linux-mag.com/2003-06/breakthroughs_01.html useful. It was written by Tom Sterling and provides some history. Doug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Tue Jan 13 16:32:15 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Tue, 13 Jan 2004 13:32:15 -0800 Subject: [Beowulf] Upcoming Beowulf Presentation - Help! In-Reply-To: References: <65F398B477B73D4BBD05E6284525226A02C61771@EMSS02M19.us.lmco.com> Message-ID: <20040113213215.GB3343@greglaptop.internal.keyresearch.com> On Tue, Jan 13, 2004 at 03:27:02PM -0500, Douglas Eadline, Cluster World Magazine wrote: > If you can find it Gregory Pfister's "In Search of Clusters" provides a > great technical overview of why clusters are going to be important, BTW > the book is pre-Linux/Beowulf There's a second edition that's post-Beowulf. When reading it, you should keep in mind that he buys the "shared memory programming is easier than message passing" dogma, and he only thinks about fairly small cpu counts. Some of his statements make a lot more sense if you remember where he comes from. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bushnell at chem.ucsb.edu Tue Jan 13 17:59:01 2004 From: bushnell at chem.ucsb.edu (John Bushnell) Date: Tue, 13 Jan 2004 14:59:01 -0800 (PST) Subject: [Beowulf] G98 on cluster In-Reply-To: <400350DF.1080508@vcu.edu> Message-ID: In the Gaussian 03 distribution, there is a script called 'subg03' in the 'bsd' subdirectory. G98 may have something similar. - John On Mon, 12 Jan 2004, Mike Davis wrote: > You will need a pbs wrapper. In addition, to run in parallel, you need > the LINDA libraries. I have a wrapper for GAMESS/G98 and a standalone > submission script (interactive, character based). Let me know if you > would like them. > > Mike > > > Gustavo Velardez wrote: > > >Hi everybody, > >I'm trying to run Gaussian98 on one or several nodes > >in a beowulf cluster. Are there another different > >option than qsub or similar? > > > >Thanks > > > > Gustavo _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 13 20:20:33 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 13 Jan 2004 20:20:33 -0500 (EST) Subject: [Beowulf] multiline string literals... In-Reply-To: <20040113065118.A20459@mas1.ats.ucla.edu> Message-ID: On Tue, 13 Jan 2004, Michael Stein wrote: > Probably because it's hard to tell the difference between the above and > an unbalanced double quote character? An unbalanced quote could > eat a big hunk of your code and possibly without any error messages... Yeah, but the stuff below has 2N-2 more opportunities to fail to close a quote and several other opportunities to screw up, and it isn't any more difficult than balancing {}'s across the miles of intervening code. I was thinking it might have to do with internationalization or something like that and not just somebody's opinion of what is easy or hard to safely code. > Try this instead: > char usage = "\n" > "Usage: foobar [-h] [-v] -f barianfile\n" > " -h for help\n" > " -v for verbose\n" > " -f barianfile to specify your foo-bar-barian\n" > " "; > > See ANSI C standard section 6.1.4 String literals which says that > adjacent strings literals are concatenated into a single string. Sigh. This works and thank you. Of course just a big string WORKS, it just complains every time the file is compiled and this shuts it up. Ah, in the good old days you could often declare a constant string like this (without complaint) and even write back into it without anything complaining. Of course in the good old days, code leaked like a sieve and buffer overwrite attacks existed in nearly every daemon or utility... > PS: You need either *usage or usage[]. -Wall is your friend... :-) I know, pointer to the data block. Just sloppy typing in the example by hand. As you say, that too is what compilers are for -- to help you find little buglets. Thanks again. I'll adopt your suggestion just so the compiler shuts up. I hate programs that don't build perfectly clean. Especially when I'm writing them. So eventually I am nagged out of deprecated habits...:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From root at pixelcatcher.de Wed Jan 14 06:06:03 2004 From: root at pixelcatcher.de (Michael =?ISO-8859-1?Q?Damb=F6ck?=) Date: Wed, 14 Jan 2004 12:06:03 +0100 Subject: [Beowulf] How to set up 2 Node Failover Cluster Message-ID: <1074078362.2246.10.camel@acernote.evonet.de> Hello everybody, I'm told to set up a 2 Node Failover Linux Cluster. Is ist possible to do this using beowulf? I've been google-ing all day but unfortuneately I didn't get what I was looking for. I hope you can help me. If it is possible to do that using beowulf please tell me where I can find Documents describing the way this can be done in general. Alternatively please tell me what other free Software is available for setting up a linux cluster. Thanks a lot! Michael -- http://www.pixelcatcher.de digitalfotografie online _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Wed Jan 14 07:28:19 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Wed, 14 Jan 2004 12:28:19 +0000 (GMT) Subject: [Beowulf] Re: Linpack Benchmark (roudy) In-Reply-To: <200401131703.i0DH3vh22722@NewBlue.scyld.com> References: <200401131703.i0DH3vh22722@NewBlue.scyld.com> Message-ID: >Hello Everybody, I have just build my cluster, and now I want to measure >its performance via hpl. IBM have a redbook on running and tuning HPL for benchmarking clusters. Take a look at: http://www.redbooks.ibm.com/ and look for "Running HPL LINPACK on IBM xSeries Linux Clusters". Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jgreenseid at wesleyan.edu Wed Jan 14 10:34:07 2004 From: jgreenseid at wesleyan.edu (Joe Greenseid) Date: Wed, 14 Jan 2004 10:34:07 -0500 (EST) Subject: [Beowulf] How to set up 2 Node Failover Cluster In-Reply-To: <1074078362.2246.10.camel@acernote.evonet.de> References: <1074078362.2246.10.camel@acernote.evonet.de> Message-ID: This type of clustering isn't really what Beowulf clusters do. You should try checking out the High Availability Linux project at http://linux-ha.org. --Joe On Wed, 14 Jan 2004, Michael [ISO-8859-1] Damb?ck wrote: > Hello everybody, > > I'm told to set up a 2 Node Failover Linux Cluster. > Is ist possible to do this using beowulf? I've been google-ing all day > but unfortuneately I didn't get what I was looking for. > > I hope you can help me. If it is possible to do that using beowulf > please tell me where I can find Documents describing the way this can be > done in general. Alternatively please tell me what other free Software > is available for setting up a linux cluster. > > Thanks a lot! > > Michael > -- > http://www.pixelcatcher.de > digitalfotografie online > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Wed Jan 14 10:32:11 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed, 14 Jan 2004 07:32:11 -0800 Subject: [Beowulf] How to set up 2 Node Failover Cluster Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C041@orsmsx402.jf.intel.com> From: Michael Damb?ck; Wednesday, January 14, 2004 3:06 AM > > Hello everybody, > > I'm told to set up a 2 Node Failover Linux Cluster. > Is ist possible to do this using beowulf? I've been google-ing all day > but unfortuneately I didn't get what I was looking for. No, and yes. No, Beowulf doesn't provide failover; yes, you can use failover within a cluster with good and desirable results. The biggest issue is what the failover will handle, as that determines the software you need. For example, do you want to provide a highly available NFS server? Web server? Cluster headnode? Other "stateless" server? Database? Queuing system? Something else? The NFS and web servers, cluster headnode, and other "stateless" servers can be handled by "heartbeat" at http://www.linux-ha.org/download/ This code allows you to define highly available resources that are managed by heartbeat, the failover manager. You'll typically have heartbeat own a well-known highly available IP address along with any other resources, e.g., shared storage. Heartbeat will only allow one of the two nodes to host the HA resources until that node fails, heartbeat will then start the resources on the other (backup) node and optionally disable the replaced server via a STONITH (shoot the other node in the head) mechanism to prevent a "split-brain" situation where two nodes think they own a resource. The limitation to heartbeat is that it cannot manage any shared state, as you would find in a database server or queuing system. For an HA database or queuing system, you'll need to look at an app that specifically provides that capability, e.g., SGE for queuing. Even in these latter cases, you still may want to use heartbeat to provide a well-known HA IP address to host the HA app. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Wed Jan 14 12:31:07 2004 From: john.hearns at clustervision.com (John Hearns) Date: Wed, 14 Jan 2004 18:31:07 +0100 (CET) Subject: [Beowulf] linuxbios In-Reply-To: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: On Wed, 14 Jan 2004, Erwan Velu wrote: > Can you develop that point ? What is really intresting in such > technologie ? Once your bios are well setup, you don't need to change it > every day..> Of course that's true. It just happens that I was just back from a customer site visit, where Iwas doing some BIOS upgrades. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jlb17 at duke.edu Wed Jan 14 11:01:03 2004 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed, 14 Jan 2004 11:01:03 -0500 (EST) Subject: [Beowulf] Dual Xeon gigabit issues Message-ID: I just moved two dual Xeon servers into a new home. One of them refuses to see a network link, before or after the machine boots. The only difference in them is a slight revision in the motherboard: the one that links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel 82546EB -- dual port gigabit). They're connected to a Cisco Catalyst 4507R. The box *will* link when connected to a 100Mb switch. I also hooked it up to a Dell 5224 (gigabit) switch that I have, and it found a link after about 10 seconds. Any ideas? I've already tried flashing the BIOS and playing with mii-tool a bit. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University _______________________________________________ Dulug mailing list Dulug at lists.dulug.duke.edu https://lists.dulug.duke.edu/mailman/listinfo/dulug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mathiasbrito at yahoo.com.br Wed Jan 14 12:36:45 2004 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Wed, 14 Jan 2004 14:36:45 -0300 (ART) Subject: [Beowulf] Nodes with or without HD? Message-ID: <20040114173646.68930.qmail@web12204.mail.yahoo.com> I would like to know how better or worse is a cluster with nodes that have HD. All my nodes in my cluster have HD with the System installed. I cannot see why it could be worse. Well if anyone have the answer, please answer. Thanks Mathias Brito ===== Mathias Brito Universidade Estadual de Santa Cruz - UESC Departamento de Ci?ncias Exatas e Tecnol?gicas Estudante do Curso de Ci?ncia da Computa??o ______________________________________________________________________ Conhe?a a nova central de informa??es anti-spam do Yahoo! Mail: http://www.yahoo.com.br/antispam _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Wed Jan 14 12:58:38 2004 From: agrajag at dragaera.net (Jag) Date: Wed, 14 Jan 2004 12:58:38 -0500 Subject: [Beowulf] Nodes with or without HD? In-Reply-To: <20040114173646.68930.qmail@web12204.mail.yahoo.com> References: <20040114173646.68930.qmail@web12204.mail.yahoo.com> Message-ID: <1074103117.4307.49.camel@pel> On Wed, 2004-01-14 at 12:36, Mathias Brito wrote: > I would like to know how better or worse is a cluster > with nodes that have HD. All my nodes in my cluster > have HD with the System installed. I cannot see why it > could be worse. Well if anyone have the answer, please > answer. There are plus and minuses. If you are not familiar with what it takes to admin a bunch of machines and keep them all in sync together (probablly using something like yum), then you may find it easier to have diskless nodes all sharing a single /usr over NFS so that its easier to update software. Likewise, diskless nodes can be quicker to add to the cluster, however if you're using kickstart and have a good kickstart config setup, nodes with disks aren't that hard to setup either. On the other side.. diskless machines require more network load in general. Unless it happens to be cached, whenever you need to access a file, you have to hit the network. This causes network congestion that can slow down MPI and such. It also means that it takes the slave node wanting the file longer to read it. So it can be quite a performance hit. But that also depends on what your networking needs are based on what kind of networking you have. Note, when you said diskless, I assumed you're talking about using NFS-root. If you're using Scyld, then things are a bit different. Their system of diskless slave nodes is a lot different. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From timm at fnal.gov Wed Jan 14 13:01:43 2004 From: timm at fnal.gov (Steven Timm) Date: Wed, 14 Jan 2004 12:01:43 -0600 (CST) Subject: [Beowulf] Nodes with or without HD? In-Reply-To: <20040114173646.68930.qmail@web12204.mail.yahoo.com> References: <20040114173646.68930.qmail@web12204.mail.yahoo.com> Message-ID: The disadvantage of having disk in the cluster is that you will increase your maintenance cost. we have 600+ nodes at Fermilab, all with system disks and data disks, and disks are almost 70% of all hardware calls that we have. To avoid disk in the cluster there is more technical work that needs to be done, we are investigating that at the moment. Steve ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division/Core Support Services Dept. Assistant Group Leader, Scientific Computing Support Group Lead of Computing Farms Team On Wed, 14 Jan 2004, [iso-8859-1] Mathias Brito wrote: > I would like to know how better or worse is a cluster > with nodes that have HD. All my nodes in my cluster > have HD with the System installed. I cannot see why it > could be worse. Well if anyone have the answer, please > answer. > > Thanks > Mathias Brito > > ===== > Mathias Brito > Universidade Estadual de Santa Cruz - UESC > Departamento de Ci?ncias Exatas e Tecnol?gicas > Estudante do Curso de Ci?ncia da Computa??o > > ______________________________________________________________________ > > Conhe?a a nova central de informa??es anti-spam do Yahoo! Mail: > http://www.yahoo.com.br/antispam > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From moloned at tcd.ie Wed Jan 14 13:46:13 2004 From: moloned at tcd.ie (david moloney) Date: Wed, 14 Jan 2004 18:46:13 +0000 Subject: [Beowulf] Market information on Beowulf clusters? Message-ID: <40058E75.7040409@tcd.ie> Could anybody point me to public and/or commercial market info. on Beowulf clusters especially: Market size (total installations, total nodes) histogram #nodes/cluster performance (GFLOPs) price cluster/node Thanks, - David _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From elfcool at yahoo.com Wed Jan 14 14:03:21 2004 From: elfcool at yahoo.com (Elif Sariarslan) Date: Wed, 14 Jan 2004 11:03:21 -0800 (PST) Subject: [Beowulf] Running MPICH on more than one node Message-ID: <20040114190321.42034.qmail@web40105.mail.yahoo.com> Hi, I have installed mpich on two machines. I am unable to execute my programs on 2 nodes. I have enabled ssh without password requirements and I have added both machines to machines.LINUX file but I still get the following error message when I execute the following command: [root at node02 basic]# ../../bin/mpirun -np 2 systest node00: Connection refused p0_26120: p4_error: Child process exited while making connection to remote process on node00.dl805.eng.ohio-state.edu: 0 ../../bin/mpirun: line 1: 26120 Broken pipe /root/mpich-1.2.5.2/examples/basic/systest -p4pg /root/mpich-1.2.5.2/examples/basic/PI26032 -p4wd /root/mpich-1.2.5.2/examples/basic Thanks, Elif __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bari at onelabs.com Wed Jan 14 14:07:47 2004 From: bari at onelabs.com (Bari Ari) Date: Wed, 14 Jan 2004 13:07:47 -0600 Subject: [Beowulf] linuxbios In-Reply-To: <20040112024517.80352.qmail@web60307.mail.yahoo.com> References: <20040112024517.80352.qmail@web60307.mail.yahoo.com> Message-ID: <40059383.4070004@onelabs.com> Andrew Latham wrote: > Just got my "Linux Journal" and was reading about the linuxbios. > It mentioned tyan (My fav.) > I searched tyan.com and found nothing related. > (just thought about googling tyan.com) > Any Tyan MB Linux BIOS users out there? > > What are the success stories out there. I am starting to play and am already > downloading some stuff I just wondered what the group thinks about this. Fwd from the LinuxBIOS list: "If you want to order a Tyan board with LinuxBIOS pre-loaded, you might need to go through some special ordering process... try asking Charlie Ding (cding at tyan.com). Currently we're working on the S2885. VGA works to an extent now, we can use the testbios utility to init the VGA BIOS on a GeForce FX and X will run with the "nv" driver. The binary driver will load and X will start, but the screen goes corrupt. So long as it's not a driver issue, this might be fixed in the near future. Getting the CLI should be first, and that will be nice since Ollie just got keyboard input working." "well there is the largest K8 cluster in the world, here at LANL, 1408 nodes. Built by linuxbios, arima mobo, linuxbios support from linux networx. Tyan will sell you K8 mobos with linuxbios on them. Hopefully the tyan people reading this will get in touch with you. There are a few other companies that will do K8 mobos with linuxbios but nothing public yet." _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tod at gust.sr.unh.edu Wed Jan 14 14:41:30 2004 From: tod at gust.sr.unh.edu (Tod Hagan) Date: 14 Jan 2004 14:41:30 -0500 Subject: [Beowulf] Running MPICH on more than one node In-Reply-To: <20040114190321.42034.qmail@web40105.mail.yahoo.com> References: <20040114190321.42034.qmail@web40105.mail.yahoo.com> Message-ID: <1074109290.16051.5.camel@haze.sr.unh.edu> Mpich uses rsh by default. Did you set 'export RSHCOMMAND=ssh' when compiling mpich? Can you run the mpich tests ('make testing') successfully? Before running mpich tests programs, you must be able to ssh in to the remote node yourself WITHOUT typing a password. See http://www.puddingonline.com/~dave/publications/SSH-with-Keys-HOWTO/document/html/SSH-with-Keys-HOWTO.html On Wed, 2004-01-14 at 14:03, Elif Sariarslan wrote: > Hi, > > I have installed mpich on two machines. I am > unable to execute my programs on 2 nodes. I have > enabled ssh without password requirements and I have > added both machines to machines.LINUX file but I still > get the following error message when I execute the > following command: > > [root at node02 basic]# ../../bin/mpirun -np 2 systest > > > node00: Connection refused > p0_26120: p4_error: Child process exited while making > connection to remote process on > node00.dl805.eng.ohio-state.edu: 0 > ../../bin/mpirun: line 1: 26120 Broken pipe > /root/mpich-1.2.5.2/examples/basic/systest -p4pg > /root/mpich-1.2.5.2/examples/basic/PI26032 -p4wd > /root/mpich-1.2.5.2/examples/basic > > Thanks, > > Elif > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 14 15:52:39 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 14 Jan 2004 15:52:39 -0500 (EST) Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: Message-ID: On Wed, 14 Jan 2004, Joshua Baker-LePain wrote: > I just moved two dual Xeon servers into a new home. One of them refuses > to see a network link, before or after the machine boots. The only > difference in them is a slight revision in the motherboard: the one that > links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't > has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel > 82546EB -- dual port gigabit). They're connected to a Cisco Catalyst > 4507R. You might first try looking for any messages in /var/log/messages when you load the driver module. I assume you are using e1000.o. Are the versions identical on both systems ? The output from things like uname -a lsmod may be a good place to start. BTW, with Intel chips-sets showing link to a switch does not necessarily imply that the kernel module is working. Doug > > The box *will* link when connected to a 100Mb switch. I also hooked it > up to a Dell 5224 (gigabit) switch that I have, and it found a link after > about 10 seconds. > > Any ideas? I've already tried flashing the BIOS and playing with mii-tool > a bit. > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From avkon at imm.uran.ru Wed Jan 14 15:03:14 2004 From: avkon at imm.uran.ru (Alexandr Konovalov) Date: Thu, 15 Jan 2004 01:03:14 +0500 Subject: [Beowulf] Bonding in 2.4.22-1.2149 Linux kernel Message-ID: <4005A082.1020701@imm.uran.ru> Hi, we have somewhat non-standart configuration of our computing cluster. Inernal computing nodes are connected by 2 (bonded) ethernet networks, but host connected only to one ethernet (i.e., connection to host not bonded for the all computing nodes). For the Linux kernel 2.4.9 - 2.4.20 it's possible to run such configuration if we use static arp tables and static routes (connection to host should be routed directly to the slave device, say eth1, not to bond0). But when we install 2.4.22-1.2149 (from Fedora 1), things are changed. Now routing to slave device became impossibe with diagnostics like ioctl(3, 0x890b, 0xbfe90b10) = -1 ENODEV (No such device) And there is no more slave devices in route output. Googling give us some reasons for this graceful changing of (yes, undocumented and unspecified) behaviour. It's compitely unacceptable for us to abandon bonding, so after drivers/net/bonding/bond_main.c reading we (probably) found a solution. Out idea is simply create new bonding policy. If a packet destination is in the bonded network it will call bond_xmit_roundrobin, if not then packed will be xmited directly to an appropriate slave device. MAC adresses for the direct xmiting will received by modiles as the load parameters. Probably we miss some opportunities and make some desing mistakes, because our hardware configuration is not very rare, so someone perhaps solve all our problems. We are interested in any feedback from the Beowulf community. -- Thanks for your attention, Alexandr Konovalov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Wed Jan 14 16:22:54 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed, 14 Jan 2004 13:22:54 -0800 Subject: [Beowulf] Market information on Beowulf clusters? In-Reply-To: <40058E75.7040409@tcd.ie> References: <40058E75.7040409@tcd.ie> Message-ID: <20040114212254.GA2127@greglaptop.internal.keyresearch.com> On Wed, Jan 14, 2004 at 06:46:13PM +0000, david moloney wrote: > Could anybody point me to public and/or commercial market info. on > Beowulf clusters especially: Two sources come to mind: 1) The Top500 list, which only has big clusters 2) Information from analysts like IDC For (2) you'll have to poke around; they usually charge $$, but they give out summaries for free sometimes. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jlb17 at duke.edu Wed Jan 14 16:29:28 2004 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed, 14 Jan 2004 16:29:28 -0500 (EST) Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: References: Message-ID: On Wed, 14 Jan 2004 at 11:01am, Joshua Baker-LePain wrote > I just moved two dual Xeon servers into a new home. One of them refuses > to see a network link, before or after the machine boots. The only > difference in them is a slight revision in the motherboard: the one that > links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't > has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel > 82546EB -- dual port gigabit). They're connected to a Cisco Catalyst > 4507R. > > The box *will* link when connected to a 100Mb switch. I also hooked it > up to a Dell 5224 (gigabit) switch that I have, and it found a link after > about 10 seconds. > > Any ideas? I've already tried flashing the BIOS and playing with mii-tool > a bit. Thanks to all who responded (on and off list). It turns out that the most likely culprit is a design flaw on certain SuperMicro dual Xeon boards that renders them unable to talk to some switches. Including Ciscos. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Wed Jan 14 16:51:08 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed, 14 Jan 2004 13:51:08 -0800 Subject: [Beowulf] Solaris on x86? In-Reply-To: References: Message-ID: <20040114215108.GA2292@greglaptop.internal.keyresearch.com> On Tue, Jan 13, 2004 at 08:40:34AM -0000, Cannon, Andrew wrote: > We are looking to get some x86 based compute power (possibly a number of > Athlon based machines) and I was wondering if anyone has any experience of > Solaris for x86, specifically, how well does it work in a cluster? Would we > be better looking at Red Hat PW or ES or some other linux distro. The > company is not too keen on going down the ROCKS etc route and would prefer a > "mainstream" solution. I see no one commented on this one. Solaris x86 is much less used for Linux clusters, with the main weaknesses being device driver support and compilers. It's certainly much less mainstream than RedHat Linux in this space, but Unix is Unix, and MPICH and most of our other favorite software runs on Solaris. As for ROCKS, it's using a pure-GPL version of RedHat Enterprise something or other. So it's less odd than you might think. You might want to explore what your company thinks "mainstream" means... I don't think I get the sense in which they use the word. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bioinformaticist at mn.rr.com Wed Jan 14 10:42:50 2004 From: bioinformaticist at mn.rr.com (Eric R Johnson) Date: Wed, 14 Jan 2004 09:42:50 -0600 Subject: [Beowulf] Schedulers for use with Scyld Version 28 Message-ID: <4005637A.2050908@mn.rr.com> I was wondering what the scheduler options are for me if I am running a Scyld version 28 machine. From everything I am finding, it appears that I only have a few choices. 1) at/bbq that ships with Scyld 2) PBS Pro 3) Clubmask/Maui I am running a very small cluster right now (16 processors) out of a University lab and PBS Pro is really out of our price range right now. I would like to add more nodes to the machine, but scheduling jobs is already a problem, so I don't think that bbq is the right choice. I have tried installing clubmask/Maui, but I ran into problems while compiling it. I was advised that it was due to the age of BProc that ships with Scyld. I was told that 3.2.5 is required. Can anybody give me some suggestions? Thanks, Eric -- ******************************************************************** Eric R A Johnson University Of Minnesota tel: (612) 626 5115 Dept. of Laboratory Medicine & Pathology fax: (612) 625 1121 7-230 BSBE e-mail: john4482 at umn.edu 312 Church Street web: www.eric-r-johnson.com Minneapolis, MN 55455 USA _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Wed Jan 14 09:42:56 2004 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Wed, 14 Jan 2004 09:42:56 -0500 Subject: [Beowulf] How to set up 2 Node Failover Cluster In-Reply-To: <1074078362.2246.10.camel@acernote.evonet.de> References: <1074078362.2246.10.camel@acernote.evonet.de> Message-ID: <20040114144255.GA15091@central.cis.upenn.edu> Just 2 nodes/servers, both are exactly the same but one is the "master" and the other is the "backup"? You're looking for a High Availability setup. Stop googling for Beowulf. Start searching for High Availability, and Failover. HTH, Dan W. > I'm told to set up a 2 Node Failover Linux Cluster. > Is ist possible to do this using beowulf? -- -- Daniel Widyono http://www.cis.upenn.edu/~widyono -- Liniac Project, CIS Dept., SEAS, University of Pennsylvania -- Mail: CIS Dept, 302 Levine 3330 Walnut St Philadelphia, PA 19104 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From avkon at imm.uran.ru Wed Jan 14 14:40:58 2004 From: avkon at imm.uran.ru (Alexandr Konovalov) Date: Thu, 15 Jan 2004 00:40:58 +0500 Subject: [Beowulf] Bonding in 2.4.22-1.2149 Linux kernel Message-ID: <40059B4A.6060701@imm.uran.ru> Hi, we have somewhat non-standart configuration of our computing cluster. Inernal computing nodes are connected by 2 (bonded) ethernet networks, but host connected only to one ethernet (i.e., connection to host not bonded for the all computing nodes). For the Linux kernel 2.4.9 - 2.4.20 it's possible to run such configuration if we use static arp tables and static routes (connection to host should be routed directly to the slave device, say eth1, not to bond0). But when we install 2.4.22-1.2149 (from Fedora 1), things are changed. Now routing to slave device became impossibe with diagnostics like ioctl(3, 0x890b, 0xbfe90b10) = -1 ENODEV (No such device) And there is no more slave devices in route output. Googling give us some reasons for this graceful changing of (yes, undocumented and unspecified) behaviour. It's compitely unacceptable for us to abandon bonding, so after drivers/net/bonding/bond_main.c reading we (probably) found a solution. Out idea is simply create new bonding policy. If a packet destination is in the bonded network it will call bond_xmit_roundrobin, if not then packed will be xmited directly to an appropriate slave device. MAC adresses for the direct xmiting will received by modiles as the load parameters. Probably we miss some opportunities and make some desing mistakes, because our hardware configuration is not very rare, so someone perhaps solve all our problems. We are interested in any feedback from the Beowulf community. -- Thanks for your attention, Alexandr Konovalov _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From erwan at mandrakesoft.com Wed Jan 14 11:04:52 2004 From: erwan at mandrakesoft.com (Erwan Velu) Date: Wed, 14 Jan 2004 17:04:52 +0100 Subject: [Beowulf] linuxbios In-Reply-To: References: Message-ID: <1074096292.310.43.camel@revolution.mandrakesoft.com> > I'm mostly interested in Linuxbios though for the controlability of > large clusters. Can you develop that point ? What is really intresting in such technologie ? Once your bios are well setup, you don't need to change it every day.. -- Erwan Velu Linux Cluster Distribution Project Manager MandrakeSoft 43 rue d'aboukir 75002 Paris Phone Number : +33 (0) 1 40 41 17 94 Fax Number : +33 (0) 1 40 41 92 00 Web site : http://www.mandrakesoft.com OpenPGP key : http://www.mandrakesecure.net/cks/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 14 20:36:16 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 14 Jan 2004 17:36:16 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) Message-ID: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> I'm building a cluster using wireless interconnects, and it is getting to be a royal pain to figure out what sort of adapters/interfaces to use. I'm casting some bread upon the waters here to see if anyone has fooled with this sort of thing. I want to go 802.11a because of spectrum usage, not because of data rate. If I go to 5GHz, I don't have to worry about accidentally connecting to JPL's 802.11b (2.4GHz) network infrstructure, for instance, which will keep netops off my back. The processors in the cluster are Mini-ITX widgets with Compact Flash (CF) drives, and, while booting off the net might be nice, I'm going to boot off CF for now. Here are some issues that have come up: 1) there's two ways to get the node to talk to the net: via the ethernet connector and an external bridge via a PCI card with a 802.11a adapter (most likely a 802.11a/b/g, since that's what's available) (D=Link, Netgear, and Linksys all have them) In all cases, I'd have an "access point" of some sort to talk to my head node/NFS, etc. Ideally, I'd like to set up the network in "ad-hoc" mode, where any node can talk to any other directly, without having to be routed through an access point. In "infrastructure" mode, many clients can talk to the access point, but clients cannot talk to clients, except by going through the access point, creating a single failure point (probably not important for my initial work, but philosophically "bad"). 2) It's unclear whether there are Linux drivers for any of the PCI based 802.11a cards. The mfrs don't seem to want to fool with that market, and, chipset mfrs are quite reticent about releasing the intellectual property needed to do a good job writing the drivers. 3) I could go with external bridging adapters (perhaps with integrated routers, in case I add another ethernet device to the node, or, to hook up a sniffer). Here the problem is that not all mfrs appear to support bridging, at least with more than 2 (i.e. they can set up a point to point bridge, but not a many to many bridge) From some reading of the various manuals, it appears that some "access points" can be set up to appear to be a "client" in infrastructure mode, however that's a problem philosophically (and in terms of latency). So, does anyone know which "access points" (i.e. a 802.11x to ethernet box) can look like a client in an ad-hoc network. (possible candidates: Netgear FWAG114, D-link DWL-774, DWL-7000AP, Linksys WAP54A*, WAP51AB, WRT51AB. *Linksys says that the WAP54A doesn't do bridging) Part 2 of the quest................. I'm also looking for suggestions on performance and timing tests to run on this cluster once it's assembled. Aside from the usual network throughput (benchmark program recommendations requested), I'm interested in techniques to look at latency, latency distribution, and dropped packets/retries, since I suspect that wireless networks will have very "unusual" statistics compared to the usual cluster interconnects. And, bearing in mind our recent lengthy thread on timing and clocks, you can be sure that I will do those sorts of tests too. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Wed Jan 14 22:08:23 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 15 Jan 2004 11:08:23 +0800 (CST) Subject: [Beowulf] Scalable PBS running at teragrid.org (900+ nodes)!! In-Reply-To: <020b01c3dae0$ea21a9b0$28b358c7@titan> Message-ID: <20040115030823.92280.qmail@web16806.mail.tpe.yahoo.com> >From the maintainer of xCAT: --- Egan Ford ???? > xCAT 1.2.0-pre4 has support for SPBS. There is an > SPBS maker and a newer Maui > Maker that works with 3.2.5 and 3.2.6. Other > scripts have been adjusted to > support SPBS and OpenPBS and the newer Maui's. > > We tested SPBS at scale at teragrid.org (900+ > nodes), but used early release Maui. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at iplink.net Wed Jan 14 23:07:37 2004 From: alvin at iplink.net (Alvin Starr) Date: Wed, 14 Jan 2004 23:07:37 -0500 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> Message-ID: <40061209.7060205@iplink.net> Jim Lux wrote: > I'm building a cluster using wireless interconnects, and it is getting > to be a royal pain to figure out what sort of adapters/interfaces to > use. I'm casting some bread upon the waters here to see if anyone has > fooled with this sort of thing. > > I want to go 802.11a because of spectrum usage, not because of data > rate. If I go to 5GHz, I don't have to worry about accidentally > connecting to JPL's 802.11b (2.4GHz) network infrstructure, for > instance, which will keep netops off my back. > > The processors in the cluster are Mini-ITX widgets with Compact Flash > (CF) drives, and, while booting off the net might be nice, I'm going to > boot off CF for now. > > Here are some issues that have come up: > 1) there's two ways to get the node to talk to the net: > via the ethernet connector and an external bridge > via a PCI card with a 802.11a adapter (most likely a 802.11a/b/g, since > that's what's available) (D=Link, Netgear, and Linksys all have them) take a look at the Actiontec cards. They have a line of prism based cards that are supported by linux and when I was talking with them they made sounds like they were going to stay with the prism chipset for a while. > > In all cases, I'd have an "access point" of some sort to talk to my head > node/NFS, etc. > > Ideally, I'd like to set up the network in "ad-hoc" mode, where any node > can talk to any other directly, without having to be routed through an > access point. In "infrastructure" mode, many clients can talk to the > access point, but clients cannot talk to clients, except by going > through the access point, creating a single failure point (probably not > important for my initial work, but philosophically "bad"). > > 2) It's unclear whether there are Linux drivers for any of the PCI > based 802.11a cards. The mfrs don't seem to want to fool with that > market, and, chipset mfrs are quite reticent about releasing the > intellectual property needed to do a good job writing the drivers. > > 3) I could go with external bridging adapters (perhaps with integrated > routers, in case I add another ethernet device to the node, or, to hook > up a sniffer). Here the problem is that not all mfrs appear to support > bridging, at least with more than 2 (i.e. they can set up a point to > point bridge, but not a many to many bridge) > > From some reading of the various manuals, it appears that some "access > points" can be set up to appear to be a "client" in infrastructure mode, > however that's a problem philosophically (and in terms of latency). > > So, does anyone know which "access points" (i.e. a 802.11x to ethernet > box) can look like a client in an ad-hoc network. > > (possible candidates: Netgear FWAG114, D-link DWL-774, DWL-7000AP, > Linksys WAP54A*, WAP51AB, WRT51AB. *Linksys says that the WAP54A > doesn't do bridging) > > > > > Part 2 of the quest................. > > I'm also looking for suggestions on performance and timing tests to run > on this cluster once it's assembled. Aside from the usual network > throughput (benchmark program recommendations requested), I'm interested > in techniques to look at latency, latency distribution, and dropped > packets/retries, since I suspect that wireless networks will have very > "unusual" statistics compared to the usual cluster interconnects. I think you will want to play with the payload size. It seems to make a big difference for what we are doing(net boot over wireless). > > And, bearing in mind our recent lengthy thread on timing and clocks, you > can be sure that I will do those sorts of tests too. > > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin at iplink.net || _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 00:05:24 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 14 Jan 2004 21:05:24 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <400606F4.3090802@iwantka.com> Message-ID: <001a01c3db26$bd9ed890$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Nathan Littlepage" To: "Jim Lux" Sent: Wednesday, January 14, 2004 7:20 PM Subject: Re: [Beowulf] Wireless clusters (Bridging 802.11a) > Jim, > > Just to add my two cents. Why not create your own AP with a linux > distribution thats available. There are many out there that can work on > hardware you already have. One that I have used, not in your situation, > but as an AP is from NYWireless called Pebble. > http://www.nywireless.net/pebble/. An added cent, I use ebtables to > screen MAC address to better secure access. > > Hope that helps in some manner. But, then, I'd still need some form of 802.11a wireless interface that's supported by Linux. I'd also have to spend some (scarce) time fooling with building APs. I'd rather have a basically off-the-shelf solution. I'm not real, real interested in understanding all the subtleties of 802.11 drivers under Linux. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 00:10:46 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed, 14 Jan 2004 21:10:46 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <40061209.7060205@iplink.net> Message-ID: <001b01c3db26$be0e6340$36a8a8c0@LAPTOP152422> > > take a look at the Actiontec cards. They have a line of prism based cards that are supported by linux and when I was > talking with them they made sounds like they were going to stay with the prism chipset for a while. While the prism chipset is interesting (for other reasons, particularly the Prism2, not so much 2.5 and 3 or even newer ones), they do not support the 802.11a at 5GHz form of wireless LAN. (for what it's worth, the prism set is interesting because it was originally a Harris product, then Intersil, and now GlobespanVirata. However, Harris has a long history of making military/space qualified components, and it turns out that the Prism 2 chip set is made on a fab line that is compatible with radiation tolerant processes) > > > > > In all cases, I'd have an "access point" of some sort to talk to my head > > node/NFS, etc. > > > > > Part 2 of the quest................. > > > > I'm also looking for suggestions on performance and timing tests to run > > on this cluster once it's assembled. Aside from the usual network > > throughput (benchmark program recommendations requested), I'm interested > > in techniques to look at latency, latency distribution, and dropped > > packets/retries, since I suspect that wireless networks will have very > > "unusual" statistics compared to the usual cluster interconnects. > > I think you will want to play with the payload size. It seems to make a big difference for what we are doing(net boot > over wireless). > > > Thanks.. I suspect that measurements vs packet size are of some interest. (something like HINT, except for size instead of computation). _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xavier at zeth.ciencias.uchile.cl Thu Jan 15 00:00:49 2004 From: xavier at zeth.ciencias.uchile.cl (Xavier Andrade) Date: Thu, 15 Jan 2004 02:00:49 -0300 (CLST) Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <4005637A.2050908@mn.rr.com> References: <4005637A.2050908@mn.rr.com> Message-ID: On Wed, 14 Jan 2004, Eric R Johnson wrote: > I was wondering what the scheduler options are for me if I am running a > Scyld version 28 machine. > From everything I am finding, it appears that I only have a few choices. > 1) at/bbq that ships with Scyld > 2) PBS Pro > 3) Clubmask/Maui > > I am running a very small cluster right now (16 processors) out of a > University lab and PBS Pro is really out of our price range right now. > I would like to add more nodes to the machine, but scheduling jobs is > already a problem, so I don't think that bbq is the right choice. I > have tried installing clubmask/Maui, but I ran into problems while > compiling it. I was advised that it was due to the age of BProc that > ships with Scyld. I was told that 3.2.5 is required. > > Can anybody give me some suggestions? > Try sun gridengine ( http://gridengine.sunsource.net ). Xavier _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Jan 15 00:38:38 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 15 Jan 2004 13:38:38 +0800 (CST) Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <4005637A.2050908@mn.rr.com> Message-ID: <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> Depending on your requirements, you can choose either gridengine (SGE) or spbs. SGE is opensource, and is the most popular batch system. Just ask any questions on its mailing lists and you will get answers very quickly, the people on the lists are helpful and friendly. http://gridengine.sunsource.net/ SPBS is a version of OpenPBS developed by another group of developers. It may be easier for you if you know PBS already. ** And now SPBS is called StORM (scalable open resource manager) http://supercluster.org/projects/storm/ Andrew. --- Eric R Johnson ???? > I was wondering what the scheduler options are for > me if I am running a > Scyld version 28 machine. > From everything I am finding, it appears that I > only have a few choices. > 1) at/bbq that ships with Scyld > 2) PBS Pro > 3) Clubmask/Maui > > I am running a very small cluster right now (16 > processors) out of a > University lab and PBS Pro is really out of our > price range right now. > I would like to add more nodes to the machine, but > scheduling jobs is > already a problem, so I don't think that bbq is the > right choice. I > have tried installing clubmask/Maui, but I ran into > problems while > compiling it. I was advised that it was due to the > age of BProc that > ships with Scyld. I was told that 3.2.5 is > required. > > Can anybody give me some suggestions? > > Thanks, > Eric > > -- > ******************************************************************** > Eric R A Johnson > University Of Minnesota tel: > (612) 626 5115 > Dept. of Laboratory Medicine & Pathology fax: > (612) 625 1121 > 7-230 BSBE e-mail: > john4482 at umn.edu > 312 Church Street web: > www.eric-r-johnson.com > Minneapolis, MN 55455 > USA > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shaeffer at neuralscape.com Thu Jan 15 02:20:13 2004 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed, 14 Jan 2004 23:20:13 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) In-Reply-To: <001a01c3db26$bd9ed890$36a8a8c0@LAPTOP152422> References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <400606F4.3090802@iwantka.com> <001a01c3db26$bd9ed890$36a8a8c0@LAPTOP152422> Message-ID: <20040115072013.GA26730@synapse.neuralscape.com> On Wed, Jan 14, 2004 at 09:05:24PM -0800, Jim Lux wrote: > > ----- Original Message ----- > From: "Nathan Littlepage" > To: "Jim Lux" > Sent: Wednesday, January 14, 2004 7:20 PM > Subject: Re: [Beowulf] Wireless clusters (Bridging 802.11a) > > > > Jim, > > > > Just to add my two cents. Why not create your own AP with a linux > > distribution thats available. There are many out there that can work on > > hardware you already have. One that I have used, not in your situation, > > but as an AP is from NYWireless called Pebble. > > http://www.nywireless.net/pebble/. An added cent, I use ebtables to > > screen MAC address to better secure access. > > > > Hope that helps in some manner. > > But, then, I'd still need some form of 802.11a wireless interface that's > supported by Linux. I'd also have to spend some (scarce) time fooling with > building APs. I'd rather have a basically off-the-shelf solution. I'm not > real, real interested in understanding all the subtleties of 802.11 drivers > under Linux. Hello, I did fool around with Linux hosted 802.11b APs. It is straightforward. I use the hostap code from: http://hostap.epitest.fi/ works like a charm. It's actually fun stuff. You've pointed out that these prism based systems do not support 802.11a. Obviously a big problem for you. But, you may not be aware that the prism chips do support the full range of operating modes, including some obscure test modes that are not published. That hostap software supports many of them. Good luck, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer at neuralscape.com http://www.neuralscape.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From pesch at attglobal.net Thu Jan 15 13:01:13 2004 From: pesch at attglobal.net (pesch at attglobal.net) Date: Thu, 15 Jan 2004 10:01:13 -0800 Subject: [Beowulf] linuxbios References: Message-ID: <4006D569.5AC38A66@attglobal.net> The time is ripe for a "cluster bios" - it could make life a lot easier if it had all the right features. What would these features be in your opinion? Paul Schenker John Hearns wrote: > I've been following it for a couple of years. > > At the Linux Beer Hike in Ireland in 2002 we did some experiments, > and got it running. I've stilll got the PC-chips motherboard which > I bought to do the tests. Antony Stone found a UK supplier if the > disk-on-chip parts. > Antony did a very good article for the UK version Linux Magazine. > > I haven't had the courage though to try Linuxbios on my precious > mini-ITXes. > > I'm mostly interested in Linuxbios though for the controlability of > large clusters. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Thu Jan 15 07:27:21 2004 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Thu, 15 Jan 2004 13:27:21 +0100 Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> References: <4005637A.2050908@mn.rr.com> <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> Message-ID: <20040115132721U.hanzl@unknown-domain> > SGE is opensource, and is the most popular batch > system. Just ask any questions on its mailing lists > and you will get answers very quickly, the people on > the lists are helpful and friendly. > > http://gridengine.sunsource.net/ SGE is superb but did anybody actually make it to work with bproc-based cluster? Is there already something better than my hack: http://noel/magi/sge+bproc.html ? If yes, please let me know, I would be happy to document any success on this page. Regards Vaclav _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 15 07:58:24 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 15 Jan 2004 07:58:24 -0500 (EST) Subject: [Beowulf] Solaris on x86? In-Reply-To: <20040114215108.GA2292@greglaptop.internal.keyresearch.com> Message-ID: On Wed, 14 Jan 2004, Greg Lindahl wrote: > As for ROCKS, it's using a pure-GPL version of RedHat Enterprise > something or other. So it's less odd than you might think. You might > want to explore what your company thinks "mainstream" means... I don't > think I get the sense in which they use the word. I think it must be like the IBM ad with the spooky kid, the orphan that nobody owned, that was adopted by the world and put to all sorts of heavy lifting work (oh child labor laws, where are you now). Maybe they take the kid out in some kind of stream to drown him? Very confusing. [And what's wrong with penguins, I want to know... what's with this "kid" metaphor that looks like it belongs on the SciFi channel?] Seriously, linux is both mainstream and pretty all-around excellent in this (cluster/server) space. You could argue that it isn't mainstream on the office desktop or as a home computer (although the kid is inexorably moving out into that stream as well in many places) but in the server room linux has been a solid player for a long time and in clusters it is by far the OS of choice. I mean, it IS only the most stable, scalable and maintainable operating system the world has ever seen... rgb, trying his coffee withOUT sugar this morning Time to go take my, um, "vitamins". -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 15 08:48:49 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 15 Jan 2004 08:48:49 -0500 (EST) Subject: [Beowulf] benchmarking network and performance In-Reply-To: <001b01c3db26$be0e6340$36a8a8c0@LAPTOP152422> Message-ID: On Wed, 14 Jan 2004, Jim Lux wrote: > Thanks.. I suspect that measurements vs packet size are of some interest. > (something like HINT, except for size instead of computation). netpipe. netperf, too, but nobody loves it any more. netpipe will let you transparently test raw TCP, PVM and MPI connections side by side, for varying packet sizes, with streams of packets long enough to get decent statistics on them, and autogenerate a nice little report on same. Most of what you learn from such a test on ethernet is that small packets are latency bound and the interface sends as many of them per second as it can (given the latency) largely independent of packet size. Consequently bandwidth grows linearly with packet size. At some point the volume of data being transmitted STARTS to actually run into the wire bandwidth (if you like, the linear growth curve gets close to the wirespeed saturation/constant line) and the linear growth curve gently bends over and saturates sort-of-exponentially, typically at some 90% of wirespeed (for a decent interface), typically when packet sizes reach roughly 1K in size (maybe 2/3 of the standard MTU). Sometimes there is interesting "structure" in the curve -- "often" and "egregious" structure for poor interfaces, less often for good ones but sometimes you'll see something like dips or peaks around certain packet size boundaries. Oh, and lmbench has a variety of network latency and bw tests (tcp and udp and localhost) mixed in with the rest of its microbenchmarks. Maybe a bit clunky to use compared to netpipe but you wanted other benchmark tools as well and this is a suite of them. Finally, for other benchmark tools there is cpu_rate, which includes stream (with malloc and a sweep of vector sizes) as well as the savage benchmark and some other stream-like arithmetic tests. If your organization participates in SPEC, there is SPEC, and of course the list archives contain pointers to various parallel performance benchmarks. rgb > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Thu Jan 15 10:46:22 2004 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Thu, 15 Jan 2004 16:46:22 +0100 Subject: [Beowulf] Schedulers for use with Scyld Version 28 In-Reply-To: <20040115132721U.hanzl@unknown-domain> References: <4005637A.2050908@mn.rr.com> <20040115053838.50835.qmail@web16805.mail.tpe.yahoo.com> <20040115132721U.hanzl@unknown-domain> Message-ID: <20040115164622K.hanzl@unknown-domain> > SGE is superb but did anybody actually make it to work with > bproc-based cluster? > > Is there already something better than my hack: > > http://noel/magi/sge+bproc.html > > ? The URL should have been: http://noel.feld.cvut.cz/magi/sge+bproc.html Sorry for my stupid mistake. Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 10:05:19 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 15 Jan 2004 07:05:19 -0800 Subject: [Beowulf] Wireless clusters (Bridging 802.11a) References: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> <40061209.7060205@iplink.net> <001b01c3db26$be0e6340$36a8a8c0@LAPTOP152422> <400698AF.4050409@iplink.net> Message-ID: <002901c3db79$431fd030$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "alvin" To: "Jim Lux" Cc: Sent: Thursday, January 15, 2004 5:42 AM Subject: Re: [Beowulf] Wireless clusters (Bridging 802.11a) > J>> > >> > >prism chipset for a while. > > > >While the prism chipset is interesting (for other reasons, particularly the > >Prism2, not so much 2.5 and 3 or even newer ones), they do not support the > >802.11a at 5GHz form of wireless LAN. (for what it's worth, the prism set is > >interesting because it was originally a Harris product, then Intersil, and > >now GlobespanVirata. However, Harris has a long history of making > >military/space qualified components, and it turns out that the Prism 2 chip > >set is made on a fab line that is compatible with radiation tolerant > >processes) > > > > > Just took a cruse through the GlobespanVirata site and it looks like > they have a new chipset that will do most of the various 802.11 protocols. > I wonder if there are linux drivers or programming information available > yet. I hope they have not gone the same way of the other chipset > manufacturers bykeeping the programming inerface propritary. You'll note that there are no datasheets on the Globespan Virata web site. Just block diagrams, and a page to contact marketing. Their latest offerings are highly integrated, and the details of the onchip processor doing the backend stuff are proprietary. > > > will you require a radation hardened product if you get to production? Probably not radiation hard, just radiation tolerant. TID of, say, 20-100 kRad, no latchup (or, some decent form of latchup mitigation). For now, though, this is just a proof of concept prototype to show that a wireless interconnect can work in this particular application. Down the road a bit, moderate integration RFICs (such as in the Prism II chipset) that are radiation tolerant would be of great interest (of course, by then, the masks will be 10 years old and you'll have to buy the fab line on E-bay.. such is life when designing flight hardware) > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 15 08:31:42 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 15 Jan 2004 08:31:42 -0500 (EST) Subject: [Beowulf] Wireless clusters (Bridging 802.11a) In-Reply-To: <5.2.0.9.2.20040114171238.030cdb38@mailhost4.jpl.nasa.gov> Message-ID: On Wed, 14 Jan 2004, Jim Lux wrote: > I'm building a cluster using wireless interconnects, and it is getting to > be a royal pain to figure out what sort of adapters/interfaces to use. I'm > casting some bread upon the waters here to see if anyone has fooled with > this sort of thing. > > I want to go 802.11a because of spectrum usage, not because of data > rate. If I go to 5GHz, I don't have to worry about accidentally connecting > to JPL's 802.11b (2.4GHz) network infrstructure, for instance, which will > keep netops off my back. I don't think I can help you with anything below, but I can point out an interesting and so far unexplained anomaly I've encountered with 802.11b. While working on a PVM article for CWM I often naturally end up sitting in my den downstairs working on my laptop where I can keep an eye on our neurotic and fey border collie puppy so he doesn't eat a table or rip a hole in a wall or something. That's fine, I've got switched 100BT systems a.k.a. "nodes" all over the house and when they aren't running Diablo II on top of winex they even have respectable performance. I can run PVM on the laptop just fine. I can login with ssh transparently (no password) to all the systems in the household just fine. I can login to one of those systems, crank PVM on it, and add the laptop (over the wireless connection) just fine. BUT, when I run PVM on the laptop and try to add the SAME host that worked fine the other way, I get: rgb at lilith|T:125>pvm pvm> add lucifer add lucifer libpvm [t40001]: pvm_addhosts(): Pvmd system error This is both not terribly informative and frustrating. PVM has an internal autodiagnostic mode these days that warns you of e.g. leftover lockfiles, inability to ssh and so forth -- this doesn't even make it that far. Running pvm -d0xffff (to get a debug trace) isn't much more helpful although it might be to somebody who really knows the PVM internals -- all the daemon's progress in setting up the connection proceeds just as it ought up to the final libpvm [t40001] mxinput() pkt src t80000000 len 36 ff 3 libpvm [t40001] mxinput() src t80000000 route t80000000 ctx 524286 tag TM_ADDHOST len 4 libpvm [t40001] mesg_input() src t80000000 ctx 524286 tag TM_ADDHOST len 4 libpvm [t40001] mxfer() txfp 0 gotem 1 tt_rxf 0 libpvm [t40001] msendrecv() from t80000000 tag TM_ADDHOST libpvm [t40001]: pvm_addhosts(): Pvmd system error and then it fails where in an identical trace going the other way it succeeds, no indication why in either case. At a guess it is either: a) I'm doing something stupid and silly somewhere. Always a possibility, although I >>have<< been using PVM since 1993 and generally can make it behave, especially since they added all the nifty diagnostics to it. b) Something "odd" about the wireless networking stack or the PCMCIA bus where (perhaps) PVM does something at a low level for speed/efficiency that just won't work with wireless. The latter is a bit disconcerting to me -- one would think that all transactions occur on top of virtual devices that keep wireless from being anything but "a network interface", although there COULD be either timeouts or packet reliability problems, of course. And the wireless network on the laptop works flawlessly otherwise as long as signal strength is adequate, and of course PVM works TO the wireless system but not FROM the wireless system. Anyway, I offer this up not because I want or need help with it (it would be silly to use my wireless laptop as a pvm master node in anything BUT writing an article or an EP task of some sort and there are other ways to accomplish both of these that are a lot less work than hacking the PVM sources to debug the failure) but because YOU might encounter similar problems and have to invest the extra time to debug the failure. Moral of story, don't assume that just because a wireless interface has device support under linux and works perfectly with simple network tests it will necessarily work with PVM, MPI, tools that access a network stack at a low level for "efficiency" reasons and presume e.g. ethernet. rgb > > The processors in the cluster are Mini-ITX widgets with Compact Flash (CF) > drives, and, while booting off the net might be nice, I'm going to boot off > CF for now. > > Here are some issues that have come up: > 1) there's two ways to get the node to talk to the net: > via the ethernet connector and an external bridge > via a PCI card with a 802.11a adapter (most likely a 802.11a/b/g, since > that's what's available) (D=Link, Netgear, and Linksys all have them) > > In all cases, I'd have an "access point" of some sort to talk to my head > node/NFS, etc. > > Ideally, I'd like to set up the network in "ad-hoc" mode, where any node > can talk to any other directly, without having to be routed through an > access point. In "infrastructure" mode, many clients can talk to the access > point, but clients cannot talk to clients, except by going through the > access point, creating a single failure point (probably not important for > my initial work, but philosophically "bad"). > > 2) It's unclear whether there are Linux drivers for any of the PCI based > 802.11a cards. The mfrs don't seem to want to fool with that market, and, > chipset mfrs are quite reticent about releasing the intellectual property > needed to do a good job writing the drivers. > > 3) I could go with external bridging adapters (perhaps with integrated > routers, in case I add another ethernet device to the node, or, to hook up > a sniffer). Here the problem is that not all mfrs appear to support > bridging, at least with more than 2 (i.e. they can set up a point to point > bridge, but not a many to many bridge) > > From some reading of the various manuals, it appears that some "access > points" can be set up to appear to be a "client" in infrastructure mode, > however that's a problem philosophically (and in terms of latency). > > So, does anyone know which "access points" (i.e. a 802.11x to ethernet box) > can look like a client in an ad-hoc network. > > (possible candidates: Netgear FWAG114, D-link DWL-774, DWL-7000AP, Linksys > WAP54A*, WAP51AB, WRT51AB. *Linksys says that the WAP54A doesn't do bridging) > > > > > Part 2 of the quest................. > > I'm also looking for suggestions on performance and timing tests to run on > this cluster once it's assembled. Aside from the usual network throughput > (benchmark program recommendations requested), I'm interested in techniques > to look at latency, latency distribution, and dropped packets/retries, > since I suspect that wireless networks will have very "unusual" statistics > compared to the usual cluster interconnects. > > And, bearing in mind our recent lengthy thread on timing and clocks, you > can be sure that I will do those sorts of tests too. > > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Jan 15 10:13:13 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Thu, 15 Jan 2004 07:13:13 -0800 Subject: [Beowulf] Re: benchmarking network and performance References: Message-ID: <003101c3db7a$18cac1e0$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Robert G. Brown" To: "Jim Lux" Cc: "Alvin Starr" ; ; Sent: Thursday, January 15, 2004 5:48 AM Subject: benchmarking network and performance > On Wed, 14 Jan 2004, Jim Lux wrote: > > > Thanks.. I suspect that measurements vs packet size are of some interest. > > (something like HINT, except for size instead of computation). > > netpipe. netperf, too, but nobody loves it any more. netpipe will let > you transparently test raw TCP, PVM and MPI connections side by side, > for varying packet sizes, with streams of packets long enough to get > decent statistics on them, and autogenerate a nice little report on > same. > > Most of what you learn from such a test on ethernet is that small > packets are latency bound and the interface sends as many of them per > second as it can (given the latency) largely independent of packet size. > Consequently bandwidth grows linearly with packet size. At some point > the volume of data being transmitted STARTS to actually run into the > wire bandwidth (if you like, the linear growth curve gets close to the > wirespeed saturation/constant line) and the linear growth curve gently > bends over and saturates sort-of-exponentially, typically at some 90% of > wirespeed (for a decent interface), typically when packet sizes reach > roughly 1K in size (maybe 2/3 of the standard MTU). Of course, since it's a wireless channel, the "wire bandwidth" is somewhat ill defined, at least in terms of a classical shared media ethernet type situation. 802.11 is also half duplex, so even though the wirespeed (the over the air bit rate) is 11 Mbps, the actual through put could be quite a bit less, especially if it's an "ack/nak" kind of protocol being used in the transfer. UDP could be a LOT faster than TCP, for instance, because the UDP packets could just stream on out. > > Sometimes there is interesting "structure" in the curve -- "often" and > "egregious" structure for poor interfaces, less often for good ones but > sometimes you'll see something like dips or peaks around certain packet > size boundaries. > > Oh, and lmbench has a variety of network latency and bw tests (tcp and > udp and localhost) mixed in with the rest of its microbenchmarks. Maybe > Thanks for the suggestions Jim _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From josip at lanl.gov Thu Jan 15 16:01:22 2004 From: josip at lanl.gov (Josip Loncaric) Date: Thu, 15 Jan 2004 14:01:22 -0700 Subject: [Beowulf] linuxbios In-Reply-To: <1074096292.310.43.camel@revolution.mandrakesoft.com> References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: <4006FFA2.3080907@lanl.gov> Erwan Velu wrote: >>I'm mostly interested in Linuxbios though for the controlability of >>large clusters. > > Can you develop that point ? What is really intresting in such > technologie ? Once your bios are well setup, you don't need to change it > every day.. Setting up BIOS is a pain. Managing 100+ machines and making sure that they all run the same version, with the same options, and upgrading BIOS if it isn't quite right is *tons* of fun, particularly if you need to plug in a monitor and keyboard to do it. By contrast, LinuxBIOS gives you much better control (see http://www.linuxbios.org/). Josip _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Thu Jan 15 15:52:57 2004 From: nixon at nsc.liu.se (Leif Nixon) Date: Thu, 15 Jan 2004 21:52:57 +0100 Subject: [Beowulf] linuxbios In-Reply-To: <1074096292.310.43.camel@revolution.mandrakesoft.com> (Erwan Velu's message of "Wed, 14 Jan 2004 17:04:52 +0100") References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: Erwan Velu writes: >> I'm mostly interested in Linuxbios though for the controlability of >> large clusters. > Can you develop that point ? What is really intresting in such > technologie ? Once your bios are well setup, you don't need to change it > every day.. Right now I'd like to evaluate the performance of a 100 node cluster with and without hyperthreading activated. I don't look forward to fiddle those BIOS settings, I can tell you. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From domingo at netcomp.com.br Thu Jan 15 08:45:50 2004 From: domingo at netcomp.com.br (Domingo Antonio) Date: Thu, 15 Jan 2004 11:45:50 -0200 Subject: [Beowulf] Re: MySQL clustering Message-ID: <001501c3db6d$e37db350$2101a8c0@blue> Hi all!! Is mysql capable to work with beowulf ? I?m intend to make a beowulf cluster to make my website more fast! Actualy a have an apache + mysql server, but they not suport requests... I tried, but no suces, to make mysql replication.. the performance was good but not good enoght... Is possible to make mysql work over beowulf? Thanks a lot!!! Domingo Antonio _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgoornaden at intnet.mu Thu Jan 15 23:12:04 2004 From: rgoornaden at intnet.mu (roudy) Date: Fri, 16 Jan 2004 08:12:04 +0400 Subject: [Beowulf] HPL References: <200401151701.i0FH1ph31020@NewBlue.scyld.com> Message-ID: <000a01c3dbe7$1358f070$b8097bca@roudy> Hi Can someone tell me what cause the installation process of HPL to stop, below is the copy of what has been done so far. Thanks Roudy [root at localhost hpl]# make arch=cluster make -f Make.top startup_dir arch=cluster make[1]: Entering directory `/home/hpl' mkdir include/cluster mkdir lib mkdir: cannot create directory `lib': File exists make[1]: [startup_dir] Error 1 (ignored) mkdir lib/cluster mkdir bin mkdir: cannot create directory `bin': File exists make[1]: [startup_dir] Error 1 (ignored) mkdir bin/cluster make[1]: Leaving directory `/home/hpl' make -f Make.top startup_src arch=cluster make[1]: Entering directory `/home/hpl' make -f Make.top leaf le=src/auxil arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/auxil ; mkdir cluster ) ( cd src/auxil/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/blas arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/blas ; mkdir cluster ) ( cd src/blas/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/comm arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/comm ; mkdir cluster ) ( cd src/comm/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/grid arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/grid ; mkdir cluster ) ( cd src/grid/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/panel arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/panel ; mkdir cluster ) ( cd src/panel/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/pauxil arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/pauxil ; mkdir cluster ) ( cd src/pauxil/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/pfact arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/pfact ; mkdir cluster ) ( cd src/pfact/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=src/pgesv arch=cluster make[2]: Entering directory `/home/hpl' ( cd src/pgesv ; mkdir cluster ) ( cd src/pgesv/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make[1]: Leaving directory `/home/hpl' make -f Make.top startup_tst arch=cluster make[1]: Entering directory `/home/hpl' make -f Make.top leaf le=testing/matgen arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/matgen ; mkdir cluster ) ( cd testing/matgen/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/timer arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/timer ; mkdir cluster ) ( cd testing/timer/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/pmatgen arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/pmatgen ; mkdir cluster ) ( cd testing/pmatgen/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/ptimer arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/ptimer ; mkdir cluster ) ( cd testing/ptimer/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make -f Make.top leaf le=testing/ptest arch=cluster make[2]: Entering directory `/home/hpl' ( cd testing/ptest ; mkdir cluster ) ( cd testing/ptest/cluster ; \ ln -s home/hpl/Make.cluster Make.inc ) make[2]: Leaving directory `/home/hpl' make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_src arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.auxil src/auxil/cluster/Makefile cp makes/Make.blas src/blas/cluster/Makefile cp makes/Make.comm src/comm/cluster/Makefile cp makes/Make.grid src/grid/cluster/Makefile cp makes/Make.panel src/panel/cluster/Makefile cp makes/Make.pauxil src/pauxil/cluster/Makefile cp makes/Make.pfact src/pfact/cluster/Makefile cp makes/Make.pgesv src/pgesv/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_tst arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.matgen testing/matgen/cluster/Makefile cp makes/Make.timer testing/timer/cluster/Makefile cp makes/Make.pmatgen testing/pmatgen/cluster/Makefile cp makes/Make.ptimer testing/ptimer/cluster/Makefile cp makes/Make.ptest testing/ptest/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_src arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.auxil src/auxil/cluster/Makefile cp makes/Make.blas src/blas/cluster/Makefile cp makes/Make.comm src/comm/cluster/Makefile cp makes/Make.grid src/grid/cluster/Makefile cp makes/Make.panel src/panel/cluster/Makefile cp makes/Make.pauxil src/pauxil/cluster/Makefile cp makes/Make.pfact src/pfact/cluster/Makefile cp makes/Make.pgesv src/pgesv/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top refresh_tst arch=cluster make[1]: Entering directory `/home/hpl' cp makes/Make.matgen testing/matgen/cluster/Makefile cp makes/Make.timer testing/timer/cluster/Makefile cp makes/Make.pmatgen testing/pmatgen/cluster/Makefile cp makes/Make.ptimer testing/ptimer/cluster/Makefile cp makes/Make.ptest testing/ptest/cluster/Makefile make[1]: Leaving directory `/home/hpl' make -f Make.top build_src arch=cluster make[1]: Entering directory `/home/hpl' ( cd src/auxil/cluster; make ) make[2]: Entering directory `/home/hpl/src/auxil/cluster' Makefile:47: Make.inc: No such file or directory make[2]: *** No rule to make target `Make.inc'. Stop. make[2]: Leaving directory `/home/hpl/src/auxil/cluster' make[1]: *** [build_src] Error 2 make[1]: Leaving directory `/home/hpl' make: *** [build] Error 2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From domingo2 at netcomp.com.br Wed Jan 14 19:07:14 2004 From: domingo2 at netcomp.com.br (Domingo Antonio) Date: Wed, 14 Jan 2004 22:07:14 -0200 Subject: [Beowulf] MySQL clustering Message-ID: <00df01c3dafb$88094060$2101a8c0@blue> Hi all!! Is mysql capable to work with beowulf ? I?m intend to make a beowulf cluster to make my website more fast! Actualy a have an apache + mysql server, but they not suport requests... I tried, but no suces, to make mysql replication.. the performance was good but not good enoght... Is possible to make mysql work over beowulf? Thanks a lot!!! Domingo Antonio -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars at meshtechnologies.com Fri Jan 16 01:48:20 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: 16 Jan 2004 06:48:20 +0000 Subject: [Beowulf] linuxbios In-Reply-To: References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: <1074235700.2279.2.camel@fermi> On Thu, 2004-01-15 at 20:52, Leif Nixon wrote: > Right now I'd like to evaluate the performance of a 100 node cluster > with and without hyperthreading activated. I don't look forward to > fiddle those BIOS settings, I can tell you. Well, the easy solution is to disable SMP support in the kernel. That can be done remote, unlike tedious BIOS config :-) best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Fri Jan 16 04:34:50 2004 From: nixon at nsc.liu.se (Leif Nixon) Date: Fri, 16 Jan 2004 10:34:50 +0100 Subject: [Beowulf] linuxbios In-Reply-To: <1074235700.2279.2.camel@fermi> (Lars Henriksen's message of "16 Jan 2004 06:48:20 +0000") References: <1074096292.310.43.camel@revolution.mandrakesoft.com> <1074235700.2279.2.camel@fermi> Message-ID: Lars Henriksen writes: > On Thu, 2004-01-15 at 20:52, Leif Nixon wrote: > >> Right now I'd like to evaluate the performance of a 100 node cluster >> with and without hyperthreading activated. I don't look forward to >> fiddle those BIOS settings, I can tell you. > > Well, the easy solution is to disable SMP support in the kernel. > That can be done remote, unlike tedious BIOS config :-) Not if the initial setting is hyperthreading deactivated. Which it is in my case. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Fri Jan 16 07:58:23 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: Fri, 16 Jan 2004 12:58:23 +0000 Subject: [Beowulf] linuxbios In-Reply-To: References: <1074096292.310.43.camel@revolution.mandrakesoft.com> <1074235700.2279.2.camel@fermi> Message-ID: <1074257903.4988.4.camel@tp1.mesh-hq> On Fri, 2004-01-16 at 09:34, Leif Nixon wrote: > >> Right now I'd like to evaluate the performance of a 100 node cluster > >> with and without hyperthreading activated. I don't look forward to > >> fiddle those BIOS settings, I can tell you. > > > > Well, the easy solution is to disable SMP support in the kernel. > > That can be done remote, unlike tedious BIOS config :-) > > Not if the initial setting is hyperthreading deactivated. Which it is > in my case. No, of course not, my bad. I assumed that it was enabled :-) In that case I can understand your point. best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5230 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Fri Jan 16 08:11:08 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Fri, 16 Jan 2004 05:11:08 -0800 (PST) Subject: [Beowulf] my beowulf problem Message-ID: <20040116131108.94673.qmail@web21506.mail.yahoo.com> hello friends i am very new at beowulf cluster i need your help i have tried following the initial advises but still unsuccesfull i get a following error i dont understand why is the lam environment req for mpirun it would be requ for lam/mpim implementataion [mpich at tung /]$ mpirun -np 4 cpilog ----------------------------------------------------------------------------- It seems that there is no lamd running on this host, which indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "mpirun" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. ----------------------------------------------------------------------------- __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 16 07:21:13 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 16 Jan 2004 07:21:13 -0500 (EST) Subject: [Beowulf] MySQL clustering In-Reply-To: <00df01c3dafb$88094060$2101a8c0@blue> Message-ID: On Wed, 14 Jan 2004, Domingo Antonio wrote: > Hi all!! > > Is mysql capable to work with beowulf ? > > I?m intend to make a beowulf cluster to make my website more fast! > > Actualy a have an apache + mysql server, but they not suport > requests... I tried, but no suces, to make mysql replication.. the > performance was good but not good enoght... > > Is possible to make mysql work over beowulf? To some extent yes, but this is the wrong list venue for the question. What you want is "high availability computing", not "high performance computing". There is a linux site devoted to it, and of course it is of great interest to and in significant use in corporate america. There is even a bit of crossover with this list -- distributing some sort of serious computation (as opposed to database transactions) front ended by a web interface, for example, but most of what HA linux focusses on is failover/reliability, delivery speed, handling lots of "transactions". BTW, two more comments. One is that in many cases web performance is at least partly bottlenecked by your net interface or I/O subsystem (disk), not the CPU. These are often difficult to share, and without sharing and parallelization there is little point in multiplexing your server arrangement and you'll do a lot of work for nothing. Multiplexing DB transactions IS one of the places where it can matter, if you can distribute multiple copies of the DB and keep them in sync as transactions proceed. A second comment is that on one or two of the lists that I'm on (not this one) I've heard significant rants against mysql as a professional quality DB. Note that this is NOT such a rant -- I use mysql myself for a small DB-driven php website, but note the key word: "small". Being a packrat, I squirrelled away the links. One is: http://openacs.org/philosophy/why-not-mysql.html and another is: http://sql-info.de/mysql/gotchas.html#1_1 A rant against mysql in general, and a fairly specific list of gotchas that have bitten other developers who looked for some specific SQL feature and found it lacking or broken. They tend to recommend using either postgres or oracle for anything more serious than alphabetizing your record collection (I use it to track campus clusters, which is just about that complex, and use SQL with a book in one hand anyway, so to me it matters little). Just FYI. rgb > > > Thanks a lot!!! > > Domingo Antonio -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From wade.hampton at nsc1.net Fri Jan 16 08:17:21 2004 From: wade.hampton at nsc1.net (Wade Hampton) Date: Fri, 16 Jan 2004 08:17:21 -0500 Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: References: Message-ID: <4007E461.6050705@nsc1.net> Douglas Eadline, Cluster World Magazine wrote: >On Wed, 14 Jan 2004, Joshua Baker-LePain wrote: > > > >>I just moved two dual Xeon servers into a new home. One of them refuses >>to see a network link, before or after the machine boots. The only >>difference in them is a slight revision in the motherboard: the one that >>links has a Supermicro X5DPE-G2 (e7501 chipset), and the one that doesn't >>has a SM P4DPE-G2 (e7500). Both have the same onboard network chip (Intel >>82546EB -- dual port gigabit). They're connected to a Cisco Catalyst >>4507R. >> >> > >You might first try looking for any messages in /var/log/messages >when you load the driver module. I assume you are using e1000.o. >Are the versions identical on both systems ? > >The output from things like > >uname -a >lsmod > >may be a good place to start. BTW, with Intel chips-sets showing link to a >switch does not necessarily imply that the kernel module is working. > >Doug > > > >>The box *will* link when connected to a 100Mb switch. I also hooked it >>up to a Dell 5224 (gigabit) switch that I have, and it found a link after >>about 10 seconds. >> >>Any ideas? I've already tried flashing the BIOS and playing with mii-tool >>a bit. >> >> I have several SuperMicro dual 2.2 xeon boxes and am having NIC problems with one as well. It is also an E7500 chipset with 82546EB NIC. The eth0 would not connect to a GB switch (don't have the make right now). Eth1worked fine so I had to use eth1 instead of eth0. When I get the box back next week, I'll find out if this was a hardware problem or similar to the one reported. Note: This box was borrowed from my cluster and is running Fedora Core 1.0 with the 2115 kernel and is somewhat unstable (several crashes) - yet another issue. Sorry I don't have more information at this time as the box is at another location. -- Wade Hampton _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Fri Jan 16 08:10:13 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Fri, 16 Jan 2004 08:10:13 -0500 (EST) Subject: [Beowulf] linuxbios In-Reply-To: Message-ID: On Thu, 15 Jan 2004, Leif Nixon wrote: > Erwan Velu writes: > > >> I'm mostly interested in Linuxbios though for the controlability of > >> large clusters. > > Can you develop that point ? What is really intresting in such > > technologie ? Once your bios are well setup, you don't need to change it > > every day.. > > Right now I'd like to evaluate the performance of a 100 node cluster > with and without hyperthreading activated. I don't look forward to > fiddle those BIOS settings, I can tell you. Depending on what you want to do, you can probably get a "feel" for the HT vs no HT with a handful of machines or even singel machine. Basically, HT is not a win for cluster/parallel applications. Things can actually run slower. Doug > > -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine Desk: 610.865.6061 Cell: 610.390.7765 Redefining High Performance Computing Fax: 610.865.6618 www.clusterworld.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Fri Jan 16 08:16:48 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Fri, 16 Jan 2004 05:16:48 -0800 (PST) Subject: [Beowulf] do lam and mpich go together Message-ID: <20040116131648.80093.qmail@web21502.mail.yahoo.com> hello everybody can anybody help me to solve mt problem i have tried mpich-1.2.5.2 implimentation of mpi but i get the foll error for mpirun do we req lam environment for mpirun comand acc to my knowld it is req for lam/mpi implementation of mpi can any body send me the link or doc for perfect instalation of mpich-1.2.5.2 on cluster of 3 machine with red hat 9.0 [mpich at tung /]$ mpirun -np 4 cpilog ----------------------------------------------------------------------------- It seems that there is no lamd running on this host, which indicates that the LAM/MPI runtime environment is not operating. The LAM/MPI runtime environment is necessary for the "mpirun" command. Please run the "lamboot" command the start the LAM/MPI runtime environment. See the LAM/MPI documentation for how to invoke "lamboot" across multiple machines. ----------------------------------------------------------------------------- __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmoyer at redhat.com Fri Jan 16 08:26:09 2004 From: jmoyer at redhat.com (Jeff Moyer) Date: Fri, 16 Jan 2004 08:26:09 -0500 Subject: [Beowulf] linuxbios In-Reply-To: References: <1074096292.310.43.camel@revolution.mandrakesoft.com> Message-ID: <16391.58993.771197.287433@segfault.boston.redhat.com> ==> Regarding Re: [Beowulf] linuxbios; Leif Nixon adds: nixon> Erwan Velu writes: >>> I'm mostly interested in Linuxbios though for the controlability of >>> large clusters. >> Can you develop that point ? What is really intresting in such >> technologie ? Once your bios are well setup, you don't need to change it >> every day.. nixon> Right now I'd like to evaluate the performance of a 100 node cluster nixon> with and without hyperthreading activated. I don't look forward to nixon> fiddle those BIOS settings, I can tell you. Well, once you've enabled HT in the BIOS, there's at leaset a kernel command line switch to disable it in the kernel (noht). -Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bkmacy at dali.ppco.com Fri Jan 16 09:58:59 2004 From: bkmacy at dali.ppco.com (Brian Macy) Date: Fri, 16 Jan 2004 08:58:59 -0600 Subject: [Beowulf] do lam and mpich go together References: <200401161329.i0GDTrh09638@NewBlue.scyld.com> Message-ID: <4007FC33.5050905@dali.ppco.com> beowulf-request at scyld.com wrote: >Message: 12 >Date: Fri, 16 Jan 2004 05:16:48 -0800 (PST) >From: prakash borade >To: beowulf at beowulf.org >Subject: [Beowulf] do lam and mpich go together > > >hello everybody > >can anybody help me to solve mt problem >i have tried mpich-1.2.5.2 implimentation of mpi > >but i get the foll error for mpirun >do we req lam environment for mpirun comand >acc to my knowld it is req for lam/mpi implementation >of mpi >can any body send me the link or doc for perfect >instalation of mpich-1.2.5.2 on cluster of 3 machine >with red hat 9.0 > > > >[mpich at tung /]$ mpirun -np 4 cpilog >----------------------------------------------------------------------------- >It seems that there is no lamd running on this host, >which indicates >that the LAM/MPI runtime environment is not operating. > The LAM/MPI >runtime environment is necessary for the "mpirun" >command. > >Please run the "lamboot" command the start the LAM/MPI >runtime >environment. See the LAM/MPI documentation for how to >invoke >"lamboot" across multiple machines. >----------------------------------------------------------------------------- > > > Prakash, It looks like you have LAM installed as well, and that you're picking up the mpirun from LAM instead of MPICH. Do a "which mpirun" to see where mpirun lives. You could provide the full path to mpirun for MPICH to ensure you're getting the right one; e.g., /usr/local/mpich-1.2.5.2/bin/mpirun -np 4 cpilog HTH, Brian _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From luis.licon at cimav.edu.mx Fri Jan 16 11:42:48 2004 From: luis.licon at cimav.edu.mx (luis.licon at cimav.edu.mx) Date: Fri, 16 Jan 2004 09:42:48 -0700 (MST) Subject: [Beowulf] Re: MySQL clustering In-Reply-To: <001501c3db6d$e37db350$2101a8c0@blue> References: <001501c3db6d$e37db350$2101a8c0@blue> Message-ID: <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> You have two options, the first one is DNS Round Robin, and the second one is Load Balance, the second one has a better performance indeed, you could try this option with PEN, and you can get it on: http://siag.nu/pen Cheerz, Luis > Hi all!! > > Is mysql capable to work with beowulf ? > I?m intend to make a beowulf cluster to make my website more fast! > Actualy a have an apache + mysql server, but they not suport > requests... I tried, but no suces, to make mysql replication.. the > performance was good but not good enoght... > > Is possible to make mysql work over beowulf? > > > Thanks a lot!!! > > Domingo Antonio > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gmpc at sanger.ac.uk Fri Jan 16 11:43:33 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Fri, 16 Jan 2004 16:43:33 +0000 (GMT) Subject: [Beowulf] MySQL clustering In-Reply-To: <200401161332.i0GDWfh10058@NewBlue.scyld.com> References: <200401161332.i0GDWfh10058@NewBlue.scyld.com> Message-ID: >quality DB. Note that this is NOT such a rant -- I use mysql myself for >a small DB-driven php website, but note the key word: "small". It depends on your value of small: df -k /mysql* Filesystem Size Used Available Capacity Mounted on mysqla#mysqla 209715200 136332008 73363504 66% /mysqla mysqlb#mysqlb 209715200 192189552 17504864 92% /mysqlb mysqlc#mysqlc 209715200 127955469 81740544 62% /mysqlc mysqld#mysqld 209715200 171582111 38133089 82% /mysqld mysqle#mysqle 209715200 100388182 109311880 48% /mysqle mysqlf#mysqlf 209715200 182784033 26910192 88% /mysqlf Mysql4 may not be fully Ansi SQL compliant, but it is fast, which if you have 1000 nodes of cluster blatting results back into it, may be a more critical requirement. Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Fri Jan 16 13:51:40 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Fri, 16 Jan 2004 10:51:40 -0800 (PST) Subject: [Beowulf] need MPI Programs Message-ID: <20040116185140.61764.qmail@web12708.mail.yahoo.com> hello all, trying to learn LAM/MPI for my final year project. I've gone through the tutorials found on the net and the MPI:Complete Reference book(which i obtained online through the beowulf mailing list.thanks) Anyway, I would be grateful if anyone would share their MPI programs with me, so that I can run/learn/test them on my cluster. Any help would be appreciated. Thanks, Toufeeq ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From atp at piskorski.com Fri Jan 16 14:38:20 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Fri, 16 Jan 2004 14:38:20 -0500 Subject: [Beowulf] MySQL clustering In-Reply-To: <200401161701.i0GH1Sh06443@NewBlue.scyld.com> References: <200401161701.i0GH1Sh06443@NewBlue.scyld.com> Message-ID: <20040116193820.GA63814@piskorski.com> > Date: Fri, 16 Jan 2004 16:43:33 +0000 (GMT) > From: Guy Coates > >quality DB. Note that this is NOT such a rant -- I use mysql myself for > >a small DB-driven php website, but note the key word: "small". > Mysql4 may not be fully Ansi SQL compliant, but it is fast, which if you > have 1000 nodes of cluster blatting results back into it, may be a more > critical requirement. >From what I've read, MySQL is only fast under conditions of no or low concurrency. Get up above 5 or so concurrent users, and performance starts going into the toilet very fast due to table lock contention. Note that 5 concurrent users is a trivial number for any large public website. SourceForge switched from MySQL to PostgreSQL a while back (two years ago?) for exactly that reason, and had some nice little articles about it at the time, showing the scaling graphs of both databases running the actual SourceForge software, etc. Better databases (Oracle, PostgreSQL, Firebird, DB2, Sybase, MS SQL Server, etc. etc.), particularly those using MVCC, will scale well to much higher numbers of concurrent users. If you don't need ACIDity, much concurrency, or powerful "real RDBMS" style features, then MySQL may be a good choice. And there are plenty of applications that either fit that, or are maybe close enough. Of course I might argue that even for most of those applications, there are likely still better choices than MySQL, but as I've never actually built such a maybe-MySQL-would-be-ok app, I've never had any reason to make a real investigation into the different options there. [shrug] YMMV. -- Andrew Piskorski http://www.piskorski.com/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From vallard at CSUA.Berkeley.EDU Fri Jan 16 15:16:45 2004 From: vallard at CSUA.Berkeley.EDU (Vallard Benincosa) Date: Fri, 16 Jan 2004 12:16:45 -0800 (PST) Subject: [Beowulf] HPL Message-ID: <20040116121329.I88241-100000@soda.csua.berkeley.edu> Just had this same problem. HPL uses a symbolic link for your $HOME directory while it's building. So since you are root, your $HOME is /root/. If you extract the tarball in /root/ instead of /home/ then it will work. Alternatively, you can take out references of $HOME in the Make.arch file. Vallard _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brbarret at lam-mpi.org Fri Jan 16 18:49:57 2004 From: brbarret at lam-mpi.org (Brian Barrett) Date: Fri, 16 Jan 2004 15:49:57 -0800 Subject: [Beowulf] my beowulf problem In-Reply-To: <20040116131108.94673.qmail@web21506.mail.yahoo.com> References: <20040116131108.94673.qmail@web21506.mail.yahoo.com> Message-ID: On Jan 16, 2004, at 5:11 AM, prakash borade wrote: > hello friends > i am very new at beowulf cluster > > i get a following error > > i dont understand why is the lam environment req for > mpirun > > it would be requ for lam/mpim implementataion It looks like the mpirun that appears in your path first is the mpirun from LAM/MPI. Both MPICH and LAM/MPI provide an "mpirun" (as well as mpicc, mpiCC, and mpif77). For LAM/MPI to operate, you must "lamboot" first, starting a small daemon on each node. You might want to take a look at our user manual: http://www.lam-mpi.org/using/docs/ As for why you are using LAM/MPI as opposed to MPICH (which is what I'm assuming you meant to ask), it's probably because LAM/MPI is installed by default on most Red Hat and Mandrake Linux installations. Hope this helps, Brian -- Brian Barrett LAM/MPI developer and all around nice guy Have a LAM/MPI day: http://www.lam-mpi.org/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Fri Jan 16 01:12:02 2004 From: gerry.creager at tamu.edu (Gerry Creager (N5JXS)) Date: Fri, 16 Jan 2004 00:12:02 -0600 Subject: [Beowulf] Looking for WRF/MM5 modellers Message-ID: <400780B2.5080908@tamu.edu> I've some questions for folks who've recently built clusters for WRF and MM5 model work. Off-list responses would probably be better. Thanks! gerry -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Office: 979.458.4020 FAX: 979.847.8578 Cell: 979.229.5301 Pager: 979.228.0173 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mperez at delta.ft.uam.es Fri Jan 16 07:11:21 2004 From: mperez at delta.ft.uam.es (Manuel J) Date: Fri, 16 Jan 2004 13:11:21 +0100 Subject: [Beowulf] Mac address collector v 1.1 Message-ID: <200401161311.21575.mperez@delta.ft.uam.es> Dear all, I am happy to announce the first version of the new mac-1.1. Mac stands for "MAC address collector", and it is a tool to automathically collect Mac addresses of a set of computers. For more information about mac and its usage, please check its man page. Mac is part of the Quattor project ( www.quattor.org ). Quattor is a system administration toolkit providing a powerful, portable and modular toolsuite for the automated installation, configuration and management of clusters and farms running UNIX derivates like Linux and Solaris. You can download mac-1.1 from http://datagrid.in2p3.fr/cvsweb/edg-quattor/util/mac/mac.cin and rename it as mac. Any comments, suggestions and critics are wellcome!! Best Regards. Manuel J. Perez Rubio Universidad Autonoma Madrid. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ken_i_m at fatair.net Fri Jan 16 23:07:59 2004 From: ken_i_m at fatair.net (ken_i_m at fatair.net) Date: Fri, 16 Jan 2004 21:07:59 -0700 Subject: [Beowulf] Dual Xeon gigabit issues In-Reply-To: <4007E461.6050705@nsc1.net> References: <4007E461.6050705@nsc1.net> Message-ID: <20040117040759.GH3362@hyper> On Fri, Jan 16, 2004 at 08:17:21AM -0500, Wade Hampton (wade.hampton at nsc1.net) wrote: > >>(Intel 82546EB -- dual port gigabit). They're connected to a Cisco > >>Catalyst 4507R. Spent an hour with a brand new HP DL380 last week that would not talk to the network dispite dmesg show both ports were detected. Once I plugged working cables into both they started to work (instead of just the one I usually use on the workbench). Remove one and networking dies... Not nice. -- Ken Dyke "Bits at the speed of light" _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From whately at lcp.coppe.ufrj.br Fri Jan 16 14:30:30 2004 From: whately at lcp.coppe.ufrj.br (Lauro Whately) Date: Fri, 16 Jan 2004 17:30:30 -0200 Subject: [Beowulf] Re: MySQL clustering In-Reply-To: <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> References: <001501c3db6d$e37db350$2101a8c0@blue> <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> Message-ID: <40083BD6.5070904@lcp.coppe.ufrj.br> Domingo, take a look in the Linux Virtual Server project (www.linuxvirtualserver.org) or try ultramonkey (www.ultramonkey.org) Lauro. luis.licon at cimav.edu.mx wrote: >You have two options, the first one is DNS Round Robin, and the second one >is Load Balance, the second one has a better performance indeed, you could >try this option with PEN, and you can get it on: http://siag.nu/pen > >Cheerz, > >Luis > > >>Hi all!! >> >> Is mysql capable to work with beowulf ? >> I?m intend to make a beowulf cluster to make my website more fast! >> Actualy a have an apache + mysql server, but they not suport >>requests... I tried, but no suces, to make mysql replication.. the >>performance was good but not good enoght... >> >> Is possible to make mysql work over beowulf? >> >> >>Thanks a lot!!! >> >>Domingo Antonio >> >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >>http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >> > > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jaimeco at dyer.rc.arizona.edu Fri Jan 16 18:02:39 2004 From: jaimeco at dyer.rc.arizona.edu (jaimeco) Date: Fri, 16 Jan 2004 16:02:39 -0700 (MST) Subject: [Beowulf] enforce resources_max.nodes Message-ID: <200401162302.i0GN2dA9002931@dyer.rc.arizona.edu> We have a cluster of alpha machines and we are running openpbs. We have created a queue with a max number of 32 cpus. We want these jobs to run on aura2 and aura4 as can be seen below. openpbs is not enforcing this unless the name of the node is used (-l nodes=aura2:ppn=32). We would like users to be more generic and use -l nodes=1:ppn=32 so that openpbs knows that this queue should be assigned to nodes 2 and 4 and will enforce it. It is not happening and jobs will usually go to aura0 --if there are enough cpus there. Does anyone know of a way to enforce this or have any experience with this? I appreciate your responses. Thanks create queue aura_32p set queue aura_32p queue_type = Execution set queue aura_32p Priority = 100 set queue aura_32p max_running = 4 set queue aura_32p acl_user_enable = True set queue aura_32p resources_max.ncpus = 32 set queue aura_32p resources_max.nodes = aura2+aura4:ppn=32 set queue aura_32p resources_default.ncpus = 16 set queue aura_32p resources_available.ncpus = 128 set queue aura_32p max_user_run = 1 set queue aura_32p enabled = True set queue aura_32p started = True --- Jaime E. Combariza Manager, Research Computing CCIT, University of Arizona (520) 626-4050 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tmerritt at email.arizona.edu Fri Jan 16 14:55:16 2004 From: tmerritt at email.arizona.edu (Todd Merritt) Date: Fri, 16 Jan 2004 12:55:16 -0700 Subject: [Beowulf] MySQL clustering In-Reply-To: <20040116193820.GA63814@piskorski.com> References: <200401161701.i0GH1Sh06443@NewBlue.scyld.com> <20040116193820.GA63814@piskorski.com> Message-ID: <1074282915.23849.50.camel@hive.CCIT.Arizona.EDU> On Fri, 2004-01-16 at 12:38, Andrew Piskorski wrote: > From what I've read, MySQL is only fast under conditions of no or low > concurrency. Get up above 5 or so concurrent users, and performance > starts going into the toilet very fast due to table lock contention. > Note that 5 concurrent users is a trivial number for any large public > website. > That may be true of multiple concurrent writes, but for mostly read access, mysql performance is stellar. We use it as the preference backend for our webmail system, as well as maintaining user mappings for postfix for ~50,000 email accounts. At peak times, there are ~1200 simultaneous webmail users and mysql has never been a bottleneck here. Todd _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Sat Jan 17 01:28:27 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Fri, 16 Jan 2004 22:28:27 -0800 (PST) Subject: [Beowulf] NEDD OPENPBS HELP In-Reply-To: <200401162302.i0GN2dA9002931@dyer.rc.arizona.edu> Message-ID: <20040117062827.10830.qmail@web21502.mail.yahoo.com> hello i am trying openpbs for the first time . iam not able to get the pbs admin_guid pdf file ( it cannot be downloaded from their page) i don't know how to make my openpbs know all my nodes it does'nt lists all my nodes in the cluster __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Sat Jan 17 07:12:53 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Sat, 17 Jan 2004 04:12:53 -0800 (PST) Subject: [Beowulf] need MPI Programs In-Reply-To: <20040117083752.83681.qmail@web12706.mail.yahoo.com> Message-ID: <20040117121253.92292.qmail@web21509.mail.yahoo.com> hello thanks for your help i have found out the bug and unistalled lam an now trying the mpirun from mpich but still un able to include my all machines though i have made entries in machine.LINUX file i get something like this about the programs yes we have lots of codes to be written usin mpi but before that i must have my clkuster working we'll work together to help each other thanks in advance [mpich]#mpirun -np 2 cpi p0_21223: (62.933408) Procgroup: p0_21223: (62.933510) entry 0: tung.unipune.ernet.in 0 0 /home/mpich/cpi mpich p0_21223: (62.933539) entry 1: dyp.unipune.ernet.in 1 1 /home/mpich/cpi mpich p0_21223: p4_error: Could not gethostbyname for host dyp.unipune.ernet.in; may be invalid name __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Sat Jan 17 08:33:07 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 17 Jan 2004 21:33:07 +0800 (CST) Subject: [Beowulf] NEDD OPENPBS HELP In-Reply-To: <20040117062827.10830.qmail@web21502.mail.yahoo.com> Message-ID: <20040117133307.2013.qmail@web16807.mail.tpe.yahoo.com> OpenPBS is dead, a lot of the bugs are not fixed. If you go to the "OpenPBS vs. PBS Pro Comparison" page on www.openpbs.org, it says: "OpenPBS works up to 32 nodes with 100s of jobs, but doesn't scale well above that due to fault tolerance problems. PBS Pro is in production use at sites with thousands of CPUS, 30,000+ jobs." You should use Scalable PBS (aka storm), it is actually the opensource version OpenPBS but with lots of fixes and new features. You can download it here: http://supercluster.org/downloads/spbs/ In storm-1.0.1p3.tar.gz, you will get admin_guide.ps in the doc/ directory. Use "ps2pdf" if you don't have the ps reader. Also, Gridengine is another good batch system and with even more features. They have the pre-compiled binaries available so you don't need to go through the source compile process, which sometimes is a pain to get all the dependencies. You can also give it a try: http://gridengine.sunsource.net/ Andrew. --- prakash borade ????> > hello > i am trying openpbs for the first > time . iam not able to get the pbs admin_guid pdf > file ( it cannot be downloaded from their page) > > i don't know how to make my openpbs know all my > nodes > it does'nt lists all my nodes in the cluster > > > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" > Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nathan at iwantka.com Sat Jan 17 14:04:32 2004 From: nathan at iwantka.com (Nathan Littlepage) Date: Sat, 17 Jan 2004 13:04:32 -0600 Subject: [Beowulf] Re: MySQL clustering In-Reply-To: <36134.148.223.46.10.1074271368.squirrel@www3.cimav.edu.mx> Message-ID: <00f201c3dd2c$cd2bdfd0$6c45a8c0@ntbrt.bigrivertelephone.com> To add an additional option that would work in conjunction with the other two. Utilize the replication ability of mySQL. http://www.mysql.com/doc/en/Replication.html > -----Original Message----- > From: beowulf-admin at scyld.com > [mailto:beowulf-admin at scyld.com] On Behalf Of luis.licon at cimav.edu.mx > Sent: Friday, January 16, 2004 10:43 AM > To: Domingo Antonio > Cc: beowulf at beowulf.org > Subject: Re: [Beowulf] Re: MySQL clustering > > > You have two options, the first one is DNS Round Robin, and > the second one > is Load Balance, the second one has a better performance > indeed, you could > try this option with PEN, and you can get it on: http://siag.nu/pen > > Cheerz, > > Luis > > Hi all!! > > > > Is mysql capable to work with beowulf ? > > I?m intend to make a beowulf cluster to make my > website more fast! > > Actualy a have an apache + mysql server, but they not suport > > requests... I tried, but no suces, to make mysql replication.. the > > performance was good but not good enoght... > > > > Is possible to make mysql work over beowulf? > > > > > > Thanks a lot!!! > > > > Domingo Antonio > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From meetsunil80x86 at yahoo.co.in Sat Jan 17 22:36:48 2004 From: meetsunil80x86 at yahoo.co.in (=?iso-8859-1?q?sunil=20kumar?=) Date: Sun, 18 Jan 2004 03:36:48 +0000 (GMT) Subject: [Beowulf] MPI Message-ID: <20040118033648.79360.qmail@web8307.mail.in.yahoo.com> Hello everybody, I am a newbie in MPI programming. Please help me. truely, sunil ________________________________________________________________________ Yahoo! India Mobile: Download the latest polyphonic ringtones. Go to http://in.mobile.yahoo.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 00:59:22 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 06:59:22 +0100 Subject: [Beowulf] Selection of Beowulf cluster type Message-ID: <400B723A.4070701@telia.com> Hi, I am going to set up my first beowulf cluster and I have no experience of this. The cluster will be built of two dual CPU workstations. Box 1 MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe 4 x 512 MB DDR SDRAM PC2100 Box 2. Intel model SE7505VB2 ATX PCI, FSB 533MHz 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB 2 x 1 GB DDR SDRAM PC2100 Reg ECC The cluster is intended for numerical simulations (FEA) with ABAQUS Please advise the type of cluster you think I shall build and what package to use. For the time being I am considering following solutions: - Warewulf 1.16 (http://www.warewulf-cluster.org) - OSCAR 3.0 ( http://oscar.openclustergroup.org/tiki-index.php) Yours faithfully Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Mon Jan 19 01:42:27 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Sun, 18 Jan 2004 22:42:27 -0800 (PST) Subject: [Beowulf] can't include all machines in my beowulf Message-ID: <20040119064227.51163.qmail@web21503.mail.yahoo.com> i was able to run cpi program on the single machine with only master included in machines.LINUX but after i have made entries of my all nodes in machines.LINUX file i get the following error i have properly configure rsh also i have mounted my mpich installed directory on all machines using NFS and all uses the same path [mpich]#mpirun -np 2 cpi > p0_21223: (62.933408) Procgroup: > p0_21223: (62.933510) entry 0: > tung.unipune.ernet.in 0 0 /home/mpich/cpi mpich > p0_21223: (62.933539) entry 1: > dyp.unipune.ernet.in 1 1 /home/mpich/cpi mpich > p0_21223: p4_error: Could not gethostbyname for > host > dyp.unipune.ernet.in; may be invalid name > __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Jan 19 08:54:55 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 19 Jan 2004 08:54:55 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400B723A.4070701@telia.com> Message-ID: On Mon, 19 Jan 2004, Per Lindstrom wrote: > Hi, > > I am going to set up my first beowulf cluster and I have no experience > of this. > > The cluster will be built of two dual CPU workstations. > > Box 1 > MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) > 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe > 4 x 512 MB DDR SDRAM PC2100 > > Box 2. > Intel model SE7505VB2 ATX PCI, FSB 533MHz > 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB > 2 x 1 GB DDR SDRAM PC2100 Reg ECC > > The cluster is intended for numerical simulations (FEA) with ABAQUS > > Please advise the type of cluster you think I shall build and what > package to use. > > For the time being I am considering following solutions: > > - Warewulf 1.16 (http://www.warewulf-cluster.org) > > - OSCAR 3.0 ( http://oscar.openclustergroup.org/tiki-index.php) You sound like you are on a good track. I've never tried warewulf but expect that it is fine. OSCAR is very popular and seems to be very easy for a novice to get going with. rgb > > Yours faithfully > Per Lindstr?m > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Mon Jan 19 12:28:55 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: 19 Jan 2004 17:28:55 +0000 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400B723A.4070701@telia.com> References: <400B723A.4070701@telia.com> Message-ID: <1074533335.2278.10.camel@fermi> On Mon, 2004-01-19 at 05:59, Per Lindstrom wrote: > Box 1 > MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) > 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe > 4 x 512 MB DDR SDRAM PC2100 > > Box 2. > Intel model SE7505VB2 ATX PCI, FSB 533MHz > 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB > 2 x 1 GB DDR SDRAM PC2100 Reg ECC > > The cluster is intended for numerical simulations (FEA) with ABAQUS First off: I do not really know ABAQUS, but i assume, that like most FE solvers, it is based on a MPI parallelization. By nature it has a strictly parallel execution flow. This would probably mean that the speed of which the execution takes place (if the problem is indeed CPU bound), is dictated by the slowest CPU in the system (beeing the 1,13 GHz P3) - You might get better performance with running on just the dual Xeon. It can also be stated as follows: Does the program execute faster with 4 x P3 1,13 GHz than on 2 x 2,8GHz Xeons communicating through memory? Though the 4 CPU setup might give better cache utilization... I'm not saying that you shouldn't build a cluster, but i i'm very interested in your findings in this case. Please correct me if i'm rambling here :-) best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5260 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 12:07:07 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 18:07:07 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <1074533335.2278.10.camel@fermi> References: <400B723A.4070701@telia.com> <1074533335.2278.10.camel@fermi> Message-ID: <400C0EBB.8030905@telia.com> Lars Henriksen wrote: >On Mon, 2004-01-19 at 05:59, Per Lindstrom wrote: > > >>Box 1 >>MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) >>2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe >>4 x 512 MB DDR SDRAM PC2100 >> >>Box 2. >>Intel model SE7505VB2 ATX PCI, FSB 533MHz >>2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB >>2 x 1 GB DDR SDRAM PC2100 Reg ECC >> >>The cluster is intended for numerical simulations (FEA) with ABAQUS >> >> > >First off: I do not really know ABAQUS, but i assume, that like most FE >solvers, it is based on a MPI parallelization. By nature it has a >strictly parallel execution flow. This would probably mean that the >speed of which the execution takes place (if the problem is indeed CPU >bound), is dictated by the slowest CPU in the system (beeing the 1,13 >GHz P3) - You might get better performance with running on just the dual >Xeon. It can also be stated as follows: Does the program execute faster >with 4 x P3 1,13 GHz than on 2 x 2,8GHz Xeons communicating through >memory? > >Though the 4 CPU setup might give better cache utilization... > >I'm not saying that you shouldn't build a cluster, but i i'm very >interested in your findings in this case. > >Please correct me if i'm rambling here :-) > >best regards >Lars > > Tanks for your thoughts Lars, :-) You are most probably right about that the Xeon machine alone will be the fastest solution, I will do some test and let you know the result and the problem used. The main plan with the cluster exercise is, in the first order, to gain know how and experience of building, managing and solving FEA (ABAQUS) problems on a cluster and/or MPI network. If everything vent well and it feels comfortable, the next step will be to replace the Tualatin machine with an identical Xeon machine i.e. giving a 4 x 2,8 GHz setup. (i.e. applied cash flow management ;-) ) Best regards Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 12:35:41 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 18:35:41 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <187D3A7CAB42A54DB61F1D05F012572201D4C05A@orsmsx402.jf.intel.com> References: <187D3A7CAB42A54DB61F1D05F012572201D4C05A@orsmsx402.jf.intel.com> Message-ID: <400C156D.90303@telia.com> Lombard, David N wrote: >From: Per Lindstrom; Sunday, January 18, 2004 9:59 PM > > >>Hi, >> >>I am going to set up my first beowulf cluster and I have no experience >>of this. >> >>The cluster will be built of two dual CPU workstations. >> >>Box 1 >>MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) >>2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe >>4 x 512 MB DDR SDRAM PC2100 >> >>Box 2. >>Intel model SE7505VB2 ATX PCI, FSB 533MHz >>2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB >>2 x 1 GB DDR SDRAM PC2100 Reg ECC >> >>The cluster is intended for numerical simulations (FEA) with ABAQUS >> >> > >The most important factor is how ABAQUS runs your jobs on the two >systems individually and on the cluster as a whole. > >If ABAQUS doesn't do any sort of load balancing, e.g., it simply cuts >the problem in half (or quarters), you will likely see the effect that >the problem solution time is controlled (limited) by the speed of the >slower system. > >You also haven't discussed the available disk (storage). I do not know >what the disk requirements of ABAQUS are. If it has significant disk >requirements, e.g., disk bandwidth, the disks will also affect the >solution time. I don't expect disk latency to be an issue with any FEA >app, and presume you have sufficiently large disks. > >Finally, the network may have some impact when running a job across the >nodes. > >Bottom line. Once you get ABAQUS running, you will likely benefit from >analyzing how well it runs in serial, SMP, and DMP modes on your >cluster. For example, you may find the "best" use is to run the cluster >in throughput (running one job per node, possibly under a queuing >system), rather than parallel performance mode (running one job across >both nodes). I put best in quotes because that quality depends on your >needs. > > > Thanks for your advices David, All the best Per Lindstro"m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 12:43:26 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 18:43:26 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400C173E.1040803@telia.com> Robert G. Brown wrote: >On Mon, 19 Jan 2004, Per Lindstrom wrote: > > > >>Hi, >> >>I am going to set up my first beowulf cluster and I have no experience >>of this. >> >>The cluster will be built of two dual CPU workstations. >> >>Box 1 >>MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) >>2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe >>4 x 512 MB DDR SDRAM PC2100 >> >>Box 2. >>Intel model SE7505VB2 ATX PCI, FSB 533MHz >>2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB >>2 x 1 GB DDR SDRAM PC2100 Reg ECC >> >>The cluster is intended for numerical simulations (FEA) with ABAQUS >> >>Please advise the type of cluster you think I shall build and what >>package to use. >> >>For the time being I am considering following solutions: >> >>- Warewulf 1.16 (http://www.warewulf-cluster.org) >> >>- OSCAR 3.0 ( http://oscar.openclustergroup.org/tiki-index.php) >> >> > >You sound like you are on a good track. I've never tried warewulf but >expect that it is fine. OSCAR is very popular and seems to be very easy >for a novice to get going with. > > rgb > > > >>Yours faithfully >>Per Lindstr?m >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> > > > I will most probably go for the Warewulf as it should be possible to use the Linux distro I'm used to (Slackware) and I think that is quite important as it allways cost time to change "environment". Or is that a blind trace? Best regards Per Lindstr?m _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From david.n.lombard at intel.com Mon Jan 19 12:09:18 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Mon, 19 Jan 2004 09:09:18 -0800 Subject: [Beowulf] Selection of Beowulf cluster type Message-ID: <187D3A7CAB42A54DB61F1D05F012572201D4C05A@orsmsx402.jf.intel.com> From: Per Lindstrom; Sunday, January 18, 2004 9:59 PM > > Hi, > > I am going to set up my first beowulf cluster and I have no experience > of this. > > The cluster will be built of two dual CPU workstations. > > Box 1 > MSI Pro266TD Master-LR2 (MS-9105) (I can not recommended MSI) > 2 x CPU Intel Tualatin 1,13 GHz with 512 L2 cashe > 4 x 512 MB DDR SDRAM PC2100 > > Box 2. > Intel model SE7505VB2 ATX PCI, FSB 533MHz > 2 x CPU Intel 2,8 GHz Xeon with 512kB L2 cache 533FSB > 2 x 1 GB DDR SDRAM PC2100 Reg ECC > > The cluster is intended for numerical simulations (FEA) with ABAQUS The most important factor is how ABAQUS runs your jobs on the two systems individually and on the cluster as a whole. If ABAQUS doesn't do any sort of load balancing, e.g., it simply cuts the problem in half (or quarters), you will likely see the effect that the problem solution time is controlled (limited) by the speed of the slower system. You also haven't discussed the available disk (storage). I do not know what the disk requirements of ABAQUS are. If it has significant disk requirements, e.g., disk bandwidth, the disks will also affect the solution time. I don't expect disk latency to be an issue with any FEA app, and presume you have sufficiently large disks. Finally, the network may have some impact when running a job across the nodes. Bottom line. Once you get ABAQUS running, you will likely benefit from analyzing how well it runs in serial, SMP, and DMP modes on your cluster. For example, you may find the "best" use is to run the cluster in throughput (running one job per node, possibly under a queuing system), rather than parallel performance mode (running one job across both nodes). I put best in quotes because that quality depends on your needs. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Mon Jan 19 15:46:27 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Mon, 19 Jan 2004 21:46:27 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400C4223.7050103@telia.com> Robert G. Brown wrote: >On Mon, 19 Jan 2004, Per Lindstrom wrote: > > > >>I will most probably go for the Warewulf as it should be possible to use >>the Linux distro I'm used to (Slackware) and I think that is quite >>important as it allways cost time to change "environment". >> >>Or is that a blind trace? >> >> > >Speaking personally as a person who started with slackware (a fairly >long time ago) and moved on, I think that you should STRONGLY consider >moving to an RPM-based distribution. Red Hat, or in the future Fedora >(for example) installed with kickstart via PXE and maintained with yum >approach the theoretical limit in operating system scalability. As in, >to install a system one turns it on. To update a system one does >nothing (a nightly cron script does it for you). To install or browse >new software from an RPM repository one uses a single command with a >very efficient interface. > >Debian also scales much better than slackware. Slackware was OK for >single systems or small networks, but it won't do for hundreds of LAN >boxes and beowulf nodes per administrator. RPM-based distros will. > > rgb > > > >>Best regards >>Per Lindstr?m >> >> >> > > > It gives that I should jump from Slack to Fedora or Debian, which distro is scaling best? Best regards Per Lindstrom _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Jan 19 15:27:26 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 19 Jan 2004 15:27:26 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400C173E.1040803@telia.com> Message-ID: On Mon, 19 Jan 2004, Per Lindstrom wrote: > I will most probably go for the Warewulf as it should be possible to use > the Linux distro I'm used to (Slackware) and I think that is quite > important as it allways cost time to change "environment". > > Or is that a blind trace? Speaking personally as a person who started with slackware (a fairly long time ago) and moved on, I think that you should STRONGLY consider moving to an RPM-based distribution. Red Hat, or in the future Fedora (for example) installed with kickstart via PXE and maintained with yum approach the theoretical limit in operating system scalability. As in, to install a system one turns it on. To update a system one does nothing (a nightly cron script does it for you). To install or browse new software from an RPM repository one uses a single command with a very efficient interface. Debian also scales much better than slackware. Slackware was OK for single systems or small networks, but it won't do for hundreds of LAN boxes and beowulf nodes per administrator. RPM-based distros will. rgb > > Best regards > Per Lindstr?m > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Mon Jan 19 17:44:57 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Mon, 19 Jan 2004 17:44:57 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400C4223.7050103@telia.com> Message-ID: On Mon, 19 Jan 2004, Per Lindstrom wrote: > It gives that I should jump from Slack to Fedora or Debian, which distro > is scaling best? Ah, the kind of question that can start a religious war. I will say that I personally prefer RH (and soon expect to prefer Fedora) because kickstart + PXE + yum is pretty incredible. However, Debianites feel that apt and friends are even better (or at least just as good). On this list there is at least one Mandrake-person who will point out that Mandrake is a good, RPM-based distribution (so yum will work) with its own RPM manipulation tools and its own cluster distro. Finally -- well, probably not finally;-) -- there is SuSE, used by a lot of Europe and reasonably popular over here where yum also should work. There. That's very ecumenical. My advise is to look over the distributions with regard to cost, how current they stay, and what other tools you want to use expect. OSCAR, for example, installs on top of one of the RH revisions, so if you go that way you'll probably want to go RH/Fedora. rgb > > Best regards > Per Lindstrom > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at comcast.net Mon Jan 19 19:00:00 2004 From: laytonjb at comcast.net (Jeffrey B. Layton) Date: Mon, 19 Jan 2004 19:00:00 -0500 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400C6F80.6060600@comcast.net> Robert G. Brown wrote: >On Mon, 19 Jan 2004, Per Lindstrom wrote: > > > >>It gives that I should jump from Slack to Fedora or Debian, which distro >>is scaling best? >> >> > >Ah, the kind of question that can start a religious war. > That's true! Although sometimes I get a little devious and like to start them (fun to watch the opinions come out). If you're goint to use Warewulf, you'll need a rpm-based distrbution. Something like RH, Mandrake, Suse, cAos, Whitebox, Fedora, etc. Warewulf gives you all of the details about PXE (don't need kickstart since there's nothing on the disk). I think yum has been added to cAos, but in general yum is pretty cool. As for performance, I didn't know one was better than the other (never heard of the problems with slackware before). Oh, I think you can add rpm capability to slack anyway (anyone comment on this?). Good Luck! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 00:50:22 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Mon, 19 Jan 2004 21:50:22 -0800 (PST) Subject: [Beowulf] can't include all machines in my beowulf In-Reply-To: Message-ID: <20040120055022.18675.qmail@web21506.mail.yahoo.com> hey tahnks for your advice i was able to find the bug thank you very much do you know what is MPE and what role does it play in mpich where cpi don't require it but cpilog requires it __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 01:54:49 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Mon, 19 Jan 2004 22:54:49 -0800 (PST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) Message-ID: <20040120065449.33171.qmail@web21507.mail.yahoo.com> hey can anybody tell me what do i choose for programing either GNU c or JAVA for implementing some networkprograms using some linux systemcalls __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Tue Jan 20 04:56:47 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Tue, 20 Jan 2004 10:56:47 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400C6F80.6060600@comcast.net> References: <400C6F80.6060600@comcast.net> Message-ID: <400CFB5F.4010104@telia.com> Jeffrey B. Layton wrote: > Robert G. Brown wrote: > >> On Mon, 19 Jan 2004, Per Lindstrom wrote: >> >> >> >>> It gives that I should jump from Slack to Fedora or Debian, which >>> distro is scaling best? >>> >> >> Ah, the kind of question that can start a religious war. >> > > That's true! Although sometimes I get a little devious and like to > start them (fun to watch the opinions come out). > > If you're goint to use Warewulf, you'll need a rpm-based distrbution. > Something like RH, Mandrake, Suse, cAos, Whitebox, Fedora, etc. > Warewulf gives you all of the details about PXE (don't need > kickstart since there's nothing on the disk). I think yum has been > added to cAos, but in general yum is pretty cool. > > As for performance, I didn't know one was better than the other > (never heard of the problems with slackware before). Oh, I think > you can add rpm capability to slack anyway (anyone comment on > this?). > ---------------------------- Throwing in the statement that Slackware has a bad scalability to a Slack fundamentalist forum, should indeed start a religious war. ; ) Here I am not running for Slack for its own sake so I will use Slacware up to two (2) nodes and when shifting to Debian when I'm connecting the remaining nodes. I think it will stop at about 10 CPUs. Until I have learned to manage Debian I will use the "slack-rpm-transfer-tool" and it will be very interesting to benchmark the two distros with each other. Thanks a lot for all help and support in his case. Best regards Per Lindstrom _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 06:59:19 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Tue, 20 Jan 2004 03:59:19 -0800 (PST) Subject: [Beowulf] can i increase priority of process on beowulf with mpich Message-ID: <20040120115919.1940.qmail@web21502.mail.yahoo.com> hello proksy here can we change the priority of the process on the cluser using mpich because i have seen that when i run the process as a root with priority -10 it is maintained only on master how can i maintain same priority on all machines __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 20 06:58:34 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Tue, 20 Jan 2004 03:58:34 -0800 (PST) Subject: [Beowulf] can i increase priority of process on beowulf with mpich Message-ID: <20040120115834.71935.qmail@web21505.mail.yahoo.com> hello proksy here can we change the priority of the process on the cluser using mpich because i have seen that when i run the process as a root with priority -10 it is maintained only on master how can i maintain same priority on all machines __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lars at meshtechnologies.com Tue Jan 20 08:05:30 2004 From: lars at meshtechnologies.com (Lars Henriksen) Date: Tue, 20 Jan 2004 13:05:30 +0000 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400CFB5F.4010104@telia.com> References: <400C6F80.6060600@comcast.net> <400CFB5F.4010104@telia.com> Message-ID: <1074603930.2417.8.camel@tp1.mesh-hq> On Tue, 2004-01-20 at 09:56, Per Lindstrom wrote: > Here I am not running for Slack for its own sake so I will use Slacware > up to two (2) nodes and when shifting to Debian when I'm connecting the > remaining nodes. I think it will stop at about 10 CPUs. > > Until I have learned to manage Debian I will use the > "slack-rpm-transfer-tool" and it will be very interesting to benchmark > the two distros with each other. If you want to use Debian as the base distribution, I can recommend SytemImager as an installation tool. http://www.systemimager.org/ Sytemimager allows you to replicate an installed client via PXE/rsync to the remaining nodes. It also allows for updates to be distributed via rsync. A feature that i really like, is that it allows for the use of several different images. Also it scales pretty well (I had the pleasure of using it in a couple of 16-32 node configurations, as well as on our own cluster) IMO it is the easiest way to deal with smaller clusters (except maybe CLIC) best regards Lars -- Lars Henriksen | MESH-Technologies A/S Systems Manager & Consultant | Forskerparken 10 www.meshtechnologies.com | DK-5230 Odense M, Denmark lars at meshtechnologies.com | mobile: +45 2291 2904 direct: +45 6315 7310 | fax: +45 6315 7314 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 20 08:05:45 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 20 Jan 2004 08:05:45 -0500 (EST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120065449.33171.qmail@web21507.mail.yahoo.com> Message-ID: On Mon, 19 Jan 2004, prakash borade wrote: > hey > > can anybody tell me > what do i choose for programing > either GNU c or JAVA for implementing some > networkprograms using some linux systemcalls Gads, that's two questions in two days with the distinct whiff of crowds of screaming peasants waving swords and torches... Let it be Jihad, then. The answer technically is "it depends". This is to make the vast horde of java programmers feel loved, and to acknowledge that for writing web applications their primary scripting language (like perl, python and many another before it) has a purpose in God's world. There. Let us bask for a moment in the serenity of our knowledge that we have the complete freedom to choose, and that there are no wrong answers. Now we can give the correct answer, which is "C". In order to give it, I have to infer an implicit "better than" and a certain degree of generality in your question is in "which language is BETTER suited to write an efficient networking program using linux systemscalls, in GENERAL". With this qualifier, C wins hands down. A variety of reasons: a) The operating system is written in C (plus a bit of assembler) b) Nearly all network daemons and utilities and program clients are written in C or perhaps an extension of C such as C++ c) Nearly all decent books on network programming (e.g. Stevens) provide excellent C templates for doing lots of network-based things d) You can do "anything" with C plus (in a few, very rare cases, a bit of inlined assembler) e) C is a compiler and produces (in the hands of a good programmer) code that runs very efficiently. Java is an interpreter and does not. f) C is an open ANSI standard implemented in linux with fully open source tools with no restrictions or covenants on their usage and with no "vendors" in any sort of position of control over the language itself or its core libraries. Java was invented by Sun, and Sun (AFAIK) still controls the distribution of its core libraries, and as an interpreted language, those libraries are sine qua non for portability. Sun is relatively linux friendly, but I personally tend not to trust giant multibillion dollar companies that own core components of something I need to rely on (or incorporate those components into my code), as they have an annoying tendency to turn around and want money for it. g) Or find themselves imbroiled in a corporate war with someboy ELSE with deep pockets advancing a different "standard"; for example I'd guess that Microsoft would like nothing better than to co-opt Java for themselves and turn around and want money for it. C is co-opt proof, and GCC is a highly reliable platform upon which to base code in the certain knowledge that the core compiler and libraries will remain royalty free. h) ...and I could go on. Note that I do not address java's OO nature, as OO C variants exist AND as OO programming is largely a matter of programming style, not language per se anyway. To conclude with a more ecumenical note because it DOES depend, I should point out that my acquaintances who code in java seem to be able to write fairly complex structured programs with the language, and I'd guess that most posix systems calls are wrapped for use within a java program as they are in perl and python and etc (I could be wrong, as java has "sandbox" issues that might well block certain systems calls from being run at all in which case this is another argument IN GENERAL for C, although there may well be SPECIFIC cases where it is an argument in favor of Java). Note also that my diatribe above is clearly biased as I don't code in Java, largely become of all of the reasons above (and then some). Speed, power, portability, open standard and source, and available code base and learning support all argue in favor of C about 10 to 1. UNLESS you are trying to write a powerful web-distributed application, which is a PITA in C and what Java was designed for. Books with good to excellent support for network programming in C: W. Richard Stevens, Networking God (deceased): http://www.kohala.com/start/ (pick almost any title, but I like "Unix Network Programming"). Even if you code in Java, you'll likely want this as I'd bet that Java just wraps up the C systems calls in interpreted tokens. Unix for Programmers and Users, Graham Glass (the current edition is with King Ables as well, but the older edition is just fine). Various websites dedicated to network programming, with examples. That's what Google is for -- I have the contents of several squirrelled away already for myself, but the web is amorphous and changes often, so you'll need to do your own search. Any of a LOT of GPL programs that can serve as code templates to get you started. xmlsysd/wulfstat, for example, contain templated code for a network daemon that can run either in forking mode or as an xinetd-managed daemon plus a client that connects to the daemon. Glass and Stevens both have related code snippets (in fact, guess where my code is derived from). But there are also a zillion other daemons in any linux distro, most of them GPL and readily available in source. I think you'd have a hard time finding 1/10 as many ready-made examples (especially ones with highly complex functionality) in Java. LOL, rgb P.S. -- I >>tried<< to be mild, but I suppose I'll need to spend the rest of the day in a bunker wearing my lead underwear and a tinfoil skullcap. Really, I love YAPL's as they pop up from year to year. I do. Marshmellows...ready! Flame on... -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nixon at nsc.liu.se Tue Jan 20 07:45:20 2004 From: nixon at nsc.liu.se (Leif Nixon) Date: Tue, 20 Jan 2004 13:45:20 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <1074603930.2417.8.camel@tp1.mesh-hq> (Lars Henriksen's message of "Tue, 20 Jan 2004 13:05:30 +0000") References: <400C6F80.6060600@comcast.net> <400CFB5F.4010104@telia.com> <1074603930.2417.8.camel@tp1.mesh-hq> Message-ID: Lars Henriksen writes: > On Tue, 2004-01-20 at 09:56, Per Lindstrom wrote: >> Here I am not running for Slack for its own sake so I will use Slacware >> up to two (2) nodes and when shifting to Debian when I'm connecting the >> remaining nodes. I think it will stop at about 10 CPUs. >> >> Until I have learned to manage Debian I will use the >> "slack-rpm-transfer-tool" and it will be very interesting to benchmark >> the two distros with each other. > > If you want to use Debian as the base distribution, I can recommend > SytemImager as an installation tool. > > http://www.systemimager.org/ Systemimager is not Debian specific. We use it together with Red Hat. Very nice indeed. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Tue Jan 20 09:06:55 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Tue, 20 Jan 2004 09:06:55 -0500 (EST) Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <400CFB5F.4010104@telia.com> Message-ID: On Tue, 20 Jan 2004, Per Lindstrom wrote: > Until I have learned to manage Debian I will use the > "slack-rpm-transfer-tool" and it will be very interesting to benchmark > the two distros with each other. > > Thanks a lot for all help and support in his case. Just to be clear, the issue is not "benchmarking two distros" for performance. It is strictly manageability and scalability. Suppose I want to (re)install a system upstairs in my house. I walk up to the system and reboot it (power cycling is OK). When it boots, a PXE prompt pops up. I tell it to boot "linux-install". The system issues a query to determine who it is (gets an IP number and various other data from a DHCP server), retrieves a kernel, and boots into a network-based install. The install is fully scripted -- it will even build/rebuild an MD RAID for me. Every package I have listed for the host is installed, its keyboard, mouse, monitor, time base, and more is set up, and in the post install routine all required account support (/etc/passwd, NFS mounts, printers, whatever) are set up. At the end the system reboots itself into production. On a fast network connection and with a fast repository server, the time required for me to fully (re)install a system can be as little as two or three minutes, at most perhaps thirty minutes (where I need be in attendance only for the first thirty seconds). In fact, I don't NEED to be in attendance. By running a script, I can tell grub on an installed system to boot linux-install by default and reboot; linux-install will end by putting a new grub in place to go back to the normal default kernel. Once installed, provided that I installed from a yummified server and installed the yum package (appropriately configured for my network and repository) the system will wake up sometime in the night, query the server over anonymous http or ftp to see if there are any packages in the repository that are more recent (have a more advanced revision number) than what is installed. If there are, it grabs the RPMs and retrieves them (plus any needed dependencies) and updates the system's installation. If I'm managing five systems or five thousand systems, to keep ALL OF THEM CURRENT within 24 hours to the latest patches and package updates requires only that I drop an RPM onto the repository and do nothing. I don't even need root privileges on all those derived clients -- yum is a client-pull tool, not a root-push tool. So if RH announces a security bug in apache, what do I care? I take the patched RPM (generally released at the same time as the bug announcement -- security bugs are not tolerated for long in linux) and drop it into my repository and by the next morning every webserver in an INSTITUTION scale linux network is once again secure, even if some of the individuals running linux-based webservers are complete idiots who don't have the foggiest clue about the need to keep the institution's network secure by means of regular system maintenance. The institution still needs a genuine clueful expert, but it only needs ONE. yum has lots of other lovely features. yum will list all packages in all repositories a system is configured to use, broken down into "installed" and "available" lists. yum will install anything that is installed, and uninstall anything that you want to remove, AND will augment or prune the dependency tree just the right amount in either case. yum always asks before doing things (unless you tell it not to in e.g. a script). yum was designed by systems adminstrators for systems administrators, so it is feature rich precisely where sysadmins need it to be and conservative in its design throughout. Some sites that maintain repositories for particular GPL tools have started to "yummify" those repositories so that you can literally get updates hot off the primary site from any network-connected system. Note that if you replace the "kickstart" step above with whatever primary installation mode an RPM-based distro favors, the other RPM-based distros are equally yummifiable. I happen to like kickstart because I've used it enough to learn it and because it is sufficiently feature rich that 95% or better of all systems can be templated to be installed without human hands (99% or better if you sensibly choose the hardware and put sane restrictions on the software) but yum actually CREATES space for new installation methodology. For example, one hassle with kickstart is that if you run it over a slow connection, e.g. DSL, it takes forever to install several GB worth of software and if you interrupt it at any point it has to start over. With yum, you can install an ABSOLUTELY MINIMAL system that takes only a few minutes to install even over DSL, and then "finish" the system with yum. Unlike kickstart, yum is perfectly happy to be interrupted and will just start where it left off until it finishes installing a given list of packages. THIS is where slackware failed -- it is package based, but its packages tended to lag, in some cases significantly, the more aggressively maintained commercial distros (as did/do Debian's packages, actually). The thing that originally recommended it was that it could be installed from floppy disks in "sets", which is precisely why I started with it back in the mid-90's. It is still distributed that way, for reasons that are now somewhat difficult to fathom. Its tgz packages don't manage dependencies and dependency issues as well as .deb or .rpm packages, and it is very, very easy for systems on a slackware-based LAN to "drift" from their installed configuration and become heterogeneous and difficult to update and maintain. Package rebuilding is also much more of an issue. It just takes more human TIME to install and maintain EACH SYSTEM on a LAN using slackware than it does using an RPM-based distro or Debian. As far as performance goes, well, performance is generally all but identical across linuces because they all tend to use the same code base EXCEPT that the most aggressively maintained distros will typically have a small edge because their underlying packages are more up to date and better maintained. A distro with an openly out of date gcc/glibc might underperform one with gcc/glibc up to date (with e.g. the newer SSE instuctions supported), for example. So for security AND performance, one typically wants an aggressively maintained distribution, yet one that doesn't advance so rapidly that it thereby becomes unstable because it lives at the bleeding edge. The major commercial linuces try to tread precisely this line, and that is in fact the primary added value in what they sell. The free linuces have to face the same issue. Debian recognizes this internally with stable and unstable versions, where unstable likely has better performance but requires a certain degree of expertise to be comfortable with. I suspect that the fedora project is now RH's "RH unstable" version, as they will no longer be able to tolerate any significant degree of instability in their moderately expensive commercial release. Caosity and a few others will likely closely track Fedora in level of relative (in)stability. I don't know what slackware does currently -- because of its relatively high degree of aggregation and relatively weak dependency resolution, I'm guessing that it tends to be very slowly maintained and advanced -- each release is stable and is advanced only very incrementally with bugfixes only until the next release, because of the difficulty in rebuilding just a single node and its dependencies without unexpected side effects. In the old days this was a plague -- every time I'd add something new to a slackware box and build it, if it required newer libraries than slackware had installed it was a nightmare to rebuild everything required to make the new package happy. So the REAL issue is not performance. It is human time, specifically your time. If you do your job well, you'll end up with a network that damn near takes care of itself, where you do work once and it is transparently replicated many times throughout your LAN. Therefore pick a distro that you are reasonably certain: a) Has an aggressive package update schedule, especially wrt security updates but also performance. b) Has intelligent package dependency resolution (in my opinion, this is rpm-based or debian). c) Similarly, makes it easy to rebuild packages from source. d) Has intelligent package management and maintenance tools, e.g. apt, yum, urpmi. e) Supports the notion of a "package repository", ideally local repositories that you can maintain yourself and that can provide high bandwidth controlled load services to your clients WITHIN your LAN. f) Supports PXE-driven fully automated installation of one sort or another. g) Has some sort of bug-feedback-resolution mechanism, e.g. bugzilla. h) Is aggressively open source and charges FAIR PRICES per client, if it charges at all. Alas, IMHO a number of the commercial distros are smelling large profits and are starting to charge like Microsoft for a product they don't even own. You get what you pay for, sure, but be sure you pay for the VALUE of what you get relative to the absolutely free linux distros if you go for any of the distros that seek to charge you per seat. i) Is used by enough people that there exists a good online support forum. Good "enough" for your level of expertise, really -- I don't mean to discourage people, especially competent system programmer people, from choosing caosity or fedora just because they are so new they don't have as many adopters yet as commercial distros or debian. Linux is the quintessential "community" product; we all own it, and we all are responsible for making it work. Adopt one tiny bit of it and help out, if you can. There are probably more features I should add, but I've got to move on. Busy list today, and also a busy teaching day for rgb (starting in one hour). rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From award at correu.andorra.ad Tue Jan 20 09:16:19 2004 From: award at correu.andorra.ad (Alan Ward i Koeck) Date: Tue, 20 Jan 2004 15:16:19 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) References: Message-ID: <400D3833.F1D97ABB@correu.andorra.ad> Well, my own preference goes to Python, which - has a syntax I prefer a lot over C - while being an interpreted language, manages to be reasonably speedy, unlike Java - has a fair amount of predefined library objects you can use to access C POSIX system routines but I do recognize that low-level stuff definately should be coded in C. Now, as to just what "low-level" means ... Best regards, Alan Ward "Robert G. Brown" wrote: (snip) > > Let it be Jihad, then. > > The answer technically is "it depends". This is to make the vast horde > of java programmers feel loved, and to acknowledge that for writing web > applications their primary scripting language (like perl, python and > many another before it) has a purpose in God's world. > > There. Let us bask for a moment in the serenity of our knowledge that > we have the complete freedom to choose, and that there are no wrong > answers. > > Now we can give the correct answer, which is "C". (snip) > -- > Robert G. Brown http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gropp at mcs.anl.gov Tue Jan 20 08:47:02 2004 From: gropp at mcs.anl.gov (William Gropp) Date: Tue, 20 Jan 2004 07:47:02 -0600 Subject: [Beowulf] can't include all machines in my beowulf In-Reply-To: <20040120055022.18675.qmail@web21506.mail.yahoo.com> References: <20040120055022.18675.qmail@web21506.mail.yahoo.com> Message-ID: <6.0.0.22.2.20040120073814.02186048@localhost> At 11:50 PM 1/19/2004, prakash borade wrote: >hey tahnks for your advice >i was able to find the bug > >thank you very much > >do you know what is MPE and what role does it play in >mpich > where cpi don't require it >but cpilog requires it cpilog uses the logging features provided by the MPE (MultiProcessing Extensions) library, which is included with MPICH and works with all MPI implementations. cpi is a basic MPI program and can be built with any MPI implementation; cpilog makes calls to the routines in the MPE library to log MPI and other operations. Bill _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Tue Jan 20 12:16:13 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Tue, 20 Jan 2004 18:16:13 +0100 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: References: Message-ID: <400D625D.9000902@telia.com> Robert G. Brown wrote: >On Tue, 20 Jan 2004, Per Lindstrom wrote: > > > >>Until I have learned to manage Debian I will use the >>"slack-rpm-transfer-tool" and it will be very interesting to benchmark >>the two distros with each other. >> >>Thanks a lot for all help and support in his case. >> >> > >Just to be clear, the issue is not "benchmarking two distros" for >performance. It is strictly manageability and scalability. > > > I got your point "total operation cost in relation to total performance". Thanks again for the knowledge, experience and information you are sharing with us. Regards Per _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Tue Jan 20 13:15:59 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Tue, 20 Jan 2004 10:15:59 -0800 (PST) Subject: [Beowulf] LAM/MPI Setup Message-ID: <20040120181559.73844.qmail@web12703.mail.yahoo.com> hi all, Have been trying (unsuccessfully) to run LAM/MPI on my 2 node network. But when I run lamboot, it says Permission Denied. the and the network is configured properly. Also the problem is with ssh. Earlier I had the same problem with rsh. I sorted it out by adding: in.rshd 192.168.0. and in.rlogind 192.168.0. although I have also added sshd 192.168.0. to the /etc/hosts.allow file I get the ssh error. I got the above from Google, if anyone has a better solution please help. Thank You Toufeeq ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From irene at syndeocorp.com Tue Jan 20 15:53:27 2004 From: irene at syndeocorp.com (Irene Koo) Date: Tue, 20 Jan 2004 12:53:27 -0800 Subject: [Beowulf] Bonding driver in Active-standby mode Message-ID: <012401c3df97$74b79570$39231504@irenexppc> Hi, I am new to using the channel bonding driver. So, any help would be appreciated. Thx. Basically, I have one 3Com 3c905c (using 3c59x driver) and an integrated Intel NIC using e100 driver on one system. Both of them are ifenslaved to bond0 with parameters mode=1 arp_interval=20 arp_target_ip=10.1.5.105 bonding.c:v2.2.5 (May 22, 2003) bond0 registered without MII link monitoring, in fault-tolerance (active-backup) mode. bond0 registered with ARP monitoring set to 20 ms with 1 target(s): 10.1.5.105 On another system, I have one 3Com 3c905b and an integrated Intel NIC using e100 driver. Both of these NICs are ifenslaved to bond0 with parameters mode=1 arp_interval=20 arp_target_ip=10.1.5.100 Two cross-over cables connect these 2 systems. The question I have is when the backup interface gets activated or when a recovered interface (the link failed and the up) get activated, the kernel recorded over 300 times of the following messages: bond0: backup interface eth0 is now up bond0: backup interface eth0 is now down bond0: backup interface eth0 is now up bond0: backup interface eth0 is now down bond0: backup interface eth0 is now up bond0: backup interface eth0 is now down .... or something like bond0: backup interface eth2 is now up last message repeated 384 times During the time these messages were displayed, traffic from ping does not go through. After about 30 sec, messages such as bond0: eth0 is up and now the active interface or bond0: eth2 is up and now the active interface was displayed, then traffic would go through again. What is causing all those repeated messages? It looks like the bonding driver was in a loop. From the experiment, I can see that there is a 30 sec down time. Is there something I can do to get ride of the down time? I also used mii-diag --monitor to monitor both interfaces. All "down/negotiating/up" messages are shown almost instantuously after the link went down/up. I also tested with arp_interval=200. Results from this set is pretty simular to the ones above To make things more interesting, I also tried miimon (without arp monitoring). Bond driver setting was miimon=20 and miimon=200 For 3Com 3c905b/c, it took close to 50 sec for the bond driver to switch over; For Intel integrated NIC, it took a bit over 1 sec. Any ideas what is causing the delay? Thanks, Irene K. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Tue Jan 20 16:01:55 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Tue, 20 Jan 2004 16:01:55 -0500 (EST) Subject: [Beowulf] can i increase priority of process on beowulf with mpich In-Reply-To: <20040120115919.1940.qmail@web21502.mail.yahoo.com> Message-ID: On Tue, 20 Jan 2004, prakash borade wrote: > hello > proksy here > > can we change the priority of the process on the > cluser using mpich > > because i have seen that when i run the process as a > root with priority -10 it is maintained only on master > how can i maintain same priority on all machines > Running MPI programs as root is not considered a good idea. Are you using a batch scheduler? Doug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From atp at piskorski.com Tue Jan 20 17:47:46 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Tue, 20 Jan 2004 17:47:46 -0500 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <200401201701.i0KH1Oh22025@NewBlue.scyld.com> References: <200401201701.i0KH1Oh22025@NewBlue.scyld.com> Message-ID: <20040120224741.GA81067@piskorski.com> On Tue, Jan 20, 2004 at 08:05:45AM -0500, Robert G. Brown wrote: > On Mon, 19 Jan 2004, prakash borade wrote: > > can anybody tell me > > what do i choose for programing > > either GNU c or JAVA for implementing some > > networkprograms using some linux systemcalls Hehe. Um, is any of this supposed to be Beowulf related? > Gads, that's two questions in two days with the distinct whiff of crowds > of screaming peasants waving swords and torches... > Now we can give the correct answer, which is "C". > W. Richard Stevens, Networking God (deceased): > http://www.kohala.com/start/ Excellent advice. The three books of his that I have are very good. Now, my own contribution to the jihad. :) None of this is particularly specific to "network programs" by the way: The conventional wisdom is that C is a "portable assembly language" for low-level bit twiddling, performance criticial library functions, and writing byte code compiles for scripting languages, and my own experience seems to bear that out. So, IMNSHO, you don't want to do all your programaming in C, you don't even want to do most of your programming in C. In fact, C should viewed like habaneros, or perhaps caviar - use where needed and called for, no more. So what other language to use? Well: - For any project of reasonable size, you HAVE to know C, because it is ubiquitous, and you will have to use it somewhere. (Your OS calls are in C, 95% of the other APIs you'll want to call are in C, etc. etc.) - Therefore pick a very high level language suitable for the most rapid development possible, and use that for as much of your code as possible. But the ability to easily interface with C is also very important. - Use C only where you really need it, which means: 1) Small amounts of performance critical code. 2) Where you have to in order to talk to other C code - vast numbers of 3rd paty libraries, etc. - For the "very high level language", so far I've found that a scripting language like Tcl works very well. (For more math and especially statistics intensive use, S/S-Plus/R can be good.) - Proper use of an RDBMS (Oracle, PostgreSQL, etc.) can make a huge difference, as it provides a high level interface to many powerful features that are hard to find elsewhere - ACIDity, concurrency, constraints to enforce data correctness, etc. A corrolary of the above is that Java isn't interesting. Java is higher level than C but not very much. I'm already using C for any really low level stuff, I want something a LOT higher level for anything else. Java would probably look pretty attractive if doing ALL my coding in C was my only other option, but it isn't. Therefore, my very limited contact with Java says, Java isn't it. There you go, the answer is Tcl. :) With it's Threads Extension (or in AOLserver), it's particularly fabulous for multi-threaded programming. (Ironically, Tcl guru's have traditonally emphasized concurrent programming using the event loop rather than threads.) And I can always drop down and implement a new Tcl command in C if I need to. I've heard other potential answers ("Standard ML for everything!", "we just need a new Lisp dialect..."), but about those, well, I can't really say. -- Andrew Piskorski http://www.piskorski.com/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Tue Jan 20 18:50:10 2004 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 21 Jan 2004 00:50:10 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: References: <20040120065449.33171.qmail@web21507.mail.yahoo.com> Message-ID: <20040120235009.GD30005@unthought.net> On Tue, Jan 20, 2004 at 08:05:45AM -0500, Robert G. Brown wrote: > On Mon, 19 Jan 2004, prakash borade wrote: > > > hey > > > > can anybody tell me > > what do i choose for programing > > either GNU c or JAVA for implementing some > > networkprograms using some linux systemcalls > > Gads, that's two questions in two days with the distinct whiff of crowds > of screaming peasants waving swords and torches... > > Let it be Jihad, then. It is better to light a flame thrower than curse the darkness. - Terry Prachett, "Men at Arms" Just keep in mind that *my* religion is *the* superior one, no matter what the infidels want you to believe! ;) > The answer technically is "it depends". This is to make the vast horde > of java programmers feel loved, and to acknowledge that for writing web > applications their primary scripting language (like perl, python and > many another before it) has a purpose in God's world. I beg to differ. It does not depend. Not when Java is in the discussion. Once Java is out, we can do the huggy-feely 'it depends' and "just because it sucks it doesn't mean it doesn't have a purpose" type discussions. For any systems related programming task, C or C++ is to prefer over Java. For the following reasons: *) C and C++ have consistent run-time performance, which Java *cannot* have due to its automatic garbage collecting (yes, programmers can apply various tricks to lessen the performance penalty and unpredictability - but that's more work for the programmer and for the administrator, and it simply cannot be as good as a well written C or C++ program) *) Portability - well-written C and C++ programs can fairly easily be ported between various UN*X type systems - and new systems can fairly easily be supported by the existing codebase. Larger Java programs are often not portable among various JVMs, and adding portability is not convenient in Java (since it lacks the "dirty" macro-processor of C and C++). In other words; wrt. portability Java is a very "pure" language, since it lacks the constructs (preprocessor) that would be unnecessary in a "perfect world" - unfortunately for Java, the world is not perfect, and therefore the "dirtier" languages have a major advantage here. *) Readability - for larger systems I recommend C++ over C - it allows you to write very efficient type-safe code, unlike C (which would either not be type-safe, or require more code) and Java (which is inefficient by design, and will do much of its type checking at run-time, decreasing programmer productivity, program performance and ease of testing) *) Reliability - well written C++ tends to be very stable - most problems will display themselves at compile time, and you have excellent tools to verify that your code is running as it should at run-time (valgrind, ElectricFence, LeakChecker), and to inspect what went wrong when your program crashed (core dumps and GDB). Most of this holds true for C as well, except that fewer errors show themselves at compile time, because the C type system is less strict than the C++ type system. > > There. Let us bask for a moment in the serenity of our knowledge that > we have the complete freedom to choose, and that there are no wrong > answers. I find your lack of faith disturbing... ;) > > Now we can give the correct answer, which is "C". Typing a little fast there, I think... The correct answer for anything larger than 1KLOC is "C++" - of course, you knew that, you were just a little fast on the keyboard ;) (KLOC = Kilo-Lines-Of-Code) > > In order to give it, I have to infer an implicit "better than" and a > certain degree of generality in your question is in "which language is > BETTER suited to write an efficient networking program using linux > systemscalls, in GENERAL". > > With this qualifier, C wins hands down. A variety of reasons: > > a) The operating system is written in C (plus a bit of assembler) This ought not to be a good argument - I guess what makes this a good argument is, that the operating system provides very good C APIs. So, any language that provides easy direct access to those C APIs have a chance of being "the one true language". > b) Nearly all network daemons and utilities and program clients are > written in C or perhaps an extension of C such as C++ I prefer to think of "C++" as "A better C", rather than a "C extension", as not all C is valid C++, and therefore C++ is not really an extension. > c) Nearly all decent books on network programming (e.g. Stevens) > provide excellent C templates for doing lots of network-based things > d) You can do "anything" with C plus (in a few, very rare cases, a bit > of inlined assembler) Amen! This goes for C++ as well though. > e) C is a compiler and produces (in the hands of a good programmer) > code that runs very efficiently. Java is an interpreter and does not. Java is not necessarily interpreted - but that doesn't help (it is still garbage collected, and it's general library routines such as lists etc. still work on "Object" types, not the real types of the elements, which makes it run-time inefficient by design). So I absolutely agree that Java sucks, just for slightly different reasons :) > f) C is an open ANSI standard implemented in linux with fully open > source tools with no restrictions or covenants on their usage and with > no "vendors" in any sort of position of control over the language itself > or its core libraries. Java was invented by Sun, and Sun (AFAIK) still > controls the distribution of its core libraries, and as an interpreted > language, those libraries are sine qua non for portability. Sun is > relatively linux friendly, but I personally tend not to trust giant > multibillion dollar companies that own core components of something I > need to rely on (or incorporate those components into my code), as they > have an annoying tendency to turn around and want money for it. > g) Or find themselves imbroiled in a corporate war with someboy ELSE > with deep pockets advancing a different "standard"; for example I'd > guess that Microsoft would like nothing better than to co-opt Java for > themselves and turn around and want money for it. C is co-opt > proof, and GCC is a highly reliable platform upon which to base code in > the certain knowledge that the core compiler and libraries will remain > royalty free. > h) ...and I could go on. Me too, oh, me too ;) > > Note that I do not address java's OO nature, as OO C variants exist AND > as OO programming is largely a matter of programming style, not language > per se anyway. Let me then. Java *forces* OO. C makes OO possible, but cumbersome and error prone. C++ makes OO both possible and safe. The fact that Java forces OO down your throat, whether you want it or not, makes it an inferior language alone. See, most real world problems are not stricly OO, or strictly procedural. They tend to be a good mix. This is why C becomes cumbersome in many larger projects, since it does not address the OO parts well. And this is why Java is hopeless for larger projects, since it simply forces a re-write of the would-be simple procedural code into something sufficiently OO-like for the Java language to be able to comprehend it. C++ on the other hand allows (and even encourages) a mix-and-match approach - using the proper "tools" for the job at hand. Oh, and not only does Java force OO, it forces inferior OO. It cannot do multiple inheritance, which makes a lot of the good arguments for OO vanish (single-inheritance encourages fat bases, which is all that OO is *not* about). > ... > > Note also that my diatribe above is clearly biased as I don't code in > Java, largely become of all of the reasons above (and then some). > Speed, power, portability, open standard and source, and available code > base and learning support all argue in favor of C about 10 to 1. UNLESS > you are trying to write a powerful web-distributed application, which is > a PITA in C and what Java was designed for. Yes - I've been told that that is what Java does well. In generall, I am pretty much of the oppinion that C++ does 99% of all problems "well" or "very well", and the remaining 1% somewhere between "poor" and "well". Java on the other hand will do 99% of all problems somewhere between "crap" and "not at all", where it will do the remaining 1% "extremely well". Unless you have a problem in that last 1%, there is probably better languages than Java. C, in my oppinion, would be somewhat like C++, except for larger problems it doesn't fare qute as well (not poorly by any means, just not as well). The only very very large problem with C++ is, that almost no people know the language. There is a mountain of absolutely crap learning material out there. This is why you see examples of "C versus C++" where the C++ code is several times larger or even less efficient than the C example, because the author felt that shovelling bad OO and braindead design after the problem seemed like a good idea. I believe that with the state of compilers today, nobody should have to start a large project in C - except if it absolutely needs to run on a platform for which no decent C++ compiler is available (maybe Novell NetWare - but that's the only such platform that comes to mind...) And I firmly believe, that in order to justify starting a project in Java, you must have *thorough* understanding of not just the problem and Java, but also about possible alternative languages - there really should be very few excuses for using the language for anything at all. ... > P.S. -- I >>tried<< to be mild, but I suppose I'll need to spend the > rest of the day in a bunker wearing my lead underwear and a tinfoil > skullcap. Really, I love YAPL's as they pop up from year to year. I > do. Marshmellows...ready! Flame on... My office is cold, and I like the flames :) Seriously though, I think that the language-flamewars are fun and informative, since so much happens in the space of compilers and real-world projects out there. So, I think it's useful to get an update every now and then, from people who have strong feelings about their langauges - oh, and a discussion in the form of a friendly flame-fest is always good fun too ;) Well, the bait is out, let's see if someone bites ;) / jakob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Wed Jan 21 01:39:57 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed, 21 Jan 2004 01:39:57 -0500 Subject: [Beowulf] fully rpm-ified NCBI toolkit available Message-ID: <1074667197.4344.28.camel@protein.scalableinformatics.com> Hi folks: I wanted to fix up my original packaging of the ncbi toolkit to make full use of the RPM mechanism for easier compilation and installation on clusters and other systems. I built a spec file for it and created a source RPM which you can pull down from http://downloads.scalableinformatics.com/ specifically http://downloads.scalableinformatics.com/downloads/NCBI-2.2.7-1.src.rpm There is nothing OS or target specific in this RPM. I will add in the Opteron patches and some other tweaks in short order, and get an updated source RPM. To build this RPM into a set of binary RPMs, do the following as root user: rpmbuild --rebuild NCBI-2.2.7-1.src.rpm If you want to build for a specific target processor, use the --target option, as in rpmbuild --rebuild --target i686 NCBI-2.2.7-1.src.rpm Please let me know if you find any problems. It has rebuilt fine on a number of machines thus far. I would like to see what happens on non-native RPM architectures (OS-X, and other Unices). -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From camm at enhanced.com Wed Jan 21 09:22:54 2004 From: camm at enhanced.com (Camm Maguire) Date: 21 Jan 2004 09:22:54 -0500 Subject: [Beowulf] Selection of Beowulf cluster type In-Reply-To: <1074603930.2417.8.camel@tp1.mesh-hq> References: <400C6F80.6060600@comcast.net> <400CFB5F.4010104@telia.com> <1074603930.2417.8.camel@tp1.mesh-hq> Message-ID: <541xptmldt.fsf@intech19.enhanced.com> Greetings! Just a brief plug for Debian here -- we've been using it on our commercial cluster for many years and across several hardware upgrades, and, after the initial installation, never futzing with the software by hand other than 'apt-get install' and/or 'apt-get dist-upgrade'! We're literally running the 'same installation' automatically upgraded via apt through at least 3 or 4 Debian release generations. And the quantity of parallel software available 'out of the box' is enormous! I'll never forget the day I could just 'apt-get install scalapack-lam-dev' and diagonalize huge matrices across the cluster with hardly a hitch. As for the initial installation, some people like FAI, though I have never tried it myself, as I've only had to install once, and that was somewhere around 1995 .... Take care, Lars Henriksen writes: > On Tue, 2004-01-20 at 09:56, Per Lindstrom wrote: > > Here I am not running for Slack for its own sake so I will use Slacware > > up to two (2) nodes and when shifting to Debian when I'm connecting the > > remaining nodes. I think it will stop at about 10 CPUs. > > > > Until I have learned to manage Debian I will use the > > "slack-rpm-transfer-tool" and it will be very interesting to benchmark > > the two distros with each other. > > If you want to use Debian as the base distribution, I can recommend > SytemImager as an installation tool. > > http://www.systemimager.org/ > > Sytemimager allows you to replicate an installed client via PXE/rsync to > the remaining nodes. It also allows for updates to be distributed via > rsync. A feature that i really like, is that it allows for the use of > several different images. > > Also it scales pretty well (I had the pleasure of using it in a couple > of 16-32 node configurations, as well as on our own cluster) > > IMO it is the easiest way to deal with smaller clusters (except maybe > CLIC) > > best regards > Lars > -- > Lars Henriksen | MESH-Technologies A/S > Systems Manager & Consultant | Forskerparken 10 > www.meshtechnologies.com | DK-5230 Odense M, Denmark > lars at meshtechnologies.com | mobile: +45 2291 2904 > direct: +45 6315 7310 | fax: +45 6315 7314 > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > -- Camm Maguire camm at enhanced.com ========================================================================== "The earth is but one country, and mankind its citizens." -- Baha'u'llah _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hanzl at noel.feld.cvut.cz Wed Jan 21 10:44:18 2004 From: hanzl at noel.feld.cvut.cz (hanzl at noel.feld.cvut.cz) Date: Wed, 21 Jan 2004 16:44:18 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120224741.GA81067@piskorski.com> References: <200401201701.i0KH1Oh22025@NewBlue.scyld.com> <20040120224741.GA81067@piskorski.com> Message-ID: <20040121164418Z.hanzl@unknown-domain> > scripting language like Tcl works very well I think tcl(/tk) fulfills most of the promisses heard about Java :-) Those tempted by Java hype might therefore well try tcl (and C/C++ of course). I use tcl for various purposes throughout the cluster and I am very happy with it. Superb integration ability. There is no better glue than tcl. But it is a pleasing language on its own. Excellent ability to go to meta-levels like adding new control structures (comparable to FORTH if anybody remembers it). Making GUI for some cluster management tool is a fun with tcl/tk. And few bonuses: - it even works on M$ Windows and Mac (shock your friends ;-) - if you ever dreamt about goodies from LISP without having to deal with Lots of Silly Idiotic Parenthesis, these dreams come true in tcl One strong detergent: Syntax which does not follow usual conventions. But you will appreciate it later, for sure. There is a very strong logic behind it. (Those who ever tried deep TeX programming: tcl syntax shock is say 10% of the TeX syntax shock ;-) Just my religious 2 cents... Vaclav Hanzl _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 21 11:46:15 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 21 Jan 2004 11:46:15 -0500 (EST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120235009.GD30005@unthought.net> Message-ID: On Wed, 21 Jan 2004, Jakob Oestergaard wrote: > > Well, the bait is out, let's see if someone bites ;) > Having been accused of having early alzheimers and forgetting some silly little symbol, what was it, oh yeah, a "++" in my even handed and totally objective diatribe, I'll have to at least nibble:-) > It is better to light a flame thrower than curse the darkness. > - Terry Prachett, "Men at Arms" Alas, my flame thrower is in the shop. The best I can do is make a nifty lamp with a handy wine bottle, some gasoline, and some detergent flakes. There, let me stick this handy rag in the neck like this, now where were we again? My memory is failing me. Oh yes. The pluses. > > There. Let us bask for a moment in the serenity of our knowledge that > > we have the complete freedom to choose, and that there are no wrong > > answers. > > I find your lack of faith disturbing... ;) Trying an old jedi mind trick on me, are you? At least your metaphor is correct. You are indeed attracted to the Power of the Dark Side...:-) > > Now we can give the correct answer, which is "C". > > Typing a little fast there, I think... The correct answer for anything > larger than 1KLOC is "C++" - of course, you knew that, you were just a > little fast on the keyboard ;) > > (KLOC = Kilo-Lines-Of-Code) Well, we COULD make a list of all the major programs involving > 1KLOC (exclusive of comments, even) that have been written entirely in C (plus a small admixture, in some cases, of assembler). Unfortunately, the list would be too long (I'd be here literally all day typing), and would include things like the kernels of pretty much all the major operating systems (certainly all the Unix derivatives and variants) and gcc itself, including its g++ extension. To be fair, that doesn't prove that your observation is wrong -- there is a rather huge list of fortran sources available in my primary field of endeavor, for example, which doesn't stop fortran from being a cruel joke. However, to be fair the OTHER way most of that fortran was written by scientists (a.k.a. "complete idiots where advanced computer programming is concerned") to do what amounts to simple arithmetic problems, or maybe even complex arithmetic problems, fine, with a trivial data interface (read the data in, write the data out). Quite a bit of the aforementioned C sources and operating systems and advanced toos were written not only by computer professionals, but by "brilliant" computing professionals. Class O stars in a sea of A, B, F and G (where fortran programmers are at best class M, or maybe white dwarfs). What you probably mean is that IF everybody knew how to program in C++, they would have written all of this in C++, right? Let's see, is there a major operating system out there that favors C++? Hmmmm, I believe there is. Are it and its applications pretty much perpetually broken, to the extent that a lot of its programmers have bolted from C++ and use things like Visual Basic instead? Could be. This isn't intended to be a purely a joke observation. I would rather advance it as evidence that contrary to expectations it is MORE difficult to write and maintain a complex application in C++. The very features of C++ that are intended to make an application "portable" and "extensible" are a deadly trap, because portability and extensibility are largely an illusion. The more you use them, the more difficult it is to go in and work under the hood, and if you DON'T go in and work under the hood, things you've ported or extended often break. To go all gooey and metaphorical, programming seems to be a holistic enterprise with lots of layering and feathering of the brush strokes to achieve a particular effect with C providing access to the entire palette (it was somebody on this list, I believe, that referred to C as a "thin veneer of upper-level language syntax on top of naked assembler"). C++, in the same metaphor, is paint by numbers. It ENCOURAGES you to create blocks to be filled in with particular colors, and adds a penalty to those that want to go in and feather. In some cases, those paint-by-numbers blocks can without doubt be very useful, I'm not arguing that. It is a question of balance. A very good C++ programmer (and I'm certain that you are one:-) very likely has developed a very good sense of this balance, as suggested by your observation that a good C++ programmer writes what amounts to procedural C where it is appropriate (which IMHO is a LARGE block of most code) and reserves C++ extensions for where its structural blocking makes sense. Who could argue with that? Of course, a very good programmer in ANY language is going to use procedural methodology where appropriate, and create "objects" where they make sense. We must not make the mistake of comparing good programming practice with the language features. Here are the real questions. Presuming "good programmers, best practice" in all cases: a) Do C's basic object features (typedefs, structs and unions, for example) suffice to meet the needs for object creation and manipulation in those code segments where they are needed? I would say of course they do. IMO, protection and inheritance are a nuisance more often than an advantage because, as I observed, real "objects" are almost never portable between programs (exceptions exist for graphical objects -- graphics is one place where OO methodology is very appropriate -- and MAYBE for DB applications where data objects are being manipulated by multiple binaries in a set). "protection" in particular I think of as being a nuisance and more likely to lead to problems than to solutions. In a large project it often translates into "live with the bugs behind this layer, you fool, mwaahhahahaha". Result: stagnation, ugly hacks, bitterness and pain. In single-developer projects, just who and what are you protecting your structs FROM? Yourself? b) Do C++'s basic object features increase or decrease the efficiencies of the eventual linked binaries one produces? As you say, C++ is more than an "extension" of C, it has some real differences. In particular, it forces (or at least, "encourages") a certain programming discipline on the programmer, one that pushes them away from the "thin veneer" aspect of pure C. I think it is clear that each additional layer of contraint INCREASES the intrinsic complexity of the compiler itself, thickens the veneer, and narrows the range of choices into particular channels. Those channels in C++ were deliberately engineered to favor somebody's idea of "good programming practice", which amounts to a particular tradeoff between ultimate speed and flexibility and code that can be extended, scaled, maintained. So I would expect C++ to be as efficient as C only in the limit that one programs it like C and to deviate away as one uses the more complex and narrow features. Efficient "enough", sure, why not? CPU clocks and Moore's Law give us cycles to burn in most cases. And when one uses objects in many cases, they ARE cases where bleeding edge efficiency IS less important than ability to create an API and "reuse" data objects via library calls (for example). Still I think C would have to win here. c) Do C's type-checking, etc. features suffice to make programming particularly easy or safe? Here I will give C++ a win, as the answer for C at least is no. C is not particularly easy and it most definitely is not safe. It is a THIN veneer on top of assembler, and assembler is as unsafe as it gets (short of writing in naked machine code), so thin that one can easily inline assembler to access and manipulate CPU registers etc and have a pretty good idea of how to smoothly move data around and switch from one programming mode to the other. Again the art metaphor is appropriate -- with color-by-numbers you are "safer" from creating really, really ugly pictures (although it is always possible, of course:-). With nothing but the canvas, the paints, and a set of brushes and palette knives you can create anything from Da Vinci or Rembrandt to a three-year-old's picture of a dog (big blob of muddy brown in the middle of the canvas that may or may not even have "eyes"). This is why I think that the C++ vs C issue is almost entirely determined by an individual's personal taste and preferences, along with (maybe) the amount and kind of code that they write. I personally prefer to eat my vegetables (objects) raw -- if I need a struct, I make a struct. If I want to allocate a struct, I use malloc or write a constructor, depending on the complexity of the struct. If I want to de-allocate a struct, I either use free or I write a destructor, again depending on the complexity (whether or not I have to recursively free the contents of the struct, and at how many levels, how many times in the code). If I want to change the struct (either structurally or by accessing or altering its contents), I change the struct, and am of course responsible for changing all the points in my program that are affected by the change. If I want to "protect" the struct, well, I don't change it, or write to it, or read it, or whatever. My choice, unchanneled by the compiler. Do I have to deal with sometimes screwing up data typing? Absolutely. Do I have to occasionally deal with working my way backwards through my programs to redo all my structs? Of course (I'm doing it right now with a program I'm writing). This process is INEVITABLE in real-world program design because we DON'T know a priori what the best shape is for the data objects is until we've programmed them at least once the wrong way, in most cases. The really interesting question is whether or not this PROCESS is any easier in C++ than in C. I can't see why it would be, but perhaps it is. I suspect it is still mostly a matter of skill and style (and how one apportions time between active code development and "planning" a program in the first place) and, of course, personal taste. > > In order to give it, I have to infer an implicit "better than" and a > > certain degree of generality in your question is in "which language is > > BETTER suited to write an efficient networking program using linux > > systemscalls, in GENERAL". > > > > With this qualifier, C wins hands down. A variety of reasons: > > > > a) The operating system is written in C (plus a bit of assembler) > > This ought not to be a good argument - I guess what makes this a good > argument is, that the operating system provides very good C APIs. > > So, any language that provides easy direct access to those C APIs have a > chance of being "the one true language". You miss my point entirely. The argument is that C is a thin veneer on top of assembler -- so thin that one CAN write an operating system in it. Imagine writing an operating system in LISP. Wait, don't do that. The results are too horrible to imagine. Imagine doing it on top of fortran, instead. That's bad, but you won't have nightmares for more than a week or two afterwards. (IIRC, somebody actually did this once.) The API issue is moot. Hell, perl and python have direct access to most of the C APIs. IN THE CONTEXT of the reply, of course, there was also the suggestion that the C APIs are a good way of writing network code since the network drivers and kernel structs those APIs provide access to were all written in C, so your access is pretty much "naked". You can often read or write directly any register or value or memory location that isn't in the protected part of the kernel, if you dare (or need to to achieve enough efficiency in your particular application). But C++ I'm sure provides the same degree of naked access and C and C++ share a common underlying data organization, really, and even Fortran (with somewhat different data organization) probably does pretty well. In other languages (especially scripting languages e.g. java, perl, python), the access is typically "wrapped" in a translation layer that is required because one has NO CONTROL over the way the interpreter actually creates data objects. They are the "ultimate" in OO programming -- instantly created by the interpreter in real time, manipulable according to a wide set of rules, they go away when they aren't being used without leaking memory (usually:-) but heaven help you if you look beneath the hood and try to manipulate the raw memory addresses. This is a lovely tradeoff in a lot of cases, which is why I hesitate to actually stick a rod up java's ass and barbeque it on a spit. I really do believe the BASIC argument associated with OO programming, that the data and what you want to do with it should dictate program design features, including choice of language and programming technique. For many problems perl or python provide a very good data interface where one DOESN'T have to mess with lots of data issues that are a major chore in C. Wanna make a hash? Use it. Wanna make your hash into an array? Change the little % TO A @ and hack your dereferencing a bit. Is your hash actually stored as a linked list, as an array of structs? Don't ask, don't tell, don't try to look. All this and direct syntactical manipulation of regular expressions, what's not to love? This is what I was trying to keep an open mind to WRT java. Perhaps there are aspects of its data manipulation methodologies and programming features that are excellent fits to particular problems. I just don't know, and would rather have my teeth drilled with a half-charged portable black and decker screwdriver (allen bit) than learn YAPL just to find out. Hell, I'm only starting to learn python under extreme duress as I swore perl was going to be the last language I ever learned and that was before PHP and now python. Somebody would have to pay me a LOT OF MONEY to get me to learn java. Yessir, a whole lot. [Anybody reading this who happens to have a lot of money is welcome to contact me to arrange for a transfer...;-)] > I prefer to think of "C++" as "A better C", rather than a "C extension", > as not all C is valid C++, and therefore C++ is not really an extension. Funny that. I tend to think of C++ as "A broken C extension" for exactly the same reason;-) If they hadn't broken it, then there really would be no reason not to use it, as if C were a strict subset of C++ all the way down to the compiler level, so that pure C code was compiled as efficiently as pure C is anyway then sure, I'd replace all the gcc's in my makefiles with g++'s. That would certainly lower the barrier to using C++; C programmers could program in C to their heart's content and IF AND WHEN their program needs a "C++ object" they could just add it without breaking or even tweaking the rest of their code. Breaking C was (IMO) a silly design decision. So was adding crap like a completely different stdin/stdout interface, which makes most C++ code, even that which doesn't use objects in any way, most non-back-portable-or-compatible to C. It is very clear that these were all deliberate design decisions INTENDED to break C and FORCE programmers to make what amounts to a religious choice instead of smoothly extend the palette of their programming possibilities. I'd honestly have to say that it is this particular aspect of C++ more than any other that irritates me the most. There is a HUGE CODE BASE in C. For a reason. > > c) Nearly all decent books on network programming (e.g. Stevens) > > provide excellent C templates for doing lots of network-based things > > d) You can do "anything" with C plus (in a few, very rare cases, a bit > > of inlined assembler) > > Amen! This goes for C++ as well though. Y'know, you won't believe this, but I actually added the (and C++) references in my original reply just thinking of you...;-) Last time we had this discussion you were profound and passionate and highly articulate in advancing C++ -- so much so that you almost convinced me. Alas, that silly barrier (which I recall your saying took you YEARS to get over yourself)... I just bounce right off of it in a state of total irritation every time I try. Another problem is that every time I work with a student C++ programmer coming out of Duke's CPS department (which now teaches C++ as standard fare) I observe that while they are just great at creating objects and so forth they are clueless about things like pointers and malloc and the actual structure of a multidimensional array of structs and how to make one and manipulate its contents. As a consequence I have to spend weeks teaching them about all of the virtues of "C as a thin veneer" in order to get them to where they can cut advanced code at all, in either one. You may not have encountered this as you did C first. Or maybe you never have cause to allocate triangular arrays of structs or to build your own linked list of structs, or maybe C++ does this and eats your meatloaf for you and I'm just to ignorant to know how. Its funny. Back in the 80's all the CPS departments taught pascal because it provided strong type checking and absolutely forced one to use a rigorous bottom-up program design methodology. Precisely the same features that caused all real programmers to run from it screaming, as somehow all the programmers I've ever met work top down initially, then top, bottom, middle, whereever until the project is finished, often with a few epiphany-driven total project reorganizations in the middle as experience teaches one what the REAL data objects are that they are manipulating and whether they need easily maintained code that can run relatively slowly or whether they need the core loop to run faster than humanly possible if they have to hand-code it in assembler to get it there. Now they teach C++, and as you observed, teach it badly. I humbly submit that the REASON they teach it badly is that they aren't teaching "C++" at all, they are teaching structured programming and programming discipline with a language that simply doesn't permit (or if you prefer encourage) the student to use methodologies with far more power but far less externally enforced structure. Pure pedagogy, in other words. I personally think the world would be a better place if they FIRST taught the students to code in naked C with no nets, and taught them that the reason for learning and using good programming discipline is BECAUSE the bare machine that they are working with comes with no nets. Then by all means, teach them C++ and object oriented design principles. I suspect that students who learn C++ in this order are, as you appear to be, really good programmers who can get the most out of C++ and its object oriented features without "forgetting" how to manipulate raw blocks of memory without any sort of OO interface per se when the code structure and extensibility requirements don't warrant all the extra overhead associated with setting one up. > C, in my oppinion, would be somewhat like C++, except for larger > problems it doesn't fare qute as well (not poorly by any means, just not > as well). For CERTAIN larger problems, you could be right. I do consider things like writing operating systems and compilers to be "larger problems" though, and C seems to do very well here;-) Your arguments from last time were very compelling, as I said. > The only very very large problem with C++ is, that almost no people know > the language. There is a mountain of absolutely crap learning material > out there. This is why you see examples of "C versus C++" where the C++ > code is several times larger or even less efficient than the C example, > because the author felt that shovelling bad OO and braindead design > after the problem seemed like a good idea. Again, I think that is because most C++ programmers learn C++ >>first<< as part of a course on structured programming, which teaches them to use C++ the way it was "intended" to be used. If one learns C first, on the other hand, you already KNOW how to write tight code and can freely switch over to using C++ constructs where they make sense. In fact, since you actually understand what a struct IS and how data allocation WORKS (a thing that, believe it or not, is largely hidden from the student in most C++ classrooms as this is precisely the under-the-hood stuff considered anathema in a world where programmers are expected to be commodity items and write commodity --interchangeable -- code) you can probably even do very clever end runs around most of the silliness that they teach as good practice whenever and whereever it suits you. > I believe that with the state of compilers today, nobody should have to > start a large project in C - except if it absolutely needs to run on a > platform for which no decent C++ compiler is available (maybe Novell > NetWare - but that's the only such platform that comes to mind...) Give me a few days paring C++ down to the bare minimum extension of C so that it is a pure superset (so ALL C CODE just plain compiles perfectly and even produces the same runfile, so that the I/O language "extensions" are moved off into a library that you can link or not, mostly not, and so that using things like classes and protection and inheritance are suddenly pure and transparent extensions of the basic ideas of structs and unions and typedefs, and I wouldn't even argue. In fact, I'd say that the argument itself becomes silly and moot. One wouldn't even have to change Makefiles, as C++ would just be a feature of gcc that is automatically processed when and where it is encountered. So the kernel would suddenly be written in "C++", sort of, and there would be a very low barrier to converting C into C++ because basically, there would be no conversion at all, only extension. This of course will never happen. Until it does, then an excellent reason to choose C instead of C++ is if you happen to know C very well but not know (or care for) C++ as well. Or if your project is derived from a code base already written in C, and you don't feel like going back and fixing everything to make a C++ compiler happy with it (especially when doing so might well "break" it so that a C compiler is no longer happy with it). Note that this isn't a "theoretical" argument about should -- it is a real-world argument on why the choice you describe is NOT made, over and over again, by most people writing code (judging strictly on the basis of the number of GPL projects written in C vs C++). Even in a world where most schools have been TEACHING C++ for close to a decade, the majority of people who become computer scientists and systems programmers and to work on the guts of computer systems underneath the hood seem to at some point crossover to C and stay there, while the majority of people who stick with C++ end up programming for Windows for some big corporation (which might well be Microsoft) -- and produce shitty, broken, huge, inefficient code as often as not. My next door neighbor is an interesting example that comes to mind. He is a professional programmer. A decade ago he taught CPS at Duke, but got irritated because he had to teach the students to program in C++, and in such a way that they never learned how data structures really work. Believe it or not, I've had to actually teach SEVERAL students who have FINISHED the intro computer courses here just how memory on a computer works -- it is deliberately taught in such a way that one DOESN'T learn that, one learns instead to "think" only about the compiler-provided memory schema. He once spent a whole afternoon in my yard ranting at me about how a struct or union was all the object support any programmer could ever need. So he quit and took over Scholastic Books software division, made a modest fortune, and STILL lives next door writing software and clipping coupons. Other faculty I know seem to view C++ the same way -- a good thing to use to teach students because it forces them to code a certain way, but THEY don't use it for THEIR research projects. Professionals seem more often than not to prefer having the entire palette and all the brushes, and don't like even MAKING color by numbers codelets. This is very definitely not exhaustive or definitive. I also know students where they exact opposite is true -- they learned C++, learned about data structs and malloc and C, and continue to code in C++ (as you do) with strictly enriched abilities. Of course they are (like you;-) Very Bright Guys...and could probably even write good Fortran code if they ever learned it. No, wait, that is an oxymoron -- couldn't happen. > Seriously though, I think that the language-flamewars are fun and > informative, since so much happens in the space of compilers and > real-world projects out there. So, I think it's useful to get an update > every now and then, from people who have strong feelings about their > langauges - oh, and a discussion in the form of a friendly flame-fest is > always good fun too ;) > > / jakob I agree, and hope you realize that I'm deliberately overarguing the case for C above for precisely that reason. I really do believe that it is as much a matter of taste and your particular experience and application space as anything else and wouldn't be surprised if some java coder DOES speak up for java as that's what THEY are really good at and they've discovered some feature that makes it all worthwhile. We've already heard from the python crowd, after all, and THAT'S a language where they even eschew such obviously useful constructs as {}'s and line terminators such as ; in favor of indentation as a matter of deliberate design/dogma. It really does come down in the end to religious belief/personal taste. In the best spirit of tongue-in-cheek, I cannot do better than end with: http://www.phy.duke.edu/~rgb/Beowulf/c++_interview/c++_interview.html Read it and weep...:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 21 12:24:50 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 21 Jan 2004 09:24:50 -0800 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040121164418Z.hanzl@unknown-domain> References: <20040120224741.GA81067@piskorski.com> <200401201701.i0KH1Oh22025@NewBlue.scyld.com> <20040120224741.GA81067@piskorski.com> Message-ID: <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> At 04:44 PM 1/21/2004 +0100, you wrote: >- if you ever dreamt about goodies from LISP without having to deal > with Lots of Silly Idiotic Parenthesis, these dreams come true in > tcl Surely you ment Lots of Idiotic Silly Parenthesis (after all, isn't LISP an acronym) And, pulling some old Intel i/PSC documentation off the shelf (what is an Intel i/PSC but the ancestor of a modern Beowulf... 8 MHz 80286 CPU (with an optional 80287), Ethernet for interprocessor comms (the venerable 82586), Xenix as the base OS).. I find a whole raft of stuff advocating CCLISP for all sorts of things: battle management, dining philosophers problems, optimizing loads of livestock and produce across a river in a small boat, all using stuff like "message passing", node streams, etc. Even such useful stuff as "Sorting a List of N elements" using things like the Sample sort of Felton, Karlin, and Otto. Lest we get too enamored with CCLISP, there's also: (roll of drums) "Concurrent Prolog" (Crash of cymbals, choir of angels) Having not actually looked at anything in Prolog for the last 15+ years, I thought it would be interesting to repeat a short program here. (I have no idea if it would work, since I don't recall CP Syntax) quicksort([X|Xs],Ys):- partition (Xs?,X,Smaller,Larger), quicksort (Smaller?,Ss), quicksort (Larger?,Ls), append(Ss>,[X|Ls?],Ys). quicksort([],[]). partition([Y|ln],X,[Y|Smaller],Larger):- X>=Y|partition(ln?,X,Smaller,Larger). partition([Y|ln],X,Smaller,[Y|Larger]):- X indicates communications name? => readonly variable, process waits until there is a value in the name before proceeding (presumably this is some sort of barrier sync?) ---------- In all of this, they make the very appropriate comments along the lines of: sure it's horribly inefficient, but, you've got: a) tons of computer power potentially available (I mean, you could have 128 of those 8 MHz 286's spinning away) b) you've got a moderately transparent way to distribute the load across those processors without worrying too much about subtlety c) human hours expensive|machine hours cheap d) The computers WILL get faster, so don't agonize about CPU resources too much.. time and the market will improve performance per dollar or wall clock hour much faster than toiling over a hot keyboard ekeing out algorithm tuning improvements. Spend your time on better high level algorithms to solve "your problem". Hey, this was in 1986,1987 or so.. In the last 15 years, I don't think much has really changed, philosophically. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From saravalli.simone at student.unife.it Wed Jan 21 14:06:20 2004 From: saravalli.simone at student.unife.it (Simone Saravalli) Date: Wed, 21 Jan 2004 20:06:20 +0100 Subject: [Beowulf] my first time Message-ID: <001601c3e051$a87182d0$68221997@simc942uomw5a0> Hi to everybody, I'm an italian student in computer science and this is the first time that I write on this mailing list. I hope that you will help this poor newbie with his first experiences with clusters. Thank you and excuse me if my english isn't so good. Simone Saravalli _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Wed Jan 21 17:31:13 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed, 21 Jan 2004 14:31:13 -0800 Subject: apropos of language wars wsa Re: [Beowulf] Which is better In-Reply-To: References: <20040120235009.GD30005@unthought.net> Message-ID: <5.2.0.9.2.20040121141457.0305ad88@mailhost4.jpl.nasa.gov> At 11:46 AM 1/21/2004 -0500, Robert G. Brown wrote: < a whole pile of stuff in response to Jakob's whole pile of stuff, all quite interesting, but irrelevant to my comments inserted below> > So, any language that provides easy direct access to those C APIs have a > > chance of being "the one true language". > >You miss my point entirely. The argument is that C is a thin veneer on >top of assembler -- so thin that one CAN write an operating system in >it. Imagine writing an operating system in LISP. Wait, don't do that. >The results are too horrible to imagine. Imagine doing it on top of >fortran, instead. That's bad, but you won't have nightmares for more >than a week or two afterwards. (IIRC, somebody actually did this once.) I should imagine that more than one person has written what amounts to an operating system (or, at least the kernel, meaning file system and scheduler/task management) in FORTRAN or other less than ideal languages. Certainly I have done both (although, thankfully, not in the same system!..).. (Gosh, I had to write a good part of an OS in MIXAL as an assignment in college, something I wouldn't wish on an enemy) Many, many laboratory/instrument control kinds of applications have been written in Basic or FORTRAN, and they tend to have all the sorts of things one expects in a scheduler (variable rate events, threads, timed execution, priorities, etc.), although, granted, with a somewhat adhoc arrangement of interfaces and so forth (and with varying degrees of independent contexts (ah, the memories of using Blank COMMON, and implementing linked lists and heaps in FORTRAN arrays). You use what's available. Some of the low level BASIC's out there for microcontrollers like the BASICStamp are quite close to a "formatted assembler". I'd also point you to ESPOL, which was the native language of Burroughs mainframes in the 70's. The OS (Master Control Program, MCP, in my recollection) was written in (only) ESPOL. It's a variant of Algol, uses polish notation for arithmetic, etc. There was no "assembler" per se for these machines (although, there must have been some binary equivalent for the ESPOL statements. In any case, the machine was designed to be programmed in high level languages, and didn't leave such essential things like array bounds checking, linked list manipuation, or memory allocation to some hacker flinging out assembler in a library. Nope, it's in the CPU as a native opcode. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From karthik at siptech.co.in Wed Jan 21 00:26:42 2004 From: karthik at siptech.co.in (Karthik) Date: Wed, 21 Jan 2004 10:56:42 +0530 Subject: [Beowulf] Re: [Ilugc] LAM/MPI Setup In-Reply-To: <20040120181559.73844.qmail@web12703.mail.yahoo.com> References: <20040120181559.73844.qmail@web12703.mail.yahoo.com> Message-ID: <200401211056.43276.karthik@siptech.co.in> I have had a successful LAM/MPI cluster some time back. Here are the steps that I have figured out after many unsuccessful attempts. These steps are my quickest way to have a LAM/MPI cluster. 1.Install Linux in the Master node 2.Set Hostname and domainname 3.Setup the Master node as NIS server 4.Export /usr, /home directories. 5.Add a user for cluster, say kluster. 6.Update the NIS maps. 7.Install lam-mpi in Master under /usr/local 8.Install NPB in /usr/npb 9.Check the path for mpicc and lam* executables. 10.In Master: login as kluster. 11.Create a text file lamhosts in ~/ with: LMIT3 #Master Node Node1 #Slave Node 12.Install Linux in Slave node node1. 13.Make slave to mount /home, /usr with NFS. 14.Setup NIS ypbind in node1. 15.Check whether NIS is OK by using ypcat. 16.In node1: Login as kluster. 17.Type recon ?v lamhosts in master to check whether the cluster is bootable. 18.Type lamboot ?v lamhosts in Master to start lam on the cluster. 19.From Master, type tping ?c5 N to ping the entire cluster (use this to check the cluster is OK). 20.In Master: mpirun N progname or mpirun ?lamd N progname to run the progname on all nodes. The second command used lamd to run the program. On Tuesday 20 January 2004 23:45, 70uf33q Hu5541n wrote: > hi all, > > Have been trying (unsuccessfully) to run LAM/MPI on my > 2 node network. But when I run lamboot, it says > Permission Denied. > > the and the network is configured properly. > > Also the problem is with ssh. Earlier I had the same > problem with rsh. I sorted it out by adding: > > in.rshd 192.168.0. and in.rlogind 192.168.0. > > although I have also added sshd 192.168.0. to the > /etc/hosts.allow file I get the ssh error. > > I got the above from Google, if anyone has a > better solution please help. > > Thank You > > Toufeeq > > > ===== > "Love is control,I'll die if I let go > I will only let you breathe > My air that you receive > Then we'll see if I let you love me." > -James Hetfield > All Within My Hands,St.Anger > Metallica > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > To unsubscribe, email ilugc-request at aero.iitm.ernet.in with > "unsubscribe address" > in the subject or body of the message. > http://www.aero.iitm.ernet.in/mailman/listinfo/ilugc > > . -- Karthik M System Administrator - Infrastructure Team SIP Technologies & Exports Limited, G4, Elnet Software City, CPT Road, Taramani, Chennai 600 113, INDIA. Phone: +91-44-22541401 +91-44-22541031 Fax: +91-44-22541475 Website: http://www.siptech.com . _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From topa_007 at yahoo.com Wed Jan 21 02:23:02 2004 From: topa_007 at yahoo.com (70uf33q Hu5541n) Date: Tue, 20 Jan 2004 23:23:02 -0800 (PST) Subject: [Beowulf] Re: [Ilugc] LAM/MPI Setup In-Reply-To: <200401211056.43276.karthik@siptech.co.in> Message-ID: <20040121072302.97063.qmail@web12702.mail.yahoo.com> hi karthick, will try out your solution. anyway the error I got was ssh_exchange_identification : Coneection refused by host. I edited my /etc/sshd/sshd_config entries to "no" where all Authentication was needed.Still same problem need help badly. Toufeeq --- Karthik wrote: > I have had a successful LAM/MPI cluster some time > back. Here are the steps > that I have figured out after many unsuccessful > attempts. These steps are my > quickest way to have a LAM/MPI cluster. > > 1.Install Linux in the Master node > 2.Set Hostname and domainname > 3.Setup the Master node as NIS server > 4.Export /usr, /home directories. > 5.Add a user for cluster, say kluster. > 6.Update the NIS maps. > 7.Install lam-mpi in Master under /usr/local > 8.Install NPB in /usr/npb > 9.Check the path for mpicc and lam* executables. > 10.In Master: login as kluster. > 11.Create a text file lamhosts in ~/ with: > LMIT3 #Master Node > Node1 #Slave Node > 12.Install Linux in Slave node node1. > 13.Make slave to mount /home, /usr with NFS. > 14.Setup NIS ypbind in node1. > 15.Check whether NIS is OK by using ypcat. > 16.In node1: Login as kluster. > 17.Type recon ???v lamhosts in master to check > whether the cluster is bootable. > 18.Type lamboot ???v lamhosts in Master to start lam > on the cluster. > 19.From Master, type tping ???c5 N to ping the > entire cluster (use this to check > the cluster is OK). > 20.In Master: mpirun N progname or mpirun ???lamd N > progname to run the progname > on all nodes. The second command used lamd to run > the program. > > > > On Tuesday 20 January 2004 23:45, 70uf33q Hu5541n > wrote: > > hi all, > > > > Have been trying (unsuccessfully) to run LAM/MPI > on my > > 2 node network. But when I run lamboot, it says > > Permission Denied. > > > > the and the network is configured > properly. > > > > Also the problem is with ssh. Earlier I had the > same > > problem with rsh. I sorted it out by adding: > > > > in.rshd 192.168.0. and in.rlogind 192.168.0. > > > > although I have also added sshd 192.168.0. to the > > /etc/hosts.allow file I get the ssh error. > > > > I got the above from Google, if anyone has a > > better solution please help. > > > > Thank You > > > > Toufeeq > > > > > > ===== > > "Love is control,I'll die if I let go > > I will only let you breathe > > My air that you receive > > Then we'll see if I let you love me." > > -James Hetfield > > All Within My Hands,St.Anger > > Metallica > > > > __________________________________ > > Do you Yahoo!? > > Yahoo! Hotjobs: Enter the "Signing Bonus" > Sweepstakes > > http://hotjobs.sweepstakes.yahoo.com/signingbonus > > _______________________________________________ > > To unsubscribe, email > ilugc-request at aero.iitm.ernet.in with > > "unsubscribe address" > > in the subject or body of the message. > > > http://www.aero.iitm.ernet.in/mailman/listinfo/ilugc > > > > . > > -- > Karthik M > System Administrator - Infrastructure Team > > SIP Technologies & Exports Limited, > G4, Elnet Software City, CPT Road, > Taramani, Chennai 600 113, > INDIA. > > Phone: +91-44-22541401 > +91-44-22541031 > Fax: +91-44-22541475 > > Website: http://www.siptech.com > > > . > > _______________________________________________ > To unsubscribe, email > ilugc-request at aero.iitm.ernet.in with > "unsubscribe address" > in the subject or body of the message. > http://www.aero.iitm.ernet.in/mailman/listinfo/ilugc ===== "Love is control,I'll die if I let go I will only let you breathe My air that you receive Then we'll see if I let you love me." -James Hetfield All Within My Hands,St.Anger Metallica __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raiders at phreaker.net Tue Jan 20 10:06:07 2004 From: raiders at phreaker.net (Shawn Raider) Date: Tue, 20 Jan 2004 23:06:07 +0800 Subject: [Beowulf] SGE vs LSF vs PBS... Message-ID: <200401202306.07794.raiders@phreaker.net> I am expanding an .edu cluster of Xeons, adding more racks. The older racks run LSF alongside all the unix boxes in the campus and the admin staff really love LSF, esp interactive logins. But we can't afford that kind of license pricing. I would be thankful if you could share your experiences in replacing LSF with SGE. I did study features etc from the respective docs/sites etc. before asking here. I am looking specifically for user experiences. Thanks, Shawn _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From afant at geekmail.cc Tue Jan 20 11:57:29 2004 From: afant at geekmail.cc (Andrew Fant) Date: Tue, 20 Jan 2004 11:57:29 -0500 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <400D3833.F1D97ABB@correu.andorra.ad> References: <400D3833.F1D97ABB@correu.andorra.ad> Message-ID: <82620000.1074617849@flux.usg.tufts.edu> Right on, Alan. Lately, I have come to think that python is a "good thing (tm)". I attended the two python tutorials at SC2003 last November, and I was very impressed with the extensions that have been made. SciPy now allows direct calls to the Atlas blas implementation from python, and offers a slew of optimized c routines as well for numerical work in python. f2py provides a mostly automatic way to wrap old fortran code so it can be loaded as a python module. Perhaps most exciting to me is the development work going on with MPI bindings in python. While I do not think python will overtake optimized C and Fortran for the bleeding-edge of scientific computing, I think that it has the potential to capture a lot of the middle ground. For teaching numerical simulations and parallel computing, it reduces the detail of the C API to a level that a part-time developer can keep straight while they are worrying about the science behind the code or that a new user can start using rapidly. Andy --On Tuesday, January 20, 2004 15:16:19 +0100 Alan Ward i Koeck wrote: > Well, my own preference goes to Python, which > > - has a syntax I prefer a lot over C > - while being an interpreted language, manages to be reasonably speedy, > unlike Java > - has a fair amount of predefined library objects you can use to access > C > POSIX system routines > > but I do recognize that low-level stuff definately should be > coded in C. Now, as to just what "low-level" means ... > > Best regards, > Alan Ward > > > "Robert G. Brown" wrote: > (snip) >> >> Let it be Jihad, then. >> >> The answer technically is "it depends". This is to make the vast horde >> of java programmers feel loved, and to acknowledge that for writing web >> applications their primary scripting language (like perl, python and >> many another before it) has a purpose in God's world. >> >> There. Let us bask for a moment in the serenity of our knowledge that >> we have the complete freedom to choose, and that there are no wrong >> answers. >> >> Now we can give the correct answer, which is "C". > (snip) >> -- >> Robert G. Brown http://www.phy.duke.edu/~rgb/ >> Duke University Dept. of Physics, Box 90305 >> Durham, N.C. 27708-0305 >> Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu >> >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From victor_ms at brturbo.com Wed Jan 21 14:38:34 2004 From: victor_ms at brturbo.com (victor_ms at brturbo.com) Date: Wed, 21 Jan 2004 17:38:34 -0200 (BRST) Subject: [Beowulf] About Linpack cluster Message-ID: <25760380.1074713914199.JavaMail.nobody@webmail1.brturbo.com> Hello everbody. I need some help. I have 17 Linux cluster Nodes with Mosix and MPI and I need some help on Linpack software. Anyone where knows where can I download some documents about Linpack? I have the Linpack Software install on my machine, but when I start it, some errors appear so, I need some documents about Linpack. Best Resguards. From camm at enhanced.com Wed Jan 21 19:14:34 2004 From: camm at enhanced.com (Camm Maguire) Date: 21 Jan 2004 19:14:34 -0500 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> References: <20040120224741.GA81067@piskorski.com> <200401201701.i0KH1Oh22025@NewBlue.scyld.com> <20040120224741.GA81067@piskorski.com> <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> Message-ID: <54oeswalg5.fsf@intech19.enhanced.com> Greetings! Jim Lux writes: > At 04:44 PM 1/21/2004 +0100, you wrote: > > > >- if you ever dreamt about goodies from LISP without having to deal > > with Lots of Silly Idiotic Parenthesis, these dreams come true in > > tcl > > Surely you ment Lots of Idiotic Silly Parenthesis (after all, isn't > LISP an acronym) > While I certainly agree that nothing comes close at present to C/Fortran for parallel numerical work, I do wonder what it would take to get LISP to be the best 'higher order' language alternative for this task. After all, 1) it has an *open* standard, 2) it compiles to native machine code, 3) speeds often within 20% of C, and 4) it can easily call libraries written in other languages. In fact, GCL already has a binding to MPI, and there has been discussion about using LISP's powerful macro system to automatically insert blas calls in place of written out floating point matrix code, and then relying on an optimized blas setup as in Debian, where several atlas packages transparently provide accelerated blas without recompilation, to get decent performance. The downside of course is that LISP has lost a lot of mindshare. Take care, -- Camm Maguire camm at enhanced.com ========================================================================== "The earth is but one country, and mankind its citizens." -- Baha'u'llah _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From webhat at xs4all.nl Wed Jan 21 06:29:28 2004 From: webhat at xs4all.nl (=?unknown-8bit?B?RGFuaetsIFcu?= Crompton) Date: Wed, 21 Jan 2004 12:29:28 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <20040120235009.GD30005@unthought.net> References: <20040120065449.33171.qmail@web21507.mail.yahoo.com> <20040120235009.GD30005@unthought.net> Message-ID: <20040121112928.GA8296@troy.rphh.org> On Wed, Jan 21, 2004 at 12:50:10AM +0100, Jakob Oestergaard wrote: > On Tue, Jan 20, 2004 at 08:05:45AM -0500, Robert G. Brown wrote: > > On Mon, 19 Jan 2004, prakash borade wrote: < ... snipped for brevity ... > > > Now we can give the correct answer, which is "C". > Typing a little fast there, I think... The correct answer for anything > larger than 1KLOC is "C++" - of course, you knew that, you were just a > little fast on the keyboard ;) While I can find litte fault with your posts I should note that there are a number of points which I would like to mention. Firstly, obviously YHBT YHL HAND. Now I've got that out of the way, there was a light weight comparative benchmark[1] on OSNews. Obviously, as you will see, they only tested some simple things, but the tests showed Sun's Java performed quite well when compaired to GCC. Compairing it to GCC is a little cheap though, as we know from the benchmarks in October edition of Dr Dobbs Journal[2]: GCC, it may have the best ANSI C/C++ support produces the bulkiest code and, although not always, slower code that a processor specific design such as Intel's C++ compiler. The have been a lot of myths[3] floating around about Java performance, and although IMO floating point precision sucks in Java, it is not as bad as people think. The main reason that I vote for Java and not my favorite C or my second favorite C++ is that the person who asks this question will just be adding more bad/exploitable C/C++ network code to the world. And I don't really think we should be encouraging that. [1] Nine Language Performance Round-up: Benchmark Math & File I/O - C.W. Cowell-Shah http://osnews.com/story.php?news_id=5602 [2] Compairing C/C++ Compilers - M. Wilson http://www.ddj.com/articles/2003/0310/ [3] Urban Performance Myths - B. Goetz http://www-106.ibm.com/developerworks/java/library/j-jtp04223.html?ca=dgr-lnxw01JavaUrbanLegends yours, Daniel (ever the debunker) blaze your trail -- redhat 'I am become Shiva, destroyer of worlds' _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 21 22:28:36 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 21 Jan 2004 22:28:36 -0500 (EST) Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: <5.2.0.9.2.20040121085813.018da6d8@mailhost4.jpl.nasa.gov> Message-ID: > > Lest we get too enamored with CCLISP, there's also: > (roll of drums) > "Concurrent Prolog" > (Crash of cymbals, choir of angels) > > Having not actually looked at anything in Prolog for the last 15+ years, I > thought it would be interesting to repeat a short program here. (I have no > idea if it would work, since I don't recall CP Syntax) > > quicksort([X|Xs],Ys):- > partition (Xs?,X,Smaller,Larger), > quicksort (Smaller?,Ss), > quicksort (Larger?,Ls), > append(Ss>,[X|Ls?],Ys). > quicksort([],[]). > > partition([Y|ln],X,[Y|Smaller],Larger):- > X>=Y|partition(ln?,X,Smaller,Larger). > partition([Y|ln],X,Smaller,[Y|Larger]):- > X partition([],X,[],[]). > > append([X|Xs],Ys,[X|Zs]):- > append(Xs?,Ys,Zs). > append([],Xs,Xs). > > There is a scribbled note (not in my handwriting) that mentions the following: > - => indicates communications > name? => readonly variable, process waits until there is a value in the > name before proceeding (presumably this is some sort of barrier sync?) I had actually helped write a parallel version of Prolog that would automatically execute the standard Prolog quicksort in parallel. It had a very interesting run-time system that automatically load balanced the parallelization - at least I thought it was cool in a "self organizing Sante Fe Institute kind of way". There was (and still is to some degree) a good argument for such languages. For those who do not understand (or hate) Prolog, it operates by interpreting the program using a "resolution mechanism". The resolution mechanism is total transparent to the user. So if you parallelize the resolution mechanism, you have parallelized all Prolog programs (which does not necessarily mean they will run faster by the way). Indeed, it is even possible (and quite simple) to write meta-interpreters in Prolog to interpret Prolog in parallel - and thus play with the parallel resolution mechanism. The runtime performance issue has also been solved so that Prolog programs can run quite fast. The trouble is that it is very difficult for most programmers to get their mind around Prolog, but once you do, it can provide a very expressive and concise language in which to program (particularly database stuff). There is a gnu Prolog compiler that works rather well (plus a few other free versions as well). Alas, most discussions of "high level" approaches are not well received these days. Seems there is certain fashion to computer languages. Finally, if you are interested, some old papers on this subject can found at http://www.clusterworld.com/deadline.html. Doug > > ---------- > > In all of this, they make the very appropriate comments along the lines of: > sure it's horribly inefficient, but, you've got: > a) tons of computer power potentially available (I mean, you could have 128 > of those 8 MHz 286's spinning away) > b) you've got a moderately transparent way to distribute the load across > those processors without worrying too much about subtlety > c) human hours expensive|machine hours cheap > d) The computers WILL get faster, so don't agonize about CPU resources too > much.. time and the market will improve performance per dollar or wall > clock hour much faster than toiling over a hot keyboard ekeing out > algorithm tuning improvements. Spend your time on better high level > algorithms to solve "your problem". > > Hey, this was in 1986,1987 or so.. In the last 15 years, I don't think much > has really changed, philosophically. > > > > > > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine Desk: 610.865.6061 Cell: 610.390.7765 Redefining High Performance Computing Fax: 610.865.6618 www.clusterworld.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Jan 22 00:42:07 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 22 Jan 2004 13:42:07 +0800 (CST) Subject: [Beowulf] SGE vs LSF vs PBS... In-Reply-To: <200401202306.07794.raiders@phreaker.net> Message-ID: <20040122054207.141.qmail@web16807.mail.tpe.yahoo.com> I did not buy LSF, but I did read the manuals of it. I compared the features of LSF and SGE, found that most of them are the same. One thing you need to understand is that if you want to have advanced scheduling algorithms, you need to use SGEEE. SGEEE is also free, but I remember there is a switch to turn on _during installation_ to get SGEEE, you can send a mail to the SGE mailing list. Another big difference is that LSF has global queues, but SGE's queue concept is different -- a queue is just a "container" of jobs. So the configuration may be different than your LSF cluster in some cases. HOWEVER, SGE 6.0 will provide LSF like queues to help people to migrate more easily, and it is coming soon too :) Since how easy or difficult it is to migrate to SGE depends on what you are looking for, the best is still to provide more info, but I think the beowulf list is not the best place to discuss about your batch system requirements. The SGE mailing list is a better place, as there are a lot of people there who used LSF but switched to SGE, like the author of GE monitor: http://www.veus.hr/linux/gemonitor.html SGE mailing list: http://gridengine.sunsource.net/project/gridengine/maillist.html Also, don't forget to find out more about SPBS. I don't have much experiece with SPBS, may be others who used LSF and SPBS can provide more info. Andrew. --- Shawn Raider ???? > I would be thankful if you could share your > experiences in replacing LSF with > SGE. I did study features etc from the respective > docs/sites etc. before > asking here. I am looking specifically for user > experiences. > > Thanks, > Shawn ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Thu Jan 22 00:49:12 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Thu, 22 Jan 2004 13:49:12 +0800 (CST) Subject: [Beowulf] Fwd: Bio-cluster Linux distribution (BioBrew announcement) Message-ID: <20040122054912.82737.qmail@web16813.mail.tpe.yahoo.com> based on Rocks cluster. Andrew. --- Glen Otero wrote: > Hi folks- > > I just wanted to announce the release of BioBrew v. 1.0 for the x86 > architecture. The three CD iso images are freely available for > download > at > http://bioinformatics.org/ftp/biobrew, a Bioinformatics.org sponsored > > and > hosted website. > > BioBrew is an open source Linux cluster distribution based on the > NPACI > Rocks (www.rocksclusters.org) cluster distribution and enhanced for > bioinformaticists and life scientists. While BioBrew looks, feels, > and > operates like ordinary Red Hat Linux, it includes popular open source > clustering and bioinformatics software e.g. the NCBI toolkit, BLAST, > mpiBLAST, HMMER, ClustalW, GROMACS, PHYLIP, WISE, FASTA, and EMBOSS. > Installing BioBrew is a simple, straightforward procedure that allows > Linux beginners and gurus alike to go from bare metal to cluster in > the > shortest time possible. > > I used this version of BioBrew to build this Infiniband cluster for > Intel > and Mellanox for Supercomputing 2003: > http://www.hpcwire.com/hpcwire/hpcwireWWW/03/1114/106388.html. > > BioBrew v. 1.0 has the following features: > > - GM 1.6 and 2.0 support for Myrinet > - Infiniband support (binary drivers only) > - Automated compute node installation > - RPM software packaging and installation methods > - Cluster management tools > - MySQL database for storing cluster node information > - Cluster status monitoring with Ganglia > - SGE batch processing and job scheduler software > - Message passing interface software (MPICH & LAM) > - Parallel virtual machine (PVM) > - Parallel virtual filesystem support (PVFS) > - Virtual Machine Interface (VMI) 2.0. > - MPICH built for VMI > > The big improvement for 1.0 is the inclusion of VMI. I can't say > enough > about VMI. I encourage everybody that uses MPICH over multiple > interconnects to read the VMI docs I've posted. Basically, by > virtualizing > all interconnects, VMI allows you to compile your application once > against > a standard version of MPICH (mpich-vmi2), and run it over any > interconnnect. VMI supports ethernet, myrinet, and Infiniband. > > The README and installation pictorial provide sufficient installation > instructions. I'll be posting the complete User's Guide in a week or > so. > > BioBrew 3.0 for x86, ia64, and x86_64 is in early beta, but will be > posted > in mid to late February. > > Biobrew-Discuss at bioinformatics.org is an email discussion list for > all > things BioBrew. You can subscribe to the list at > http://bioinformatics.org/lists/biobrew-discuss. > > Glen > > (Sorry about any double posting) > > > Glen Otero, Ph.D. > Linux Prophet > 619.917.1772 > > _______________________________________________ > Bioclusters maillist - Bioclusters at bioinformatics.org > https://bioinformatics.org/mailman/listinfo/bioclusters ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From brian.dobbins at yale.edu Thu Jan 22 11:57:02 2004 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Thu, 22 Jan 2004 11:57:02 -0500 (EST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: Hi Brent, A while back, I had done a portion of what you were looking for, just to make my job easier while testing new kernels on our nodes. We had a small cluster connected to an APC masterswitch which could reboot the nodes (by powering them off, then on again). The nodes themselves were diskless, and the process was: a) If a newer image was located on the file server, reboot with the new image b) If we don't get a status OK from the node after 5 minutes (done via a simple file write in rc.local in the ramdisk), revert back to the old image. (This was all done on a fairly small scale, mind you.) For us, with diskless nodes, this was very easy - we didn't have to modify GRUB's boot parameters at all. I'm not even remotely knowledgeable about GRUB's options, but perhaps there's an easy way to do configure it to boot a specified kernel just once, then revert back to a 'safe' one. (You can do this in startup, if you're guaranteed startup will succeed, but that's not likely in your situation!) If there's a chance you can do this disklessly in terms of booting, let me know, I can see if I have any of the old configuration files still around. This doesn't mean you can't use disks, but with RAM so cheap these days, booting from a ramdisk doesn't cost you much, and makes some things a lot easier. You can always mount the disk during init. (Ie, you'd have the script called from the job manager execute a 'mkelf-linux' or 'mknbi' binary with a standard ramdisk, and the only thing that changes is the bzImage file supplied to the command. From memory, something like:) mkelf-linux --append="root=/dev/ram" ${bzImage_custom} RamDisk.gz mem=xG > /tftpboot/${user}_${date}_${bzImage_custom}.elf Unfortunately, I can't help you on the job manager stuff, but I'm sure there are plenty of people here who know heaps more and will certainly chime in. Good luck! - Brian Brian Dobbins Yale University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Thu Jan 22 10:29:09 2004 From: bclem at rice.edu (Brent M. Clements) Date: Thu, 22 Jan 2004 09:29:09 -0600 (CST) Subject: [Beowulf] Hypothetical Situation Message-ID: We have a request by some of our research scientists to form a beowulf cluster that provides the following abilities: 1. It must be part of a shared computing facility controlled by a batch queuing system. 2. A normal user must be able to compile a customized kernel, then submit a job with a variable pointing to that kernel. The batch queuing system must then load that kernel onto the allocated nodes and reboot the nodes allocated. 2a. If the node doesn't come back up after rebooting, the job must be canceled and the node rebuilt automatically with a stable kernel/image. 3. When the job is finished, the node must be rebuilt automatically using a stable kernel/image. The admin's here have come up with a name for this: "user provisioned os imaging". Our gut feeling is that this can be done but it will be a highly customized system. What is everyone's thoughts, experiences, etc etc concerning doing something like this? Thanks, Brent Clements Linux Technology Specialist Information Technology Rice University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Thu Jan 22 12:25:31 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Thu, 22 Jan 2004 09:25:31 -0800 Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> At 09:29 AM 1/22/2004 -0600, Brent M. Clements wrote: >We have a request by some of our research scientists to form a beowulf >cluster that provides the following abilities: > >1. It must be part of a shared computing facility controlled by a batch >queuing system. >2. A normal user must be able to compile a customized kernel, then submit >a job with a variable pointing to that kernel. The batch queuing system >must then load that kernel onto the allocated nodes and reboot the nodes >allocated. >2a. If the node doesn't come back up after rebooting, the job must be >canceled and the node rebuilt automatically with a stable kernel/image. >3. When the job is finished, the node must be rebuilt automatically using >a stable kernel/image. > >The admin's here have come up with a name for this: "user >provisioned os imaging". Our gut feeling is that this can be done but it >will be a highly customized system. What is everyone's thoughts, >experiences, etc etc concerning doing something like this? I think this cries out for diskless nodes!! The only moderately tricky part will be the mangement of the cluster so that you can feed off the right images when the node reboots and asks for its image. Some fairly simple database table of MAC, Node ID, image, etc. would probably take care of it. Obviously, you'll need some sort of remote controlled booting (power cycle?). Possibly, you might want to have centrally managed network switches so you can isolate the subclusters, in case someone builds a kernel that runs amok. You'll need to spend a few work weeks/months building some nice quasi automated scripts and documentation to help the folks build their kernels and put them into an appropriate form. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Florent.Calvayrac at univ-lemans.fr Thu Jan 22 12:11:15 2004 From: Florent.Calvayrac at univ-lemans.fr (Florent Calvayrac) Date: Thu, 22 Jan 2004 18:11:15 +0100 Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: <40100433.4000903@univ-lemans.fr> Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? > > with PXE and/or bproc, (two kernel monte) I see it easily. If the node doesn"t come up, just send a magic packet to reboot it and download another kernel. -- Florent Calvayrac | Tel : 02 43 83 26 26 Laboratoire de Physique de l'Etat Condense | Fax : 02 43 83 35 18 UMR-CNRS 6087 | http://www.univ-lemans.fr/~fcalvay Universite du Maine-Faculte des Sciences | 72085 Le Mans Cedex 9 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 22 13:00:11 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 22 Jan 2004 13:00:11 -0500 (EST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: On Thu, 22 Jan 2004, Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? See the "COD" (Cluster on Demand) project in the Duke CPS department (Jeff Chase, primarily, along with his student Justin Moore). It is PRECISELY what you are looking for. I mean precisely. I don't know that current status of the project, but that's just what COD is designed for. Justin Moore, BTW, is the ex-protege of this list's own Greg Lindahl, and a very savvy computer guy. http://issg.cs.duke.edu/cod/ >From its summary: Users of a shared cluster should be free to select the software environments that best support their needs. Cluster-on-Demand (COD) is a system to enable rapid, automated, on-the-fly partitioning of a physical cluster into multiple independent virtual clusters. A virtual cluster (vcluster) is a group of machines (physical or virtual) configured for a common purpose, with associated user accounts and storage resources, a user-specified software environment, and a private IP address block and DNS naming domain. COD vclusters are dynamic; their node allotments may change according to demand or resource availability (This includes shipping each node its own custom kernel and filesystem image -- full setup and boot into cluster, followed by teardown and restoration of the resource to the pool when done, "arbitrary" kernels, operating systems, and runtime node images.) I'm very peripherally connected to this project, as in we all talk about it, but I haven't contributed any code, only (maybe) a very few ideas. Justin is probably who you'd want to talk to: http://www.cs.duke.edu/~justin/ He's on this list; maybe he'll speak up if he's around. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From epaulson at cs.wisc.edu Thu Jan 22 11:21:06 2004 From: epaulson at cs.wisc.edu (Erik Paulson) Date: Thu, 22 Jan 2004 10:21:06 -0600 Subject: [Beowulf] Hypothetical Situation In-Reply-To: ; from bclem@rice.edu on Thu, Jan 22, 2004 at 09:29:09AM -0600 References: Message-ID: <20040122102105.A5743@chopin.cs.wisc.edu> On Thu, Jan 22, 2004 at 09:29:09AM -0600, Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? > Auto rebuilding with custom images isn't difficult. The Utah folks probably have the best complete setup for this: http://www.emulab.net/index.php3 Read "An Integrated Experimental Environment for Distributed Systems and Networks" from OSDI 2002, it's a good overview. Their setup is aimed more at building a network-emulation testbed, but it will provision up machines, install the right OS, reboot nodes, muck with switches, etc etc. Tossing a batch system into that image wouldn't be hard (it'd probably be a bit tricky with PBS, since it doesn't like nodes coming and going, Condor would work fine, and I know nothing about LSF or SGE.) -Erik _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jmoyer at redhat.com Thu Jan 22 12:59:14 2004 From: jmoyer at redhat.com (Jeff Moyer) Date: Thu, 22 Jan 2004 12:59:14 -0500 Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: <16400.3954.325319.622862@segfault.boston.redhat.com> ==> Regarding [Beowulf] Hypothetical Situation; "Brent M. Clements" adds: bclem> We have a request by some of our research scientists to form a bclem> beowulf cluster that provides the following abilities: bclem> 1. It must be part of a shared computing facility controlled by a bclem> batch queuing system. 2. A normal user must be able to compile a bclem> customized kernel, then submit a job with a variable pointing to bclem> that kernel. The batch queuing system must then load that kernel bclem> onto the allocated nodes and reboot the nodes allocated. 2a. If the bclem> node doesn't come back up after rebooting, the job must be canceled bclem> and the node rebuilt automatically with a stable kernel/image. bclem> 3. When the job is finished, the node must be rebuilt automatically bclem> using a stable kernel/image. Check out STP. Besides being a pretty good band, it stands for Scalable Test Platform. http://sourceforge.net/projects/stp/ -Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Thu Jan 22 13:58:26 2004 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Thu, 22 Jan 2004 19:58:26 +0100 (CET) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: On Thu, 22 Jan 2004, Brent M. Clements wrote: > 1. It must be part of a shared computing facility controlled by a batch > queuing system. Use the "epilogue" facility to run a script that installs a sane image and then reboots or first reboots and PXE+something (recent thread mentioned what this something can be :-)) takes care of the rest. The trickier part is starting the user compiled kernel, because the "epilogue" of a job would need to know about the requirements of the next job. Making the change in the "prologue" of the user's job might not work properly because the node is rebooted in the process of changing the kernel and this breaks the link to the batch system daemon... > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. I did something similar for a test and after I realized what are the implications I never put it into production, even though I would have allowed only kernels compiled by myself :-) Basically, user compiled kernel means that the kernel can do anything; that means reading the content of /etc/shadow and dumping it into an unprotected file, disturbing the network activity on the switch(es) where it's connected, overloading file-servers, making connectiong from priviledged ports (which can upset the batch system :-)), destroying local file-systems. It can even go to destroying hardware by driving it with out-of-spec parameters. Completely hijacking the node is another thing that can be done; there's no guarantee with a user compiled kernel that the batch system daemons are started, for example, so the master node might loose any control over the node, requiring a human to press a button. Now if these kernels come only from trusted persons, there are still many things that can go wrong. For example, running with a RHEL 3.0 user space and a non-NTPL kernel will break threaded applications. Compiling most (all) drivers in-kernel while user-space expects modules might break the whole system. Not including IP autoconfig when root FS is over NFS will not allow booting. And so on... > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. "doesn't come back up after rebooting" is a pretty vague description. Probably a watchdog (even the software one) would help, but might still not catch all cases. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. If the user compiled kernel plays nice and gives back the node to the batch system... just reboot and instruct PXE to give a sane kernel/image. You might have better chances of keeping things under control using some kind of virtualization technique, UML or VMWare/Bochs. Limits can be imposed on the host system, but performance might go down a lot. One solution that I used at some point and will probably use again in the near future is to have a (small) set of sane kernels that are allowed for such purposes; that would allow for example conflicting drivers (like GM and PM for Myrinet cards) to be used - for each job the kernel that corresponds to the right driver is chosen. This greatly reduces the risks as the kernels are compiled only by the admin who is supposed to know what is doing :-) -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Thu Jan 22 12:30:21 2004 From: john.hearns at clustervision.com (John Hearns) Date: Thu, 22 Jan 2004 18:30:21 +0100 (CET) Subject: [Beowulf] Hypothetical Situation In-Reply-To: Message-ID: On Thu, 22 Jan 2004, Brent M. Clements wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? I've always wondered if there is a place for UML (User Mode Linux) in cluster or grid computing. This might be an interesting area to apply it - though I admit it would take effort. If you are not familiar with the concept, UML allows Linux to be run in a virtual machine. SO the real machine can be running the standard distro. The users get their own machine, and run it under their control. So what they would have to do is cmpile up their own kernel, and UML instance. On a more conventional Beowulf setup, what you would have to do is make a software image with the custom kernel. Reboot the machine, install that image. Run the job, change the image which should be installed on that node then reboot. We support per-node chhsing of the software image to be installed, and I'm sure lots of others do also. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bogdan.costescu at iwr.uni-heidelberg.de Thu Jan 22 14:02:48 2004 From: bogdan.costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Thu, 22 Jan 2004 20:02:48 +0100 (CET) Subject: [Beowulf] Hypothetical Situation In-Reply-To: <40100433.4000903@univ-lemans.fr> Message-ID: On Thu, 22 Jan 2004, Florent Calvayrac wrote: > If the node doesn"t come up, just send a magic packet to reboot it Magic packet only wakes up a computer. There is no magic packet to reboot or turn off (or at least put back to sleep) a computer. Imagine the security consequences... -- Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From laytonjb at comcast.net Thu Jan 22 16:22:02 2004 From: laytonjb at comcast.net (Jeffrey B. Layton) Date: Thu, 22 Jan 2004 16:22:02 -0500 Subject: [Beowulf] Hypothetical Situation In-Reply-To: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> References: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> Message-ID: <40103EFA.6060900@comcast.net> Jim Lux wrote: > At 09:29 AM 1/22/2004 -0600, Brent M. Clements wrote: > >> We have a request by some of our research scientists to form a beowulf >> cluster that provides the following abilities: >> >> 1. It must be part of a shared computing facility controlled by a batch >> queuing system. >> 2. A normal user must be able to compile a customized kernel, then >> submit >> a job with a variable pointing to that kernel. The batch queuing system >> must then load that kernel onto the allocated nodes and reboot the nodes >> allocated. >> 2a. If the node doesn't come back up after rebooting, the job must be >> canceled and the node rebuilt automatically with a stable kernel/image. >> 3. When the job is finished, the node must be rebuilt automatically >> using >> a stable kernel/image. >> >> The admin's here have come up with a name for this: "user >> provisioned os imaging". Our gut feeling is that this can be done but it >> will be a highly customized system. What is everyone's thoughts, >> experiences, etc etc concerning doing something like this? > > > I think this cries out for diskless nodes!! The only moderately tricky > part will be the mangement of the cluster so that you can feed off the > right images when the node reboots and asks for its image. Some fairly > simple database table of MAC, Node ID, image, etc. would probably take > care of it. Obviously, you'll need some sort of remote controlled > booting (power cycle?). Possibly, you might want to have centrally > managed network switches so you can isolate the subclusters, in case > someone builds a kernel that runs amok. I've got just the thing for this - Warewulf. WW can already handle hetero diskless clusters. I'm working on a write-up on how to do. I've tested it on my home cluster and it works fine. Enjoy! Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Thu Jan 22 18:33:12 2004 From: bclem at rice.edu (Brent M. Clements) Date: Thu, 22 Jan 2004 17:33:12 -0600 (CST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: We have actually come up with an idea called a "kernel preload" system based on ideas from alot of places. Basically what we have is as follows: The system admins provide the source code to a bunch of "stock" kernels known to work with the hardware we are using in the cluster. The user takes a copy of this source code and makes his/her changes. We have a script that's called skj(submit kernel job). This is a perl script that does the following 1. Makes a reservation within job scheduling system for 1 hour, reserving what ever number of nodes the user requests. 2. Creates a patch from their modified source code and then builds the modified kernel. Please note that the script will do tests to make sure that the patch file does not remove any needed hardware drivers to ensure that the machine will actually boot. 4. The script makes it a tftp'd kernel and modifies the dhcp file to reflect the kernel changes for the nodes reserved. 5. The machines are power cycled/rebooted 6. The script waits up to 15 minutes for the machines that have the new kernel installed to come back up. If after 15 minutes there are no responses from any of the machines, the script reblasts the system with the stock kernel and outputs an error message. We will actually speed up the boot process by installing linuxbios, so our waittime will be like 5 minutes. 7. But If everything goes well, the script modifies the reservation and creates a pbs job script, or gives them an interactive session to the allocated nodes. Voila, we have solved our issue. This could also work if we wanted to actually reblast an entirely new image onto the allocated nodes rather than just the kernel. Remember our requirements/issue was that we needed this to work easily with our job schedulars(pbs/maui). I am going to be posting a document outlining the requirements,and the solution next week if anyone is interested. Thanks to everyone who helped out! Sincerely, Brent Clements Linux Technology Specialist Information Technology Rice University On Thu, 22 Jan 2004, Mark Hahn wrote: > > You'll need to spend a few work weeks/months building some nice quasi > > automated scripts and documentation to help the folks build their kernels > > and put them into an appropriate form. > > I can't imagine why this would take even a day of work, > assuming you use plain old PXE to boot the nodes, already > have a table of MAC addrs, and are comfortable with simple > suid script programming: > > pushkernel(machinename, kernelfile): > > $mac = grep $machinename mac-table-file > ln -s $kernelfile /var/tftp/pxelinux.cfg/$mac > powercontrol -r $machinename > sleep 180 > if (cant ping $machinename) > ln -s fallbackimage /var/tftp/pxelinux.cfg/$mac > > (powercontrol is a simple perl script I use to talk to my APC power thingies.) > hooking this into a batch system shouldn't be hard. > > physically installing the kernel on a machine and booting it locally > seems pointless. "lilo -R" is great for trying out new kernels, though. > I understand someone has added this feature to grub, but haven't seen it myself. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Thu Jan 22 18:18:05 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Thu, 22 Jan 2004 18:18:05 -0500 (EST) Subject: [Beowulf] Hypothetical Situation In-Reply-To: <5.2.0.9.2.20040122092208.018d90c0@mailhost4.jpl.nasa.gov> Message-ID: > You'll need to spend a few work weeks/months building some nice quasi > automated scripts and documentation to help the folks build their kernels > and put them into an appropriate form. I can't imagine why this would take even a day of work, assuming you use plain old PXE to boot the nodes, already have a table of MAC addrs, and are comfortable with simple suid script programming: pushkernel(machinename, kernelfile): $mac = grep $machinename mac-table-file ln -s $kernelfile /var/tftp/pxelinux.cfg/$mac powercontrol -r $machinename sleep 180 if (cant ping $machinename) ln -s fallbackimage /var/tftp/pxelinux.cfg/$mac (powercontrol is a simple perl script I use to talk to my APC power thingies.) hooking this into a batch system shouldn't be hard. physically installing the kernel on a machine and booting it locally seems pointless. "lilo -R" is great for trying out new kernels, though. I understand someone has added this feature to grub, but haven't seen it myself. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From anand at mecheng.iisc.ernet.in Fri Jan 23 00:10:43 2004 From: anand at mecheng.iisc.ernet.in (Anand TNC) Date: Fri, 23 Jan 2004 10:40:43 +0530 (IST) Subject: [Beowulf] cluster on suse In-Reply-To: <40103EFA.6060900@comcast.net> Message-ID: Hi, I'm new to clustering...does anyone know of some clustering software which works on Suse 8.2 or Suse 9.0? Thanks regards, Anand -- Anand TNC PhD Student, Engine Research Laboratory U-55 IISc Hostels, Dept. of Mechanical Engg., Indian Institute of Science, Indian Institute of Science, Bangalore 560 012. Bangalore 560 012. Ph: 080 293 2591 Lab Ph: 293 2352 080 293 2624 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From siegert at sfu.ca Fri Jan 23 12:37:34 2004 From: siegert at sfu.ca (Martin Siegert) Date: Fri, 23 Jan 2004 09:37:34 -0800 Subject: [Beowulf] cluster on suse In-Reply-To: References: <40103EFA.6060900@comcast.net> Message-ID: <20040123173734.GA17568@stikine.ucs.sfu.ca> Hi Anand, On Fri, Jan 23, 2004 at 10:40:43AM +0530, Anand TNC wrote: > I'm new to clustering...does anyone know of some clustering software which > works on Suse 8.2 or Suse 9.0? You actually do not have to look very far: SuSE comes with a lot of software for clusters. Here is a list of SuSE-9.0 rpms you may want to look at: mpich-1.2.5-241.i586.rpm mpich-devel-1.2.5-241.i586.rpm lam-7.0-77.i586.rpm lam-devel-7.0-77.i586.rpm fftw-mpi-2.1.3-960.i586.rpm fftw-mpi-devel-2.1.3-960.i586.rpm OpenPBS-2.3.16-466.i586.rpm OpenPBS-clients-2.3.16-466.i586.rpm OpenPBS-devel-2.3.16-466.i586.rpm OpenPBS-mom-2.3.16-466.i586.rpm OpenPBS-scheduler-2.3.16-466.i586.rpm OpenPBS-server-2.3.16-466.i586.rpm OpenPBS-utils-2.3.16-466.i586.rpm and probably more. As an aside: that version of OpenPBS is actually by far superior to the version that you can download from the PBS web site. The SuSE rpm has patches for many bugs in the (unsupported) openpbs source code. Thus, even if you are not running SuSE you may want to download their OpenPBS src.rpm and recompile it for your distribution (alternatively you could use SPBS). Cheers, Martin -- Martin Siegert Manager, Research Services WestGrid Site Manager Academic Computing Services phone: (604) 291-4691 Simon Fraser University fax: (604) 291-4242 Burnaby, British Columbia email: siegert at sfu.ca Canada V5A 1S6 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From afant at geekmail.cc Thu Jan 22 11:29:30 2004 From: afant at geekmail.cc (Andrew Fant) Date: Thu, 22 Jan 2004 11:29:30 -0500 Subject: [Beowulf] Hypothetical Situation In-Reply-To: References: Message-ID: <34120000.1074788970@flux.usg.tufts.edu> Brent, Good to see someone from Rice on here. I spent a couple years at Rice in the mid-ninetys, myself in Biochemistry and Cell Biology. To be clear, are you wanting to use a "real" beowulf with single system image and unified process space, or are you more interested in a compute farm, where each system runs it's own copy of the OS? If it's the former, I can't be much help. If it is the latter, is there a reason why user-mode linux is not an option for the researchers? Are they part of the CITI crew and doing hardcore OS work, or are they computational science types who have software that is just very anal about kernel, etc. versioning? It would probably be far easier to keep a stable minimal OS on each node and mount loopback filesystems that the users keep their custom environments in. Alternately, if you have to go the bare-metal route, systemimager might be an option, if the delay in time to reimage the system after job submission is not deemed excessive; in any case you may need to examine a fast dedicated network for reimaging to keep the time cost low. Normally, I am a big fan of LSF, but from the sounds of the demands, you would probably do better to work with SGE (Grid Engine), since the situation you describe would require a couple of reboots if real installs are needed. I don't think any available batch system can handle reboots as part of a prelude/prologue script cleanly, but with SGE, you at least have a chance to hack the code to try and fix it. Have you given any thought to which distro you intend to use? I will admit to being a Gentoo fanatic, but it sounds as if you might want to look into it. Gentoo is a source-based Linux distribution that allows the installation to be fairly tightly tailored to the application at hand. If the users are demanding their preferred kernels, I wouldn't be surprised if they started wanting their own tool-chains and application versions in the near future. Gentoo would allow them to build exactly what they want in a chrooted directory that could then either be used as the loopback filesystem for uml, or as the golden client to upload to systemimager. You can find out more from www.gentoo.org or check on irc.freenode.net in #gentoo-server. This sounds like an interesting problem. I'd be glad to answer more questions or serve as a sounding board if you want. Is Cathy F. still working on networking stuff in Mudd? If she is, and you see her, please give her my regards. Hope this Helps, Andrew Fant Tufts University --On Thursday, January 22, 2004 09:29:09 -0600 "Brent M. Clements" wrote: > We have a request by some of our research scientists to form a beowulf > cluster that provides the following abilities: > > 1. It must be part of a shared computing facility controlled by a batch > queuing system. > 2. A normal user must be able to compile a customized kernel, then submit > a job with a variable pointing to that kernel. The batch queuing system > must then load that kernel onto the allocated nodes and reboot the nodes > allocated. > 2a. If the node doesn't come back up after rebooting, the job must be > canceled and the node rebuilt automatically with a stable kernel/image. > 3. When the job is finished, the node must be rebuilt automatically using > a stable kernel/image. > > The admin's here have come up with a name for this: "user > provisioned os imaging". Our gut feeling is that this can be done but it > will be a highly customized system. What is everyone's thoughts, > experiences, etc etc concerning doing something like this? > > Thanks, > > Brent Clements > Linux Technology Specialist > Information Technology > Rice University > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From andrewxwang at yahoo.com.tw Fri Jan 23 23:09:59 2004 From: andrewxwang at yahoo.com.tw (=?big5?q?Andrew=20Wang?=) Date: Sat, 24 Jan 2004 12:09:59 +0800 (CST) Subject: [Beowulf] SGE vs LSF vs PBS... Message-ID: <20040124040959.85793.qmail@web16806.mail.tpe.yahoo.com> I almost forgot, Platform Computing is willing to cut the price of LSF, what used to be $1000 is now $50 per CPU. I think you might want to stay with LSF if your admin likes it, since switching means new trainings, service intereptions, and user complains. You can get that discount provided that you are from an edu site and buy enough. http://www.bohnsack.com/lists/archives/xcat-user/2104.html I believe they will cut the price even more when SGE 6.0 comes out later this year, or SPBS gets more popular. This is good news since we will benefit. Andrew. > But we can't afford that kind of license > pricing. ----------------------------------------------------------------- ??? Yahoo!?? ?????????????????????? http://tw.promo.yahoo.com/mail_premium/stationery.html _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Fri Jan 23 09:41:28 2004 From: jakob at unthought.net (Jakob Oestergaard) Date: Fri, 23 Jan 2004 15:41:28 +0100 Subject: [Beowulf] Which is better GNU C or JAVA (for network programing) In-Reply-To: References: <20040120235009.GD30005@unthought.net> Message-ID: <20040123144127.GA814@unthought.net> On Wed, Jan 21, 2004 at 11:46:15AM -0500, Robert G. Brown wrote: ... > > I find your lack of faith disturbing... ;) > > Trying an old jedi mind trick on me, are you? > > At least your metaphor is correct. You are indeed attracted to the > Power of the Dark Side...:-) Hey! Making jokes when we're having an *argument* - now that's just plain unfair! ;) ... > Well, we COULD make a list of all the major programs involving > 1KLOC > (exclusive of comments, even) that have been written entirely in C (plus > a small admixture, in some cases, of assembler). Unfortunately, the > list would be too long (I'd be here literally all day typing), and would > include things like the kernels of pretty much all the major operating > systems (certainly all the Unix derivatives and variants) and gcc > itself, including its g++ extension. For good reason too See - C++ was standardized in 1998 - but until around 2001 or so, there were no decent compilers generally available. Even today, "certain" platforms still have problems on the C++ compiler front. How many compilers and operating system kernels have been written from scratch since, say, 2001? Yeah, probably some, but none that I know of either :) As I see it, C++ is a very new language when you consider languages that are "mature" and "practically usable". That's why I understand (and appreciate) that big things like the Linux kernel or GCC are written in C - anything else at that time would have been plain silly. > > To be fair, that doesn't prove that your observation is wrong -- there > is a rather huge list of fortran sources available in my primary field > of endeavor, for example, which doesn't stop fortran from being a cruel > joke. Did he just say the F-word? ;) (oh, the memories of parallelizing serial chemistry code in Fortran pre-77 from Cray to SP/2 with MPI.... The guy who wrote it had kept the Amdahl-induced line serial numbers after column 73 in his code as a sort of revision control - so he could see which lines he wrote before and after 1970... I have a big scar on my chest from back then... The scars in my mind will probably never heal though ;) ... > What you probably mean is that IF everybody knew how to program in C++, > they would have written all of this in C++, right? *if* they knew the language and *if* they started over from scratch today, then 100% affirmative *yes*. (Starting a new project was the spawn of this thread) > Let's see, is there > a major operating system out there that favors C++? Hmmmm, I believe > there is. Are it and its applications pretty much perpetually broken, > to the extent that a lot of its programmers have bolted from C++ and use > things like Visual Basic instead? Could be. What operating system would that be? :) I make a living from writing a piece of commercial software that runs on a bunch of different operating systems, one of them being the one you mention. We use the Intel C++ compiler on that platform. We have to. Because the opearting system vendor do not have a C++ compiler themselves - they do have a compiler called "C++", but it does not understand the language. We only have two major problems left with C++ support on that platform; that Intel rely on the OS vendor's linker and STL - those are the two weak links that are left. So, honestly, I cannot say that they favour C++. If they do, they have one sick way of showing it ;) They do favour "bad C with extensions" though - I've seen a lot of such code (on many platforms) being labelled as C++, so that might be where the confusion comes from ;) Now, you can look at my statements as I'm arguing for a solution which would fit in an ideal world (which we do not live in) - this is not my intention. I am just trying to point out that things have changed the past 10 years, and what was true about a language under development 10 years ago just may not be the entire truth about what that language has evolved into today. > This isn't intended to be a purely a joke observation. I would rather > advance it as evidence that contrary to expectations it is MORE > difficult to write and maintain a complex application in C++. I think this boils down to how well written the application is - and that will hold true for any language out there. I have seen unreadable ML - something I was told at university was impossible (yeah, I didn't believe them either). > The very > features of C++ that are intended to make an application "portable" and > "extensible" are a deadly trap, because portability and extensibility > are largely an illusion. Maybe I misunderstand, or maybe you know a dark corner of C++ that I don't (that's not impossible at all :) But for platform portability I use makefile rules to compile different implementations of code prototyped in a common header file. Same as I do in C. ... > To go all gooey and metaphorical, programming seems to be a holistic > enterprise with lots of layering and feathering of the brush strokes to > achieve a particular effect with C providing access to the entire > palette (it was somebody on this list, I believe, that referred to C as > a "thin veneer of upper-level language syntax on top of naked > assembler"). C++, in the same metaphor, is paint by numbers. It > ENCOURAGES you to create blocks to be filled in with particular colors, > and adds a penalty to those that want to go in and feather. Hmm.... I would say, C++ allows you to write "quite" efficient code really darn easily. If "quite" is not good enough, it allows you to do what C would have forced you to; to write the stuff yourself. With no run-time performance difference between C and C++. You can say C++ might encourage people to accept the "quite" efficient solution, whereas a "real" C programmer would have spent that extra week writing yet another AVL tree implementation instead of using the existing RB tree provided by the STL. But there is nothing that stops you from spending that week and re-inventing the AVL tree in C++ either. Now some C programmers would say "but there are standard implementations for all those data structures for C as well!" - the really funny part being, that all generic data structure implementations in C that I have seen, use void pointers to point to key/data members, which makes it impossible for the compiler to do clever inlining, and therefore a type-agnostic C implementation of an RB tree will actually be run-time *slower* than the corresponding STL C++ RB tree ;) This is the sort of thing that made me move from C to C++ (where feasible - yes, part of my job is also writing software that will actually function on a number of platforms including NetWare, so I do write and maintain a good piece of C code as well ;) ... > a) Do C's basic object features (typedefs, structs and unions, for > example) suffice to meet the needs for object creation and manipulation > in those code segments where they are needed? I would say of course > they do. I agree. But I must add, that for example having "destructors" can take a lot of pain out of memory management, when used cleverly and appropriately. I have not found any usable substitute for that in C. (Yes, you can write "destructor functions" and call them instead of free(), it will work but it is less convenient). Once you start using exceptions in error handling, destructors can become quite essential in assuring that memory leaks cannot happen. > IMO, protection and inheritance are a nuisance more often than > an advantage because, as I observed, real "objects" are almost never > portable between programs (exceptions exist for graphical objects -- > graphics is one place where OO methodology is very appropriate -- and > MAYBE for DB applications where data objects are being manipulated by > multiple binaries in a set). "protection" in particular I think of as > being a nuisance and more likely to lead to problems than to solutions. > In a large project it often translates into "live with the bugs behind > this layer, you fool, mwaahhahahaha". Result: stagnation, ugly hacks, > bitterness and pain. In single-developer projects, just who and what > are you protecting your structs FROM? Yourself? Never heard that argument before - but it's a good one. I do not see protection as being a powerful feature in C++ (but I have not thought about it before now - scary, you make me think! ;) I see protection more as a way of commenting your code, with a compiler assisted check that someone read your 'comment'. For example, "private" means, "when you use this class/struct, you're not supposed to be referring directly to this member". Someone new to the project can look at the struct/class in question, and will know (assuming it is well written code), that probably the original author had a good reason for not wanting others to address those members directly. So, I guess one could think of protection as "enforced comments". In a sense one could argue that this is an unnecessary language feature, because it make up for three reserved words (private, protected, public), and only allows us to "enforce comments" in one particular (but rather common) case. Anyway, it's there. It does not give run-time overhead. Use it if you want to. Or just use 'struct' if you feel better with that. These two type declarations are *identical* (will yield exactly the same code, memory layout, run-time performance, everything...): struct A { B b; }; class A { public: B b; }; If what you need is the first, it'd be silly to write the longer one ;) I have quite a lot of C++ code which obviously uses the first and shorter example. Anything else would just be plain stupid. > > b) Do C++'s basic object features increase or decrease the efficiencies > of the eventual linked binaries one produces? As you say, C++ is more > than an "extension" of C, it has some real differences. In particular, > it forces (or at least, "encourages") a certain programming discipline on > the programmer, one that pushes them away from the "thin veneer" aspect > of pure C. I disagree that it "forces" anything. This is a delusion, but a common one at that. I do agree though, that it can be difficult to keep people "sharp", when you use a language that allows you to write really bloated code easily. In C it is more difficult to write code that magically results in about a billion nested function calls, even more dynamic memory allocations, etc. etc. In C++ this can be a trap. In order to write lean code in C++, you need to have some sort of dicipline (or a screaming violent psychopath in the corner of the office - that's me ;) So, in a way I agree that C++ can be more dangerous wrt. writing poorly performing code. But it is not a language limitation, it is a disciplinary problem :) The same features that lead to this trap, are the same features that allow you to write code that is as fast as "C the way you would have written it if you had had the time", correct and maintainable. If the programmer does not understand the language (ie. when to use what), then I guess he has a good chance of writing something that's really crap. But to a large extent that would go for any other language as well. The real incompatibilities from C to C++ (except for a few reserved words which do cause real (but easy to remedy) problems), are mostly C braindamages that noone really depends on anyway - at least this is my experience... If anyone has counter examples, I'm always happy to be enlightened. For example: struct A { struct B { int a; }; }; struct B b; The above is valid C and invalid C++, because C++ does not forget scope checking in this special case. C++ would require you to write: struct A::B b; Now, I'd be rather surprised if something like this is going to make all your C code not compile with a C++ compiler ;) > I think it is clear that each additional layer of contraint > INCREASES the intrinsic complexity of the compiler itself, thickens the > veneer, and narrows the range of choices into particular channels. It absolutely increases the complexity of the compiler front-end. Compiling C++ just takes longer. That's a real and practical problem (but a good excuse for a Beowulf ;) However, the language was carefully designed so that "you pay for what you use". If you take a piece of C code that happens to also be valid C++, and compile it with g++ instead of gcc, there is no valid reason for the resulting code to be slower than the original gcc-compiled code. > Those channels in C++ were deliberately engineered to favor somebody's > idea of "good programming practice", which amounts to a particular > tradeoff between ultimate speed and flexibility and code that can be > extended, scaled, maintained. So I would expect C++ to be as efficient > as C only in the limit that one programs it like C and to deviate away > as one uses the more complex and narrow features. One very powerful feature of C++ is templates. It has no run-time overhead, compared to how you would solve the problem in C. They do allow more complex problems to be solved, where in C you would fall back to using void pointers and casts (instead of the proper types), which would give you a *slower* C implementation than what C++ makes feasible. Let me throw out a challenge: "Write a program in C that reads from standard input until EOF, and prints the number of distinct words found in that input" I give you my solution here, in C++: ---------------------------------------- #include #include #include int main(int argc, char **argv) { // Declare and Initialize some variables std::string word; std::set wordcount; // Read words and insert in rb-tree while (std::cin >> word) wordcount.insert(word); // Print the result std::cout << "Words: " << wordcount.size() << std::endl; return 0; } ---------------------------------------- I use an RB-tree to store the words I find, and print the number of elements in the final tree. A comparable C solution shouldn't be more than a few thousand lines ;) Oh, right, I cheated and used STL. Well, STL is a big part of why C++ is really handy, but go right ahead and use "glib" or whatever. I still guarantee you two things: 1) Your code will be longer 2) Your program will be slower As always, I love to be proven wrong ;) > Efficient "enough", > sure, why not? CPU clocks and Moore's Law give us cycles to burn in > most cases. And when one uses objects in many cases, they ARE cases > where bleeding edge efficiency IS less important than ability to create > an API and "reuse" data objects via library calls (for example). Still > I think C would have to win here. I'm just dying to see that piece of C which cannot possibly be made to parse as C++, without fundamentally changing the structure of the code. ...[lots of good arguments nipped for brevity]... > unchanneled by the compiler. Yup, I understand what you mean. And I'm not arguing that C++ is some magic silver bullet either. All I'm saying is, that I think for 99% of the larger projects you start today, C++ can provide you with some really convenient benefits over C, with no practical downsides. That is, if you know the language, if the people who will maintain your code knows the language, and if you can rely on having a reasonably good compiler. These are definitely practical obstacles out there in the scary real world of coding, and I am aware of that. Just trying to point out that *if* those practical obstacles can be ovecome, then there *is* practical benefits. And today, just *maybe* it is not universally a bad idea to at least consider whether the practical obstacles can be overcome in order to reap the benefits that can be provided. Uh, I'm starting to sound like some middle-ground-searching tree-hugging weed-smoking hippie. That was not my intention! ;) > Do I have to deal with sometimes screwing up data typing? Absolutely. > Do I have to occasionally deal with working my way backwards through my > programs to redo all my structs? Of course (I'm doing it right now with > a program I'm writing). This process is INEVITABLE in real-world > program design because we DON'T know a priori what the best shape is for > the data objects is until we've programmed them at least once the wrong > way, in most cases. The really interesting question is whether or not > this PROCESS is any easier in C++ than in C. I can't see why it would > be, but perhaps it is. In my experience, not much, if at all. You screw up, decide to rewrite, and rewrite. One thing which could give C++ an edge (depending on your program of course), is, if your code is 1/10th the size in C++ - you would have less code to rewrite :) ... > This is a lovely tradeoff in a lot of cases, which is why I hesitate to > actually stick a rod up java's ass and barbeque it on a spit. I really > do believe the BASIC argument associated with OO programming, that the > data and what you want to do with it should dictate program design > features, including choice of language and programming technique. For > many problems perl or python provide a very good data interface where > one DOESN'T have to mess with lots of data issues that are a major chore > in C. Wanna make a hash? Use it. Wanna make your hash into an array? > Change the little % TO A @ and hack your dereferencing a bit. Is your > hash actually stored as a linked list, as an array of structs? Don't > ask, don't tell, don't try to look. All this and direct syntactical > manipulation of regular expressions, what's not to love? A hash of hashes, is not to love in perl ;) At least, that's one thing they got right with STL and templates - templates may not have the prettiest syntax imaginable, but at least it's consistent. A map from int to int: std::map A; a[42] = 5; A map from int to a map from int to int: std::map > A; A[42][5] = 8; I can't remember how many explicit '{' and '%' you'd need in perl to do that last one, but I do remember it wasn't pretty ;) To be fair, I have never seen anything remotely as powerful as Perl's regular expression handling. That's just sweet. Unreadable, but sweet. ... > > > I prefer to think of "C++" as "A better C", rather than a "C extension", > > as not all C is valid C++, and therefore C++ is not really an extension. > > Funny that. I tend to think of C++ as "A broken C extension" for > exactly the same reason;-) Good one! > > If they hadn't broken it, then there really would be no reason not to > use it, as if C were a strict subset of C++ all the way down to the > compiler level, so that pure C code was compiled as efficiently as pure > C is anyway then sure, I'd replace all the gcc's in my makefiles with > g++'s. That would certainly lower the barrier to using C++; C > programmers could program in C to their heart's content and IF AND WHEN > their program needs a "C++ object" they could just add it without > breaking or even tweaking the rest of their code. Have you actually tried this and found that your code would not compile with g++ ? Anyway, I'm not arguing that all that is C should become C++. C has its uses (sort of like automatic weapons ;) I think for new projects C++ is worth considering. If I had a 10KLOC C program that worked really well, I probably wouldn't bother to convert it (even if no conversion was necessary). Why risk to break the application on some obscure platform out there, if you are not going to take advantage of the new language features anyway? I've never tried such a conversion on a large program, and for the small programs where I did it I ended up pretty much rewriting them anyway (having 60% void* "generic" code with casts, and 40% template-based code is just *messy* to look at). > Breaking C was (IMO) a silly design decision. Well, yes, but I believe the largest brekage is due to the new keywords, which really coulnd't have been avoided (unless "static" should have gotten yet another five-ten different meanings depending on which context you use it in ;) The other breakage (my scope example earlier) is more like a braindamage in C that virtually noone depends on which was fixed. I like that. > So was adding crap like a > completely different stdin/stdout interface, which makes most C++ code, > even that which doesn't use objects in any way, most > non-back-portable-or-compatible to C. The C library is readily available, and writing fprintf() instead of std::cout in your C++ code does not make you a bad person :) > It is very clear that these were > all deliberate design decisions INTENDED to break C Ok, Bjarne explains quite well in his books why each breakage was introduced - they did deliberately break compatibility, but they paid a lot of attention to not breaking "more than absolutely necessary". You may disagree with the definition of "absolutely necessary" of course - I do too at times. > and FORCE > programmers to make what amounts to a religious choice instead of > smoothly extend the palette of their programming possibilities. I'd > honestly have to say that it is this particular aspect of C++ more than > any other that irritates me the most. Ok. I just do not see the breakage as such a deliberate attempt to piss people off ;) ... > > > > Amen! This goes for C++ as well though. > > Y'know, you won't believe this, but I actually added the (and C++) > references in my original reply just thinking of you...;-) Hehe, thanks ;) > Last time we had this discussion you were profound and passionate and > highly articulate in advancing C++ -- so much so that you almost > convinced me. :) > Alas, that silly barrier (which I recall your saying took > you YEARS to get over yourself)... I just bounce right off of it in a > state of total irritation every time I try. I understand. I don't know of any easy way to do that. Having too much time on ones hands is a good start I think. Yeah, I had that once, I can almost remember it, if I think about it hard ;) It was worse when I moved from Pascal to C though (quite a few years earlier, I guess I was 16 back then - time does fly ;) > Another problem is that > every time I work with a student C++ programmer coming out of Duke's CPS > department (which now teaches C++ as standard fare) I observe that while > they are just great at creating objects and so forth they are clueless > about things like pointers and malloc and the actual structure of a > multidimensional array of structs and how to make one and manipulate its > contents. As a consequence I have to spend weeks teaching them about > all of the virtues of "C as a thin veneer" in order to get them to where > they can cut advanced code at all, in either one. Some people argue that it is better to start with C++, then "move on" to C if necessary (or at least the subset of C++ that is also C). I think they have many valid points. And so do you. I don't know. For sure, you cannot be an excellent "computer programmer" if you do not understand in detail how memory works, and all those other "details" that *can* be abstracted away in C++. Those details will still bite you, no matter how far they are abstracted away, and if you do not understand them, you cannot hope to solve the problem. I guess we agree on that :) > You may not have encountered this as you did C first. Or maybe you > never have cause to allocate triangular arrays of structs or to build > your own linked list of structs, or maybe C++ does this and eats your > meatloaf for you and I'm just to ignorant to know how. C++ does not make the need for understanding go away. Other "more pure" languages make understanding irrelevant though - no matter how clever you are, you cannot solve the problem because the language won't let you ;) > Its funny. Back in the 80's all the CPS departments taught pascal > because it provided strong type checking and absolutely forced one to > use a rigorous bottom-up program design methodology. Precisely the same > features that caused all real programmers to run from it screaming, as > somehow all the programmers I've ever met work top down initially, then > top, bottom, middle, whereever until the project is finished, often with > a few epiphany-driven total project reorganizations in the middle as > experience teaches one what the REAL data objects are that they are > manipulating and whether they need easily maintained code that can run > relatively slowly or whether they need the core loop to run faster than > humanly possible if they have to hand-code it in assembler to get it > there. The difference, as I see it, is that Pascal *forces* you to use a very rigorous (and limiting) type system, or, work around it in extremely unsafe ways. C++ *allows* you to use a fairly rigorous type system, and "fall back" to casting where required. But even the casts have been made "safer", for example, the static_cast will not cast away const, you must explicitly add a const_cast if you really want to do that. Just an example. Typing the longer casts can be a little tedous - but hey, \textbf{} is that long because if you used it frequently your text would look ugly (don't remember if it was Lamport or Knuth who gave that explanation). Same goes with cast-ridden code. The longer and more difficult a cast is to type, the better ;) But if you disagree with me, C++ allows you to use C-style casts as well. Just don't come back complaining to me when they bite you ;) > Now they teach C++, and as you observed, teach it badly. I humbly > submit that the REASON they teach it badly is that they aren't teaching > "C++" at all, they are teaching structured programming and programming > discipline with a language that simply doesn't permit (or if you prefer > encourage) the student to use methodologies with far more power but far > less externally enforced structure. Pure pedagogy, in other words. Thank you, I do prefer "encourage", and I would argue that "permit" would be false. ;) Handing a person a copy of Webster's will not make him a poet. I'm sure we can agree on that last one (we can, right? ;) It's a pity, really, if teaching of "real programming languages" is really in such a sad state. I got some learning material from the univeristy here (have a cousin who started there recently), and that was *bad*. It was C though, at least I think it was meant to be (yes, it was hard to tell). Giving 3rd semester students example code where you do inter-process communication by writing to a file from one process, and polling it from another (hoping that other write finished completely), now that's just sick. Best part; the delay function (from keeping the polling from taking too much CPU time or whatever) was a busy wait - yes, you know, a for() loop which counts to a gazillion and a half, with a neat little arithmetic function in it to keep the compiler from optimizing the loop away. I almost cried. Then came the rage. Oh, and the voices in my head. [...good stuff snipped...] > I suspect that students who learn C++ in this order are, as you appear > to be, really good programmers who can get the most out of C++ and its > object oriented features without "forgetting" how to manipulate raw > blocks of memory without any sort of OO interface per se when the code > structure and extensibility requirements don't warrant all the extra > overhead associated with setting one up. I really don't know how one ought to teach people "properly". But I absolutely agree with you that one must understand the machine that one intends to program. At least to a resonable extend (and that, in my book as well as in yours it seems, includes knowing about layout of datastructures in memory, knowing that caches exist, etc.) ...[much more good stuff snipped]... > Give me a few days paring C++ down to the bare minimum extension of C so > that it is a pure superset (so ALL C CODE just plain compiles perfectly > and even produces the same runfile, so that the I/O language > "extensions" are moved off into a library that you can link or not, You would need to take out the 'template' reserved word (among others) to gain that full compatiblity. That alone would make C++ woth only half as much. Templates are *extremely* powerful (when in combination with either existing C and new C++ constructs). Again, I believe that there were good reasons for the breakage introduced. Sure, you can probably point out things that were stupid, and I may even agree with you, but a major part of the breakage was definitely a "lesser evil", at least the way that I see it :) > mostly not, and so that using things like classes and protection and > inheritance are suddenly pure and transparent extensions of the basic > ideas of structs and unions and typedefs, and I wouldn't even argue. In > fact, I'd say that the argument itself becomes silly and moot. Exactly! >From where I stand, in many situations (in particular, the situation that spawned this very thread - the start of a new project), we may already be at the point where there is no practical downside to using C++ as compared to C. That leaves you with some possible benefits you can use where you see fit. Assuming, as before, that the author knows the language etc. etc. And the whole argument becomes silly and moot :) > One > wouldn't even have to change Makefiles, as C++ would just be a feature > of gcc that is automatically processed when and where it is encountered. [albatross:joe] $ g++ -o wc wc.cc [albatross:joe] $ gcc -o wc wc.cc -lstdc++ We're there already - sort of ;) Just rename your files and you're done (the wc.cc file is the "challenge" from above) ;) > So the kernel would suddenly be written in "C++", sort of, and there > would be a very low barrier to converting C into C++ because basically, > there would be no conversion at all, only extension. Well, yes and no. Thinking about all this, I am actually not so much of a proponent of converting existing C projects into C++. I believe it is better to make a "clean break" - when starting a new project, go C++ from the beginning. Do templates. Use STL. Manage your memory safely. Compared to a hybrid project, where some percentage of the code is "very" safe, some percentage is "rather unsafe", and some of the safe code may need to call unsafe code and end up being unsafe itself. I don't know actually - I have a gut feeling that such a conversion would be a pain. The problem is not the conversion itself, the problem is that you end up with a so-called C++ program, which has never taken advantage of the advantages that C++ could have provided. So, in effect, your old "Good C" program becomes a "Bad C++" program. I am not so sure that this is a change to the better. In fact, you may even make me say that it is probably not... And this may be a good reason to keep the good ol' "gcc" for a few more decades. For all the code that is "Good C". > This of course will never happen. Until it does, then an excellent > reason to choose C instead of C++ is if you happen to know C very well > but not know (or care for) C++ as well. That's a good reason for sure! :) It's even language neutral ;) [...good stuff snipped yet again...] > majority of people who stick with C++ end up programming for Windows for > some big corporation (which might well be Microsoft) -- and produce > shitty, broken, huge, inefficient code as often as not. Hey, I will claim that: *) "Bad C++" has a strange attraction to crappy programmers *) Windows has a strange attraction to crappy programmers Thus, I end up observing the very same thing as you; that a lot of people write crappy C++ on windows. I'd just rather attack people than languages ;) (uh, *some* languages that is... Don't get me started on the J-word ;) [...more good stuff snipped yet again...] > lives next door writing software and clipping coupons. Other faculty I > know seem to view C++ the same way -- a good thing to use to teach > students because it forces them to code a certain way, but THEY don't > use it for THEIR research projects. I think it's a matter of time until more people get to learn the language and get used to it. Sure, a struct may be all you need. But then one day you have a struct which just needs a dynamically allocated member. And then, maybe, you find that it is convenient that you can put the deallocation of that member in the struct destructor. It's the little things, at first. Small conveniences that just come in handy. As far as my arguments have gone, C++ is more like a three year old language, in the "mature languages" world. No wonder it's not ubiquitous. Especially when looking at production systems or even research projects that may have been 10 or 20 years in development. > > Professionals seem more often than not to prefer having the entire > palette and all the brushes, and don't like even MAKING color by numbers > codelets. I continue to claim, that you do have the whole palette. It's just, that for the little trivial things where colour by number is more convenient and fully accomplishes the task at hand, you can actually do it, instead of starting to cut out the support frame for your future canvas on which you will eventually paint your masterpiece. You should still know how to produce a canvas, and how to paint properly. But you should (in my oppinion) not be absolutely required to start from scratch for every little trivial problem that needs solving. Any large project will solve countless small trivial problems, and a number of absolutely nontrivial problems too probably. It's darn convenient to be able to solve the trivial ones trivially ;) It will take a long time before C++ is generally accepted as "A Better C", I'm sure, but the time will come eventually (bets, anyone?) ;) > > This is very definitely not exhaustive or definitive. I also know > students where they exact opposite is true -- they learned C++, learned > about data structs and malloc and C, and continue to code in C++ (as you > do) with strictly enriched abilities. Of course they are (like you;-) > Very Bright Guys... *blush* ;) > and could probably even write good Fortran code if > they ever learned it. No, wait, that is an oxymoron -- couldn't happen. Point! ... > I agree, and hope you realize that I'm deliberately overarguing the case > for C above for precisely that reason. I really do believe that it is > as much a matter of taste and your particular experience and application > space as anything else and wouldn't be surprised if some java coder DOES > speak up for java as that's what THEY are really good at and they've > discovered some feature that makes it all worthwhile. I see it as serious fun :) Besides, it's no fun to hear about a langauge from someone who doesn't care about it. You need to poke people, call them sissies, tell them their language sucks and things about their mother, and *then* you can get a good language discussion going ;) A good language discussion needs *passion*, *anger*, *love*, true feelings. Or maybe that's just me... ;) > We've already heard from the python crowd, after all, and THAT'S a > language where they even eschew such obviously useful constructs as {}'s > and line terminators such as ; in favor of indentation as a matter of > deliberate design/dogma. It really does come down in the end to > religious belief/personal taste. They do that to some extent in Haskell as well - I code a little in Haskell from time to time, but haven't gotten around to Python yet. The indentation idea is, well, "interesting", I'd say :) > > In the best spirit of tongue-in-cheek, I cannot do better than end with: > > http://www.phy.duke.edu/~rgb/Beowulf/c++_interview/c++_interview.html > > Read it and weep...:-) Will do. Both, I fear ;) Cheers, / jakob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From amb_lew at yahoo.com Sat Jan 24 14:14:20 2004 From: amb_lew at yahoo.com (ambrose lewis) Date: Sat, 24 Jan 2004 11:14:20 -0800 (PST) Subject: [Beowulf] MPI_File experience??? Message-ID: <20040124191420.82560.qmail@web14204.mail.yahoo.com> Hi All: I have a small cluster running a remote sensing code. I'm using MPICH on top of Mac OS X. Right now, my parallel performance isn't so great, because my communication time is far outweighing my computations. Does anybody have experience/suggestions/lessons learned about using the various MPI-IO routines??? I'm particularly interesting in learning more about successful use of the non-blocking read and write calls. Any pointers to sample codes would be greatly appreciated! THANXS! amb __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Sat Jan 24 16:36:06 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Sat, 24 Jan 2004 22:36:06 +0100 Subject: [Beowulf] MPI_File experience??? In-Reply-To: <20040124191420.82560.qmail@web14204.mail.yahoo.com> References: <20040124191420.82560.qmail@web14204.mail.yahoo.com> Message-ID: <4012E546.8010309@telia.com> Ambrose, This sounds interesting, how is the price/prestanda compared to PCs? /pelle ambrose lewis wrote: >Hi All: >I have a small cluster running a remote sensing code. >I'm using MPICH on top of Mac OS X. >Right now, my parallel performance isn't so great, >because my communication time is far outweighing my >computations. >Does anybody have experience/suggestions/lessons >learned about using the various MPI-IO routines??? >I'm particularly interesting in learning more about >successful use of the non-blocking read and write >calls. >Any pointers to sample codes would be greatly >appreciated! >THANXS! >amb > >__________________________________ >Do you Yahoo!? >Yahoo! SiteBuilder - Free web site building tool. Try it! >http://webhosting.yahoo.com/ps/sb/ >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kums at mpi-softtech.com Sun Jan 25 13:23:00 2004 From: kums at mpi-softtech.com (Kumaran Rajaram) Date: Sun, 25 Jan 2004 12:23:00 -0600 (CST) Subject: [Beowulf] MPI_File experience??? In-Reply-To: <20040124191420.82560.qmail@web14204.mail.yahoo.com> Message-ID: Ambrose, If your application is such that only root process performs file I/O and distributes raw data/ aggregates processed data to/from other processes, its recommended to shift to the MPI-2 model and use MPI-IO interface to perform file I/O. In order to gain performance from the MPI-IO non-blocking I/O interface, the middleware/implementation needs to as well have a good asynchronous model. Please read the following for more info: http://www.mpi-softtech.com/company/publications/files/Mercutio.pdf You should be able to find sample MPI-IO codes and other information in the following website: http://www.cs.dartmouth.edu/pario/ Hope this helps. Cheers, -Kums On Sat, 24 Jan 2004, ambrose lewis wrote: > Hi All: > I have a small cluster running a remote sensing code. > I'm using MPICH on top of Mac OS X. > Right now, my parallel performance isn't so great, > because my communication time is far outweighing my > computations. > Does anybody have experience/suggestions/lessons > learned about using the various MPI-IO routines??? > I'm particularly interesting in learning more about > successful use of the non-blocking read and write > calls. > Any pointers to sample codes would be greatly > appreciated! > THANXS! > amb > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shriram1976 at yahoo.com Sun Jan 25 16:40:51 2004 From: shriram1976 at yahoo.com (Shriram R) Date: Sun, 25 Jan 2004 13:40:51 -0800 (PST) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node Message-ID: <20040125214051.84151.qmail@web11408.mail.yahoo.com> Hi, We have a 24 node/48 procs linux cluster running Redhat. The queueing system that we use is PBS. One of the nodes, node15, conked out completely and is not restarting. "pbsnodes -a" shows the state of the node15 as "down". However, jobs which had been running on node15 still show up when I do a "qstat". I tried to use "qdel" and "qsig" to delete these jobs, but the server complains that it is unable to contact pbs_mom, which is obvious since node15 is down. Can someone tell me how do I delete these jobs from the output of "qstat" ? TIA. -shriram __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Sun Jan 25 17:56:26 2004 From: bclem at rice.edu (Brent M. Clements) Date: Sun, 25 Jan 2004 16:56:26 -0600 (CST) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node In-Reply-To: <20040125214051.84151.qmail@web11408.mail.yahoo.com> References: <20040125214051.84151.qmail@web11408.mail.yahoo.com> Message-ID: YOu have to remove the job files from both the mom_priv/jobs directory on the moms that the job was running on as well as the server_priv/jobs directory on the pbs server. -B Brent Clements Linux Technology Specialist Information Technology Rice University On Sun, 25 Jan 2004, Shriram R wrote: > Hi, > > We have a 24 node/48 procs linux cluster running > Redhat. The queueing system that we use is PBS. One > of the nodes, node15, conked out completely and is not > restarting. > > "pbsnodes -a" shows the state of the node15 as "down". > > However, jobs which had been running on node15 still > show up when I do a "qstat". > > I tried to use "qdel" and "qsig" to delete these jobs, > but the server complains that it is unable to contact > pbs_mom, which is obvious since node15 is down. > > Can someone tell me how do I delete these jobs from > the output of "qstat" ? > > TIA. > -shriram > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Mon Jan 26 00:43:53 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Mon, 26 Jan 2004 00:43:53 -0500 Subject: [Beowulf] mpiBLAST 1.2.0 RPM's available Message-ID: <1075095833.4481.542.camel@protein.scalableinformatics.com> Hi Folks: We built some source and binary RPM's for mpiBLAST 1.2.0. You can find them at http://downloads.scalableinformatics.com/downloads/mpiblast/ or linked off of our main page http://scalableinformatics.com . These RPM's assume you have an operational and functional mpich installation. They should work with most architectures, though I am working on fixing the Opteron build. We created these as people seemed to have some trouble building the package, and installing it on their systems. If you grab the binary version, please edit the /usr/local/etc/mpiblast.conf file as per the instructions on the mpiBLAST page at http://mpiblast.lanl.gov . Please send us a note on any problems you encounter with this packaging. To install the binary version, download the appropriate binary and as root, run rpm -Uvh mpiBLAST-1.2.0-3.i686.rpm This assumes you have grabbed the i686 architecture binary. If you would prefer to build from source, make sure you have a functional mpich install with mpe, have it in your path and be able to compile with mpiCC, and then run: rpmbuild --rebuild mpiBLAST-1.2.0-3.src.rpm to build a binary. Then use the resulting binary RPM to install. Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shriram1976 at yahoo.com Mon Jan 26 01:03:30 2004 From: shriram1976 at yahoo.com (Shriram R) Date: Sun, 25 Jan 2004 22:03:30 -0800 (PST) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node In-Reply-To: Message-ID: <20040126060330.30554.qmail@web11411.mail.yahoo.com> Hi, This is with ref the reply below. I deleted the files from server_priv/jobs. But the node on which the jobs were running has completely crashed - doesnt even reboot. So, cant delete the files from mom_priv/jobs on the corresponding node. Hence the jobs are still showing up in "qstat". Any suggestions ? TIA -shriram --- "Brent M. Clements" wrote: > YOu have to remove the job files from both the > mom_priv/jobs directory on > the moms that the job was running on as well as the > server_priv/jobs > directory on the pbs server. > -B > > Brent Clements > Linux Technology Specialist > Information Technology > Rice University > > > On Sun, 25 Jan 2004, Shriram R wrote: > > > Hi, > > > > We have a 24 node/48 procs linux cluster running > > Redhat. The queueing system that we use is PBS. > One > > of the nodes, node15, conked out completely and is > not > > restarting. > > > > "pbsnodes -a" shows the state of the node15 as > "down". > > > > However, jobs which had been running on node15 > still > > show up when I do a "qstat". > > > > I tried to use "qdel" and "qsig" to delete these > jobs, > > but the server complains that it is unable to > contact > > pbs_mom, which is obvious since node15 is down. > > > > Can someone tell me how do I delete these jobs > from > > the output of "qstat" ? > > > > TIA. > > -shriram > > > > __________________________________ > > Do you Yahoo!? > > Yahoo! SiteBuilder - Free web site building tool. > Try it! > > http://webhosting.yahoo.com/ps/sb/ > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or > unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From franz.marini at mi.infn.it Mon Jan 26 04:41:02 2004 From: franz.marini at mi.infn.it (Franz Marini) Date: Mon, 26 Jan 2004 10:41:02 +0100 (CET) Subject: [Beowulf] PBS : deleting jobs that were running on a crashed node In-Reply-To: <20040126060330.30554.qmail@web11411.mail.yahoo.com> References: <20040126060330.30554.qmail@web11411.mail.yahoo.com> Message-ID: Hi, On Sun, 25 Jan 2004, Shriram R wrote: > Hi, > > > This is with ref the reply below. I deleted the files > from server_priv/jobs. But the node on which the jobs > were running has completely crashed - doesnt even > reboot. So, cant delete the files from mom_priv/jobs > on the corresponding node. > > Hence the jobs are still showing up in "qstat". Any > suggestions ? Have you restarted pbs_server *after* deleting the jobs under server_priv/jobs ? (and maybe even maui, if you use it) Franz --------------------------------------------------------- Franz Marini Sys Admin and Software Analyst, Dept. of Physics, University of Milan, Italy. email : franz.marini at mi.infn.it --------------------------------------------------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Mon Jan 26 20:17:35 2004 From: shewa at inel.gov (Andrew Shewmaker) Date: Mon, 26 Jan 2004 18:17:35 -0700 Subject: [Beowulf] C vs C++ challenge In-Reply-To: <20040123144127.GA814@unthought.net> References: <20040120235009.GD30005@unthought.net> <20040123144127.GA814@unthought.net> Message-ID: <4015BC2F.8030405@inel.gov> Jakob Oestergaard wrote: > Let me throw out a challenge: > > "Write a program in C that reads from standard input until EOF, and > prints the number of distinct words found in that input" > > > I use an RB-tree to store the words I find, and print the number of > elements in the final tree. > > A comparable C solution shouldn't be more than a few thousand lines ;) > > Oh, right, I cheated and used STL. Well, STL is a big part of why C++ > is really handy, but go right ahead and use "glib" or whatever. > > I still guarantee you two things: > 1) Your code will be longer > 2) Your program will be slower > > As always, I love to be proven wrong ;) Here's a similar program...word frequency...instead of word count. http://www.cs.bell-labs.com/cm/cs/pearls/sec151.html His C++ code is shorter, but his C code is faster. With a trivial modification, his C code can meet your challenge too. http://www.cs.bell-labs.com/cm/cs/pearls/wordfreq.c I downloaded the KJV of the Bible, Shakespeare's Complete Works, War and Peace, part of the human genome, and a few other files off of Project Gutenberg. The above C code was much faster on small files and twice as fast when I lumped everything together into a 35 MB file with 460000+ distinct words. The only test I ran where C++ beat C (by 20+%) was a 187 MB file with over 2.5 million distinct words, where I used PGI's hash_set instead of its regular set. Have a great day, Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 27 01:04:37 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Mon, 26 Jan 2004 22:04:37 -0800 (PST) Subject: [Beowulf] How to enable MPE environment in MPICH-1.2.5.2 In-Reply-To: <6.0.0.22.2.20040120073814.02186048@localhost> Message-ID: <20040127060437.45877.qmail@web21501.mail.yahoo.com> how to i enable MPE environment in my cluster do i have do build it explicitly or just some changes while installing mpich __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From swap_project at yahoo.com Tue Jan 27 01:40:27 2004 From: swap_project at yahoo.com (swapna ghosh) Date: Mon, 26 Jan 2004 22:40:27 -0800 (PST) Subject: [Beowulf] Clustering on Redhat9.0 In-Reply-To: Message-ID: <20040127064027.41294.qmail@web40808.mail.yahoo.com> Hi I am a newbee to the clustering and my next project is clustering with max 2 nodes. The servers are Redhat 9.0. I was going through few clustering procedure.. - beowolf with PVM - openMosix - SSI Cluster for linux There are lots many but i need the *free* software... Though the last one(SSI Clustering..) the document i read , it is on Redhat 9.0 openMosix is kernel patch oriented.. So my little knowledge says that it will be difficult everytime to upgrade if not rpm...[ i may be wrong ] beowolf with PVM, i did not get any such document(installation steps) till yet on Redhat 9.0. It will be really appreciated if anybody points me the document/url of the procedure *installation of clustering with beowolf PVM on RH-9.0* Lastly can anybody suggest me which one will be suitable for redhat 9.0? Thanks in advance -Swapna __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Tue Jan 27 04:50:19 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Tue, 27 Jan 2004 01:50:19 -0800 (PST) Subject: [Beowulf] Remote procedure call (error) Message-ID: <20040127095019.30138.qmail@web21506.mail.yahoo.com> have any body tried RPC programing on Linux i get rpc unknown protocol error i am trying to diaply the message on the remote host __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Tue Jan 27 09:56:04 2004 From: bclem at rice.edu (Brent M. Clements) Date: Tue, 27 Jan 2004 08:56:04 -0600 (CST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: <20040127095019.30138.qmail@web21506.mail.yahoo.com> References: <20040127095019.30138.qmail@web21506.mail.yahoo.com> Message-ID: Are the rpc daemons started up or enabled in xinetd/inetd? -Brent Brent Clements Linux Technology Specialist Information Technology Rice University On Tue, 27 Jan 2004, prakash borade wrote: > have any body tried RPC programing on Linux > i get > rpc unknown protocol error > i am trying to diaply the message on the remote host > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From shewa at inel.gov Tue Jan 27 10:37:19 2004 From: shewa at inel.gov (Andrew Shewmaker) Date: Tue, 27 Jan 2004 08:37:19 -0700 Subject: [Beowulf] Clustering on Redhat9.0 In-Reply-To: <20040127064027.41294.qmail@web40808.mail.yahoo.com> References: <20040127064027.41294.qmail@web40808.mail.yahoo.com> Message-ID: <401685AF.2000901@inel.gov> swapna ghosh wrote: > Hi > > I am a newbee to the clustering and my next project is clustering > with max 2 nodes. > > The servers are Redhat 9.0. I was going through few clustering > procedure.. > > - beowolf with PVM > - openMosix > - SSI Cluster for linux > > There are lots many but i need the *free* software... > > Though the last one(SSI Clustering..) the document i read , it is on > Redhat 9.0 > > openMosix is kernel patch oriented.. So my little knowledge says > that it will be difficult everytime to upgrade if not rpm...[ i may be > wrong ] OpenSSI also involves a patched kernel, so it sounds like you'll probably want to avoid it too, if you are avoiding openMosix. > beowolf with PVM, i did not get any such document(installation > steps) till yet on Redhat 9.0. It will be really appreciated if anybody > points me the document/url of the procedure *installation of clustering > with beowolf PVM on RH-9.0* RH-9 includes pvm-3.4.4-12.i386.rpm on the second CD and pvm-gui-3.4.4-12.i386.rpm on the third. You still have to set it up to use ssh or rsh, though. Search through the archives at http://marc.theaimsgroup.com/?l=beowulf&r=1&w=2v and you will find links to howtos or descriptions on how to set up rsh. I recommend learning how to set up ssh for host based authentication. http://www.omega.telia.net/vici/openssh/ has good instructions, but note that the latest OpenSSH requires "EnableSSHKeysign yes" to be set in /ssh_config. Also, you can simplify the instructions if you treat the cluster as a single system and copy the same host keys to all of the machines. Rocks does it that way. If you don't want to learn about that sort of thing, then install a cluster environment/distribution. > Lastly can anybody suggest me which one will be suitable for redhat > 9.0? You should look at Rocks or OSCAR. Rocks is a distribution based on RH and OSCAR can be installed on top of RH-9. They will set up PVM and MPI for you. http://rocksclusters.org http://oscar.openclustergroup.org Andrew -- Andrew Shewmaker, Associate Engineer Phone: 1-208-526-1415 Idaho National Eng. and Environmental Lab. P.0. Box 1625, M.S. 3605 Idaho Falls, Idaho 83415-3605 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Tue Jan 27 09:55:34 2004 From: bclem at rice.edu (Brent M. Clements) Date: Tue, 27 Jan 2004 08:55:34 -0600 (CST) Subject: [Beowulf] How to enable MPE environment in MPICH-1.2.5.2 In-Reply-To: <20040127060437.45877.qmail@web21501.mail.yahoo.com> References: <20040127060437.45877.qmail@web21501.mail.yahoo.com> Message-ID: By default, I believe mpe is built. But you may want to check the output from ./configure --help -B Brent Clements Linux Technology Specialist Information Technology Rice University On Mon, 26 Jan 2004, prakash borade wrote: > > how to i enable MPE environment in my cluster > do i have do build it explicitly or just some changes > while installing mpich > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From tkrewson at korrnet.org Tue Jan 27 14:15:55 2004 From: tkrewson at korrnet.org (Tom A Krewson) Date: Tue, 27 Jan 2004 14:15:55 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade Message-ID: Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dtj at uberh4x0r.org Tue Jan 27 16:35:40 2004 From: dtj at uberh4x0r.org (Dean Johnson) Date: Tue, 27 Jan 2004 15:35:40 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <1075239339.8252.2.camel@terra> On Tue, 2004-01-27 at 13:15, Tom A Krewson wrote: > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > They say that the upgraded machine will give off less heat. If I am not mistaken, its the same damn cpus in a different box and the power supply will be roughly the same ratings. Am missing something? -- -Dean _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at fnal.gov Tue Jan 27 17:33:53 2004 From: djholm at fnal.gov (Don Holmgren) Date: Tue, 27 Jan 2004 16:33:53 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> References: <1075239660.18659.14.camel@localhost.localdomain> Message-ID: Xserve boxes use new G5's based on a 90nm process. See http://www.theregister.co.uk/content/39/35057.html The claim is that power consumption per processor drops to ~ 25 Watt at 2 GHz from ~ 50 Watt at 1.8 GHz. Don Holmgren On Tue, 27 Jan 2004, Vann H. Walke wrote: > "The new system will take much less power and generate less heat and > free up space." > > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Removing video cards / hard drives? Do the video cards > really use a significant amount of power if they aren't being used? Or, > is there something fundamentally different about the Xserve? > > Vann > > On Tue, 2004-01-27 at 14:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jamesr at best.com Tue Jan 27 17:25:45 2004 From: jamesr at best.com (James Rogers) Date: 27 Jan 2004 14:25:45 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> References: <1075239660.18659.14.camel@localhost.localdomain> Message-ID: <1075242345.14220.7.camel@localhost.localdomain> On Tue, 2004-01-27 at 13:41, Vann H. Walke wrote: > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Removing video cards / hard drives? Do the video cards > really use a significant amount of power if they aren't being used? Or, > is there something fundamentally different about the Xserve? Different fab processes for the CPUs. The PPC970 XServes use a 90nm PPC970 versus the 130nm PPC970 used in the G5 towers. These are the first chips off the new process line, and use about half the power of the 130nm ones IIRC. -James Rogers jamesr at best.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jim at ks.uiuc.edu Tue Jan 27 17:33:38 2004 From: jim at ks.uiuc.edu (Jim Phillips) Date: Tue, 27 Jan 2004 16:33:38 -0600 (CST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239339.8252.2.camel@terra> Message-ID: As I understand, the Xserve G5 is using a newer process G5 that gives off less heat (and can therefore fit in a 1U chassis). -Jim On Tue, 27 Jan 2004, Dean Johnson wrote: > On Tue, 2004-01-27 at 13:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > > > They say that the upgraded machine will give off less heat. If I am not > mistaken, its the same damn cpus in a different box and the power supply > will be roughly the same ratings. Am missing something? > > -- > > -Dean > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From robl at mcs.anl.gov Tue Jan 27 18:02:17 2004 From: robl at mcs.anl.gov (Robert Latham) Date: Tue, 27 Jan 2004 17:02:17 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> <1075239339.8252.2.camel@terra> References: <1075239660.18659.14.camel@localhost.localdomain> <1075239339.8252.2.camel@terra> Message-ID: <20040127230217.GK25328@mcs.anl.gov> On Tue, Jan 27, 2004 at 03:35:40PM -0600, Dean Johnson wrote: > They say that the upgraded machine will give off less heat. If I am not > mistaken, its the same damn cpus in a different box and the power supply > will be roughly the same ratings. Am missing something? On Tue, Jan 27, 2004 at 04:41:01PM -0500, Vann H. Walke wrote: > "The new system will take much less power and generate less heat and > free up space." > > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Different cpus. http://www.theregister.co.uk/content/39/35057.html new 90nm chips throw off 24W at 1.4 GHz, down from 51 W at 1.8 GHz in the older 130nm models ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Labs, IL USA B29D F333 664A 4280 315B _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mbanck at gmx.net Tue Jan 27 17:54:49 2004 From: mbanck at gmx.net (Michael Banck) Date: Tue, 27 Jan 2004 23:54:49 +0100 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239339.8252.2.camel@terra> References: <1075239339.8252.2.camel@terra> Message-ID: <20040127225449.GM638@blackbird.oase.mhn.de> On Tue, Jan 27, 2004 at 03:35:40PM -0600, Dean Johnson wrote: > On Tue, 2004-01-27 at 13:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > They say that the upgraded machine will give off less heat. If I am not > mistaken, its the same damn cpus in a different box and the power supply > will be roughly the same ratings. Am missing something? The latest XServe are reportedly equipped with 90nm CPUs, not 120nm CPUs like the G5, thus apparently spreading far less heat. Michael _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From xyzzy at speakeasy.org Tue Jan 27 18:04:11 2004 From: xyzzy at speakeasy.org (Trent Piepho) Date: Tue, 27 Jan 2004 15:04:11 -0800 (PST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message-ID: On Tue, 27 Jan 2004, Joel Jaeggli wrote: > one wonders why they didn't just wait g5 the close relationship to apple > and the fact that xserves were basically justaround the corner. unless of > course it had to be in this years top-500 list and the motivation is as > much political/econmic as practical... Exactly. They said before that one of the reasons they went with apple was because they could get them the system in the timeframe (top-500 list it would seem) that they needed. The xserve has ECC memory, which will bring the performance of their cluster up to what they say it is. Their performance now is half the published value, since they need to run everything at least twice to have a reasonable assurance that their results are correct. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From walkev at presearch.com Tue Jan 27 16:41:01 2004 From: walkev at presearch.com (Vann H. Walke) Date: Tue, 27 Jan 2004 16:41:01 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <1075239660.18659.14.camel@localhost.localdomain> "The new system will take much less power and generate less heat and free up space." Ok, the space reduction is obvious. Anyone know why the power/heat is going down? Removing video cards / hard drives? Do the video cards really use a significant amount of power if they aren't being used? Or, is there something fundamentally different about the Xserve? Vann On Tue, 2004-01-27 at 14:15, Tom A Krewson wrote: > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From joelja at darkwing.uoregon.edu Tue Jan 27 17:22:05 2004 From: joelja at darkwing.uoregon.edu (Joel Jaeggli) Date: Tue, 27 Jan 2004 14:22:05 -0800 (PST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075239660.18659.14.camel@localhost.localdomain> Message-ID: radeon 9600's are probably good for 6-10w idle at the default clock speed... not all that much... just from the specs the dual g5 cluster node specifices 5.5amp peak vs 6.5amp for the dual g5 dekstop. I can't imagine that the difference would be too massive however between a dekstop and a node each with one drive. the space savings have to be enormous thought g5 towers are basically 5u even in the best circumstances.... one wonders why they didn't just wait g5 the close relationship to apple and the fact that xserves were basically justaround the corner. unless of course it had to be in this years top-500 list and the motivation is as much political/econmic as practical... joelja On Tue, 27 Jan 2004, Vann H. Walke wrote: > "The new system will take much less power and generate less heat and > free up space." > > Ok, the space reduction is obvious. Anyone know why the power/heat is > going down? Removing video cards / hard drives? Do the video cards > really use a significant amount of power if they aren't being used? Or, > is there something fundamentally different about the Xserve? > > Vann > > On Tue, 2004-01-27 at 14:15, Tom A Krewson wrote: > > Virginia Tech is upgrading their G5 cluster to Xserv.....very smart. > > > > > > http://www.reuters.co.uk/newsArticle.jhtml?type=technologyNews&storyID=4213216§ion=news > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja at darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From canon at nersc.gov Tue Jan 27 19:00:36 2004 From: canon at nersc.gov (canon at nersc.gov) Date: Tue, 27 Jan 2004 16:00:36 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message from Dean Johnson of "Tue, 27 Jan 2004 15:35:40 CST." <1075239339.8252.2.camel@terra> Message-ID: <200401280000.i0S00ax0015946@pookie.nersc.gov> I think the Xserve boxes use a version of the PPC with smaller feature size (90nm vs 130 or something like that). So its a different version of the same damn chip. :-) --Shane ------------------------------------------------------------------------ Shane Canon voice: 510-486-6981 PSDF Project Lead fax: 510-486-7520 National Energy Research Scientific Computing Center 1 Cyclotron Road Mailstop 943-256 Berkeley, CA 94720 canon at nersc.gov ------------------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Tue Jan 27 21:09:33 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Tue, 27 Jan 2004 21:09:33 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message-ID: I draw several conclusions from the VT cluster: - HPL is very cache-friendly. the dual-G5 has an impressive peak flops rating, but pretty mediocre memory bandwidth. anyone have a Stream result for a G5? I'd guess around 2.5 GB/s. - HPL also doesn't penalize fractional bisection bandwidth clusters, which is pretty interesting, since it *does* seem to reward post-gigabit interconnect. latency, perhaps? - the press isn't smart enough to notice that the VT cluster was far too unreliable (no ECC) and used an aweful lot of floorspace. it *is* neat that clusters are commodities nowadays, but is that news? somehow I don't think very many future clusters will be built by pizza-motivated volunteers... - a good PR stunt buys a lot of vendor discount. remember: the G5s list at close to $5M total, and the IB kit is pushing $2M. I'm guessing that VT can sell the G5's at a 30% discount. and Xserves cost about 50% more than desktops, so VT winds up with around $10M from Apple, for around $3M investment, nice! > The xserve has ECC memory, which will bring the performance of their cluster > up to what they say it is. oh, come on. MTBF is ~10 hours, so short jobs are perfectly fine ;) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 00:14:32 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 00:14:32 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <4015BC2F.8030405@inel.gov> Message-ID: On Mon, 26 Jan 2004, Andrew Shewmaker wrote: > Jakob Oestergaard wrote: > > I still guarantee you two things: > > 1) Your code will be longer > > 2) Your program will be slower > > > > As always, I love to be proven wrong ;) > > Here's a similar program...word frequency...instead of word count. > > http://www.cs.bell-labs.com/cm/cs/pearls/sec151.html > > His C++ code is shorter, but his C code is faster. > > With a trivial modification, his C code can meet your challenge too. > > http://www.cs.bell-labs.com/cm/cs/pearls/wordfreq.c There are two or three very interesting observations to make about both Jakob's challenge and this response. One is that the code is indeed longer. Or is it? One obvious feature of the C++ code is that it relies heavily on the STL. Is this cheating? Perhaps not exactly, or not quite, but neither is it totally fair WHEN COMPARING CODE LENGTH. One could encapsulate all the code in ANY such program into a single subroutine call and then say "look how short my program is", and although perhaps the library IS a standard feature of C++, it is also very much a library whose functionality could, in all probability, be pretty much duplicated in C. And of course as expected C++ appears to lose the speed challenge (which should be no great surprise, as the C code that DOESN'T encapsulate a solution is pretty close to as efficient as assembler and is in any event dominated by I/O. So let's give C++ points for having a powerful library. Let's take points away because it runs slower and because HAVING the library, especially a POWERFUL library, tends to channel your program design in ways that can take advantage of the library but that might not be horribly efficient. Let's also not give it too MANY points for shorter program length, both because the library hides a LOT of code, you betcha, and because using the library renders the program totally opaque to somebody that doesn't know C++ and the library. As in maybe it's just me, but when I look at Jakob's code I have no idea how it works. What are these routines? What do they do? Where's the logic? Oh, I can read the comments and see the keywords and sort of guess that all sorts of dark magic is being done; I just have no idea how it does it, any more than I would if one took Jakob's code or my code or anybody's code for the same task and wrapped it up in a single routine: int count_words_from_stdin(); main() { int count; count = count_words_from_stdin(); printf("I counted %d words\n",count); } Simple code -- pretty readable without any comments at all. However, not terribly understandable or useful, without being able to see what the routine count_words_from_stdin() in fact does in however many lines of code it hides. It could even be written IN assembler, and take a LOT of lines and be VERY efficient. It would be my hope and expectation that C++ is more than just a big library with some nifty stuff added on top of the already fairly hefty (but incrementally includeable, you only link to what you need) and relatively low level library set one has to work with in C. In fact, to convince me to try C++ you'd have to show me code that is NOT this compact -- code that in fact defines all the various class/objects used the hard way, uses (and I mean really USES) inheritance and protection and so forth, in ways that cannot be trivially duplicated in C code (and that, in fact, doesn't look very much LIKE the associated C code). The fundamental issue is whether or not "objects" are somehow managed better in C++. Beyond that, even its relatively strong typing and powerful library are not language issues, they are features added to C++ as a more recent revision of C and could probably be added to C as well, if anybody really cared. As the KR to ANSI transition shows, perhaps somebody one day will care, so this isn't an entirely moot point. You might argue, perhaps with complete accuracy, that WRITING such a high level library in C would be difficult because it exploits many of the language features of C++. I'd argue back that projects such as the GSL suggest otherwise -- the GSL code is certainly written in a very "object-oriented" way, and there are constructors and destructors and so forth where they make sense, but underlying those entities are ordinary C structs as the essential objects, and you can read the GSL headers and work directly with the structs if you prefer to work with the naked data for reasons of speed even in cases where they do provide a get/set method for accessing components of an object. In my own code I typically use lots of objects, and when it is reasonable to do so (when they are very complex or have multiple layers of pointers that have to be freed in a particular order) I use constructors and destructors for those objects. Lots of times I don't. I NEVER use "methods" for objects -- if I want to operate on the contents of a struct I operate on the contents of a struct, with a subroutine or without it as the needs of the code dictate. I think such a argument would eventually end up back where I left us. C++ a) shapes code in certain ways that go beyond the mere use of "objects" or "object oriented methods" in programming which is ALWAYS an option in any language that supports some sort of implementation of a struct or union or (in the worst possible case) even a raw data block with suitable offsets (fortran, anyone:-); b) provides strong typing, which I'm willing to acknowledge could be a real advantage (although one CAN cast void types in C and accomplish very subtle things one does have to be a programming god to do it and not occasionally get bitten by it); c) provides library extensions (quite possibly powerful extensions) that encapsulate a variety of useful tasks so that they can be done with a relatively few lines of the concept of YOUR code (plus all the hidden lines of library code, of course); d) at the cost of transparency and user-controllable logic and user access to some components of the code. This is just restating a) again, and around we'd go. If I were to write code to read lines from stdin, parse them into words on the basis of a set of separators e.g. space, tab, comma, period, put them into a vector of strings (uniqued or not) and count and print the final result on EOF it would certainly be longer than the C++, no doubt, but it would also be literally as fast as it could be at execution (probably comparing decently to assembler) and any programmer, even someone that didn't program in C but minimally understood the concept of the malloc family for memory management, could likely make out the logic. Preferring this to writing a program that just calls STL functions or somebody else's count_words_from_stdin() is doubtless a matter of taste, and there are doubtless programmers that span the range (taste-wise) from "real programmers only program in assembler" to perl or python (where the code would also be very short -- a single transparent loop and an output statement in perl) to "I like LISP". So tastes differ, experiences differ, and needs differ as well. Is the C++/C debate about taste, experiences, needs? When I talk to CPS faculty, they make no bones about it -- they teach C++ so that their students can learn structured programming in an environment with strong typing and lots of structure, and they feel that in today's programming world performance is less important than ease of programming and structure and having a powerful library. They compromise helping students develop skill in the "details" of procedural programming for the ability to quickly put together projects using big building blocks in a collegial/corporate programming environment. They KNOW (in many cases) that they are doing this; they just consider ability to be a good, diligent programmer that follows the rules and colors within the lines to be more important than knowing how pointers and malloc works. Is this a good tradeoff? I dunno. This is a bit different from when I learned to code -- there WERE no powerful libraries, and a whole lot of our programming exercises were what amounted to writing code to process lists of words or numbers in various ways, to sort, to build little code machines on a microscopic level within the confines of the language. In one course we even wrote an assembler simulator/interpreter and a compiler interpreter WRITTEN in our assembler interpreter (all punched out on cards for our IBM 370 in the upper level language du jour, which I think was PL/1 at the time). Useless for developing programming skills, unless you think that good programming skills are developed writing a really complex project that forces you to duplicate the way a computer really really works, and then write a complex program that works on your simulated computer... A lot of the kids learning programming now don't learn "how can one sort a list"; they learn how to call a routine named "sort", and extensions thereof (at least until they are a relatively upper level class). This works just marvey, of course, as long as the entity they need to sort and the sort criterion can be fed to the subroutine successfully in a form that it can work with... but one cannot help but feel that something is lost, just as something was lost even with C when people who needed random numbers just naively called "rand()" and were a bit surprised when, um, the results weren't horribly random... So perhaps my problem is that I'm just an Old Guy(tm) and punched too many cards in too many mediocre languages in my day. For me learning C was liberating, exhilerating, empowering, and I continue to discover clever little things it can do to this very day. Portable assembler, that's what C is, plus a bit of upper-level dressing (some of which COULD be a lot nicer, no doubt). But I will, just maybe, give C++ another try some day in the future when I have time and have found a really good book on it. Every time in the past I was working without a decent text and with a questionable (Microsoft or Borland) compiler. If g++ is, as you suggest, a pure superset of gcc (a few extra keywords don't matter) I can always give using a class a try sometime when I want to make a new struct, and see just how many hoops I have to jump through to do what I want to do. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 00:47:47 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 00:47:47 -0500 (EST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: <20040127095019.30138.qmail@web21506.mail.yahoo.com> Message-ID: On Tue, 27 Jan 2004, prakash borade wrote: > have any body tried RPC programing on Linux > i get > rpc unknown protocol error > i am trying to diaply the message on the remote host To do rpm programming you probably need a bunch of stuff configured and running that might or might not be turned off for security reasons. Check, for example, to see if the portmapper is running. Also check iptables and/or ipchains to see if they are running and if so if any rules block access to the portmapper. rpcinfo is also your friend: rgb at lilith|T:3#rpcinfo -p lilith program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 32768 status 100024 1 tcp 32768 status 391002 2 tcp 32769 sgi_fam As a general rule, I'd consider the portmapper and rpc stuff to be a moderate security risk; lilith lives inside firewalls only. I have had direct experience of systems cracked in the past through portmapper bugs, which perhaps makes me a bit paranoid. What exactly are you trying to do? rgb > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Wed Jan 28 01:26:17 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Tue, 27 Jan 2004 22:26:17 -0800 (PST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: Message-ID: hi ya robert On Wed, 28 Jan 2004, Robert G. Brown wrote: > rpcinfo is also your friend: > > rgb at lilith|T:3#rpcinfo -p lilith > program vers proto port > 100000 2 tcp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 32768 status > 100024 1 tcp 32768 status > 391002 2 tcp 32769 sgi_fam > > As a general rule, I'd consider the portmapper and rpc stuff to be a > moderate security risk; lilith lives inside firewalls only. I have had > direct experience of systems cracked in the past through portmapper > bugs, which perhaps makes me a bit paranoid. did you run the secure versions of each ?? secure rpc secure portmap secure nfs http://www.Linux-Sec.net/FileSystem/#NFS and yes, guess if security is an issue, dont use "nfs" but than again, what's the options .. at least encrypt as much as one can (afford) .. including the filesystem at "*.duke.edu" guess you folks would be a prime target... c ya alvin _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 00:42:54 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 00:42:54 -0500 (EST) Subject: [Beowulf] Clustering on Redhat9.0 In-Reply-To: <20040127064027.41294.qmail@web40808.mail.yahoo.com> Message-ID: On Mon, 26 Jan 2004, swapna ghosh wrote: > beowolf with PVM, i did not get any such document(installation > steps) till yet on Redhat 9.0. It will be really appreciated if anybody > points me the document/url of the procedure *installation of clustering > with beowolf PVM on RH-9.0* > > Lastly can anybody suggest me which one will be suitable for redhat > 9.0? Find the pvm and pvm-gui binary rpm's on whatever RH 9 medium you used for the install or from their website or from a mirror (such as the one at mirror.dulug.duke.edu). Download them if necessary. CD into their directory. As root, run: rpm -Uvh pvm*.rpm Configure ssh to permit login with no password. Use google to find a micro-HOWTO on how to do this, if you can't follow the man pages -- it is straightforward. Configure your environment to support PVM. It will work best/easiest if your two nodes share a home directory filesystem (NFS exported from one to the other) and have identical accounts. That's all there is to it, really. Login to one system, start pvm or xpvm (the latter is more fun) and add the other as a node. Write code, have fun. There is a functional pvm program template on my personal website under general. I think it is even semi-documented. Give it a try. This is also going to be the focus of my column in Cluster World in March and April. Consider subscribing. It has also been covered in past issues of Forrest's column in Linux Magazine (IIRC). Search LM's archive. Finally, there is a good book on introductory PVM from MIT press. Consider buying it via e.g. Amazon. rgb > > Thanks in advance > -Swapna > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hpcatcnc at yahoo.com Wed Jan 28 04:09:12 2004 From: hpcatcnc at yahoo.com (prakash borade) Date: Wed, 28 Jan 2004 01:09:12 -0800 (PST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: Message-ID: <20040128090912.51497.qmail@web21506.mail.yahoo.com> rgb thanks for your support well i am trying to wite an client server application ( monitoring sofware ) for my beowulf cluster tell me what would be efficient either sockets or RPC would open socket on each machine put an extra load on the cluster or is it managable --- "Robert G. Brown" wrote: > On Tue, 27 Jan 2004, prakash borade wrote: > > > have any body tried RPC programing on Linux > > i get > > rpc unknown protocol error > > i am trying to diaply the message on the remote > host > > To do rpm programming you probably need a bunch of > stuff configured and > running that might or might not be turned off for > security reasons. > Check, for example, to see if the portmapper is > running. Also check > iptables and/or ipchains to see if they are running > and if so if any > rules block access to the portmapper. > > rpcinfo is also your friend: > > rgb at lilith|T:3#rpcinfo -p lilith > program vers proto port > 100000 2 tcp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 32768 status > 100024 1 tcp 32768 status > 391002 2 tcp 32769 sgi_fam > > As a general rule, I'd consider the portmapper and > rpc stuff to be a > moderate security risk; lilith lives inside > firewalls only. I have had > direct experience of systems cracked in the past > through portmapper > bugs, which perhaps makes me a bit paranoid. > > What exactly are you trying to do? > > rgb > > > > > __________________________________ > > Do you Yahoo!? > > Yahoo! SiteBuilder - Free web site building tool. > Try it! > > http://webhosting.yahoo.com/ps/sb/ > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or > unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- > Robert G. Brown > http://www.phy.duke.edu/~rgb/ > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone: 1-919-660-2567 Fax: 919-660-2525 > email:rgb at phy.duke.edu > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctibirna at giref.ulaval.ca Wed Jan 28 09:49:10 2004 From: ctibirna at giref.ulaval.ca (Cristian Tibirna) Date: Wed, 28 Jan 2004 09:49:10 -0500 Subject: [Beowulf] C vs C++ challenge In-Reply-To: References: Message-ID: <200401280949.10516.ctibirna@giref.ulaval.ca> Le Mercredi 28 Janvier 2004 00:14, Robert G. Brown a ?crit: > As in maybe it's just me, but when I look at Jakob's code I have no idea > how it works. What are these routines? What do they do? Where's the > logic? Oh, I can read the comments and see the keywords and sort of > guess that all sorts of dark magic is being done; I just have no idea > how it does it, any more than I would if one took Jakob's code or my > code or anybody's code for the same task and wrapped it up in a single > routine: And here I argue that this is where C++'s strength becomes evident. STL is _standard_ (hence its name). Thus, for a C++ programmer, Jakob's code is comprehensible in one 15second long glimpse and this programmer can immediately use (trust?) the code. With the equivalent C code from the posted link, I personally have the problem that I need at least 15 minutes to read and understand it, and even then, I can't be sure I identified all gotchas and thus I can use it without major crashes and problems. I argue that the biggest problem with C up until glib and gsl (thus very recently) was exactly the lack of high level routines collected in a ubiquous library. > A lot of the kids learning programming now don't learn "how can one sort > a list"; they learn how to call a routine named "sort", and extensions > thereof (at least until they are a relatively upper level class). This > works just marvey, of course, as long as the entity they need to sort > and the sort criterion can be fed to the subroutine successfully in a > form that it can work with... And once again C++ fills the expectation here, IMO. Paraphrasing you: "This works just marvey... with C++ exactly because any... entity they need to sort and the sort criterion can be fed to the subroutine successfully and the subroutine will happily work with". This, of course, thanks to C++'s templates. And the result is even highly readable and quite compact usually (and this is the reason I am ready to cope with performance penalties, because I can put out code 10-20 times faster than in C -- OK, this is a subjective measure -- and that code will only run 2-3 times slower -- citing another poster "that's what clusters are for" ;-). > But I will, just maybe, give C++ > another try some day in the future when I have time and have found a > really good book on it. Lippman's "C++ Primer". Thanks for listening. -- Cristian Tibirna (1-418-) 656-2131 / 4340 Universit? Laval - Qu?bec, CAN ... http://www.giref.ulaval.ca/~ctibirna Professionnel de recherche GIREF ... ctibirna at giref.ulaval.ca ?tudiant PhD G?nie Chimique ... tibirna at gch.ulaval.ca _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jakob at unthought.net Wed Jan 28 08:54:02 2004 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed, 28 Jan 2004 14:54:02 +0100 Subject: [Beowulf] C vs C++ challenge In-Reply-To: References: <4015BC2F.8030405@inel.gov> Message-ID: <20040128135402.GM814@unthought.net> On Wed, Jan 28, 2004 at 12:14:32AM -0500, Robert G. Brown wrote: ... Guys, thanks a lot for your replies! Interesting, and a challenge. One I intend to follow up on. The C example ran 10 times faster than my C++ example here. So that's a 1-0 for "the world" vs. "me". However, I had to extend the stack-allocated variable that holds a temporary string, in order to stop the program from segfaulting. The only way to fix the problem was manual code inspection, as neither ElectricFence, Valgrind or plain GDB could display the problem because of the significant stack scribbling. So, I'll take the liberty of calling it 1-1 so far - I do intend to follow up on this to make it a win though ;) I have *tons* of work to do before saturday, where I board a plane for france, to spend next eight days skiing in the alps. I don't think I will be able to give you a proper followup until after the vacation. Say, in about one and a half week from now. However, once back, I'll be fully energized and ready to take on the world - my asbestos underwear will be ready, my tinfoil hat polished up, and by then I think my lithium prescription should have run out again. All in all, I'll be back! :) Cheers, / jakob _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Wed Jan 28 11:19:09 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed, 28 Jan 2004 11:19:09 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <200401280949.10516.ctibirna@giref.ulaval.ca> Message-ID: > And here I argue that this is where C++'s strength becomes evident. STL is > _standard_ (hence its name). Thus, for a C++ programmer, Jakob's code is > comprehensible in one 15second long glimpse and this programmer can I'm less enthusiastic about STL for a couple reasons. first, it's not always obvious whether it's being used correctly. STL is pretty big - more in the way that Java is basically a huge library, incidentally glued together with a mediocre scripting language and what amounts to emacs bytecodes. STL is probably a valuable toolkit for people who've invested in learning it. frankly, I haven't, though I'm a C++ programmer. I find that most of my experimental coding happens in Perl (which is quite a lot handier than STL, I think). and for non-experimental code, I'm usually too concerned with space or time efficiency to want to trust a big library like STL. > > another try some day in the future when I have time and have found a > > really good book on it. > > Lippman's "C++ Primer". I loved this in its first edition (more than a few years ago). in fact, I must have raved about it a lot, since someone made off with my copy. I looked at a later edition in a bookstore and was un-enthralled, though. perhaps I'm just surlier than I used to be, or perhaps the book changed. I remember being very impressed with Coplien's "Advanced C++". Stroustrup's books are also worth reading, at least parts of. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Tue Jan 27 23:43:54 2004 From: bill at math.ucdavis.edu (Bill Broadley) Date: Tue, 27 Jan 2004 20:43:54 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <20040128044354.GA10117@cse.ucdavis.edu> On Tue, Jan 27, 2004 at 09:09:33PM -0500, Mark Hahn wrote: > I draw several conclusions from the VT cluster: > > - HPL is very cache-friendly. the dual-G5 has an impressive peak > flops rating, but pretty mediocre memory bandwidth. anyone have a > Stream result for a G5? I'd guess around 2.5 GB/s. gcc -v # default for osx-10.3 Reading specs from /usr/libexec/gcc/darwin/ppc/3.3/specs Thread model: posix gcc version 3.3 20030304 (Apple Computer, Inc. build 1495) gcc -O1 stream_d.c second_wall.c Total memory required = 183.1 MB. Function Rate (MB/s) RMS time Min time Max time Copy: 2040.0621 0.0629 0.0627 0.0632 Scale: 2055.3702 0.0624 0.0623 0.0631 Add: 2141.2825 0.0901 0.0897 0.0908 Triad: 2100.9326 0.0917 0.0914 0.0921 -O2 Function Rate (MB/s) RMS time Min time Max time Copy: 2058.1437 0.0626 0.0622 0.0630 Scale: 1987.8218 0.0646 0.0644 0.0647 Add: 2150.3450 0.0897 0.0893 0.0902 Triad: 2155.9465 0.0893 0.0891 0.0895 -O3 Function Rate (MB/s) RMS time Min time Max time Copy: 2053.5855 0.0625 0.0623 0.0629 Scale: 1985.6382 0.0646 0.0645 0.0651 Add: 2148.7557 0.0897 0.0894 0.0901 Triad: 2155.1214 0.0894 0.0891 0.0896 -- Bill Broadley Computational Science and Engineering UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 13:01:19 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 13:01:19 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <20040128135402.GM814@unthought.net> Message-ID: On Wed, 28 Jan 2004, Jakob Oestergaard wrote: > I have *tons* of work to do before saturday, where I board a plane for > france, to spend next eight days skiing in the alps. I don't think I > will be able to give you a proper followup until after the vacation. > Say, in about one and a half week from now. > > However, once back, I'll be fully energized and ready to take on the > world - my asbestos underwear will be ready, my tinfoil hat polished up, > and by then I think my lithium prescription should have run out again. > > All in all, I'll be back! :) Awwwww, that hash-table example has tons of unnecessary overhead. Are you gonna make me actually WRITE a C variant to compete? Maybe I should start by posting a related program I wrote just for fun. I like to play hangman, and strategically a knowledge of letter frequency in the English language seemed like a useful thing to have. After all, everybody knows that 'e' is the most common letter, but what about the second, third, twenty-third most common letter? So one day, bored, I wrote letcount, which takes a filename as an argument, opens the file, reads it, and counts the letters. It was the first program I ever wrote that compiled and worked perfectly the first try! Not a lot of lines, but still...:-) Note that if all one wishes is to count the words, this program could be trivially modified with a snippet to determine word boundaries and increment a counter. If one wishes to count the words and save them (non-uniqued) in order, an additional snippet that moves each non-word-break character into a wor buffer, together with a realloc and a malloc (realloc the **char word array, malloc the spac for the word) would yield a scalable and storage-minimal solution. If one REALLY wants performance (and don't care about memory usage, sensible on a large memory modern system) one could realloc in largish blocks or simply malloc "more than enough" space (storing a million words less than or equal to rgb at lilith|T:144>echo supercalifragilisticexpialadotious | wc 1 1 35 in length requires a whopping 36 MB of memory) hard allocation of char words[1000000][36]; would yield the fastest possible access with no indirected pointer address resolution at all at the expense of being horribly wasteful in space that is probably idle anyway. Note that this wouldn't be efficient for smaller files as it probably takes the kernel some time to free up a block of memory that size; page size requests would presumably be much faster up to a point. Again, in addition to being perhaps the fastest possible code (note that scanning the entire stream of characters inline in this way probably would beat using strtok(), although not by much since this is likely pretty much what strtok does:-) the code (included below) is READABLE! At least I think so. I didn't even comment it the way I usually do -- open the file, stream it in, count the letters, trivially and inefficiently sort the count vector, print the sorted result in place. As the result of running the code on the linux words dictionary shows, it is pretty fast -- it literally returns a count as my finger is lifting from depressing the enter key, and the bulk of the DELAY is writing to stdout at 9600 baud or whatever an xterm supports, not the 0.03 seconds it actually takes to run. So there are LOTS of ways to solve your challenge problem in C; static memory allocation (wasteful but fast, especially for big files/streams), dynamic memory allocation (probably optimal, especially if one allocates roughly a page at a time on demand as the previous page is filled and maintain your own table of word boundaries as pointer offsets in an otherwise linear buffer), dynamic allocation the simpler and probably more readable/portable way (a word at a time in a char **words list), dynamic allocation of the words as "simple objects" (build a linked list struct of some sort to hold words and perhaps indices for use in a later sort or uniqueing step), and of course the hash. In each case the performance tradeoffs for using one memory management approach or the other are fairly transparent, and in each case one can choose between using a tool like e.g. strtok or doing it yourself. Of course, some of these have significant risks associated with them in the hands of a novice programmer or result in code that isn't horribly portable -- writing the incoming stream as rapidly as possible into a resize-on-demand buffer and recording word offsets in a linear integer (pointer) vector requires visualizing the way the memory space is laid out and is very close to the way one would likely do this in assembler, but accessing a word as (char *) buffer[offset[13]] isn't everybody's cup of tea (including mine, unless the code HAD to be really fast). So sure, give it your best shot in C++ using C++ constructs, but I guarantee you won't beat the best one can do in C and probably won't come terribly close unless you write the C++ code, after all, in C. I'll see if I can spare an hour or so this afternoon to hack letcount into wordcount using one or another of the buffer storage mechanisms I mentioned above -- the buffer[offset[]] actually seems kinda cute, come to think of it, but I may use strtok() instead of my code as a base to make the code appear shorter. Especially since it RETURNS the offset in buffer -- I think my entire program would be reading the file into a big buffer and a single while loop where strtok extracts the offsets, puts them into a vector, and counts. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu rgb at lilith|T:141>/usr/bin/time ./letcount /usr/share/dict/linux.words Opening /usr/share/dict/linux.words for counting e: 42715 i: 31459 s: 29652 a: 28984 r: 27064 n: 26981 t: 24614 o: 21613 l: 19484 c: 15018 d: 13857 u: 11723 g: 10342 p: 10073 m: 9807 h: 7812 b: 7371 y: 6006 f: 4932 v: 3973 k: 3209 w: 3073 z: 1631 x: 1054 j: 727 q: 682 Command exited with non-zero status 7 0.03user 0.00system 0:00.40elapsed 7%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (93major+14minor)pagefaults 0swaps %< snip snip ======================================================= #include main(int argc, char **argv) { char filename[128],buf[256]; int i,j,biggest,bigind,count[26]; FILE *infile; if(argc != 2){ fprintf(stderr,"Usage: letcount filename\n"); exit(1); } argv++; sprintf(filename,"%s",*argv); printf("Opening %s for counting\n",filename); infile = fopen(filename,"r"); for(i=0;i<26;i++) count[i] = 0; while(fgets(buf,256,infile)!= NULL){ /* printf("%s",buf); */ i = 0; while(buf[i] != 0){ if(buf[i] > 64 && buf[i] < 92) count[buf[i]-65]++; if(buf[i] > 96 && buf[i] < 124) count[buf[i]-97]++; i++; } } for(i=0;i<26;i++){ biggest = 0; for(j=0;j<26;j++){ if(count[j] > biggest) { biggest = count[j]; bigind = j; } } printf("%c: %d\n",bigind+97,biggest); count[bigind] = 0; } } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at fnal.gov Wed Jan 28 13:16:26 2004 From: djholm at fnal.gov (Don Holmgren) Date: Wed, 28 Jan 2004 12:16:26 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: With gcc 2.95.3, -O3, I get on a dual 2.0 GHz G5: Total memory required = 183.1 MB, walltime clock ------------------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Copy: 2848.7459 0.0450 0.0449 0.0457 Scale: 2851.4134 0.0453 0.0449 0.0457 Add: 3470.4630 0.0556 0.0553 0.0557 Triad: 3456.0981 0.0557 0.0556 0.0561 and with the beta IBM compiler (VAC 6.0, -O5): ------------------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Copy: 2923.5762 0.0439 0.0438 0.0441 Scale: 2566.9303 0.0500 0.0499 0.0502 Add: 2304.2857 0.0834 0.0833 0.0836 Triad: 2339.1539 0.0822 0.0821 0.0823 The data bus is a bit unconventional, in that it's split - 32 bits read-only, 32 bits write-only. The two halves can run at the same time. For moving data, that configuration is nicely balanced. For computations, it may not be so well balanced. For example, most of the flops in one of our codes occur doing small complex matrix-vector multiplies. One typical kernel reads 96 single precision operands, and writes only 6 single precision results. The effective memory bandwidth for this is much reduced - instead of a 1 GHz FSB, it's more like an effective 500 MHz FSB. Don Holmgren Fermilab On Tue, 27 Jan 2004, Mark Hahn wrote: > I draw several conclusions from the VT cluster: > > - HPL is very cache-friendly. the dual-G5 has an impressive peak > flops rating, but pretty mediocre memory bandwidth. anyone have a > Stream result for a G5? I'd guess around 2.5 GB/s. > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 13:55:42 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 13:55:42 -0500 (EST) Subject: [Beowulf] C vs C++ challenge In-Reply-To: <200401280949.10516.ctibirna@giref.ulaval.ca> Message-ID: On Wed, 28 Jan 2004, Cristian Tibirna wrote: > And once again C++ fills the expectation here, IMO. Paraphrasing you: "This > works just marvey... with C++ exactly because any... entity they need to sort > and the sort criterion can be fed to the subroutine successfully and the > subroutine will happily work with". This, of course, thanks to C++'s > templates. And the result is even highly readable and quite compact usually > (and this is the reason I am ready to cope with performance penalties, > because I can put out code 10-20 times faster than in C -- OK, this is a > subjective measure -- and that code will only run 2-3 times slower -- citing > another poster "that's what clusters are for" ;-). Well, let's get serious here. A cluster is a large and costly instrument. You don't really mean that you are willing to waste 1/2-2/3 of the practical capacity of up to a million dollars worth of hardware by writing in C++'s generic objects in order to save a few weeks of writing truly efficient code in C. That makes no sense from a CBA point of view -- especially when it requires one to purchase 2-3x as many nodes to get the same time of completion of the job OR requires the project to run 2-3x as long (3 year instead of one, for example). You can write and debug a hell of a lot of C code in a year. You can also avoid being burned in effigy by competing groups wanting access to the resource. Everything else you say, well, that's just precisely the attitude of many C++ programmers and teachers of same and is perfectly OK with me. In fact, it is the attitude of most perl and python coders as well -- why bother with all the overhead of a compiler when you can have run-time checking, dynamically self-rewriting code (there's a clever perl trick -- code that writes itself!) threads and more, especially string parsing and regular expressions in conditionals), and never have to actually manage memory at all? For many tasks they are only a few times slower than compilers (they effectively tokenize/"compile" and then run library calls just like a compiled program) but they can make developing various classes of applications must faster. In fact, if I were to actually want to write a word counter to parse stdin (where I/O is already a rate limiter and speed is irrelevant) I'd almost certainly use perl myself, just as others (who like a more OO interface) would use python. Neither perl nor python is suitable for use on a beowulf to do real work; C++ doubtless is, but only if it is used "correctly" so that parts and data structures that need to be in plain old C for reasons of efficiency are. This is why I'm ecumenical, however strong my own prejud-- I mean "calmly reasoned and wise opinions":-). There is a spectrum of languages, and there are tradeoffs on ease of use and encapsulation across the spectrum. As computers get ever faster and larger, the overhead cost associated with using interpreters and higher-level languages and libraries goes down not in relative terms but in absolute ones, often to the point of irrelevancy. Notable exceptions include HIGH PERFORMANCE COMPUTING (the principle topic of this list:-) where performance is a PRIMARY focus, and a whole variety of tasks that tend to be studied by computer scientists because of their unfavorable scaling properties and consequent large demands on whatever system is running them. These categories are far from static as Moore's Law does its thing -- the reason PL/1 is a vanished language is in part because it WAS a high-overhead language for its era (IBM mainframe computing) and the resource it wasted (CPU time at truly exorbitant rates) was EXPENSIVE. The fact that the compiler would take monkeytype and turn it into SOME sort of program with its syntax error correction probably didn't help...given that one paid for the bad runs along with the good.;-) Also, Real Computer Scientists work out algorithms and port code and turn it into a library routine and suddenly a difficult problem DOES have a pretty good and even efficient solution in a given language/library. And then there is ATLAS, where the library isn't just good, it is dynamically self-optimizing. So by all means, seek your own comfort level, and optimize tradeoffs in accord with your own abilities and needs... > Lippman's "C++ Primer". > > Thanks for listening. Thanks for the pointer to the book. I'll give it a look next time I'm in B&N. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 11:29:18 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 11:29:18 -0500 (EST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: Message-ID: On Tue, 27 Jan 2004, Alvin Oga wrote: > > hi ya robert > > On Wed, 28 Jan 2004, Robert G. Brown wrote: > > > rpcinfo is also your friend: > > > > rgb at lilith|T:3#rpcinfo -p lilith > > program vers proto port > > 100000 2 tcp 111 portmapper > > 100000 2 udp 111 portmapper > > 100024 1 udp 32768 status > > 100024 1 tcp 32768 status > > 391002 2 tcp 32769 sgi_fam > > > > As a general rule, I'd consider the portmapper and rpc stuff to be a > > moderate security risk; lilith lives inside firewalls only. I have had > > direct experience of systems cracked in the past through portmapper > > bugs, which perhaps makes me a bit paranoid. > > did you run the secure versions of each ?? > secure rpc > secure portmap > secure nfs This was a few years ago. Remember, I've been a Unix sysadmin since, oh, 1986 or 1987 or thereabouts. As in I remember the Morris worm, a slew of sendmail and lpd attacks, an emacs bug, syslogd attacks, the good old days where any user armed with a userspace nfs kit could access any user's NFS space, and much, much, more. The portmapper exploit was particularly annoying and was one of the things (to remain vaguely on topic:-) that propelled us from Slackware to Red Hat as a departmental base -- our previous head sysadmin (by this time I was a "part time" sysadmin of our department network and our primary sysadmin kept pretty tight control of the LAN so I basically did ver little actual admin work) had left our department, and his first replacement turned out to be a midlevel disaster and permitted sigificant spread in the actual set of installed packages on all the various systems in the department. As in there were probably three or four different revisions and patchlevels of portmap running on different systems. Well, THAT sysadmin went away, I found myself once again in charge of the network literally overnight until we hired a replacement, and almost the first thing that happened was that about three or four departmental systems turned out to be vulnerable to a portmap buffer overwrite attack, were exploited, and I spent week of my life cleaning up. This is a salutary lesson to the novices out there; to quote from A Thousand Nights and a Night (various places) "it should be graven on the eyelid corners as a warner to whomso would be warned". It is pretty much essential to: a) keep systems patched and up to date, especially wrt security fixes but of course performance and bugfixes are also important. b) keep all systems in a LAN patched at the SAME level -- all it takes is one bad system and your whole network is compromised, as many attackers who get a user account can find a way to promote to root, and a user who gets root on one system in a LAN has many routes to eventually promote to root on the department servers and LAN. Heterogeneity is an enemy of good systems administration, as it dilutes effort and limits scalability. c) In case it isn't obvious, this suggests that sound management practice includes something like: pick a packaged distribution that is aggressively maintained, especially wrt security; use kickstart or other scripting/imaging tools to construct system templates so that all your systems will be as similar as possible (same packages, same general configuration); use yum or related tools in other distributions to fully automate system updates from a common package repository, so that keeping the package repository up to date is precisely equivalent to keeping all your systems up to date. d) Minimize exposure. This means that unless you need a daemon or service, don't run it. Buffer overwrite attacks via "essential" systems daemons have probably been the single most common avenue of successful attack in the history of the non-Windows internet, and they have potentially disasterous scaling properties as the Morris worm (and more contemporaneously, the Slammer) clearly demonstrated. Worms like this might be classified as dangerous terrorist munitions, as they can cause more economic damage in a day than a hundred car bombs, especially with the Internet at this point being the base of a huge amount of economic activity. e) Since there are always daemons and services that are more or less REQUIRED inside a LAN in order not to lose productivity THAT way run some sort of firewall to isolate access to the exposed ports to only the hosts that need the access. It is relatively easy to police these hosts and their users with tool-enhanced vigilance (syslog-ng etc) and a sucker rod (read man syslogd if the reference escapes you:-) to school any local users who want to try any 1337 crap or mad skills). iptables or ipchains permit one remarkably fine-grained control of port access rights even within a LAN -- every host runs its own mini-firewall. NFS in particular and most RPC services are very definition, BTW, of services that should have carefully regulated access. NFS should pretty much NEVER be used over a WAN (this may have changed, of course, but Old Guys will always be paranoid and not really believe that it has:-). f) A lot of folks organize their internal LAN in security layers, with highly restricted access to servers, very limited privileges to clients (to make it more difficult for a successful exploit of a client to be promotable to the entire LAN and especially to the servers, ESPECIALLY to install repositories). There is no substitute for vigilance, though -- most exploits are discovered by admins or users noting and reporting something "odd" or something "broken" and following up aggressively. Again kickstart/yum/PXE are your friend, as cleanup nowadays is just Ctrl-Alt-Del into a full reinstall of a compromised client, coupled with a scrubbing of any affected user accounts. g) Keep Good Backups. With highly automated install/maintenance tools, cleanup is relatively cheap, but you may well have to restore corrupted files or home directories or (in the worst of cases) rebuild servers or repositories. Security is always a trade-off; you give up the ability to do certain kinds of work easily and quickly in order to achieve a "hardened" status, but the lost work and productivity is also a "cost", one that accrues linearly in time, and can easily exceed the "cost" of a successful exploit. Many linuces now install by default into what amounts to a tuckee-tail hard firewalled mode with access to virtually all ports blocked via iptables and turned off in chkconfig. This is a good thing, but it can mean that lots of things "mysteriously" don't work unless you know what your are doing and know how to enable them. Which is ALSO a good thing; if you don't know how to turn them on, could you possibly know enough to be vigilant and keep a system with them enabled secure? Learning the one should always be accompanied by learning the other... > at "*.duke.edu" guess you folks would be a prime target... I think that everybody is a prime target. Many, even most, exploits these days are driven by fully automated engines; there are relatively few incidents of pimply-faced cracker script-kiddies targeting systems one at a time (working through the usual string of intermediary successes). Even the script-kids use tools that simply sweep IP spaces probing for open/vulnerable ports, and the worms combine a successful sweep with exploitation of the port, implanting of a clone of the worm, and a continuation of propagation process. At Duke, at least, we have a VERY competent security officer (Chris Cramer) who does a damn fine job coping with and minimizing our exposure on an institutional basis. He's a beowulfer, BTW, and might even read (eventually) these words of glowing praise and wonder what the hell I want out of him now:-). (Lunch, Chris, Lunch. Love those free lunches:-) He is backed up by an equally excellent collective sysadmin group; they keep mot of the organized LAN spaces professionally secured without his constant browbeating so he can focus on the bleeding, supperating wound that is Windows. Today is of course not a good day for him on that front as YAWE (yet another windows exploit) got enough traction yesterday that answering my email was distinctly tedious (note that I do NOT use the word Hi in my subject line above, not wishing to be procmailed out by others in the same pickle). I'm certain that it is popping up in the dorms if nowhere else, as we obviously have a hard time controlling "private" student systems in the dorms, who are both "inside" and "outside" Duke's trusted space (as outside as we can make them, but nevertheless possessing trusted access to certain resources). So sure, Duke is a major target by virtue of its large IP-space volume if nothing else, but it is also fairly aggressively and competently policed. rgb > > c ya > alvin > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rmiguel at usmp.edu.pe Wed Jan 28 14:46:31 2004 From: rmiguel at usmp.edu.pe (Richard Miguel) Date: Wed, 28 Jan 2004 14:46:31 -0500 Subject: [Beowulf] about compilers for cluster beowulf References: Message-ID: <000001c3e5db$8d6d1280$1101000a@cpn.senamhi.gob.pe> Hi ... this is my first post . So, i have a beowulf cluster with 27 nodes PIV 2.4 Ghz Intel, my question is if is possible that Fortran Intel compiler and PGI compiler are installed in a same machine without conflicts (server node). thanks for yor answers.. R. Miguel _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nordquist at geosci.uchicago.edu Wed Jan 28 15:13:46 2004 From: nordquist at geosci.uchicago.edu (Russell Nordquist) Date: Wed, 28 Jan 2004 14:13:46 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <1075320826.5224.7.camel@vitalstatistix> in other news here is an article written by the dean of the college of engineering on the bigMac http://news.com.com/2010-7337-5149087.html i finally know what is was built to do....solve "grand challenge" problems. BTW, does anyone out there own a smallMac cluster that would be willing to field some questions offline. were looking at smallish (~30 cpus) g5 cluster. we have several linux clusters but this is my would be my first stab at Xserves. russell _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 28 17:47:37 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 28 Jan 2004 17:47:37 -0500 (EST) Subject: [Beowulf] About a survey request In-Reply-To: Message-ID: Has anyone else received an HPC survey request from: microsoftresearch at surveysite.com I received one today. I declined the survey as they will not make the results available to the participants/community. Just curious. Doug _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jan 28 11:42:05 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed, 28 Jan 2004 11:42:05 -0500 (EST) Subject: [Beowulf] Remote procedure call (error) In-Reply-To: <20040128090912.51497.qmail@web21506.mail.yahoo.com> Message-ID: On Wed, 28 Jan 2004, prakash borade wrote: > rgb thanks for your support > > well i am trying to wite an client server application > ( monitoring sofware ) for my beowulf cluster > tell me what would be efficient either sockets or RPC Raw sockets. Eschew RPC. You should also just visit http://www.phy.duke.edu/~rgb/Beowulf/beowulf.php or brahma and pull xmlsysd and wulfstat. It IS client/server monitoring software, fully GPL, and will build and install and (hopefully) just plain work for you. If you are interested in this as a "learning project" and not because you just need a monitoring package, consider using xmlsysd as a base and contributing to the project rather than reinventing all the wheels that it contains. For example, it already contains forking daemon and xinetd code (so the daemon part is "done". It already knows how to open most of the "interesting" files in /proc, read them and encapsulate their contents in XML and send them to clients on demand, and even uses tricks such as rewinding the files instead of closing and reopening them to minimize the fairly significant overhead associated with opening a file in the first place. I think you'll find that xmlsysd is pretty efficiently written and lightweight and stable and functional and non-leaky and (due to its use of xml to encapsulate results) scalable and accessible to a variety of clients. It is even moderately extensible -- it is fairly easy to add a new thing to be monitored, once you know the code (not quite "follow the instructions" easy, though -- at this point I'm probably the only person who could do it without extensively studying the code). The thing that really needs to be done is develop more monitoring (client) tools. wulfstat isn't bad, really, especially for smallish clusters, but it would be lovely to have "gwulfstat" (gtk) and "wwulfstat" (web) monitoring tools. It would also be lovely to have some command line tools that could be backgrounded and poll results in a timed loop and write them out to a named logfile (both as a historical log trace of node load and to facilitate the creation of tools that could graph the traces and generate reports). SO if you are interested at all in helping out and joining the project, I could certainly use the help. I have too many projects running all at the same time to give this one the full attention it deserves. rgb > > would open socket on each machine put an extra load > on the cluster or is it managable > > --- "Robert G. Brown" wrote: > > On Tue, 27 Jan 2004, prakash borade wrote: > > > > > have any body tried RPC programing on Linux > > > i get > > > rpc unknown protocol error > > > i am trying to diaply the message on the remote > > host > > > > To do rpm programming you probably need a bunch of > > stuff configured and > > running that might or might not be turned off for > > security reasons. > > Check, for example, to see if the portmapper is > > running. Also check > > iptables and/or ipchains to see if they are running > > and if so if any > > rules block access to the portmapper. > > > > rpcinfo is also your friend: > > > > rgb at lilith|T:3#rpcinfo -p lilith > > program vers proto port > > 100000 2 tcp 111 portmapper > > 100000 2 udp 111 portmapper > > 100024 1 udp 32768 status > > 100024 1 tcp 32768 status > > 391002 2 tcp 32769 sgi_fam > > > > As a general rule, I'd consider the portmapper and > > rpc stuff to be a > > moderate security risk; lilith lives inside > > firewalls only. I have had > > direct experience of systems cracked in the past > > through portmapper > > bugs, which perhaps makes me a bit paranoid. > > > > What exactly are you trying to do? > > > > rgb > > > > > > > > __________________________________ > > > Do you Yahoo!? > > > Yahoo! SiteBuilder - Free web site building tool. > > Try it! > > > http://webhosting.yahoo.com/ps/sb/ > > > _______________________________________________ > > > Beowulf mailing list, Beowulf at beowulf.org > > > To change your subscription (digest mode or > > unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > > -- > > Robert G. Brown > > http://www.phy.duke.edu/~rgb/ > > Duke University Dept. of Physics, Box 90305 > > Durham, N.C. 27708-0305 > > Phone: 1-919-660-2567 Fax: 919-660-2525 > > email:rgb at phy.duke.edu > > > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or > > unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > __________________________________ > Do you Yahoo!? > Yahoo! SiteBuilder - Free web site building tool. Try it! > http://webhosting.yahoo.com/ps/sb/ > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ctibirna at giref.ulaval.ca Wed Jan 28 15:17:54 2004 From: ctibirna at giref.ulaval.ca (Cristian Tibirna) Date: Wed, 28 Jan 2004 15:17:54 -0500 Subject: [Beowulf] C vs C++ challenge In-Reply-To: References: Message-ID: <200401281517.54712.ctibirna@giref.ulaval.ca> Le Mercredi 28 Janvier 2004 13:55, Robert G. Brown a ?crit: > > 2-3 times slower -- citing another poster "that's what clusters are for" > > ;-). > > Well, let's get serious here. A cluster is a large and costly > instrument. You don't really mean that you are willing to waste 1/2-2/3 > of the practical capacity of up to a million dollars worth of hardware > by writing in C++'s generic objects in order to save a few weeks of > writing truly efficient code in C. Of course not. Thus the smiley. And of course I (very often) get frustrated at the "laziness" of the popular C++ compilers as well at the performance drop (that sometimes becomes unacceptable by any standards). But getting frustrated at compilers is a bearable psychosociological inconvenience compared with the rage (accompanied sometimes with physical violence ;-) that badly written code in a too flexible (or too poor) language can inflict on me. There's the aspect of software engineering in temporally, spacially and knoweldgeably sparse teams. In projects with many millions LOCs, developed over many (>5) years, employing 30 to 40 persons (many of them passage students with brief interventions), especially projects with a pedagogic penchant, anything less strict and with a lesser standardized high level library than C++ wouldn't cut it. But you of course already touched the topic of pedagogic intents before in this thread. I dare extract a conclusion (as the thread drifts fastly;-): "the right tool for the job at hand" remains the essential wisdom to keep in mind when comes to choosing languages. -- Cristian Tibirna (1-418-) 656-2131 / 4340 Laval University - Quebec, CAN ... http://www.giref.ulaval.ca/~ctibirna Research professional at GIREF ... ctibirna at giref.ulaval.ca PhD Student - Chemical Engng ... tibirna at gch.ulaval.ca _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Wed Jan 28 14:58:16 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Wed, 28 Jan 2004 11:58:16 -0800 (PST) Subject: [Beowulf] C vs C++ challenge - hw too In-Reply-To: Message-ID: hi ya robert On Wed, 28 Jan 2004, Robert G. Brown wrote: > On Wed, 28 Jan 2004, Cristian Tibirna wrote: > > > (and this is the reason I am ready to cope with performance penalties, > > because I can put out code 10-20 times faster than in C -- OK, this is a > > subjective measure -- and that code will only run 2-3 times slower -- citing > > another poster "that's what clusters are for" ;-). yupperz... people will do things the fastest/easiest way .. :-) > Well, let's get serious here. A cluster is a large and costly > instrument. You don't really mean that you are willing to waste 1/2-2/3 > of the practical capacity of up to a million dollars worth of hardware > by writing in C++'s generic objects in order to save a few weeks of > writing truly efficient code in C. That makes no sense from a CBA point > of view -- especially when it requires one to purchase 2-3x as many > nodes to get the same time of completion of the job OR requires the > project to run 2-3x as long (3 year instead of one, for example). You > can write and debug a hell of a lot of C code in a year. You can also > avoid being burned in effigy by competing groups wanting access to the > resource. sometimes ... people dont do the "full total analysis" and/or it can be swayed one way vs another sometimes cheaper to hire another programmer to do the work .. while one carries on with one's own area of expertise -- and dont forget to add the issues of hw costs to the list of too - soemtimes i wonder why people want dual-cpu or quad-cpu - found 1 guy ( company ) that says that their custom code runs linearly with increasing number of cpu's ... ( they write custom finite element stuff ) - otherwise it might be cheapr to run 2 jobs on 2 machines than 1.5 jobs on one dual-cpu machines ( aka 2 machines ) have fun alvin -- latest fun puzzle ... one can fit 4 1Us in one 4U shelf space one can also fit 10 1Us in one 4U shelf space ( magic of atx blades .. 2.5x the capacity in the same volume ) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From kdunder at sandia.gov Wed Jan 28 15:35:49 2004 From: kdunder at sandia.gov (Keith D. Underwood) Date: Wed, 28 Jan 2004 13:35:49 -0700 Subject: [Beowulf] C vs C++ challenge In-Reply-To: <200401280949.10516.ctibirna@giref.ulaval.ca> References: <200401280949.10516.ctibirna@giref.ulaval.ca> Message-ID: <1075305637.4039.31.camel@thinkpad> > And once again C++ fills the expectation here, IMO. Paraphrasing you: "This > works just marvey... with C++ exactly because any... entity they need to sort > and the sort criterion can be fed to the subroutine successfully and the > subroutine will happily work with". This, of course, thanks to C++'s > templates. And the result is even highly readable and quite compact usually > (and this is the reason I am ready to cope with performance penalties, > because I can put out code 10-20 times faster than in C -- OK, this is a > subjective measure -- and that code will only run 2-3 times slower -- citing > another poster "that's what clusters are for" ;-). So, in some very real examples, the performance difference between an existing Fortran code and the first version of a C++ code was more like a factor of 10. When you look at the impact of that in real dollars on, say, an ASCI scale computing platform that costs $90+ million, it is huge. If that code is used for very long, you are wasting 90% of the machine or $80+ million (and we won't count the very significant cost of electricity or cooling for a machine of that scale). So, then the programmers get to spend their days trying to get some of that performance back. In one example, a Fortran code was being replaced by a C++ code. The C++ code was way too slow and the core computational routines were replaced with their Fortran equivalent. The code was still a factor of 2 to 3 behind. By the end of it, a lot more effort had been spent on trying to make the C++ code run fast than was spent when the C++ code was scrapped and the Fortran code was updated instead. Ease of programming is good, but the performance implications cannot be ignored. Even a factor of 2 is a lot of money when you talk about a large computing resource. Even a large computing resources as inexpensive as the Virginia Tech Mac cluster. Keith _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From deadline at linux-mag.com Wed Jan 28 17:59:37 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed, 28 Jan 2004 17:59:37 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: Message-ID: The first thought I had was "what will they do with all the old systems?" Then it hit me. They put a fancy sticker on each box that says "This machine was part of the third fastest supercomputer on the planet Nov. 2003" or something similar. Also put a serial number on the tag and provide a "certificate of authenticity" from VT. My guess is they can make a little bit on the whole deal. I wager they would sell rather quickly. Alumni eat this kind of thing up. For those interested, my old www.cluster-rant.com site has morphed into the new www.clusterworld.com site. You can check out issue contents, submit stories, check out the polls, and rant about clusters. Doug -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From patrick at myri.com Wed Jan 28 19:03:27 2004 From: patrick at myri.com (Patrick Geoffray) Date: Wed, 28 Jan 2004 19:03:27 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075320826.5224.7.camel@vitalstatistix> References: <1075320826.5224.7.camel@vitalstatistix> Message-ID: <40184DCF.8090800@myri.com> Hi Russel, Russell Nordquist wrote: > to field some questions offline. were looking at smallish (~30 cpus) g5 > cluster. we have several linux clusters but this is my would be my first > stab at Xserves. You can also look at the JS-20 blades from IBM, 28 cpus in 7U. Maybe a little pricey though. Patrick -- Patrick Geoffray Myricom, Inc. http://www.myri.com _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From nordquist at geosci.uchicago.edu Wed Jan 28 20:58:42 2004 From: nordquist at geosci.uchicago.edu (Russell Nordquist) Date: Wed, 28 Jan 2004 19:58:42 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <40184DCF.8090800@myri.com> References: <1075320826.5224.7.camel@vitalstatistix> <40184DCF.8090800@myri.com> Message-ID: <1075341521.5224.34.camel@vitalstatistix> On Wed, 2004-01-28 at 18:03, Patrick Geoffray wrote: > Hi Russel, > > Russell Nordquist wrote: > > to field some questions offline. were looking at smallish (~30 cpus) g5 > > cluster. we have several linux clusters but this is my would be my first > > stab at Xserves. > > You can also look at the JS-20 blades from IBM, 28 cpus in 7U. Maybe a > little pricey though. > I am working with an IBM rep to get a login to a JS-20 blade. From the ibm website a JS20 blade w/512M ram is $2,699.00. The chassis is 2,789.00 so.... around $40.5K. i think i'm missing a GiGe switch module though. in comparison, for g5 Xserves are $3k x 14 = $42. so it's similar. however the g5's are 2Ghz the JS-20's are 1.6Ghz..... on a related note....in the next day I am going to post (or prompt someone to post) about an interesting issue we are having with some code. basically it will run on g4's and g5's, but not Xeons/Athlons due to what appears to be underflow issues. the odd thing is that it doesn't work on an Opteron (in 64 bit mode) either. call this a teaser since i am going to get the author of the code to explain a bit more about it. this particular problem has me a bit vexed....i can't figure if it's a architecture or compiler issue. more on this tomorrow. russell > Patrick _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bill at math.ucdavis.edu Wed Jan 28 21:20:22 2004 From: bill at math.ucdavis.edu (Bill Broadley) Date: Wed, 28 Jan 2004 18:20:22 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <40184DCF.8090800@myri.com> References: <1075320826.5224.7.camel@vitalstatistix> <40184DCF.8090800@myri.com> Message-ID: <20040129022022.GA19317@cse.ucdavis.edu> On Wed, Jan 28, 2004 at 07:03:27PM -0500, Patrick Geoffray wrote: > Hi Russel, > > Russell Nordquist wrote: > >to field some questions offline. were looking at smallish (~30 cpus) g5 > >cluster. we have several linux clusters but this is my would be my first > >stab at Xserves. > > You can also look at the JS-20 blades from IBM, 28 cpus in 7U. Maybe a > little pricey though. An IBM guy said (informally) approximately $2500 list per blade. Same ball park as the .edu dual g5 1u compute node. -- Bill Broadley Information Architect Computational Science and Engineering UC Davis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dc at hemeris.com Thu Jan 29 09:55:55 2004 From: dc at hemeris.com (dc) Date: Thu, 29 Jan 2004 15:55:55 +0100 Subject: [Beowulf] C vs C++ challenge (java version) Message-ID: Jakob Oestergaard wrote: > Let me throw out a challenge: > > "Write a program in C that reads from standard input until EOF, and > prints the number of distinct words found in that input" > > > I use an RB-tree to store the words I find, and print the number of > elements in the final tree. > > A comparable C solution shouldn't be more than a few thousand lines ;) > > Oh, right, I cheated and used STL. Well, STL is a big part of why C++ > is really handy, but go right ahead and use "glib" or whatever. > > I still guarantee you two things: > 1) Your code will be longer > 2) Your program will be slower > > As always, I love to be proven wrong ;) Here is another try at that, this time in Java. C++ code compiled with g++ 3.3 (-O3), times are 'real time' reported by the bash time function, the client and server columns correspond to java runs using the -client and -server options respectively. Java compiler and VM are the hotspot 1.4. file size C++ j client j server wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s big.txt 39389424 0m29.120s 0m13.972s 0m12.776s vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s Oh right, I cheated and used Java standard collections ;-) Seriously tough, Java is not as slow as some seem to believe. And optimizing the code at run-time (hotspot, code morphing,...) has the advantage that the optimizer has access to potentially useful information, both about the code itself and about the executing environment. -------------------------------------------------------------- import java.util.*; import java.util.regex.Pattern; import java.io.*; public class wordcount { public static void main (String args[]) { HashSet hs = new HashSet(); try { BufferedReader input = new BufferedReader(new InputStreamReader(System.in)); String s, words[]; Pattern p = Pattern.compile("\\s"); while ((s = input.readLine()) != null) { words = p.split(s); for (int i = 0; i < words.length; i++) hs.add(words[i]); } hs.remove(""); } catch (Throwable e) { e.printStackTrace(); } System.out.println("Words: " + hs.size()); } } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From djholm at fnal.gov Thu Jan 29 14:45:05 2004 From: djholm at fnal.gov (Don Holmgren) Date: Thu, 29 Jan 2004 13:45:05 -0600 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <40192F0C.5010809@lmco.com> References: <40192F0C.5010809@lmco.com> Message-ID: Here you go: 1 Copy of the IBM VAC version: Function Rate (MB/s) RMS time Min time Max time Copy: 2956.1260 0.0434 0.0433 0.0435 Scale: 2620.4804 0.0491 0.0488 0.0495 Add: 2308.5529 0.0833 0.0832 0.0837 Triad: 2371.3380 0.0810 0.0810 0.0812 2 simultaneous copies: Function Rate (MB/s) RMS time Min time Max time Copy: 1671.2143 0.0820 0.0766 0.0886 Scale: 1462.9393 0.0915 0.0875 0.0941 Add: 1296.6226 0.1519 0.1481 0.1575 Triad: 1378.7155 0.1425 0.1393 0.1507 Function Rate (MB/s) RMS time Min time Max time Copy: 1569.2244 0.0845 0.0816 0.0893 Scale: 1470.4361 0.0891 0.0870 0.0929 Add: 1324.9507 0.1505 0.1449 0.1545 Triad: 1481.1049 0.1448 0.1296 0.1509 These results are consistent with the Fortran MPI version, run with np=1 and np=2: bash-2.05a$ mpirun -np 1 stream_mpi ... ---------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 2910.9423 .0220 .0220 .0220 Scale: 2577.4299 .0249 .0248 .0249 Add: 2385.3296 .0403 .0402 .0404 Triad: 2456.4980 .0391 .0391 .0392 bash-2.05a$ mpirun -np 2 stream_mpi ... ---------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 2780.1300 .0465 .0460 .0469 Scale: 2779.6478 .0464 .0460 .0469 Add: 2649.7750 .0728 .0725 .0736 Triad: 2777.8189 .0698 .0691 .0709 Don On Thu, 29 Jan 2004, Jeff Layton wrote: > Don, > > Is there any way you could run two copies of Stream > at the same time, one on each CPU? I'd like to see the > effect of the memory subsystem. > > Thanks! > > Jeff _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jeffrey.b.layton at lmco.com Thu Jan 29 11:04:28 2004 From: jeffrey.b.layton at lmco.com (Jeff Layton) Date: Thu, 29 Jan 2004 11:04:28 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: References: Message-ID: <40192F0C.5010809@lmco.com> Don, Is there any way you could run two copies of Stream at the same time, one on each CPU? I'd like to see the effect of the memory subsystem. Thanks! Jeff > > With gcc 2.95.3, -O3, I get on a dual 2.0 GHz G5: > > > Total memory required = 183.1 MB, walltime clock > ------------------------------------------------------------- > Function Rate (MB/s) RMS time Min time Max time > Copy: 2848.7459 0.0450 0.0449 0.0457 > Scale: 2851.4134 0.0453 0.0449 0.0457 > Add: 3470.4630 0.0556 0.0553 0.0557 > Triad: 3456.0981 0.0557 0.0556 0.0561 > > and with the beta IBM compiler (VAC 6.0, -O5): > > ------------------------------------------------------------- > Function Rate (MB/s) RMS time Min time Max time > Copy: 2923.5762 0.0439 0.0438 0.0441 > Scale: 2566.9303 0.0500 0.0499 0.0502 > Add: 2304.2857 0.0834 0.0833 0.0836 > Triad: 2339.1539 0.0822 0.0821 0.0823 > > > The data bus is a bit unconventional, in that it's split - 32 bits > read-only, 32 bits write-only. The two halves can run at the same time. > > For moving data, that configuration is nicely balanced. For > computations, it may not be so well balanced. For example, most of the > flops in one of our codes occur doing small complex matrix-vector > multiplies. One typical kernel reads 96 single precision operands, and > writes only 6 single precision results. The effective memory bandwidth > for this is much reduced - instead of a 1 GHz FSB, it's more like an > effective 500 MHz FSB. > > Don Holmgren > Fermilab > > > On Tue, 27 Jan 2004, Mark Hahn wrote: > > > I draw several conclusions from the VT cluster: > > > > - HPL is very cache-friendly. the dual-G5 has an impressive peak > > flops rating, but pretty mediocre memory bandwidth. anyone > have a > > Stream result for a G5? I'd guess around 2.5 GB/s. > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Dr. Jeff Layton Aerodynamics and CFD Lockheed-Martin Aeronautical Company - Marietta _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pathscale.com Thu Jan 29 15:25:09 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 29 Jan 2004 12:25:09 -0800 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075341521.5224.34.camel@vitalstatistix> References: <1075320826.5224.7.camel@vitalstatistix> <40184DCF.8090800@myri.com> <1075341521.5224.34.camel@vitalstatistix> Message-ID: <20040129202509.GB1630@greglaptop.internal.keyresearch.com> On Wed, Jan 28, 2004 at 07:58:42PM -0600, Russell Nordquist wrote: > on a related note....in the next day I am going to post (or prompt > someone to post) about an interesting issue we are having with some > code. basically it will run on g4's and g5's, but not Xeons/Athlons due > to what appears to be underflow issues. The G5 in particular has the usual IBM feature that some float computations are done as double, with the answer only being truncated when variables are stored to memory. The IBM compiler has a switch that rounds things more normally. If this feature allows your code to run, then your code is probably the most guilty party. -- greg _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 29 15:07:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 29 Jan 2004 15:07:08 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (C version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, dc wrote: > > I still guarantee you two things: > > 1) Your code will be longer > > 2) Your program will be slower > > > > As always, I love to be proven wrong ;) > > > Here is another try at that, this time in Java. > > file size C++ j client j server ... > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s And here is a version in C. It is longer, no question. It does its own memory management, in detail, in pages (which should be nearly optimally efficient). It is moderately robust, and smart enough to recognize all sorts of separators (when counting words, separators matter -- hence this program will find more words than e.g. wc because it splits things differently). It's timing (compared to a REALLY efficient wc): rgb at ganesh|T:406>/usr/bin/time wordcount shaks12.txt Opening shaks12.txt for counting Total Number of Words = 902303 Command exited with non-zero status 31 0.46user 0.02system 0:00.47elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (93major+2334minor)pagefaults 0swaps rgb at ganesh|T:407>/usr/bin/time wc shaks12.txt 124456 901325 5582655 shaks12.txt 0.17user 0.02system 0:00.19elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (130major+21minor)pagefaults 0swaps 0.47 wall clock time is far, far faster than C++ or java. It is still 2x and change wc, but it is far from optimally efficient, for many reasons: a) It uses fgets() instead of raw read(). It would be easy to change this but I'm lazy and haven't bothered as even the version I've come up with blows away the competition. At a guess, given an I/O dominated process it would cut processing time roughly in half if I did. b) It uses strsep() to tokenize lines of input read directly into a buffer (but a line at a time). wc works with the character stream directly (and splits words on fewer characters) and is much faster. c) It saves all the words as separate strings in the buffer and creates a list of offsets so each word can be addressed by its own index. This meets the requirement that not only do I count the words, I read them into a list where they can be accessed individually and (presumably) processed further. d) As noted, I'm lazy. If anyone complains about accessing the words via words+word_offset[i], I should point out that it is absolutely trivial to repack the offsets (now that we have them) in a char **wordlist so that the words can be addressed as wordlist[i]. If anyone complains that we "just got a char ** list of words", I should ALSO point out that with trivial modifications the code could allocate a struct { char *word; int index; int length; } *wordlist; and fill IT in so that each word is in a word "object". Neither of these steps would significantly increase the time required (more than a few percent). As for the code, well, 144 lines including all comments isn't horrible, and if I cut the comments and empty lines it's probably about 80. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu #include #include #include #include #define PAGE 4096 #define K 1024 static char delim[] = { ' ', /* blank */ (char) 9, /* tab */ ',', /* comma */ (char) 10, /* LF */ (char) 13, /* CR */ '.', /* . */ '!', /* ! */ '?', /* ? */ '"', /* " */ '*', /* * */ '<', /* < */ '>', /* > */ '@', /* @ */ '[', ']', '(', ')', '{', '}', '+', '=', '/', '\\', '#', ':', (char) 0 /* terminator */ }; main(int argc, char **argv) { char filename[128],*word,*words,*words_head,*words_tail,*words_last; int *word_offset; size_t offset,word_length,words_length,wordlist_length,word_count,i; FILE *infile; if(argc != 2){ fprintf(stderr,"Usage: wordcount filename\n"); exit(1); } argv++; sprintf(filename,"%s",*argv); printf("Opening %s for counting\n",filename); infile = fopen(filename,"r"); /* * The word list as a single buffer. We allocate two pages to start with, and * reallocate whenever the free space in the buffer drops below one page. We * also maintain a pointer to the beginning of the unused part of this buffer. */ words_length = 2*PAGE; words = (char *) malloc(words_length); words_head = words; words_tail = words + words_length; /* * We do the exact same thing here, sort of. We allocate a PAGE of offsets * (relative to the words head!) and reallocate whenever the number remaining * drops below K. i will both index the wordlist and count the words as we go. */ wordlist_length = PAGE; word_offset = (int *) malloc(wordlist_length*sizeof(int)); word_count = 0; /* * Each pass through the while loop reads a single line from * infile into words at offset wptr (current tail). We will * write the code to manage lines up to 1K in length. */ word_offset[0] = 0; while(fgets(words_head,(words_tail - words_head),infile)!= NULL){ /* * Check to add a PAGE to the words vector/buffer. Since realloc * may move pointer, we have to recompute offsets. */ if((words_tail - words_head) < PAGE) { words_length += PAGE; /* save offset */ offset = words_head - words; words = (char *) realloc(words,words_length); /* displace by saved offset */ words_head = words + offset; words_tail = words + words_length; } /* * Check to add a PAGE to the wordlist vector/buffer. Here we * don't have to worry about offsets. */ if((wordlist_length - word_count) < K) { wordlist_length += PAGE; word_offset = (int *) realloc(word_offset,wordlist_length*sizeof(int)); } /* * Now we rip through the line. */ while(1){ /* Get a word */ word = strsep(&words_head,delim); if(words_head == NULL) break; /* Save its address for after we are done with this line */ words_last = words_head; /* Save its offset in the offset vector */ if(word[0] != (char)0){ word_offset[word_count++] = word - words; /* for debugging offsets etc */ /* printf("%d: word at %0x = %s, offset is %d\n",word_count,word,word,word_offset[word_count - 1]); */ } } /* * strsep finishes parsing a line with a NULL. Restore the right pointer. */ words_head = words_last; } /* * That's about it. We print the results. * (or not, in a speed trial). for(i=0;i (Alvin Oga's message of "Tue, 27 Jan 2004 22:26:17 -0800 (PST)") References: Message-ID: Alvin Oga writes: > On Wed, 28 Jan 2004, Robert G. Brown wrote: > >> As a general rule, I'd consider the portmapper and rpc stuff to be a >> moderate security risk; lilith lives inside firewalls only. I have had >> direct experience of systems cracked in the past through portmapper >> bugs, which perhaps makes me a bit paranoid. > > did you run the secure versions of each ?? > secure rpc > secure portmap > secure nfs Well, "secure" in that context means "encrypted". That might protect your traffic against snoopers, but you're still as exposed to security holes due to portmapper bugs. I've seen too many RPC related buffer overrun holes in the past to expose any RPC services to the Internet at large, whether "secure" or not. -- Leif Nixon Systems expert ------------------------------------------------------------ National Supercomputer Centre Linkoping University ------------------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Thu Jan 29 15:47:38 2004 From: agrajag at dragaera.net (Jag) Date: Thu, 29 Jan 2004 15:47:38 -0500 Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075320826.5224.7.camel@vitalstatistix> References: <1075320826.5224.7.camel@vitalstatistix> Message-ID: <1075409258.4438.19.camel@pel> On Wed, 2004-01-28 at 15:13, Russell Nordquist wrote: > in other news here is an article written by the dean of the college of > engineering on the bigMac > > http://news.com.com/2010-7337-5149087.html > Interesting article. It seems they don't know what their cluster is called. The article wants to call it 'Virginia Tech X', whereas the website for the cluster wants to call it Terascale: http://computing.vt.edu/research_computing/terascale/ Another interesting thing is the article mentions Deja Vu, a proprietary piece of software. If you check this site (http://mac.excaliburworld.com/teracluster.html), it seems that Deja Vu is written in house. I actually went up and saw the cluster last month. When I was there, they still weren't production ready, and didn't have any real users. Now they're suddenly upgrading everything.. sounds very curious to me. Btw, when I was there, it was indicated that the Deja Vu software was written by a faculty member in their computer science department. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 29 18:22:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 29 Jan 2004 18:22:08 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, Selva Nair wrote: > But this one does not count unique words, does it? Hmmm. I read "distinct words" as being words distinguished from one another by separators, not as unique words. As in "two and two" are one line, three distinct words, but only two unique words. I'll have to hack a bit tomorrow to make the C do only distinct words. For that the hash table solution is probably optimal, but I might as well see how much extra time the loops take. > Here is the script: > > #!/bin/awk -f > { > for(i = 1; i <= NF; i++) { > if (words[$i]) continue; > words[$i] = 1 ; > ++nwords; > } > } > END { > printf "Number of distinct words = %i\n", nwords; > } I always have liked awk;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Jan 29 17:56:41 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Thu, 29 Jan 2004 17:56:41 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: <1075408742.7848.73.camel@squash.scalableinformatics.com> Message-ID: On Thu, 29 Jan 2004, Joe Landman wrote: > On Thu, 2004-01-29 at 09:55, dc wrote: > > file size C++ j client j server > > wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s > > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > > big.txt 39389424 0m29.120s 0m13.972s 0m12.776s > > vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s > > Where did these files come from? Would be nice to try out > non-C++/non-Java solutions with. At least shaks12.txt is from project gutenberg and shows up as a direct link to there, first thing, from Google on a "shaks12 txt" search. I haven't tried the rest of them -- if a tool works for Shakespeare it SHOULD work for any of the rest. The only dicey point will arise when the filesize starts to compete with available free memory, as one of the rules of the "competition" is to save the list of words in addressable form. At least my algorithm should scale boringly linearly up to where the system is forced to really clear buffers and cache and then swap. Theirs appear to as well. So I'd suggest using Shakespeare from Gutenberg and maybe /usr/share/dict/linux_words as a baseline, although the latter might vary from version to version of linux. I'd also suggest that each contestant run their code at least once or twice with commented out lines that demonstrate that their code actually is parsing the text and saving the words (and that what it saves ARE words, or at least word-like tokens) in an addressable retrievable form. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at yahoo.com Thu Jan 29 18:59:37 2004 From: raysonlogin at yahoo.com (Rayson Ho) Date: Thu, 29 Jan 2004 15:59:37 -0800 (PST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <20040129202509.GB1630@greglaptop.internal.keyresearch.com> Message-ID: <20040129235937.80631.qmail@web11404.mail.yahoo.com> IBM did not release its compilers for the G4, so I believe he used gcc to compiler his code. May be it's due to the little vs big-endian issue?? Rayson --- Greg Lindahl wrote: > The G5 in particular has the usual IBM feature that some float > computations are done as double, with the answer only being truncated > when variables are stored to memory. The IBM compiler has a switch > that rounds things more normally. If this feature allows your code to > run, then your code is probably the most guilty party. > > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf __________________________________ Do you Yahoo!? Yahoo! SiteBuilder - Free web site building tool. Try it! http://webhosting.yahoo.com/ps/sb/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From selva.nair at utoronto.ca Thu Jan 29 17:59:13 2004 From: selva.nair at utoronto.ca (Selva Nair) Date: Thu, 29 Jan 2004 17:59:13 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, rgb wrote: > On Thu, 29 Jan 2004, dc wrote: > > > > I still guarantee you two things: > > > 1) Your code will be longer > > > 2) Your program will be slower > > > > > > As always, I love to be proven wrong ;) > > > > > > Here is another try at that, this time in Java. > > > > file size C++ j client j server > ... > > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > > And here is a version in C. It is longer, no question. It does its own > memory management, in detail, in pages (which should be nearly optimally > efficient). It is moderately robust, and smart enough to recognize all > sorts of separators (when counting words, separators matter -- hence > this program will find more words than e.g. wc because it splits things > differently). But this one does not count unique words, does it? Here is my version in awk. It beats C++ by 1 line in length and 1.5 times in speed (1.86s versus 2.83s elapsed time) with shaks12.txt as input. [selva at scala distinct_words]$ wc shaks12.txt 124456 901325 5458199 shaks12.txt This copy of shaks12.txt has been filtered by dos2unix. Timings: First my awk script (with GNU awk 3.1.0) [selva at scala distinct_words]$ /usr/bin/time ./dwc.awk shaks12.txt Number of distinct words = 67505 1.82user 0.04system 0:01.86elapsed 99%CPU Now the original C++ code (compiled by g++ 2.96). [selva at scala distinct_words]$ /usr/bin/time ./dwc < shaks12.txt Words: 67505 2.79user 0.04system 0:02.83elapsed 100%CPU Here is the script: #!/bin/awk -f { for(i = 1; i <= NF; i++) { if (words[$i]) continue; words[$i] = 1 ; ++nwords; } } END { printf "Number of distinct words = %i\n", nwords; } Selva _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eccf at super.unam.mx Thu Jan 29 20:16:13 2004 From: eccf at super.unam.mx (Eduardo Cesar Cabrera Flores) Date: Thu, 29 Jan 2004 19:16:13 -0600 (CST) Subject: [Beowulf] PVM BEOLIN & SMP Message-ID: Are there anybody that could run sucessfully PVM BEOLIN conf into a cluster using both of the processors of the dual machines? Cause i got the next, output (i'm using OpenPBS to submit my job): [eccf at mixbaal BEOLIN]$ more ~/pvml.10006 [pvmd pid1213] 01/29 19:43:50 PROC_LIST=nodo13:nodo13:nodo12:nodo12:nodo11:nodo1 1:nodo10:nodo10: [pvmd pid1213] 01/29 19:43:50 8 nodes in list. [t80040000] 01/29 19:43:50 nodo13 (192.168.1.13:32770) BEOLIN 3.4.4 [t80040000] 01/29 19:43:50 ready Thu Jan 29 19:43:50 2004 [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60001 [t80040000] 01/29 19:43:50 mpp_find: Task not found [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60001 [t80040000] 01/29 19:43:50 mpp_find: Task not found [t80040000] 01/29 19:43:50 mpp_new() sp=806f510 sp->n_link=806f510 [t80040000] 01/29 19:43:50 mpp_new() sp=806f510 sp->n_link=80828e8 [t80040000] 01/29 19:43:50 mppload(): Forking to nodo13 [t80040000] 01/29 19:43:50 mppload(): Forking to nodo13 [t80040000] 01/29 19:43:50 mppload(): Forking to nodo12 [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60000 [t80040000] 01/29 19:43:50 mpp_find looking for t60002 [t80040000] 01/29 19:43:50 [t60000] [t80040000] 01/29 19:43:50 [t60000] Estoy en el nodo nodo13 [t80040000] 01/29 19:43:50 mpp_free() sp=80828e8 sp->n_link=806f510 [t80040000] 01/29 19:43:50 mpp_free() n_ptype = 0, ptype = 0 [eccf at mixbaal BEOLIN]$ And that's it. The process aparently is running but nothing else happens!!!! Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 9947.mixbaal.su jlgr q_250m_2 chel 1247 2 -- 200mb 08:00 R 01:42 nodo04/0+nodo03/0 9952.mixbaal.su eccf q_500m_4 myjob.tcsh 1246 4 -- 500mb 16:00 R 00:00 nodo09/1+nodo09/0+nodo08/1+nodo08/0+nodo07/1+nodo07/0+nodo05/1+nodo05/0 9953.mixbaal.su eccf q_500m_4 shel 1146 4 -- 500mb 16:00 R 00:00 nodo13/1+nodo13/0+nodo12/1+nodo12/0+nodo11/1+nodo11/0+nodo10/1+nodo10/0 This is my shell scrit: #!/bin/bash #PBS -q q_500m_4n_2h #PBS -l mem=500Mb #PBS -l nodes=4:ppn=2 #PBS -o temporal1 #PBS -e tiempo_temporal cd /home/staff/eccf/pvm3/bin/BEOLIN cat $PBS_NODEFILE > salida export PROC_LIST=`gawk 'BEGIN{ORS=":"} {getline a < FILENAME; print a}' salida` echo "El valor de PROC_LIST = `echo $PROC_LIST`" /local/pvm3/lib/BEOLIN/pvm /home/staff/eccf/pvm3/bin/BEOLIN/master1 echo "halt" | pvm Any idea? If i run 8 processes in 8 different nodes it perfectly runs!!!! Thanks a lot cafe _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu Jan 29 15:39:02 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Thu, 29 Jan 2004 15:39:02 -0500 Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: References: Message-ID: <1075408742.7848.73.camel@squash.scalableinformatics.com> On Thu, 2004-01-29 at 09:55, dc wrote: > file size C++ j client j server > wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > big.txt 39389424 0m29.120s 0m13.972s 0m12.776s > vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s Where did these files come from? Would be nice to try out non-C++/non-Java solutions with. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Thu Jan 29 14:56:34 2004 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Thu, 29 Jan 2004 13:56:34 -0600 Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: References: Message-ID: <20040129195634.GG23892@mikee.ath.cx> On Thu, 29 Jan 2004, dc wrote: > Jakob Oestergaard wrote: > > > Let me throw out a challenge: > > > > "Write a program in C that reads from standard input until EOF, and > > prints the number of distinct words found in that input" > > > > > > > I use an RB-tree to store the words I find, and print the number of > > elements in the final tree. > > > > A comparable C solution shouldn't be more than a few thousand lines ;) > > > > Oh, right, I cheated and used STL. Well, STL is a big part of why C++ > > is really handy, but go right ahead and use "glib" or whatever. > > > > I still guarantee you two things: > > 1) Your code will be longer > > 2) Your program will be slower > > > > As always, I love to be proven wrong ;) > > > Here is another try at that, this time in Java. > > C++ code compiled with g++ 3.3 (-O3), times are 'real time' reported by the > bash time function, the client and server columns correspond to java runs using > the -client and -server options respectively. > Java compiler and VM are the hotspot 1.4. > > > file size C++ j client j server > wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s > shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s > big.txt 39389424 0m29.120s 0m13.972s 0m12.776s > vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s > > Oh right, I cheated and used Java standard collections ;-) > > Seriously tough, Java is not as slow as some seem to believe. > > And optimizing the code at run-time (hotspot, code morphing,...) has the > advantage that the optimizer has access to potentially useful information, both > about the code itself and about the executing environment. > > -------------------------------------------------------------- > import java.util.*; > import java.util.regex.Pattern; > import java.io.*; > > > public class wordcount { > > public static void main (String args[]) { > HashSet hs = new HashSet(); > try { > BufferedReader input = new BufferedReader(new InputStreamReader(System.in)); > > String s, words[]; > Pattern p = Pattern.compile("\\s"); > > while ((s = input.readLine()) != null) { > words = p.split(s); > for (int i = 0; i < words.length; i++) > hs.add(words[i]); > } > hs.remove(""); > } > catch (Throwable e) { > e.printStackTrace(); > } > System.out.println("Words: " + hs.size()); > } > } Do you have some time to run the same test using the gcc front-end to java and posting those results? Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From scyld at jasons.us Fri Jan 30 00:44:26 2004 From: scyld at jasons.us (scyld at jasons.us) Date: Fri, 30 Jan 2004 00:44:26 -0500 (EST) Subject: [Beowulf] Virginia Tech upgrade In-Reply-To: <1075409258.4438.19.camel@pel> References: <1075320826.5224.7.camel@vitalstatistix> <1075409258.4438.19.camel@pel> Message-ID: <20040130004018.J28799@torgo.bigbroncos.org> On Thu, 29 Jan 2004, Jag wrote: > Interesting article. It seems they don't know what their cluster is > called. The article wants to call it 'Virginia Tech X', whereas the > website for the cluster wants to call it Terascale: > http://computing.vt.edu/research_computing/terascale/ When I was there last month they referred to it as "System X" and as part of the Terascale project. YMMV. -Jason ----- Jason K. Schechner - check out www.cauce.org and help ban spam-mail. "All HELL would break loose if time got hacked." - Bill Kearney 02-04-03 ---There is no TRUTH. There is no REALITY. There is no CONSISTENCY.--- ---There are no ABSOLUTE STATEMENTS I'm very probably wrong.--- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From mikee at mikee.ath.cx Fri Jan 30 08:16:19 2004 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Fri, 30 Jan 2004 07:16:19 -0600 Subject: [Beowulf] [OT] HA clusters vs. grids? Message-ID: <20040130131619.GA32707@mikee.ath.cx> Ok, so this is on the edge of the mailing list.... I have been told I must suggest a plan for the purchase of a new server, that I can't low-ball the situation by only adding core to a few servers. One pair of servers is in a AIX HA cluster, one node is the primary and the other is simply a hot-spare (always in standby). I understand what the HA is doing with its heartbeats, fail over, and such. If I have to buy a new server, is it possible I could improve uptime by having a grid? This is a homegrown application, the primary application in the company, and it uses SNA, ISAM files, and lots of serial connections. Can this type of application (that needs these external devices) work in a grid environment? Mike _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From dc at hemeris.com Fri Jan 30 04:31:45 2004 From: dc at hemeris.com (Didier Carlier) Date: Fri, 30 Jan 2004 10:31:45 +0100 Subject: [Beowulf] C vs C++ challenge (java version) In-Reply-To: References: Message-ID: <1F08D319-5307-11D8-84E2-003065E5B6BA@hemeris.com> On Jan 29, 2004, at 23:56, Robert G. Brown wrote: > On Thu, 29 Jan 2004, Joe Landman wrote: > >> On Thu, 2004-01-29 at 09:55, dc wrote: >>> file size C++ j client j server >>> wrnpc10.txt 3282452 0m2.746s 0m1.941s 0m1.643s >>> shaks12.txt 5582655 0m4.476s 0m3.321s 0m2.842s >>> big.txt 39389424 0m29.120s 0m13.972s 0m12.776s >>> vbig.txt 196947120 2m23.882s 1m5.707s 1m2.350s >> >> Where did these files come from? Would be nice to try out >> non-C++/non-Java solutions with. > > At least shaks12.txt is from project gutenberg and shows up as a direct > link to there, first thing, from Google on a "shaks12 txt" search. I > haven't tried the rest of them -- if a tool works for Shakespeare it > SHOULD work for any of the rest. The only dicey point will arise when > the filesize starts to compete with available free memory [...] Exactly. The rnpc10.txt is "War and Peace" (also from Gutenberg), big.txt is simply 12 copies of wrnp10.txt concatenated, and vbig,txt is 60 copies of wrnp10.txt concatenated. I included the 2 large files because Java makes little sense for run-times of a couple of seconds where more time is spent compiling the code than executing it. Also I didn't think that run-times of 1 or 2 seconds were of particular interest on this list. Looking at the (elapsed time)/(file size) ratio for the 2 big files, both the C++ and the Java elapsed times seem to scale linearly with the file size, but (* in this particular test *) java beats c++ by a factor of 2.3 for large files and only 1.6 for the smaller one. Didier _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Fri Jan 30 13:37:53 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Fri, 30 Jan 2004 13:37:53 -0500 Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: References: Message-ID: <1075487873.8580.36.camel@squash.scalableinformatics.com> On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > Note that dwc finds 67505 "distinct words". This is very interesting, This is what I find in the Perl version as well. I wish I could get that Java code compiling on this platform so I could compare the two. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From selva.nair at utoronto.ca Fri Jan 30 16:39:19 2004 From: selva.nair at utoronto.ca (Selva Nair) Date: Fri, 30 Jan 2004 16:39:19 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: <1075487873.8580.36.camel@squash.scalableinformatics.com> Message-ID: On Fri, 30 Jan 2004, Joe Landman wrote: > On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > > > Note that dwc finds 67505 "distinct words". This is very interesting, > > This is what I find in the Perl version as well. I wish I could get > that Java code compiling on this platform so I could compare the two. The awk script I posted earlier counts tokens separated by [:blank:] and I guess that is what the C++ version also does -- tokens not words as rgb carfully pointed out. I was trying to mimic the C++ code, so case and punctuations were not ignored. Anyway, there is no easy answer to what a word is -- e.g, simplistic removal of punctuations makes 3 "words" out of 100,000,000. But all that is beside the point and the point itself has nothing to do with beowulf. It is still tempting to to change the awk script to match RGB's C version ;-) My first attempt with 3 extra lines of awk gives: [selva at scala distinct_words]$ /usr/bin/time ./dwc.awk < shaks12.txt Total number of words = 902299 Number of distinct words = 31384 2.04user 0.03system 0:02.06elapsed 100%CPU Only 10% slower than the earlier version that was not punctuation/capitalisation-agnostic. The total number of words is still off by 4 and unique words are about 6000 smaller than the C version (see footnote). It may not be hard to hunt down the difference, but that does not appear worthwhile. I think the point has been made -- use your favourite scripting language for quick answers or even for testing algorithms. "For everything else there is C" -- be it for extensibility or performance or all the other reasons eloguantly expressed by rgb. I can not see any reason to learn or use C++ simply for sets or hashes or trees in STL. By the way, the performance of C++ STL appears to vary by up to a factor of 10 between different versions of g++. The number I quoted (2.8 s) was the fastest I could get and that was with g++ 2.96. For some reason version 3.0.x runs much slower here.. Selva Footnotes: 1. Googling shows that Shakespeare used 31,534 different words and a total of 884,647 words in his complete works (http://www-math.cudenver.edu/~wbriggs/qr/shakespeare.html) And some statistical projections indicate he probably knew that many more words but did not use. In any case, the awk result is surprisingly close to 31,534 inspite of the much smaller sample we used -- there is some over counting due to numeric tokens and other "junk" in the Gutenberg text, but I am still surprised by the close agreement. Also I wonder why the C version comes up with as high as 37,000+ unique words. 2. FWIW, here is the new script: #!/bin/awk -f # Count number of distinct or unique words in a file # Usage: dwc.awk inputfile { $0 = tolower($0); # be case insensitive gsub(/[\r\n\t,.!?;"*<>@\[\](){}+=/\\\#:]/, " "); # remove punctuations etc.. # the above list is chosen to match the C version -- ', ` etc missing $0 = $0; # hack to re-split $0 for(i = 1; i <= NF; ++i) { # printf "line:%i word:%i %s\n", NR, i, $i; # DEBUG ++nwords; # increment total number of words if (words[$i]) continue; # word is not new - skip words[$i] = 1; ++uwords; # increment number of unique words } } END { printf("Total number of words = %i\n", nwords); printf("Number of distinct or unique words = %i\n", uwords); } _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From selva.nair at utoronto.ca Fri Jan 30 16:55:11 2004 From: selva.nair at utoronto.ca (Selva Nair) Date: Fri, 30 Jan 2004 16:55:11 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) -- correction In-Reply-To: Message-ID: Replying to my own post.. On Fri, 30 Jan 2004, Selva Nair wrote: > knew that many more words but did not use. In any case, the awk > result is surprisingly close to 31,534 inspite of the > much smaller sample we used -- there is some over counting Ignore that comment about "smaller smaple". I did not realize that shaks12.txt indeed contains the complete works including even the sonnets. Sorry for the confusion. Selva _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 30 16:31:18 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 30 Jan 2004 16:31:18 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: <1075487873.8580.36.camel@squash.scalableinformatics.com> Message-ID: On Fri, 30 Jan 2004, Joe Landman wrote: > On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > > > Note that dwc finds 67505 "distinct words". This is very interesting, > > This is what I find in the Perl version as well. I wish I could get > that Java code compiling on this platform so I could compare the two. It might also be worthwhile to look at the list of words it produces to see what it is counting as a word. As I said, even my version where I'm controlling the split into words contains lots of non-words. Of course, I could have a bug in my code, as well:-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Fri Jan 30 19:27:59 2004 From: rossini at blindglobe.net (A.J. Rossini) Date: Fri, 30 Jan 2004 16:27:59 -0800 Subject: [Beowulf] HVAC and room cooling... Message-ID: <85k739x8q8.fsf@servant.blindglobe.net> I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual athlon or opteron machines). Can anyone suggest where I might look to find information for this? Also, does anyone have any brilliant thoughts for cooling an internal room that can't affordably get chilled water? (I've been suggesting to people that it isn't possible, but someone brought up "portable liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying to get stable systems, not instability :-). best, -tony -- rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 30 18:46:44 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 30 Jan 2004 18:46:44 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Fri, 30 Jan 2004, Selva Nair wrote: > 1. Googling shows that Shakespeare used 31,534 different > words and a total of 884,647 words in his complete works > (http://www-math.cudenver.edu/~wbriggs/qr/shakespeare.html) > And some statistical projections indicate he probably > knew that many more words but did not use. In any case, the awk > result is surprisingly close to 31,534 inspite of the > much smaller sample we used -- there is some over counting > due to numeric tokens and other "junk" in the Gutenberg text, > but I am still surprised by the close agreement. Also I wonder > why the C version comes up with as high as 37,000+ unique words. I didn't use toupper or tolower, simply said that to get closer to a correct count of unique "words" (as opposed to tokens) one should. If I do (trivial addition, completed) then I get 31384 words, which is again curious because a) I don't strip off the Gutenberg headers -- this is on raw shaks12.txt, so it contains words like cdroms and etexts and of course this is one reason the raw word count is high as well; and b) yeah, it still miscounts hyphenations and apostrophes and various other constructions I see scanning the actual list of unique words. It appears that the net effect is negative. OR, as always, I could have a bug. Or the google produced count could be wrong -- depends on how it was made and the sources, after all. If you (or anyone) care I can put the wordlist it generates on my website or something so you can grab it. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Fri Jan 30 20:00:52 2004 From: bclem at rice.edu (Brent M. Clements) Date: Fri, 30 Jan 2004 19:00:52 -0600 (CST) Subject: [Beowulf] Authentication within beowulf clusters. Message-ID: So, we were just going through out standards and procedures on how we do various things on our beowulf clusters. One of the procedures that we are looking at is the way to do authentication. My question is this, how does everyone handle authentication? Do you run standalone ldap or nis services on the master/management node of their beowulf clusters? We'd like to get an idea of how everyone else handles authentication on their clusters. If your company/university uses a centralized authentication service...please explain how you get your private network nodes to work with that configuration. THanks, Brent Clements Linux Technology Specialist Information Technology Rice University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bushnell at chem.ucsb.edu Fri Jan 30 20:16:59 2004 From: bushnell at chem.ucsb.edu (John Bushnell) Date: Fri, 30 Jan 2004 17:16:59 -0800 (PST) Subject: [Beowulf] HVAC and room cooling... Message-ID: >I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual >athlon or opteron machines). > >Can anyone suggest where I might look to find information for this? You can probably find some approx. formula on a Beowulf site. I know I've seen one on this list, which I found useful. (list archive) (So many watts) times 'x' equals how many "tons" of AC. Multiply by at least two of course ;-) >Also, does anyone have any brilliant thoughts for cooling an internal >room that can't affordably get chilled water? (I've been suggesting >to people that it isn't possible, but someone brought up "portable >liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >to get stable systems, not instability :-). You can have an external heat exchanger. If you are lucky and are, say, on the first floor somewhere close to an external wall, it is pretty simple to run a small pipe between the internal AC and the heat exchanger outside. Don't know how far it is practical to run one though. We have one in our computer room, but it is only six feet or so from the exchanger outside. Our newer AC runs on chilled water which was quoted for a lot less than another inside/outside combo, but we already had a leftover chilled water supply in the computer room. Hope that helps - John > >best, >-tony > >-- >rossini at u.washington.edu http://www.analytics.washington.edu/ >Biomedical and Health Informatics University of Washington >Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research >Center >UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable >FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Fri Jan 30 20:31:16 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Fri, 30 Jan 2004 20:31:16 -0500 Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: References: Message-ID: <1075512676.4915.207.camel@protein.scalableinformatics.com> On Fri, 2004-01-30 at 16:39, Selva Nair wrote: > On Fri, 30 Jan 2004, Joe Landman wrote: > > > On Fri, 2004-01-30 at 11:37, Robert G. Brown wrote: > > > > > Note that dwc finds 67505 "distinct words". This is very interesting, > > > > This is what I find in the Perl version as well. I wish I could get > > that Java code compiling on this platform so I could compare the two. > > The awk script I posted earlier counts tokens separated by [:blank:] and I > guess that is what the C++ version also does -- tokens not words as rgb > carfully pointed out. I was trying to mimic the C++ code, so case and Ok. I changed mine slightly. Very naive case here Perl 5.8.3 wrnpc11.txt 1.21 +/- 0.01 s shaks12.txt 2.09 +/- 0.04 s big.txt 14.31 +/- 0.09 s vbig.txt 1m11.49 +/- 0.52 s I could easily optimize it more (do the work on a larger buffer at a once), but I think enough waste heat has been created here. This is a simple 2500+ Athlon XP box (nothing fancy) running 2.4.24-pre3. ---- #!/opt/perl583/bin/perl my $infile; my $filename = shift; my (%word,$buffer); die "Usage: wcount2.pl filename\n" if (!defined($filename)); open($infile,"<".$filename); printf "opened file ...\n"; while ( $buffer = lc(<$infile>) ) { @word{($buffer=~ /(\w+)/g)}=1; } printf "Words: %s\n",1 + keys %word; ---- [...] > eloguantly expressed by rgb. I can not see any reason to learn or use C++ > simply for sets or hashes or trees in STL. The view I have is to use the most appropriate tool for the task. Some tools are in search of tasks for which they are appropriate. Marketing has in some cases supplanted common sense. > By the way, the performance of C++ STL appears to vary by up to a factor > of 10 between different versions of g++. The number I quoted (2.8 s) was > the fastest I could get and that was with g++ 2.96. For some reason > version 3.0.x runs much slower here.. Alex Stepanov had a benchmark on the cost of the STL and the advanced C++ features. Some compilers did rather poorly as I remember. You might get better results from Intel's compiler or the PathScale compilers (for the Opteron). > > Selva > > Footnotes: > > 1. Googling shows that Shakespeare used 31,534 different > words and a total of 884,647 words in his complete works Hmmm... time ./wcount2.pl shaks12.txt opened file ... Words: 23903 2.000u 0.020s 0:02.04 99.0% 0+0k 0+0io 360pf+0w I am using /(\w+)/g which will grab words into capture buffers. A word is a collection of alphanumeric plus others. I could build other regexes. Using /(\S+)/g gives 59509. Using /\b(.*)\b/g I get 110557. > (http://www-math.cudenver.edu/~wbriggs/qr/shakespeare.html) > And some statistical projections indicate he probably > knew that many more words but did not use. In any case, the awk > result is surprisingly close to 31,534 inspite of the > much smaller sample we used -- there is some over counting > due to numeric tokens and other "junk" in the Gutenberg text, > but I am still surprised by the close agreement. Also I wonder > why the C version comes up with as high as 37,000+ unique words. It goes to what the definition of "word" is. I am lower-casing everything. Without lower-casing I get 29673. Not sure what the issue is, but not willing to spend more time on it. Of course this has little to do with Beowulf-ery. Language choice is largely dictated by the problem, the developers etc. Force fitting solutions is often a recipe for disaster (systems designed to fail often do). Joe -- Joseph Landman, Ph.D Scalable Informatics LLC email: landman at scalableinformatics.com web: http://scalableinformatics.com phone: +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Fri Jan 30 22:32:20 2004 From: rossini at blindglobe.net (A.J. Rossini) Date: Fri, 30 Jan 2004 19:32:20 -0800 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: (John Bushnell's message of "Fri, 30 Jan 2004 17:16:59 -0800 (PST)") References: Message-ID: <85y8rosshn.fsf@servant.blindglobe.net> John Bushnell writes: > (So many watts) times 'x' equals how many "tons" of AC. Multiply > by at least two of course ;-) Or 3, sigh... >>Also, does anyone have any brilliant thoughts for cooling an internal >>room that can't affordably get chilled water? (I've been suggesting >>to people that it isn't possible, but someone brought up "portable >>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>to get stable systems, not instability :-). > > You can have an external heat exchanger. If you are lucky and are, > say, on the first floor somewhere close to an external wall, it is > pretty simple to run a small pipe between the internal AC and the > heat exchanger outside. Don't know how far it is practical to run > one though. We have one in our computer room, but it is only six > feet or so from the exchanger outside. Our newer AC runs on chilled > water which was quoted for a lot less than another inside/outside > combo, but we already had a leftover chilled water supply in the > computer room. I've looked at the chilled-water approach. They estimated between $40k-$80k. oops (this room is REALLY in the middle of the building. Great for other computing purposes, but not for cooling). I'm looking for the proverbial vent-free A/C. Sort of like frictionless tables and similar devices I recall from undergraduate physics... Thanks for the comments! best, -tony -- rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Sat Jan 31 00:18:15 2004 From: bclem at rice.edu (Brent M. Clements) Date: Fri, 30 Jan 2004 23:18:15 -0600 (CST) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <85y8rosshn.fsf@servant.blindglobe.net> References: <85y8rosshn.fsf@servant.blindglobe.net> Message-ID: I have found that the best thing to do is outsource the colocation of your equipment. The cost of installing and maintaining the proper type of cooling and ventilation for mid-large size clusters costs more than to colocate. We are currently exploring placing our larger clusters in colocation facilities right now. The only downside that we have is that we can't find colocation facilities that will give us 24/7 physical access to our equipment. As you all know...researchers push beowulf hardware to the limits and the meantime to failure is higher. -B Brent Clements Linux Technology Specialist Information Technology Rice University On Fri, 30 Jan 2004, A.J. Rossini wrote: > John Bushnell writes: > > > (So many watts) times 'x' equals how many "tons" of AC. Multiply > > by at least two of course ;-) > > Or 3, sigh... > > >>Also, does anyone have any brilliant thoughts for cooling an internal > >>room that can't affordably get chilled water? (I've been suggesting > >>to people that it isn't possible, but someone brought up "portable > >>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > >>to get stable systems, not instability :-). > > > > You can have an external heat exchanger. If you are lucky and are, > > say, on the first floor somewhere close to an external wall, it is > > pretty simple to run a small pipe between the internal AC and the > > heat exchanger outside. Don't know how far it is practical to run > > one though. We have one in our computer room, but it is only six > > feet or so from the exchanger outside. Our newer AC runs on chilled > > water which was quoted for a lot less than another inside/outside > > combo, but we already had a leftover chilled water supply in the > > computer room. > > I've looked at the chilled-water approach. They estimated between > $40k-$80k. oops (this room is REALLY in the middle of the building. > Great for other computing purposes, but not for cooling). > > I'm looking for the proverbial vent-free A/C. Sort of like > frictionless tables and similar devices I recall from undergraduate > physics... > > Thanks for the comments! > > best, > -tony > > -- > rossini at u.washington.edu http://www.analytics.washington.edu/ > Biomedical and Health Informatics University of Washington > Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center > UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable > FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email > > CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be > confidential and privileged. If you received this message in error, > please destroy it and notify the sender. Thank you. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at clustervision.com Sat Jan 31 06:22:44 2004 From: john.hearns at clustervision.com (John Hearns) Date: Sat, 31 Jan 2004 12:22:44 +0100 (CET) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <85k739x8q8.fsf@servant.blindglobe.net> Message-ID: On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > to people that it isn't possible, but someone brought up "portable > liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > to get stable systems, not instability :-). > Daft idea. What happens when the liquid nitrogen vapourises? It turns to nitrogen gas. And if you are unlucky anyone in the room chokes... I was reminded of this in the lift of a chemistry department where we have an opteron cluster - a large sign reminding that LNO2 and dangerous chemicals are not allowed in passenger lifts. BTW, when arranging air flows try to get cold air ducted into the front of the rack - node tend to draw air in the front and exhaust to the rear. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Sat Jan 31 06:57:49 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Sat, 31 Jan 2004 12:57:49 +0100 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: References: <85y8rosshn.fsf@servant.blindglobe.net> Message-ID: <401B983D.8070801@telia.com> Hi, As a Marine Engineer is the cooling aspect the absolute smallest problem with a Linux cluster. The cooling capacity required is calculated with by means of the "Funnel principle" i.e. every thing that you are putting in will come out. This give that you have to calculate the total heat in-put in your cluster-room as that is what you want transport out of the room by means of an air-refrigerant equipment. The simplest way to found out the cooling effect required will be to: A) Ask a professional air refrigerant company to recommend the capacity for the actual1 location when it is used as an office. (They should have experience and know-how to calculate the heat in-put from the sun-shine and climate in your area) B) Calculate your cluster's maximum power consumption 3 Calculate the total cooling capacity required by the formula: ? > (A+B) x 1.1 Best regards Per Lindstro"m Brent M. Clements wrote: >I have found that the best thing to do is outsource the colocation of your >equipment. The cost of installing and maintaining the proper type of >cooling and ventilation for mid-large size clusters costs more than to >colocate. > >We are currently exploring placing our larger clusters in colocation >facilities right now. > >The only downside that we have is that we can't find colocation facilities >that will give us 24/7 physical access to our equipment. As you all >know...researchers push beowulf hardware to the limits and the meantime to >failure is higher. > >-B > >Brent Clements >Linux Technology Specialist >Information Technology >Rice University > > >On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > >>John Bushnell writes: >> >> >> >>>(So many watts) times 'x' equals how many "tons" of AC. Multiply >>>by at least two of course ;-) >>> >>> >>Or 3, sigh... >> >> >> >>>>Also, does anyone have any brilliant thoughts for cooling an internal >>>>room that can't affordably get chilled water? (I've been suggesting >>>>to people that it isn't possible, but someone brought up "portable >>>>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>>>to get stable systems, not instability :-). >>>> >>>> >>> You can have an external heat exchanger. If you are lucky and are, >>>say, on the first floor somewhere close to an external wall, it is >>>pretty simple to run a small pipe between the internal AC and the >>>heat exchanger outside. Don't know how far it is practical to run >>>one though. We have one in our computer room, but it is only six >>>feet or so from the exchanger outside. Our newer AC runs on chilled >>>water which was quoted for a lot less than another inside/outside >>>combo, but we already had a leftover chilled water supply in the >>>computer room. >>> >>> >>I've looked at the chilled-water approach. They estimated between >>$40k-$80k. oops (this room is REALLY in the middle of the building. >>Great for other computing purposes, but not for cooling). >> >>I'm looking for the proverbial vent-free A/C. Sort of like >>frictionless tables and similar devices I recall from undergraduate >>physics... >> >>Thanks for the comments! >> >>best, >>-tony >> >>-- >>rossini at u.washington.edu http://www.analytics.washington.edu/ >>Biomedical and Health Informatics University of Washington >>Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center >>UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable >>FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email >> >>CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be >>confidential and privileged. If you received this message in error, >>please destroy it and notify the sender. Thank you. >>_______________________________________________ >>Beowulf mailing list, Beowulf at beowulf.org >>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Fri Jan 30 11:37:52 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Fri, 30 Jan 2004 11:37:52 -0500 (EST) Subject: [Beowulf] C vs C++ challenge (awk version) In-Reply-To: Message-ID: On Thu, 29 Jan 2004, Selva Nair wrote: > But this one does not count unique words, does it? > > Here is my version in awk. It beats C++ by 1 line in length and > 1.5 times in speed (1.86s versus 2.83s elapsed time) with shaks12.txt as > input. > A modified (hashing/unique word) version of the C yields: rgb at lilith|T:245>/usr/bin/time wordcount shaks12.txt Opening shaks12.txt for counting Total Number of Words = 902303; total number of UNIQUE words = 37499 Command exited with non-zero status 69 1.15user 0.12system 0:01.29elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (100major+4093minor)pagefaults 0swaps This is using gnu's hcreate, hsearch hash functions (ordinary POSIX library functions, as it were) and simply adds a secondary loop through all the words to do the count. This is still not terribly close to efficient -- the reason it has to be done in a secondary loop is because realloc moves the base pointer to the primary data buffer where all the words are stored, which makes hcreate/hsearch unhappy in midstream. The best way around this are to split the code into a segment to read the whole file into memory efficiently (probably using read()/realloc in 32K chunks) and a single loop to strsep() parse into words and to hsearch() them into the hash. I don't have a good feel for how much time this would save, but at a guess one could get the overall time down to maybe a second from 1.29 seconds wallclock, mostly on the I/O side. Oh, and one wouldn't need to maintain a secondary indexing vector at all since the hash would do it all, although I'm sure it isn't horribly expensive. Of course this STILL doesn't REALLY count unique "words", only unique "tokens". To do words we'd have to toupper() all the words before saving/hashing them, as The and the aren't really unique. In Shakespeare, things are even worse as he uses his own characteristic blend of "interesting" contractions, what say'st though? Idiomatic, as it were. So lots of tokens included a ' and are contractions, others are caps, still others are real mistakes -- hyphenated words split across a line but recorded as "two unique words". So the real count is more likely a few thousand smaller. Still, Shakespeare seems pretty literate -- ~38K unique tokens when linux_words only has order of 45K total. I don't know that I'll bother to redo this one more time to speed it any further as described above -- it IS a toy project. The main point is that adding hashing required about 13 lines of code, and in a rewrite that eliminated the indexing vector I'd get those back and change. So using POSIX/Gnu functions (sloppily) it still isn't a horribly long program, and is still fairly efficient (running in less than half the time of C++). One final point: > [selva at scala distinct_words]$ /usr/bin/time ./dwc < shaks12.txt > Words: 67505 > 2.79user 0.04system 0:02.83elapsed 100%CPU > > Here is the script: > > #!/bin/awk -f > { > for(i = 1; i <= NF; i++) { > if (words[$i]) continue; > words[$i] = 1 ; > ++nwords; > } > } > END { > printf "Number of distinct words = %i\n", nwords; > } Note that dwc finds 67505 "distinct words". This is very interesting, as wordcount only finds 38000-odd distinct words, where "distinct" has to do with just how words are parsed in the first place and where I can watch, via inserted printf commands, to be dead certain that the process of word selection and uniquifying is proceeding correctly. In fact in C this is ALL strictly under my control -- I can alter the list of word delimiters as I wish, and I can even (with a bit of effort but not a huge penalty in time or lines of code) "fix" the hyphenation problem: a) stop parsing on LF/CR (currently they are delimiters b) IF a word ends on LF and/or CR AND the preceeding character is a hyphen c) THEN shift the following word left N spaces (to be computed) in the buffer to reconnect the word halves. d) AND save the word as the reconnected halves (via offsets or hash or both. Fix >>this<< in C++ using the code already presented -- hmmm, not so easy. Fix it in awk (or even figure out what is wrong with the script above that it is returning almost twice as many words as I count, where my script and e.g. wc return precisely the same count on /usr/share/dict/linux_words and where I can as noted watch it do what it does and hence have a fair bit of confidence in it). Fix it so that (in both cases) one still has the entire text in memory, indexed by and retrievable by words one at a time AND has access to the hash of unique words only without significantly increasing the time or storage requirement. I bring these up simply to reiterate that real world code is rarely simple. This is hardly a "complex project" at first glance, but if one examines what one actually GETS as unique words when one just reads in a block of arbitrary (but real) text, parses it badly (or at least in a way that is out of your control) one finds that it is pretty unlikely to return what you really want if you use a one-size-fits-all library function to do all the work. To reassert control and fix special cases without microscopic access to the raw data stream requires more and more contortion, and can easily introduce multiple passes across the data to do first one thing, then another, then another, when they could all be done in a single loop if one did it "right". It is precisely this that I was referring to when I said that using a library, ESPECIALLY a powerful high level library, shapes your code, sometimes in highly inefficient ways. This isn't surprising -- a variety of philosophers e.g. Dorothy Lee and Bronislaw Malinoski have long argued that the very way we think is governed by the language we think in, and this is obviously even more true when it comes to computer "languages", where commands encapsulate entire algorithms with very specific and complex results. Similar tradeoffs are visible in pictographic languages (where every "thought" has its own unique token that must be learned in order to read it or write it) and phonetic languages (where every thought has a nearly unique VERBAL token, but where a relatively small set of building blocks can represent all the phonemic encodings so that one can learn to read and write years earlier, and where the language is far more flexible and capable of growth and evolution at a far greater rate). So my final conclusion (this thread has probably gone on long enough and at least SOME people are likely bored with it by now:-) is my original one -- raw C (or C++ written as procedural C with minimal and sensible use of the extensions, a la Mark Hahn:-) will nearly always result in the fastest execution times, and gives one COMPLETE control of your data and execution loops. Using objects and object-oriented methodology is still entirely possible and I'd even say best practice in BOTH language variants. C++ (including the STL library) has a wider set of "standard" tools, although those tools come with a very distinct ease of programming vs execution time tradeoff, and with another more subtle tradeoff in the way that they shape your code (which may be in ways that are just fine, or MAY be in ways that are moderately detrimental when the library calls aren't terribly good fits to what you want to do but your reliance on them has slowed the development of skills to do what you want without them). Again, this isn't an intrinsic criticism of C++ -- it is just an empirical observation and I'd make an entirely similar observation about many other high level tools including markup languages vs word processors and more. If you use a nail gun all the time doing rough carpentry, don't be surprised if you hit your thumb and leave divots in the work when it comes time to hammer finish nails into the trim or cut nails into a cinder block wall with an actual hammer. (Don't you just love metaphors?:-) Other languages have similar tradeoffs -- few will beat C in executable speed, although it is possible (even humble fortran might beat C in certain tasks, partly because the fixed data language structure permits some things to be done by the compiler very efficiently where C uses library calls on more arbitrary pointer based data structures less efficiently). Many have things C does not that make PROGRAMMING a lot easier and may even enable certain things (such as writing a self-modifying program) POSSIBLE that are horribly difficult in raw C. SO I like C, but I also like PHP, Perl, might one day like C++ and python. I even like awk, although aside from very short awk programs to do very specialized things perl is infinitely easier. I therefore remain highly ecumenical -- I might tease people for using other languages and do think that one should match language to task with some care, but I can hardly justify being overly zealous in advancing any single language as being THE best as it is always a matter of which tradeoffs one prefers or are led to by one's problem. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu #include #include #include #include #define _GNU_SOURCE #include #define PAGE 4096 #define K 1024 static char delim[] = { ' ', /* blank */ (char) 9, /* tab */ ',', /* comma */ (char) 10, /* LF */ (char) 13, /* CR */ '.', /* . */ '!', /* ! */ '?', /* ? */ ';', /* ; */ '"', /* " */ '*', /* * */ '<', /* < */ '>', /* > */ '@', /* @ */ '[', ']', '(', ')', '{', '}', '+', '=', '/', '\\', '#', ':', (char) 0 /* terminator */ }; static char dummy[] = "aapvoinwe"; main(int argc, char **argv) { char filename[128],*word,*words,*words_head,*words_tail,*words_last; int *word_offset; size_t offset,word_length,words_length,wordlist_length, word_count,unique_count,i,j,nel; FILE *infile; ENTRY item,*search_result; if(argc != 2){ fprintf(stderr,"Usage: wordcount filename\n"); exit(1); } argv++; sprintf(filename,"%s",*argv); printf("Opening %s for counting\n",filename); infile = fopen(filename,"r"); /* * The word list as a single buffer. We allocate two pages to start with, and * reallocate whenever the free space in the buffer drops below one page. We * also maintain a pointer to the beginning of the unused part of this buffer. */ words_length = 2*PAGE; words = (char *) malloc(words_length); words_head = words; words_tail = words + words_length; /* * We do the exact same thing here, sort of. We allocate a PAGE of offsets * (relative to the words head!) and reallocate whenever the number remaining * drops below K. i will both index the wordlist and count the words as we go. */ wordlist_length = PAGE; word_offset = (int *) malloc(wordlist_length*sizeof(int)); word_count = 0; /* * Each pass through the while loop reads a single line from * infile into words at offset wptr (current tail). We will * write the code to manage lines up to 1K in length. */ word_offset[0] = 0; /* * Set up a hash table to extract/count only UNIQUE words. 300000 * words could come close to doing the OED, at least OTC versions. * Not much speed penalty for making nel bigger, BIG penalty for * making it too small... */ nel = 300000; hcreate(nel); unique_count = 0; while(fgets(words_head,(words_tail - words_head),infile)!= NULL){ /* * Check to add a PAGE to the words vector/buffer. Since realloc * may move pointer, we have to recompute offsets. */ if((words_tail - words_head) < PAGE) { words_length += PAGE; /* save offset */ offset = words_head - words; words = (char *) realloc(words,words_length); /* displace by saved offset */ words_head = words + offset; words_tail = words + words_length; /* printf("reallocating words at %0x, new length = %d\n",words,words_length); */ } /* * Check to add a PAGE to the wordlist vector/buffer. Here we * don't have to worry about offsets. */ if((wordlist_length - word_count) < K) { wordlist_length += PAGE; word_offset = (int *) realloc(word_offset,wordlist_length*sizeof(int)); /* printf("reallocating words_offset at %0x, new length = %d\n",word_offset,wordlist_length); */ } /* * Now we rip through the line. */ while(1){ /* Get a word */ word = strsep(&words_head,delim); if(words_head == NULL) break; /* Save its address for after we are done with this line */ words_last = words_head; /* Save its offset in the offset vector */ if(word[0] != (char)0){ word_offset[word_count] = word - words; /* for debugging offsets etc */ /* printf("%d: word at %0x = %s, offset is %d\n",word_count,word,word,word_offset[word_count]); */ word_count++; } } /* * strsep finishes parsing a line with a NULL. Restore the right pointer. */ words_head = words_last; } /* * Let's see, we can try sorting and uniq'ing here. As expected, it is * gawdawful slow and scales terribly with size and even though we COULD * move the first loop up into the code above easily enough, this is a * poor way to do it. We'll leave the fragment here, disabled. for(i=0;i Message-ID: On Fri, 30 Jan 2004, A.J. Rossini wrote: > John Bushnell writes: > > > (So many watts) times 'x' equals how many "tons" of AC. Multiply > > by at least two of course ;-) > > Or 3, sigh... I should have posted this in the earlier reply, sorry -- 3500 watts/ton, within a watt or four. From the sound of it, you need at least one ton, but probably not over two. BTW, I should have said, but consider talking to an architect or HVAC engineer. They are professionals, after all...:-) > I'm looking for the proverbial vent-free A/C. Sort of like > frictionless tables and similar devices I recall from undergraduate > physics... Hahahahahaha... (ROTFL). Send me one if you find it. Then I can build my perpetual motion heat engine. rgb > > Thanks for the comments! > > best, > -tony > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 10:03:56 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 07:03:56 -0800 Subject: [Beowulf] HVAC and room cooling... References: <85k739x8q8.fsf@servant.blindglobe.net> Message-ID: <002d01c3e80b$809b94f0$36a8a8c0@LAPTOP152422> There were some posts on this last summer, and rgb's book on the Brahma site at Duke has a chapter that runs through the calculations. You just ballpark the power consumption (figure a couple hundred watts/processor to be conservative)... 24*200 = 4800W... convert that to BTU/hr and you know how big an a/c you'll need. All airconditioning just moves heat from one place to another, whether by moving water or by pumping Freon or ammonia around. Where are you going to reject that 5kW of heat? The hallway outside? (it's been done..) There's a variety of a/c equipment that only requires about a 2" diameter hole/passage between cold end and hot end. ----- Original Message ----- From: "A.J. Rossini" To: Sent: Friday, January 30, 2004 4:27 PM Subject: [Beowulf] HVAC and room cooling... > > I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual > athlon or opteron machines). > > Can anyone suggest where I might look to find information for this? > > Also, does anyone have any brilliant thoughts for cooling an internal > room that can't affordably get chilled water? (I've been suggesting > to people that it isn't possible, but someone brought up "portable > liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > to get stable systems, not instability :-). > > best, > -tony > > -- > rossini at u.washington.edu http://www.analytics.washington.edu/ > Biomedical and Health Informatics University of Washington > Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center > UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable > FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email > > CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be > confidential and privileged. If you received this message in error, > please destroy it and notify the sender. Thank you. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Sat Jan 31 10:32:58 2004 From: agrajag at dragaera.net (Jag) Date: 31 Jan 2004 10:32:58 -0500 Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: References: Message-ID: <1075563177.2560.4.camel@loiosh> On Fri, 2004-01-30 at 20:00, Brent M. Clements wrote: > So, we were just going through out standards and procedures on how we do > various things on our beowulf clusters. One of the procedures that we are > looking at is the way to do authentication. > > My question is this, how does everyone handle authentication? > > Do you run standalone ldap or nis services on the master/management node > of their beowulf clusters? > > We'd like to get an idea of how everyone else handles authentication on > their clusters. If your company/university uses a centralized > authentication service...please explain how you get your private network > nodes to work with that configuration. I work at a large university that has a central kerberos server. How we do it is we use NIS locally in the cluster, but *only* for name service information. We then configure PAM to use kerberos off the main campus servers to do the actual authentication. We do this so we can use the central campus resources, but still remap home directories and such. The main key to that is making sure that the usernames on the cluster match the campus username. In order to use this on the compute nodes, we use hostbased authentication with ssh. This is along the same lines of doing hosts.equiv with rsh. We have 4 head nodes that are on the public network and the internal cluster network. A user simply logs into there, and when their mpich jobs call ssh, ssh lets them log into any of the cluster nodes without asking for a password. We can get away with this as we have a private LAN for all the nodes, and physical access to the cluster is restricted. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From widyono at cis.upenn.edu Fri Jan 30 20:11:22 2004 From: widyono at cis.upenn.edu (Daniel Widyono) Date: Fri, 30 Jan 2004 20:11:22 -0500 Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: References: Message-ID: <20040131011122.GA19068@central.cis.upenn.edu> We have one form of authentication (based on University standards) to login to the head node. We use our own system for authenticating within the internal network (which is only used by the cluster). We distribute /etc/passwd for account information via cron (and via useradd and userdel wrappers). We update an authentication token on each node as it becomes assigned to a user by the scheduling system. Ssh checks this authentication token (via PAM module) before executing on a node, and Bproc uses an ownership database on the server side. Regards, Dan W. On Fri, Jan 30, 2004 at 07:00:52PM -0600, Brent M. Clements wrote: > > So, we were just going through out standards and procedures on how we do > various things on our beowulf clusters. One of the procedures that we are > looking at is the way to do authentication. > > My question is this, how does everyone handle authentication? > > Do you run standalone ldap or nis services on the master/management node > of their beowulf clusters? > > We'd like to get an idea of how everyone else handles authentication on > their clusters. If your company/university uses a centralized > authentication service...please explain how you get your private network > nodes to work with that configuration. > > THanks, > > > Brent Clements > Linux Technology Specialist > Information Technology > Rice University > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- -- Daniel Widyono http://www.cis.upenn.edu/~widyono -- Liniac Project, CIS Dept., SEAS, University of Pennsylvania -- Mail: CIS Dept, 302 Levine 3330 Walnut St Philadelphia, PA 19104 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Jan 31 10:25:58 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 31 Jan 2004 10:25:58 -0500 (EST) Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: <20040131011122.GA19068@central.cis.upenn.edu> Message-ID: On Fri, 30 Jan 2004, Daniel Widyono wrote: > We have one form of authentication (based on University standards) to login > to the head node. > > We use our own system for authenticating within the internal network (which > is only used by the cluster). We distribute /etc/passwd for account > information via cron (and via useradd and userdel wrappers). We update an > authentication token on each node as it becomes assigned to a user by the > scheduling system. Ssh checks this authentication token (via PAM module) > before executing on a node, and Bproc uses an ownership database on the > server side. ... > > My question is this, how does everyone handle authentication? > > > > Do you run standalone ldap or nis services on the master/management node > > of their beowulf clusters? NIS works fine for many purposes as well, but be warned -- in certain configurations and for certain tasks it becomes a very high overhead protocol. In particular, it adds an NIS hit to every file stat, for example, so that it can check groups and permissions. I think it was Greg who pointed out many moons ago that there are was around this (IIRC, making every node an NIS slave so that the hit is local?). NIS also at one point in time was a bleeding wound as far as security is concerned. How fondly I remember reaching out across campus and grabbing the encrypted passwd's of whole departments with a simple userspace tool to feed into crack and grab the passwords of the five silly beanies who used something like "password" for their password. Or using tftp to accomplish the same thing on systems configured out of the box to provide it. (Shame on you! I wasn't really cracking, I was a white hat checking the nets in question with permission...:-) Even though they've largely fixed it, I believe NIS remains susceptible to a vast range of attacks -- snooping (sending stuff in cleartext where it can be read by anyone on a shared line), spoofing (a host is known by IP number only) and more. So I'd strongly urge using it only inside a firewall on a trusted LAN space. I suspect LDAP has the same issues, but we've never used it so I'm not certain. Kerberos is a lot better for this sort of thing, especially if the cluster you're talking about is a "public cluster" to be accessed from a WAN or a large LAN with multiple internal LAN boundaries (e.g. department LANs) of varying degrees of trust and quality of administration. It's also fairly straightforward to use e.g. rsync to keep passwd, groups, shadow in sync across a cluster, but one does have the problem of propagating a passwd change, which NIS manages for you transparently. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From agrajag at dragaera.net Sat Jan 31 11:30:56 2004 From: agrajag at dragaera.net (Jag) Date: 31 Jan 2004 11:30:56 -0500 Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: References: Message-ID: <1075566655.2560.8.camel@loiosh> On Sat, 2004-01-31 at 10:25, Robert G. Brown wrote: > NIS works fine for many purposes as well, but be warned -- in certain > configurations and for certain tasks it becomes a very high overhead > protocol. In particular, it adds an NIS hit to every file stat, for > example, so that it can check groups and permissions. A good way around this is to run nscd (Name Services Caching Daemon). Its part of glibc and will cache a lot of that info on the local nodes. That way if you're stating a bunch of files all owned by the same user, it'll go over the network once to get the info, but all future requests will be handled by nscd, and you won't see any network traffic for it. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From jj.renes at hccnet.nl Sat Jan 31 19:28:19 2004 From: jj.renes at hccnet.nl (joop renes) Date: Sat, 31 Jan 2004 16:28:19 -0800 Subject: [Beowulf] C/C++ discussion Message-ID: <200401311628.19673.jj.renes@hccnet.nl> Hi, I would like to offer the attached remarks to that discussion. best regards. ++jopa -------------- next part -------------- A non-text attachment was scrubbed... Name: discussion1.lyx Type: text/x-lyx Size: 2522 bytes Desc: not available URL: From daniel.pfenniger at obs.unige.ch Sat Jan 31 12:40:33 2004 From: daniel.pfenniger at obs.unige.ch (Daniel Pfenniger) Date: Sat, 31 Jan 2004 18:40:33 +0100 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: References: Message-ID: <401BE891.708@obs.unige.ch> John Hearns wrote: > On Fri, 30 Jan 2004, A.J. Rossini wrote: >>to people that it isn't possible, but someone brought up "portable >>liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>to get stable systems, not instability :-). >> > > Daft idea. > What happens when the liquid nitrogen vapourises? It turns to nitrogen > gas. And if you are unlucky anyone in the room chokes... > I was reminded of this in the lift of a chemistry department where > we have an opteron cluster - a large sign reminding that LNO2 and > dangerous chemicals are not allowed in passenger lifts. When liquid N2 vaporizes in a closed room the natural fraction of N2 in air, already 78% by volume, just increases. N2 is thus not toxic, the only risk by increasing N2 volume is to reach a too low O2 concentration for mammals beings breathing in the room. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at physics.mcmaster.ca Sat Jan 31 12:54:25 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat, 31 Jan 2004 12:54:25 -0500 (EST) Subject: [Beowulf] Authentication within beowulf clusters. In-Reply-To: Message-ID: > My question is this, how does everyone handle authentication? our clusters are accessible via ssh, with no auth within. > Do you run standalone ldap or nis services on the master/management node > of their beowulf clusters? no, rsync'ed passwd/shadow files; we'll eventually consider switching to ldap or a simple sql-backed passwd/shadow generator. NIS is out of the question. > their clusters. If your company/university uses a centralized > authentication service...please explain how you get your private network > nodes to work with that configuration. I don't see the advantage of "centralized" auth, except has a form of centralized control (which is inherently bad). consider that a random user *never* has access to every resource on campus, so whoever admins the central auth has to be constantly managing permissions for random resources that pop up. not to mention that it creates a single point of failure and a network hotspot (yes, yes, those can be engineered around.) single point of compromise, too. what's the benefit? ssh already does password-less, secure logins, so there's no advantage there. if everything were not ssh-based, I'm sure the analysis would be different. I'd sooner consider going to pure ssh-key logins, rather than involving some external authentication oracle. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From bclem at rice.edu Sat Jan 31 13:38:26 2004 From: bclem at rice.edu (Brent M. Clements) Date: Sat, 31 Jan 2004 12:38:26 -0600 (CST) Subject: [Beowulf] How do vendors win you over? Message-ID: Hi Everyone, In addition to our beowulf standards that we are working on, I'm streamlining our research and purchasing process of vendors for beowulf clusters within IT. 1. What are the qualities in vendors that you look for. 2. What requirements of a vendor do you require before you make a purchase? 3. Would you go with a relatively new company? If so, what would make you choose them over someone who is more knowledgeble about clusters or who has been in the computer business for a good number of years. 4. What type of support options do you all require. Do you just require warranty support or do you require full support(software and hardware, patches, etc etc) 5. Finally, what is the number one thing that always sways your vote towards a particular vendor? Thanks Alot! Brent Clements Brent Clements Information Technology Rice University _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From klamman.gard at telia.com Sat Jan 31 14:54:47 2004 From: klamman.gard at telia.com (Per Lindstrom) Date: Sat, 31 Jan 2004 20:54:47 +0100 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <401BE891.708@obs.unige.ch> References: <401BE891.708@obs.unige.ch> Message-ID: <401C0807.4000209@telia.com> ATTENTION! Liquid Nitrogen is deadly dangerous as it will shut off the breathing function of the human body when the Oxygen content falls below 19.5 % and thereby causing suffocation. This is not an unusual accident type in the Oil- & Gas- as well as in the Shipping industry. For the sake of your safety I'm recommending the use of an Industrial Safe Working Practice as a guideline before any work commence. A reference: **http://www.medicine.uiowa.edu/biochemstores/Pages/ln2msds.htm Best regards Per Lindstrom Daniel Pfenniger wrote: > John Hearns wrote: > >> On Fri, 30 Jan 2004, A.J. Rossini wrote: > > >>> to people that it isn't possible, but someone brought up "portable >>> liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying >>> to get stable systems, not instability :-). >>> >> >> Daft idea. >> What happens when the liquid nitrogen vapourises? It turns to >> nitrogen gas. And if you are unlucky anyone in the room chokes... >> I was reminded of this in the lift of a chemistry department where >> we have an opteron cluster - a large sign reminding that LNO2 and >> dangerous chemicals are not allowed in passenger lifts. > > > When liquid N2 vaporizes in a closed room the natural fraction of N2 > in air, already 78% by volume, just increases. N2 is thus not toxic, > the only risk by increasing N2 volume is to reach a too low O2 > concentration > for mammals beings breathing in the room. > > Dan > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel.pfenniger at obs.unige.ch Sat Jan 31 16:59:26 2004 From: daniel.pfenniger at obs.unige.ch (Pfenniger Daniel) Date: Sat, 31 Jan 2004 21:59:26 +0000 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <401C0807.4000209@telia.com> References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> Message-ID: <401C253E.9040206@obs.unige.ch> Per Lindstrom wrote: > ATTENTION! > > Liquid Nitrogen is deadly dangerous as it will shut off the breathing > function of the human body when the Oxygen content falls below 19.5 % > and thereby causing suffocation. > Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly dangerous when inhaled in too large quantities. The quoted 19.5% are probably referring to the human body (blood?) concentration, not the inhaled air. Otherwise most of the people in pubs or small rooms would rapidly die as the normal O2 concentration in air is 20%. Note that in the responded message John was confusing N2 and NO2. Dan _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From gerry.creager at tamu.edu Sat Jan 31 17:30:47 2004 From: gerry.creager at tamu.edu (Gerry Creager (N5JXS)) Date: Sat, 31 Jan 2004 16:30:47 -0600 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <401C253E.9040206@obs.unige.ch> References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> <401C253E.9040206@obs.unige.ch> Message-ID: <401C2C97.8020903@tamu.edu> At the risk of turning this technical, the real problem is usually the partial pressure of available oxygen. Generally, if the PaO2 (arterial blood partial pressure) drops below about 80 mmHg, one begins to suffer the effects of hypoxia. Most folks (University administrators seem to be immune to this) start dying at or below a PaO2 of 40 mmHg. In general, if one considers a standard pressure to be 760 mmHg at sea level, and noting that, in fact the normal concentration of oxygen available ambiently is roughly 20.95%, one sees the ambient PO2 at sea level is approximately 159 mmHg. Upon inspiration, one adds significant water vapor to the inhaled air (for reference, the medical and aviation world tend to assume the partial pressure of water in ambient conditions is about 4 mmHg), usually, about 40 mmHg at saturation... which is how the inspired air is supposed to be when it hits the lungs. So, taking that off-the-top from the partial pressure of oxygen in inspired air we have 159 - 40= 119 mmHg. Now, there's a gradient that helps control the transfer of oxygen across the alveolar-capillary barrier, where gas exchange occurs. Further, there's some transfer at the same interface of carbon dioxide leaving the blood cells and plasma. This combines to a net pressure gradient (I'm working from memory here, and I've not had to give this lecture for several years, so the exact numbers could be off) of 20-25 mmHg. The net result is simple: The PaO2 (arterial O2 concentration) is about 95 mmHg for normal individuals. Nitrogen in an enclosed space tends to act simply as an oxygen displacer. Thus, you go from roughly 78% N2 in ambient air to however much you ramp it up, and the O2 concentration decreases from a nominal 21% (rounding error, you understand) to whatever's left. If you note, while there's trace gases in ambient air, we can generally ignore them and worry about N2 and O2. YMMV for increased concentrations of CO2 and CO. That's the end of gas exchange physiology I. There will be a short quiz Monday. We'll continue with the next module. I encourage everyone to have read the Pulmonary Medicine chapters in Harrison's for the next lecture. Gerry Pfenniger Daniel wrote: > Per Lindstrom wrote: > >> ATTENTION! >> >> Liquid Nitrogen is deadly dangerous as it will shut off the breathing >> function of the human body when the Oxygen content falls below 19.5 % >> and thereby causing suffocation. >> > Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly > dangerous when inhaled in too large quantities. > The quoted 19.5% are probably referring to the human body (blood?) > concentration, not the inhaled air. Otherwise most of the people in > pubs or small rooms would rapidly die as the normal O2 concentration > in air is 20%. > > Note that in the responded message John was confusing N2 and NO2. > > > Dan > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager at tamu.edu Network Engineering -- AATLT, Texas A&M University Office: 979.458.4020 FAX: 979.847.8578 Cell: 979.229.5301 Pager: 979.228.0173 _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 18:26:55 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 15:26:55 -0800 Subject: [Beowulf] HVAC and room cooling... References: <401BE891.708@obs.unige.ch> Message-ID: <005d01c3e851$b78dbba0$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Daniel Pfenniger" To: "John Hearns" Cc: ; Sent: Saturday, January 31, 2004 9:40 AM Subject: Re: [Beowulf] HVAC and room cooling... > John Hearns wrote: > > On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > > Daft idea. > > What happens when the liquid nitrogen vapourises? It turns to nitrogen > > gas. And if you are unlucky anyone in the room chokes... > > I was reminded of this in the lift of a chemistry department where > > we have an opteron cluster - a large sign reminding that LNO2 and > > dangerous chemicals are not allowed in passenger lifts. > > When liquid N2 vaporizes in a closed room the natural fraction of N2 in > air, already 78% by volume, just increases. N2 is thus not toxic, the > only risk by increasing N2 volume is to reach a too low O2 concentration > for mammals beings breathing in the room. Which is precisely the problem. You can fill a room with nitrogen and there's no obvious sign of it. Someone opens the door, walks in, and is unconscious in a remarkably short time. The problem is that you've also displaced the CO2, so there's no "breathe reflex", and you just breath faster trying to get the PO2 in your blood up, which aggravates the problem, then you pass out. All that oxygen poor air is now flowing out through the door so eventually you'll get oxygen into the room, as well, but you're really relying on diffusion (there's not a huge density difference). This is one of the classic "silent but deadly" hazards. This is what "confined space" training is all about. People have climbed down into a tank or crawlspace and not climbed back out, for instance. Here at JPL, even in ordinary labs, if there's nitrogen plumbed in (we use it to purge "dry boxes" or temperature test chambers, e.g. to prevent frost from forming when it's cooled), we have oxygen monitors and big placards on the door. The risk is that the hose comes off, and that even a fairly low flow (say, 1-2 CFM, typically) can flush all the oxygen out overnight, for instance. Say the lab is 20x20x8 feet.. that's 3200 cubic feet, less a few hundred for the stuff that's in the lab (shelves, file cabinets, equipment racks, benches, etc). The hose pops off unnoticed and 12 hours later at a couple CGM, and the lab's full of nitrogen when you arrive the next morning. I can't recall the exact number for where you go unconscious, but certainly, somewhere around 10% O2 would probably be sufficient. You'd only have to displace half the air to get there. With LN2 there's also the problem that the gas is cold, so it pools more readily. LN2 and elevators is a bad combination if you get a catastrophic leak, because the elevator shaft fills. People do, in fact, die from this, every year. (though, thankfully, not at JPL).. There's cases of stagehands quietly suffocating below stage when they're using dryice foggers (CO2 is very much more dense and pools quite nicely). In stage shows where they use large quantities of LN2 for making fog (we're talking hundreds of gallons here!), they generally use a mix of LN2 and LO2 for just this reason. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rossini at blindglobe.net Sat Jan 31 19:52:27 2004 From: rossini at blindglobe.net (A.J. Rossini) Date: Sat, 31 Jan 2004 16:52:27 -0800 Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <007701c3e855$6c6c58d0$36a8a8c0@LAPTOP152422> (Jim Lux's message of "Sat, 31 Jan 2004 15:53:27 -0800") References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> <401C253E.9040206@obs.unige.ch> <007701c3e855$6c6c58d0$36a8a8c0@LAPTOP152422> Message-ID: <85llnnr584.fsf@servant.blindglobe.net> Thanks, all! Looks like LN2 isn't as bad as I thought. I still don't think we'll be using it as a temporary cooling measure, though :-). Looks like I'll be doing a local co-lo, and getting my collaborators to pay for the computing lab infrastructure... best, -tony "Jim Lux" writes: > Found some definitions: > An oxygen-deficient atmosphere means that there is not enough oxygen in the > space. > Normal air is made up of 20.8 percent oxygen. > > An oxygen-deficient atmosphere has less than 19.5 percent of oxygen. > > Levels below 10 percent can cause unconsciousness and levels below 8 > percent can quickly cause death. > > That's probably where Per's 19.5% came from > > And, a delightful paragraph from OSHA: > Oxygen deprivation is one form of asphyxiation. While it is desirable to > maintain the atmospheric oxygen level at 21% by volume, the body can > tolerate deviation from this ideal. When the oxygen level falls to 17%, the > first sign of hypoxia is a deterioration to night vision which is not > noticeable until a normal oxygen concentration is restored. Physiologic > effects are increased breathing volume and accelerated heartbeat. Between > 14-16% physiologic effects are increased breathing volume, accelerated > heartbeat, very poor muscular coordination, rapid fatigue, and intermittent > respiration. Between 6-10% the effects are nausea, vomiting, inability to > perform, and unconsciousness. Less than 6%, spasmatic breathing, convulsive > movements, and death in minutes. > > > ----- Original Message ----- > From: "Pfenniger Daniel" > To: "Per Lindstrom" > Cc: "John Hearns" ; > ; > Sent: Saturday, January 31, 2004 1:59 PM > Subject: Re: [Beowulf] HVAC and room cooling... > > >> Per Lindstrom wrote: >> > ATTENTION! >> > >> > Liquid Nitrogen is deadly dangerous as it will shut off the breathing >> > function of the human body when the Oxygen content falls below 19.5 % >> > and thereby causing suffocation. >> > >> Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly >> dangerous when inhaled in too large quantities. >> The quoted 19.5% are probably referring to the human body (blood?) >> concentration, not the inhaled air. Otherwise most of the people in >> pubs or small rooms would rapidly die as the normal O2 concentration >> in air is 20% > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- rossini at u.washington.edu http://www.analytics.washington.edu/ Biomedical and Health Informatics University of Washington Biostatistics, SCHARP/HVTN Fred Hutchinson Cancer Research Center UW (Tu/Th/F): 206-616-7630 FAX=206-543-3461 | Voicemail is unreliable FHCRC (M/W): 206-667-7025 FAX=206-667-4812 | use Email CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be confidential and privileged. If you received this message in error, please destroy it and notify the sender. Thank you. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 18:53:27 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 15:53:27 -0800 Subject: [Beowulf] HVAC and room cooling... References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> <401C253E.9040206@obs.unige.ch> Message-ID: <007701c3e855$6c6c58d0$36a8a8c0@LAPTOP152422> Found some definitions: An oxygen-deficient atmosphere means that there is not enough oxygen in the space. Normal air is made up of 20.8 percent oxygen. An oxygen-deficient atmosphere has less than 19.5 percent of oxygen. Levels below 10 percent can cause unconsciousness and levels below 8 percent can quickly cause death. That's probably where Per's 19.5% came from And, a delightful paragraph from OSHA: Oxygen deprivation is one form of asphyxiation. While it is desirable to maintain the atmospheric oxygen level at 21% by volume, the body can tolerate deviation from this ideal. When the oxygen level falls to 17%, the first sign of hypoxia is a deterioration to night vision which is not noticeable until a normal oxygen concentration is restored. Physiologic effects are increased breathing volume and accelerated heartbeat. Between 14-16% physiologic effects are increased breathing volume, accelerated heartbeat, very poor muscular coordination, rapid fatigue, and intermittent respiration. Between 6-10% the effects are nausea, vomiting, inability to perform, and unconsciousness. Less than 6%, spasmatic breathing, convulsive movements, and death in minutes. ----- Original Message ----- From: "Pfenniger Daniel" To: "Per Lindstrom" Cc: "John Hearns" ; ; Sent: Saturday, January 31, 2004 1:59 PM Subject: Re: [Beowulf] HVAC and room cooling... > Per Lindstrom wrote: > > ATTENTION! > > > > Liquid Nitrogen is deadly dangerous as it will shut off the breathing > > function of the human body when the Oxygen content falls below 19.5 % > > and thereby causing suffocation. > > > Yeah sure, this remind me the joke about dihydrogen oxyd (H2O) deadly > dangerous when inhaled in too large quantities. > The quoted 19.5% are probably referring to the human body (blood?) > concentration, not the inhaled air. Otherwise most of the people in > pubs or small rooms would rapidly die as the normal O2 concentration > in air is 20% _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Sat Jan 31 18:38:36 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Sat, 31 Jan 2004 15:38:36 -0800 Subject: [Beowulf] HVAC and room cooling... References: <401BE891.708@obs.unige.ch> <401C0807.4000209@telia.com> Message-ID: <006e01c3e853$592f0850$36a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Per Lindstrom" To: "Daniel Pfenniger" Cc: "John Hearns" ; ; Sent: Saturday, January 31, 2004 11:54 AM Subject: Re: [Beowulf] HVAC and room cooling... > ATTENTION! > > Liquid Nitrogen is deadly dangerous as it will shut off the breathing > function of the human body when the Oxygen content falls below 19.5 % > and thereby causing suffocation. > That high? I'm not so sure.. Consider for instance that conscious ness has mostly to do with partial pressure of oxygen in the gas being breathed, which then transfers to the PO2 in your blood. As a rule of thumb, the pressure drops by half with every 18,000 feet of elevation change (5500 m). Most people can tolerate a sudden pressure drop to an equivalent altitude of 5000 ft without losing consciousness, and, for instance, the Federal Aviation Administration, uses 10,000 ft as a cutoff. 19.5% oxygen, compared to the usual 21%, is a partial O2 pressure change of about 2000 ft or so (600m). I would go for 9.5% (it's well established that suddenly taking someone to 5000 meters will cause unconsciousness, although not that quickly... you have to go to something like 15,000 m/40,000 ft to get rapid unconsciousness). They do this sort of training in altitude chambers for pilots to impress on them the need to "keep that mask fastened tightly, even if you think you're getting sick" We're sort of off topic here, but think in terms of those little oxygen masks that drop "in case of a sudden cabin depressurization"... Not only will there be nosebleeds and broken eardrums a plenty if this happens, but I think you'd better move pretty fast to get that mask on. By the way, if you know the loss of oxygen is coming, you can hold your breath (or just not breathe) and be conscious for quite a while (30 seconds to minute would be fairly easy..). The problem with the nitrogen purge is that you don't realize it, and you keep on breathing, flushing the oxygen from your lungs, until it's too late. My math may be all wrong here, but that's the general idea. > This is not an unusual accident type in the Oil- & Gas- as well as in > the Shipping industry. For the sake of your safety I'm recommending the > use of an Industrial Safe Working Practice as a guideline before any > work commence. > > A reference: **http://www.medicine.uiowa.edu/biochemstores/Pages/ln2msds.htm > > Best regards > Per Lindstrom > > > Daniel Pfenniger wrote: > > > John Hearns wrote: > > > >> On Fri, 30 Jan 2004, A.J. Rossini wrote: > > > > > >>> to people that it isn't possible, but someone brought up "portable > >>> liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > >>> to get stable systems, not instability :-). > >>> > >> > >> Daft idea. > >> What happens when the liquid nitrogen vapourises? It turns to > >> nitrogen gas. And if you are unlucky anyone in the room chokes... > >> I was reminded of this in the lift of a chemistry department where > >> we have an opteron cluster - a large sign reminding that LNO2 and > >> dangerous chemicals are not allowed in passenger lifts. > > > > > > When liquid N2 vaporizes in a closed room the natural fraction of N2 > > in air, already 78% by volume, just increases. N2 is thus not toxic, > > the only risk by increasing N2 volume is to reach a too low O2 > > concentration > > for mammals beings breathing in the room. > > > > Dan > > _______________________________________________ > > Beowulf mailing list, Beowulf at beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Jan 31 08:34:55 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat, 31 Jan 2004 08:34:55 -0500 (EST) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: <85k739x8q8.fsf@servant.blindglobe.net> Message-ID: On Fri, 30 Jan 2004, A.J. Rossini wrote: > > I'm trying to do an HVAC analysis of a small beowulf cluster (12 dual > athlon or opteron machines). > > Can anyone suggest where I might look to find information for this? There are some documents on the brahma site (and my personal website) that might help. Quite a bit of their content has been drawn from extended threads on this list, so a look through the list archives is also in order. My book on the brahma site and an article in Linux Magazine and I think one other document should tell you how to work out required capacity and so forth. > Also, does anyone have any brilliant thoughts for cooling an internal > room that can't affordably get chilled water? (I've been suggesting > to people that it isn't possible, but someone brought up "portable > liquid nitrogen" -- for the room, NOT for overclocking -- I'm trying > to get stable systems, not instability :-). How big a room? Is it ducted for air conditioning of any sort? Does it have any windows or outside walls (probably not, given that "internal room", eh?). It isn't a matter of brilliance. The electrical cables running into the room can be thought of as pipes bringing in pure heat and pouring it out onto the floor. Some of that heat will leak out, driven by temperature differences, through the walls and ceiling, and the temperature of the room will rise until equilibrium of a sort is established (probably stabilizing too hot for most hardware). To run systems in there, you have to provide a means for removing that heat. Chilled water can be heated and removed/recycled. Chilled air can be heated and removed/recycled. Cooler air from surrounding air conditioned rooms can be forcably driven through, heated, and indirectly removed/recycled. You will have to provide some sort of mechanism for removing the heat, and they'll all involve moving cooled mass in and warmed mass out. Sorry. rgb > > best, > -tony > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From alvin at Mail.Linux-Consulting.com Sat Jan 31 21:51:01 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Sat, 31 Jan 2004 18:51:01 -0800 (PST) Subject: [Beowulf] HVAC and room cooling... In-Reply-To: Message-ID: hi ya On Sat, 31 Jan 2004, John Hearns wrote: > What happens when the liquid nitrogen vapourises? It turns to nitrogen > gas. And if you are unlucky anyone in the room chokes... > I was reminded of this in the lift of a chemistry department where > we have an opteron cluster - a large sign reminding that LNO2 and > dangerous chemicals are not allowed in passenger lifts. > > > BTW, when arranging air flows try to get cold air ducted into the front of > the rack - node tend to draw air in the front and exhaust to the rear. it'd be better, if one could, to make it bottom to top air flow and all inside a sealed ( reasonably closed ) rack if co-location is an option... the best choice would be alsaka or canada and leave the 3rd/4th story window open to the outside ?? ( wonder how it'd run during the short summers ) - and yes.. lots of people forget about regulatory issues when building clusters or too many pc's in the office - not enough building power being the first major "oops" - fire hazard being the 2nd violations ( they(fire dept) don't allow daisy chained power strips ) - not enough hvac or costs of cooling .. ( sometimes big fans, in the right direction, is enough to keep the pc cooler ) - too much regulations also kills the project :-0 c ya alvin _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf