From hahn at mcmaster.ca Thu Mar 1 01:16:49 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 1 Mar 2012 01:16:49 -0500 (EST) Subject: [Beowulf] amd buys seamicro Message-ID: http://www.eetimes.com/electronics-news/4237271/AMD-to-buy-microserver-startup-SeaMicro this is interesting. most of the coverage seems to interpret this as using opterons (which makes some sense, given the direction bulldozer is going, towards lots of space/power-effective cores.) but here's another prospect: a box with lots of APU chips that max out GPU density. lotsa gflops/watt, very compact... seamicro says their interconnect is special, low-lat, high-bw, but it sounds like an onboard 10Gb chip to me. calxeda's onboard distributed switch might be more interesting. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From tegner at renget.se Thu Mar 1 01:52:36 2012 From: tegner at renget.se (Jon Tegner) Date: Thu, 01 Mar 2012 07:52:36 +0100 Subject: [Beowulf] Functionality of schedulers Message-ID: <4F4F1CB4.4010303@renget.se> Hi list! Is there any scheduler which has the functionality to automatically put a running job on hold when another job with higher priority is submitted? Preferably the state of the first job should be frozen, and saved to disk, so that it can be restarted again when the higher priority job has finished. Is this at all possible (we are using torque/maui, and I couldn't find this feature there)? Regards, /jon _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From j.wender at science-computing.de Thu Mar 1 02:20:28 2012 From: j.wender at science-computing.de (Jan Wender) Date: Thu, 01 Mar 2012 08:20:28 +0100 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: <4F4F233C.3000507@science-computing.de> Hi Jon, Jon Tegner schrieb: > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? AFAIK at least LSF has this as a feature called preemption. > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. That depends probably mostly on the application. If the application offers it, then the batch system can use it to save state. I don't know much about kernel level checkpointing, though. Cheerio, Jan -- ---- Company Information ---- Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Roland Niemeier, Dr. Arno Steitz, Dr. Ingrid Zech Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: Philippe Miltin Sitz/Registered Office: Tuebingen Registergericht/Registration Court: Stuttgart Registernummer/Commercial Register No.: HRB 382196 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- A non-text attachment was scrubbed... Name: j_wender.vcf Type: text/x-vcard Size: 367 bytes Desc: not available URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at gmail.com Thu Mar 1 02:22:12 2012 From: raysonlogin at gmail.com (Rayson Ho) Date: Thu, 1 Mar 2012 02:22:12 -0500 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: Jon, Almost every cluster job scheduler supports preemption (use it as the google keyword and you will find lots of references...). Torque has job preemption. I have not used Torque for a while (I used to be an OpenPBS user) so I am not the best person to answer the question for Torque. However, if you google "job preemption"+torque, you should be able to find some useful info. In Grid Engine (and now Open Grid Scheduler) there is Subordinate Queues: http://docs.oracle.com/cd/E19080-01/n1.grid.eng6/817-5677/i998889/index.html Condor is well-known for cycle stealing, and it also offers the checkpoint restart library for Open Grid Scheduler/Grid Engine and other batch systems: http://research.cs.wisc.edu/condor/checkpointing.html http://gridscheduler.sourceforge.net/howto/checkpointing.html (I am sure that you can integrate checkpointing with Torque so you don't need to look for a new batch system to get what you need.) Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ On Thu, Mar 1, 2012 at 1:52 AM, Jon Tegner wrote: > Hi list! > > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? > > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. > > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? > > Regards, > > /jon > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From raysonlogin at gmail.com Thu Mar 1 03:22:16 2012 From: raysonlogin at gmail.com (Rayson Ho) Date: Thu, 1 Mar 2012 03:22:16 -0500 Subject: [Beowulf] LSF Job Preemption & Checkpointing Message-ID: On Thu, Mar 1, 2012 at 2:20 AM, Jan Wender wrote: > AFAIK at least LSF has this as a feature called preemption. IMO, LSF has the best job preemption & checkpointing support, with the least integration effort needed from the end user & cluster administrator. And resource preemption and license preemption are the more advanced features of LSF. (There are more manual configuration needed for Grid Engine & Open Grid Scheduler and/or other batch systems - not impossible, but needs knowledge on how to tune the scheduler.) > That depends probably mostly on the application. If the application offers > it, then the batch system can use it to save state. > I don't know much about kernel level checkpointing, though. There are 3 types of checkpointing supported by LSF: 1) kernel-level 2) user-level 3) application-level Kernel level is easy, the OS kernel handles everything for the user (for interactively processes) & the batch system (for jobs). However, only IRIX, Cray UNICOS, and NEC SUPER-UX support kernel-level checkpointing. On Linux, you usually need to patch the kernel: - "Checkpoint/restart: it's complicated": http://lwn.net/Articles/414264/ - "Kernel-based checkpoint and restart": http://lwn.net/Articles/293575/ (Lots of discussions on kernel-level checkpointing in the past few years but still we don't have anything in the official tree yet...) Or even kernel assisted user-level checkpointing: - "Preparing for user-space checkpoint/restore": http://lwn.net/Articles/478111/ And there is also the famous Berkeley Lab Checkpoint/Restart (BLCR), which is a kernel module and thus you can use your distribution's stock kernel: - "RCE 12: BLCR": http://www.rce-cast.com/Podcast/rce-12-blcr.html - "Checkpointing under Linux with Berkeley Lab Checkpoint/Restart": http://gridscheduler.sourceforge.net/howto/APSTC-TB-2004-005.pdf For user-level, you will need to link against a checkpointing library shipped with LSF, which (I think) has some object file level init routines that perform initializations to properly save the state of stuff and also need to wrap around standard libc functions & system calls (I forgot the actual details, lots of academic papers published 15 years ago and I recall reading a few of them, but just don't recall the content :-D ). See "Standalone Checkpointing": http://research.cs.wisc.edu/condor/checkpointing.html With user-level checkpointing & restart, you usually need to relink your application (unless you use the LD_PRELOAD trick). So for operating systems that don't support kernel-level checkpointing (ie. most of the OSes), user-level checkpointing usually works for most general applications (I *think* Platform Computing even ported the LSF checkpointing library to Windows as well - or at least that's what I was told). For application-level checkpointing, the applications will handle everything. But of course each application needs to have its own built-in support for checkpoint & restart. Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ > > Cheerio, Jan > -- > ---- Company Information ---- > Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, > Dr. Roland Niemeier, Dr. Arno Steitz, Dr. Ingrid Zech > Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: > Philippe Miltin > Sitz/Registered Office: Tuebingen Registergericht/Registration Court: > Stuttgart > Registernummer/Commercial Register No.: HRB 382196 > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From john.hearns at mclaren.com Thu Mar 1 06:37:58 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Thu, 1 Mar 2012 11:37:58 -0000 Subject: [Beowulf] Pbsnodes xml format Message-ID: <207BB2F60743C34496BE41039233A8090BC87925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Slightly off topic. Would some kind soul who is running Torque send me some sample output from pbsnodes -x Thanks The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From samuel at unimelb.edu.au Thu Mar 1 06:46:47 2012 From: samuel at unimelb.edu.au (Chris Samuel) Date: Thu, 1 Mar 2012 22:46:47 +1100 Subject: [Beowulf] Functionality of schedulers In-Reply-To: References: <4F4F1CB4.4010303@renget.se> Message-ID: <201203012246.47618.samuel@unimelb.edu.au> On Thursday 01 March 2012 18:22:12 Rayson Ho wrote: > Torque has job preemption. I have not used Torque for a while (I > used to be an OpenPBS user) so I am not the best person to answer > the question for Torque. However, if you google "job > preemption"+torque, you should be able to find some useful info. It also integrates with BLCR (Berkeley Lab Checkpoint/Restart) to provide kernel level checkpointing support for Linux, including for MPI applications if you are using Open-MPI. Useful links: BLCR: https://ftg.lbl.gov/projects/CheckpointRestart/ Open-MPI and BLCR: http://www.open-mpi.org/faq/?category=ft Torque and checkpoint/restore and BLCR: http://www.clusterresources.com/torquedocs/2.6jobcheckpoint.shtml Hope this helps! Chris -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ntmoore at gmail.com Thu Mar 1 08:57:52 2012 From: ntmoore at gmail.com (Nathan Moore) Date: Thu, 1 Mar 2012 07:57:52 -0600 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: I think the command you're looking for is, > sudo nice -n -20 ./my_important_program On Thu, Mar 1, 2012 at 12:52 AM, Jon Tegner wrote: > Hi list! > > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? > > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. > > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? > > Regards, > > /jon > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- - - - - - - -?? - - - - - - -?? - - - - - - - Nathan Moore Associate Professor, Physics Winona State University - - - - - - -?? - - - - - - -?? - - - - - - - _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 1 10:44:06 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 1 Mar 2012 10:44:06 -0500 (EST) Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. well, maybe. that process (checkpoint/restore) really makes sense only if the preemtor is giant and/or long. otherwise SIGSTOP is a much better solution (it implies that you should have swap, but you should have swap anyway.) > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? this code (even moab) has all sorts of problems keeping track of suspension. the weak spot is usually that when you suspend a parallel job and the preemptor doesn't use all the cpus, you can't go starting random other jobs on these pseudo-free cpus. LSF wasn't all that great about this little detail either, at least back in 6.x versions. it's kind of amazing how poor all the schedulers are, really. classic example of how projects get sclerotic by adding features... regards, mark hahn _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 2 10:39:57 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 02 Mar 2012 10:39:57 -0500 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: <4F50E9CD.1050704@ias.edu> On 03/01/2012 01:52 AM, Jon Tegner wrote: > Hi list! > > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? > > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. > > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? > SGE can do this more or less, but it doesn't write the the job state to disk, unless you count swapping. In SGE you can create multiple queues, where one is subordinate (lower priority) to another. When a job is running in the subordinate queue and then a job submitted in a superior queue, the job is the subordinate queue will be paused while the higher priority job runs. The problem with this that the subordinate job is only paused it stays in memory, and it doesn't free up any resources it consumed (software licenses, etc), and it won't be migrated to other hosts If you want a job to be written to disk so it can be completely, or be able to be migrated to other hosts, you might want to look at checkpoint restart options like BLCR, if that's applicable to your situation. -- Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From trainor at presciencetrust.org Sat Mar 3 11:04:19 2012 From: trainor at presciencetrust.org (Douglas J. Trainor) Date: Sat, 3 Mar 2012 11:04:19 -0500 Subject: [Beowulf] Ninja Blocks (Re: Raspberry Pi) In-Reply-To: <4F4EAB27.2080201@unimelb.edu.au> References: <4F4EAB27.2080201@unimelb.edu.au> Message-ID: <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> Ninja Blocks are being funded by a Kickstarter campaign -- http://www.kickstarter.com/projects/ninja/ninja-blocks-connect-your-world-with-the-web This little computer group is emphasizing the software and sensors, and they're build to order. douglas On Feb 29, 2012, at 5:48 PM, Christopher Samuel wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 01/03/12 08:40, Douglas J. Trainor wrote: > >> thought some people here should see the Raspberry Pi -- > > Been following this for a while, they're rather neat little devices.. > > http://www.raspberrypi.org/#modelb > > http://en.wikipedia.org/wiki/Raspberry_Pi#Hardware > > Their usual site is down at the moment as it couldn't cope with demand > (but then again, neither could Farnell or RS, their retailers :-) ). > > cheers, > Chris > - -- > Christopher Samuel - Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.unimelb.edu.au/ > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Sun Mar 4 14:35:59 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Sun, 4 Mar 2012 14:35:59 -0500 (EST) Subject: [Beowulf] seamicro fabric? Message-ID: Has anyone found an informative description of the Seamicro fabric? various heavily masticated reports on the web say it's a 3d torus, "low latency" and 160 GB/s (per link? bisecection? in-chassis?) thanks, mark. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Mon Mar 5 09:35:06 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Mon, 05 Mar 2012 09:35:06 -0500 Subject: [Beowulf] Ninja Blocks (Re: Raspberry Pi) In-Reply-To: <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> References: <4F4EAB27.2080201@unimelb.edu.au> <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> Message-ID: <4F54CF1A.1060709@ias.edu> Ninja Blocks looks like Arduinos with sexier packaging. Prentice On 03/03/2012 11:04 AM, Douglas J. Trainor wrote: > Ninja Blocks are being funded by a Kickstarter campaign -- > > http://www.kickstarter.com/projects/ninja/ninja-blocks-connect-your-world-with-the-web > > This little computer group is emphasizing the software and sensors, and they're build to order. > > douglas > > On Feb 29, 2012, at 5:48 PM, Christopher Samuel wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 01/03/12 08:40, Douglas J. Trainor wrote: >> >>> thought some people here should see the Raspberry Pi -- >> Been following this for a while, they're rather neat little devices.. >> >> http://www.raspberrypi.org/#modelb >> >> http://en.wikipedia.org/wiki/Raspberry_Pi#Hardware >> >> Their usual site is down at the moment as it couldn't cope with demand >> (but then again, neither could Farnell or RS, their retailers :-) ). >> >> cheers, >> Chris >> - -- >> Christopher Samuel - Senior Systems Administrator >> VLSCI - Victorian Life Sciences Computation Initiative >> Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 >> http://www.vlsci.unimelb.edu.au/ >> > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Mon Mar 5 15:07:10 2012 From: eugen at leitl.org (Eugen Leitl) Date: Mon, 5 Mar 2012 21:07:10 +0100 Subject: [Beowulf] [FoRK] seamicro fabric? Message-ID: <20120305200710.GW9891@leitl.org> ----- Forwarded message from Aaron Burt ----- From: Aaron Burt Date: Mon, 5 Mar 2012 11:31:37 -0800 To: fork at xent.com Subject: Re: [FoRK] [Beowulf] seamicro fabric? User-Agent: Mutt/1.5.20 (2009-06-14) Reply-To: Friends of Rohit Khare On Mon, Mar 05, 2012 at 05:23:02AM +0100, Eugen Leitl wrote: > From: Mark Hahn > > Has anyone found an informative description of the Seamicro fabric? I hope Mark has repeated his search; http://google.com?q=anil+rao+fabric and http://google.com?q=seamicro+interconnect turn up lots of info. A 2011 PDF[1] at Seamicro sez, "It is a three-dimensional torus, with both path redundancy and diversity. The fabric is FLIT-based and wormhole- routed, with integrated virtual-channel technology to manage congestion, and has a throughput of 1.2 Terabits/sec." Looks like proc boards use PCIe interconnects, like the old cPCI blades. The more interesting bit (to me) is the possibility raised in an older article[2] that Rao and Lauterbach are thinking of turning their interconnect into a datacenter-level fabric. Infiniband II, anyone? Remembering the days of physically separate frontend and I/O processors, Aaron [1] http://www.seamicro.com/sites/default/files/TO1_SM10000_Technology_Overview.pdf [2] http://www.datacenterknowledge.com/archives/2010/01/26/seamicro-more-than-just-low-power-servers/ _______________________________________________ FoRK mailing list http://xent.com/mailman/listinfo/fork ----- End forwarded message ----- -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From raysonlogin at gmail.com Mon Mar 5 15:20:36 2012 From: raysonlogin at gmail.com (Rayson Ho) Date: Mon, 5 Mar 2012 15:20:36 -0500 Subject: [Beowulf] [FoRK] seamicro fabric? In-Reply-To: <20120305200710.GW9891@leitl.org> References: <20120305200710.GW9891@leitl.org> Message-ID: You can find lots of info from this EE380 talk at Stanford: "Architectural tradeoffs in the Sea Micro SM 10000 Server" http://www.stanford.edu/class/ee380/Abstracts/100922.html Video archive: http://ee380.stanford.edu/cgi-bin/videologger.php?target=100922-ee380-300.asx Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ On Mon, Mar 5, 2012 at 3:07 PM, Eugen Leitl wrote: > ----- Forwarded message from Aaron Burt ----- > > From: Aaron Burt > Date: Mon, 5 Mar 2012 11:31:37 -0800 > To: fork at xent.com > Subject: Re: [FoRK] [Beowulf] seamicro fabric? > User-Agent: Mutt/1.5.20 (2009-06-14) > Reply-To: Friends of Rohit Khare > > On Mon, Mar 05, 2012 at 05:23:02AM +0100, Eugen Leitl wrote: >> From: Mark Hahn >> >> Has anyone found an informative description of the Seamicro fabric? > > I hope Mark has repeated his search; http://google.com?q=anil+rao+fabric > and http://google.com?q=seamicro+interconnect turn up lots of info. > > A 2011 PDF[1] at Seamicro sez, "It is a three-dimensional torus, with both > path redundancy and diversity. The fabric is FLIT-based and wormhole- > routed, with integrated virtual-channel technology to manage congestion, > and has a throughput of 1.2 Terabits/sec." > > Looks like proc boards use PCIe interconnects, like the old cPCI blades. > > The more interesting bit (to me) is the possibility raised in an older > article[2] that Rao and Lauterbach are thinking of turning their > interconnect into a datacenter-level fabric. ?Infiniband II, anyone? > > Remembering the days of physically separate frontend and I/O processors, > ?Aaron > > [1] http://www.seamicro.com/sites/default/files/TO1_SM10000_Technology_Overview.pdf > [2] http://www.datacenterknowledge.com/archives/2010/01/26/seamicro-more-than-just-low-power-servers/ > _______________________________________________ > FoRK mailing list > http://xent.com/mailman/listinfo/fork > > ----- End forwarded message ----- > -- > Eugen* Leitl leitl http://leitl.org > ______________________________________________________________ > ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org > 8B29F6BE: 099D 78BA 2FD3 B014 B08A ?7779 75B0 2443 8B29 F6BE > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From samuel at unimelb.edu.au Tue Mar 6 01:19:17 2012 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Tue, 06 Mar 2012 17:19:17 +1100 Subject: [Beowulf] Ninja Blocks (Re: Raspberry Pi) In-Reply-To: <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> References: <4F4EAB27.2080201@unimelb.edu.au> <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> Message-ID: <4F55AC65.8080405@unimelb.edu.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/03/12 03:04, Douglas J. Trainor wrote: > Ninja Blocks are being funded by a Kickstarter campaign -- I don't think they run Linux though, unlike Raspberry Pi. - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9VrGUACgkQO2KABBYQAh+CVwCfUeRkHYPxHFMuL14zWfMG/AOE 3IkAoI9HlQAXY6hb903f1b3gSFxKO+0V =dHho -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From john.hearns at mclaren.com Thu Mar 8 08:43:10 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Thu, 8 Mar 2012 13:43:10 -0000 Subject: [Beowulf] Supercomputers - iPad versus Cray Message-ID: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ A rather nice Register article on costs for supercomputers, adjusted to 2010 dollars, And a rather interesting cost per megaflop table on the second page. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Mar 8 09:27:15 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 06:27:15 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: Of some interest is Jack Dongarra's measurements on the iPad http://bits.blogs.nytimes.com/2011/05/09/the-ipad-in-your-hand-as-fast-as-a-supercomputer-of-yore/ And his speculation on a cluster. (wouldn't be a Beowulf: not commodity hardware, not open source OS, etc.) While it might do the flops, the interconnect bandwidth is quite low. I assume it would have to use WiFi (I don't know how fast the tether port is, but I don't think it's a real ball o'fire. It's hard to measure. You can plug a USB disk drive into the photo adapter so maybe your interconnect could simulate that. From: "Hearns, John" > Date: Thu, 8 Mar 2012 05:43:10 -0800 To: "beowulf at beowulf.org" > Subject: [Beowulf] Supercomputers - iPad versus Cray http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ A rather nice Register article on costs for supercomputers, adjusted to 2010 dollars, And a rather interesting cost per megaflop table on the second page. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Mar 8 09:37:38 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 06:37:38 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: What would be interesting is to know what is the value not the cost of a FLOP? I tend to look at computational horsepower in the tool sense. If my job is cutting wood, a chainsaw lets me cut a lot more wood than I could with a hand saw. Since there's some value in a unit of wood cutting, one could come up with a chainsaw value in cords cut. Likewise, we tend to evaluate computing solutions in terms of "how much more work can an engineer get done using the computation, than some alternative" (leaving aside the delightful negative productivity from Angry Birds on an iPad, but gosh it's great on a long plane flight) There's stories about nuclear reaction dynamics computations using Electric Accounting Machinery (EAM): tabulators, sorters, etc, which was then replaced by early computers. And the EAM equipment replaced rooms of computers (the living breathing kind with rows of Marchand calculators). SO there's some data there to compare with. But I think that simplifying it to instruction counts (valid for EAM vs room of calculators) doesn't account for "change in way of solving problem" Back in the day, people designed antennas by empiricism and some analytic approximations. But now, there are FEM codes, and the analytic forms are used more to validate that test cases run correctly on the FEM, and for "quick and dirty" estimates. Likewise, structural design trades many simple calculations in a FEM code for longer form analytical approaches, and the ability to do the FEM enables the design of structures that are MUCH more complex. What is the "value" of being able to do a more complex structure? Or design optics that aren't simple spherical and cylinder sections? From: "Hearns, John" > Date: Thu, 8 Mar 2012 05:43:10 -0800 To: "beowulf at beowulf.org" > Subject: [Beowulf] Supercomputers - iPad versus Cray http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ A rather nice Register article on costs for supercomputers, adjusted to 2010 dollars, And a rather interesting cost per megaflop table on the second page. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at mclaren.com Thu Mar 8 10:08:05 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Thu, 8 Mar 2012 15:08:05 -0000 Subject: [Beowulf] Supercomputers - iPad versus Cray References: Message-ID: <207BB2F60743C34496BE41039233A8090C0C3E1A@MRL-PWEXCHMB02.mil.tagmclarengroup.com> What would be interesting is to know what is the value?not the cost?of a FLOP? I tend to look at computational horsepower in the tool sense. ?If my job is cutting wood, a chainsaw lets me cut a lot more wood than I could with a hand saw. ?Since there's some value in a unit of wood cutting, one could come up with a chainsaw value in cords cut. Likewise, we tend to evaluate computing solutions in terms of "how much more work can an engineer get done using the computation, than some alternative" (leaving aside the delightful negative productivity from Angry Birds on an iPad, but gosh it's great on a long plane flight) That is a very good point. There is also, as we know on here, value in 'time to solution' - which is what supercomputers with more mega/terafloppage can offer. Calculations which were possible years ago, but which would take so long that you couldn't be bothered to wait for the result are now done. One example was given by Jon 'Maddog' Hall in the very first talk about Beowulfery which I attended (at Compaq's offices in London). A group in South America had built a Beowulf cluster, which was used for recognition of (I think) mammograms. In the past, women who had to travel for maybe hours or days had to make a return visit to the clinic to get their diagnosis. If the recognition could be run within the time of one clinic then the results are so much more useful. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 10:57:44 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 10:57:44 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: <4F58D6F8.5050708@cse.psu.edu> On 03/08/2012 08:43 AM, Hearns, John wrote: > http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ > > A rather nice Register article on costs for supercomputers, adjusted to > 2010 dollars, Without being overly pessimistic, I get a vibe of "contrived" throughout the article. For instance, as you mention, the supers are CPI adjusted for present day worth, which is quite interesting to see the trend of increasing cost for supers. However, it just ends there -- our curiosity as to the much more important question, why that trend exists, goes unsatisfied. I have some guesses about why, but it would be far more interesting to see hard evidence (or even squishy evidence, really any evidence) as to why than to simply speculate. For instance, I'm sure there are figures on the older systems at least that break down the overall cost into costs of development versus hardware, or perhaps even more interesting, the respective costs of different pieces of hardware in the system (how much is spent on I/O versus CPU versus Net, etc). Seeing a rapid growth in development costs or that new systems are spending way more on the interconnect than the CPU or vice-versa would shed much needed light on an otherwise unexplained finding. > And a rather interesting cost per megaflop table on the second page. I actually hate this table. If we want to compare flops/cost between machines, we should at least level the playing field: find Linpack numbers on individual machines in the listed clusters and compare them to your desktop or iPad. After all, in the latter cases, we have no interconnect issues to worry about (just imagine Linpack numbers over 3G or even Wifi to other iPads...probably worse for 2 than for 1). I think this table does a disservice to all the complexities that super-designers have to deal with, dumbing down the great hurdles they must cope with to get Linpack to scale across the entire cluster. Just my 2c, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 13:42:57 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 13:42:57 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: <4F58FDB1.4080008@cse.psu.edu> On 03/08/2012 01:30 PM, Mark Hahn wrote: > including how you could possibly > spend $10k on a dual-socket box Yea, I was giving him the benefit of the doubt on that one and just hoping he bought the parts some years back, or has crazy expensive water cooling, but in essence yes the 120 number for flops is embarrassing given how much he paid. Also, his wife having bought that computer for 1700 is equally brow-raising. My desktop directly compares to that desktop machine but was bought for $800 in early 2009... It seems given his inclination to overpay for things and the interest in using iPads for clustering, he has too much money on his hands :). The article with Dongarra is particularly weird however, as that is a guy who /does/ know a lot about computing. Perhaps it was just a sensationalist spin for the NYT since so many now have iPads at home so the average iJoe will connect with the article (read: Oh I have a dated supercomputer in my hands!). Either way, it still surprised me as the article should have just ended before the laughable suggestion that we tether these chic monitors with a brain together. That was a little disturbing coming from him. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 8 13:30:57 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 8 Mar 2012 13:30:57 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F58D6F8.5050708@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: >> http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ >> >> A rather nice Register article on costs for supercomputers, adjusted to >> 2010 dollars, he really should have talked to someone who knows computers first, though. lot of embarassing nonsense in that article (including how you could possibly spend $10k on a dual-socket box. or why you'd choose an ipad (65% profit margin, and whose BOM is mainly display.)) > Without being overly pessimistic, I get a vibe of "contrived" throughout > the article. as opposed to other reg articles? ;) > For instance, as you mention, the supers are CPI adjusted > for present day worth, which is quite interesting to see the trend of > increasing cost for supers. However, it just ends there -- our > curiosity as to the much more important question, why that trend exists, > goes unsatisfied. supers are in a kind of crazy arms race. what I'd really like to see is an article that explores exactly what code runs on the largest (say top100) computers. I'm not saying I disbelieve any code would scale to 700k cores (K computer) just that I don't know of any science that would. I'm certainly first to admit I don't know anything about world-class HPC (Canadian HPC is pretty much a flop, no pun intended) but would science be better off with 100x 7k-core centi-K computers? I suspect so. there are a couple premises that should be questioned: - are scale-limited problems where the interesting science is? I talk to cosmologists a lot - they seem to be many orders of magnitude from being able to resolve their physics. I'm not sure the same applies to MD, q-chem, sequence analysis, etc. I'm also not sure that just because a field is scale-limited, that's where the effort/money should go. - is there some assumption that larger computers provide economies of scale? surely this is untrue: past a certain point, larger computers require more overhead (interconnect is often nonlinear, practical issues require greater attention to cooling, density, and reliability dictates that you simply can't use the same parts at 700kcore as you can at 7kcore.) >> And a rather interesting cost per megaflop table on the second page. > > I actually hate this table. If we want to compare flops/cost between > machines, we should at least level the playing field: find Linpack > numbers on individual machines in the listed clusters and compare them > to your desktop or iPad. using iPad is just stupid, since it's an embarassingly expensive piece of eyecandy, not a computer. (yeah, yeah, you might love yours, but you still wrote a cheque for 65% of what you paid that landed directly on Apple's big pile of cash.) there are dozens of low-overhead chips out there that make a better comparison. > After all, in the latter cases, we have no > interconnect issues to worry about (just imagine Linpack numbers over 3G > or even Wifi to other iPads...probably worse for 2 than for 1). I think it's not worth thinking about, wifi being half-duplex. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 8 14:01:27 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 11:01:27 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Mark Hahn Sent: Thursday, March 08, 2012 10:31 AM To: Beowulf Mailing List Subject: Re: [Beowulf] Supercomputers - iPad versus Cray using iPad is just stupid, since it's an embarassingly expensive piece of eyecandy, not a computer. (yeah, yeah, you might love yours, but you still wrote a cheque for 65% of what you paid that landed directly on Apple's big pile of cash.) Ahh.. but precisely because it is eyecandy means that $/FLOP means you're comparing the wrong metric. iPads are not computers. They're media storage and display devices. So the value (not the cost) needs to be compared to alternatives. It's "is my iPad cheaper than buying and carrying hardback books, watching pay-per-view videos,etc on an airplane or hotel room or airport concourse." I decided to get an eReader/iPad when I was stuck at Phoenix SkyHarbor, having just paid over $20 for some hardcover book (which was a reasonable purchase price), and now lugging the book around. And realizing that I had carried various and sundry hard and softcover books on most trips over the past couple years. And then, contemplating a 2 week long trip to Egypt without internet connectivity, but with teenage children and a couple of 20 hour-ish plane trips (counting layovers, etc), where the ability to get some mindless entertainment that I could just "watch" would be nice. I can load up my iPad with dozens of books (not to mention countless .pdfs of journal articles that I never seem to have time to read) and with a half dozen movies ripped from DVDs I own. *that* is what makes the multi-hundred dollar expense worth it. Whether it cost Steve&Co a penny to manufacture, and Steve pocketed the rest, or whether they took a loss is immaterial. It was worth the cash, to me. As it happens, now that I have the device, I have found other uses (makes a dandy user interface to stuff controlled by a web-server interface... when I get my arduino temperature/humidity controller done, I'm going to use the iPad (or an iPhone/iPodTouch) as the "touchable and viewable" part of the UI. Would it make a good computational element? I don't know. It's reasonably power efficient in a joules/computation basis, and certainly, as an element in a toy/gimmick cluster to demonstrate things it might be nice. (but a better solution might be a bunch of androids or iPodTouch-es... cheaper, smaller, still make a cool tabletop demo, especially if you could figure out a way to do different "tinkerable" interconnects) I can see a "mess o'iTouch" thing to do demonstrations of mesh network routing, for instance. Or perhaps to demonstrate aspects of cooperative processing in a fractionated spacecraft. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 8 14:02:51 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 11:02:51 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F58FDB1.4080008@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Ellis H. Wilson III Sent: Thursday, March 08, 2012 10:43 AM To: beowulf at beowulf.org Subject: Re: [Beowulf] Supercomputers - iPad versus Cray The article with Dongarra is particularly weird however, as that is a guy who /does/ know a lot about computing. Perhaps it was just a sensationalist spin for the NYT since so many now have iPads at home so the average iJoe will connect with the article (read: Oh I have a dated supercomputer in my hands!). Either way, it still surprised me as the article should have just ended before the laughable suggestion that we tether these chic monitors with a brain together. That was a little disturbing coming from him. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From deadline at eadline.org Thu Mar 8 14:19:00 2012 From: deadline at eadline.org (Douglas Eadline) Date: Thu, 8 Mar 2012 14:19:00 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: > http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ > > > > A rather nice Register article on costs for supercomputers, adjusted to > 2010 dollars, > > And a rather interesting cost per megaflop table on the second page. The table is a bit silly, but as another data point, my Limulus box can do 200 GFLOPS (running HPL) for less than $5K, that puts it at $.025/MFLOP, or better than anything in the table. BTW, they should be launching real soon. -- Doug > > > The contents of this email are confidential and for the exclusive use of > the intended recipient. If you receive this email in error you should not > copy it, retransmit it, use it or disclose its contents but should return > it to the sender immediately and delete your copy. > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Doug -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From prentice at ias.edu Thu Mar 8 14:45:00 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Thu, 08 Mar 2012 14:45:00 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: <4F590C3C.5060705@ias.edu> Man climbed the mountain, because it was there, and techies will continue to build clusters and other things out of unusual hardware merely because they are there, too. At the risk of being flamed, I think some of you are missing the point. Neither the Register article or Dongarra's article were advocating for the building clusters from iPads, I think they only meant to show how far computing hardware has come in terms of power and cost. To say that people should only build clusters out of the proper hardware is like telling artists to throw out their paints and brushes since we now have cameras, that do a better job than they can. BTW - I'd never waste my time building a cluster out of iPads, but I defend the right of others to do so if that's what turns them on. -- Prentice On 03/08/2012 02:02 PM, Lux, Jim (337C) wrote: > Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. > > -----Original Message----- > From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Ellis H. Wilson III > Sent: Thursday, March 08, 2012 10:43 AM > To: beowulf at beowulf.org > Subject: Re: [Beowulf] Supercomputers - iPad versus Cray > > The article with Dongarra is particularly weird however, as that is a guy who /does/ know a lot about computing. Perhaps it was just a sensationalist spin for the NYT since so many now have iPads at home so the average iJoe will connect with the article (read: Oh I have a dated supercomputer in my hands!). Either way, it still surprised me as the article should have just ended before the laughable suggestion that we tether these chic monitors with a brain together. That was a little disturbing coming from him. > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 15:24:52 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 15:24:52 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: <4F591594.6090504@cse.psu.edu> On 03/08/2012 02:01 PM, Lux, Jim (337C) wrote: > -----Original Message----- > From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Mark Hahn > Sent: Thursday, March 08, 2012 10:31 AM > To: Beowulf Mailing List > Subject: Re: [Beowulf] Supercomputers - iPad versus Cray > using iPad is just stupid, since it's an embarassingly expensive piece of eyecandy, not a computer. (yeah, yeah, you might love yours, but you still wrote a cheque for 65% of what you paid that landed directly on Apple's big pile of cash.) > > > Ahh.. but precisely because it is eyecandy means that $/FLOP means you're comparing the wrong metric. iPads are not computers. They're media storage and display devices. So the value (not the cost) needs to be compared to alternatives. It's "is my iPad cheaper than buying and carrying hardback books, watching pay-per-view videos,etc on an airplane or hotel room or airport concourse." I fully agree that iPads and similar tablet devices fill a much-needed gap between tiny-screened and underpowered phones and full-blown computers. However, I certainly didn't intend to, nor did I get from other peoples posts, to argue about the iPads worth for the purpose it was created. I argued this is a foolish, sensationalist use of it for a cluster. Whether or not an iPad is a good tablet I take no position on, having never owned one myself nor shopped around for them. Regarding power/computation, I think it's a silly metric to consider for this usage (probably just plugged into the wall). If we're concerned about the cost to power a cluster when all nodes are on, that's a fair concern, but it has to scale first. These machines won't allow for scaling with their included hardware, so that's not a reasonable consideration. If instead we want to have a cluster that is low power because we'll be running off battery for some duration, then it's foolish to use these since they rely on wifi or 3g to communicate, not to mention having a touchscreen, accelerometer and all kinds of other gadgets that eat battery-life. In short, I think the power/computation argument is totally contrived to give a scientific spin to an otherwise sensationalist "research" directive. In fact, I'd faster support a cluster of furbies because it's overtly absurd and frankly, funnier, than a cluster of iPads. This has the dangerous potential to being taken seriously (i.e. calling it "research" and aligning it with the directives of a big university such as Tennessee). If he just came out and said, "this is ridiculous, but we're bored and have some funding to burn" then I'd be on-board ;D. I'm sure Engadget or someone similar would love an article like that, not to mention Apple. Last, regarding educational purposes for mesh network routing, there are plenty of systems that are cheap as dirt that would be better for this kind of stuff, such as the ones we've been discussing on the list recently, that are far cheaper. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 15:47:29 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 15:47:29 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F590C3C.5060705@ias.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> <4F590C3C.5060705@ias.edu> Message-ID: <4F591AE1.6080305@cse.psu.edu> On 03/08/2012 02:45 PM, Prentice Bisbal wrote: > Man climbed the mountain, because it was there, and techies will > continue to build clusters and other things out of unusual hardware > merely because they are there, too. I'm not certain the mountain metaphor follows. For instance, I'm not going to jump out of the window next to my desk just because it is there, because surviving it will be challenging, or because it will entertain people watching me attempt it. There's more going on with mountains and with this iPad clustering. > At the risk of being flamed, I think some of you are missing the point. > Neither the Register article or Dongarra's article were advocating for > the building clusters from iPads, I think they only meant to show how > far computing hardware has come in terms of power and cost. I remember reading a similar article comparing TI-83s and Eniac a long time ago. It was excellent, interesting, pointed, and hit on the latter point you mention (progress of computer science). However, that article didn't suggest that we should take TI-83s and make a cluster out of them, or even worse, that doing so is appealing for power reasons. Whether or not the article on the Reg argues to use them as a cluster is up in the air, but the the NYT article absolutely suggests that there could be some benefits to it: snip "Meanwhile, the fact that the iPad runs off a battery and is air-cooled has given Dr. Dongarra some ideas, like building a supercomputer composed of a couple of stacks of the tablets. ... ?It could be done and provide a very power-friendly cluster,? he said. snip > To say that people should only build clusters out of the proper hardware > is like telling artists to throw out their paints and brushes since we > now have cameras, that do a better job than they can. Again, weird analogy IMHO. I don't pay an artist to have a picture of something so I know what it looks like or can remember it later. If this was the 1200s and we didn't have photographs, then yes, that would be exactly what would happen. While back then many got portraits who could afford them, and higher prices were paid for higher fidelity pictures relative to reality, as soon as cameras hit the scene the demand lessened and artists indeed put down their brushes for that purpose. Modern art is (mostly) interpretive, and tries as hard as possible to avoid doing work that cameras can do. > BTW - I'd never waste my time building a cluster out of iPads, but I > defend the right of others to do so if that's what turns them on. I'm right there with you Prentice -- I'd even enjoy reading an article about just such a cluster if one was built, for the same absurdity factor that the furbie cluster was supported. Weird things are fun. But I don't defend the right of intelligent people to mislead the public (via the Reg and NYT) into actually believing there is something scientifically appealing about doing supercomputing with iPads. Sorry Apple, even you guys can't make hardcore computer science chic. It just doesn't work that way. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 8 17:09:31 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 8 Mar 2012 17:09:31 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: > Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. well, playstations had cell chips, which were, in their day, pretty respectable. to me, that's pure beowulf, in the classic sense. clustering very light-weight machines is sort of amusing, but it's hard to imagine a less appropriate substrate than an iPad. completely mundane hardware, extremely expensive display and touch panel, slow, small memory, no ethernet. a cluster of pogoplug mobiles would be pretty amusing instead, though pandaboards and the like have better cpus. the latter always seem overpriced, since you can get whole netbook for less. OK, I did think of a way that would be pretty cool in a demo/kluge, almost performance-art way: have the iPads communicate by modulating their LCD, and receiving via the screen-side camera. I didn't say it was practical ;) it would be hillarious to see a ring of ipads flashing QR codes at each other while running linpack. well, to me at least! anyone want to donate a pile of iPads? actually, a pile of android tabs would be even better, since you can put a decent/open dev env on them. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 8 17:13:58 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 8 Mar 2012 17:13:58 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F590C3C.5060705@ias.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> <4F590C3C.5060705@ias.edu> Message-ID: > the building clusters from iPads, I think they only meant to show how > far computing hardware has come in terms of power and cost. yes, but ipads suck as an example of this! it _is_ very interesting how far gflop-level performance has spread, but it's not really new, and any smartphone is, if anything, a better example. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 8 19:33:18 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 16:33:18 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Mark Hahn Sent: Thursday, March 08, 2012 2:10 PM To: beowulf at beowulf.org Subject: Re: [Beowulf] Supercomputers - iPad versus Cray OK, I did think of a way that would be pretty cool in a demo/kluge, almost performance-art way: have the iPads communicate by modulating their LCD, and receiving via the screen-side camera. I didn't say it was practical ;) it would be hillarious to see a ring of ipads flashing QR codes at each other while running linpack. well, to me at least! >>> This is a cool idea.. and not so far fetched. I've been involved in optical interconnects which weren't that far off.. multipixel area sensors (i.e. cameras) looking at multiple other computers (all as nodes on the surface of a sphere, looking inward) each with a matrixed light emitting display. With the right optics, you could light up just the leds that shine on the desired recipients (or all of them for broadcast). The practical application is optical comm among a bunch of satellites flying in a not very well controlled formation. So, for the demo, the ring o'pads isn't a bad idea, especially if you put a wide angle lens on the front facing camera. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From rtomek at ceti.pl Fri Mar 9 14:26:16 2012 From: rtomek at ceti.pl (Tomasz Rola) Date: Fri, 9 Mar 2012 20:26:16 +0100 (CET) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: On Thu, 8 Mar 2012, Mark Hahn wrote: > > Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. > > well, playstations had cell chips, which were, in their day, pretty > respectable. to me, that's pure beowulf, in the classic sense. > > clustering very light-weight machines is sort of amusing, but it's hard > to imagine a less appropriate substrate than an iPad. completely mundane > hardware, extremely expensive display and touch panel, slow, small memory, > no ethernet. I remember, some (ten?) years ago on this list, there was an article about making a cluster of Compaq Ipaqs PDAs. Those particular models only had serial port and IR, so the author(s) went infrared. The performance of them was dog ugly, AFAIR, compared to what I could do with one iPaq with Linux and Python program doing similar stuff (prime sieve). I remember iPaqs were at that time touted as would be student's digital assistants, were to be introduced into US schools etc etc. Strange, seems like history repeats itself. Am I a digital prophet? Is Apple going to be bought by HP? :-) > OK, I did think of a way that would be pretty cool in a demo/kluge, > almost performance-art way: have the iPads communicate > by modulating their LCD, and receiving via the screen-side camera. > I didn't say it was practical ;) > > it would be hillarious to see a ring of ipads flashing QR codes at each other > while running linpack. well, to me at least! > > anyone want to donate a pile of iPads? actually, a pile of android tabs > would be even better, since you can put a decent/open dev env on them. Extremely cool idea. How about stacking them all against a huge mirror? Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola at bigfoot.com ** _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Fri Mar 9 17:19:47 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Fri, 9 Mar 2012 17:19:47 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: > Extremely cool idea. How about stacking them all against a huge mirror? a grid of ipads, all admiring themselves in a mirror would be very apt ;) maybe between *two* mirrors, since the _underside_ of apple products are so greatly admired by apple people... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Fri Mar 9 19:37:10 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Fri, 9 Mar 2012 16:37:10 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: Message-ID: Two mirrors face to face, so each device can "see" both front and back. The back facing camera is also higher resolution. Or, if you arranged them in a circle, so that A looks at B which looks at C which looks at D.... which looks at A. Mirrors could provide a redundant optical path. On 3/9/12 2:19 PM, "Mark Hahn" wrote: >> Extremely cool idea. How about stacking them all against a huge mirror? > >a grid of ipads, all admiring themselves in a mirror would be very apt ;) >maybe between *two* mirrors, since the _underside_ of apple products >are so greatly admired by apple people... >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From rtomek at ceti.pl Sat Mar 10 10:30:50 2012 From: rtomek at ceti.pl (Tomasz Rola) Date: Sat, 10 Mar 2012 16:30:50 +0100 (CET) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: Message-ID: On Fri, 9 Mar 2012, Lux, Jim (337C) wrote: > Two mirrors face to face, so each device can "see" both front and back. > The back facing camera is also higher resolution. Or, if you arranged > them in a circle, so that A looks at B which looks at C which looks at > D.... which looks at A. > > Mirrors could provide a redundant optical path. And we finally come to nice algorithmic problem: given a number of mirrors, their sizes and orientation compute maximum number of iphones that could be arranged into an optical cluster. Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola at bigfoot.com ** _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Wed Mar 14 11:22:24 2012 From: eugen at leitl.org (Eugen Leitl) Date: Wed, 14 Mar 2012 16:22:24 +0100 Subject: [Beowulf] oil immersion cooled blades Message-ID: <20120314152224.GU9891@leitl.org> http://www.heise.de/newsticker/meldung/Server-Blades-in-Oel-1471734.html (translation courtesy Google Translate): Server blades in oil Hardcore Computer LSS 200 Photo: Boston The U.S. company has with the hardcore computer Submerged Liquid server developed (LSS 200), a server blade in the format of immersion cooling uses: The entire unit sits in a closed housing which oil flows. This "Core Coolant" is according to the safety of a non-toxic and biodegradable compound based on a synthetic wax. The advantages of this cooling method is called Hardcore Computer eliminated including a more efficient cooling, because the coolant can be for example of a data center transported directly to heat exchangers, and going through cold air. Because no special rack delivers warm air directly into the environment, it can also be operated in locations without air conditioning. The server manufacturer in Boston, the LSS 200 is added to its product line and offers it in Germany. However, neither prices nor called Boston delivery, and plugged in the server rack now slightly dusty Technology: In comparison to the recently announced Xeon E5600 Xeon E5 falls off significantly. Also, the data sheet (PDF file) of the CLS 200 with hardcore computer can open questions. For example, detailed information is lacking on the power supplies, the special chassis and to the oil-cooling equipment. The CLS 200 will also allow the use of a PCIe expansion card, such as a Tesla accelerator or InfiniBand adapter card - if it works well both at the same time remains uncertain. Finally, the question remains open on the disk, at least be mentioned only SSDs in 2.5-inch format. The entire board is surrounded by oil. Picture: Hardcore Computer In the United States sold the hardcore computer desktops and Reactor Reactor X, and the detonator Workstation with Immersion cooling. Oil as a coolant has a lower specific heat capacity than water, but cools better than air and leaks caused by short circuits. When complete immersion of the coolant reaches all the components, while in the water cooling individual heat sinks are needed which do not reach all critical components of any assemblies. Optimal cooling water can be exploited if the motherboard design optimized to be. ( ciw ) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Wed Mar 14 18:02:57 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed, 14 Mar 2012 18:02:57 -0400 (EDT) Subject: [Beowulf] oil immersion cooled blades In-Reply-To: <20120314152224.GU9891@leitl.org> References: <20120314152224.GU9891@leitl.org> Message-ID: > Server blades in oil sounds like some brutalist take on molecular gastronomy! > Hardcore Computer LSS 200 http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf seems kind of uninspiring, in hardware specs. 1366 socket, not 2011, optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really a density play. I guess I'm a bit skeptical about the utility of this approach - would be nice if they had some technical literature. something about thermal resistance. define how the oil bath dumps the heat (water hookups in the back?) comparison to modern heatpipe-based solutions, etc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Wed Mar 14 23:12:47 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Wed, 14 Mar 2012 20:12:47 -0700 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: Message-ID: Immersing motherboards in oil has been done for years (probably even mentioned on this list). There's countless hacker versions using various and sundry off the shelf mobos and power supplies etc in various and sundry containers (ice chests, aquariums, purpose built acrylic boxes) using a variety of coolants. The vast majority of these have no "thermal engineering".. They're more "we tried it and it seemed to work". There's not a lot of content at the hardcore computer website. It could be ok or it might not. They're not real clear on how they actually get the heat out of the enclosures. The MSDS for their coolant says that it's basically synthetic mineral oil with a antioxidant (you can look up the CAS number to get more info). I'd be a bit nervous about the fire hazard aspects of a machine room full of the stuff. So here's my experience using oil as an insulator/coolant. 1) it wicks up insulated wire, particularly stranded. Put the mobo in oil and the power supply outside, and pretty soon your power supply will be full of oil. 2) oil leaks. There is *nothing* that is oil insulated that doesn't have a fine film of oil on its surface eventually, unless it is in a hermetically sealed can with welded/crimped seals. 3) oil is a mess when you need to fix something The Cray-2 used Fluorinert(tm) FC-74 as the coolant, which is very nice to work with, although expensive. It doesn't wet things very well, so when you pull something out of the bath, it doesn't bring much fluid with it. The Cray used it as a heat transfer medium to water coolant. I think they had a way to drain it into a tank quickly for servicing. It can be used for ebullient (boiling) cooling by picking the right vapor pressure/BP grade (the Cray didn't use this). Ebullient cooling is quite efficient at moving the heat away because it's a phase change, and the bubbling causes good circulation, but it does require careful design so you don't get film boiling/Leidenfrost effect (the phenomenon that protects your feet when walking across burning coals barefoot) On 3/14/12 3:02 PM, "Mark Hahn" wrote: >> Server blades in oil > >sounds like some brutalist take on molecular gastronomy! > >> Hardcore Computer LSS 200 > >http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf > >seems kind of uninspiring, in hardware specs. 1366 socket, not 2011, >optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really >a density play. > >I guess I'm a bit skeptical about the utility of this approach - >would be nice if they had some technical literature. something about >thermal resistance. define how the oil bath dumps the heat (water >hookups in the back?) comparison to modern heatpipe-based solutions, etc. >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Thu Mar 15 06:51:43 2012 From: eugen at leitl.org (Eugen Leitl) Date: Thu, 15 Mar 2012 11:51:43 +0100 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: Message-ID: <20120315105143.GY9891@leitl.org> On Wed, Mar 14, 2012 at 08:12:47PM -0700, Lux, Jim (337C) wrote: > The Cray-2 used Fluorinert(tm) FC-74 as the coolant, which is very nice to > work with, although expensive. I've worked with a few 3M fluorinerts for blood substitutes (nothing kills quite like liberated fluoride during sonickating) and for cooling/partial liquid ventilation purposes. Heavy stuff, twice the density of water. Seems to do havoc to your immune system at long-term exposures. > It doesn't wet things very well, so when you pull something out of the > bath, it doesn't bring much fluid with it. The Cray used it as a heat > transfer medium to water coolant. I think they had a way to drain it into > a tank quickly for servicing. > > > It can be used for ebullient (boiling) cooling by picking the right vapor > pressure/BP grade (the Cray didn't use this). Ebullient cooling is quite The Cray X1 used evaporative spray cooling at least http://www.cse.ohio-state.edu/~panda/875/class_slides/cray-jaguar.pdf > efficient at moving the heat away because it's a phase change, and the > bubbling causes good circulation, but it does require careful design so > you don't get film boiling/Leidenfrost effect (the phenomenon that > protects your feet when walking across burning coals barefoot) There's a K/Na eutectic which is cheap (unlike Ga eutectics) and is liquid at RT. Extreme fire hazard if you'll get a leak in air, though. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Thu Mar 15 21:57:27 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Thu, 15 Mar 2012 21:57:27 -0400 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> Message-ID: <4F629E07.2000209@ias.edu> On 3/14/2012 6:02 PM, Mark Hahn wrote: >> Server blades in oil > > sounds like some brutalist take on molecular gastronomy! > >> Hardcore Computer LSS 200 > > http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf > > seems kind of uninspiring, in hardware specs. 1366 socket, not 2011, > optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really > a density play. > > I guess I'm a bit skeptical about the utility of this approach - > would be nice if they had some technical literature. something about > thermal resistance. define how the oil bath dumps the heat (water > hookups in the back?) comparison to modern heatpipe-based solutions, etc. No need for water hookups. You can circulate the oil through an oil to air heat exchanger. That's what those green cooing guys at the SC conferences do. Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 15 23:47:42 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 15 Mar 2012 23:47:42 -0400 (EDT) Subject: [Beowulf] oil immersion cooled blades In-Reply-To: <4F629E07.2000209@ias.edu> References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: >> I guess I'm a bit skeptical about the utility of this approach - >> would be nice if they had some technical literature. something about >> thermal resistance. define how the oil bath dumps the heat (water >> hookups in the back?) comparison to modern heatpipe-based solutions, etc. > > No need for water hookups. You can circulate the oil through an oil to > air heat exchanger. That's what those green cooing guys at the SC is that really better than going to air directly? I guess I'd like to see the the numbers - to my way of thinking, it's almost all about the thermal resistance. transferring heat to oil, then to air, means two stages of resistance. using oil would permit a bigger air interface, though I suppose. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ntmoore at gmail.com Fri Mar 16 00:47:59 2012 From: ntmoore at gmail.com (Nathan Moore) Date: Thu, 15 Mar 2012 23:47:59 -0500 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: If you take a cm^3 of space, right next to the cpu and fill it either with air, or with oil, you'll have many, many, more atomic/molecular degrees of freedom to fill with energy in the cm^3 of oil. Getting that energy out of the cooling medium seems primarily like a fluid-flow problem - given oil's higher heat capacity, you can leave it around something hot, and still have it serve as an effective heat sink for a longer period of time than you can with air. My point is, the fan for the air has to run much faster than the oil pump for the oil coolant. I'm too young for this, but didn't VW and Porche cool some of their engines with oil through the early 1980's? On Thu, Mar 15, 2012 at 10:47 PM, Mark Hahn wrote: > >> I guess I'm a bit skeptical about the utility of this approach - > >> would be nice if they had some technical literature. something about > >> thermal resistance. define how the oil bath dumps the heat (water > >> hookups in the back?) comparison to modern heatpipe-based solutions, > etc. > > > > No need for water hookups. You can circulate the oil through an oil to > > air heat exchanger. That's what those green cooing guys at the SC > > is that really better than going to air directly? > I guess I'd like to see the the numbers - to my way of thinking, > it's almost all about the thermal resistance. transferring heat to oil, > then to air, means two stages of resistance. using oil would permit > a bigger air interface, though I suppose. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- - - - - - - - - - - - - - - - - - - - - - Nathan Moore Associate Professor, Physics Winona State University - - - - - - - - - - - - - - - - - - - - - -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From prentice at ias.edu Fri Mar 16 10:51:49 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 16 Mar 2012 10:51:49 -0400 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: <4F635385.5040606@ias.edu> On 03/15/2012 11:47 PM, Mark Hahn wrote: >>> I guess I'm a bit skeptical about the utility of this approach - >>> would be nice if they had some technical literature. something about >>> thermal resistance. define how the oil bath dumps the heat (water >>> hookups in the back?) comparison to modern heatpipe-based solutions, etc. >> No need for water hookups. You can circulate the oil through an oil to >> air heat exchanger. That's what those green cooing guys at the SC > is that really better than going to air directly? > I guess I'd like to see the the numbers - to my way of thinking, > it's almost all about the thermal resistance. transferring heat to oil, > then to air, means two stages of resistance. using oil would permit > a bigger air interface, though I suppose. Oh boy... I'm having flashbacks of college. The short answer is "yes" that is much better than going to air directly. I'm tying to think of a simple way to explain why, and I'm starting to have flashbacks of my Chem Eng. classes: Thermodynamics, Transport Phenomena... ugh. Here's an attempt at a longer explanation. Sorry if I'm rehashing what you already know. Yes, going from oil, or any liquid, to air is better than using air directly. That is why car engines are liquid-cooled instead of air cooled. It comes down to two physical properties: Thermal conductivity, and thermal capacity. Thermal conductivity is the ability for a material to move energy in the form of heat from one place to another. Thermal capacity is how much energy in the form of heat can be stored in a given quantity (mass or volume) of a substance. In general, liquids and solids have better thermal conductivity and capacity than gases. We use liquid for cooling because the liquid's superior conductivity draws the heat away from the object to be cooled quicker, requiring less surface area, and less "residence time" for the coolant. The superior capacity allows us to use less volume of coolant to store the same amount of "heat" (energy). And liquids are better than solids because they flow, and the bulk movement allows us to easily transport the heat stored in the coolant away easier than if were trying to move solids. Using liquids allows to do draw heat out of tiny confined spaces, like a 1U server, or a cars engine block, that we can't do effectively with air. Once we draw that heat out, it needs to go somewhere, which, in most cases, is atmospheric are, so we pump the liquids to a cooling tower outside, the radiator in the front of our engine bay, or somewhere where is a large amount of air and room for all the surface area the heat exchanger will need (since air has a low thermal conductivity, it needs a lot more surface area to transfer the heat into it than a liquid does). This last point you hit on in the last sentence of your post. That is why car radiators are large and have lots of think aluminium fins in them (more fins = more surface area for the air to contact), and we have massive cooling towers or heat exchangers outside our office buildings. You are correct in that there are inefficiencies when you transfer energy from one material to another, but those are usually trivial and offset by the greater effectiveness of liquids in pulling the heat away from the heat source in the first place. I hope that helps. Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 16 11:08:16 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 16 Mar 2012 11:08:16 -0400 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: <4F635760.3020305@ias.edu> On 03/16/2012 12:47 AM, Nathan Moore wrote: > If you take a cm^3 of space, right next to the cpu and fill it either > with air, or with oil, you'll have many, many, more atomic/molecular > degrees of freedom to fill with energy in the cm^3 of oil. Getting > that energy out of the cooling medium seems primarily like a > fluid-flow problem - given oil's higher heat capacity, you can leave > it around something hot, and still have it serve as an effective heat > sink for a longer period of time than you can with air. My point is, > the fan for the air has to run much faster than the oil pump for the > oil coolant. > > I'm too young for this, but didn't VW and Porche cool some of their > engines with oil through the early 1980's? Yes, I and was going to mention that in the e-mail. I just posted. The original VW "boxer" engines were air cooled. If you look at the cylinders, they have heat fins on them like a CPU heat sink or a motorcycle engine to promote heat transfer to the air. For VW fanatics (of which I used to be one) there two types of VWs: air-cooled, and everything else. I was in the "everything else" camping, owning several rabbits and GTis. The engines in the early Porsches and the 911s trace their lineage all the way back to the first VW beetle engines, and were also air-cooled boxer engines (boxer = horizontally-opposed cylinders). Porsche 911s retained air-cooled engines well into the 1990s, and went to water-cooled engines because it was getting too hard to meet tougher and tougher noise and emissions requirements with air-cooled engines. As a bonus, I think liquid-cooling really helped increase how much horse power they could put out, too. But you lose out on the very distinctive sound that the air-cooled 911 had. :( Do you remember the 911 turbos with that big "whale tale" rear wing" That wing wasn't really there for aerodynamics (although it did help for that, too) It was there to house the massive air-to-air intercoolers needed to cool the turbocharged air. Since they went to liquid cooled engines, you really don't see that any more. Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From john.hearns at mclaren.com Wed Mar 21 10:11:51 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Wed, 21 Mar 2012 14:11:51 -0000 Subject: [Beowulf] Google greywater cooling Message-ID: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Flagging up yet another Register article I'm afraid, but it is interesting http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_coo ling/ The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ellis at cse.psu.edu Wed Mar 21 20:45:46 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Wed, 21 Mar 2012 20:45:46 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: <4F6A763A.3080404@cse.psu.edu> On 03/21/2012 10:11 AM, Hearns, John wrote: > Flagging up yet another Register article I?m afraid, but it is interesting > > http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ Certainly interesting. I missed this one in my El Reg RSS. Does anyone have references on a (modern) picture of what their servers look like? In other words, they are particularly vague in the article and the associated video on how they actually use this water to do the cooling. I assume it is not "real" water cooling, but utilizing that water in their air conditioning units somehow. I cannot imagine they use the water to re-humidify their server rooms, since rooms smelling of piss and chlorine probably isn't appreciated by any but the most green of environmentalists. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From jmdavis1 at vcu.edu Wed Mar 21 22:17:41 2012 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed, 21 Mar 2012 22:17:41 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6A763A.3080404@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> Message-ID: <4F6A8BC5.4060207@vcu.edu> Greywater isn't sewage, though it is often dealt with in the same way. Greywater could be from sinks, plant watering systems, condensation, etc. It could also be recaptured and reused fresh water from some cooling systems. On 3/21/2012 8:45 PM, Ellis H. Wilson III wrote: > On 03/21/2012 10:11 AM, Hearns, John wrote: >> Flagging up yet another Register article I?m afraid, but it is interesting >> >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > Certainly interesting. I missed this one in my El Reg RSS. Does anyone > have references on a (modern) picture of what their servers look like? > In other words, they are particularly vague in the article and the > associated video on how they actually use this water to do the cooling. > I assume it is not "real" water cooling, but utilizing that water in > their air conditioning units somehow. I cannot imagine they use the > water to re-humidify their server rooms, since rooms smelling of piss > and chlorine probably isn't appreciated by any but the most green of > environmentalists. > > Best, > > ellis > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From bob at drzyzgula.org Wed Mar 21 22:52:09 2012 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Wed, 21 Mar 2012 22:52:09 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6A8BC5.4060207@vcu.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: And it certainly is not the case that greywater would be used as a potable -- or breathable -- source of water or humidity. To obtain greywater in the quantities useful for data center cooling, it typically has to come from the local public water company, and generally requires a local ordinance allowing its use. Moreover, such ordinances will specify exactly what the water can be used for, and plumbing codes will ensure that the greywater will be kept separate from the potable water, much as are waste and storm water. --Bob On Wed, Mar 21, 2012 at 10:17 PM, Mike Davis wrote: > Greywater isn't sewage, though it is often dealt with in the same way. > Greywater could be from sinks, plant watering systems, condensation, > etc. It could also be recaptured and reused fresh water from some > cooling systems. > > > On 3/21/2012 8:45 PM, Ellis H. Wilson III wrote: >> On 03/21/2012 10:11 AM, Hearns, John wrote: >>> Flagging up yet another Register article I?m afraid, but it is interesting >>> >>> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ >> Certainly interesting. ?I missed this one in my El Reg RSS. ?Does anyone >> have references on a (modern) picture of what their servers look like? >> In other words, they are particularly vague in the article and the >> associated video on how they actually use this water to do the cooling. >> ? ?I assume it is not "real" water cooling, but utilizing that water in >> their air conditioning units somehow. ?I cannot imagine they use the >> water to re-humidify their server rooms, since rooms smelling of piss >> and chlorine probably isn't appreciated by any but the most green of >> environmentalists. >> >> Best, >> >> ellis >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 22 11:04:19 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 22 Mar 2012 11:04:19 -0400 (EDT) Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: > http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ I grew up in a city cooled by a central evaporative water-chilling plant, so I'm always surprised how unusual this sort of project is. does anyone have comments on how to enable/encourage/foster less conventional datacenter engineering? I've participated in two DC projects, and both went with safe, boring and inefficient solutions, simply because power costs weren't high enough to motivate any deviation from utter conventionality. power costs are somewhat hidden on uni campuses, and people involved are very, extremely, hyper-risk-averse... I suppose part of the problem is in retrofitting a green DC into existing buildings - it's a lot easier if you can plan a new building to, for instance, take advantage of waste heat. I'm also interested in anyone who is currently running clusters at higher than conventional/ASHRAE temperatures. thanks, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Thu Mar 22 11:19:41 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Thu, 22 Mar 2012 16:19:41 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: <4F6B430D.3070503@gmail.com> Those are some interesting thoughts, I live on the island of Malta, where electrical prices are expensive. The Dc i used to work at charged 58 euros per kwh of electricity used if you colocate. I have a few ideas im working on proposing to vodafone malta, and if they deem it feasible it would be something I coudl move forward with for them. Here in Malta we have alot of wind and solar, I dunno why to be honest electrical companies dont burn organic wastes to help generate electricity. I am actually thinking of approaching the government with the idea of using organic wastes to generate electricity locally, and start to cut our dependence on oil for electrical generation. On 22/03/2012 16:04, Mark Hahn wrote: >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > I grew up in a city cooled by a central evaporative water-chilling plant, > so I'm always surprised how unusual this sort of project is. > > does anyone have comments on how to enable/encourage/foster less > conventional datacenter engineering? > > I've participated in two DC projects, and both went with safe, boring > and inefficient solutions, simply because power costs weren't high enough > to motivate any deviation from utter conventionality. power costs are > somewhat hidden on uni campuses, and people involved are very, extremely, > hyper-risk-averse... > > I suppose part of the problem is in retrofitting a green DC into > existing buildings - it's a lot easier if you can plan a new building > to, for instance, take advantage of waste heat. > > I'm also interested in anyone who is currently running clusters at > higher than conventional/ASHRAE temperatures. > > thanks, mark hahn. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From brian.ropers.huilman at gmail.com Thu Mar 22 12:03:38 2012 From: brian.ropers.huilman at gmail.com (Brian D. Ropers-Huilman) Date: Thu, 22 Mar 2012 11:03:38 -0500 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6B430D.3070503@gmail.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: > Those are some interesting thoughts, I live on the island of Malta, > where electrical prices are expensive. The Dc i used to work at charged > 58 euros per kwh of electricity used if you colocate. I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay ~$0.11 per kw/hr residential here in the States (after various fees are thrown in, too). I see a "reality check" error in your ~700x cost ... -- Brian D. Ropers-Huilman 612.234.7778 (m) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Thu Mar 22 12:16:41 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Thu, 22 Mar 2012 17:16:41 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: <4F6B5069.2060800@gmail.com> When i was looking to colocate a server for my business there thats what i was told. The problem with where I am is that we rely on imported oil which is used to generate electricity. So basically it depends on the price of oil which causes electrical prices to fluctuate greatly here. I have an idea which i need to draft up a business proposal for vodafone malta to see if they can see if its feasible to go completely green, and if so that would be a project that I would head up and move forward with. On 22/03/2012 17:03, Brian D. Ropers-Huilman wrote: > On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: >> Those are some interesting thoughts, I live on the island of Malta, >> where electrical prices are expensive. The Dc i used to work at charged >> 58 euros per kwh of electricity used if you colocate. > I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's > conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay > ~$0.11 per kw/hr residential here in the States (after various fees > are thrown in, too). I see a "reality check" error in your ~700x cost > ... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Thu Mar 22 12:17:44 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Thu, 22 Mar 2012 17:17:44 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: <4F6B50A8.8010005@gmail.com> On a residential electricity note. between 3 people in this house we pay about 800 euros which is about 266 euros a person which equates to about 88 euros a month for electricity. On 22/03/2012 17:03, Brian D. Ropers-Huilman wrote: > On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: >> Those are some interesting thoughts, I live on the island of Malta, >> where electrical prices are expensive. The Dc i used to work at charged >> 58 euros per kwh of electricity used if you colocate. > I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's > conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay > ~$0.11 per kw/hr residential here in the States (after various fees > are thrown in, too). I see a "reality check" error in your ~700x cost > ... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 22 20:14:26 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 22 Mar 2012 17:14:26 -0700 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6B430D.3070503@gmail.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: 58 Euro? Or 0.58 Euro? (and I think $0.34/kWhr in California is high...) -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina Sent: Thursday, March 22, 2012 8:20 AM To: beowulf at beowulf.org Subject: Re: [Beowulf] Google greywater cooling Those are some interesting thoughts, I live on the island of Malta, where electrical prices are expensive. The Dc i used to work at charged 58 euros per kwh of electricity used if you colocate. I have a few ideas im working on proposing to vodafone malta, and if they deem it feasible it would be something I coudl move forward with for them. Here in Malta we have alot of wind and solar, I dunno why to be honest electrical companies dont burn organic wastes to help generate electricity. I am actually thinking of approaching the government with the idea of using organic wastes to generate electricity locally, and start to cut our dependence on oil for electrical generation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 22 20:23:22 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 22 Mar 2012 17:23:22 -0700 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6B50A8.8010005@gmail.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> <4F6B50A8.8010005@gmail.com> Message-ID: Much of the electricity in the US is actually generated by burning oil, as well.. As it happens, enemalta has a web presence.. There's a fixed service charge of ?65 for single phase residential, ?195 for three phase.. .. that's where your big cost is.. The actual electricity is ?0.161 to ?0.620 per kWh in 5 tiers. (it doesn't actually say kWh, but I assume that's what the units are..) So that's high on a per kWh basis (compared to, say, California) but not exceedingly high.. (maybe 2x-3x.) For businesses: The flat fee is ?360 for 3phase, and rates go from 0.162 up to 0.215 (at 20-60 MWh) and then down to 0.144 at 5GWh. This is a pretty typical sort of rate schedule.. gradually rising, then falling. (and of course, they stick you for reactive power, too, just like in the U.S.) -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina Sent: Thursday, March 22, 2012 9:18 AM To: Brian D. Ropers-Huilman Cc: beowulf at beowulf.org Subject: Re: [Beowulf] Google greywater cooling On a residential electricity note. between 3 people in this house we pay about 800 euros which is about 266 euros a person which equates to about 88 euros a month for electricity. On 22/03/2012 17:03, Brian D. Ropers-Huilman wrote: > On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: >> Those are some interesting thoughts, I live on the island of Malta, >> where electrical prices are expensive. The Dc i used to work at >> charged >> 58 euros per kwh of electricity used if you colocate. > I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's > conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay > ~$0.11 per kw/hr residential here in the States (after various fees > are thrown in, too). I see a "reality check" error in your ~700x cost > ... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Fri Mar 23 03:22:09 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Fri, 23 Mar 2012 08:22:09 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: <4F6C24A1.5040802@gmail.com> 58 euros I can dig through my emails and see if i find the quote for colocating a server. On 23/03/2012 01:14, Lux, Jim (337C) wrote: > 58 Euro? Or 0.58 Euro? (and I think $0.34/kWhr in California is high...) > > -----Original Message----- > From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina > Sent: Thursday, March 22, 2012 8:20 AM > To: beowulf at beowulf.org > Subject: Re: [Beowulf] Google greywater cooling > > Those are some interesting thoughts, I live on the island of Malta, where electrical prices are expensive. The Dc i used to work at charged > 58 euros per kwh of electricity used if you colocate. > > I have a few ideas im working on proposing to vodafone malta, and if they deem it feasible it would be something I coudl move forward with for them. > > Here in Malta we have alot of wind and solar, I dunno why to be honest electrical companies dont burn organic wastes to help generate electricity. > > I am actually thinking of approaching the government with the idea of using organic wastes to generate electricity locally, and start to cut our dependence on oil for electrical generation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 23 09:47:18 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 23 Mar 2012 09:47:18 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6A763A.3080404@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> Message-ID: <4F6C7EE6.4060802@ias.edu> On 03/21/2012 08:45 PM, Ellis H. Wilson III wrote: > On 03/21/2012 10:11 AM, Hearns, John wrote: >> Flagging up yet another Register article I?m afraid, but it is interesting >> >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > Certainly interesting. I missed this one in my El Reg RSS. Does anyone > have references on a (modern) picture of what their servers look like? > In other words, they are particularly vague in the article and the > associated video on how they actually use this water to do the cooling. > I assume it is not "real" water cooling, but utilizing that water in > their air conditioning units somehow. I cannot imagine they use the > water to re-humidify their server rooms, since rooms smelling of piss > and chlorine probably isn't appreciated by any but the most green of > environmentalists. > That water is first sent through a Google-built sidestream treatment plant, which sterilizes, filters, and clorinates it, cleaning it just enough for it to be used in the data center's evaporative cooling towers. The water that doesn't evaporate away during that process is then further purified in Google's on-premises Effluent Treatment Plant. "There, we treat the water once again to disinfect it," Brown writes, "remove mineral solids and send it back out to the Chattahoochee ? clean, clear and safe." Based on the above description, this is the most likely scenario: The greywater flows down the outside of heat exchangers in the cooling tower. As it evaporates, the change of state removes a lot of the heat from the liquid inside the heat exchanger (probably also water, but clean, treated water). The chilled water inside the heat exchanger then flows into heat exchangers inside the data center CRACs where it absorbs heat from the warm data center air., cooling the air. Any time you use water in a closed-loop system (chilled water cooling or steam heating, for example). It needs to be very clean, and have additional chemicals added to it prevent corrosion or mineral buildup inside the pipes, so greywater would probably never be used inside one of these systems. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 23 09:59:37 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 23 Mar 2012 09:59:37 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: <4F6C81C9.4010604@ias.edu> On 03/22/2012 11:04 AM, Mark Hahn wrote: >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > I grew up in a city cooled by a central evaporative water-chilling plant, > so I'm always surprised how unusual this sort of project is. > > does anyone have comments on how to enable/encourage/foster less > conventional datacenter engineering? > > I've participated in two DC projects, and both went with safe, boring > and inefficient solutions, simply because power costs weren't high enough > to motivate any deviation from utter conventionality. power costs are > somewhat hidden on uni campuses, and people involved are very, extremely, > hyper-risk-averse... > > I suppose part of the problem is in retrofitting a green DC into > existing buildings - it's a lot easier if you can plan a new building > to, for instance, take advantage of waste heat. > I think you need to get the bean-counters involved, and sell them on the money savings, not just the planet savings. Princeton University just completed contruction of their High-Performance Computing Research Center (HPCRC. The name is a complete misnomer - its a production datacenter that houses ALL university computing, not just HPC, and it's not a research center). It's a pretty amazing building, and is shooting for some level of LEED certification. To me, the place looks like an over-engineered monster (I've toured it several times), but the engineers giving the tours always point out many different ways the datacenter is both saving the planet and saving the university a lot of money at the same time. They've done an amazing job with it, and I think that the end of the day, the monetary savings is what made it happen. http://www.princeton.edu/facilities/info/major_projects/HPCRC/ I know there are some lurkers from PU on this list who are involved with the HPCRC. Maybe they'll pipe up on this topic. -- Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From diep at xs4all.nl Sun Mar 25 14:47:00 2012 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sun, 25 Mar 2012 20:47:00 +0200 Subject: [Beowulf] Nvidia's quantum leap in 28 nm Message-ID: <630CE216-EC95-4A42-B7B3-070549C803B3@xs4all.nl> It's been some year or 12 that a genius visited me. His expertise being the same like Einsteins, it's not much of a question what his research topics were. Though not deep into computer hardware he told me that for massive computing, just above the 1Ghz border would prove to be a big barrier as electrons basically move at around 1/3 of the lightspeed, which translates to 1.3Ghz in metals like aluminium. At copper so he said that barrier might be a tad higher than aluminium, yet even then the power needed for such speeds would prove to be massive. At that moment intel's marketing department shouted out loud their P4's would clock 10Ghz by 2010. Well the P4 never got there and we got into the megacore count game for HPC. AMD Now AMD needs 4 PE's for doing double precision, so their core count of 1536 actually wasn't more than the 5000 series with 1600. Their new 7970 gpu with 2048 pe's has the double precision equivalent in core count of 512 compute cores. Actually the 7970 mostly profits from a 100Mhz higher frequency with some boosting to 1Ghz at some overclocked cards, it gets impressive game scores. As for gpgpu of course, moving from 1536 cores to 2048 is an interesting improvement, yet far away from a doubling. The 7970 is said to have around 4.31B transistors (see http://www.anandtech.com/show/5261/amd-radeon-hd-7970-review ) NVIDIA FERMI Fermi, nvidia's 40 nm gpu which currently gets used in HPC, it has 3 bilion transistors. Here at home i have a few 2075 Tesla's with 448 cores producing a tad more than 0.5 Tflop which was its a big improvement over the previous generation. The Nvidia Fermi on the other hand in the form of the GTX 560 clocks 1.644Ghz and the 580 clocks 1.544Ghz. For gpgpu this is on the risky side as getting far over that 1Ghz seems to be a problem. The tesla's therefore are clocked safely 1.15Ghz NVIDIA KEPLER 2012 The new kid on the block from Nvidia is the Kepler. It's in the 28 nm proces technology, just like AMD's 7970. Now i'm not gonna redo a review for games, there is great sites for that. http://www.anandtech.com/show/5699/nvidia-geforce-gtx-680-review/1 Over here we are interested in the implications for the beowulf systems of course, i read that as HPC implications. Let's look to facts and then speculate what that means for HPC: I'm still trying to full understand the differences, yet it seems as if nvidia clocked back to 1Ghz the cores. That should make it easier to release a gpu for gpgpu as well. In the meantime core count went up to 1536. The chip itself has 3.5 billion transistors. Just 500M more than Fermi, meanwhile at a factor 2.04 smaller proces, that means it will consume less juice and a lot less juice. Benchmarks at anandtech confirm this. Now that's a MASSIVE quantumleap. Basically factor 3 the number of cores available to HPC. Additional to that the memory is 256 bits wide, versus 384 bits for Fermi. This should make it easier to release 2 gpu's on a single card. Whether nvidia has those plans for gpgpu tesla's we can only speculate about, as the chip eats less juice, it sure fits this time within the power envelope. So where the gamer kids with sureness can expect a 690 gpu, for HPC we of course cheer if nvidia manages to improve to 1.5 - 1.7 Tflop for their new gpu, with the option to move to 3 - 3.4 Tflop double precision for a 2 gpu Tesla card. Note that some might argue that the 680 has less double precision capabilities than the 580. However for the Tesla this doesn't matter, as what happens for gamerscards is that they disable some transistors; so the Tesla gpu will be the exact same chip like the kids has, just with the double precision enabled. The same thing was the case with Fermi, so it's logical to expect that to happen with Kepler as well. Seems like intel can also scrap their current corner project as they have a new goal, namely 4 Tflop, rather than a 1 Tflop manycore :) As for Nvidia, releasing a new chip that's factor 3 the power of your previous one for gpgpu sure is a big quantum leap! _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From samuel at unimelb.edu.au Sun Mar 25 19:53:10 2012 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Mon, 26 Mar 2012 10:53:10 +1100 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6C7EE6.4060802@ias.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6C7EE6.4060802@ias.edu> Message-ID: <4F6FAFE6.8020607@unimelb.edu.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24/03/12 00:47, Prentice Bisbal wrote: > Any time you use water in a closed-loop system (chilled water > cooling or steam heating, for example). It needs to be very clean, > and have additional chemicals added to it prevent corrosion or > mineral buildup inside the pipes, so greywater would probably never > be used inside one of these systems. In our experience they also add anti-fungals like azoles to them too. cheers, Chris - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9vr+YACgkQO2KABBYQAh/zzQCfS5p5Z4jEu0VPukZPgo8auD0Q B6MAnArzv9TgtPUctu1X0nb5ktscjQgn =Xjyl -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Tue Mar 27 11:15:51 2012 From: eugen at leitl.org (Eugen Leitl) Date: Tue, 27 Mar 2012 17:15:51 +0200 Subject: [Beowulf] Everything You Know (about Parallel Programming) Is Wrong!: A Wild Screed about the Future Message-ID: <20120327151551.GI17245@leitl.org> http://splashcon.org/2011/program/dls/245-invited-talk-2 Mon 2:00-3:00 pm - Pavilion East Everything You Know (about Parallel Programming) Is Wrong!: A Wild Screed about the Future invited speakerDavid Ungar, IBM Research, USA In the 1970?s, researchers at Xerox PARC gave themselves a glimpse of the future by building computers that, although wildly impractical at the time, let them experience plentiful fast cycles and big memories. PARC researchers invented Smalltalk, and the freedom afforded by such a dynamic, yet safe, language, led them to create a new experience of computing, which has become quite mainstream today. In the end of the first decade of the new century, chips such as Tilera?s can give us a glimpse of a future in which manycore microprocessors will become commonplace: every (non-hand-held) computer?s CPU chip will contain 1,000 fairly homogeneous cores. Such a system will not be programmed like the cloud, or even a cluster because communication will be much faster relative to computation. Nor will it be programmed like today?s multicore processors because the illusion of instant memory coherency will have been dispelled by both the physical limitations imposed by the 1,000-way fan-in to the memory system, and the comparatively long physical lengths of the inter- vs. intra-core connections. In the 1980?s we changed our model of computation from static to dynamic, and when this future arrives we will have to change our model of computation yet again. If we cannot skirt Amdahl?s Law, the last 900 cores will do us no good whatsoever. What does this mean? We cannot afford even tiny amounts of serialization. Locks?! Even lock-free algorithms will not be parallel enough. They rely on instructions that require communication and synchronization between cores? caches. Just as we learned to embrace languages without static type checking, and with the ability to shoot ourselves in the foot, we will need to embrace a style of programming without any synchronization whatsoever. In our Renaissance project at IBM, Brussels, and Portland State, (http://soft.vub.ac.be/~smarr/renaissance/) we are investigating what we call ?anti-lock,? ?race-and-repair,? or ?end-to-end nondeterministic? computing. As part of this effort, we have build a Smalltalk system that runs on the 64-core Tilera chip, and have experimented with dynamic languages atop this system. When we give up synchronization, we of necessity give up determinism. There seems to be a fundamental tradeoff between determinism and performance, just as there once seemed to be a tradeoff between static checking and performance. The obstacle we shall have to overcome, if we are to successfully program manycore systems, is our cherished assumption that we write programs that always get the exactly right answers. This assumption is deeply embedded in how we think about programming. The folks who build web search engines already understand, but for the rest of us, to quote Firesign Theatre: Everything You Know Is Wrong! David Ungar is an out-of-the-box thinker who enjoys the challenge of building computer software systems that work like magic and fit a user's mind like a glove. He received the 2009 Dahl-Nygaard award for outstanding career contributions in the field of object-orientation, and was honored as an ACM Fellow in 2010. Three of his papers have been honored by the Association for Computing Machinery for lasting impact over ten to twenty-four years: for the design of the prototype-based Self language, dynamic optimization techniques, and the application of cartoon animation ideas to user interfaces. He enjoys a position at IBM Research, where he is taking on a new challenge: investigating how application programmers can exploit manycore systems, and testing those ideas to see if they can help scale up analytics. [NOTE] this session is organized as a joint event with the AGERE! workshop _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From deadline at eadline.org Thu Mar 29 07:58:21 2012 From: deadline at eadline.org (Douglas Eadline) Date: Thu, 29 Mar 2012 07:58:21 -0400 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <20120327151551.GI17245@leitl.org> References: <20120327151551.GI17245@leitl.org> Message-ID: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> I am glad some one is talking about this. I have wondered about this myself, but never had a chance to look into it. http://www.isgtw.org/feature/does-computation-threaten-scientific-method -- Doug -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From diep at xs4all.nl Thu Mar 29 08:39:31 2012 From: diep at xs4all.nl (Vincent Diepeveen) Date: Thu, 29 Mar 2012 14:39:31 +0200 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: <711D3DE6-3858-45BA-B65B-7F8D04FA7AA8@xs4all.nl> Why don't you ask Robert G Brown to write something about it? It seems like a problem at universities, not so much a problem in companies, which can't or do not want to publish their codes anyway. On Mar 29, 2012, at 1:58 PM, Douglas Eadline wrote: > > > I am glad some one is talking about this. I have wondered > about this myself, but never had a chance to look into it. > > > http://www.isgtw.org/feature/does-computation-threaten-scientific- > method > > -- > Doug > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 29 10:24:38 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 29 Mar 2012 07:24:38 -0700 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: It is an interesting problem, and one relevant to HPC, just because HPC tends to be used in applications where errors can have big effects (although one could argue that a bad 1 page spreadsheet in the hands of a Fortune 10 CEO might result in a bad decision that has huge impacts..) With respect to "the scientific method", is it really different than say, a very complex or esoteric math proof. The SM relies on (attempted) replication for validation and in some ways, it's more a funding thing (that is, if your results depend on 100 work years of software development, replicating requires investing 100 work years). "big science" has always had that problem: Do you build a copy of the LHC? And, given the small number of people actually doing it, would a replication really be an independent replication. Odds are, some of the same developers would wind up on the new project, because of the limited pool of people who really know, say, numerical weather modeling. (again, this is partly funding.. If there were public investments of trillions of dollars in numerical weather modeling, there would be a veritable army of numerical weather modelers emerging from the halls of academe from which to choose for your replication effort. We don't, and there aren't.) And one has to be very careful about measures of defect density (usually given as defects/KLOC). Not only are there the "what's a LOC" questions, but different defects have different consequences, and the reporting of defects varies as a result. The LOC question is particularly an issue with the use of autogenerated code, although I tend to think this is no different than a compiler. Do you count source lines or machine instructions? If you make the (big) leap that the compiler is validated, you're working on the assumption that there's some underlying defect rate per "human operation". At JPL, we have a defect density of .1 to 4 defects/KSLOC spanning development test, system test, and operations. Mostly, we're in the less than 0.5 defects/KSLOC in operational use. Comparing with DoD data, it's in the same ballpark (DoD reports 0.2 to 0.6). But this is for "flight software" (that which runs on a computer in the spacecraft) which has a much more rigorous development process, over a shorter time span, than a lot of what the article was talking about. The data is somewhat skewed by the observation that until very recently, flight code is small because of the limited processor and memory space. Our productivity is gradually increasing, but not by leaps and bounds. We generated about 100 lines of code per work-month in the early 80s and about 50% more right now, and that's at the low end of the 1-5 line of code/work hour you see bandied about. I would attribute the low rate to the large amount of oversight and rigor. The shuttle software at $500M/500KSLOC is an interesting thing. Let's say that was mostly done in the early 80s, when a high end work hour cost about $50 (= 50K/yr salary+ overheads). $500M is then 10M work hours, so they achieved the spectacularly low productivity of 0.05 SLOC/hour. The shuttle software is widely known to have been VERY low productivity. What's also interesting is that it's not clear that they achieved a commensurate reduction of defect rate (that is, is the defect rate 20 times lower?). Nancy Leveson, among others, has some papers talking about this. I suspect that most of the software being described in the article doesn't have anywhere near the process rigor, nor does it have the inspectability, of the flight software at JPL. In general, I think it's probably tested by comparing results with expectation or with actual data, and the code is adjusted to make the model output match the observed data. Whether the code adjustments are actually "model parameters" or "unfounded logic changes" is sort of the question. And, of course, the "software" is really a conglomeration of libraries, other people's programs, etc. There are pieces of the puzzle that are no doubt rigorously verified: For instance not only is the source code for the Numerical Electromagnetics Code, NEC, published, but there's hundreds of pages of theory of operation documentation, as well as lots of published validation studies comparing against theoretical calculation and empirical measurements. I suspect that something like LAPACK is pretty well validated as well. But I would imagine that there is little *published* information on the validation of the overall assemblage. I think a lot of people just assume the library or popular tool "just works" and probably don't look at analyzing "well, how would we know if NEC or LAPACK was wrong". I'll speculate, too, that while a given research effort and the scientists attached to it may have a long duration (and, so, retain some corporate knowledge of the software architecture, design, and validation), there is more turnover on the software developers, and knowledge retention/transfer probably isn't all that good. An interesting observation from JPL is that the vast majority of the 1000+ people developing software don't have any training in software development processes/engineering/etc, beyond practical experience (which is a form of training). This is, in part, because most software development is quite diffuse. Over half the software development projects are smaller than 2 work years, and software development is just part of the overall bigger task. That is, the scientist or engineer is doing software development as a tool to do their job, not as a job in itself. I suspect a similar phenomenon is true in academe. How much "I'll just whip up this Matlab module to analyze the data" winds up being the basis for "production code". On 3/29/12 4:58 AM, "Douglas Eadline" wrote: > > >I am glad some one is talking about this. I have wondered >about this myself, but never had a chance to look into it. > > >http://www.isgtw.org/feature/does-computation-threaten-scientific-method > >-- >Doug > >-- >This message has been scanned for viruses and >dangerous content by MailScanner, and is >believed to be clean. > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From brian.dobbins at yale.edu Thu Mar 29 12:22:49 2012 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Thu, 29 Mar 2012 12:22:49 -0400 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: <4F748C59.4000700@yale.edu> To borrow from an old joke, I'd say the short answer is "No.", and the long answer? "Nooooooooooo." Reproducibility is an interesting issue - on the surface, it seems like a binary thing: something is or is not reproducible. In reality, though, things are almost never duplicated exactly, and there exists some fuzzy threshold at which point things are considered good enough to be a reproduction. I can go down to a local store and buy a print of the Mona Lisa and, to me, it might be a really great reproduction, yet even writing that sentence has some art critic screaming in agony. Similarly, in computing, if I run some model on two different systems and get two different results, that can either be indicative of a potential issue or it can be completely fine, because those differences are below a certain threshold and thus the runs were, in scientific terms, 'reproducible' with respect to each other. On a small scale (meaning a lab, code or project), this is a key issue - I've seen grad students and faculty alike be dismayed by trivial differences, and when this happens, more often than not the mentality is, "My first results are correct - make this code give them back to me", without understanding that the later, different results are quite possibly equally valid, and possibly more so. Back in the early Beowulf days, I remember switching some codes from an RS/6000 platform to an x86-based one, and the internal precision of the x86 FPU was 80-bits, not 64, so sequences of FP math could produce small differences unless this option was specifically disabled via compiler switches. Which a lot of people did, not because the situation was carefully considered, but because with it on, it gave 'wrong' results. Another example would be an algorithm that was orders of magnitude faster than one previously in use, but wasn't adopted because ultimately the results were different. The catch here? Reordering the input data while still using the original algorithm gave similarly different answers - the nature of the code was that single runs were useless, and ensemble runs were a necessity. Ultimately, the issues here come down to the common perception of computers - "They give you THE answer!" - versus the reality of computers - "They give you AN answer!", with the latter requiring additional effort to provide some error margin or statistical analysis of results. This happens in certain computational disciplines far more often than others. On the larger scales - whether reproducibility is an issue in scientific /fields/ - again, I'd say the answer is no. The scientific method is resilient, but it never made any claims to be 'fast'. Would it speed things up to have researchers publish their code and data? Probably. Or, rather, it'd certainly speed up the verification of results, but it might also inhibit new approaches to doing the same thing. Some people here might recall Michael Abrash's "Graphics Programming Black Book", which had a wonderful passage where about a word-counting program. It focused explicitly on performance tuning, with the key lesson being that nobody thought there was a better way of doing the task... until someone showed there was. And that lead to a flurry of new ideas. Similarly, having software that does things in a certain way often convinces people that that is THE way of doing things, whereas if they knew it could be done but not how, newer methods might develop. There's probably some happy medium here, since having so many different codes, mostly with a single author who isn't a software developer by training, seems less efficient and flexible than a large code with good documentation, a good community and the ability to use many of those methods previously in the one-off codes. In other words, we can probably do better, but science itself isn't threatened by the inefficiency in verifying results, or even bad results - in the absolute worst case, with incorrect ideas being laid down as the foundation for new science and no checking done on them, progress will happen until it can't... at which point people will backtrack until the discover the underlying principle they thought was correct and will fix it. The scientific method is a bit like a game of chutes and ladders in this respect. Ultimately, in a lot of ways, I think computational science has it better than other disciplines. There was news earlier this week [1] about problems reproducing some early-stage cancer research - specifically, Amgen tried to reproduce 53 'landmark' conclusions, and were only able to do so with 11% of them. Again, that's OK - it will correct itself, albeit in slow fashion, but what's interesting here is that these sorts of experiments, especially those involving mice (and often other wet-lab methods), don't have something like Moore's Law making them more accessible over time. To reproduce a study involving the immune system of a mouse, I need mice. And I need to wait the proper number of days. Yet with computational science, what today may take a top end supercomputer can probably be done in a few years on a departmental cluster. A few years after that? Maybe a workstation. In our field, data doesn't really change or degrade over time and the ability to analyze it in countless different ways becomes more and more accessible all the time. In short (hah, nothing about this was short!), can we do better with our scientific approaches? Probably. But is the scientific method threatened by computation? Nooooooooo. :-) That's my two cents, - Brian [1] http://vitals.msnbc.msn.com/_news/2012/03/28/10905933-rethinking-how-we-confront-cancer-bad-science-and-risk-reduction Or, more directly (if you have access to Nature) : http://www.nature.com/nature/journal/v483/n7391/full/483531a.html (PS. The one thing which can threaten science is a lack of education - it can decrease the signal-to-noise ratio of 'good' science, amongst other things. That's a whole essay in itself.) (PPS. This was a long answer, and yet not nearly long enough... but I didn't want to be de-invited from future Beowulf Bashes by writing even more!) On 3/29/2012 7:58 AM, Douglas Eadline wrote: > > I am glad some one is talking about this. I have wondered > about this myself, but never had a chance to look into it. > > > http://www.isgtw.org/feature/does-computation-threaten-scientific-method > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From samuel at unimelb.edu.au Thu Mar 29 20:27:56 2012 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Fri, 30 Mar 2012 11:27:56 +1100 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: <4F74FE0C.9050607@unimelb.edu.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 29/03/12 22:58, Douglas Eadline wrote: > I am glad some one is talking about this. I have wondered about > this myself, but never had a chance to look into it. IIRC this issue was discussed at CCGrid 2005 in Cardiff, but provenance and reproducibility are hard - we need to keep upgrading to get security fixes but we get other changes into the bargain. You could work around that by having jobs run in VM images that are set in stone, but then you will take a performance penalty (from a small amount to large amount, depending on the level of virtualisation you need). Then of course there's the issue of hardware errata, not to mention availability if you ran it on an Alphaserver SC in 2002.. cheers. Chris - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk90/gwACgkQO2KABBYQAh8o3QCfd3Q23fD09gYYanNm4ehxIVgM YI0An0JSC9iNfC5FMvk2FepSaAv9Xnrx =gG3x -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From diep at xs4all.nl Thu Mar 29 20:45:44 2012 From: diep at xs4all.nl (Vincent Diepeveen) Date: Fri, 30 Mar 2012 02:45:44 +0200 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <4F748C59.4000700@yale.edu> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> <4F748C59.4000700@yale.edu> Message-ID: Brian - using medical examples probably is not a good idea in this discussion. To introduce anew medicine is something of a 3 phase introduction to get it from acceptance to the market; getting allowance at a political level is the hardest. Now a big problem is simply that you can already get a 'go' there when you have a 95% confidence - or around 200 persons which react positive at whatever you were doing at the SHORT TERM. That's fairly little actually, especially for the trillion turnover that nowadays psychological medicines alltogether make. So to speak that's science from 1946. The computation risk there is a different one - namely that in short term cocain always works - that's what kids get now basically, that is, the new medicines for examle for ADHD look frightening much like cocain and in long term also have those same side effects. Yet they get prescribed massively, also for kids that do not need it. The real problem is not so much the computation, but more the government rules of allowing something based upon 200 cases short term. If you sell already a product for billions, even later corrections onto what you do, will not soon make it into mainstream, so it will get used for years and years until someone says STOP. That STOP is very tough to give if they produced synthetical cocain and in short term got through the 200 cases that reacted positive. This is more of a government problem of course - they are still 65+ years behind. So to mix lapack statements with pharmaceuticals is not a good idea i'd say. If you suck everywhere as a scientist, you can still go produce a new medicine and sell it worldwide. Government should really undertake action there. Also directly modify that DSM classification once again so that less people get diagnosed, as then it's not always the doctors that want to put medicines into someone, but for teachers it's very great to do so and one of the reports, though i don't know whether that's accurate, seems so though, said 33% more children now are diagnosed thanks to a few changes in the classification system! As usual of course that was taken over from USA, as each individual small nation in Europe is too small to do things like that on itself, wit hsometimes devastating consequences. Vincent On Mar 29, 2012, at 6:22 PM, Brian Dobbins wrote: > > To borrow from an old joke, I'd say the short answer is "No.", and > the long answer? "Nooooooooooo." > > Reproducibility is an interesting issue - on the surface, it seems > like a binary thing: something is or is not reproducible. In > reality, though, things are almost never duplicated exactly, and > there exists some fuzzy threshold at which point things are > considered good enough to be a reproduction. I can go down to a > local store and buy a print of the Mona Lisa and, to me, it might > be a really great reproduction, yet even writing that sentence has > some art critic screaming in agony. Similarly, in computing, if I > run some model on two different systems and get two different > results, that can either be indicative of a potential issue or it > can be completely fine, because those differences are below a > certain threshold and thus the runs were, in scientific terms, > 'reproducible' with respect to each other. > > On a small scale (meaning a lab, code or project), this is a key > issue - I've seen grad students and faculty alike be dismayed by > trivial differences, and when this happens, more often than not the > mentality is, "My first results are correct - make this code give > them back to me", without understanding that the later, different > results are quite possibly equally valid, and possibly more so. > Back in the early Beowulf days, I remember switching some codes > from an RS/6000 platform to an x86-based one, and the internal > precision of the x86 FPU was 80-bits, not 64, so sequences of FP > math could produce small differences unless this option was > specifically disabled via compiler switches. Which a lot of people > did, not because the situation was carefully considered, but > because with it on, it gave 'wrong' results. Another example > would be an algorithm that was orders of magnitude faster than one > previously in use, but wasn't adopted because ultimately the > results were different. The catch here? Reordering the input data > while still using the original algorithm gave similarly different > answers - the nature of the code was that single runs were useless, > and ensemble runs were a necessity. > > Ultimately, the issues here come down to the common perception of > computers - "They give you THE answer!" - versus the reality of > computers - "They give you AN answer!", with the latter requiring > additional effort to provide some error margin or statistical > analysis of results. This happens in certain computational > disciplines far more often than others. > > On the larger scales - whether reproducibility is an issue in > scientific fields - again, I'd say the answer is no. The > scientific method is resilient, but it never made any claims to be > 'fast'. Would it speed things up to have researchers publish their > code and data? Probably. Or, rather, it'd certainly speed up the > verification of results, but it might also inhibit new approaches > to doing the same thing. Some people here might recall Michael > Abrash's "Graphics Programming Black Book", which had a wonderful > passage where about a word-counting program. It focused explicitly > on performance tuning, with the key lesson being that nobody > thought there was a better way of doing the task... until someone > showed there was. And that lead to a flurry of new ideas. > Similarly, having software that does things in a certain way often > convinces people that that is THE way of doing things, whereas if > they knew it could be done but not how, newer methods might > develop. There's probably some happy medium here, since having so > many different codes, mostly with a single author who isn't a > software developer by training, seems less efficient and flexible > than a large code with good documentation, a good community and the > ability to use many of those methods previously in the one-off codes. > > In other words, we can probably do better, but science itself isn't > threatened by the inefficiency in verifying results, or even bad > results - in the absolute worst case, with incorrect ideas being > laid down as the foundation for new science and no checking done on > them, progress will happen until it can't... at which point people > will backtrack until the discover the underlying principle they > thought was correct and will fix it. The scientific method is a > bit like a game of chutes and ladders in this respect. > > Ultimately, in a lot of ways, I think computational science has it > better than other disciplines. There was news earlier this week > [1] about problems reproducing some early-stage cancer research - > specifically, Amgen tried to reproduce 53 'landmark' conclusions, > and were only able to do so with 11% of them. Again, that's OK - > it will correct itself, albeit in slow fashion, but what's > interesting here is that these sorts of experiments, especially > those involving mice (and often other wet-lab methods), don't have > something like Moore's Law making them more accessible over time. > To reproduce a study involving the immune system of a mouse, I need > mice. And I need to wait the proper number of days. Yet with > computational science, what today may take a top end supercomputer > can probably be done in a few years on a departmental cluster. A > few years after that? Maybe a workstation. In our field, data > doesn't really change or degrade over time and the ability to > analyze it in countless different ways becomes more and more > accessible all the time. > > In short (hah, nothing about this was short!), can we do better > with our scientific approaches? Probably. But is the scientific > method threatened by computation? Nooooooooo. :-) > > That's my two cents, > - Brian > > [1] http://vitals.msnbc.msn.com/_news/2012/03/28/10905933- > rethinking-how-we-confront-cancer-bad-science-and-risk-reduction > Or, more directly (if you have access to Nature) : http:// > www.nature.com/nature/journal/v483/n7391/full/483531a.html > > (PS. The one thing which can threaten science is a lack of > education - it can decrease the signal-to-noise ratio of 'good' > science, amongst other things. That's a whole essay in itself.) > (PPS. This was a long answer, and yet not nearly long enough... > but I didn't want to be de-invited from future Beowulf Bashes by > writing even more!) > > > On 3/29/2012 7:58 AM, Douglas Eadline wrote: >> >> I am glad some one is talking about this. I have wondered about >> this myself, but never had a chance to look into it. http:// >> www.isgtw.org/feature/does-computation-threaten-scientific-method > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 1 01:16:49 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 1 Mar 2012 01:16:49 -0500 (EST) Subject: [Beowulf] amd buys seamicro Message-ID: http://www.eetimes.com/electronics-news/4237271/AMD-to-buy-microserver-startup-SeaMicro this is interesting. most of the coverage seems to interpret this as using opterons (which makes some sense, given the direction bulldozer is going, towards lots of space/power-effective cores.) but here's another prospect: a box with lots of APU chips that max out GPU density. lotsa gflops/watt, very compact... seamicro says their interconnect is special, low-lat, high-bw, but it sounds like an onboard 10Gb chip to me. calxeda's onboard distributed switch might be more interesting. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From tegner at renget.se Thu Mar 1 01:52:36 2012 From: tegner at renget.se (Jon Tegner) Date: Thu, 01 Mar 2012 07:52:36 +0100 Subject: [Beowulf] Functionality of schedulers Message-ID: <4F4F1CB4.4010303@renget.se> Hi list! Is there any scheduler which has the functionality to automatically put a running job on hold when another job with higher priority is submitted? Preferably the state of the first job should be frozen, and saved to disk, so that it can be restarted again when the higher priority job has finished. Is this at all possible (we are using torque/maui, and I couldn't find this feature there)? Regards, /jon _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From j.wender at science-computing.de Thu Mar 1 02:20:28 2012 From: j.wender at science-computing.de (Jan Wender) Date: Thu, 01 Mar 2012 08:20:28 +0100 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: <4F4F233C.3000507@science-computing.de> Hi Jon, Jon Tegner schrieb: > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? AFAIK at least LSF has this as a feature called preemption. > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. That depends probably mostly on the application. If the application offers it, then the batch system can use it to save state. I don't know much about kernel level checkpointing, though. Cheerio, Jan -- ---- Company Information ---- Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Roland Niemeier, Dr. Arno Steitz, Dr. Ingrid Zech Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: Philippe Miltin Sitz/Registered Office: Tuebingen Registergericht/Registration Court: Stuttgart Registernummer/Commercial Register No.: HRB 382196 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- A non-text attachment was scrubbed... Name: j_wender.vcf Type: text/x-vcard Size: 367 bytes Desc: not available URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From raysonlogin at gmail.com Thu Mar 1 02:22:12 2012 From: raysonlogin at gmail.com (Rayson Ho) Date: Thu, 1 Mar 2012 02:22:12 -0500 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: Jon, Almost every cluster job scheduler supports preemption (use it as the google keyword and you will find lots of references...). Torque has job preemption. I have not used Torque for a while (I used to be an OpenPBS user) so I am not the best person to answer the question for Torque. However, if you google "job preemption"+torque, you should be able to find some useful info. In Grid Engine (and now Open Grid Scheduler) there is Subordinate Queues: http://docs.oracle.com/cd/E19080-01/n1.grid.eng6/817-5677/i998889/index.html Condor is well-known for cycle stealing, and it also offers the checkpoint restart library for Open Grid Scheduler/Grid Engine and other batch systems: http://research.cs.wisc.edu/condor/checkpointing.html http://gridscheduler.sourceforge.net/howto/checkpointing.html (I am sure that you can integrate checkpointing with Torque so you don't need to look for a new batch system to get what you need.) Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ On Thu, Mar 1, 2012 at 1:52 AM, Jon Tegner wrote: > Hi list! > > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? > > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. > > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? > > Regards, > > /jon > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From raysonlogin at gmail.com Thu Mar 1 03:22:16 2012 From: raysonlogin at gmail.com (Rayson Ho) Date: Thu, 1 Mar 2012 03:22:16 -0500 Subject: [Beowulf] LSF Job Preemption & Checkpointing Message-ID: On Thu, Mar 1, 2012 at 2:20 AM, Jan Wender wrote: > AFAIK at least LSF has this as a feature called preemption. IMO, LSF has the best job preemption & checkpointing support, with the least integration effort needed from the end user & cluster administrator. And resource preemption and license preemption are the more advanced features of LSF. (There are more manual configuration needed for Grid Engine & Open Grid Scheduler and/or other batch systems - not impossible, but needs knowledge on how to tune the scheduler.) > That depends probably mostly on the application. If the application offers > it, then the batch system can use it to save state. > I don't know much about kernel level checkpointing, though. There are 3 types of checkpointing supported by LSF: 1) kernel-level 2) user-level 3) application-level Kernel level is easy, the OS kernel handles everything for the user (for interactively processes) & the batch system (for jobs). However, only IRIX, Cray UNICOS, and NEC SUPER-UX support kernel-level checkpointing. On Linux, you usually need to patch the kernel: - "Checkpoint/restart: it's complicated": http://lwn.net/Articles/414264/ - "Kernel-based checkpoint and restart": http://lwn.net/Articles/293575/ (Lots of discussions on kernel-level checkpointing in the past few years but still we don't have anything in the official tree yet...) Or even kernel assisted user-level checkpointing: - "Preparing for user-space checkpoint/restore": http://lwn.net/Articles/478111/ And there is also the famous Berkeley Lab Checkpoint/Restart (BLCR), which is a kernel module and thus you can use your distribution's stock kernel: - "RCE 12: BLCR": http://www.rce-cast.com/Podcast/rce-12-blcr.html - "Checkpointing under Linux with Berkeley Lab Checkpoint/Restart": http://gridscheduler.sourceforge.net/howto/APSTC-TB-2004-005.pdf For user-level, you will need to link against a checkpointing library shipped with LSF, which (I think) has some object file level init routines that perform initializations to properly save the state of stuff and also need to wrap around standard libc functions & system calls (I forgot the actual details, lots of academic papers published 15 years ago and I recall reading a few of them, but just don't recall the content :-D ). See "Standalone Checkpointing": http://research.cs.wisc.edu/condor/checkpointing.html With user-level checkpointing & restart, you usually need to relink your application (unless you use the LD_PRELOAD trick). So for operating systems that don't support kernel-level checkpointing (ie. most of the OSes), user-level checkpointing usually works for most general applications (I *think* Platform Computing even ported the LSF checkpointing library to Windows as well - or at least that's what I was told). For application-level checkpointing, the applications will handle everything. But of course each application needs to have its own built-in support for checkpoint & restart. Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ > > Cheerio, Jan > -- > ---- Company Information ---- > Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, > Dr. Roland Niemeier, Dr. Arno Steitz, Dr. Ingrid Zech > Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: > Philippe Miltin > Sitz/Registered Office: Tuebingen Registergericht/Registration Court: > Stuttgart > Registernummer/Commercial Register No.: HRB 382196 > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From john.hearns at mclaren.com Thu Mar 1 06:37:58 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Thu, 1 Mar 2012 11:37:58 -0000 Subject: [Beowulf] Pbsnodes xml format Message-ID: <207BB2F60743C34496BE41039233A8090BC87925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Slightly off topic. Would some kind soul who is running Torque send me some sample output from pbsnodes -x Thanks The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From samuel at unimelb.edu.au Thu Mar 1 06:46:47 2012 From: samuel at unimelb.edu.au (Chris Samuel) Date: Thu, 1 Mar 2012 22:46:47 +1100 Subject: [Beowulf] Functionality of schedulers In-Reply-To: References: <4F4F1CB4.4010303@renget.se> Message-ID: <201203012246.47618.samuel@unimelb.edu.au> On Thursday 01 March 2012 18:22:12 Rayson Ho wrote: > Torque has job preemption. I have not used Torque for a while (I > used to be an OpenPBS user) so I am not the best person to answer > the question for Torque. However, if you google "job > preemption"+torque, you should be able to find some useful info. It also integrates with BLCR (Berkeley Lab Checkpoint/Restart) to provide kernel level checkpointing support for Linux, including for MPI applications if you are using Open-MPI. Useful links: BLCR: https://ftg.lbl.gov/projects/CheckpointRestart/ Open-MPI and BLCR: http://www.open-mpi.org/faq/?category=ft Torque and checkpoint/restore and BLCR: http://www.clusterresources.com/torquedocs/2.6jobcheckpoint.shtml Hope this helps! Chris -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ntmoore at gmail.com Thu Mar 1 08:57:52 2012 From: ntmoore at gmail.com (Nathan Moore) Date: Thu, 1 Mar 2012 07:57:52 -0600 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: I think the command you're looking for is, > sudo nice -n -20 ./my_important_program On Thu, Mar 1, 2012 at 12:52 AM, Jon Tegner wrote: > Hi list! > > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? > > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. > > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? > > Regards, > > /jon > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- - - - - - - -?? - - - - - - -?? - - - - - - - Nathan Moore Associate Professor, Physics Winona State University - - - - - - -?? - - - - - - -?? - - - - - - - _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 1 10:44:06 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 1 Mar 2012 10:44:06 -0500 (EST) Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. well, maybe. that process (checkpoint/restore) really makes sense only if the preemtor is giant and/or long. otherwise SIGSTOP is a much better solution (it implies that you should have swap, but you should have swap anyway.) > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? this code (even moab) has all sorts of problems keeping track of suspension. the weak spot is usually that when you suspend a parallel job and the preemptor doesn't use all the cpus, you can't go starting random other jobs on these pseudo-free cpus. LSF wasn't all that great about this little detail either, at least back in 6.x versions. it's kind of amazing how poor all the schedulers are, really. classic example of how projects get sclerotic by adding features... regards, mark hahn _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 2 10:39:57 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 02 Mar 2012 10:39:57 -0500 Subject: [Beowulf] Functionality of schedulers In-Reply-To: <4F4F1CB4.4010303@renget.se> References: <4F4F1CB4.4010303@renget.se> Message-ID: <4F50E9CD.1050704@ias.edu> On 03/01/2012 01:52 AM, Jon Tegner wrote: > Hi list! > > Is there any scheduler which has the functionality to automatically put > a running job on hold when another job with higher priority is submitted? > > Preferably the state of the first job should be frozen, and saved to > disk, so that it can be restarted again when the higher priority job has > finished. > > Is this at all possible (we are using torque/maui, and I couldn't find > this feature there)? > SGE can do this more or less, but it doesn't write the the job state to disk, unless you count swapping. In SGE you can create multiple queues, where one is subordinate (lower priority) to another. When a job is running in the subordinate queue and then a job submitted in a superior queue, the job is the subordinate queue will be paused while the higher priority job runs. The problem with this that the subordinate job is only paused it stays in memory, and it doesn't free up any resources it consumed (software licenses, etc), and it won't be migrated to other hosts If you want a job to be written to disk so it can be completely, or be able to be migrated to other hosts, you might want to look at checkpoint restart options like BLCR, if that's applicable to your situation. -- Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From trainor at presciencetrust.org Sat Mar 3 11:04:19 2012 From: trainor at presciencetrust.org (Douglas J. Trainor) Date: Sat, 3 Mar 2012 11:04:19 -0500 Subject: [Beowulf] Ninja Blocks (Re: Raspberry Pi) In-Reply-To: <4F4EAB27.2080201@unimelb.edu.au> References: <4F4EAB27.2080201@unimelb.edu.au> Message-ID: <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> Ninja Blocks are being funded by a Kickstarter campaign -- http://www.kickstarter.com/projects/ninja/ninja-blocks-connect-your-world-with-the-web This little computer group is emphasizing the software and sensors, and they're build to order. douglas On Feb 29, 2012, at 5:48 PM, Christopher Samuel wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 01/03/12 08:40, Douglas J. Trainor wrote: > >> thought some people here should see the Raspberry Pi -- > > Been following this for a while, they're rather neat little devices.. > > http://www.raspberrypi.org/#modelb > > http://en.wikipedia.org/wiki/Raspberry_Pi#Hardware > > Their usual site is down at the moment as it couldn't cope with demand > (but then again, neither could Farnell or RS, their retailers :-) ). > > cheers, > Chris > - -- > Christopher Samuel - Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.unimelb.edu.au/ > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Sun Mar 4 14:35:59 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Sun, 4 Mar 2012 14:35:59 -0500 (EST) Subject: [Beowulf] seamicro fabric? Message-ID: Has anyone found an informative description of the Seamicro fabric? various heavily masticated reports on the web say it's a 3d torus, "low latency" and 160 GB/s (per link? bisecection? in-chassis?) thanks, mark. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Mon Mar 5 09:35:06 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Mon, 05 Mar 2012 09:35:06 -0500 Subject: [Beowulf] Ninja Blocks (Re: Raspberry Pi) In-Reply-To: <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> References: <4F4EAB27.2080201@unimelb.edu.au> <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> Message-ID: <4F54CF1A.1060709@ias.edu> Ninja Blocks looks like Arduinos with sexier packaging. Prentice On 03/03/2012 11:04 AM, Douglas J. Trainor wrote: > Ninja Blocks are being funded by a Kickstarter campaign -- > > http://www.kickstarter.com/projects/ninja/ninja-blocks-connect-your-world-with-the-web > > This little computer group is emphasizing the software and sensors, and they're build to order. > > douglas > > On Feb 29, 2012, at 5:48 PM, Christopher Samuel wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 01/03/12 08:40, Douglas J. Trainor wrote: >> >>> thought some people here should see the Raspberry Pi -- >> Been following this for a while, they're rather neat little devices.. >> >> http://www.raspberrypi.org/#modelb >> >> http://en.wikipedia.org/wiki/Raspberry_Pi#Hardware >> >> Their usual site is down at the moment as it couldn't cope with demand >> (but then again, neither could Farnell or RS, their retailers :-) ). >> >> cheers, >> Chris >> - -- >> Christopher Samuel - Senior Systems Administrator >> VLSCI - Victorian Life Sciences Computation Initiative >> Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 >> http://www.vlsci.unimelb.edu.au/ >> > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Mon Mar 5 15:07:10 2012 From: eugen at leitl.org (Eugen Leitl) Date: Mon, 5 Mar 2012 21:07:10 +0100 Subject: [Beowulf] [FoRK] seamicro fabric? Message-ID: <20120305200710.GW9891@leitl.org> ----- Forwarded message from Aaron Burt ----- From: Aaron Burt Date: Mon, 5 Mar 2012 11:31:37 -0800 To: fork at xent.com Subject: Re: [FoRK] [Beowulf] seamicro fabric? User-Agent: Mutt/1.5.20 (2009-06-14) Reply-To: Friends of Rohit Khare On Mon, Mar 05, 2012 at 05:23:02AM +0100, Eugen Leitl wrote: > From: Mark Hahn > > Has anyone found an informative description of the Seamicro fabric? I hope Mark has repeated his search; http://google.com?q=anil+rao+fabric and http://google.com?q=seamicro+interconnect turn up lots of info. A 2011 PDF[1] at Seamicro sez, "It is a three-dimensional torus, with both path redundancy and diversity. The fabric is FLIT-based and wormhole- routed, with integrated virtual-channel technology to manage congestion, and has a throughput of 1.2 Terabits/sec." Looks like proc boards use PCIe interconnects, like the old cPCI blades. The more interesting bit (to me) is the possibility raised in an older article[2] that Rao and Lauterbach are thinking of turning their interconnect into a datacenter-level fabric. Infiniband II, anyone? Remembering the days of physically separate frontend and I/O processors, Aaron [1] http://www.seamicro.com/sites/default/files/TO1_SM10000_Technology_Overview.pdf [2] http://www.datacenterknowledge.com/archives/2010/01/26/seamicro-more-than-just-low-power-servers/ _______________________________________________ FoRK mailing list http://xent.com/mailman/listinfo/fork ----- End forwarded message ----- -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From raysonlogin at gmail.com Mon Mar 5 15:20:36 2012 From: raysonlogin at gmail.com (Rayson Ho) Date: Mon, 5 Mar 2012 15:20:36 -0500 Subject: [Beowulf] [FoRK] seamicro fabric? In-Reply-To: <20120305200710.GW9891@leitl.org> References: <20120305200710.GW9891@leitl.org> Message-ID: You can find lots of info from this EE380 talk at Stanford: "Architectural tradeoffs in the Sea Micro SM 10000 Server" http://www.stanford.edu/class/ee380/Abstracts/100922.html Video archive: http://ee380.stanford.edu/cgi-bin/videologger.php?target=100922-ee380-300.asx Rayson ================================= Open Grid Scheduler / Grid Engine http://gridscheduler.sourceforge.net/ Scalable Grid Engine Support Program http://www.scalablelogic.com/ On Mon, Mar 5, 2012 at 3:07 PM, Eugen Leitl wrote: > ----- Forwarded message from Aaron Burt ----- > > From: Aaron Burt > Date: Mon, 5 Mar 2012 11:31:37 -0800 > To: fork at xent.com > Subject: Re: [FoRK] [Beowulf] seamicro fabric? > User-Agent: Mutt/1.5.20 (2009-06-14) > Reply-To: Friends of Rohit Khare > > On Mon, Mar 05, 2012 at 05:23:02AM +0100, Eugen Leitl wrote: >> From: Mark Hahn >> >> Has anyone found an informative description of the Seamicro fabric? > > I hope Mark has repeated his search; http://google.com?q=anil+rao+fabric > and http://google.com?q=seamicro+interconnect turn up lots of info. > > A 2011 PDF[1] at Seamicro sez, "It is a three-dimensional torus, with both > path redundancy and diversity. The fabric is FLIT-based and wormhole- > routed, with integrated virtual-channel technology to manage congestion, > and has a throughput of 1.2 Terabits/sec." > > Looks like proc boards use PCIe interconnects, like the old cPCI blades. > > The more interesting bit (to me) is the possibility raised in an older > article[2] that Rao and Lauterbach are thinking of turning their > interconnect into a datacenter-level fabric. ?Infiniband II, anyone? > > Remembering the days of physically separate frontend and I/O processors, > ?Aaron > > [1] http://www.seamicro.com/sites/default/files/TO1_SM10000_Technology_Overview.pdf > [2] http://www.datacenterknowledge.com/archives/2010/01/26/seamicro-more-than-just-low-power-servers/ > _______________________________________________ > FoRK mailing list > http://xent.com/mailman/listinfo/fork > > ----- End forwarded message ----- > -- > Eugen* Leitl leitl http://leitl.org > ______________________________________________________________ > ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org > 8B29F6BE: 099D 78BA 2FD3 B014 B08A ?7779 75B0 2443 8B29 F6BE > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- ================================================== Open Grid Scheduler - The Official Open Source Grid Engine http://gridscheduler.sourceforge.net/ _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From samuel at unimelb.edu.au Tue Mar 6 01:19:17 2012 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Tue, 06 Mar 2012 17:19:17 +1100 Subject: [Beowulf] Ninja Blocks (Re: Raspberry Pi) In-Reply-To: <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> References: <4F4EAB27.2080201@unimelb.edu.au> <15E75ADD-0A85-4C60-9E0C-61DE4A7C1BDB@presciencetrust.org> Message-ID: <4F55AC65.8080405@unimelb.edu.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/03/12 03:04, Douglas J. Trainor wrote: > Ninja Blocks are being funded by a Kickstarter campaign -- I don't think they run Linux though, unlike Raspberry Pi. - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9VrGUACgkQO2KABBYQAh+CVwCfUeRkHYPxHFMuL14zWfMG/AOE 3IkAoI9HlQAXY6hb903f1b3gSFxKO+0V =dHho -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From john.hearns at mclaren.com Thu Mar 8 08:43:10 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Thu, 8 Mar 2012 13:43:10 -0000 Subject: [Beowulf] Supercomputers - iPad versus Cray Message-ID: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ A rather nice Register article on costs for supercomputers, adjusted to 2010 dollars, And a rather interesting cost per megaflop table on the second page. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Mar 8 09:27:15 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 06:27:15 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: Of some interest is Jack Dongarra's measurements on the iPad http://bits.blogs.nytimes.com/2011/05/09/the-ipad-in-your-hand-as-fast-as-a-supercomputer-of-yore/ And his speculation on a cluster. (wouldn't be a Beowulf: not commodity hardware, not open source OS, etc.) While it might do the flops, the interconnect bandwidth is quite low. I assume it would have to use WiFi (I don't know how fast the tether port is, but I don't think it's a real ball o'fire. It's hard to measure. You can plug a USB disk drive into the photo adapter so maybe your interconnect could simulate that. From: "Hearns, John" > Date: Thu, 8 Mar 2012 05:43:10 -0800 To: "beowulf at beowulf.org" > Subject: [Beowulf] Supercomputers - iPad versus Cray http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ A rather nice Register article on costs for supercomputers, adjusted to 2010 dollars, And a rather interesting cost per megaflop table on the second page. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Mar 8 09:37:38 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 06:37:38 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: What would be interesting is to know what is the value not the cost of a FLOP? I tend to look at computational horsepower in the tool sense. If my job is cutting wood, a chainsaw lets me cut a lot more wood than I could with a hand saw. Since there's some value in a unit of wood cutting, one could come up with a chainsaw value in cords cut. Likewise, we tend to evaluate computing solutions in terms of "how much more work can an engineer get done using the computation, than some alternative" (leaving aside the delightful negative productivity from Angry Birds on an iPad, but gosh it's great on a long plane flight) There's stories about nuclear reaction dynamics computations using Electric Accounting Machinery (EAM): tabulators, sorters, etc, which was then replaced by early computers. And the EAM equipment replaced rooms of computers (the living breathing kind with rows of Marchand calculators). SO there's some data there to compare with. But I think that simplifying it to instruction counts (valid for EAM vs room of calculators) doesn't account for "change in way of solving problem" Back in the day, people designed antennas by empiricism and some analytic approximations. But now, there are FEM codes, and the analytic forms are used more to validate that test cases run correctly on the FEM, and for "quick and dirty" estimates. Likewise, structural design trades many simple calculations in a FEM code for longer form analytical approaches, and the ability to do the FEM enables the design of structures that are MUCH more complex. What is the "value" of being able to do a more complex structure? Or design optics that aren't simple spherical and cylinder sections? From: "Hearns, John" > Date: Thu, 8 Mar 2012 05:43:10 -0800 To: "beowulf at beowulf.org" > Subject: [Beowulf] Supercomputers - iPad versus Cray http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ A rather nice Register article on costs for supercomputers, adjusted to 2010 dollars, And a rather interesting cost per megaflop table on the second page. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at mclaren.com Thu Mar 8 10:08:05 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Thu, 8 Mar 2012 15:08:05 -0000 Subject: [Beowulf] Supercomputers - iPad versus Cray References: Message-ID: <207BB2F60743C34496BE41039233A8090C0C3E1A@MRL-PWEXCHMB02.mil.tagmclarengroup.com> What would be interesting is to know what is the value?not the cost?of a FLOP? I tend to look at computational horsepower in the tool sense. ?If my job is cutting wood, a chainsaw lets me cut a lot more wood than I could with a hand saw. ?Since there's some value in a unit of wood cutting, one could come up with a chainsaw value in cords cut. Likewise, we tend to evaluate computing solutions in terms of "how much more work can an engineer get done using the computation, than some alternative" (leaving aside the delightful negative productivity from Angry Birds on an iPad, but gosh it's great on a long plane flight) That is a very good point. There is also, as we know on here, value in 'time to solution' - which is what supercomputers with more mega/terafloppage can offer. Calculations which were possible years ago, but which would take so long that you couldn't be bothered to wait for the result are now done. One example was given by Jon 'Maddog' Hall in the very first talk about Beowulfery which I attended (at Compaq's offices in London). A group in South America had built a Beowulf cluster, which was used for recognition of (I think) mammograms. In the past, women who had to travel for maybe hours or days had to make a return visit to the clinic to get their diagnosis. If the recognition could be run within the time of one clinic then the results are so much more useful. The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 10:57:44 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 10:57:44 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: <4F58D6F8.5050708@cse.psu.edu> On 03/08/2012 08:43 AM, Hearns, John wrote: > http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ > > A rather nice Register article on costs for supercomputers, adjusted to > 2010 dollars, Without being overly pessimistic, I get a vibe of "contrived" throughout the article. For instance, as you mention, the supers are CPI adjusted for present day worth, which is quite interesting to see the trend of increasing cost for supers. However, it just ends there -- our curiosity as to the much more important question, why that trend exists, goes unsatisfied. I have some guesses about why, but it would be far more interesting to see hard evidence (or even squishy evidence, really any evidence) as to why than to simply speculate. For instance, I'm sure there are figures on the older systems at least that break down the overall cost into costs of development versus hardware, or perhaps even more interesting, the respective costs of different pieces of hardware in the system (how much is spent on I/O versus CPU versus Net, etc). Seeing a rapid growth in development costs or that new systems are spending way more on the interconnect than the CPU or vice-versa would shed much needed light on an otherwise unexplained finding. > And a rather interesting cost per megaflop table on the second page. I actually hate this table. If we want to compare flops/cost between machines, we should at least level the playing field: find Linpack numbers on individual machines in the listed clusters and compare them to your desktop or iPad. After all, in the latter cases, we have no interconnect issues to worry about (just imagine Linpack numbers over 3G or even Wifi to other iPads...probably worse for 2 than for 1). I think this table does a disservice to all the complexities that super-designers have to deal with, dumbing down the great hurdles they must cope with to get Linpack to scale across the entire cluster. Just my 2c, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 13:42:57 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 13:42:57 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: <4F58FDB1.4080008@cse.psu.edu> On 03/08/2012 01:30 PM, Mark Hahn wrote: > including how you could possibly > spend $10k on a dual-socket box Yea, I was giving him the benefit of the doubt on that one and just hoping he bought the parts some years back, or has crazy expensive water cooling, but in essence yes the 120 number for flops is embarrassing given how much he paid. Also, his wife having bought that computer for 1700 is equally brow-raising. My desktop directly compares to that desktop machine but was bought for $800 in early 2009... It seems given his inclination to overpay for things and the interest in using iPads for clustering, he has too much money on his hands :). The article with Dongarra is particularly weird however, as that is a guy who /does/ know a lot about computing. Perhaps it was just a sensationalist spin for the NYT since so many now have iPads at home so the average iJoe will connect with the article (read: Oh I have a dated supercomputer in my hands!). Either way, it still surprised me as the article should have just ended before the laughable suggestion that we tether these chic monitors with a brain together. That was a little disturbing coming from him. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 8 13:30:57 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 8 Mar 2012 13:30:57 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F58D6F8.5050708@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: >> http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ >> >> A rather nice Register article on costs for supercomputers, adjusted to >> 2010 dollars, he really should have talked to someone who knows computers first, though. lot of embarassing nonsense in that article (including how you could possibly spend $10k on a dual-socket box. or why you'd choose an ipad (65% profit margin, and whose BOM is mainly display.)) > Without being overly pessimistic, I get a vibe of "contrived" throughout > the article. as opposed to other reg articles? ;) > For instance, as you mention, the supers are CPI adjusted > for present day worth, which is quite interesting to see the trend of > increasing cost for supers. However, it just ends there -- our > curiosity as to the much more important question, why that trend exists, > goes unsatisfied. supers are in a kind of crazy arms race. what I'd really like to see is an article that explores exactly what code runs on the largest (say top100) computers. I'm not saying I disbelieve any code would scale to 700k cores (K computer) just that I don't know of any science that would. I'm certainly first to admit I don't know anything about world-class HPC (Canadian HPC is pretty much a flop, no pun intended) but would science be better off with 100x 7k-core centi-K computers? I suspect so. there are a couple premises that should be questioned: - are scale-limited problems where the interesting science is? I talk to cosmologists a lot - they seem to be many orders of magnitude from being able to resolve their physics. I'm not sure the same applies to MD, q-chem, sequence analysis, etc. I'm also not sure that just because a field is scale-limited, that's where the effort/money should go. - is there some assumption that larger computers provide economies of scale? surely this is untrue: past a certain point, larger computers require more overhead (interconnect is often nonlinear, practical issues require greater attention to cooling, density, and reliability dictates that you simply can't use the same parts at 700kcore as you can at 7kcore.) >> And a rather interesting cost per megaflop table on the second page. > > I actually hate this table. If we want to compare flops/cost between > machines, we should at least level the playing field: find Linpack > numbers on individual machines in the listed clusters and compare them > to your desktop or iPad. using iPad is just stupid, since it's an embarassingly expensive piece of eyecandy, not a computer. (yeah, yeah, you might love yours, but you still wrote a cheque for 65% of what you paid that landed directly on Apple's big pile of cash.) there are dozens of low-overhead chips out there that make a better comparison. > After all, in the latter cases, we have no > interconnect issues to worry about (just imagine Linpack numbers over 3G > or even Wifi to other iPads...probably worse for 2 than for 1). I think it's not worth thinking about, wifi being half-duplex. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 8 14:01:27 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 11:01:27 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Mark Hahn Sent: Thursday, March 08, 2012 10:31 AM To: Beowulf Mailing List Subject: Re: [Beowulf] Supercomputers - iPad versus Cray using iPad is just stupid, since it's an embarassingly expensive piece of eyecandy, not a computer. (yeah, yeah, you might love yours, but you still wrote a cheque for 65% of what you paid that landed directly on Apple's big pile of cash.) Ahh.. but precisely because it is eyecandy means that $/FLOP means you're comparing the wrong metric. iPads are not computers. They're media storage and display devices. So the value (not the cost) needs to be compared to alternatives. It's "is my iPad cheaper than buying and carrying hardback books, watching pay-per-view videos,etc on an airplane or hotel room or airport concourse." I decided to get an eReader/iPad when I was stuck at Phoenix SkyHarbor, having just paid over $20 for some hardcover book (which was a reasonable purchase price), and now lugging the book around. And realizing that I had carried various and sundry hard and softcover books on most trips over the past couple years. And then, contemplating a 2 week long trip to Egypt without internet connectivity, but with teenage children and a couple of 20 hour-ish plane trips (counting layovers, etc), where the ability to get some mindless entertainment that I could just "watch" would be nice. I can load up my iPad with dozens of books (not to mention countless .pdfs of journal articles that I never seem to have time to read) and with a half dozen movies ripped from DVDs I own. *that* is what makes the multi-hundred dollar expense worth it. Whether it cost Steve&Co a penny to manufacture, and Steve pocketed the rest, or whether they took a loss is immaterial. It was worth the cash, to me. As it happens, now that I have the device, I have found other uses (makes a dandy user interface to stuff controlled by a web-server interface... when I get my arduino temperature/humidity controller done, I'm going to use the iPad (or an iPhone/iPodTouch) as the "touchable and viewable" part of the UI. Would it make a good computational element? I don't know. It's reasonably power efficient in a joules/computation basis, and certainly, as an element in a toy/gimmick cluster to demonstrate things it might be nice. (but a better solution might be a bunch of androids or iPodTouch-es... cheaper, smaller, still make a cool tabletop demo, especially if you could figure out a way to do different "tinkerable" interconnects) I can see a "mess o'iTouch" thing to do demonstrations of mesh network routing, for instance. Or perhaps to demonstrate aspects of cooperative processing in a fractionated spacecraft. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 8 14:02:51 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 11:02:51 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F58FDB1.4080008@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Ellis H. Wilson III Sent: Thursday, March 08, 2012 10:43 AM To: beowulf at beowulf.org Subject: Re: [Beowulf] Supercomputers - iPad versus Cray The article with Dongarra is particularly weird however, as that is a guy who /does/ know a lot about computing. Perhaps it was just a sensationalist spin for the NYT since so many now have iPads at home so the average iJoe will connect with the article (read: Oh I have a dated supercomputer in my hands!). Either way, it still surprised me as the article should have just ended before the laughable suggestion that we tether these chic monitors with a brain together. That was a little disturbing coming from him. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From deadline at eadline.org Thu Mar 8 14:19:00 2012 From: deadline at eadline.org (Douglas Eadline) Date: Thu, 8 Mar 2012 14:19:00 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: > http://www.theregister.co.uk/2012/03/08/supercomputing_vs_home_usage/ > > > > A rather nice Register article on costs for supercomputers, adjusted to > 2010 dollars, > > And a rather interesting cost per megaflop table on the second page. The table is a bit silly, but as another data point, my Limulus box can do 200 GFLOPS (running HPL) for less than $5K, that puts it at $.025/MFLOP, or better than anything in the table. BTW, they should be launching real soon. -- Doug > > > The contents of this email are confidential and for the exclusive use of > the intended recipient. If you receive this email in error you should not > copy it, retransmit it, use it or disclose its contents but should return > it to the sender immediately and delete your copy. > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Doug -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From prentice at ias.edu Thu Mar 8 14:45:00 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Thu, 08 Mar 2012 14:45:00 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: <4F590C3C.5060705@ias.edu> Man climbed the mountain, because it was there, and techies will continue to build clusters and other things out of unusual hardware merely because they are there, too. At the risk of being flamed, I think some of you are missing the point. Neither the Register article or Dongarra's article were advocating for the building clusters from iPads, I think they only meant to show how far computing hardware has come in terms of power and cost. To say that people should only build clusters out of the proper hardware is like telling artists to throw out their paints and brushes since we now have cameras, that do a better job than they can. BTW - I'd never waste my time building a cluster out of iPads, but I defend the right of others to do so if that's what turns them on. -- Prentice On 03/08/2012 02:02 PM, Lux, Jim (337C) wrote: > Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. > > -----Original Message----- > From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Ellis H. Wilson III > Sent: Thursday, March 08, 2012 10:43 AM > To: beowulf at beowulf.org > Subject: Re: [Beowulf] Supercomputers - iPad versus Cray > > The article with Dongarra is particularly weird however, as that is a guy who /does/ know a lot about computing. Perhaps it was just a sensationalist spin for the NYT since so many now have iPads at home so the average iJoe will connect with the article (read: Oh I have a dated supercomputer in my hands!). Either way, it still surprised me as the article should have just ended before the laughable suggestion that we tether these chic monitors with a brain together. That was a little disturbing coming from him. > > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 15:24:52 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 15:24:52 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> Message-ID: <4F591594.6090504@cse.psu.edu> On 03/08/2012 02:01 PM, Lux, Jim (337C) wrote: > -----Original Message----- > From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Mark Hahn > Sent: Thursday, March 08, 2012 10:31 AM > To: Beowulf Mailing List > Subject: Re: [Beowulf] Supercomputers - iPad versus Cray > using iPad is just stupid, since it's an embarassingly expensive piece of eyecandy, not a computer. (yeah, yeah, you might love yours, but you still wrote a cheque for 65% of what you paid that landed directly on Apple's big pile of cash.) > > > Ahh.. but precisely because it is eyecandy means that $/FLOP means you're comparing the wrong metric. iPads are not computers. They're media storage and display devices. So the value (not the cost) needs to be compared to alternatives. It's "is my iPad cheaper than buying and carrying hardback books, watching pay-per-view videos,etc on an airplane or hotel room or airport concourse." I fully agree that iPads and similar tablet devices fill a much-needed gap between tiny-screened and underpowered phones and full-blown computers. However, I certainly didn't intend to, nor did I get from other peoples posts, to argue about the iPads worth for the purpose it was created. I argued this is a foolish, sensationalist use of it for a cluster. Whether or not an iPad is a good tablet I take no position on, having never owned one myself nor shopped around for them. Regarding power/computation, I think it's a silly metric to consider for this usage (probably just plugged into the wall). If we're concerned about the cost to power a cluster when all nodes are on, that's a fair concern, but it has to scale first. These machines won't allow for scaling with their included hardware, so that's not a reasonable consideration. If instead we want to have a cluster that is low power because we'll be running off battery for some duration, then it's foolish to use these since they rely on wifi or 3g to communicate, not to mention having a touchscreen, accelerometer and all kinds of other gadgets that eat battery-life. In short, I think the power/computation argument is totally contrived to give a scientific spin to an otherwise sensationalist "research" directive. In fact, I'd faster support a cluster of furbies because it's overtly absurd and frankly, funnier, than a cluster of iPads. This has the dangerous potential to being taken seriously (i.e. calling it "research" and aligning it with the directives of a big university such as Tennessee). If he just came out and said, "this is ridiculous, but we're bored and have some funding to burn" then I'd be on-board ;D. I'm sure Engadget or someone similar would love an article like that, not to mention Apple. Last, regarding educational purposes for mesh network routing, there are plenty of systems that are cheap as dirt that would be better for this kind of stuff, such as the ones we've been discussing on the list recently, that are far cheaper. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ellis at cse.psu.edu Thu Mar 8 15:47:29 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Thu, 08 Mar 2012 15:47:29 -0500 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F590C3C.5060705@ias.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> <4F590C3C.5060705@ias.edu> Message-ID: <4F591AE1.6080305@cse.psu.edu> On 03/08/2012 02:45 PM, Prentice Bisbal wrote: > Man climbed the mountain, because it was there, and techies will > continue to build clusters and other things out of unusual hardware > merely because they are there, too. I'm not certain the mountain metaphor follows. For instance, I'm not going to jump out of the window next to my desk just because it is there, because surviving it will be challenging, or because it will entertain people watching me attempt it. There's more going on with mountains and with this iPad clustering. > At the risk of being flamed, I think some of you are missing the point. > Neither the Register article or Dongarra's article were advocating for > the building clusters from iPads, I think they only meant to show how > far computing hardware has come in terms of power and cost. I remember reading a similar article comparing TI-83s and Eniac a long time ago. It was excellent, interesting, pointed, and hit on the latter point you mention (progress of computer science). However, that article didn't suggest that we should take TI-83s and make a cluster out of them, or even worse, that doing so is appealing for power reasons. Whether or not the article on the Reg argues to use them as a cluster is up in the air, but the the NYT article absolutely suggests that there could be some benefits to it: snip "Meanwhile, the fact that the iPad runs off a battery and is air-cooled has given Dr. Dongarra some ideas, like building a supercomputer composed of a couple of stacks of the tablets. ... ?It could be done and provide a very power-friendly cluster,? he said. snip > To say that people should only build clusters out of the proper hardware > is like telling artists to throw out their paints and brushes since we > now have cameras, that do a better job than they can. Again, weird analogy IMHO. I don't pay an artist to have a picture of something so I know what it looks like or can remember it later. If this was the 1200s and we didn't have photographs, then yes, that would be exactly what would happen. While back then many got portraits who could afford them, and higher prices were paid for higher fidelity pictures relative to reality, as soon as cameras hit the scene the demand lessened and artists indeed put down their brushes for that purpose. Modern art is (mostly) interpretive, and tries as hard as possible to avoid doing work that cameras can do. > BTW - I'd never waste my time building a cluster out of iPads, but I > defend the right of others to do so if that's what turns them on. I'm right there with you Prentice -- I'd even enjoy reading an article about just such a cluster if one was built, for the same absurdity factor that the furbie cluster was supported. Weird things are fun. But I don't defend the right of intelligent people to mislead the public (via the Reg and NYT) into actually believing there is something scientifically appealing about doing supercomputing with iPads. Sorry Apple, even you guys can't make hardcore computer science chic. It just doesn't work that way. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 8 17:09:31 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 8 Mar 2012 17:09:31 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: > Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. well, playstations had cell chips, which were, in their day, pretty respectable. to me, that's pure beowulf, in the classic sense. clustering very light-weight machines is sort of amusing, but it's hard to imagine a less appropriate substrate than an iPad. completely mundane hardware, extremely expensive display and touch panel, slow, small memory, no ethernet. a cluster of pogoplug mobiles would be pretty amusing instead, though pandaboards and the like have better cpus. the latter always seem overpriced, since you can get whole netbook for less. OK, I did think of a way that would be pretty cool in a demo/kluge, almost performance-art way: have the iPads communicate by modulating their LCD, and receiving via the screen-side camera. I didn't say it was practical ;) it would be hillarious to see a ring of ipads flashing QR codes at each other while running linpack. well, to me at least! anyone want to donate a pile of iPads? actually, a pile of android tabs would be even better, since you can put a decent/open dev env on them. regards, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 8 17:13:58 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 8 Mar 2012 17:13:58 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: <4F590C3C.5060705@ias.edu> References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> <4F590C3C.5060705@ias.edu> Message-ID: > the building clusters from iPads, I think they only meant to show how > far computing hardware has come in terms of power and cost. yes, but ipads suck as an example of this! it _is_ very interesting how far gflop-level performance has spread, but it's not really new, and any smartphone is, if anything, a better example. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 8 19:33:18 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 8 Mar 2012 16:33:18 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Mark Hahn Sent: Thursday, March 08, 2012 2:10 PM To: beowulf at beowulf.org Subject: Re: [Beowulf] Supercomputers - iPad versus Cray OK, I did think of a way that would be pretty cool in a demo/kluge, almost performance-art way: have the iPads communicate by modulating their LCD, and receiving via the screen-side camera. I didn't say it was practical ;) it would be hillarious to see a ring of ipads flashing QR codes at each other while running linpack. well, to me at least! >>> This is a cool idea.. and not so far fetched. I've been involved in optical interconnects which weren't that far off.. multipixel area sensors (i.e. cameras) looking at multiple other computers (all as nodes on the surface of a sphere, looking inward) each with a matrixed light emitting display. With the right optics, you could light up just the leds that shine on the desired recipients (or all of them for broadcast). The practical application is optical comm among a bunch of satellites flying in a not very well controlled formation. So, for the demo, the ring o'pads isn't a bad idea, especially if you put a wide angle lens on the front facing camera. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From rtomek at ceti.pl Fri Mar 9 14:26:16 2012 From: rtomek at ceti.pl (Tomasz Rola) Date: Fri, 9 Mar 2012 20:26:16 +0100 (CET) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: On Thu, 8 Mar 2012, Mark Hahn wrote: > > Any more bizarre than a cluster of Furbies? Or PS/2s, or any other weird hardware. > > well, playstations had cell chips, which were, in their day, pretty > respectable. to me, that's pure beowulf, in the classic sense. > > clustering very light-weight machines is sort of amusing, but it's hard > to imagine a less appropriate substrate than an iPad. completely mundane > hardware, extremely expensive display and touch panel, slow, small memory, > no ethernet. I remember, some (ten?) years ago on this list, there was an article about making a cluster of Compaq Ipaqs PDAs. Those particular models only had serial port and IR, so the author(s) went infrared. The performance of them was dog ugly, AFAIR, compared to what I could do with one iPaq with Linux and Python program doing similar stuff (prime sieve). I remember iPaqs were at that time touted as would be student's digital assistants, were to be introduced into US schools etc etc. Strange, seems like history repeats itself. Am I a digital prophet? Is Apple going to be bought by HP? :-) > OK, I did think of a way that would be pretty cool in a demo/kluge, > almost performance-art way: have the iPads communicate > by modulating their LCD, and receiving via the screen-side camera. > I didn't say it was practical ;) > > it would be hillarious to see a ring of ipads flashing QR codes at each other > while running linpack. well, to me at least! > > anyone want to donate a pile of iPads? actually, a pile of android tabs > would be even better, since you can put a decent/open dev env on them. Extremely cool idea. How about stacking them all against a huge mirror? Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola at bigfoot.com ** _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Fri Mar 9 17:19:47 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Fri, 9 Mar 2012 17:19:47 -0500 (EST) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C016925@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F58D6F8.5050708@cse.psu.edu> <4F58FDB1.4080008@cse.psu.edu> Message-ID: > Extremely cool idea. How about stacking them all against a huge mirror? a grid of ipads, all admiring themselves in a mirror would be very apt ;) maybe between *two* mirrors, since the _underside_ of apple products are so greatly admired by apple people... _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Fri Mar 9 19:37:10 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Fri, 9 Mar 2012 16:37:10 -0800 Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: Message-ID: Two mirrors face to face, so each device can "see" both front and back. The back facing camera is also higher resolution. Or, if you arranged them in a circle, so that A looks at B which looks at C which looks at D.... which looks at A. Mirrors could provide a redundant optical path. On 3/9/12 2:19 PM, "Mark Hahn" wrote: >> Extremely cool idea. How about stacking them all against a huge mirror? > >a grid of ipads, all admiring themselves in a mirror would be very apt ;) >maybe between *two* mirrors, since the _underside_ of apple products >are so greatly admired by apple people... >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From rtomek at ceti.pl Sat Mar 10 10:30:50 2012 From: rtomek at ceti.pl (Tomasz Rola) Date: Sat, 10 Mar 2012 16:30:50 +0100 (CET) Subject: [Beowulf] Supercomputers - iPad versus Cray In-Reply-To: References: Message-ID: On Fri, 9 Mar 2012, Lux, Jim (337C) wrote: > Two mirrors face to face, so each device can "see" both front and back. > The back facing camera is also higher resolution. Or, if you arranged > them in a circle, so that A looks at B which looks at C which looks at > D.... which looks at A. > > Mirrors could provide a redundant optical path. And we finally come to nice algorithmic problem: given a number of mirrors, their sizes and orientation compute maximum number of iphones that could be arranged into an optical cluster. Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola at bigfoot.com ** _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Wed Mar 14 11:22:24 2012 From: eugen at leitl.org (Eugen Leitl) Date: Wed, 14 Mar 2012 16:22:24 +0100 Subject: [Beowulf] oil immersion cooled blades Message-ID: <20120314152224.GU9891@leitl.org> http://www.heise.de/newsticker/meldung/Server-Blades-in-Oel-1471734.html (translation courtesy Google Translate): Server blades in oil Hardcore Computer LSS 200 Photo: Boston The U.S. company has with the hardcore computer Submerged Liquid server developed (LSS 200), a server blade in the format of immersion cooling uses: The entire unit sits in a closed housing which oil flows. This "Core Coolant" is according to the safety of a non-toxic and biodegradable compound based on a synthetic wax. The advantages of this cooling method is called Hardcore Computer eliminated including a more efficient cooling, because the coolant can be for example of a data center transported directly to heat exchangers, and going through cold air. Because no special rack delivers warm air directly into the environment, it can also be operated in locations without air conditioning. The server manufacturer in Boston, the LSS 200 is added to its product line and offers it in Germany. However, neither prices nor called Boston delivery, and plugged in the server rack now slightly dusty Technology: In comparison to the recently announced Xeon E5600 Xeon E5 falls off significantly. Also, the data sheet (PDF file) of the CLS 200 with hardcore computer can open questions. For example, detailed information is lacking on the power supplies, the special chassis and to the oil-cooling equipment. The CLS 200 will also allow the use of a PCIe expansion card, such as a Tesla accelerator or InfiniBand adapter card - if it works well both at the same time remains uncertain. Finally, the question remains open on the disk, at least be mentioned only SSDs in 2.5-inch format. The entire board is surrounded by oil. Picture: Hardcore Computer In the United States sold the hardcore computer desktops and Reactor Reactor X, and the detonator Workstation with Immersion cooling. Oil as a coolant has a lower specific heat capacity than water, but cools better than air and leaks caused by short circuits. When complete immersion of the coolant reaches all the components, while in the water cooling individual heat sinks are needed which do not reach all critical components of any assemblies. Optimal cooling water can be exploited if the motherboard design optimized to be. ( ciw ) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Wed Mar 14 18:02:57 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed, 14 Mar 2012 18:02:57 -0400 (EDT) Subject: [Beowulf] oil immersion cooled blades In-Reply-To: <20120314152224.GU9891@leitl.org> References: <20120314152224.GU9891@leitl.org> Message-ID: > Server blades in oil sounds like some brutalist take on molecular gastronomy! > Hardcore Computer LSS 200 http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf seems kind of uninspiring, in hardware specs. 1366 socket, not 2011, optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really a density play. I guess I'm a bit skeptical about the utility of this approach - would be nice if they had some technical literature. something about thermal resistance. define how the oil bath dumps the heat (water hookups in the back?) comparison to modern heatpipe-based solutions, etc. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Wed Mar 14 23:12:47 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Wed, 14 Mar 2012 20:12:47 -0700 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: Message-ID: Immersing motherboards in oil has been done for years (probably even mentioned on this list). There's countless hacker versions using various and sundry off the shelf mobos and power supplies etc in various and sundry containers (ice chests, aquariums, purpose built acrylic boxes) using a variety of coolants. The vast majority of these have no "thermal engineering".. They're more "we tried it and it seemed to work". There's not a lot of content at the hardcore computer website. It could be ok or it might not. They're not real clear on how they actually get the heat out of the enclosures. The MSDS for their coolant says that it's basically synthetic mineral oil with a antioxidant (you can look up the CAS number to get more info). I'd be a bit nervous about the fire hazard aspects of a machine room full of the stuff. So here's my experience using oil as an insulator/coolant. 1) it wicks up insulated wire, particularly stranded. Put the mobo in oil and the power supply outside, and pretty soon your power supply will be full of oil. 2) oil leaks. There is *nothing* that is oil insulated that doesn't have a fine film of oil on its surface eventually, unless it is in a hermetically sealed can with welded/crimped seals. 3) oil is a mess when you need to fix something The Cray-2 used Fluorinert(tm) FC-74 as the coolant, which is very nice to work with, although expensive. It doesn't wet things very well, so when you pull something out of the bath, it doesn't bring much fluid with it. The Cray used it as a heat transfer medium to water coolant. I think they had a way to drain it into a tank quickly for servicing. It can be used for ebullient (boiling) cooling by picking the right vapor pressure/BP grade (the Cray didn't use this). Ebullient cooling is quite efficient at moving the heat away because it's a phase change, and the bubbling causes good circulation, but it does require careful design so you don't get film boiling/Leidenfrost effect (the phenomenon that protects your feet when walking across burning coals barefoot) On 3/14/12 3:02 PM, "Mark Hahn" wrote: >> Server blades in oil > >sounds like some brutalist take on molecular gastronomy! > >> Hardcore Computer LSS 200 > >http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf > >seems kind of uninspiring, in hardware specs. 1366 socket, not 2011, >optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really >a density play. > >I guess I'm a bit skeptical about the utility of this approach - >would be nice if they had some technical literature. something about >thermal resistance. define how the oil bath dumps the heat (water >hookups in the back?) comparison to modern heatpipe-based solutions, etc. >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Thu Mar 15 06:51:43 2012 From: eugen at leitl.org (Eugen Leitl) Date: Thu, 15 Mar 2012 11:51:43 +0100 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: Message-ID: <20120315105143.GY9891@leitl.org> On Wed, Mar 14, 2012 at 08:12:47PM -0700, Lux, Jim (337C) wrote: > The Cray-2 used Fluorinert(tm) FC-74 as the coolant, which is very nice to > work with, although expensive. I've worked with a few 3M fluorinerts for blood substitutes (nothing kills quite like liberated fluoride during sonickating) and for cooling/partial liquid ventilation purposes. Heavy stuff, twice the density of water. Seems to do havoc to your immune system at long-term exposures. > It doesn't wet things very well, so when you pull something out of the > bath, it doesn't bring much fluid with it. The Cray used it as a heat > transfer medium to water coolant. I think they had a way to drain it into > a tank quickly for servicing. > > > It can be used for ebullient (boiling) cooling by picking the right vapor > pressure/BP grade (the Cray didn't use this). Ebullient cooling is quite The Cray X1 used evaporative spray cooling at least http://www.cse.ohio-state.edu/~panda/875/class_slides/cray-jaguar.pdf > efficient at moving the heat away because it's a phase change, and the > bubbling causes good circulation, but it does require careful design so > you don't get film boiling/Leidenfrost effect (the phenomenon that > protects your feet when walking across burning coals barefoot) There's a K/Na eutectic which is cheap (unlike Ga eutectics) and is liquid at RT. Extreme fire hazard if you'll get a leak in air, though. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Thu Mar 15 21:57:27 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Thu, 15 Mar 2012 21:57:27 -0400 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> Message-ID: <4F629E07.2000209@ias.edu> On 3/14/2012 6:02 PM, Mark Hahn wrote: >> Server blades in oil > > sounds like some brutalist take on molecular gastronomy! > >> Hardcore Computer LSS 200 > > http://www.hardcorecomputer.com/Resources/assets/Documents/LSS-specs.pdf > > seems kind of uninspiring, in hardware specs. 1366 socket, not 2011, > optional fast network, gpu, etc, builtin ipmi. 8x in 5U, so not really > a density play. > > I guess I'm a bit skeptical about the utility of this approach - > would be nice if they had some technical literature. something about > thermal resistance. define how the oil bath dumps the heat (water > hookups in the back?) comparison to modern heatpipe-based solutions, etc. No need for water hookups. You can circulate the oil through an oil to air heat exchanger. That's what those green cooing guys at the SC conferences do. Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 15 23:47:42 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 15 Mar 2012 23:47:42 -0400 (EDT) Subject: [Beowulf] oil immersion cooled blades In-Reply-To: <4F629E07.2000209@ias.edu> References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: >> I guess I'm a bit skeptical about the utility of this approach - >> would be nice if they had some technical literature. something about >> thermal resistance. define how the oil bath dumps the heat (water >> hookups in the back?) comparison to modern heatpipe-based solutions, etc. > > No need for water hookups. You can circulate the oil through an oil to > air heat exchanger. That's what those green cooing guys at the SC is that really better than going to air directly? I guess I'd like to see the the numbers - to my way of thinking, it's almost all about the thermal resistance. transferring heat to oil, then to air, means two stages of resistance. using oil would permit a bigger air interface, though I suppose. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From ntmoore at gmail.com Fri Mar 16 00:47:59 2012 From: ntmoore at gmail.com (Nathan Moore) Date: Thu, 15 Mar 2012 23:47:59 -0500 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: If you take a cm^3 of space, right next to the cpu and fill it either with air, or with oil, you'll have many, many, more atomic/molecular degrees of freedom to fill with energy in the cm^3 of oil. Getting that energy out of the cooling medium seems primarily like a fluid-flow problem - given oil's higher heat capacity, you can leave it around something hot, and still have it serve as an effective heat sink for a longer period of time than you can with air. My point is, the fan for the air has to run much faster than the oil pump for the oil coolant. I'm too young for this, but didn't VW and Porche cool some of their engines with oil through the early 1980's? On Thu, Mar 15, 2012 at 10:47 PM, Mark Hahn wrote: > >> I guess I'm a bit skeptical about the utility of this approach - > >> would be nice if they had some technical literature. something about > >> thermal resistance. define how the oil bath dumps the heat (water > >> hookups in the back?) comparison to modern heatpipe-based solutions, > etc. > > > > No need for water hookups. You can circulate the oil through an oil to > > air heat exchanger. That's what those green cooing guys at the SC > > is that really better than going to air directly? > I guess I'd like to see the the numbers - to my way of thinking, > it's almost all about the thermal resistance. transferring heat to oil, > then to air, means two stages of resistance. using oil would permit > a bigger air interface, though I suppose. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- - - - - - - - - - - - - - - - - - - - - - Nathan Moore Associate Professor, Physics Winona State University - - - - - - - - - - - - - - - - - - - - - -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From prentice at ias.edu Fri Mar 16 10:51:49 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 16 Mar 2012 10:51:49 -0400 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: <4F635385.5040606@ias.edu> On 03/15/2012 11:47 PM, Mark Hahn wrote: >>> I guess I'm a bit skeptical about the utility of this approach - >>> would be nice if they had some technical literature. something about >>> thermal resistance. define how the oil bath dumps the heat (water >>> hookups in the back?) comparison to modern heatpipe-based solutions, etc. >> No need for water hookups. You can circulate the oil through an oil to >> air heat exchanger. That's what those green cooing guys at the SC > is that really better than going to air directly? > I guess I'd like to see the the numbers - to my way of thinking, > it's almost all about the thermal resistance. transferring heat to oil, > then to air, means two stages of resistance. using oil would permit > a bigger air interface, though I suppose. Oh boy... I'm having flashbacks of college. The short answer is "yes" that is much better than going to air directly. I'm tying to think of a simple way to explain why, and I'm starting to have flashbacks of my Chem Eng. classes: Thermodynamics, Transport Phenomena... ugh. Here's an attempt at a longer explanation. Sorry if I'm rehashing what you already know. Yes, going from oil, or any liquid, to air is better than using air directly. That is why car engines are liquid-cooled instead of air cooled. It comes down to two physical properties: Thermal conductivity, and thermal capacity. Thermal conductivity is the ability for a material to move energy in the form of heat from one place to another. Thermal capacity is how much energy in the form of heat can be stored in a given quantity (mass or volume) of a substance. In general, liquids and solids have better thermal conductivity and capacity than gases. We use liquid for cooling because the liquid's superior conductivity draws the heat away from the object to be cooled quicker, requiring less surface area, and less "residence time" for the coolant. The superior capacity allows us to use less volume of coolant to store the same amount of "heat" (energy). And liquids are better than solids because they flow, and the bulk movement allows us to easily transport the heat stored in the coolant away easier than if were trying to move solids. Using liquids allows to do draw heat out of tiny confined spaces, like a 1U server, or a cars engine block, that we can't do effectively with air. Once we draw that heat out, it needs to go somewhere, which, in most cases, is atmospheric are, so we pump the liquids to a cooling tower outside, the radiator in the front of our engine bay, or somewhere where is a large amount of air and room for all the surface area the heat exchanger will need (since air has a low thermal conductivity, it needs a lot more surface area to transfer the heat into it than a liquid does). This last point you hit on in the last sentence of your post. That is why car radiators are large and have lots of think aluminium fins in them (more fins = more surface area for the air to contact), and we have massive cooling towers or heat exchangers outside our office buildings. You are correct in that there are inefficiencies when you transfer energy from one material to another, but those are usually trivial and offset by the greater effectiveness of liquids in pulling the heat away from the heat source in the first place. I hope that helps. Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 16 11:08:16 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 16 Mar 2012 11:08:16 -0400 Subject: [Beowulf] oil immersion cooled blades In-Reply-To: References: <20120314152224.GU9891@leitl.org> <4F629E07.2000209@ias.edu> Message-ID: <4F635760.3020305@ias.edu> On 03/16/2012 12:47 AM, Nathan Moore wrote: > If you take a cm^3 of space, right next to the cpu and fill it either > with air, or with oil, you'll have many, many, more atomic/molecular > degrees of freedom to fill with energy in the cm^3 of oil. Getting > that energy out of the cooling medium seems primarily like a > fluid-flow problem - given oil's higher heat capacity, you can leave > it around something hot, and still have it serve as an effective heat > sink for a longer period of time than you can with air. My point is, > the fan for the air has to run much faster than the oil pump for the > oil coolant. > > I'm too young for this, but didn't VW and Porche cool some of their > engines with oil through the early 1980's? Yes, I and was going to mention that in the e-mail. I just posted. The original VW "boxer" engines were air cooled. If you look at the cylinders, they have heat fins on them like a CPU heat sink or a motorcycle engine to promote heat transfer to the air. For VW fanatics (of which I used to be one) there two types of VWs: air-cooled, and everything else. I was in the "everything else" camping, owning several rabbits and GTis. The engines in the early Porsches and the 911s trace their lineage all the way back to the first VW beetle engines, and were also air-cooled boxer engines (boxer = horizontally-opposed cylinders). Porsche 911s retained air-cooled engines well into the 1990s, and went to water-cooled engines because it was getting too hard to meet tougher and tougher noise and emissions requirements with air-cooled engines. As a bonus, I think liquid-cooling really helped increase how much horse power they could put out, too. But you lose out on the very distinctive sound that the air-cooled 911 had. :( Do you remember the 911 turbos with that big "whale tale" rear wing" That wing wasn't really there for aerodynamics (although it did help for that, too) It was there to house the massive air-to-air intercoolers needed to cool the turbocharged air. Since they went to liquid cooled engines, you really don't see that any more. Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From john.hearns at mclaren.com Wed Mar 21 10:11:51 2012 From: john.hearns at mclaren.com (Hearns, John) Date: Wed, 21 Mar 2012 14:11:51 -0000 Subject: [Beowulf] Google greywater cooling Message-ID: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Flagging up yet another Register article I'm afraid, but it is interesting http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_coo ling/ The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From ellis at cse.psu.edu Wed Mar 21 20:45:46 2012 From: ellis at cse.psu.edu (Ellis H. Wilson III) Date: Wed, 21 Mar 2012 20:45:46 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> Message-ID: <4F6A763A.3080404@cse.psu.edu> On 03/21/2012 10:11 AM, Hearns, John wrote: > Flagging up yet another Register article I?m afraid, but it is interesting > > http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ Certainly interesting. I missed this one in my El Reg RSS. Does anyone have references on a (modern) picture of what their servers look like? In other words, they are particularly vague in the article and the associated video on how they actually use this water to do the cooling. I assume it is not "real" water cooling, but utilizing that water in their air conditioning units somehow. I cannot imagine they use the water to re-humidify their server rooms, since rooms smelling of piss and chlorine probably isn't appreciated by any but the most green of environmentalists. Best, ellis _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From jmdavis1 at vcu.edu Wed Mar 21 22:17:41 2012 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed, 21 Mar 2012 22:17:41 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6A763A.3080404@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> Message-ID: <4F6A8BC5.4060207@vcu.edu> Greywater isn't sewage, though it is often dealt with in the same way. Greywater could be from sinks, plant watering systems, condensation, etc. It could also be recaptured and reused fresh water from some cooling systems. On 3/21/2012 8:45 PM, Ellis H. Wilson III wrote: > On 03/21/2012 10:11 AM, Hearns, John wrote: >> Flagging up yet another Register article I?m afraid, but it is interesting >> >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > Certainly interesting. I missed this one in my El Reg RSS. Does anyone > have references on a (modern) picture of what their servers look like? > In other words, they are particularly vague in the article and the > associated video on how they actually use this water to do the cooling. > I assume it is not "real" water cooling, but utilizing that water in > their air conditioning units somehow. I cannot imagine they use the > water to re-humidify their server rooms, since rooms smelling of piss > and chlorine probably isn't appreciated by any but the most green of > environmentalists. > > Best, > > ellis > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From bob at drzyzgula.org Wed Mar 21 22:52:09 2012 From: bob at drzyzgula.org (Bob Drzyzgula) Date: Wed, 21 Mar 2012 22:52:09 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6A8BC5.4060207@vcu.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: And it certainly is not the case that greywater would be used as a potable -- or breathable -- source of water or humidity. To obtain greywater in the quantities useful for data center cooling, it typically has to come from the local public water company, and generally requires a local ordinance allowing its use. Moreover, such ordinances will specify exactly what the water can be used for, and plumbing codes will ensure that the greywater will be kept separate from the potable water, much as are waste and storm water. --Bob On Wed, Mar 21, 2012 at 10:17 PM, Mike Davis wrote: > Greywater isn't sewage, though it is often dealt with in the same way. > Greywater could be from sinks, plant watering systems, condensation, > etc. It could also be recaptured and reused fresh water from some > cooling systems. > > > On 3/21/2012 8:45 PM, Ellis H. Wilson III wrote: >> On 03/21/2012 10:11 AM, Hearns, John wrote: >>> Flagging up yet another Register article I?m afraid, but it is interesting >>> >>> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ >> Certainly interesting. ?I missed this one in my El Reg RSS. ?Does anyone >> have references on a (modern) picture of what their servers look like? >> In other words, they are particularly vague in the article and the >> associated video on how they actually use this water to do the cooling. >> ? ?I assume it is not "real" water cooling, but utilizing that water in >> their air conditioning units somehow. ?I cannot imagine they use the >> water to re-humidify their server rooms, since rooms smelling of piss >> and chlorine probably isn't appreciated by any but the most green of >> environmentalists. >> >> Best, >> >> ellis >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From hahn at mcmaster.ca Thu Mar 22 11:04:19 2012 From: hahn at mcmaster.ca (Mark Hahn) Date: Thu, 22 Mar 2012 11:04:19 -0400 (EDT) Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: > http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ I grew up in a city cooled by a central evaporative water-chilling plant, so I'm always surprised how unusual this sort of project is. does anyone have comments on how to enable/encourage/foster less conventional datacenter engineering? I've participated in two DC projects, and both went with safe, boring and inefficient solutions, simply because power costs weren't high enough to motivate any deviation from utter conventionality. power costs are somewhat hidden on uni campuses, and people involved are very, extremely, hyper-risk-averse... I suppose part of the problem is in retrofitting a green DC into existing buildings - it's a lot easier if you can plan a new building to, for instance, take advantage of waste heat. I'm also interested in anyone who is currently running clusters at higher than conventional/ASHRAE temperatures. thanks, mark hahn. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Thu Mar 22 11:19:41 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Thu, 22 Mar 2012 16:19:41 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: <4F6B430D.3070503@gmail.com> Those are some interesting thoughts, I live on the island of Malta, where electrical prices are expensive. The Dc i used to work at charged 58 euros per kwh of electricity used if you colocate. I have a few ideas im working on proposing to vodafone malta, and if they deem it feasible it would be something I coudl move forward with for them. Here in Malta we have alot of wind and solar, I dunno why to be honest electrical companies dont burn organic wastes to help generate electricity. I am actually thinking of approaching the government with the idea of using organic wastes to generate electricity locally, and start to cut our dependence on oil for electrical generation. On 22/03/2012 16:04, Mark Hahn wrote: >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > I grew up in a city cooled by a central evaporative water-chilling plant, > so I'm always surprised how unusual this sort of project is. > > does anyone have comments on how to enable/encourage/foster less > conventional datacenter engineering? > > I've participated in two DC projects, and both went with safe, boring > and inefficient solutions, simply because power costs weren't high enough > to motivate any deviation from utter conventionality. power costs are > somewhat hidden on uni campuses, and people involved are very, extremely, > hyper-risk-averse... > > I suppose part of the problem is in retrofitting a green DC into > existing buildings - it's a lot easier if you can plan a new building > to, for instance, take advantage of waste heat. > > I'm also interested in anyone who is currently running clusters at > higher than conventional/ASHRAE temperatures. > > thanks, mark hahn. > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From brian.ropers.huilman at gmail.com Thu Mar 22 12:03:38 2012 From: brian.ropers.huilman at gmail.com (Brian D. Ropers-Huilman) Date: Thu, 22 Mar 2012 11:03:38 -0500 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6B430D.3070503@gmail.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: > Those are some interesting thoughts, I live on the island of Malta, > where electrical prices are expensive. The Dc i used to work at charged > 58 euros per kwh of electricity used if you colocate. I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay ~$0.11 per kw/hr residential here in the States (after various fees are thrown in, too). I see a "reality check" error in your ~700x cost ... -- Brian D. Ropers-Huilman 612.234.7778 (m) _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Thu Mar 22 12:16:41 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Thu, 22 Mar 2012 17:16:41 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: <4F6B5069.2060800@gmail.com> When i was looking to colocate a server for my business there thats what i was told. The problem with where I am is that we rely on imported oil which is used to generate electricity. So basically it depends on the price of oil which causes electrical prices to fluctuate greatly here. I have an idea which i need to draft up a business proposal for vodafone malta to see if they can see if its feasible to go completely green, and if so that would be a project that I would head up and move forward with. On 22/03/2012 17:03, Brian D. Ropers-Huilman wrote: > On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: >> Those are some interesting thoughts, I live on the island of Malta, >> where electrical prices are expensive. The Dc i used to work at charged >> 58 euros per kwh of electricity used if you colocate. > I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's > conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay > ~$0.11 per kw/hr residential here in the States (after various fees > are thrown in, too). I see a "reality check" error in your ~700x cost > ... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Thu Mar 22 12:17:44 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Thu, 22 Mar 2012 17:17:44 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: <4F6B50A8.8010005@gmail.com> On a residential electricity note. between 3 people in this house we pay about 800 euros which is about 266 euros a person which equates to about 88 euros a month for electricity. On 22/03/2012 17:03, Brian D. Ropers-Huilman wrote: > On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: >> Those are some interesting thoughts, I live on the island of Malta, >> where electrical prices are expensive. The Dc i used to work at charged >> 58 euros per kwh of electricity used if you colocate. > I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's > conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay > ~$0.11 per kw/hr residential here in the States (after various fees > are thrown in, too). I see a "reality check" error in your ~700x cost > ... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 22 20:14:26 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 22 Mar 2012 17:14:26 -0700 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6B430D.3070503@gmail.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: 58 Euro? Or 0.58 Euro? (and I think $0.34/kWhr in California is high...) -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina Sent: Thursday, March 22, 2012 8:20 AM To: beowulf at beowulf.org Subject: Re: [Beowulf] Google greywater cooling Those are some interesting thoughts, I live on the island of Malta, where electrical prices are expensive. The Dc i used to work at charged 58 euros per kwh of electricity used if you colocate. I have a few ideas im working on proposing to vodafone malta, and if they deem it feasible it would be something I coudl move forward with for them. Here in Malta we have alot of wind and solar, I dunno why to be honest electrical companies dont burn organic wastes to help generate electricity. I am actually thinking of approaching the government with the idea of using organic wastes to generate electricity locally, and start to cut our dependence on oil for electrical generation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 22 20:23:22 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 22 Mar 2012 17:23:22 -0700 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6B50A8.8010005@gmail.com> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> <4F6B50A8.8010005@gmail.com> Message-ID: Much of the electricity in the US is actually generated by burning oil, as well.. As it happens, enemalta has a web presence.. There's a fixed service charge of ?65 for single phase residential, ?195 for three phase.. .. that's where your big cost is.. The actual electricity is ?0.161 to ?0.620 per kWh in 5 tiers. (it doesn't actually say kWh, but I assume that's what the units are..) So that's high on a per kWh basis (compared to, say, California) but not exceedingly high.. (maybe 2x-3x.) For businesses: The flat fee is ?360 for 3phase, and rates go from 0.162 up to 0.215 (at 20-60 MWh) and then down to 0.144 at 5GWh. This is a pretty typical sort of rate schedule.. gradually rising, then falling. (and of course, they stick you for reactive power, too, just like in the U.S.) -----Original Message----- From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina Sent: Thursday, March 22, 2012 9:18 AM To: Brian D. Ropers-Huilman Cc: beowulf at beowulf.org Subject: Re: [Beowulf] Google greywater cooling On a residential electricity note. between 3 people in this house we pay about 800 euros which is about 266 euros a person which equates to about 88 euros a month for electricity. On 22/03/2012 17:03, Brian D. Ropers-Huilman wrote: > On Thu, Mar 22, 2012 at 10:19, Jonathan Aquilina wrote: >> Those are some interesting thoughts, I live on the island of Malta, >> where electrical prices are expensive. The Dc i used to work at >> charged >> 58 euros per kwh of electricity used if you colocate. > I'm sorry, but ... are your numbers right? ?58 per kw/hr? At today's > conversion rate of $1.3167 == ?1, that's ~$76.37 per kw/hr. I pay > ~$0.11 per kw/hr residential here in the States (after various fees > are thrown in, too). I see a "reality check" error in your ~700x cost > ... > _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eagles051387 at gmail.com Fri Mar 23 03:22:09 2012 From: eagles051387 at gmail.com (Jonathan Aquilina) Date: Fri, 23 Mar 2012 08:22:09 +0100 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> <4F6B430D.3070503@gmail.com> Message-ID: <4F6C24A1.5040802@gmail.com> 58 euros I can dig through my emails and see if i find the quote for colocating a server. On 23/03/2012 01:14, Lux, Jim (337C) wrote: > 58 Euro? Or 0.58 Euro? (and I think $0.34/kWhr in California is high...) > > -----Original Message----- > From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina > Sent: Thursday, March 22, 2012 8:20 AM > To: beowulf at beowulf.org > Subject: Re: [Beowulf] Google greywater cooling > > Those are some interesting thoughts, I live on the island of Malta, where electrical prices are expensive. The Dc i used to work at charged > 58 euros per kwh of electricity used if you colocate. > > I have a few ideas im working on proposing to vodafone malta, and if they deem it feasible it would be something I coudl move forward with for them. > > Here in Malta we have alot of wind and solar, I dunno why to be honest electrical companies dont burn organic wastes to help generate electricity. > > I am actually thinking of approaching the government with the idea of using organic wastes to generate electricity locally, and start to cut our dependence on oil for electrical generation. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 23 09:47:18 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 23 Mar 2012 09:47:18 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6A763A.3080404@cse.psu.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> Message-ID: <4F6C7EE6.4060802@ias.edu> On 03/21/2012 08:45 PM, Ellis H. Wilson III wrote: > On 03/21/2012 10:11 AM, Hearns, John wrote: >> Flagging up yet another Register article I?m afraid, but it is interesting >> >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > Certainly interesting. I missed this one in my El Reg RSS. Does anyone > have references on a (modern) picture of what their servers look like? > In other words, they are particularly vague in the article and the > associated video on how they actually use this water to do the cooling. > I assume it is not "real" water cooling, but utilizing that water in > their air conditioning units somehow. I cannot imagine they use the > water to re-humidify their server rooms, since rooms smelling of piss > and chlorine probably isn't appreciated by any but the most green of > environmentalists. > That water is first sent through a Google-built sidestream treatment plant, which sterilizes, filters, and clorinates it, cleaning it just enough for it to be used in the data center's evaporative cooling towers. The water that doesn't evaporate away during that process is then further purified in Google's on-premises Effluent Treatment Plant. "There, we treat the water once again to disinfect it," Brown writes, "remove mineral solids and send it back out to the Chattahoochee ? clean, clear and safe." Based on the above description, this is the most likely scenario: The greywater flows down the outside of heat exchangers in the cooling tower. As it evaporates, the change of state removes a lot of the heat from the liquid inside the heat exchanger (probably also water, but clean, treated water). The chilled water inside the heat exchanger then flows into heat exchangers inside the data center CRACs where it absorbs heat from the warm data center air., cooling the air. Any time you use water in a closed-loop system (chilled water cooling or steam heating, for example). It needs to be very clean, and have additional chemicals added to it prevent corrosion or mineral buildup inside the pipes, so greywater would probably never be used inside one of these systems. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From prentice at ias.edu Fri Mar 23 09:59:37 2012 From: prentice at ias.edu (Prentice Bisbal) Date: Fri, 23 Mar 2012 09:59:37 -0400 Subject: [Beowulf] Google greywater cooling In-Reply-To: References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6A8BC5.4060207@vcu.edu> Message-ID: <4F6C81C9.4010604@ias.edu> On 03/22/2012 11:04 AM, Mark Hahn wrote: >> http://www.theregister.co.uk/2012/03/20/google_greywater_data_center_cooling/ > I grew up in a city cooled by a central evaporative water-chilling plant, > so I'm always surprised how unusual this sort of project is. > > does anyone have comments on how to enable/encourage/foster less > conventional datacenter engineering? > > I've participated in two DC projects, and both went with safe, boring > and inefficient solutions, simply because power costs weren't high enough > to motivate any deviation from utter conventionality. power costs are > somewhat hidden on uni campuses, and people involved are very, extremely, > hyper-risk-averse... > > I suppose part of the problem is in retrofitting a green DC into > existing buildings - it's a lot easier if you can plan a new building > to, for instance, take advantage of waste heat. > I think you need to get the bean-counters involved, and sell them on the money savings, not just the planet savings. Princeton University just completed contruction of their High-Performance Computing Research Center (HPCRC. The name is a complete misnomer - its a production datacenter that houses ALL university computing, not just HPC, and it's not a research center). It's a pretty amazing building, and is shooting for some level of LEED certification. To me, the place looks like an over-engineered monster (I've toured it several times), but the engineers giving the tours always point out many different ways the datacenter is both saving the planet and saving the university a lot of money at the same time. They've done an amazing job with it, and I think that the end of the day, the monetary savings is what made it happen. http://www.princeton.edu/facilities/info/major_projects/HPCRC/ I know there are some lurkers from PU on this list who are involved with the HPCRC. Maybe they'll pipe up on this topic. -- Prentice _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From diep at xs4all.nl Sun Mar 25 14:47:00 2012 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sun, 25 Mar 2012 20:47:00 +0200 Subject: [Beowulf] Nvidia's quantum leap in 28 nm Message-ID: <630CE216-EC95-4A42-B7B3-070549C803B3@xs4all.nl> It's been some year or 12 that a genius visited me. His expertise being the same like Einsteins, it's not much of a question what his research topics were. Though not deep into computer hardware he told me that for massive computing, just above the 1Ghz border would prove to be a big barrier as electrons basically move at around 1/3 of the lightspeed, which translates to 1.3Ghz in metals like aluminium. At copper so he said that barrier might be a tad higher than aluminium, yet even then the power needed for such speeds would prove to be massive. At that moment intel's marketing department shouted out loud their P4's would clock 10Ghz by 2010. Well the P4 never got there and we got into the megacore count game for HPC. AMD Now AMD needs 4 PE's for doing double precision, so their core count of 1536 actually wasn't more than the 5000 series with 1600. Their new 7970 gpu with 2048 pe's has the double precision equivalent in core count of 512 compute cores. Actually the 7970 mostly profits from a 100Mhz higher frequency with some boosting to 1Ghz at some overclocked cards, it gets impressive game scores. As for gpgpu of course, moving from 1536 cores to 2048 is an interesting improvement, yet far away from a doubling. The 7970 is said to have around 4.31B transistors (see http://www.anandtech.com/show/5261/amd-radeon-hd-7970-review ) NVIDIA FERMI Fermi, nvidia's 40 nm gpu which currently gets used in HPC, it has 3 bilion transistors. Here at home i have a few 2075 Tesla's with 448 cores producing a tad more than 0.5 Tflop which was its a big improvement over the previous generation. The Nvidia Fermi on the other hand in the form of the GTX 560 clocks 1.644Ghz and the 580 clocks 1.544Ghz. For gpgpu this is on the risky side as getting far over that 1Ghz seems to be a problem. The tesla's therefore are clocked safely 1.15Ghz NVIDIA KEPLER 2012 The new kid on the block from Nvidia is the Kepler. It's in the 28 nm proces technology, just like AMD's 7970. Now i'm not gonna redo a review for games, there is great sites for that. http://www.anandtech.com/show/5699/nvidia-geforce-gtx-680-review/1 Over here we are interested in the implications for the beowulf systems of course, i read that as HPC implications. Let's look to facts and then speculate what that means for HPC: I'm still trying to full understand the differences, yet it seems as if nvidia clocked back to 1Ghz the cores. That should make it easier to release a gpu for gpgpu as well. In the meantime core count went up to 1536. The chip itself has 3.5 billion transistors. Just 500M more than Fermi, meanwhile at a factor 2.04 smaller proces, that means it will consume less juice and a lot less juice. Benchmarks at anandtech confirm this. Now that's a MASSIVE quantumleap. Basically factor 3 the number of cores available to HPC. Additional to that the memory is 256 bits wide, versus 384 bits for Fermi. This should make it easier to release 2 gpu's on a single card. Whether nvidia has those plans for gpgpu tesla's we can only speculate about, as the chip eats less juice, it sure fits this time within the power envelope. So where the gamer kids with sureness can expect a 690 gpu, for HPC we of course cheer if nvidia manages to improve to 1.5 - 1.7 Tflop for their new gpu, with the option to move to 3 - 3.4 Tflop double precision for a 2 gpu Tesla card. Note that some might argue that the 680 has less double precision capabilities than the 580. However for the Tesla this doesn't matter, as what happens for gamerscards is that they disable some transistors; so the Tesla gpu will be the exact same chip like the kids has, just with the double precision enabled. The same thing was the case with Fermi, so it's logical to expect that to happen with Kepler as well. Seems like intel can also scrap their current corner project as they have a new goal, namely 4 Tflop, rather than a 1 Tflop manycore :) As for Nvidia, releasing a new chip that's factor 3 the power of your previous one for gpgpu sure is a big quantum leap! _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From samuel at unimelb.edu.au Sun Mar 25 19:53:10 2012 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Mon, 26 Mar 2012 10:53:10 +1100 Subject: [Beowulf] Google greywater cooling In-Reply-To: <4F6C7EE6.4060802@ias.edu> References: <207BB2F60743C34496BE41039233A8090C66EB11@MRL-PWEXCHMB02.mil.tagmclarengroup.com> <4F6A763A.3080404@cse.psu.edu> <4F6C7EE6.4060802@ias.edu> Message-ID: <4F6FAFE6.8020607@unimelb.edu.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24/03/12 00:47, Prentice Bisbal wrote: > Any time you use water in a closed-loop system (chilled water > cooling or steam heating, for example). It needs to be very clean, > and have additional chemicals added to it prevent corrosion or > mineral buildup inside the pipes, so greywater would probably never > be used inside one of these systems. In our experience they also add anti-fungals like azoles to them too. cheers, Chris - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9vr+YACgkQO2KABBYQAh/zzQCfS5p5Z4jEu0VPukZPgo8auD0Q B6MAnArzv9TgtPUctu1X0nb5ktscjQgn =Xjyl -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From eugen at leitl.org Tue Mar 27 11:15:51 2012 From: eugen at leitl.org (Eugen Leitl) Date: Tue, 27 Mar 2012 17:15:51 +0200 Subject: [Beowulf] Everything You Know (about Parallel Programming) Is Wrong!: A Wild Screed about the Future Message-ID: <20120327151551.GI17245@leitl.org> http://splashcon.org/2011/program/dls/245-invited-talk-2 Mon 2:00-3:00 pm - Pavilion East Everything You Know (about Parallel Programming) Is Wrong!: A Wild Screed about the Future invited speakerDavid Ungar, IBM Research, USA In the 1970?s, researchers at Xerox PARC gave themselves a glimpse of the future by building computers that, although wildly impractical at the time, let them experience plentiful fast cycles and big memories. PARC researchers invented Smalltalk, and the freedom afforded by such a dynamic, yet safe, language, led them to create a new experience of computing, which has become quite mainstream today. In the end of the first decade of the new century, chips such as Tilera?s can give us a glimpse of a future in which manycore microprocessors will become commonplace: every (non-hand-held) computer?s CPU chip will contain 1,000 fairly homogeneous cores. Such a system will not be programmed like the cloud, or even a cluster because communication will be much faster relative to computation. Nor will it be programmed like today?s multicore processors because the illusion of instant memory coherency will have been dispelled by both the physical limitations imposed by the 1,000-way fan-in to the memory system, and the comparatively long physical lengths of the inter- vs. intra-core connections. In the 1980?s we changed our model of computation from static to dynamic, and when this future arrives we will have to change our model of computation yet again. If we cannot skirt Amdahl?s Law, the last 900 cores will do us no good whatsoever. What does this mean? We cannot afford even tiny amounts of serialization. Locks?! Even lock-free algorithms will not be parallel enough. They rely on instructions that require communication and synchronization between cores? caches. Just as we learned to embrace languages without static type checking, and with the ability to shoot ourselves in the foot, we will need to embrace a style of programming without any synchronization whatsoever. In our Renaissance project at IBM, Brussels, and Portland State, (http://soft.vub.ac.be/~smarr/renaissance/) we are investigating what we call ?anti-lock,? ?race-and-repair,? or ?end-to-end nondeterministic? computing. As part of this effort, we have build a Smalltalk system that runs on the 64-core Tilera chip, and have experimented with dynamic languages atop this system. When we give up synchronization, we of necessity give up determinism. There seems to be a fundamental tradeoff between determinism and performance, just as there once seemed to be a tradeoff between static checking and performance. The obstacle we shall have to overcome, if we are to successfully program manycore systems, is our cherished assumption that we write programs that always get the exactly right answers. This assumption is deeply embedded in how we think about programming. The folks who build web search engines already understand, but for the rest of us, to quote Firesign Theatre: Everything You Know Is Wrong! David Ungar is an out-of-the-box thinker who enjoys the challenge of building computer software systems that work like magic and fit a user's mind like a glove. He received the 2009 Dahl-Nygaard award for outstanding career contributions in the field of object-orientation, and was honored as an ACM Fellow in 2010. Three of his papers have been honored by the Association for Computing Machinery for lasting impact over ten to twenty-four years: for the design of the prototype-based Self language, dynamic optimization techniques, and the application of cartoon animation ideas to user interfaces. He enjoys a position at IBM Research, where he is taking on a new challenge: investigating how application programmers can exploit manycore systems, and testing those ideas to see if they can help scale up analytics. [NOTE] this session is organized as a joint event with the AGERE! workshop _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From deadline at eadline.org Thu Mar 29 07:58:21 2012 From: deadline at eadline.org (Douglas Eadline) Date: Thu, 29 Mar 2012 07:58:21 -0400 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <20120327151551.GI17245@leitl.org> References: <20120327151551.GI17245@leitl.org> Message-ID: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> I am glad some one is talking about this. I have wondered about this myself, but never had a chance to look into it. http://www.isgtw.org/feature/does-computation-threaten-scientific-method -- Doug -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From diep at xs4all.nl Thu Mar 29 08:39:31 2012 From: diep at xs4all.nl (Vincent Diepeveen) Date: Thu, 29 Mar 2012 14:39:31 +0200 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: <711D3DE6-3858-45BA-B65B-7F8D04FA7AA8@xs4all.nl> Why don't you ask Robert G Brown to write something about it? It seems like a problem at universities, not so much a problem in companies, which can't or do not want to publish their codes anyway. On Mar 29, 2012, at 1:58 PM, Douglas Eadline wrote: > > > I am glad some one is talking about this. I have wondered > about this myself, but never had a chance to look into it. > > > http://www.isgtw.org/feature/does-computation-threaten-scientific- > method > > -- > Doug > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From james.p.lux at jpl.nasa.gov Thu Mar 29 10:24:38 2012 From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C)) Date: Thu, 29 Mar 2012 07:24:38 -0700 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: It is an interesting problem, and one relevant to HPC, just because HPC tends to be used in applications where errors can have big effects (although one could argue that a bad 1 page spreadsheet in the hands of a Fortune 10 CEO might result in a bad decision that has huge impacts..) With respect to "the scientific method", is it really different than say, a very complex or esoteric math proof. The SM relies on (attempted) replication for validation and in some ways, it's more a funding thing (that is, if your results depend on 100 work years of software development, replicating requires investing 100 work years). "big science" has always had that problem: Do you build a copy of the LHC? And, given the small number of people actually doing it, would a replication really be an independent replication. Odds are, some of the same developers would wind up on the new project, because of the limited pool of people who really know, say, numerical weather modeling. (again, this is partly funding.. If there were public investments of trillions of dollars in numerical weather modeling, there would be a veritable army of numerical weather modelers emerging from the halls of academe from which to choose for your replication effort. We don't, and there aren't.) And one has to be very careful about measures of defect density (usually given as defects/KLOC). Not only are there the "what's a LOC" questions, but different defects have different consequences, and the reporting of defects varies as a result. The LOC question is particularly an issue with the use of autogenerated code, although I tend to think this is no different than a compiler. Do you count source lines or machine instructions? If you make the (big) leap that the compiler is validated, you're working on the assumption that there's some underlying defect rate per "human operation". At JPL, we have a defect density of .1 to 4 defects/KSLOC spanning development test, system test, and operations. Mostly, we're in the less than 0.5 defects/KSLOC in operational use. Comparing with DoD data, it's in the same ballpark (DoD reports 0.2 to 0.6). But this is for "flight software" (that which runs on a computer in the spacecraft) which has a much more rigorous development process, over a shorter time span, than a lot of what the article was talking about. The data is somewhat skewed by the observation that until very recently, flight code is small because of the limited processor and memory space. Our productivity is gradually increasing, but not by leaps and bounds. We generated about 100 lines of code per work-month in the early 80s and about 50% more right now, and that's at the low end of the 1-5 line of code/work hour you see bandied about. I would attribute the low rate to the large amount of oversight and rigor. The shuttle software at $500M/500KSLOC is an interesting thing. Let's say that was mostly done in the early 80s, when a high end work hour cost about $50 (= 50K/yr salary+ overheads). $500M is then 10M work hours, so they achieved the spectacularly low productivity of 0.05 SLOC/hour. The shuttle software is widely known to have been VERY low productivity. What's also interesting is that it's not clear that they achieved a commensurate reduction of defect rate (that is, is the defect rate 20 times lower?). Nancy Leveson, among others, has some papers talking about this. I suspect that most of the software being described in the article doesn't have anywhere near the process rigor, nor does it have the inspectability, of the flight software at JPL. In general, I think it's probably tested by comparing results with expectation or with actual data, and the code is adjusted to make the model output match the observed data. Whether the code adjustments are actually "model parameters" or "unfounded logic changes" is sort of the question. And, of course, the "software" is really a conglomeration of libraries, other people's programs, etc. There are pieces of the puzzle that are no doubt rigorously verified: For instance not only is the source code for the Numerical Electromagnetics Code, NEC, published, but there's hundreds of pages of theory of operation documentation, as well as lots of published validation studies comparing against theoretical calculation and empirical measurements. I suspect that something like LAPACK is pretty well validated as well. But I would imagine that there is little *published* information on the validation of the overall assemblage. I think a lot of people just assume the library or popular tool "just works" and probably don't look at analyzing "well, how would we know if NEC or LAPACK was wrong". I'll speculate, too, that while a given research effort and the scientists attached to it may have a long duration (and, so, retain some corporate knowledge of the software architecture, design, and validation), there is more turnover on the software developers, and knowledge retention/transfer probably isn't all that good. An interesting observation from JPL is that the vast majority of the 1000+ people developing software don't have any training in software development processes/engineering/etc, beyond practical experience (which is a form of training). This is, in part, because most software development is quite diffuse. Over half the software development projects are smaller than 2 work years, and software development is just part of the overall bigger task. That is, the scientist or engineer is doing software development as a tool to do their job, not as a job in itself. I suspect a similar phenomenon is true in academe. How much "I'll just whip up this Matlab module to analyze the data" winds up being the basis for "production code". On 3/29/12 4:58 AM, "Douglas Eadline" wrote: > > >I am glad some one is talking about this. I have wondered >about this myself, but never had a chance to look into it. > > >http://www.isgtw.org/feature/does-computation-threaten-scientific-method > >-- >Doug > >-- >This message has been scanned for viruses and >dangerous content by MailScanner, and is >believed to be clean. > >_______________________________________________ >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From brian.dobbins at yale.edu Thu Mar 29 12:22:49 2012 From: brian.dobbins at yale.edu (Brian Dobbins) Date: Thu, 29 Mar 2012 12:22:49 -0400 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: <4F748C59.4000700@yale.edu> To borrow from an old joke, I'd say the short answer is "No.", and the long answer? "Nooooooooooo." Reproducibility is an interesting issue - on the surface, it seems like a binary thing: something is or is not reproducible. In reality, though, things are almost never duplicated exactly, and there exists some fuzzy threshold at which point things are considered good enough to be a reproduction. I can go down to a local store and buy a print of the Mona Lisa and, to me, it might be a really great reproduction, yet even writing that sentence has some art critic screaming in agony. Similarly, in computing, if I run some model on two different systems and get two different results, that can either be indicative of a potential issue or it can be completely fine, because those differences are below a certain threshold and thus the runs were, in scientific terms, 'reproducible' with respect to each other. On a small scale (meaning a lab, code or project), this is a key issue - I've seen grad students and faculty alike be dismayed by trivial differences, and when this happens, more often than not the mentality is, "My first results are correct - make this code give them back to me", without understanding that the later, different results are quite possibly equally valid, and possibly more so. Back in the early Beowulf days, I remember switching some codes from an RS/6000 platform to an x86-based one, and the internal precision of the x86 FPU was 80-bits, not 64, so sequences of FP math could produce small differences unless this option was specifically disabled via compiler switches. Which a lot of people did, not because the situation was carefully considered, but because with it on, it gave 'wrong' results. Another example would be an algorithm that was orders of magnitude faster than one previously in use, but wasn't adopted because ultimately the results were different. The catch here? Reordering the input data while still using the original algorithm gave similarly different answers - the nature of the code was that single runs were useless, and ensemble runs were a necessity. Ultimately, the issues here come down to the common perception of computers - "They give you THE answer!" - versus the reality of computers - "They give you AN answer!", with the latter requiring additional effort to provide some error margin or statistical analysis of results. This happens in certain computational disciplines far more often than others. On the larger scales - whether reproducibility is an issue in scientific /fields/ - again, I'd say the answer is no. The scientific method is resilient, but it never made any claims to be 'fast'. Would it speed things up to have researchers publish their code and data? Probably. Or, rather, it'd certainly speed up the verification of results, but it might also inhibit new approaches to doing the same thing. Some people here might recall Michael Abrash's "Graphics Programming Black Book", which had a wonderful passage where about a word-counting program. It focused explicitly on performance tuning, with the key lesson being that nobody thought there was a better way of doing the task... until someone showed there was. And that lead to a flurry of new ideas. Similarly, having software that does things in a certain way often convinces people that that is THE way of doing things, whereas if they knew it could be done but not how, newer methods might develop. There's probably some happy medium here, since having so many different codes, mostly with a single author who isn't a software developer by training, seems less efficient and flexible than a large code with good documentation, a good community and the ability to use many of those methods previously in the one-off codes. In other words, we can probably do better, but science itself isn't threatened by the inefficiency in verifying results, or even bad results - in the absolute worst case, with incorrect ideas being laid down as the foundation for new science and no checking done on them, progress will happen until it can't... at which point people will backtrack until the discover the underlying principle they thought was correct and will fix it. The scientific method is a bit like a game of chutes and ladders in this respect. Ultimately, in a lot of ways, I think computational science has it better than other disciplines. There was news earlier this week [1] about problems reproducing some early-stage cancer research - specifically, Amgen tried to reproduce 53 'landmark' conclusions, and were only able to do so with 11% of them. Again, that's OK - it will correct itself, albeit in slow fashion, but what's interesting here is that these sorts of experiments, especially those involving mice (and often other wet-lab methods), don't have something like Moore's Law making them more accessible over time. To reproduce a study involving the immune system of a mouse, I need mice. And I need to wait the proper number of days. Yet with computational science, what today may take a top end supercomputer can probably be done in a few years on a departmental cluster. A few years after that? Maybe a workstation. In our field, data doesn't really change or degrade over time and the ability to analyze it in countless different ways becomes more and more accessible all the time. In short (hah, nothing about this was short!), can we do better with our scientific approaches? Probably. But is the scientific method threatened by computation? Nooooooooo. :-) That's my two cents, - Brian [1] http://vitals.msnbc.msn.com/_news/2012/03/28/10905933-rethinking-how-we-confront-cancer-bad-science-and-risk-reduction Or, more directly (if you have access to Nature) : http://www.nature.com/nature/journal/v483/n7391/full/483531a.html (PS. The one thing which can threaten science is a lack of education - it can decrease the signal-to-noise ratio of 'good' science, amongst other things. That's a whole essay in itself.) (PPS. This was a long answer, and yet not nearly long enough... but I didn't want to be de-invited from future Beowulf Bashes by writing even more!) On 3/29/2012 7:58 AM, Douglas Eadline wrote: > > I am glad some one is talking about this. I have wondered > about this myself, but never had a chance to look into it. > > > http://www.isgtw.org/feature/does-computation-threaten-scientific-method > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From samuel at unimelb.edu.au Thu Mar 29 20:27:56 2012 From: samuel at unimelb.edu.au (Christopher Samuel) Date: Fri, 30 Mar 2012 11:27:56 +1100 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> Message-ID: <4F74FE0C.9050607@unimelb.edu.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 29/03/12 22:58, Douglas Eadline wrote: > I am glad some one is talking about this. I have wondered about > this myself, but never had a chance to look into it. IIRC this issue was discussed at CCGrid 2005 in Cardiff, but provenance and reproducibility are hard - we need to keep upgrading to get security fixes but we get other changes into the bargain. You could work around that by having jobs run in VM images that are set in stone, but then you will take a performance penalty (from a small amount to large amount, depending on the level of virtualisation you need). Then of course there's the issue of hardware errata, not to mention availability if you ran it on an Alphaserver SC in 2002.. cheers. Chris - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk90/gwACgkQO2KABBYQAh8o3QCfd3Q23fD09gYYanNm4ehxIVgM YI0An0JSC9iNfC5FMvk2FepSaAv9Xnrx =gG3x -----END PGP SIGNATURE----- _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From diep at xs4all.nl Thu Mar 29 20:45:44 2012 From: diep at xs4all.nl (Vincent Diepeveen) Date: Fri, 30 Mar 2012 02:45:44 +0200 Subject: [Beowulf] Does computation threaten the scientific method? In-Reply-To: <4F748C59.4000700@yale.edu> References: <20120327151551.GI17245@leitl.org> <7219832b87625990e515bb6f9ddb621d.squirrel@mail.eadline.org> <4F748C59.4000700@yale.edu> Message-ID: Brian - using medical examples probably is not a good idea in this discussion. To introduce anew medicine is something of a 3 phase introduction to get it from acceptance to the market; getting allowance at a political level is the hardest. Now a big problem is simply that you can already get a 'go' there when you have a 95% confidence - or around 200 persons which react positive at whatever you were doing at the SHORT TERM. That's fairly little actually, especially for the trillion turnover that nowadays psychological medicines alltogether make. So to speak that's science from 1946. The computation risk there is a different one - namely that in short term cocain always works - that's what kids get now basically, that is, the new medicines for examle for ADHD look frightening much like cocain and in long term also have those same side effects. Yet they get prescribed massively, also for kids that do not need it. The real problem is not so much the computation, but more the government rules of allowing something based upon 200 cases short term. If you sell already a product for billions, even later corrections onto what you do, will not soon make it into mainstream, so it will get used for years and years until someone says STOP. That STOP is very tough to give if they produced synthetical cocain and in short term got through the 200 cases that reacted positive. This is more of a government problem of course - they are still 65+ years behind. So to mix lapack statements with pharmaceuticals is not a good idea i'd say. If you suck everywhere as a scientist, you can still go produce a new medicine and sell it worldwide. Government should really undertake action there. Also directly modify that DSM classification once again so that less people get diagnosed, as then it's not always the doctors that want to put medicines into someone, but for teachers it's very great to do so and one of the reports, though i don't know whether that's accurate, seems so though, said 33% more children now are diagnosed thanks to a few changes in the classification system! As usual of course that was taken over from USA, as each individual small nation in Europe is too small to do things like that on itself, wit hsometimes devastating consequences. Vincent On Mar 29, 2012, at 6:22 PM, Brian Dobbins wrote: > > To borrow from an old joke, I'd say the short answer is "No.", and > the long answer? "Nooooooooooo." > > Reproducibility is an interesting issue - on the surface, it seems > like a binary thing: something is or is not reproducible. In > reality, though, things are almost never duplicated exactly, and > there exists some fuzzy threshold at which point things are > considered good enough to be a reproduction. I can go down to a > local store and buy a print of the Mona Lisa and, to me, it might > be a really great reproduction, yet even writing that sentence has > some art critic screaming in agony. Similarly, in computing, if I > run some model on two different systems and get two different > results, that can either be indicative of a potential issue or it > can be completely fine, because those differences are below a > certain threshold and thus the runs were, in scientific terms, > 'reproducible' with respect to each other. > > On a small scale (meaning a lab, code or project), this is a key > issue - I've seen grad students and faculty alike be dismayed by > trivial differences, and when this happens, more often than not the > mentality is, "My first results are correct - make this code give > them back to me", without understanding that the later, different > results are quite possibly equally valid, and possibly more so. > Back in the early Beowulf days, I remember switching some codes > from an RS/6000 platform to an x86-based one, and the internal > precision of the x86 FPU was 80-bits, not 64, so sequences of FP > math could produce small differences unless this option was > specifically disabled via compiler switches. Which a lot of people > did, not because the situation was carefully considered, but > because with it on, it gave 'wrong' results. Another example > would be an algorithm that was orders of magnitude faster than one > previously in use, but wasn't adopted because ultimately the > results were different. The catch here? Reordering the input data > while still using the original algorithm gave similarly different > answers - the nature of the code was that single runs were useless, > and ensemble runs were a necessity. > > Ultimately, the issues here come down to the common perception of > computers - "They give you THE answer!" - versus the reality of > computers - "They give you AN answer!", with the latter requiring > additional effort to provide some error margin or statistical > analysis of results. This happens in certain computational > disciplines far more often than others. > > On the larger scales - whether reproducibility is an issue in > scientific fields - again, I'd say the answer is no. The > scientific method is resilient, but it never made any claims to be > 'fast'. Would it speed things up to have researchers publish their > code and data? Probably. Or, rather, it'd certainly speed up the > verification of results, but it might also inhibit new approaches > to doing the same thing. Some people here might recall Michael > Abrash's "Graphics Programming Black Book", which had a wonderful > passage where about a word-counting program. It focused explicitly > on performance tuning, with the key lesson being that nobody > thought there was a better way of doing the task... until someone > showed there was. And that lead to a flurry of new ideas. > Similarly, having software that does things in a certain way often > convinces people that that is THE way of doing things, whereas if > they knew it could be done but not how, newer methods might > develop. There's probably some happy medium here, since having so > many different codes, mostly with a single author who isn't a > software developer by training, seems less efficient and flexible > than a large code with good documentation, a good community and the > ability to use many of those methods previously in the one-off codes. > > In other words, we can probably do better, but science itself isn't > threatened by the inefficiency in verifying results, or even bad > results - in the absolute worst case, with incorrect ideas being > laid down as the foundation for new science and no checking done on > them, progress will happen until it can't... at which point people > will backtrack until the discover the underlying principle they > thought was correct and will fix it. The scientific method is a > bit like a game of chutes and ladders in this respect. > > Ultimately, in a lot of ways, I think computational science has it > better than other disciplines. There was news earlier this week > [1] about problems reproducing some early-stage cancer research - > specifically, Amgen tried to reproduce 53 'landmark' conclusions, > and were only able to do so with 11% of them. Again, that's OK - > it will correct itself, albeit in slow fashion, but what's > interesting here is that these sorts of experiments, especially > those involving mice (and often other wet-lab methods), don't have > something like Moore's Law making them more accessible over time. > To reproduce a study involving the immune system of a mouse, I need > mice. And I need to wait the proper number of days. Yet with > computational science, what today may take a top end supercomputer > can probably be done in a few years on a departmental cluster. A > few years after that? Maybe a workstation. In our field, data > doesn't really change or degrade over time and the ability to > analyze it in countless different ways becomes more and more > accessible all the time. > > In short (hah, nothing about this was short!), can we do better > with our scientific approaches? Probably. But is the scientific > method threatened by computation? Nooooooooo. :-) > > That's my two cents, > - Brian > > [1] http://vitals.msnbc.msn.com/_news/2012/03/28/10905933- > rethinking-how-we-confront-cancer-bad-science-and-risk-reduction > Or, more directly (if you have access to Nature) : http:// > www.nature.com/nature/journal/v483/n7391/full/483531a.html > > (PS. The one thing which can threaten science is a lack of > education - it can decrease the signal-to-noise ratio of 'good' > science, amongst other things. That's a whole essay in itself.) > (PPS. This was a long answer, and yet not nearly long enough... > but I didn't want to be de-invited from future Beowulf Bashes by > writing even more!) > > > On 3/29/2012 7:58 AM, Douglas Eadline wrote: >> >> I am glad some one is talking about this. I have wondered about >> this myself, but never had a chance to look into it. http:// >> www.isgtw.org/feature/does-computation-threaten-scientific-method > > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin > Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.