<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Arial; font-size: 12pt; color: #000000'><br><br><br><br>----- Forwarded Message -----<br>From: "richard walsh" <richard.walsh@comcast.net><br>To: "Craig Tierney" <Craig.Tierney@noaa.gov><br>Sent: Thursday, April 8, 2010 5:19:14 PM GMT -05:00 US/Canada Eastern<br>Subject: Re: [Beowulf] QDR InfiniBand interconnect architectures ... approaches ...<br><br><div style="font-family: Arial; font-size: 12pt; color: #000000"><br>On Thursday, April 8, 2010 2:42:49 PM Craig Tierney wrote:<div><br></div><div>>We have been telling our vendors to design a multi-level tree using<br>>36 port switches that provides approximately 70% bisection bandwidth.<br>>On a 448 node Nehalem cluster, this has worked well (weather, hurricane, and<br>>some climate modeling). This design (15 up/21 down) allows us to<br>>scale the system to 714 nodes.<br><br></div><div>Hey Craig,</div><div><br></div><div>Thanks for the information. So are you driven mostly by the need</div><div>for incremental expandability with this design, or do you disagree</div><div>with Greg and think that the cost is as good or better than a chassis</div><div>based approach? What about reliability (assuming the vendor is</div><div>putting it together for you) and maintenance headaches? Not so</div><div>bad? What kind of cabling are you using? </div><div><br></div><div>Trying to do the math on the design ... for the 448 nodes you would</div><div>need 22 switches for the first tier (22 * 21 = 462 down). That gives</div><div>you (15 * 22 = 330 uplinks), so you need at least 10 switches in the</div><div>second tier (10 * 36 = 360) which leaves you some spare ports for</div><div>other things. Am I getting this right? Could you lay out the design</div><div>in a bit more detail? Did you consider building things from medium</div><div>size switches (say 108 port models)? Are you paying a premium </div><div>for incremental expandability or not? How many ports are you using</div><div>for your file server? </div><div><br></div><div>Our system is likely to come in at 192 nodes with some additional</div><div>ports for file server connection. I would like to compare the cost</div><div>of a 216 port switch to your 15/21 design using 36 port switches.</div><div><br></div><div>Thanks much,</div><div><br></div><div>rbw</div><div><br><br><br><br><br>> Thanks, <br>> <br>> <br>> rbw <br>> <br>> <br>> ------------------------------------------------------------------------<br>> <br>> _______________________________________________<br>> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing<br>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf<br><br></div></div></div></body></html>