<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Arial; font-size: 12pt; color: #000000'><br>Mark Hahn wrote:<br><br>>> Doesn't this assume worst case all-to-all type communication<br>>> patterns.<br>><br>>I'm assuming random point-to-point communication, actually.<div><br></div><div>A sub-case of all-to-all (possibly all-to-all). So you are assuming</div><div>random point-to-point is a common pattern in HPC ... mmm ... I</div><div>would call it a worse case pattern, something more typical of </div><div>graph searching codes like they run at the NSA. Sure a high</div><div>radix switch (or better yet a global memory address space, Cray</div><div>X1E) is good and designed for this worst-case, but not sure this</div><div>is the common case data reference pattern in HPC ... if it were</div><div>they would be selling more global memory systems at Cray and</div><div>SGI (not just to the NSA).</div><div><br></div><div>There you might also want a machine like the Cray XMT where</div><div>the memory is flat and stalled threads can be switched out for</div><div>another thread. </div><div><br>>> If you are just trading ghost cell data with your neighbors<br>>> and you have placed your job smartly on the torus the fan out<br>>> advantage mentioned is irrelevant. No?<br>><br>>if your comms are nearest-neighbor, then yes, a nearest-neighbor</div><div>>fabric is your friend ;)</div><div><br></div><div>I think that if you look at the HPC space globally there is still a lot</div><div>of locality that you can rely on. Familiar with the "7 dwarves" paper</div><div>from Berkeley? <br><br>>how often does that actually happen? to work out so neatly would <br>>preclude, for instance, adaptive meshes, right? it seems like mostly<br>>I see jobs with no obvious regular structure to their communication.<br></div><div><br></div><div>Really ... must be doing a lot of turbulent flow simulations with shedding</div><div>vortices, crash simulations with self-penetrating meshes ... tough stuff</div><div>for your average cluster or even your above average cluster. Even AMR</div><div>codes usually attempt to discover new neighbors and localize them. </div><div><br></div><div>Not disrespecting switches, but they are in a sense designed for worse</div><div>case scenarios (the design asserts that "there are no neighborhoods") </div><div>... a torus design appeals to the middle ground were locality is not banished. </div><div><br></div><div>rbw</div></div></body></html>