James.P.Lux at jpl.nasa.gov
Mon Jan 16 21:13:35 EST 2006
At 05:03 PM 1/16/2006, Joe Landman wrote:
>Craig Tierney wrote:
>>Mike Davis wrote:
>>>geeks out there, what happens when a folder has 50,000 files in it. Can
>>>you say SLOOOOOOOOOWWWW?
>>First, pick the right filesystem.
>>Second, rewrite your code so you don't have 50k+ files in one directory.
>>There must be some straightforward way to solve the problem if
>>you have too many files in one directory.
>Lots of the informatics codes were not written with such input (or
>database) scaling in mind. For them, 10-100 files in a directory isn't
>much of a problem. Its when you start to scale up that the bugs and
I think that unforeseen scaling problems are probably the single biggest
challenge with HPC. If it's not the plethora of files, or crippling
interprocessor comm needs, it's something like timing races and implied
barriers (in a brute force master doling out identical work packets to all
the slaces.. they all finish at the same time).
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit
James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf