[Beowulf] Accelerator for data compressing

Mark Hahn hahn at mcmaster.ca
Thu Oct 2 22:31:48 EDT 2008

> Currently I generate nearly one TB data every few days and I need to pass it

Bill's right - 6 MB/s is really not much to ask from even a complex WAN.
I think the first thing you should do is find the bottleneck.  to me it
sounds like you have a sort of ropey path with a 100 Mbps hop somewhere.

> thinking about compressing it (most tiff format image data) as much as I can

tiff is a fairly generic container that can hold anything from a horrible
uncompressed 4-byte-per-pixel to jpeg or rle.  looking at the format you're
really using would be wise.  I'm guessing that if you transcode to png,
you'll get better compression than gzip/etc.  dictionary-based compression
is fundamentally inappropriate for most non-text data - not images, not 
double-precision dumps of physical simulations, etc.  png is quite a lot 
smarter about most kinds of images than older formats, and can be lossy or

hardware compression would be a serious mistake unless you've already 
pursued these routes.  specialized hardware is a very short-term and 
quite narrow value proposition.  I would always prefer to improve the 

> The information transmitted in this electronic communication is intended only

uh, email is publication.

regards, mark hahn.
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list