[Beowulf] Accelerator for data compressing
hahn at mcmaster.ca
Thu Oct 2 22:31:48 EDT 2008
> Currently I generate nearly one TB data every few days and I need to pass it
Bill's right - 6 MB/s is really not much to ask from even a complex WAN.
I think the first thing you should do is find the bottleneck. to me it
sounds like you have a sort of ropey path with a 100 Mbps hop somewhere.
> thinking about compressing it (most tiff format image data) as much as I can
tiff is a fairly generic container that can hold anything from a horrible
uncompressed 4-byte-per-pixel to jpeg or rle. looking at the format you're
really using would be wise. I'm guessing that if you transcode to png,
you'll get better compression than gzip/etc. dictionary-based compression
is fundamentally inappropriate for most non-text data - not images, not
double-precision dumps of physical simulations, etc. png is quite a lot
smarter about most kinds of images than older formats, and can be lossy or
hardware compression would be a serious mistake unless you've already
pursued these routes. specialized hardware is a very short-term and
quite narrow value proposition. I would always prefer to improve the
> The information transmitted in this electronic communication is intended only
uh, email is publication.
regards, mark hahn.
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf