[Beowulf] OT: recoverable optical media archive format?

David Mathog mathog at caltech.edu
Tue Jun 8 13:44:55 EDT 2010


This is off topic so I will try to keep it short:  is there an
"archival" format for large binary files which contains enough error
correction to that all original data may be recovered even if there is a
little data loss in the storage media?  

For my purposes these are disk images, sometimes .tar.gz, other times
gunzip -c of dd dumps of whole partitions which have been "cleared" by
filling the empty space with one big file full of zero, and then that
file deleted.  I'm thinking of putting this information on DVD's (only
need to keep it for a few years at a time) but I don't trust that media
not to lose a sector here or there - having watched far too many
scratched DVD movies with playback problems.

Unlike an SDLT with a bad section, the good parts of a DVD are still
readable when there is a bad block (using dd or ddrescue) but of course
even a single missing chunk makes it impossible to decompress a .gz file
correctly.  So what I'm looking for is some sort of .img.gz.ecc format,
where the .ecc puts in enough redundant information to recover the
underlying img.gz even when sectors or data are missing.   If no such
tool/format exists then two copies should be enough to recover all of an
.img.gz so long as the same data wasn't lost on both media, and if bad
DVD sectors always come back as "failed read", never ever showing up as
a good read but actually containing bad data.  Perhaps the frame
checksum on a DVD is enough to guarantee that?

Thanks,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



More information about the Beowulf mailing list