<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>FW: Node cloning</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Did you all hear that the English actually did this first. The hostname of the first node cloned on their server farm was "dolly", but it keeps crashing due to bad code transmitted in the cloning process . . .</FONT></P>
<P><FONT SIZE=2>Sorry . . . it's Friday and I couldn't resist.</FONT>
</P>
<P><FONT SIZE=2>Richard Schilling</FONT>
<BR><FONT SIZE=2>Web Integration Programmer/Webmaster</FONT>
<BR><FONT SIZE=2>phone: 360.856.7129</FONT>
<BR><FONT SIZE=2>fax: 360.856.7166</FONT>
<BR><FONT SIZE=2>URL: <A HREF="http://www.affiliatedhealth.org" TARGET="_blank">http://www.affiliatedhealth.org</A></FONT>
</P>
<P><FONT SIZE=2>Affiliated Health Services</FONT>
<BR><FONT SIZE=2>Information Systems</FONT>
<BR><FONT SIZE=2>1971 Highway 20</FONT>
<BR><FONT SIZE=2>Mount Vernon, WA USA</FONT>
</P>
<BR>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Richard C Ferri [<A HREF="mailto:rcferri@us.ibm.com">mailto:rcferri@us.ibm.com</A>] </FONT>
<BR><FONT SIZE=2>Sent: Thursday, April 05, 2001 6:41 PM</FONT>
<BR><FONT SIZE=2>To: Robert G. Brown</FONT>
<BR><FONT SIZE=2>Cc: Giovanni Scalmani; beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>Subject: Re: Node cloning</FONT>
</P>
<BR>
<BR>
<BR>
<P><FONT SIZE=2> Since cloning continues to be a fertile topic, I'll jump right in...</FONT>
<BR><FONT SIZE=2>if you're not interested in node installation or cloning, skip this note...</FONT>
</P>
<P><FONT SIZE=2> I feel that the node installation problem using the NFS root/tftp/pxe</FONT>
<BR><FONT SIZE=2>boot approach has been solved by LUI (oss.software.ibm.com/lui) and others.</FONT>
<BR><FONT SIZE=2>I don't see why anyone would need to roll their own solution. When one</FONT>
<BR><FONT SIZE=2>defines a node to LUI, LUI creates a custom remote NFS root and updates</FONT>
<BR><FONT SIZE=2>dhcpd.conf or /etc/bootptab with an entry for the node. One chooses a set</FONT>
<BR><FONT SIZE=2>of resources to install on the node, and creates a disk partition table.</FONT>
<BR><FONT SIZE=2>Resources are lists of RPMs or tar files, custom kernels, individual files</FONT>
<BR><FONT SIZE=2>(/etc/hosts, /etc/resolv.conf, /etc/shadow would be good examples). You</FONT>
<BR><FONT SIZE=2>either pxe boot the node, or boot from diskette using etherboot technology.</FONT>
<BR><FONT SIZE=2>The node boots, gets a custom boot kernel over the network via tftp, and</FONT>
<BR><FONT SIZE=2>transfers control. The kernel mounts the remote root, and reads the list</FONT>
<BR><FONT SIZE=2>of allocated resources. Based on the resources, the node partitions the</FONT>
<BR><FONT SIZE=2>harddrive,creates FSs, installs RPMS or tar files, copies any specified</FONT>
<BR><FONT SIZE=2>files, installs a custom kernel, and so on. The software configures the</FONT>
<BR><FONT SIZE=2>eth0 device based on the IP info for that particular node, assigns a</FONT>
<BR><FONT SIZE=2>default route and runs lilo to make the node ready to boot. If you allow</FONT>
<BR><FONT SIZE=2>rsh, LUI will also remove the /etc/bootptab entry, and optionally reboot</FONT>
<BR><FONT SIZE=2>the node. It keeps a log of all activity during install.</FONT>
</P>
<P><FONT SIZE=2> The goal of the LUI project is to install any distro on any</FONT>
<BR><FONT SIZE=2>architecture (ia-32, itanium, PowerPC and alpha). So far RedHat and ia-32</FONT>
<BR><FONT SIZE=2>are supported, but Suse and PowerPC are in test but not ready for prime</FONT>
<BR><FONT SIZE=2>time. It's an open source project, and open to contributors. Since LUI is</FONT>
<BR><FONT SIZE=2>resource based, and resources are reusable, it's perfect for heterogenous</FONT>
<BR><FONT SIZE=2>clusters, clusters where nodes have different requirements. Many people</FONT>
<BR><FONT SIZE=2>have said that the NFS/tftp/pxe solution doesn't scale and should be</FONT>
<BR><FONT SIZE=2>abandoned. Well, users have installed 80-way clusters using LUI, and while</FONT>
<BR><FONT SIZE=2>that's not huge, it's not dog meat either.</FONT>
</P>
<P><FONT SIZE=2> Simple cloning, basically copying an image from one golden node to</FONT>
<BR><FONT SIZE=2>another, changing some rudimentary info along the way, is performed today</FONT>
<BR><FONT SIZE=2>by SystemImager, based on rsync technology. rysync is superior to simple</FONT>
<BR><FONT SIZE=2>copy in that you can easily exclude files or directories (/var for example)</FONT>
<BR><FONT SIZE=2>and can be used for maintainence as well. rsync does intelligent copying</FONT>
<BR><FONT SIZE=2>for maintenance -- it copies only files that are different on the source</FONT>
<BR><FONT SIZE=2>and target systems, and copies only the parts of the file that have</FONT>
<BR><FONT SIZE=2>changed. SystemImager and rsync are good solutions when the nodes in your</FONT>
<BR><FONT SIZE=2>cluster are basically the same, except for IP info and disk size.</FONT>
</P>
<P><FONT SIZE=2> Then there's kickstart. Well, it's ok if you do RedHat.</FONT>
</P>
<P><FONT SIZE=2> I think the real burning issue is not how to install nodes, but</FONT>
<BR><FONT SIZE=2>*whether* to install nodes or embrace the beowulf 2 technology from SCYLD.</FONT>
<BR><FONT SIZE=2>I think SCYLD is close to becoming the linux beowulf appliance, a turnkey</FONT>
<BR><FONT SIZE=2>commodity supercomputer. It will be interesting to see how many new</FONT>
<BR><FONT SIZE=2>clusters adopt traditional beowulf solutions, and how many adopt beowulf</FONT>
<BR><FONT SIZE=2>2...</FONT>
</P>
<P><FONT SIZE=2>the view from here, Rich</FONT>
</P>
<P><FONT SIZE=2>Richard Ferri</FONT>
<BR><FONT SIZE=2>IBM Linux Technology Center</FONT>
<BR><FONT SIZE=2>rcferri@us.ibm.com</FONT>
<BR><FONT SIZE=2>845.433.7920</FONT>
</P>
<P><FONT SIZE=2>"Robert G. Brown" <rgb@phy.duke.edu>@beowulf.org on 04/05/2001 06:47:46 PM</FONT>
</P>
<P><FONT SIZE=2>Sent by: beowulf-admin@beowulf.org</FONT>
</P>
<BR>
<P><FONT SIZE=2>To: Giovanni Scalmani <Giovanni@lsdm.dichi.unina.it></FONT>
<BR><FONT SIZE=2>cc: <beowulf@beowulf.org></FONT>
<BR><FONT SIZE=2>Subject: Re: Node cloning</FONT>
</P>
<BR>
<BR>
<P><FONT SIZE=2>On Thu, 5 Apr 2001, Giovanni Scalmani wrote:</FONT>
</P>
<P><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> Hi!</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> On Thu, 5 Apr 2001, Oscar Roberto [iso-8859-1] López Bonilla wrote:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> > And then use the command (this will take long, so you can do it</FONT>
<BR><FONT SIZE=2>overnight)</FONT>
<BR><FONT SIZE=2>> > cp /dev/hda /dev/hdb ; cp /dev/hda /dev/hdc ; cp /dev/hda</FONT>
<BR><FONT SIZE=2>/dev/hdd</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> I also did this way for my cluster, BUT I've experienced instability</FONT>
<BR><FONT SIZE=2>> for some nodes (3/4 over 20). My guess was that "cp /dev/hda /dev/hdb"</FONT>
<BR><FONT SIZE=2>> copied also the bad-blocks list of hda onto hdb and this looks wrong</FONT>
<BR><FONT SIZE=2>> to me. So I partitioned and made the filesystems on each node and then</FONT>
<BR><FONT SIZE=2>> cloned the content of each filesystem. Those nodes are now stable.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>> A question to the 'cp gurus' out there: is my guess correct about</FONT>
<BR><FONT SIZE=2>> the bad blocks list?</FONT>
</P>
<P><FONT SIZE=2>One of many possible problems, actually. This approach to cloning</FONT>
<BR><FONT SIZE=2>makes me shudder -- things like the devices in /dev generally have to</FONT>
<BR><FONT SIZE=2>built, not copied, there are issues with the boot blocks and bad block</FONT>
<BR><FONT SIZE=2>lists and the bad blocks themselves on both target and host. raw</FONT>
<BR><FONT SIZE=2>devices are dangerous things to use as if they were flatfiles.</FONT>
</P>
<P><FONT SIZE=2>Tarpipes (with tar configured the same way it would be for a</FONT>
<BR><FONT SIZE=2>backup|restore but writing/reading stdout) are a much safer way to</FONT>
<BR><FONT SIZE=2>proceed. Or dump/restore pipes on systems that have it -- either one is</FONT>
<BR><FONT SIZE=2>equivalent to making a backup and restoring it onto the target disk.</FONT>
<BR><FONT SIZE=2>One reason I gave up cloning (after investing many months writing a</FONT>
<BR><FONT SIZE=2>first generation cloning tool for nodes (which booted a diskless</FONT>
<BR><FONT SIZE=2>configuration, formatted a local disk, and cloned itself onto the local</FONT>
<BR><FONT SIZE=2>disk) and started a second generation GUI-driven one) was that just</FONT>
<BR><FONT SIZE=2>cloning isn't enough. There is all sorts of stuff that needs to be done</FONT>
<BR><FONT SIZE=2>to the clones to give them a unique identity (even something as simple</FONT>
<BR><FONT SIZE=2>as their own ssh keys), one needs to rerun lilo, it requires that you</FONT>
<BR><FONT SIZE=2>keep one "pristine" host to use as the master to clone or you have the</FONT>
<BR><FONT SIZE=2>very host configuration creep you set out to avoid. Either way you end</FONT>
<BR><FONT SIZE=2>up inevitably having to upgrade all the nodes or install security or</FONT>
<BR><FONT SIZE=2>functionality updates.</FONT>
</P>
<P><FONT SIZE=2>These days there are just better ways (in my opinion) to proceed if your</FONT>
<BR><FONT SIZE=2>goal is simple installation and easy upgrade/update and low maintenance.</FONT>
<BR><FONT SIZE=2>Cloning is also very nearly an irreversible decision -- if you adopt</FONT>
<BR><FONT SIZE=2>clone methods it can get increasingly difficult to maintain your cluster</FONT>
<BR><FONT SIZE=2>without ALSO developing tools and methods that could just as easily have</FONT>
<BR><FONT SIZE=2>been used to install and clean things up post-install.</FONT>
</P>
<P><FONT SIZE=2>Even so, if you are going to clone, I think that the diskless->local</FONT>
<BR><FONT SIZE=2>clone is a very good way to proceed, because it facilitates</FONT>
<BR><FONT SIZE=2>reinstallation and emergency operation of a node even if a hard disk</FONT>
<BR><FONT SIZE=2>crashes (you can run it diskless while getting a replacement). It does</FONT>
<BR><FONT SIZE=2>require either a floppy drive ($15) or a PXE chip, but this is a pretty</FONT>
<BR><FONT SIZE=2>trivial investment per node.</FONT>
</P>
<P><FONT SIZE=2> rgb</FONT>
</P>
<P><FONT SIZE=2>--</FONT>
<BR><FONT SIZE=2>Robert G. Brown <A HREF="http://www.phy.duke.edu/~rgb/" TARGET="_blank">http://www.phy.duke.edu/~rgb/</A></FONT>
<BR><FONT SIZE=2>Duke University Dept. of Physics, Box 90305</FONT>
<BR><FONT SIZE=2>Durham, N.C. 27708-0305</FONT>
<BR><FONT SIZE=2>Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu</FONT>
</P>
<BR>
<BR>
<BR>
<P><FONT SIZE=2>_______________________________________________</FONT>
<BR><FONT SIZE=2>Beowulf mailing list, Beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>To change your subscription (digest mode or unsubscribe) visit</FONT>
<BR><FONT SIZE=2><A HREF="http://www.beowulf.org/mailman/listinfo/beowulf" TARGET="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</A></FONT>
</P>
<BR>
<BR>
<BR>
<P><FONT SIZE=2>_______________________________________________</FONT>
<BR><FONT SIZE=2>Beowulf mailing list, Beowulf@beowulf.org</FONT>
<BR><FONT SIZE=2>To change your subscription (digest mode or unsubscribe) visit <A HREF="http://www.beowulf.org/mailman/listinfo/beowulf" TARGET="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</A></FONT>
</P>
</BODY>
</HTML>