[Beowulf] Remote console management
ballen at gravity.phys.uwm.edu
Fri Sep 23 00:33:34 EDT 2005
> We (my company) have set something like this up for one of our customers
> with the cluster they bought, where the admins can be (in theory) 1/2 world
> away and completely control the machines.
> It is quite possible to do this, but you have to be ready for the sticker
> shock on the KVM over IP. Lets address the remote power first.
> There are lots of power strips out there, many folks use APC units of one
> form or the other. We use NBB-1600 from WTI (http://www.wti.com/nbb16.htm)
> to handle power control. They work, quite well at that. The only issue is
> that you will need a few of them for full racks, and you would need to set
> the bios to power on on restoration of power. If you have an IPMI capable
> machine (Sun v20z), you plug in to the service processor via a managment
> network (make it separate from your main network, but routeable and visible
> from the WAN), and you can power cycle the machine. You get some nice
> diagnostic bits in there for free as well.
We don't need the power strips. The inexpensive IPMI cards let us to both
hard and soft power cycle.
> Now onto to the remote consoles. I am sure I am going to regret saying
> this, but ... I have seen many people purchase cyclades, and I cannot say I
> have ever seen them being used. One customer asked me what it was for, as
> their last cluster vendor required it as part of their package, and they (the
> customer) never used it. In part because it was never attached.
We WOULD attach them!
> IPMI gives you the ability to do a console over IPMI (text only). I am
> not sure how well this works in general.
The main issue is 'where is the console output buffered'. Look back at my
original posting. I want to see the same display that someone plugging in
a monitor would see. This means that at least the last 24*80 bytes have
to be buffered. And ideally somewhat more.
> I prefer KVM over IP. Yes, it is expensive.
> Rariton KX232's are about 6000-8000$US/unit these days, and the CIMs that you
> attach to the machines are about $100/unit. 128 CIMs will run you 12k$. And
> will require 4 of these Raritan units. The Raritans also have an IE-only
> (grrr!!!) interface. Some others are saner and use either a java applet, or
> some sort of portable console. I have heard a rumor that someone uses vnc,
> but have not looked hard enough to find out if this is true.
> Combining these two technologies will get you that level of control.
My impression is that the a serial port terminal server
would be more cost effective, IF it works.
> That said, some of the higher end vendors management processors integrate
> all/most of these things. It would be overkill to get one of those units
> with the aforementioned power/kvm technologies.
Actually I am surprised that IPMI 2.0 doesn't already do ALL of what I
want. The main issue seems to be this one: where is the serial console
> Raritan also integrates power control if you prefer single vendors.
I didn't know about Raritan. Just had a look on the web. They look to be
about $150 - $200 per node in quantity one. So they might be reasonable
in quantity fifteen or so.
How can they do power control (for a locked system) with just Keyboard,
Video and Mouse connectors?
> Bruce Allen wrote:
>> We're getting ready to put together our next large Linux compute cluster.
>> This time around, we'd like to be able to interact with the machines
>> remotely. By this I mean that if a machine is locked up, we'd like to be
>> able to see what's on the console, power cycle it, mess with BIOS settings,
>> and so on, WITHOUT having to drive to work, go into the cluster room, etc.
>> One possible solution is to buy nodes that have IPMI cards. These
>> piggyback on the ethernet LAN and let you interact with the machine even in
>> the absence of an OS. With the appropriate tools running on a remote
>> machine, you can interact with the nodes even if they have no OS on them or
>> are hung.
>> Another solution is to use the DB9 serial ports of the nodes. You have an
>> 'administrative' box containing lots of high-port-count serial cards (eg,
>> Cyclades 32 or 64 port cards) and then run a serial cable from each node to
>> this box. By remotely logging into this admin box you can access the
>> serial ports of the machines, and if the BIOS has the right
>> settings/support, this lets you have keyboard/console access.
>> Or one can do both IPMI + remote serial port access.
>> Could people on this list please report their experiences with these or
>> other approaches? In particular, does someone have a simple and
>> inexpensive solution (say < $100/node) which lets them remotely:
>> - power cycle a machine
>> - examine/set BIOS values
>> - look at console output even for a dead/locked/unresponsive box
>> - ???
>> Bruce Allen
>> U. of Wisconsin - Milwaukee Physics Department
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web : http://www.scalableinformatics.com
> phone: +1 734 786 8423
> fax : +1 734 786 8452
> cell : +1 734 612 4615
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf