[Beowulf] Remote procedure call (error)

Robert G. Brown rgb at phy.duke.edu
Wed Jan 28 11:29:18 EST 2004

On Tue, 27 Jan 2004, Alvin Oga wrote:

> hi ya robert
> On Wed, 28 Jan 2004, Robert G. Brown wrote:
> > rpcinfo is also your friend:
> > 
> > rgb at lilith|T:3#rpcinfo -p lilith
> >    program vers proto   port
> >     100000    2   tcp    111  portmapper
> >     100000    2   udp    111  portmapper
> >     100024    1   udp  32768  status
> >     100024    1   tcp  32768  status
> >     391002    2   tcp  32769  sgi_fam
> > 
> > As a general rule, I'd consider the portmapper and rpc stuff to be a
> > moderate security risk; lilith lives inside firewalls only.  I have had
> > direct experience of systems cracked in the past through portmapper
> > bugs, which perhaps makes me a bit paranoid.
> did you run the secure versions of each ??
> 	secure rpc
> 	secure portmap
> 	secure nfs

This was a few years ago.  Remember, I've been a Unix sysadmin since,
oh, 1986 or 1987 or thereabouts.  As in I remember the Morris worm, a
slew of sendmail and lpd attacks, an emacs bug, syslogd attacks, the
good old days where any user armed with a userspace nfs kit could access
any user's NFS space, and much, much, more.

The portmapper exploit was particularly annoying and was one of the
things (to remain vaguely on topic:-) that propelled us from Slackware
to Red Hat as a departmental base -- our previous head sysadmin (by this
time I was a "part time" sysadmin of our department network and our
primary sysadmin kept pretty tight control of the LAN so I basically did
ver little actual admin work) had left our department, and his first
replacement turned out to be a midlevel disaster and permitted
sigificant spread in the actual set of installed packages on all the
various systems in the department.  As in there were probably three or
four different revisions and patchlevels of portmap running on different
systems.  Well, THAT sysadmin went away, I found myself once again in
charge of the network literally overnight until we hired a replacement,
and almost the first thing that happened was that about three or four
departmental systems turned out to be vulnerable to a portmap buffer
overwrite attack, were exploited, and I spent week of my life cleaning

This is a salutary lesson to the novices out there; to quote from A
Thousand Nights and a Night (various places) "it should be graven on the
eyelid corners as a warner to whomso would be warned".  It is pretty
much essential to: 

  a) keep systems patched and up to date, especially wrt security fixes
but of course performance and bugfixes are also important.

  b) keep all systems in a LAN patched at the SAME level -- all it takes
is one bad system and your whole network is compromised, as many
attackers who get a user account can find a way to promote to root, and
a user who gets root on one system in a LAN has many routes to
eventually promote to root on the department servers and LAN.
Heterogeneity is an enemy of good systems administration, as it dilutes
effort and limits scalability.

  c) In case it isn't obvious, this suggests that sound management
practice includes something like: pick a packaged distribution that is
aggressively maintained, especially wrt security; use kickstart or other
scripting/imaging tools to construct system templates so that all your
systems will be as similar as possible (same packages, same general
configuration); use yum or related tools in other distributions to fully
automate system updates from a common package repository, so that
keeping the package repository up to date is precisely equivalent to
keeping all your systems up to date.

  d) Minimize exposure.  This means that unless you need a daemon or
service, don't run it.  Buffer overwrite attacks via "essential" systems
daemons have probably been the single most common avenue of successful
attack in the history of the non-Windows internet, and they have
potentially disasterous scaling properties as the Morris worm (and more
contemporaneously, the Slammer) clearly demonstrated.  Worms like this
might be classified as dangerous terrorist munitions, as they can cause
more economic damage in a day than a hundred car bombs, especially with
the Internet at this point being the base of a huge amount of economic

  e) Since there are always daemons and services that are more or less
REQUIRED inside a LAN in order not to lose productivity THAT way run
some sort of firewall to isolate access to the exposed ports to only the
hosts that need the access.  It is relatively easy to police these hosts
and their users with tool-enhanced vigilance (syslog-ng etc) and a
sucker rod (read man syslogd if the reference escapes you:-) to school
any local users who want to try any 1337 crap or mad skills).  iptables
or ipchains permit one remarkably fine-grained control of port access
rights even within a LAN -- every host runs its own mini-firewall.  NFS
in particular and most RPC services are very definition, BTW, of
services that should have carefully regulated access.  NFS should pretty
much NEVER be used over a WAN (this may have changed, of course, but Old
Guys will always be paranoid and not really believe that it has:-).

  f) A lot of folks organize their internal LAN in security layers, with
highly restricted access to servers, very limited privileges to clients
(to make it more difficult for a successful exploit of a client to be
promotable to the entire LAN and especially to the servers, ESPECIALLY
to install repositories).  There is no substitute for vigilance, though
-- most exploits are discovered by admins or users noting and reporting
something "odd" or something "broken" and following up aggressively.
Again kickstart/yum/PXE are your friend, as cleanup nowadays is just
Ctrl-Alt-Del into a full reinstall of a compromised client, coupled with
a scrubbing of any affected user accounts.

  g) Keep Good Backups.  With highly automated install/maintenance
tools, cleanup is relatively cheap, but you may well have to restore
corrupted files or home directories or (in the worst of cases) rebuild
servers or repositories.

Security is always a trade-off; you give up the ability to do certain
kinds of work easily and quickly in order to achieve a "hardened"
status, but the lost work and productivity is also a "cost", one that
accrues linearly in time, and can easily exceed the "cost" of a
successful exploit.  Many linuces now install by default into what
amounts to a tuckee-tail hard firewalled mode with access to virtually
all ports blocked via iptables and turned off in chkconfig.  This is a
good thing, but it can mean that lots of things "mysteriously" don't
work unless you know what your are doing and know how to enable them.
Which is ALSO a good thing; if you don't know how to turn them on, could
you possibly know enough to be vigilant and keep a system with them
enabled secure?  Learning the one should always be accompanied by
learning the other...

> at "*.duke.edu" guess you folks would be a prime target...

I think that everybody is a prime target.  Many, even most, exploits
these days are driven by fully automated engines; there are relatively
few incidents of pimply-faced cracker script-kiddies targeting systems
one at a time (working through the usual string of intermediary
successes).  Even the script-kids use tools that simply sweep IP spaces
probing for open/vulnerable ports, and the worms combine a successful
sweep with exploitation of the port, implanting of a clone of the worm,
and a continuation of propagation process.

At Duke, at least, we have a VERY competent security officer (Chris
Cramer) who does a damn fine job coping with and minimizing our exposure
on an institutional basis.  He's a beowulfer, BTW, and might even read
(eventually) these words of glowing praise and wonder what the hell I
want out of him now:-).  (Lunch, Chris, Lunch.  Love those free
lunches:-) He is backed up by an equally excellent collective sysadmin
group; they keep mot of the organized LAN spaces professionally secured
without his constant browbeating so he can focus on the bleeding,
supperating wound that is Windows.  

Today is of course not a good day for him on that front as YAWE (yet
another windows exploit) got enough traction yesterday that answering my
email was distinctly tedious (note that I do NOT use the word Hi in my
subject line above, not wishing to be procmailed out by others in the
same pickle).  I'm certain that it is popping up in the dorms if nowhere
else, as we obviously have a hard time controlling "private" student
systems in the dorms, who are both "inside" and "outside" Duke's trusted
space (as outside as we can make them, but nevertheless possessing
trusted access to certain resources). So sure, Duke is a major target by
virtue of its large IP-space volume if nothing else, but it is also
fairly aggressively and competently policed.


> c ya
> alvin

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list