Click on our Sponsors to help Support SunWorld
Sysadmin by Hal Stern

How you can make the network yente work for you

Need to find the IP address of a network node? Let ARP be your matchmaker

March  1997
[Next story]
[Table of Contents]
Subscribe to SunWorld, it's free!

This month, we'll give you the network handy-person's guide to ARP, starting with an overview of how the protocol fits into the TCP/IP world. We'll look at how Solaris manages the ARP protocol and associated caches, and see how ARP deals with routers and other devices that don't pass along its gentle probes for data. We'll conclude with a list of symptoms and fixes for common ARP-related woes. (2,800 words)

Mail this
article to
a friend

In the Broadway show and movie "Fiddler On The Roof," five young women turn to Yente, the arranger of marriages in their small Russian town. Yente's job is that of the matchmaker, finding the best fit for a young lady, or at least a job close enough to call complete. Are we crass enough to take this slice of classic show business and turn it into a network analogy? Right -- of course right!

Whether you realize it or not, the address resolution protocol (ARP) is hard at work constantly matching IP addresses to their Ethernet hardware addresses on your network. When ARP is working well, packets flow in both directions, across routers and bridges without a moment's notice. If you're suffering from ARP difficulties, you'll see intermittent grand pauses in network traffic, reductions in usable network bandwidth, and generally poor performance for various network applications.

Like Yente, ARP has to deal with liars, two- and three-timers, and other less-than-trusty network citizens like those middlemen we usually call routers. Figuring out who's telling the truth, who needs a sharp reprimand, and what's going wrong is, of course, your job. Right? Of course right!

A many-to-one splendored thing: Mapping IP addresses
First of all, it's permissable to sound out the protocol acronym, rhyming it with "harp," even if you do sound like Network the dog with a bit error in his bark. ARP can also be used as a verb as in, "Who is arping for that address?" It's not part of the IP protocol per se but instead is a peer protocol that is nearly always included in an implementation of TCP/IP. ARP is described in gory detail in RFC 826.

ARP's job appears simple on the surface: it finds the hardware or MAC (media access control) address corresponding to a given IP address. ARP builds up relations of 32-bit IP addresses into the 48-bit MAC address space, maintaining these many-to-one mappings on a per-host basis. Each machine that runs TCP/IP needs to track the mappings constructed up by ARP traffic, building its own map of the local network. The fact that ARP management represents the ultimate in distributed state is what also makes it prone to strange failure modes.


A bit of background on MAC addresses is in order before explaining ARP's tasks. A hardware, or MAC, address is assigned per host - one per system, no matter how many network interfaces it has. The MAC address uniquely identifies the system in the "universe" of Ethernet devices. MAC addresses are written as a series of 6 colon-separated hex values, for example, 8:0:20:74:b:6. The upper 24 bits are reserved for a vendor ID and the lower 24 bits represent a unique sequence number within that vendor's space. You'll find all of your Sun systems have MAC addresses that start with 8:0:20. RFC XXX, Assigned Numbers, provides a listing of MAC address prefixes and vendors, allowing you to match up a network packet and the flavor of machine it came from. You'll also see that Sun assigns one MAC address per system, burned into the ID or EEPROM of the system, a fact that startles most people adding additional network interfaces.

The IP-MAC relationship is N:1 when a given host has IP addresses on multiple network interfaces but uses the default MAC address on each one. For the most part, a given MAC address maps back to one IP address per network -- so a network client can maintain a 1:1 table of IP addresses and associated hardware addresses. Looking at it another way, IP addresses are logical conventions, assigned by system administrators to make the TCP/IP protocol work. MAC addresses are physical conventions, assigned by the vendor; ARP's job is to bridge the logical world onto the physical one.

Who are you? ARP in action
Here's how a typical ARP request and reply play out:

Not all network protocol suites require ARP: DECnet uses an algorithm to convert DECnet area:node addresses into Ethernet MAC addresses. This requires the DECnet protocol to change the local MAC address to match the DECnet configuration, a change that can confuse bridges if DECnet is enabled after the node has already spoken TCP/IP with its neighbors. ARP traffic may seem like an unreasonable bit of overhead added to the startup of each conversation with a new host. It's the price paid for flexibility -- you can assign IP addresses as you like to your hosts, even re-using the same addresses on disjoint networks, without upsetting the inner workings of the Ethernet one bit.

Digging data out of ARP
ARP maintains its cache of IP:MAC mappings on each machine. Dump it out using arp -a:

luey% arp -a
Net to Media Table
Device   IP Address               Mask      Flags   Phys Addr 
------ -------------------- --------------- ----- ---------------
le0    duey              08:00:20:1d:0b:92
le0 U     
le0    huey        SP    08:00:20:21:15:13
le0    fred              08:00:20:1a:f2:02
le0         SM    01:00:5e:00:00:00

You can get a slightly better report from ndd:

luey% ndd -get /dev/arp arp_cache_report
ifname   proto addr      proto mask      hardware addr     flags
le0 08:00:20:1f:34:9b
le0 08:00:20:04:cd:1e
le0 08:00:20:18:1c:e5
le0 08:00:20:11:27:af
le0 08:00:20:7a:5c:2c
le0 08:00:20:21:15:13 PERM PUBLISH
le0 08:00:20:1a:f2:02
le0 08:00:20:1e:26:5c
le0 01:00:5e:00:00:00 PERM MAPPING

Completed entries represent queries that returned at least one reply. Those marked "U" or "UNRESOLVED" are from queries that were never answered, because the IP address is not on the local network or because the host owning it was down or not listening at the moment the request was sent.

Entries marked "S" (static) or "PERM" will never be removed from the cache; typically these are inserted on behalf of devices (like printers) that have limited TCP/IP implementations and don't answer their ARP requests, or to work around network configuration quirks. Create a permanent entry using arp as root:

luey# arp -s 06:1:4:bc:a:34 

A "published" entry is one for which the local host acts as a proxy server. When the local host sees an ARP request come by for that IP address, it answers with the MAC address in the cache, not its own MAC address. Publishing ARP entries allows one machine to answer ARP requests for machines that can't hear the broadcasts because they're on the far side of a router, or because they're not listening. Published entries are marked with a "P" or "PUBLISH" in the arp cache report. Load ARP information from a table using arp -f to publish several ARP addresses at once.

Using permanent entries prevents the local machine from sending ARP requests; creating a published entry allows the local machine to answer ARP requests coming from other machines, on behalf of hosts that can't handle their own ARP traffic. We'll discuss proxy ARP configurations and the networks that require them next month.

ARP ages entries in the cache. An entry that is not completed within 5 minutes is removed, and entries not in use are flushed every 20 minutes. These timeouts are governed by the variables ip_ire_cleanup_interval and ip_ire_flush_interval, settable with the ndd interface:

luey# ndd -set /dev/ip ip_ire_flush_interval 600000

Note that the values are in milliseconds.

Given that ARP is relatively simple, it seems it should have few failure modes. However, it sits at the heart of each nascent TCP/IP connection, and flakey ARP behavior is the root cause of many network dog-days. The first thing that can go wrong is that your router or switch gets upset with a multi-homed machine's ARP replies.

Love me two times: Multi-homed hosts and MAC addresses
How and why might you want to change your own ARP information? If you're connecting multiple network interfaces to a switch or hub that wants to see a unique MAC address on each port, you'll need to modify the default Sun configuration. Start by dumping out the MAC address for each interface using ifconfig -a:

luey# ifconfig -a
lo0: flags=849 mtu 8232
	inet netmask ff000000 
qe0: flags=863 mtu 1500
	inet netmask ffffff00 broadcast
	ether 8:0:20:21:15:13 
qe1: flags=863 mtu 1500
	inet netmask ffffff00 broadcast
	ether 8:0:20:21:15:13 

You'll notice that the same Ethernet address shows up for every interface, creating an N:1 mapping for that machine that ARP manages just fine on multiple networks. When you connect those interfaces to the same network, for example, multiple ports of a switch - to gain improved input bandwidth to the server using 2 IP addresses on dedicated switch segments - the switch is likely to be confused by seeing the same MAC address show up on multiple ports.

The easiest way to fix this behavior is to assign each interface its own MAC address. The "ether" parameter to ifconfig allows you to set the MAC address on a per-interface basis. One easy configuration is to add a digit to the first hex value in the default MAC address:

luey# ifconfig qe1 duey-qe1 ether 18:0:20:4:56:7
luey# ifconfig qe2 duey-qe2 ether 28:0:20:4:56:7
luey# ifconfig qe3 duey-qe2 ether 38:0:20:4:56:7

Be careful not to set the last bit in the first value in the MAC address, as this bit is reserved for indicating multicast addresses. For example, 18:0:20:x:x:x is a valid station address but 9:0:20:x:x:x is a multicast address.

Wary the scary middleman: ARP and routers
Dealing with single hosts and switches is relatively simple. How does ARP play when there's a router involved? ARP is the magic glue that guides the packet from the sender to the router without cheating on the value of the real, destination IP address. Here's how it works:

There are two rules to be taken from this quick look at packet routing. First, never try to associate MAC addresses and IP addresses merely from the contents of an average, TCP or UDP packet. The only real answers come from ARP replies. Traffic that is incident to a router will have the router's MAC address but the real, final destination's IP address, confusing you if you want to build IP:MAC mappings that way. Second, keeping a full and useful ARP cache reduces network latency by taking an extra round-trip request-reply pair out of the connection set up. We'll wrap up with some tips on keeping ARP running smoothly and detecting trouble spots when it's not.

ARPing up the wrong tree: Dealing with trouble in Network doggie land
ARP can produce a whole host of problems that create grand pauses or performance bottlenecks across an entire network:

ARP is one of the protocols you'd like to learn about and forget, but each time you have a puzzling network performance problem that can't be explained by application or system software design, make sure your network traffic is flowing to the right places. Everybody loves the network that does as its supposed to, quietly and unevently, like a good doggie companion.

Click on our Sponsors to help Support SunWorld


What did you think of this article?
-Very worth reading
-Worth reading
-Not worth reading
-Too long
-Just right
-Too short
-Too technical
-Just right
-Not technical enough

[Table of Contents]
Subscribe to SunWorld, it's free!
[Next story]
Sun's Site

[(c) Copyright  Web Publishing Inc., and IDG Communication company]

If you have technical problems with this magazine, contact

Last modified: